分组聚合、过滤、转换

简书大神SeanCheney的译作，我作了些格式调整和文章目录结构的变化，更适合自己阅读，以后翻阅是更加方便自己查找吧

import pandas as pd
import numpy as np

设定最大列数和最大行数

pd.set_option(‘max_columns‘,8 , ‘max_rows‘, 8)

1 聚合

读取flights数据集，查询头部

flights = pd.read_csv(‘data/flights.csv‘)

flights.head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

5 rows × 14 columns

1.1 单列聚合

按照AIRLINE分组，使用agg方法，传入要聚合的列和聚合函数

flights.groupby(‘AIRLINE‘).agg({‘ARR_DELAY‘:‘mean‘}).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

或者要选取的列使用索引，聚合函数作为字符串传入agg

flights.groupby(‘AIRLINE‘)[‘ARR_DELAY‘].agg(‘mean‘).head()

AIRLINE
AA    5.542661
AS   -0.833333
B6    8.692593
DL    0.339691
EV    7.034580
Name: ARR_DELAY, dtype: float64

也可以向agg中传入NumPy的mean函数

flights.groupby(‘AIRLINE‘)[‘ARR_DELAY‘].agg(np.mean).head()

AIRLINE
AA    5.542661
AS   -0.833333
B6    8.692593
DL    0.339691
EV    7.034580
Name: ARR_DELAY, dtype: float64

也可以直接使用mean()函数

flights.groupby(‘AIRLINE‘)[‘ARR_DELAY‘].mean().head()

AIRLINE
AA    5.542661
AS   -0.833333
B6    8.692593
DL    0.339691
EV    7.034580
Name: ARR_DELAY, dtype: float64

1.2 多列聚合

每家航空公司每周平均每天取消的航班数

flights.groupby([‘AIRLINE‘, ‘WEEKDAY‘])[‘CANCELLED‘].agg(‘sum‘).head(7)

AIRLINE  WEEKDAY
AA       1          41
         2           9
         3          16
         4          20
         5          18
         6          21
         7          29
Name: CANCELLED, dtype: int64

分组可以是多个

选取可以是多组

聚合函数也可以是多个

每周每家航空公司取消或改变航线的航班总数和比例

flights.groupby([‘AIRLINE‘, ‘WEEKDAY‘])[‘CANCELLED‘, ‘DIVERTED‘].agg([‘sum‘, ‘mean‘]).head(7)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead tr th { text-align: left }
.dataframe thead tr:last-of-type th { text-align: right }

用列表和嵌套字典对多列分组和聚合

对于每条航线，找到总航班数，取消的数量和比例，飞行时间的平均时间和方差

group_cols = [‘ORG_AIR‘, ‘DEST_AIR‘]

agg_dict = {‘CANCELLED‘:[‘sum‘, ‘mean‘, ‘size‘],
            ‘AIR_TIME‘:[‘mean‘, ‘var‘]}

flights.groupby(group_cols).agg(agg_dict).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead tr th { text-align: left }
.dataframe thead tr:last-of-type th { text-align: right }

1.3 DataFrameGroupBy对象

groupby方法产生的是一个DataFrameGroupBy对象

college = pd.read_csv(‘data/college.csv‘)
grouped = college.groupby([‘STABBR‘, ‘RELAFFIL‘])

查看分组对象的类型

type(grouped)

pandas.core.groupby.groupby.DataFrameGroupBy

用dir函数找到该对象所有的可用函数

print([attr for attr in dir(grouped) if not attr.startswith(‘_‘)])

[‘CITY‘, ‘CURROPER‘, ‘DISTANCEONLY‘, ‘GRAD_DEBT_MDN_SUPP‘, ‘HBCU‘, ‘INSTNM‘, ‘MD_EARN_WNE_P10‘, ‘MENONLY‘, ‘PCTFLOAN‘, ‘PCTPELL‘, ‘PPTUG_EF‘, ‘RELAFFIL‘, ‘SATMTMID‘, ‘SATVRMID‘, ‘STABBR‘, ‘UG25ABV‘, ‘UGDS‘, ‘UGDS_2MOR‘, ‘UGDS_AIAN‘, ‘UGDS_ASIAN‘, ‘UGDS_BLACK‘, ‘UGDS_HISP‘, ‘UGDS_NHPI‘, ‘UGDS_NRA‘, ‘UGDS_UNKN‘, ‘UGDS_WHITE‘, ‘WOMENONLY‘, ‘agg‘, ‘aggregate‘, ‘all‘, ‘any‘, ‘apply‘, ‘backfill‘, ‘bfill‘, ‘boxplot‘, ‘corr‘, ‘corrwith‘, ‘count‘, ‘cov‘, ‘cumcount‘, ‘cummax‘, ‘cummin‘, ‘cumprod‘, ‘cumsum‘, ‘describe‘, ‘diff‘, ‘dtypes‘, ‘expanding‘, ‘ffill‘, ‘fillna‘, ‘filter‘, ‘first‘, ‘get_group‘, ‘groups‘, ‘head‘, ‘hist‘, ‘idxmax‘, ‘idxmin‘, ‘indices‘, ‘last‘, ‘mad‘, ‘max‘, ‘mean‘, ‘median‘, ‘min‘, ‘ndim‘, ‘ngroup‘, ‘ngroups‘, ‘nth‘, ‘nunique‘, ‘ohlc‘, ‘pad‘, ‘pct_change‘, ‘pipe‘, ‘plot‘, ‘prod‘, ‘quantile‘, ‘rank‘, ‘resample‘, ‘rolling‘, ‘sem‘, ‘shift‘, ‘size‘, ‘skew‘, ‘std‘, ‘sum‘, ‘tail‘, ‘take‘, ‘transform‘, ‘tshift‘, ‘var‘]

用ngroups属性查看分组的数量

grouped.ngroups

112

查看每个分组的唯一识别标签

groups属性是一个字典，包含每个独立分组与行索引标签的对应

groups = list(grouped.groups.keys())
groups[:6]

[(‘AK‘, 0), (‘AK‘, 1), (‘AL‘, 0), (‘AL‘, 1), (‘AR‘, 0), (‘AR‘, 1)]

用get_group，传入分组标签的元组

例如，获取佛罗里达州所有与宗教相关的学校

grouped.get_group((‘FL‘, 1)).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

5 rows × 27 columns

groupby对象是一个可迭代对象，可以挨个查看每个独立分组

i = 0
for name, group in grouped:
    print(name)
    display(group.head(2))
    i += 1
    if i == 5:
     break

(‘AK‘, 0)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

2 rows × 27 columns

(‘AK‘, 1)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

2 rows × 27 columns

(‘AL‘, 0)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

2 rows × 27 columns

(‘AL‘, 1)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

2 rows × 27 columns

(‘AR‘, 0)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

2 rows × 27 columns

groupby对象使用head方法，可以在一个DataFrame钟显示每个分组的头几行

grouped.head(2).head(6)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

6 rows × 27 columns

nth方法可以选出每个分组指定行的数据，下面选出的是第1行和最后1行

grouped.nth([1, -1]).head(8)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

8 rows × 25 columns

2 聚合函数

college = pd.read_csv(‘data/college.csv‘)
college.head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

5 rows × 27 columns

2.1 自定义聚合函数

求出每个州的本科生的平均值和标准差

college.groupby(‘STABBR‘)[‘UGDS‘].agg([‘mean‘, ‘std‘]).round(0).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

远离平均值的标准差的最大个数，写一个自定义函数

def max_deviation(s):
    std_score = (s - s.mean()) / s.std()
    return std_score.abs().max()

agg聚合函数在调用方法时，直接引入自定义的函数名

college.groupby(‘STABBR‘)[‘UGDS‘].agg(max_deviation).round(1).head()

STABBR
AK    2.6
AL    5.8
AR    6.3
AS    NaN
AZ    9.9
Name: UGDS, dtype: float64

自定义的聚合函数也适用于多个数值列

college.groupby(‘STABBR‘)[‘UGDS‘, ‘SATVRMID‘, ‘SATMTMID‘].agg(max_deviation).round(1).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

自定义聚合函数也可以和预先定义的函数一起使用

college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘, ‘SATVRMID‘, ‘SATMTMID‘].agg([max_deviation, ‘mean‘, ‘std‘]).round(1).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead tr th { text-align: left }
.dataframe thead tr:last-of-type th { text-align: right }

5 rows × 9 columns

Pandas使用函数名作为返回列的名字；你可以直接使用rename方法修改，或通过__name__属性修改

max_deviation.__name__

‘max_deviation‘

max_deviation.__name__ = ‘Max Deviation‘

college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘, ‘SATVRMID‘, ‘SATMTMID‘]    .agg([max_deviation, ‘mean‘, ‘std‘]).round(1).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead tr th { text-align: left }
.dataframe thead tr:last-of-type th { text-align: right }

5 rows × 9 columns

2.2 用 *args 和 **kwargs 自定义聚合函数

自定义一个返回去本科生人数在1000和3000之间的比例的函数

def pct_between_1_3k(s):
    return s.between(1000, 3000).mean()

用州和宗教分组，再聚合

college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘].agg(pct_between_1_3k).head(9)

STABBR  RELAFFIL
AK      0           0.142857
        1           0.000000
AL      0           0.236111
        1           0.333333
                      ...
AR      1           0.111111
AS      0           1.000000
AZ      0           0.096774
        1           0.000000
Name: UGDS, Length: 9, dtype: float64

但是这个函数不能让用户自定义上下限，再新写一个函数

def pct_between(s, low, high):
    return s.between(low, high).mean()

使用这个自定义聚合函数，并传入最大和最小值

college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘].agg(pct_between, 1000, 10000).head(9)

STABBR  RELAFFIL
AK      0           0.428571
        1           0.000000
AL      0           0.458333
        1           0.375000
                      ...
AR      1           0.166667
AS      0           1.000000
AZ      0           0.233871
        1           0.111111
Name: UGDS, Length: 9, dtype: float64

显示指定最大和最小值

college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘].agg(pct_between, high=10000, low=1000).head(9)

STABBR  RELAFFIL
AK      0           0.428571
        1           0.000000
AL      0           0.458333
        1           0.375000
                      ...
AR      1           0.166667
AS      0           1.000000
AZ      0           0.233871
        1           0.111111
Name: UGDS, Length: 9, dtype: float64

也可以关键字参数和非关键字参数混合使用，只要非关键字参数在后面

college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘].agg(pct_between, 1000, high=10000).head(9)

STABBR  RELAFFIL
AK      0           0.428571
        1           0.000000
AL      0           0.458333
        1           0.375000
                      ...
AR      1           0.166667
AS      0           1.000000
AZ      0           0.233871
        1           0.111111
Name: UGDS, Length: 9, dtype: float64

Pandas不支持多重聚合时，使用参数

用闭包自定义聚合函数

def make_agg_func(func, name, *args, **kwargs):
     def wrapper(x):
         return func(x, *args, **kwargs)
     wrapper.__name__ = name
     return wrapper

my_agg1 = make_agg_func(pct_between, ‘pct_1_3k‘, low=1000, high=3000)

college.groupby([‘STABBR‘, ‘RELAFFIL‘])[‘UGDS‘].agg([my_agg1,make_agg_func(pct_between, ‘pct_10_30k‘, 10000, 30000)])

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

112 rows × 2 columns

3 聚合后去除多级索引

读取数据

flights = pd.read_csv(‘data/flights.csv‘)

flights.head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

5 rows × 14 columns

按‘AIRLINE‘, ‘WEEKDAY‘分组，分别对DIST和ARR_DELAY聚合

airline_info = flights.groupby([‘AIRLINE‘, ‘WEEKDAY‘])    .agg({‘DIST‘:[‘sum‘, ‘mean‘],‘ARR_DELAY‘:[‘min‘, ‘max‘]})    .astype(int)
airline_info.head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead tr th { text-align: left }
.dataframe thead tr:last-of-type th { text-align: right }

行和列都有两级索引

3.1 拼接列索引

get_level_values(0)取出第一级索引

level0 = airline_info.columns.get_level_values(0)

get_level_values(1)取出第二级索引

level1 = airline_info.columns.get_level_values(1)

一级和二级索引拼接成新的列索引

airline_info.columns = level0 + ‘_‘ + level1

airline_info.head(7)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

3.2 重置行索引

reset_index()可以将行索引变成单级

airline_info.reset_index().head(7)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

Pandas默认会在分组运算后，将所有分组的列放在索引中，as_index设为False可以避免这么做。

分组后使用reset_index，也可以达到同样的效果

flights.groupby([‘AIRLINE‘], as_index=False)[‘DIST‘].agg(‘mean‘).round(0)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

14 rows × 2 columns

4 过滤聚合

college = pd.read_csv(‘data/college.csv‘, index_col=‘INSTNM‘)
grouped = college.groupby(‘STABBR‘)
grouped.ngroups

59

这等于求出不同州的个数，nunique()可以得到同样的结果

college[‘STABBR‘].nunique()

59

自定义一个计算少数民族学生总比例的函数，如果比例大于阈值，还返回True

def check_minority(df, threshold):
    minority_pct = 1 - df[‘UGDS_WHITE‘]
    total_minority = (df[‘UGDS‘] * minority_pct).sum()
    total_ugds = df[‘UGDS‘].sum()
    total_minority_pct = total_minority / total_ugds
    return total_minority_pct > threshold

grouped变量有一个filter方法，可以接收一个自定义函数，决定是否保留一个分组

college_filtered = grouped.filter(check_minority, threshold=.5)

college_filtered.head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

5 rows × 26 columns

通过查看形状，可以看到过滤了60%，只有20个州的少数学生占据多数

college.shape

(7535, 26)

college_filtered.shape

(3028, 26)

college_filtered[‘STABBR‘].nunique()

20

用一些不同的阈值，检查形状和不同州的个数

college_filtered_20 = grouped.filter(check_minority, threshold=.2)

college_filtered_20.shape,college_filtered_20[‘STABBR‘].nunique()

((7461, 26), 57)

college_filtered_70 = grouped.filter(check_minority, threshold=.7)

college_filtered_70.shape,college_filtered_70[‘STABBR‘].nunique()

((957, 26), 10)

college_filtered_95 = grouped.filter(check_minority, threshold=.95)

college_filtered_95.shape,college_filtered_95[‘STABBR‘].nunique()

((156, 26), 7)

5 apply函数

apply函数是pandas里面所有函数中自由度最高的函数

读取college，‘UGDS‘, ‘SATMTMID‘, ‘SATVRMID‘三列如果有缺失值则删除行

college = pd.read_csv(‘data/college.csv‘)
subset = [‘UGDS‘, ‘SATMTMID‘, ‘SATVRMID‘]
college2 = college.dropna(subset=subset)
college.shape,college2.shape

((7535, 27), (1184, 27))

5.1 apply与agg

自定义一个求SAT数学成绩的加权平均值的函数

def weighted_math_average(df):
     weighted_math = df[‘UGDS‘] * df[‘SATMTMID‘]
     return int(weighted_math.sum() / df[‘UGDS‘].sum())

5.1.1 apply应用聚合函数

按州分组，并调用apply方法，传入自定义函数

college2.groupby(‘STABBR‘).apply(weighted_math_average).head()

STABBR
AK    503
AL    536
AR    529
AZ    569
CA    564
dtype: int64

5.1.2 agg应用聚合函数

college2.groupby(‘STABBR‘).agg(weighted_math_average).head()

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

5 rows × 26 columns

如果将列限制到SATMTMID，会报错。这是因为不能访问UGDS。

# college2.groupby(‘STABBR‘)[‘SATMTMID‘].agg(weighted_math_average)

5.2 apply创建新列

apply的一个不错的功能是通过返回Series，创建多个新的列

from collections import OrderedDict
def weighted_average(df):
    data = OrderedDict()
    weight_m = df[‘UGDS‘] * df[‘SATMTMID‘]
    weight_v = df[‘UGDS‘] * df[‘SATVRMID‘]

    data[‘weighted_math_avg‘] = weight_m.sum() / df[‘UGDS‘].sum()
    data[‘weighted_verbal_avg‘] = weight_v.sum() / df[‘UGDS‘].sum()
    data[‘math_avg‘] = df[‘SATMTMID‘].mean()
    data[‘verbal_avg‘] = df[‘SATVRMID‘].mean()
    data[‘count‘] = len(df)
    return pd.Series(data, dtype=‘int‘)

college2.groupby(‘STABBR‘).apply(weighted_average).head(10)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

10 rows × 5 columns

5.3 apply创建dataframe

自定义一个返回DataFrame的函数

使用NumPy的函数average计算加权平均值，使用SciPy的gmean和hmean计算几何和调和平均值

from scipy.stats import gmean, hmean
def calculate_means(df):
    df_means = pd.DataFrame(index=[‘Arithmetic‘, ‘Weighted‘, ‘Geometric‘, ‘Harmonic‘])
    cols = [‘SATMTMID‘, ‘SATVRMID‘]
    for col in cols:
        arithmetic = df[col].mean()
        weighted = np.average(df[col], weights=df[‘UGDS‘])
        geometric = gmean(df[col])
        harmonic = hmean(df[col])
        df_means[col] = [arithmetic, weighted, geometric, harmonic]
    df_means[‘count‘] = len(df)
    return df_means.astype(int)

college2.groupby(‘STABBR‘)    .filter(lambda x: len(x) != 1)    .groupby(‘STABBR‘)    .apply(calculate_means).head(10)

.dataframe tbody tr th:only-of-type { vertical-align: middle }
.dataframe tbody tr th { vertical-align: top }
.dataframe thead th { text-align: right }

10 rows × 3 columns

原文地址：https://www.cnblogs.com/shiyushiyu/p/9757766.html