问题描述
我有一个参考名单
ref = ['September', 'August', 'July', 'June', 'May', 'April', 'March']
还有一个数据框
df = pd.DataFrame({'Month_List': [['July'], ['August'], ['July', 'June'], ['May', 'April', 'March']]})
df
Month_List
0 [July]
1 [August]
2 [July, June]
3 [May, April, March]
我想检查引用列表中的哪些元素存在于每一行中,然后转换为二进制列表
I want to check which elements from reference list is present in each row, and convert into binary list
我可以使用apply
def convert_month_to_binary(ref,lst):
s = pd.Series(ref)
return s.isin(lst).astype(int).tolist()
df['Binary_Month_List'] = df['Month_List'].apply(lambda x: convert_month_to_binary(ref, x))
df
Month_List Binary_Month_List
0 [July] [0, 0, 1, 0, 0, 0, 0]
1 [August] [0, 1, 0, 0, 0, 0, 0]
2 [July, June] [0, 0, 1, 1, 0, 0, 0]
3 [May, April, March] [0, 0, 0, 0, 1, 1, 1]
但是,在大型数据集上使用apply
非常慢,因此我希望使用numpy向量化.如何改善我的表现?
However, using apply
on large datasets is very slow and hence I am looking to use numpy vectorization. How can I improve my performance?
扩展名:
我想使用numpy vectorization
,因为我现在需要在此列表上应用另一个功能
I wanted to use numpy vectorization
because I need to now apply another function on this list
我正在尝试这样,但是性能非常慢.与apply
I am trying like this, but performance is very slow. Similar results with apply
def count_one(lst):
index = [i for i, e in enumerate(lst) if e != 0]
return len(index)
vfunc = np.vectorize(count_one)
df['Value'] = vfunc(df['Binary_Month_List'])
推荐答案
在熊猫中更好不要这样使用list
,但可以使用 MultiLabelBinarizer
a>和 DataFrame.reindex
对于添加的缺少类别,最后将值转换为numpy数组,如果性能很重要,则将其转换为list
:
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
df1 = pd.DataFrame(mlb.fit_transform(df['Month_List']),columns=mlb.classes_)
df['Binary_Month_List'] = df1.reindex(columns=ref, fill_value=0).values.tolist()
或使用 Series.str.join
, Series.str.get_dummies
和reindex
:
df['Binary_Month_List'] = (df['Month_List'].str.join('|')
.str.get_dummies()
.reindex(columns=ref, fill_value=0)
.values
.tolist())
print (df)
Month_List Binary_Month_List
0 [July] [0, 0, 1, 0, 0, 0, 0]
1 [August] [0, 1, 0, 0, 0, 0, 0]
2 [July, June] [0, 0, 1, 1, 0, 0, 0]
3 [May, April, March] [0, 0, 0, 0, 1, 1, 1]
性能不同:
df = pd.concat([df] * 1000, ignore_index=True)
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
In [338]: %timeit (df['Month_List'].str.join('|').str.get_dummies().reindex(columns=ref, fill_value=0).values.tolist())
31.4 ms ± 1.41 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [339]: %timeit pd.DataFrame(mlb.fit_transform(df['Month_List']),columns=mlb.classes_).reindex(columns=ref, fill_value=0).values.tolist()
5.57 ms ± 94.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [340]: %timeit df['Binary_Month_List2'] =df.Month_List.explode().str.get_dummies().sum(level=0).reindex(columns=ref, fill_value=0).values.tolist()
58.6 ms ± 461 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
这篇关于使用numpy向量化检查pandas列中的参考列表的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!