我想找出我的数据的每一列中NaN的数量。
当前回答
我写了一个简短的函数(Python 3)来生成.info作为pandas数据框架,然后可以写入excel:
df1 = pd.DataFrame({'a':[1,2,np.nan], 'b':[np.nan,1,np.nan]})
def info_as_df (df):
null_counts = df.isna().sum()
info_df = pd.DataFrame(list(zip(null_counts.index,null_counts.values))\
, columns = ['Column', 'Nulls_Count'])
data_types = df.dtypes
info_df['Dtype'] = data_types.values
return info_df
print(df1.info())
print(info_as_df(df1))
这使:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 a 2 non-null float64
1 b 1 non-null float64
dtypes: float64(2)
memory usage: 176.0 bytes
None
Column Nulls_Count Dtype
0 a 1 float64
1 b 2 float64
其他回答
import pandas as pd
import numpy as np
# example DataFrame
df = pd.DataFrame({'a':[1,2,np.nan], 'b':[np.nan,1,np.nan]})
# count the NaNs in a column
num_nan_a = df.loc[ (pd.isna(df['a'])) , 'a' ].shape[0]
num_nan_b = df.loc[ (pd.isna(df['b'])) , 'b' ].shape[0]
# summarize the num_nan_b
print(df)
print(' ')
print(f"There are {num_nan_a} NaNs in column a")
print(f"There are {num_nan_b} NaNs in column b")
给出输出:
a b
0 1.0 NaN
1 2.0 1.0
2 NaN NaN
There are 1 NaNs in column a
There are 2 NaNs in column b
使用isna()方法(或者它的别名isnull(),这也兼容较旧的pandas版本< 0.21.0),然后求和来计算NaN值。其中一列:
>>> s = pd.Series([1,2,3, np.nan, np.nan])
>>> s.isna().sum() # or s.isnull().sum() for older pandas versions
2
对于一些列,这也适用:
>>> df = pd.DataFrame({'a':[1,2,np.nan], 'b':[np.nan,1,np.nan]})
>>> df.isna().sum()
a 1
b 2
dtype: int64
下面的代码将按降序打印所有Nan列。
df.isnull().sum().sort_values(ascending = False)
or
下面将按降序打印前15个Nan列。
df.isnull().sum().sort_values(ascending = False).head(15)
另一种完整的方法是使用np。带有.isna()的count_non0:
np.count_nonzero(df.isna())
%timeit np.count_nonzero(df.isna())
512 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
使用1000005行× 16列的数据框架与顶部答案进行比较:
%timeit df.isna().sum()
492 ms ± 55.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit df.isnull().sum(axis = 0)
478 ms ± 34.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit count_nan = len(df) - df.count()
484 ms ± 47.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
数据:
raw_data = {'first_name': ['Jason', np.nan, 'Tina', 'Jake', 'Amy'],
'last_name': ['Miller', np.nan, np.nan, 'Milner', 'Cooze'],
'age': [22, np.nan, 23, 24, 25],
'sex': ['m', np.nan, 'f', 'm', 'f'],
'Test1_Score': [4, np.nan, 0, 0, 0],
'Test2_Score': [25, np.nan, np.nan, 0, 0]}
results = pd.DataFrame(raw_data, columns = ['first_name', 'last_name', 'age', 'sex', 'Test1_Score', 'Test2_Score'])
# big dataframe for %timeit
big_df = pd.DataFrame(np.random.randint(0, 100, size=(1000000, 10)), columns=list('ABCDEFGHIJ'))
df = pd.concat([big_df,results]) # 1000005 rows × 16 columns
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.count.html#pandas.Series.count
pandas.Series.count
Series.count(level=None)[source]
返回系列中非na /null观测值的个数
推荐文章
- 证书验证失败:无法获得本地颁发者证书
- 当使用pip3安装包时,“Python中的ssl模块不可用”
- 无法切换Python与pyenv
- Python if not == vs if !=
- 确定每列中NA值的个数
- 如何从scikit-learn决策树中提取决策规则?
- 为什么在Mac OS X v10.9 (Mavericks)的终端中apt-get功能不起作用?
- 将旋转的xtick标签与各自的xtick对齐
- 为什么元组可以包含可变项?
- 如何合并字典的字典?
- 如何创建类属性?
- 如何结合多个条件子集数据帧使用“或”?
- 不区分大小写的“in”
- 在Python中获取迭代器中的元素个数
- 解析日期字符串并更改格式