How to fill null values
Null (nan) values can be filled by using fillna() method (DataFrame.fillna())
It can be applied to series as well as for data frame objects
Example 1
** all the nan have been filled with 0
List of Important Parameters fillna() takes
value ---- "which value to be filled "
method ---- "different methods to fill nan "
axis ---- " Axis along which to fill missing values" [0 or ‘index’, 1 or ‘columns]
inplace ---- "True or False " default will be False. If True, do operation inplace and return None
limit ---- "To limit the fill for methods used to fill nan "
value
which values to be filled In nan position
If a single value is passed then all the nan will be replaced with that single value passed
Example 2
we can also pass dictionaries where keys will indicate the column name and values indicating which value to replace with nan in that particular column
Example 3
* values passed should be logical to that column otherwise total column datatype will get effected
Example:- in eps we can fill nan with mean of that column and in tickers we can fill with the mode of that column
inplace
When ever we are using fillna method on the data structures it is not affecting the original data structures .It is returning a new data structure object
if we want to modified the data in place,we have to use inplace = True which means it will return nothing and the dataframe is now updated.
If inplace = False which means it will return new dataframe and the dataframe is now updated
Example 4
Example 5
method
We also use some methods to fill na values
We can use "ffill" method knows as forward fill .fill is used to forward fill the missing values in the dataset
Example 6
** in 2 index row in "pe" column the nan values is filled with 1 index value in "pe" column (forward fill)
We can use "bfill" method knows as backward fill .bfill is used to backward fill the missing values in the dataset
Example 7
** in 2 index row in "pe" column the nan values is filled with 3 index value in "pe" column (backward fill)
axis
While using method parameters to fill nan values we can also use axis to specify which axis to consider whether column or rows to fill nan values
axis=0 means column axis=1 means rows
Example 8
Example 9
** when axis = 1 it is doing ffill row wise thats why in "pe" column in 2 index the nan value has filled the values from 2 index form "eps" column
limit
limit parameter is used to limit the maximum number of consecutive NaN values to be filled by the method.
Example 10
** all the consecutive nan values are filled
Example 11
** only the first consecutive nan value are filled
Example 12
** only the two consecutive nan values are filled
How to drop null values
Null (nan) values can be droped by using dropna() method (DataFrame.dropna())
It can be applied to series as well as for data frame objects
It will drop the rows or columns containing nan values
Default it will drop rows
Example 13
** dropped 2,3,4,6 indexed rows (because all this rows contain nan values
List of Important Parameters dropna() takes
how ---- "how to drop the rows "
axis ---- "columns or rows to drop"
inplace ---- "True or False " default will be False. If True, do operation inplace and return None
thresh ---- "Require that many non-NA values "
how
how to drop the rows
how parameters take two argument "all" or" any"
Example 14
"all" -- If all values are NA, drop that row or columns
"any" -- If any values are NA, drop that row or columns
Example 15
axis
Here also we can apply axis to specify which axis (column or rows) to drop if it contains nan values
Example 16
** axis = 0 drop rows containing nan values
Example 17
** axis = 1 drop columns containing nan values
inplace
When ever we are using fillna method on the data structures it is not affecting the original data structures .It is returning a new data structure object
if we want to modified the data in place,we have to use inplace = True which means it will return nothing and the dataframe is now updated.
If inplace = False which means it will return new dataframe and the dataframe is now updated
Example 18
thresh
How many non nan values are required to remove rows or columns
Here we are dealing with non nan values not nan values
Example 19
** drop those rows without 1 non nan values
Example 20
** drop those rows without 3 non nan values
Example 21
** drop those rows without 4 non nan values