Missing data is a common data quality issue that every data practitioner has to deal with. Missing data affects the accuracy of analysis and the performance of the model. Most machine learning algorithms are sensitive to missing data. In the data cleaning process checking and dealing with missing data is a common task. Missing data can be presented as **NaN**, **Null** or **None**. The strategies for handling missing data can be as simple as deleting the rows or columns with missing values or as complex as developing a model to predict the value to be used to impute missing data. In this post we will look at different techniques to handle missing data and how to implement them in Python.

**Causes of Missing Data**

There are many reasons that can results to missing values in a datasets; some of the most causes are as listed below;

- Data entry error
- Rare observations that cannot be found
- The species/individual under study can die or drop off before sampling concludes
- Lack of response to the survey questions

It is important to understand the root cause of the missing data/value to prevent future occurrence or device a mechanism to handle it.

**Types of Missing Data**

**Missing Completely at Random (MCAR).**This is when the probability of an instance to have a missing value for an attribute does not depend on any other value from the dataset.**Missing at Random (MAR).**This is when the probability of an instance to have a missing value for an attribute depends on some other value from the dataset.**Not Missing at Random (NMAR).**This is when the probability that a value is missing does depend on the value(s) of the target variable to be imputed, and possibly also on the values of auxiliary variables. This is the case of the value of the variable that’s missing is related to the reason it’s missing such as if men failed to fill in a depression survey because of their level of depression.

**Handling Missing Data**

There are various approaches to deal with missing values/data ranging from simple techniques such as dropping the entire record to creating a sophisticated model for data imputation. Below are some of the most common techniques to deal with missing data.

- Deleting records with missing values.

- Imputation. This involves filling the missing values with other values either manually determined or statistical computed value.

- Interpolation. This involves constructing new data points to fill in the missing values.

- Full Analysis. We use different approaches such as generative or discriminative approaches to impute missing data.

- Model-based techniques. We can develop a machine learning model to predict the probable value for the missing data point. Sklearn comes with
**KNNImputer**class that is based on K-Nearest neighbors. It uses the distance based metric to compute the nearest data to impute with missing one.

**Create DataFrame**

import pandas as pd import numpy as np missing_data_df = pd.DataFrame( { "Students": ["Tom", "Peter",np.nan, "Mary", "Tom","King","Tom","Mary",np.nan], "Exam_Date": ["15/01/2021", "16/01/2021", "19/01/2021", "27/01/2021", "16/01/2021", "16/01/2021", "16/01/2021", "16/01/2021", "16/01/2021"], "Math": [79.00, 67.00,np.nan, np.nan, 70.00,np.nan,90.00,76.00,np.nan], "Physics":[63.00, np.nan, 60.00, np.nan,84.00, 77.00,55.00,np.nan,66.00], "Computer":[np.nan,78.00, 57.00, 88.00, np.nan,np.nan,np.nan,70.00,np.nan], } ) missing_data_df

**Show null values and return false if not null**

missing_data_df.isnull().head()

**Show not null values and return true if not null**

missing_data_df.notnull().head()

**Count of null values**

missing_data_df.isnull().sum()

## Techniques for Handling Missing Values

### 1. Deleting records with missing values

**Drop entire row with all values null**

missing_data_df.dropna(how='all')

**Drop column with any null value**

missing_data_df.dropna(how='any',axis=1)

**Drop rows with any of specified columns have null**

missing_data_df.dropna(subset=['Math', 'Physics'], how='any')

**Drop rows with all of specified columns have null**

missing_data_df.dropna(subset=['Math', 'Physics'], how='all')

**Drop row with a given number of null values**

missing_data_df.dropna(axis=1,thresh=2)

### 2. Imputation

**Replace null values with a scalar value**

missing_data_df.fillna(-999) # replace null values with -999

**Backward Fill with Next Value**

missing_data_df.fillna(method='bfill')

**Forward Fill with Previous Value**

missing_data_df.fillna(method='ffill')

**Statistical Imputation**

missing_data_df.fillna(missing_data_df.Math.mean()) # fillna null value in Math column with mean of the Math missing_data_df.fillna(missing_data_df.Students.mode()) # fillna null value in Students column with mode of the Students missing_data_df.fillna(missing_data_df.Computer.median()) # fillna null value in Computer column with median of the Computer

### 3. Interpolate missing values

**Interpolate missing data in forward direction**

missing_data_df.interpolate(method='linear', limit_direction ='forward')

**Interpolate missing data in backward direction**

missing_data_df.interpolate(method='linear', limit_direction ='backward')

### 4. Model-based Imputation techniques

**Nearest neighbors imputation**

**DataFrame with Missing Values**

from sklearn.impute import KNNImputer imputer = KNNImputer(n_neighbors=2, weights="uniform") imputed_data=imputer.fit_transform(missing_data_df[['Math','Physics','Computer']].to_numpy()) imputed_data

Join imputed data to DataFrame

pd.concat([missing_data_df[['Students','Exam_Date']],pd.DataFrame(imputed_data,columns=['Math','Physics','Computer'])],axis=1)

For complete code check notebook from github **here**.

**Conclusion**

Missing data is as a results of data quality issues. It affects the results of the analysis and performance of machine learning model as many machine learning algorithm are sensitive to missing data. To deal with missing values we can either impute the missing values with other values, drop records with missing values, interpolate or develop machine learning model that can predict the data point to fill in the missing value. In this post we have looked at what is the missing value, causes of missing value and how to handle it. In the next post we will learn about **handling imbalanced data**.