# 3 Methods to Handle Missing Values in Dataset Using Pandas

Pandas provide numerous functions and methods to clean and preprocess the dataset to make it production-ready.

In this article, we'll see the methods provided by pandas to **handle missing values** in a dataset.

## df.fillna()

The [`DataFrame.fillna()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html) is used to fill in the missing values with the desired value. Let's see how we can use it.

```python
data = {"A": [2, np.nan, 19, 34, np.nan],
        "B": [np.nan, 23, 12, 34, np.nan]}
df = pd.DataFrame(data)
df
--------------------
      A     B
0   2.0   NaN
1   NaN  23.0
2  19.0  12.0
3  34.0  34.0
4   NaN   NaN
```

### Filling Arbitrary Value

```python
filled_df = df.fillna(0)
print(filled_df)
--------------------
      A     B
0   2.0   0.0
1   0.0  23.0
2  19.0  12.0
3  34.0  34.0
4   0.0   0.0
```

We passed an arbitrary value (`0`) to fill those `NaN` values in the dataset `df`.

### Fill Using a Dataset

We can also use a dataset to fill in the missing values.

```python
df2 = pd.DataFrame({"A": [1,2,3,4,5], "B": [6,7,8,9,10]})
fill_using_df = df.fillna(df2)
print(fill_using_df)
--------------------
      A     B
0   2.0   6.0
1   2.0  23.0
2  19.0  12.0
3  34.0  34.0
4   5.0  10.0
```

When using the `fillna()` method with `df2`, the `NaN` values in the original DataFrame `df` are replaced by the corresponding values in `df2`. If a cell in `df` is `NaN`, the method will look for the corresponding value in `df2` (at the same position) and use that value to fill in the `NaN`.

### Filling Different Values in Each Column

If we want to fill in different values in each column, we can use the following approach.

```python
values = {"A": 100, "B": 200}
diff_val = df.fillna(value=values)
print(diff_val)
--------------------
       A      B
0    2.0  200.0
1  100.0   23.0
2   19.0   12.0
3   34.0   34.0
4  100.0  200.0
```

The value dictionary holds values to fill `NaN` in columns `A` and `B` in the dataset. By using `df.fillna(value=values)`, the `NaN` value in column `A` is filled with the value `100` and `NaN` value in column `B` is filled with the value `200`.

## df.interpolate()

The [`DataFrame.interpolate()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.interpolate.html) method provides various **interpolation techniques** to fill in the missing values.

Instead of filling in hard-coded values, we can use an interpolation method to fill missing values that make the dataset even more expressive and real.

### Filling Computed Value

```python
df2 = df.interpolate()  # default: linear method and axis=0
print(df2)
--------------------
      A     B
0   2.0   NaN
1  10.5  23.0
2  19.0  12.0
3  34.0  34.0
4  34.0  34.0
```

When we use `df.interpolate()`, the default `linear` interpolation method is used that fills the `NaN` values equally spaced ignoring the index.

For example, in column `A`, `10.5` is filled which is equally spaced between the values `2.0` and `19.0` with the difference of `8.5`.

But if we see the fourth row in both columns, they are filled with the same value (`34.0`) as above them because there were no values to compute in the fifth row.

### Filling Nearest Values

```python
data = {"A": [3, np.nan, 2, np.nan, 4],
        "B": [1, 4, np.nan, 2, 5]}
df = pd.DataFrame(data)
df3= df.interpolate(method='nearest')
print(df3)
--------------------
     A    B
0  3.0  1.0
1  3.0  4.0
2  2.0  4.0
3  2.0  2.0
4  4.0  5.0
```

When we use `method='nearest'`, the `NaN` values are filled with the nearest valid values.

In this case, the second row in column `A` is filled with the value of `3.0`. **Why so?** The nearest value is decided based on the index close to the `NaN` value index. The index `0` (`3.0`) is closest to the index `1`. The same is applied to all the `NaN` values.

### Filling Values Considering Index Values

```python
data = {"A": [3, np.nan, 9, np.nan, 4],
        "B": [1, 10, np.nan, 20, 5]}
df = pd.DataFrame(data)
df4= df.interpolate(method='values')  # or method='index'
print(df4)
--------------------
     A     B
0  3.0   1.0
1  6.0  10.0
2  9.0  15.0
3  6.5  20.0
4  4.0   5.0
```

The `NaN` values are filled equally spaced considering the values of the index surrounding the `NaN` value index.

## df.ffill() and df.bfill()

The [`DataFrame.ffill()`](https://geekpython.in/ffill-and-bfill-in-pandas) method is used to fill the **last** valid value in the missing place whereas the [`DataFrame.bfill()`](https://geekpython.in/ffill-and-bfill-in-pandas) method is used to fill the **next** valid value.

### Forward Filling With ffill()

```python
data = {"A": [2, np.nan, 19, 34, np.nan],
        "B": [np.nan, 23, 12, 34, np.nan]}
df = pd.DataFrame(data)

forward_fill = df.ffill()
print(forward_fill)
--------------------
      A     B
0   2.0   NaN
1   2.0  23.0
2  19.0  12.0
3  34.0  34.0
4  34.0  34.0
```

We can see that `NaN` values are filled with the preceding valid values, for instance, the second row of column `A` is filled with `2.0` which is the same value above it.

### Backward Filling With bfill()

```python
data = {"A": [2, np.nan, 19, 34, np.nan],
        "B": [np.nan, 23, 12, 34, np.nan]}
df = pd.DataFrame(data)

backward_fill = df.bfill()
print(backward_fill)
--------------------
      A     B
0   2.0  23.0
1  19.0  23.0
2  19.0  12.0
3  34.0  34.0
4   NaN   NaN
```

In this case, the `NaN` is filled with the next valid values, for instance, the first row of column `B` is filled with `23.0` which is the next value in the column.

We can also see that the fifth row of columns `A` and `B` remains unfilled (`NaN`) due to the absence of the next valid value in the dataset.

---

🏆**Other articles you might be interested in if you liked this one**

✅[Pandas df.ffill() and df.bfill() to handle missing values](https://geekpython.in/ffill-and-bfill-in-pandas).

✅[Merge, combine, and concatenate multiple datasets using pandas](https://geekpython.in/multiple-datasets-integration-using-pandas).

✅[Find and delete duplicate rows from the dataset using pandas](https://geekpython.in/find-and-delete-duplicate-rows-from-dataset).

✅[How to efficiently manage memory use when working with large datasets in pandas](https://geekpython.in/copy-on-write-in-pandas)?

✅[How to find and delete mismatched columns from datasets in pandas](https://geekpython.in/find-and-delete-mismatched-columns-from-dataframes-using-pandas)?

✅[How does the learning rate affect the ML and DL models](https://geekpython.in/impact-of-learning-rates-on-ml-and-dl-models)?

---

**That's all for now**

**Keep Coding✌✌**