---
title: "Missing Data"
author: "Data Wrangling in R"
---


```{r, include = FALSE}
library(knitr)
library(tidyverse)
opts_chunk$set(comment = "")
```

# Dealing with Missing Data

## Missing data types
One of the most important aspects of data cleaning is missing values.  

Types of "missing" data:

* `NA` - general missing data
* `NaN` - stands for "**N**ot **a** **N**umber", happens when you do 0/0.
* `Inf` and `-Inf` - Infinity, happens when you take a positive number (or negative number) by 0.


## Missing Data with Logicals

Logical operations return `NA` for `NA` values.  Think about it, the data could be `> 2` or not we don't know, so `R` says there is no `TRUE` or `FALSE`, so that is missing:

```{r}
x <- c(0, NA, 2, 3, 4)
x > 2
```

## Missing Data Issues

Mathematical operations with `NA` result in `NA`s.

```{r}
y <- c(1,2,3,NA)
sum(y)
mean(y)
```


## Missing Data Issues

Logicals: `TRUE` is evaluated as 1 and `FALSE` is evaluated as 0.

```{r}
x <- c(TRUE, TRUE, TRUE, TRUE, FALSE, NA)
sum(x)
sum(x, na.rm = TRUE)
```


## Finding Missing data {.small}

-   `is.na` - looks for `NAN` and `NA`
-   `is.nan`- looks for `NAN`

```{r}
test <- c(0,NA, -1, NaN)
is.na(test)
is.nan(test)
```


## Useful checking functions {.small}

Do we have any `NA`s? (`any` can help)

```{r}
A <- c(1, 2, 3, NA)
B <- c(1, 2, 3, 4)
```
<br>
```{r}
any(is.na(A)) # are there any NAs - YES/TRUE
any(is.na(B)) # are there any NAs- NO/FALSE
```


## Finding `NA` values with `count()`{.codesmall}

::: {style="color: red;"}
Check the values for your variables, are they what you expect?
:::

`count()` is a great option because it gives you:

1)  The unique values
2)  The amount of these values

Check if rare values make sense.


## Finding `NA` values with `count()`{.codesmall}

```{r, message=FALSE}
library(readr)
library(janitor)
ufo <- read_csv(
  "https://sisbid.github.io/Data-Wrangling/data/ufo/ufo_data_complete.csv")
ufo <- clean_names(ufo)

count(ufo, country)
``` 


## `naniar`

Sometimes you need to look at lots of data... the [`naniar`
package](https://cran.r-project.org/web/packages/naniar/vignettes/getting-started-w-naniar.html) is a good option.

```{r, error=FALSE}
#install.packages("naniar")
library(naniar)
```


## naniar: `pct_complete()`{.small}

This can tell you if there are missing values in the dataset.

```{R}
pct_complete(ufo)
```

Or for a particular variable:

```{R}
ufo |> select(shape) |>
pct_complete()
```

## naniar:`miss_var_summary()`{.codesmall}

To get the percent missing (and counts) for each variable as a table, use this function.

```{r}
miss_var_summary(ufo)
```


## `naniar` plots of number missing

The `gg_miss_var()` function creates a nice plot about the number of
missing values for each variable, (need a data frame).

```{r, fig.height=4, warning=FALSE, fig.align='center'}
gg_miss_var(ufo)
```

## `naniar` plots of percent missing

The `gg_miss_var()` function creates a nice plot about the number of
missing values for each variable, (need a data frame).

```{r, fig.height=4, warning=FALSE, fig.align='center'}
gg_miss_var(ufo, show_pct = TRUE)
```

## filter() and missing data

Be **careful** with missing data using subsetting!

**`filter()` removes missing values by default.** Because R can't tell
for sure if an `NA` value meets the condition. To keep them need to add
`is.na()` conditional.

Think about if this is OK or not - it depends on your data!


## filter() and missing data{.codesmall}

What if NA values represent values that are so low it is undetectable?

Filter will drop them from the data.

```{r}
count(ufo, country)
ufo |> filter(country == "de") |> dim()

```

## filter() and missing data

`is.na()` can help us keep them.

```{r}
ufo |> filter(country == "de" | is.na(country)) |> dim()
```

## To remove **rows** with `NA` values for a **variable** use `drop_na()`{.codesmall}

A function from the `tidyr` package. (Need a data frame to start!)

Disclaimer: Don't do this unless you have thought about if dropping `NA` values makes sense based on knowing what these values mean in your data.

```{r}
ufo
```

## To remove **rows** with `NA` values for a **variable** use `drop_na()`{.codesmall}

```{r}
ufo |> drop_na(state)
```

## Drop all NAs with `drop_na()`{.codesmall}

Drops rows with **any** missing data in **any** column.

```{r}
ufo |> drop_na() |> dim()
ufo |> drop_na()
```

## Find columns with any missing values

Use the `miss_var_which()` function from `naniar`

```{r}
miss_var_which(ufo) # which columns have missing values
```

## Drop **columns** with any missing values {.codesmall}

`miss_var_which` and  function from `naniar` (need a data frame)

```{r}
ufo |> select(!miss_var_which(ufo))
```

## Change a value to be `NA` {.codesmall}

Let's say we think that all 0 values should be `NA`.


The `na_if()` function of `dplyr` can be helpful for changing all 0 values to `NA`. More on `mutate()` soon! (Here we save the change)


```{r}
count(ufo, duration_seconds) |> tail()
```

## Change a value to be `NA` {.codesmall}

```{r}
ufo <- ufo |> 
  mutate(duration_seconds = na_if(duration_seconds, 0))

count(ufo, duration_seconds) |> tail()

```

## Change `NA` to be a value {.codesmall}

The `replace_na()` function (part of the `tidyr` package), can do the opposite of `na_if()`. (Here we just print the change)

```{r}

count(ufo, shape) |> tail()
```

## Change `NA` to be a value {.codesmall}

The `replace_na()` function (part of the `tidyr` package), can do the opposite of `na_if()`. (Here we just print the change)

```{r}
ufo |> 
  mutate(shape = replace_na(shape, "unknown")) |>
  count(shape) |> tail()
```


## Think about `NA`

::: {style="color: red;"}
 THINK ABOUT YOUR DATA FIRST! 
::: 


⚠️ Sometimes removing `NA` values leads to distorted math - be careful!

⚠️ Think about what your `NA` means for your data (are you sure ?).

- Is an `NA` for values so low they could not be reported? 

- Or is it if it was too low and also if there was a different issue (like no one reported)?

## Think about `NA`

If it is something more like a zero then you might want it included in
your data like a zero instead of an `NA`.

Example: - survey reports `NA` if student has never tried cigarettes -
survey reports 0 if student has tried cigarettes but did not smoke that
week


⚠️ You might want to keep the `NA` values so that you know the original sample size. 


## Word of caution {.codesmall}


⚠️ Calculating percentages will give you a different result depending on your choice to include NA values.!

This is because the denominator changes.


## Word of caution - Percentages with `NA`


```{r}
count(ufo, country) |> mutate(percent = (n/(sum(n)) *100)) |> 
  arrange(desc(percent))
```

## Word of caution - Percentages with `NA`

```{r}

ufo |> drop_na(country) |>
count(country) |> mutate(percent = (n/(sum(n)) *100)) |> 
  arrange(desc(percent))

```

Should you be dividing by the total count with `NA` values included?   
It depends on your data and what `NA` might mean.   
Pay attention to your data and your `NA` values!


## Summary

-   `is.na()`,`any(is.na())`, `count()`, and functions from `naniar`
    like `gg_miss_var()` can help determine if we have `NA` values
-   `filter()` automatically removes `NA` values - can't confirm or deny
    if condition is met (need `| is.na()` to keep them)
-   `drop_na()` can help you remove `NA` values from a variable or an
    entire data frame
-   `NA` values can change your calculation results
-   think about what `NA` values represent - don't drop them if you shouldn't

[Link to Lab](https://sisbid.github.io/Data-Wrangling/07_Missing_Data/lab/missing-data-lab.Rmd)