For example, let’s say we have a variable about treatment or control conditions coded as treatment, T, treat, Treat, C, Cont, cont, cOnt, Control, and control. Using Excel to find all of these would be a matter of filtering and changing all by hand or using if statements.

Sometimes though, it’s not so simple. That’s where functions that find patterns come to be very useful.

set.seed(4) # random sample below - make sure same every time
status <- sample(c("treatment", "T", "treat", 
                   "Traet", "Treat", "C", "Cont",
                   "cont", "cOnt", "Control", "control"),
                 1000, replace = TRUE)
data_case = tibble(status)

count(data_case, status)

# A tibble: 11 × 2
   status        n
   <chr>     <int>
 1 C            81
 2 Cont         90
 3 Control      91
 4 T            91
 5 Traet       105
 6 Treat       100
 7 cOnt         79
 8 cont         83
 9 control      98
10 treat        86
11 treatment    96

In R, you could use case_when():

#case_when way:
data_case <-data_case |> mutate(status = 
            case_when(status 
          %in% c("C", "cont", "cOnt", "Cont", "control", "Control")
                                ~ "Control",
                           .default =  status)) 
count(data_case, status)

# A tibble: 6 × 2
  status        n
  <chr>     <int>
1 Control     522
2 T            91
3 Traet       105
4 Treat       100
5 treat        86
6 treatment    96

Oh dear! This only fixes some values! It is difficult to notice values like "Traet".

Like dplyr, the stringr package:

• Makes some things more intuitive
• Is different than base R
• Is used on forums for answers
• Has a standard format for most functions: str_
  • the first argument is a string like first argument is a data.frame in dplyr

Useful String functions from base R and stringr

• toupper(), tolower() - uppercase or lowercase your data

• str_sentence() - uppercase just the first character (in the stringr package)

• paste() - paste strings together with a space

• paste0 - paste strings together with no space as default

• str_trim() (in the stringr package) or trimws in base

  • will trim whitespace

• nchar - get the number of characters in a string

data_case <-data_case |>
                mutate(status = str_to_sentence(status))
count(data_case, status)

# A tibble: 5 × 2
  status        n
  <chr>     <int>
1 Control     522
2 T            91
3 Traet       105
4 Treat       186
5 Treatment    96

#case_when way:
data_case <-data_case |>
                mutate(status = str_to_sentence(status)) |>
                mutate(status = 
                      case_when(status %in% 
                    c("Treatment", "T", "Treat", "Traet", "Treat")
                                ~ "Treatment",
                           .default = status)) 
count(data_case, status)

# A tibble: 2 × 2
  status        n
  <chr>     <int>
1 Control     522
2 Treatment   478

OK, now we are getting somewhere!

Now we have a chance to keep but clean these values!

ufo <-read_csv(
  "https://sisbid.github.io/Data-Wrangling/data/ufo/ufo_data_complete.csv", 
  col_types = cols(`duration (seconds)` = "c"))

Warning: One or more parsing issues, call `problems()` on your data frame for details, e.g.:
  dat <- vroom(...)
  problems(dat)

colnames(ufo)

 [1] "datetime"             "city"                 "state"               
 [4] "country"              "shape"                "duration (seconds)"  
 [7] "duration (hours/min)" "comments"             "date posted"         
[10] "latitude"             "longitude"           

ufo_clean <- clean_names(ufo)
colnames(ufo_clean)

 [1] "datetime"           "city"               "state"             
 [4] "country"            "shape"              "duration_seconds"  
 [7] "duration_hours_min" "comments"           "date_posted"       
[10] "latitude"           "longitude"         

Now, let’s remove the backticks.

ufo_clean  <- ufo_clean |> 
  mutate(duration_seconds = 
           str_remove(string = duration_seconds,
                     pattern = "`"))

And let’s check it:

ufo_clean |> 
  filter(str_detect( 
    string = duration_seconds, 
    pattern = "`"))

# A tibble: 0 × 11
# ℹ 11 variables: datetime <chr>, city <chr>, state <chr>, country <chr>,
#   shape <chr>, duration_seconds <chr>, duration_hours_min <chr>,
#   comments <chr>, date_posted <chr>, latitude <chr>, longitude <chr>

stringr

• str_sub(x, start, end) - substrings from position start to position end

Examples:

str_sub("I like friesian horses", 8,12)

[1] "fries"

#123456789101112
#I like fries
str_sub(c("Site A", "Site B", "Site C"), 6,6)

[1] "A" "B" "C"

• R can do much more than find exact matches for a whole string
• Like Perl and other languages, it can use regular expressions.
• What are regular expressions?
  • Ways to search for specific strings
  • Can be very complicated or simple
  • Highly Useful - think “Find” on steroids

• https://rstudio.github.io/cheatsheets/strings.pdf
• They can use to match a large number of strings in one statement
• . matches any single character
• * means repeat as many (even if 0) more times the last character
• ? makes a pattern optional (i.e. it matches 0 or 1 times)
• ^ matches start of vector ^a - starts with “a”
• $ matches end of vector b$ - ends with “b”

str_detect, str_subset, str_replace, and str_replace_all search for matches to argument pattern within each element of a character vector: they differ in the format of and amount of detail in the results.

• str_detect - returns TRUE if pattern is found
• str_subset - returns only the strings where the pattern were detected
• str_extract - returns only the pattern that was detected
• str_replace - replaces pattern with replacement the first time
• str_replace_all - replaces pattern with replacement as many times matched

str_extract extracts just the matched string

ufo_clean |> 
  mutate(aliens  = str_extract(comments, "two aliens")) |>
  count(aliens)

# A tibble: 2 × 2
  aliens         n
  <chr>      <int>
1 two aliens     2
2 <NA>       88873

• Look for any comment that starts with “aliens”

ufo_clean |> pull(comments) |> str_subset( "^aliens")

[1] "aliens speak german???" "aliens exist"           "aliens in srilanka"    

pull(ufo_clean, duration_hours_min) |> head(n = 20)

 [1] "45 minutes"      "1-2 hrs"         "20 seconds"      "1/2 hour"       
 [5] "15 minutes"      "5 minutes"       "about 3 mins"    "20 minutes"     
 [9] "3  minutes"      "several minutes" "5 min."          "3 minutes"      
[13] "30 min."         "3 minutes"       "30 seconds"      "20minutes"      
[17] "2 minutes"       "20-30 min"       "20 sec."         "2 min"          

Let’s say we wanted to make the time information more consistent. Using case_when() could be very tedious and error-prone!

We can use str_replace() to do so.

ufo_clean |> mutate(duration_hours_min = 
                 str_replace(string = duration_hours_min, 
                             pattern = "minutes", 
                             replacement ="mins")) |>
  pull(duration_hours_min) |>
  head(8)

[1] "45 mins"      "1-2 hrs"      "20 seconds"   "1/2 hour"     "15 mins"     
[6] "5 mins"       "about 3 mins" "20 mins"     

Better yet, you might notice that this data isn’t tidy- there are more than two entries for each value - amount of time and unit. We could separate this using separate() from the tidyr package.

ufo_clean |> separate(duration_hours_min,
                 into = c("duration_amount", "duration_unit"),
                 sep = " ") |>
  select(duration_amount, duration_unit) |> head()

# A tibble: 6 × 2
  duration_amount duration_unit
  <chr>           <chr>        
1 45              minutes      
2 1-2             hrs          
3 20              seconds      
4 1/2             hour         
5 15              minutes      
6 5               minutes      

As you can see there is still plenty of cleaning to do!

ufo_clean <- ufo_clean |> separate(datetime,
                 into = c("date", "time"),
                 sep = " ")
ufo_clean |> select(date, time) |> head()

# A tibble: 6 × 2
  date       time 
  <chr>      <chr>
1 10/10/1949 20:30
2 10/10/1949 21:00
3 10/10/1955 17:00
4 10/10/1956 21:00
5 10/10/1960 20:00
6 10/10/1961 19:00

str_detect(string = c("abcdd", "two"), pattern = "dd")

[1]  TRUE FALSE

str_subset(string = c("abcdd", "two"), pattern = "dd")

[1] "abcdd"

str_extract(string = c("abcdd", "two"), pattern = "dd")

[1] "dd" NA  

str_sub(string = c("abcdd", "two"), start = 1, end = 3)

[1] "abc" "two"

• stringr package has lots of helpful functions that work on vectors or variables in a data frame
• str_detect helps find patterns
• str_detect and filter can help you filter data based on patterns within value
• str_extract helps extract a pattern
• str_sub extracts pieces of strings based on the position of the the characters
• str_subset gives the values that match a pattern
• separate can separate columns into two
• ^ indicates the start of a string
• $ indicates the end of a string
• the lubridate package is useful for dates and times

https://sisbid.github.io/Data-Wrangling/09_Data_Cleaning/lab/data-cleaning-lab-part2.Rmd