3/13/2019

The tidyverse

A collection of packages particularly helpful for working with tables.

  • dplyr - manipulating data frames
  • ggplot2 - data visualization
  • readr - importing data
  • stringr - string manipulation
  • lubridate - dates and times
  • purrr package for working with functions

The tidyverse

Load the main packages like this:

library(tidyverse)

Tidy data

  • Each row represents one observation
  • Columns represent the different variables

Tidy data

The murders dataset is an example of a tidy data frame.

library(dslabs)
data(murders)
head(murders)
##        state abb region population total
## 1    Alabama  AL  South    4779736   135
## 2     Alaska  AK   West     710231    19
## 3    Arizona  AZ   West    6392017   232
## 4   Arkansas  AR  South    2915918    93
## 5 California  CA   West   37253956  1257
## 6   Colorado  CO   West    5029196    65

Wide data

This is not tidy:

##       country 1960 1961 1962 1963 1964 1965
## 1     Germany 2.41 2.44 2.47 2.49 2.49 2.48
## 2 South Korea 6.16 5.99 5.79 5.57 5.36 5.16

Later we learn how to wrangle into tidy format.

Tidy data

Same information in tidy format.

##        country year fertility
## 1      Germany 1960      2.41
## 2  South Korea 1960      6.16
## 3      Germany 1961      2.44
## 4  South Korea 1961      5.99
## 5      Germany 1962      2.47
## 6  South Korea 1962      5.79
## 7      Germany 1963      2.49
## 8  South Korea 1963      5.57
## 9      Germany 1964      2.49
## 10 South Korea 1964      5.36
## 11     Germany 1965      2.48
## 12 South Korea 1965      5.16

Manipulating data frames with dplyr

  • mutate - change or add columns
  • filter- filter rows based on a rule
  • select - Subset the data by selecting specific columns

Adding a column with mutate

data("murders")
murders <- mutate(murders, rate = total / population * 100000)

Notice that here we used total and population inside the function, which are objects that are not defined in our workspace.

Adding a column with mutate

We can see the new column now

head(murders)
##        state abb region population total     rate
## 1    Alabama  AL  South    4779736   135 2.824424
## 2     Alaska  AK   West     710231    19 2.675186
## 3    Arizona  AZ   West    6392017   232 3.629527
## 4   Arkansas  AR  South    2915918    93 3.189390
## 5 California  CA   West   37253956  1257 3.374138
## 6   Colorado  CO   West    5029196    65 1.292453

Subsetting with filter

filter(murders, rate <= 0.71)
##           state abb        region population total      rate
## 1        Hawaii  HI          West    1360301     7 0.5145920
## 2          Iowa  IA North Central    3046355    21 0.6893484
## 3 New Hampshire  NH     Northeast    1316470     5 0.3798036
## 4  North Dakota  ND North Central     672591     4 0.5947151
## 5       Vermont  VT     Northeast     625741     2 0.3196211

Selecting columns with select

new_table <- select(murders, state, region, rate)
filter(new_table, rate <= 0.71)
##           state        region      rate
## 1        Hawaii          West 0.5145920
## 2          Iowa North Central 0.6893484
## 3 New Hampshire     Northeast 0.3798036
## 4  North Dakota North Central 0.5947151
## 5       Vermont     Northeast 0.3196211

The pipe: %>%

\[ \mbox{original data } \rightarrow \mbox{ select } \rightarrow \mbox{ filter } \]

The pipe: %>%

When using the pipe, functions take left hand side as first argumemt:

murders %>% select(state, region, rate) %>% filter(rate <= 0.71)
##           state        region      rate
## 1        Hawaii          West 0.5145920
## 2          Iowa North Central 0.6893484
## 3 New Hampshire     Northeast 0.3798036
## 4  North Dakota North Central 0.5947151
## 5       Vermont     Northeast 0.3196211

More pipe example

sqrt(16)
## [1] 4
16 %>% sqrt()
## [1] 4
log2(sqrt(16))
## [1] 2
16 %>% sqrt() %>% log2()
## [1] 2

Example: Use arguments with pipe

16 %>% sqrt() %>% log(base = 2)
## [1] 2

Pipe example

murders %>% 
  select(state, region, rate) %>% 
  filter(rate <= 0.71)

Summarizing data

Summarizing variables is very common operation.

Average female heights:

data(heights)
s <- heights %>% 
  filter(sex == "Female") %>%
  summarize(average = mean(height))

Tidyverse always returns tables

s$average
## [1] 64.93942

Summarizing data

You can compute more than one summary

data(heights)
heights %>% 
  filter(sex == "Female") %>%
  summarize(median = median(height), 
            q25 = quantile(height, .25),
            q75 = quantile(height, .75))
##     median q25 q75
## 1 64.98031  63  67

Summarize only wors with one variable summaries

You can compute more than one summary

data(heights)
heights %>% 
  filter(sex == "Female") %>%
  summarize(qs = quantile(height, c(1,2,3) / 4))
Error in summarise_impl(.data, dots) : 
  Column `qs` must be length 1 (a summary value), not 3

pull

The us_murder_rate object defined above represents just one number. Yet we are storing it in a data frame:

s <- heights %>% 
  filter(sex == "Female") %>%
  summarize(average = mean(height))
class(s)
## [1] "data.frame"

pull

s %>% pull(average)
## [1] 64.93942

Group then summarize

heights %>% group_by(sex)
## # A tibble: 1,050 x 2
## # Groups:   sex [2]
##    sex    height
##    <fct>   <dbl>
##  1 Male       75
##  2 Male       70
##  3 Male       68
##  4 Male       74
##  5 Male       61
##  6 Female     65
##  7 Female     66
##  8 Female     62
##  9 Female     66
## 10 Male       67
## # ... with 1,040 more rows
  • The result does not look very different from heights, except we see Groups: sex [2] when we print the object.
  • Although not immediately obvious from its appearance, this is now a special data frame called a grouped data frame

Group then summarize

heights %>% 
  group_by(sex) %>%
  summarize(average = mean(height), sd = sd(height))
## # A tibble: 2 x 3
##   sex    average    sd
##   <fct>    <dbl> <dbl>
## 1 Female    64.9  3.76
## 2 Male      69.3  3.61

The dot operator

  • When using the pipe, the object being passed by the expression on the left side of the pipe can be accessed via .

  • Here is an example that is equivalent to pull:

s %>% .$average
## [1] 64.93942

do

  • We saw that summarize does not permit the use of functions that return more than one scaler

  • One way to apply such function is with do

  • Think of do as a connection between R base and the tidyverse.

  • General idea, write a function that returns what you want in a data frame, then call it with do.

do

qs <- function(x){
  res <- quantile(x,  c(1,2,3) / 4)
  data_frame(q25 = res[1], median = res[2],  q75 = res[3])
}
heights %>% 
  group_by(sex) %>%
  do(qs(.$height))
## # A tibble: 2 x 4
## # Groups:   sex [2]
##   sex      q25 median   q75
##   <fct>  <dbl>  <dbl> <dbl>
## 1 Female    63   65.0    67
## 2 Male      67   69      72

Sorting data frames

murders %>%
  arrange(population) %>%
  head()
##                  state abb        region population total       rate
## 1              Wyoming  WY          West     563626     5  0.8871131
## 2 District of Columbia  DC         South     601723    99 16.4527532
## 3              Vermont  VT     Northeast     625741     2  0.3196211
## 4         North Dakota  ND North Central     672591     4  0.5947151
## 5               Alaska  AK          West     710231    19  2.6751860
## 6         South Dakota  SD North Central     814180     8  0.9825837

Sorting data frames

murders %>% 
  arrange(rate) %>% 
  head()
##           state abb        region population total      rate
## 1       Vermont  VT     Northeast     625741     2 0.3196211
## 2 New Hampshire  NH     Northeast    1316470     5 0.3798036
## 3        Hawaii  HI          West    1360301     7 0.5145920
## 4  North Dakota  ND North Central     672591     4 0.5947151
## 5          Iowa  IA North Central    3046355    21 0.6893484
## 6         Idaho  ID          West    1567582    12 0.7655102

Sorting data frames

murders %>% 
  arrange(desc(rate)) %>% 
  head()
##                  state abb        region population total      rate
## 1 District of Columbia  DC         South     601723    99 16.452753
## 2            Louisiana  LA         South    4533372   351  7.742581
## 3             Missouri  MO North Central    5988927   321  5.359892
## 4             Maryland  MD         South    5773552   293  5.074866
## 5       South Carolina  SC         South    4625364   207  4.475323
## 6             Delaware  DE         South     897934    38  4.231937

Nested sorting

murders %>% 
  arrange(region, rate) %>% 
  head()
##           state abb    region population total      rate
## 1       Vermont  VT Northeast     625741     2 0.3196211
## 2 New Hampshire  NH Northeast    1316470     5 0.3798036
## 3         Maine  ME Northeast    1328361    11 0.8280881
## 4  Rhode Island  RI Northeast    1052567    16 1.5200933
## 5 Massachusetts  MA Northeast    6547629   118 1.8021791
## 6      New York  NY Northeast   19378102   517 2.6679599

The top \(n\)

murders %>% top_n(10, rate)
##                   state abb        region population total      rate
## 1               Arizona  AZ          West    6392017   232  3.629527
## 2              Delaware  DE         South     897934    38  4.231937
## 3  District of Columbia  DC         South     601723    99 16.452753
## 4               Georgia  GA         South    9920000   376  3.790323
## 5             Louisiana  LA         South    4533372   351  7.742581
## 6              Maryland  MD         South    5773552   293  5.074866
## 7              Michigan  MI North Central    9883640   413  4.178622
## 8           Mississippi  MS         South    2967297   120  4.044085
## 9              Missouri  MO North Central    5988927   321  5.359892
## 10       South Carolina  SC         South    4625364   207  4.475323

Tibbles

data(murders)
class(murders)
## [1] "data.frame"
  • Remember group_by permits stratifying data before computing summary statistics.
  • But where is the group information stored in the data frame?

Tibbles

murders %>% group_by(region) %>% head()
## # A tibble: 6 x 5
## # Groups:   region [2]
##   state      abb   region population total
##   <chr>      <chr> <fct>       <dbl> <dbl>
## 1 Alabama    AL    South     4779736   135
## 2 Alaska     AK    West       710231    19
## 3 Arizona    AZ    West      6392017   232
## 4 Arkansas   AR    South     2915918    93
## 5 California CA    West     37253956  1257
## 6 Colorado   CO    West      5029196    65

Tibbles

murders %>% group_by(region) %>% class()
## [1] "grouped_df" "tbl_df"     "tbl"        "data.frame"

Tibles versus data frames

  • Tibbles display better
  • Subsets of tibbles are tibbles
class(murders[,4])
## [1] "numeric"
class(as_tibble(murders)[,4])
## [1] "tbl_df"     "tbl"        "data.frame"

Tibles versus data frames

murders$Population
## NULL
as_tibble(murders)$Population
## Warning: Unknown or uninitialised column: 'Population'.
## NULL

Tibles versus data frames

  • Tibbles can have complex entries
tibble(id = c(1, 2, 3), func = c(mean, median, sd))
## # A tibble: 3 x 2
##      id func  
##   <dbl> <list>
## 1     1 <fn>  
## 2     2 <fn>  
## 3     3 <fn>

Tibles versus data frames

  • Tibbles can be grouped

Create a tibble using data_frame

grades <- data_frame(names = c("John", "Juan", "Jean", "Yao"), 
                     exam_1 = c(95, 80, 90, 85), 
                     exam_2 = c(90, 85, 85, 90))
grades
## # A tibble: 4 x 3
##   names exam_1 exam_2
##   <chr>  <dbl>  <dbl>
## 1 John      95     90
## 2 Juan      80     85
## 3 Jean      90     85
## 4 Yao       85     90

Another difference

grades <- data.frame(names = c("John", "Juan", "Jean", "Yao"), 
                     exam_1 = c(95, 80, 90, 85), 
                     exam_2 = c(90, 85, 85, 90))
class(grades$names)
## [1] "factor"

Another difference

  • To avoid this, we use the rather cumbersome argument stringsAsFactors:
grades <- data.frame(names = c("John", "Juan", "Jean", "Yao"), 
                     exam_1 = c(95, 80, 90, 85), 
                     exam_2 = c(90, 85, 85, 90),
                     stringsAsFactors = FALSE)
class(grades$names)
## [1] "character"

Conver to tibble

as_tibble(grades) %>% class()
## [1] "tbl_df"     "tbl"        "data.frame"

Conditionals

  • You should be familiar with the ifelse function. But let's review it as it is very useful in the tidyverse

  • ifelse vectorizes conditional assignments. Here is an example

x <- seq(-5, 5)
ifelse(x>0, "positive", "non-positive")
##  [1] "non-positive" "non-positive" "non-positive" "non-positive"
##  [5] "non-positive" "non-positive" "positive"     "positive"    
##  [9] "positive"     "positive"     "positive"

Conditionals

  • The tidyverse offers another useful tool: case_when
x <- seq(-5, 5)
case_when(x > 0 ~ "positive",
          x < 0 ~ "negative",
          TRUE ~ "zero")
##  [1] "negative" "negative" "negative" "negative" "negative" "zero"    
##  [7] "positive" "positive" "positive" "positive" "positive"

Remapping

The recode function is a related useful function for dealing with factor names.

x <- seq(-5, 5)
recode(sign(x), "1" = "positive", "0" = "zero", "-1" = "nonpositive")
##  [1] "nonpositive" "nonpositive" "nonpositive" "nonpositive" "nonpositive"
##  [6] "zero"        "positive"    "positive"    "positive"    "positive"   
## [11] "positive"

Mainly useful for changing codes to actual meaningful level names.