```{r setup, include=FALSE}
knitr::opts_chunk$set(fig.align = 'center')
sessionInfo()
```

## Announcement

- Remember to cite your data source.  
  
  - A web link is less stable, but works for the course project
  
  - Cite the original publication

## Acknowledgement

Dr. Hua Zhou's [slides](https://ucla-biostat203b-2020winter.github.io/slides/07-dplyr/dplyr.html)


```{r}
rm(list = ls()) # clean-up workspace
```

A typical data science project:

<p align="center">
<img src="./data-science.png" height="275">
</p>

## nycflights13 data

- Available from the nycflights13 package. 

- 336,776 flights that departed from New York City in 2013:
    ```{r}
    library("tidyverse")
    library("nycflights13")
    flights
    ```

- To display more rows:
    ```{r}
    flights %>% print(n = 20)
    ```
  Note `%>%` is the pipe is tidyverse. Above command is equivalent to `print(flights, n = 20)`.

- To display all rows:
    ```{r, eval = FALSE}
    flights %>% print(n = nrow(.))
    ```
    Do **not** try this on the `flights` data, which has too many rows.
    
- To display more columns (variables):
    ```{r}
    flights %>% print(width = Inf)
    ```
    The `width` argument controls the screen width.

## dplyr basics

* Pick observations (rows) by their values: `filter()`.

* Reorder the rows: `arrange()`.

* Pick variables (columns) by their names: `select()`.

* Create new variables with functions of existing variables: `mutate()`.

* Collapse many values down to a single summary: `summarise()`.

in conjunction with __`group_by()`__ changes the scope of each function from operating on the entire dataset to operating on it group-by-group.

# Manipulate rows (cases)

## Filter rows with `filter()`

- Flights on Jan 1st:
    ```{r}
    # same as filter(flights, month == 1 & day == 1)
    filter(flights, month == 1, day == 1)
    ```

----

- Flights in Nov or Dec:
    ```{r}
    filter(flights, month == 11 | month == 12)
    ```
    
    equavalent as
    ```{r}
    filter(flights, month %in% c(11, 12))
    ```

## Missing values

`filter()` only includes rows where the condition is `TRUE`; it excludes both `FALSE` and `NA` values.

If you want to preserve missing values, ask for them explicitly
```{r}
df <- tibble(x = c(1, NA, 3))
filter(df, x > 1)
filter(df, is.na(x) | x > 1)
```

## Remove rows with duplicate values

- One row from each month:
    ```{r}
    distinct(flights, month, .keep_all = TRUE)
    ```
    
- With `.keep_all = FALSE`, only distinct values of the variable are selected:
    ```{r}
    distinct(flights, month)
    ```

## Sample rows

- Randomly select `n` rows:
    ```{r}
    slice_sample(flights, n = 10, replace = TRUE)
    ```

----

- Randomly select fraction of rows:
    ```{r}
    slice_sample(flights, prop = 0.1, replace = TRUE)
    ```

## Select rows by position

- Select rows by position:
    ```{r}
    slice(flights, 1:5)
    ```

----

- Top `n` rows:
    ```{r}
    slice_head(flights, n = 5)
    ```
    Use `prop` argument to specify proportions instead
    
- Bottom `n` rows:
    ```{r}
    slice_tail(flights, n = 5)
    ```

## Arrange rows with `arrange()`

- Sort in ascending order:
    ```{r}
    arrange(flights, year, month, day)
    ```

----

- Sort in descending order:
    ```{r}
    arrange(flights, desc(arr_delay))
    ```

Missing values are always sorted at the end.

# Manipulate columns (variables)

## Select columns with `select()`

- Select columns by variable name:
    ```{r}
    select(flights, year, month, day)
    ```

- Pull values of _one_ column as a vector:
    ```{r, eval = FALSE}
    pull(flights, year)
    ```

----

- Select all columns between two variables (inclusive):
    ```{r}
    select(flights, year:day)
    ```

----

- Select all columns _except_ those between two variables:
    ```{r, }
    select(flights, -(year:day))
    ```

  Read the documentation of `select` to learn more about allowed operations:
    
    - `:` for selecting a range of consecutive variables
    
    - `!` for taking the complement of a set of variables
    
    - `&` and `|` for selecting the intersection or the union of two sets of variables
    
    - `c()` for combining selections
    

----

- Select columns by positions:
    ```{r}
    select(flights, seq(1, 10, by = 2))
    ```

----

- Move variables to the start of data frame:
    ```{r, }
    select(flights, time_hour, air_time, everything())
    ```

----
 
- Helper functions.

    * `starts_with("abc")`: matches names that begin with “abc”.

    * `ends_with("xyz")`: matches names that end with “xyz”.

    * `contains("ijk")`: matches names that contain “ijk”.

    * `matches("(.)\\1")`: selects variables that match a regular expression.

    * `num_range("x", 1:3)`: matches x1, x2 and x3.
    
    * `one_of()`




