PPOL670 | Introduction to Data Science for Public Policy Week 3 Reproducibility in Practice

# PPOL670 | Introduction to Data Science for Public Policy Week 3 Reproducibility in Practice 
###  Prof. Eric Dunford  ◆  Georgetown University  ◆  McCourt School of Public Policy  ◆  <a href="mailto:eric.dunford@georgetown.edu" class="email">eric.dunford@georgetown.edu</a>

---

<div class="slide-footer"> 
PPOL670 | Introduction to Data Science for Public Policy

&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;

Week 3

&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;

Reproducibility in Practice

</div>

---
class: outline

# Outline for Today

![:space 3]

- More on **_programming in `R`_**
  
  - **_![:text_color darkred](indexing)_** and **_![:text_color darkred](iteration)_**
  
  - **_![:text_color darkred](control statements)_**

- Using **_`R` Markdown_** to generate reproducible documents.

- Generating **_Reproducible Examples_**

---

# Iteration

---

### Indexing Refresher

Recall that we can index data objects in `R`. By taping into each object's structure and using integers to extract data that belong to specific **positions**.

```r
x = c(-5:5)
x
```

```
##  [1] -5 -4 -3 -2 -1  0  1  2  3  4  5
```

```r
x[1]
```

```
## [1] -5
```

```r
x[9]
```

```
## [1] 3
```

---

### Indexing Refresher

We can call hole ranges of numbers using colon (`:`) operator.

```r
x[4:8]
```

```
## [1] -2 -1  0  1  2
```

And when using the negative, we can use indices as exclusion calls (i.e. "give me back everything _but_ these positions")

```r
x[-4:-8]
```

```
## [1] -5 -4 -3  3  4  5
```

We can also use specific index position to extract elements out of the original order.

```r
x[c(1,10,5,2)]
```

```
## [1] -5  4 -1 -4
```

---

### Indexing Refresher

Likewise, we can do the same with `data.frame` or `matric` objects -- except this time we have two index positions to keep track of: `rows` and `columns`.

```r
str(mtcars)
```

```
## 'data.frame':	32 obs. of  11 variables:
##  $ mpg : num  21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
##  $ cyl : num  6 6 4 6 8 6 8 4 4 6 ...
##  $ disp: num  160 160 108 258 360 ...
##  $ hp  : num  110 110 93 110 175 105 245 62 95 123 ...
##  $ drat: num  3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
##  $ wt  : num  2.62 2.88 2.32 3.21 3.44 ...
##  $ qsec: num  16.5 17 18.6 19.4 17 ...
##  $ vs  : num  0 0 1 1 0 1 0 1 1 1 ...
##  $ am  : num  1 1 1 0 0 0 0 0 0 0 ...
##  $ gear: num  4 4 4 3 3 3 3 4 4 4 ...
##  $ carb: num  4 4 1 1 2 1 4 2 2 4 ...
```

---

### Indexing Refresher

![:space 3]

```r
mtcars[1,]
```

```
##           mpg cyl disp  hp drat   wt  qsec vs am gear carb
## Mazda RX4  21   6  160 110  3.9 2.62 16.46  0  1    4    4
```

```r
mtcars[,1]
```

```
##  [1] 21.0 21.0 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 17.8 16.4 17.3 15.2 10.4
## [16] 10.4 14.7 32.4 30.4 33.9 21.5 15.5 15.2 13.3 19.2 27.3 26.0 30.4 15.8 19.7
## [31] 15.0 21.4
```

```r
mtcars[1:2,1:2]
```

```
##               mpg cyl
## Mazda RX4      21   6
## Mazda RX4 Wag  21   6
```

---

## `for` loops

A **loop** is offers a way to iterate through a series of items stored as data object in `R`.

The below codes pulls out each item once at a time, stores it in the object `i` and then prints it.

```r
items <- c("grapes","bananas","chocolate","bread")

for(i in items){
  print(i)
}
```

```
## [1] "grapes"
## [1] "bananas"
## [1] "chocolate"
## [1] "bread"
```

---

The code in the last slide is equivalent to this.

```r
i <- items[1]
print(i)
```

```
## [1] "grapes"
```

```r
i <- items[2]
print(i)
```

```
## [1] "bananas"
```

```r
i <- items[3]
print(i)
```

```
## [1] "chocolate"
```

```r
i <- items[4]
print(i)
```

```
## [1] "bread"
```

---

### `for` loop components

There are two thing really important features of a loop.

1. the **_`for()` function_** which we use to specify 
  
  a. what object we're drawing from and 
  
  b. what object we are writing to.

```{}
            for( i  in  items  )
                 ^        ^
                 |        |___ object we are drawing from
                 |
          obj. we write each item to 
```

---

### `for` loop components

Second, the **_brackets `{}`_**. Inside the brackets we house the code that is going to happen each iteration.

```{}
 for( i in items ){
 |~~~~~~~~~~~~~~~~| 
 |~~~~~~~~~~~~~~~~|
 |~~~~~~~~~~~~~~~~| code we need perform on each iteration.
 |~~~~~~~~~~~~~~~~|
 |~~~~~~~~~~~~~~~~|
 }
```

Together, `for` loops allow us to repeat complicated processes. This prevents us from having to arduously repeat the same chunk of code over and over again.

---

### `for` loop components

```r
# Create an array of numbers
numbers = c(10,100,-1000,345,-7,999,21345,444457)

# Iterate from the 2nd index until the last
for(i in 2:length(numbers)){
 
 # Multiply the two numbers together 
 new_number <- numbers[i]*numbers[i-1] 
 
 print(new_number) # print the numbers 
}
```

```
## [1] 1000
## [1] -1e+05
## [1] -345000
## [1] -2415
## [1] -6993
## [1] 21323655
## [1] 9486934665
```

---

## `while()` loops

Similar to `for` loops, `while()` loops only stop when some condition is met. As long as the statement within `while()` is `TRUE`, will the loop keep looping.

In the below example, as long as `i` is less than `5` the loop will keep looping.

```r
i <- 0
while( i < 5 ){
 print(i)
 i <- i + 1
}
```

```
## [1] 0
## [1] 1
## [1] 2
## [1] 3
## [1] 4
```

---

### `for()` loops and storing output

```r
letters
```

```
##  [1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j" "k" "l" "m" "n" "o" "p" "q" "r" "s"
## [20] "t" "u" "v" "w" "x" "y" "z"
```

```r
# Container with 10 spaces
container <- rep(0,10)
container
```

```
##  [1] 0 0 0 0 0 0 0 0 0 0
```

```r
# Each loop, we store the output of some code.
for(i in 1:10){
 container[i] <- paste0(letters[i],letters[i+1])
}
container
```

```
##  [1] "ab" "bc" "cd" "de" "ef" "fg" "gh" "hi" "ij" "jk"
```

---

### Container &rarr; any non-scalar data structure

```r
dat_container <- as.data.frame(matrix(0, nrow = 5,ncol = 2))
dat_container
```

```
##   V1 V2
## 1  0  0
## 2  0  0
## 3  0  0
## 4  0  0
## 5  0  0
```

```r
for( i in 1:5){
 dat_container[i,1] <- i
 dat_container[i,2] <- letters[i]
}
dat_container
```

```
##   V1 V2
## 1  1  a
## 2  2  b
## 3  3  c
## 4  4  d
## 5  5  e
```

---

## `for()` loops and binding output

Likewise, we can build objects up by binding them together.

```r
containter2 <- c()
containter2
```

```
## NULL
```

```r
tmp_data <- data.frame(v1=1,v2=2)
tmp_data
```

```
##   v1 v2
## 1  1  2
```

```r
containter2 <- rbind(containter2,tmp_data)
containter2 <- rbind(containter2,tmp_data)
containter2 
```

```
##   v1 v2
## 1  1  2
## 2  1  2
```

---

## `for()` loops and binding output

```r
containter2 <- c()
for( i in 1:10){
 
 tmp_data <- data.frame(v1=i,v2=letters[i])
 
 containter2 <- rbind(containter2,tmp_data)
 
}
containter2
```

```
##    v1 v2
## 1   1  a
## 2   2  b
## 3   3  c
## 4   4  d
## 5   5  e
## 6   6  f
## 7   7  g
## 8   8  h
## 9   9  i
## 10 10  j
```

---

## Which is most efficient?

![:space 5]

**Contender 1**: Filling a pre-defined container

```r
require(tictoc) # for counting how long it takes R to run
N = 5000
tic()
dat_container <- as.data.frame(matrix(0, nrow = N,ncol = 2))
for( i in 1:N){
 dat_container[i,1] <- i
 dat_container[i,2] <- 1/i
}
toc()
```

```
## 0.694 sec elapsed
```

---

## Which is most efficient?

![:space 5]

**Contender 2**: Binding each iteration

```r
N = 5000
tic()
dat_container <- c()
for( i in 1:N){
 tmp <- data.frame(V1 = i,V2 = 1/i)
 dat_container <- rbind(dat_container,tmp)
}
toc()
```

```
## 2.527 sec elapsed
```

---

# Control Statements

---

## `if` - `else`

```r
if(TRUE){
  print("Hello")
}else{
  print("Goodbye")
}
```

```
## [1] "Hello"
```

```r
if(FALSE){
  print("Hello")
}else{
  print("Goodbye")
}
```

```
## [1] "Goodbye"
```

---

**_Boolean statement dictates whether one code chunk is executed, or another is executed_**.

We can combine these conditionals with `for()` loops to powerful effect.

```r
for(i in 1:10){
 if(i <= 5){
 print("Hello")
 }else{
 print("Goodbye")
 }
}
```

```
## [1] "Hello"
## [1] "Hello"
## [1] "Hello"
## [1] "Hello"
## [1] "Hello"
## [1] "Goodbye"
## [1] "Goodbye"
## [1] "Goodbye"
## [1] "Goodbye"
## [1] "Goodbye"
```

---

## `ifelse()`

Same idea just vectorized.

```r
ifelse(T,"Hello","Goodbye")
```

```
## [1] "Hello"
```

```r
ifelse(F,"Hello","Goodbye")
```

```
## [1] "Goodbye"
```

```r
x <- 1:10
ifelse(x<5,"Hello","Goodbye")
```

```
##  [1] "Hello"   "Hello"   "Hello"   "Hello"   "Goodbye" "Goodbye" "Goodbye"
##  [8] "Goodbye" "Goodbye" "Goodbye"
```

---

# `R` Markdown

---

## Toward Reproducibility
 
 
 
Throughout the course, we will use `Rmarkdown` (or otherwise referred to as `R` "notebooks") to write code and to document our analyses.

`Rmarkdown` embodies all the principles we discussed above. Allowing for use to implement best practices in our research and empirical work.

For the "pitch" and valuable tutorials, see here

## <center> [`R` Markdown](https://rmarkdown.rstudio.com/lesson-1.html) </center>

---

# R Notebooks

---

# R Notebooks

---

# R Notebooks

---

# R Notebooks

---

# R Notebooks

---

### Compiling documents

To turn code into a report (`.html`,`.pdf`,`.doc`, ect) we need to **_knit_** (or "compile" the document).

The YAML headers gives `R` the instructions for how to do this. Specifically, we tell `R` what type of output we want.

`.html`
```
    output: html_document 
```
`.docx`
```
    output: word_document
```
`.pdf` ... requires a [LaTex distribution](https://miktex.org/download)
```
    output: pdf_document
```

---

### Compiling documents

To turn code into a report (`.html`,`.pdf`,`.doc`, ect) we need to **_knit_** (or "compile" the document).

The YAML headers gives `R` the instructions for how to do this. Specifically, we tell `R` what type of output we want.

`.nb.html`
```
output: html_notebook
```

An **_`R` Markdown Notebook_** allows you to write code and then see the rendered code in _real time_.

---

### Compiling documents

We can **`knit`** a R Markdown document in one of three ways:

![:space 2]

i. click the `Knit` button in `RStudio`

![:space 2]

ii. Use the keyboard shortcut

- Mac: `command + shift + k`
- Windows: `control + shift + k`

![:space 2]

ii. Knit the document using the `knit()` function in the `knitr` package.

---

### Using Markdown

![:space 10]

# Header

## Header

### Header

#### Header
```
]

]
]

---

### Using Markdown

![:space 10]

```
**Bold** or _italize_ text. Or **_both_**! ~~Even cross things out~~
```

```
> We can emphasize text and quotes.
```

> We can emphasize text and quotes.

---

### Using Markdown

![:space 10]

**Generate lists**
```
- Important point!
    + And another...
    + And another...
```

- Important point!
    + And another...
    + And another...
    
---

### Using Markdown

![:space 7]

**Enumerated Lists**

```
Things to do:

1. "Do Research"
2. Stare blankly at the wall and wonder why I came to grad school.
3. Eat chipotle.
```

Things to do:

1. "Do Research"
2. Stare blankly at the wall and wonder why I came to grad school.
3. Eat chipotle.

---

### Using Markdown

Even write some fancy math!

**inline**
```
...assume $x \sim N(\mu,\sigma)$ and $x \ge 0$ so that....
```
.center[&darr;]

...assume `$x \sim N(\mu,\sigma)$` and `$x \ge 0$` so that....

**its own line**
```
$$\hat{y_i} = \beta_0 + \beta_1x_i + \beta_2_ix^2 + \epsilon_i$$
```

`$$\hat{y_i} = \beta_0 + \beta_1x_i + \beta_2x_i^2 + \epsilon_i$$`
---

### Code Chunks

The real point of `R` Markdown is to embed your R code in your working script so that the document is **_reproducible_** and **_transparent_**.

**_To write code_**, we need to create a **_code chunk_**. We can do this by:

- click `insert`, and select an `R` code chunk.
- press `cmd` + `option/alt` + `i` or `ctrl` + `option/alt` + `i`

This will yield a discolored chunk that looks like this. Everything written in this chunk will be evaluated as `R` code. Everything written outside of it will be evaluated as prose.

---

### Code Chunks

![:space 15]

Chunk output can be customized with options, arguments supplied to chunk header. Knitr provides almost [60 options](http://yihui.name/knitr/options/) that you can use to customize your code chunks.

---

### Code Chunks

.center[
|Option |Run code |	Show code |	Output	|Plots |	Messages |	Warnings| errors|
|:-----|:-----:|:-----|:-----:|:-----:|:-----:|:-----:|:-----:|
|`eval = FALSE` | &#10006; | | &#10006; |&#10006; | &#10006; |&#10006; |&#10006;
|`include = FALSE` | | &#10006; | &#10006;| &#10006;| &#10006;| &#10006;|&#10006;
|`echo = FALSE` |	| &#10006; | | | | | |
|`results = "hide"` | | |&#10006; | | | ||
|`fig.show = "hide"` | | | | &#10006;|  | | |
|`message = FALSE`	 | | | | |&#10006; | | |
|`warning = FALSE`	 | | | | | |&#10006; | |
|`error = FALSE`	 | | | | | | | &#10006; |
]

---

### YAML Header

Finally there are a bunch of different ways that the YAML can be set up. Different configurations yield different layouts.

As we already saw, we can change how the document is compiled.

```
---
title: "Markdown Basics"
author: "Prof. Dunford"
date: "Fall 2019"
output: html_notebook
---
```

```
---
title: "Markdown Basics"
author: "Prof. Dunford"
date: "Fall 2019"
output: pdf_notebook
---
```

---

### YAML Header

We can customize the YAML to include different output themes, table of contents, parameters, and more!

```
---
title: "Markdown Basics"
author: "Prof. Dunford"
date: "Fall 2019"
output:
  html_document:
    theme: spacelab
    highlight: espresso
    toc: true
    toc_depth: 2
---
```

---

## Best practices

### 1. Code chunks should be broken up

### 2. No excessive output
  
  - i.e don't print of pages and pages of a data frame.
  
### 3. Figures should be appropriately sized for the rendered document

### 4. All data and code should be self-contained

- Given the data and the `.Rmd` file, the `R` Markdown down document should knit.

---

# Reprex

---

### Life as an new `R` user....

So you ran into an issue...

![:space 5]

You can't find answer on the internet...

![:space 5]

![:space 5]

When asking for help, you need to make the problem as transparent and easy to work through as possible.

---

### Generating reproducible examples

![:space 5]

```r
# install.packages("reprex")
require(reprex)
```

---

### Generating reproducible examples

**Steps**:

1. **_Reduce the problem down into a workable example_**.
  
  - Small toy example that doesn't require loading in data.
  
  - Replicate the issue.
  
2. **_Copy that code_**.

3. **_Then run `reprex()` in the consol_**.

- a html rendering of the example will render.
  
  - everything you need to post will be copied to your clipboard
  
---

### Generating a small toy example

![:space 5 ]

You'll need to run this in your console.

```r
# Copy this...
x = 1:5
for(i in x){
  print(i)
}

# Then run this in the console...
reprex()

# Finally, click 'paste' in where ever you're 
# posting the issue (e.g. slack)
```

---

### Output examples for any venue...

**_Github_** (or Slack!)

```r
reprex(venue = "gh")
```

**_Stack Overflow_**

```r
reprex(venue = "so")
```

**_HTML only slides_**

```r
reprex(venue = "html")
```

Finally, produce an example with all your **_session info_** (really useful for Stack Overflow)

```r
reprex(venue = "gh",si = T) 
```

---

### Generating an example of the data

`reprex` requires that **_all the code and data exist in the example_**. This is so that one can reproduce it by copying and pasting the example into their R console.

Your **_issue might be due to something wrong with your data_**, so often, it helps to give a little sample of the data you're using.

To reproduce the problem, we need some **data**:

- Example code without data only gets us halfway.

- Example code with **all the data** isn't practical and/or useful (try copying and pasting a gig of data)

- Need a **_small, representative subset_** of data that is concise enough to copy and paste but capable of reproducing the error.

---

### Generating an example of the data

![:space 5]

```r
# install.packages("datapasta")
require(datapasta)
```

---

### Generating an example of the data

Steps for **generating reproducible data** (for your reproducible example):

1. Generate a **_small subset_** of data (when you're copying and pasting, size matters)

- use `head()`, `tail()` or sample randomly (we'll learn this next time!)
  
2. Break the data down into a `tribble` or `data.frame` (copy/paste friendly version of the data) using:
  - `df_format()` &rarr; copies a data.frame structure to clipboard
  - `tribble_format()` &rarr; copies a tibble structure to clipboard

3. **_Paste_** this raw data **_at the top of the code_** you're aiming to reproduce.

---