Back to Course Website

The Signal and the Noise
Introduction to Statistical Learning

PPOL 564 | Data Science I | Foundations

Lecture Materials for Week 9

---

Learning Objectives

In the Asynchronous Lecture

• Discuss virtual environments how they work and why they’re useful.
• Get a general introduction to statistical learning.
• Talk about supervised learning problems.
• Think about cross-validation as a strategy to deal with over-fitting.

In the Synchronous Lecture

• General introduction to the scikit-learn api.
• Delve into data pre-processing, why we do it, and how to do it in sklearn
• Talk about how to implement cross-validation methods in sklearn
• Talk about ml pipelines and why they’re useful.

If you have any questions while watching the pre-recorded material, be sure to write them down and to bring them up during the synchronous portion of the lecture.

---

Synchronous Materials

• Lecture Scripts:
  • introduction-to-sklearn.py
  • introduction-to-data-preprocessing.py
  • introduction-to-CV.py
  • introduction-to-pipelines.py

---

Asynchronous Materials

The following tabs contain pre-recorded lecture materials for class this week. Please review these materials prior to the synchronous lecture.

Total time: Approx. 1 hour and 20 minutes

_

---

Virtual Environments

Code from the video

Setting up a virtual environment

# Bash/Unix 

# Create a folder where your project will be
mkdir my_project && cd my_project

# Create a virtual environment in your project 
python3 -m venv env

Note that this bit python3 -m venv creates the environment, and this bit env is the name we gave the environment.

Activating your virtual environment

# Bash/Unix 

# Assuming you're still in your project directory 
source env/bin/activate

You’ll note that the commandline prompt has changed. You can now install packages using pip. This python environment will only contain the modules that come with python’s standard library, so you’ll have to import any other modules. That’s the whole point!

# Bash/Unix 
pip install pandas

You can fire Python version up by activating the Python3 REPL.

# Bash/Unix 
python3

To deactivate your virtual environment just type:

# Bash/Unix 
deactivate

Getting your virtual environment to play along with Jupyter/Hydrogen

To be able to activate our virtual environment, we need to register this python kernel with jupyter.

First, activate your virtual environment

# Bash/Unix 
source env/bin/activate

Next, install ipykernel

# Bash/Unix 
python -m pip install ipykernel
python -m ipykernel install --user --name=env

The final bit there (--name=env) registers your virtual environment under a specific name (so give it a good name so you can remember which environment is associated with which project). When we run a jupyter notebook, we’ll see env as a kernel option.

Likewise, in Atom we can access our virtual environment kernel. We first need to update our kernels in hydrogen. Press cmd-shift-p (or cnt-shift-p on a Windows), find Hydrogen: Update Kernels (you can also do this from the packages/hydrogen menu). Finally, press cmd-shift-p again and select Hydrogen: Start Local Kernel. You’ll see your evnronment (which we named “env”) appear as a kernel option.

You can see all your available kernels running the following command.

# Bash/Unix 
jupyter kernelspec list

To uninstall your virtual environment kernel, just type

# Bash/Unix 
jupyter kernelspec uninstall env

Where again env is what we named our virtual environment.

This all seems involved, but the reproducibility benefits of virtual environments make the hassle well worth it, especially when you need to fire up a project a year or so later and all the dependencies may have changed.

---

Statistical Learning

Relevant Slides

---

Supervised Learning

Relevant Slides

---

Cross-Validation

Relevant Slides

---

 

---

The following materials were generated for students enrolled in PPOL564. Please do not distribute without permission.

ed769@georgetown.edu | www.ericdunford.com