Lab Session 1: Git & The Shell#

  • Statistics 159/259, Spring 2022

  • Prof. F. Pérez and GSI F. Sapienza, Department of Statistics, UC Berkeley.

  • 01/27/2023

Menu for today:

  1. Recap on Lab 00. Last lab we did a quick overview of how to define functions and write errors and exceptions. As an option assignment, you can complete homework 00 to practice your Python skills. We will also see how the submission of homeworks will work.

  2. JupyterLab and JupyterHub.

  3. Using the Unix Shell. We are going to manipulate files and directories in GitHub but using the unit shell.

  4. Managing git repositories in local.

Useful links:

An overview of Jupyter tools#

If you are not familiar with Jupyter notebooks, you can take a look into the quick, practical introduction to the Jupyter Notebook posted in the course website. These are tools we are going to use during the semester, so it is a good idea to start using them from the beginning.

Some basic notions on Jupyter notebook include:

  • iPython: beyond plain Python

  • Different types of cells (code, markdowm, raw)

  • Navigation between cells.

  • The Kernel: what is it? What are good practices when working in a notebook?

Notebooks are very flexible and allow you to include a lot of scientific and computing elements together: code, text, figures, equations, media.

Another important tool to learn how to use is iPython, which expands the traditional Python capabilities to make it more interactive. In a sense, we can think in iPython as a meta-language on top of Python that allow us to interact with Python code in a more complete way. An example of this are magic functions, that are indicated with the % operator. You can take a look to all the magic commands by entering %quickref in a cell.

Exercise No1#

We are going to follow the Cell Profiling exercise from the Introduction to Jupyter and JupyterLab. This exercise is about cell profiling, but you will get practice in working with magics and cells.

Copy-paste the following code into a cell:

import numpy as np
import matplotlib.pyplot as plt

def step():
    import random
    return 1. if random.random() > .5 else -1.

def walk(n):
    x = np.zeros(n)
    dx = 1. / n
    for i in range(n - 1):
        x_new = x[i] + dx * step()
        if x_new > 5e-3:
            x[i + 1] = 0.
        else:
            x[i + 1] = x_new
    return x

n = 100000
x = walk(n)

  1. Split up the functions over 4 cells (either via Edit menu or keyboard shortcut Ctrl-Shift-minus).

  2. Plot the random walk trajectory using plt.plot(x).

  3. Time the execution of walk() with a line magic.

If you feel energetic, continue with the other exercises in the tutorial where you explore the %prun magic command.

Warming up with the Unix Shell#

Working from the shell may seem intimidating, but it is actually not that difficult.

hackerman

For this section, we are going to follow Chapters 2 & 3 of Research Software Engineering with Python. Less fancy than a graphical user interface (GUIs), command-line interfaces (CLI) are still very popular since they are highly efficient. The command shell is a REPL (read-evaluate-print-loop) software that allows us to interact with our machine. The most popular shell today is bash, and it is the one we will use in this course. What we call the terminal is the GUI we use to see the shell.

There we can find a comprehensive explanation about how to use different commands in the command shell.

These are some useful commands (for an extensive list of commands, refer to this Bash Shell Reference)

  • whoami

  • pwd: Print Working Directory

  • ls: Listing

  • cd: Change Directory

  • mkdir: Make Directory

  • mv: Move. This command can be also used to rename a file.

  • cp: Copy

  • rm. Remove

  • touch. Touches a file that already exists. If not, it creates a new one.

  • cat. Reads a file and prints it.

Remember that we can always communicate with the Shell on a Jupyter Notebook by adding ! at the beginning of a line or with the magic command %%bash just at the beginning of the cell. We can also add flags to these commands (usually indicated with -) as arguments for these functions.

There are also shortcuts to move in the shell. For example, Ctrl+A and Ctrl+E move the cursor to the beginning and end of the line, respectively, and Ctrl+K deletes text in front of the cursor.

Exercise No2#

We are going to do some basic manipulation of files and folders from the terminal.

  1. Open a terminal using the launcher (+ bottom in the upper left corner).

  2. Perform some basic scanning about where you are right now. Try whoami, pwd and ls.

  3. Create a new folder called folder_test using the mkdir command.

  4. Navigate inside that folder using cd. If you effectively moved inside the folder, you should see the changes in the current working directory reflected if you use pwd.

  5. Let’s create a new file in this folder. We are going to do this in three different ways:

    1. Create a text file text1.txt using the file browser in JypyterLab.

    2. Create a text file text2.txt using the echo function. You will need to explore how to use the > special character.

    3. Create a new file text3.txt using the touch function. This will just create an empty file. Try to open it with the file browser and add some content inside it. Save your changes and print its contents from the terminal using cat.

  6. Change the name of text3.txt.

  7. Comeback to your home directory and delete the folder folder_test with all the files inside (you will need to explore what the flag -r stands for).

Exercise No3#

Now, let’s write a small bash script that we can later execute in the terminal and that combines some of the operations we did in the previous exercise. Create a file called test.sh with the following. content:

#!/bin/bash

# Create a new text file with "touch"

# Add some content in that file with "echo". Do we really need "touch" before this?

# Print the contents of the file with "cat"

The header of the file has the shebang (#!) that indicates that this is an executable file. Complete the spaces after each commented line with the corresponding instruction. Then, execute the bash script by entering

./test.sh

from the terminal. Does it work? It is (very) likely you may need to change the permissions to the file in order to execute it. Explore the chmod command in bash in order to make the file test.sh executable.

A note about text editors#

There are different text editors we can use, both in our local and remote machines. You can launch text editors and manipulate them directly from the shell. It is also useful to know that you can change the default editor used by git in your computer (we will come back to this later).

Some popular text editors you may (and probably will) encounter include

  1. Vi. This is the default editor in git, so it is important to know how to manipulate it.

  2. Nano

  3. emacs

To open a text file with any of these editors, you enter <editor> <filename> from the terminal (fir example, vi text.txt).

Unless you work a lot with the text editor, no one remembers all the line commands to manipulate these text editors. However, it is useful to know at least the basic commands so we never get lost when we encounter one of them.

Command

Vi

Nano

emacs

Exit

:q

Ctrl+X

Ctrl+X Ctrl+C

Save

:w

Ctrl+S

Ctrl+X Ctrl+S

For Vi, notice that we can navigate through the text but not write until we enter i (insert). At the same time, when we want to exit the editing mode, we need to enter esc.

Git, always git#

We will be practicing the file and directory commands for manipulations of files inside a local test repository. During the lecture we walk through the Git Tutorial). Here, we are going to follow similar steps but using just the terminal.

Exercise No4#

Basic git manipulation:

  1. Create a new folder called git_test and move inside that folder.

  2. Initialize a git repository with git init.

  3. Create a new file inside the folder.

  4. Make git keeping track of the changes in that file (git add) and make an initial commit to the repository.

  5. Make changes to the same file and commit those changes. This time, instead of using git commit -m"my commit" use just git commit. This will launch the default git editor from where you can write and save your commit message.

  6. Change the editor used in git with git config --global core.editor <editor>. Repeat the procedure with the different editors listed above (vi, nano, emacs). Which editor do you find more easy to use?

  7. What do we do if we enter a wrong commit message? Try using git commit --amend.

Extra: Midnight commander#

Having fun with the terminal and want to explore another GUI to navigate through your system? This is how the midnight commander looks like.

mc

The same allows you to do the same manipulations we covered before, but with a nicer visual interface. You can open the midnight commander from the terminal by just entering mc. Here you have a manual of how to use it. Learning how to use it is a little bit more challenging than bash, but very useful, especially when we want to manipulate multiple files between different directories.