Using psh magics

pshnb adds %bash and %%bash functions to Jupyter and IPython, which execute expressions in a persistent shell.

Installation

Install pshnb with:

pip install pshnb

Once that’s complete, you can install the magics to all IPython and Jupyter sessions automatically by running in your terminal:

pshnb_install

If you don’t want to install the automatic magic, you can instead run in notebooks where you want to use it:

%load_ext pshnb

What’s the point?

In jupyter and ipython, you can run a shell command using the ! prefix:

!pwd

/Users/jhoward/aai-ws/pshnb

However, each time you run this command, a new shell is created and then removed, so for instance, cd doesn’t actually do anything if you run another command afterwards:

!cd ..
!pwd

/Users/jhoward/aai-ws/pshnb

As you see from the !pwd output, the directory hasn’t actually changed!

%bash, on the other hand, creates a persistent shell, which solves this problem:

%load_ext pshnb

%bash pwd

/Users/jhoward/aai-ws/pshnb

%bash cd ..
%bash pwd

/Users/jhoward/aai-ws

With %bash, you can implement, and document in notebooks, multi-step stateful shell interactions, including setting environment variables, sourcing scripts, and changing directories.

Features

Cell magic

You can use the %%bash cell magic to run multi-line shell commands, such as here-docs. For instance:

%%bash
cat > tmp << EOF
hi
there
EOF

This creates a file called tmp containing two lines. Let’s check it worked, and then remove it – as you see, you can also use the cell magic to run multiple commands:

%%bash
cat tmp
rm tmp

hi
there

Variable expansion

You can pipe commands together just like in a regular shell, and use standard unix utilities like head to process the output. For instance, here we show just the first 3 lines of the directory listing:

%bash ls | head -3

_nbs
_proc
addnew.py

You can use Python variables in your shell commands by prefixing them with @{}. For instance, here we create a variable n and then display it using echo:

n = 2

%bash echo @{n}

2

Here we use n to show just the first two entries from the directory listing:

%bash ls | head -@{n}

_nbs
_proc

Background tasks

You can run commands in the background in the shell by adding & at the end of a command. The parentheses (...) group commands together to run as one unit. In this example, we first print “starting”, and then create a background process that will wait for 1 second (using sleep 1) and then print “finished”. The shell immediately shows us “starting” and tells us it created background process number 1 (with a process ID):

%%bash
echo starting
(sleep 1; echo finished) &

starting
[1] 18618

For this demonstration, we wait for 1.1 seconds (slightly longer than the background process needs). During this time, the background process will complete in the background. But we won’t see any output from it yet.

from time import sleep
sleep(1.1)

The next time we run any psh magic we will also see any output that has occurred in our persistent shell since the last command. Run %bash by itself to only see those updates, e.g here we see that “finished” was printed, and the shell tells us that background job 1 completed successfully.

%bash

finished

[1]+  Done                    ( sleep 1; echo finished )

Flags

You can get help on the %bash magic’s options using -h.

%bash -h

::

  %bash [-h] [-r [RESET]] [-o] [-x] [-X] [-s] [-S] [-t TIMEOUT]
            [command ...]

Run line or cell in persistent shell

positional arguments:
  command               The command to run

options:
  -h, --help            Show this help
  -r <[RESET]>, --reset <[RESET]>
                        Reset the shell interpreter (optionally choose shell)
  -o, --obj             Return this magic object
  -x, --expand          Enable variable expansion
  -X, --no-expand       Disable variable expansion
  -s, --sudo            Enable sudo
  -S, --no-sudo         Disable sudo
  -t TIMEOUT, --timeout TIMEOUT
                        Set timeout in seconds

You can reset the shell to its initial state using the -r flag. Let’s first check our current directory:

%bash pwd

/Users/jhoward/aai-ws

Now let’s reset the shell:

%bash -r

As you can see, after resetting we’re back in our starting directory:

%bash pwd

/Users/jhoward/aai-ws/pshnb

The -s flag enables sudo mode, which runs commands as the root user, and -S disables it. For instance, here we first enable sudo mode:

%bash -s

Then we can check which user we’re running as:

%bash whoami

root

As you can see, we’re now running as root. We can disable sudo mode:

%bash -S

And when we check again, we’re back to our regular user:

%bash whoami

jhoward

You can set a timeout (in seconds) using the -t flag, which will raise a TIMEOUT exception if a command takes too long. For instance, here we set a 1-second timeout:

%bash -t 1

Then we try to run a command that sleeps for 2 seconds – since this is longer than our timeout, we’ll get a timeout error:

from pexpect import TIMEOUT

try: get_ipython().run_line_magic('bash', 'sleep 2')
except TIMEOUT: print("timed out")

timed out