JupyterLab + SoS Suite setup

Install miniconda3 the Python development environment

We recommend using miniconda over anaconda and customize your installation as needed after install this minimal version of conda. To install please follow instructions on this page. Please go for miniconda3.

After you successfully installed the latest version of miniconda3, please follow prompts below to setup a JupyterLab + SoS Suite environment for daily computing.

Note: maybe you already have a version of anaconda or miniconda on your computer. If you are very familiar with conda then please try to work with your existing version by either upgrading or create separate env under it to install additional packages. You might also want to start afresh and retire your older version (but keep the installation around for a while just in case). To do so, find in your ~/.bashrc or ~/.bash_profile a line like this:

export PATH=$HOME/anaconda3/bin:$PATH

that actually points to the folder you installed your anaconda3 or earlier version of miniconda3. You can remove this line and restart your bash terminal to enable the setting.

Alternatively, a simpler but less elegant approach is to rename your anaconda3 folder to, say anaconda3_bak.

conda vs pip for package installation

With miniconda there are two ways to install Python packages: either using conda install or pip install. I wouldn’t discuss too much details on what each does and pros and cons. I’d just say that 1) it is recommended to consistently use either conda or pip and not a combination of them, and 2) I recommend using pip over conda.

(Do not) use conda to install R and R packages

From my experience, this is not recommended — it creates more issues than convenience at least to me. On a cluster you can try to load the R software that the cluster system has already installed, then install packages to your home directory. You should be asked to set or confirm the path to install R packages to in your HOME directory. If not … <FIXME: instructions to set that path manually>

Jupyter Notebook and kernels

Note: if pip install in commands below generates timeout errors on your cluster system,

• On Columbia CUMC cluster, you need to run the commands below to set network proxy:

export http_proxy=http://bcp3.cumc.columbia.edu:8080
export https_proxy=http://bcp3.cumc.columbia.edu:8080

• If you are in China you might need to try a different mirror, depending on your location. For example use a mirror at Tsinghua University, pip install -i https://pypi.tuna.tsinghua.edu.cn/simple ..., may help. You can also configure bash or pypi to use alternative mirrors by default.

Base notebook

pip install notebook jupyterlab jupyter_contrib_nbextensions

Bash kernel

pip install bash_kernel --no-cache-dir
python -m bash_kernel.install

Markdown kernel

pip install markdown-kernel --no-cache-dir
python -m markdown_kernel.install 

R kernel

You need to install R first. Here are some tips for Debian based Linux (possibly outdated). For MacOS you can download the R software installer from CRAN and install from there.

To install R kernel for Jupyter after you installed R,

R --slave -e "IRkernel::installspec()"

If you get a complaint that IRkernel package is not available, please install it in R, eg install.packages('IRkernel'), before you run the command above.

A ipynb to docx converter

This will allow you to save ipynb file to a docx file for various purposes

pip install jupyter-docx-bundler --no-cache-dir
jupyter bundlerextension enable --py jupyter_docx_bundler --sys-prefix

nbdime to work with git

This will override the default git diff and display better the changes to IPython notebooks

pip install nbdime
nbdime config-git --enable --global

SoS Suite

pip install docker markdown wand graphviz imageio pillow nbformat feather-format --no-cache-dir
pip install sos sos-notebook sos-r sos-python sos-bash -U --no-cache-dir
python -m sos_notebook.install
pip install jupyterlab-sos -U --no-cache-dir

What if Jupyter kernels keep dying?

This happened to us several times, and solution on this ticket was the rescue.

Install Docker

Notice: docker cannot be installed on many HPC cluster environments due to security reasons. We may use singularity instead of docker to run some applications on cluster. But still having docker configured on your laptop or desktop computer can be useful.

We use Docker a lot running various software that are hard to install. SoS also provides an interface to run Docker images.

To install Docker on Linux,

• Run commands below:

curl -fsSL get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

• Log out and log back in (no need to reboot computer)

To install it on MacOS, visit https://www.docker.com/products/docker-desktop and download & install the Docker Desktop installer.