JupyterLab + SoS Suite setup
Install miniconda3 the Python development environment
We recommend using
anaconda and customize your installation as needed after install this minimal version of
To install please follow instructions on this page. Please go for
After you successfully installed the latest version of
miniconda3, please follow prompts below to setup
a JupyterLab + SoS Suite environment for daily computing.
Note: maybe you already have a version of
miniconda on your computer. If you are very familiar with
conda then please try to work with your existing version by either upgrading or create separate
env under it to install additional packages. You might also want to start afresh and retire your older version (but keep the installation around for a while just in case). To do so, find in your
~/.bash_profile a line like this:
that actually points to the folder you installed your
anaconda3 or earlier version of
miniconda3. You can remove this line and restart your bash terminal to enable the setting.
Alternatively, a simpler but less elegant approach is to rename your
anaconda3 folder to, say
pip for package installation
miniconda there are two ways to install Python packages: either using
conda install or
I wouldn’t discuss too much details on what each does and pros and cons. I’d just say that 1) it is recommended to consistently use either
pip and not a combination of them, and 2) I recommend using
(Do not) use
conda to install R and R packages
From my experience, this is not recommended — it creates more issues than convenience at least to me. On a cluster you can try to load the R software that the cluster system has already installed, then install packages to your home directory. You should be asked to set or confirm the path to install R packages to in your
HOME directory. If not … <FIXME: instructions to set that path manually>
Jupyter Notebook and kernels
pip install in commands below generates timeout errors on your cluster system,
- On Columbia CUMC cluster, you need to run the commands below to set network proxy:
export http_proxy=http://bcp3.cumc.columbia.edu:8080 export https_proxy=http://bcp3.cumc.columbia.edu:8080
- If you are in China you might need to try a different mirror, depending on your location. For example use a mirror at Tsinghua University,
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple ..., may help. You can also configure
pypito use alternative mirrors by default.
pip install notebook jupyterlab jupyter_contrib_nbextensions
pip install bash_kernel --no-cache-dir python -m bash_kernel.install
pip install markdown-kernel --no-cache-dir python -m markdown_kernel.install
To install R kernel for Jupyter after you installed R,
R --slave -e "IRkernel::installspec()"
If you get a complaint that
IRkernel package is not available, please install it in R, eg
install.packages('IRkernel'), before you run the command above.
A ipynb to docx converter
This will allow you to save
ipynb file to a
docx file for various purposes
pip install jupyter-docx-bundler --no-cache-dir jupyter bundlerextension enable --py jupyter_docx_bundler --sys-prefix
nbdime to work with git
This will override the default
git diff and display better the changes to IPython notebooks
pip install nbdime nbdime config-git --enable --global
pip install docker markdown wand graphviz imageio pillow nbformat feather-format --no-cache-dir pip install sos sos-notebook sos-r sos-python sos-bash -U --no-cache-dir python -m sos_notebook.install pip install jupyterlab-sos -U --no-cache-dir
What if Jupyter kernels keep dying?
This happened to us several times, and solution on this ticket was the rescue.
Notice: docker cannot be installed on many HPC cluster environments due to security reasons. We may use
singularity instead of
docker to run some applications on cluster. But still having docker configured on your laptop or desktop computer can be useful.
We use Docker a lot running various software that are hard to install. SoS also provides an interface to run Docker images.
To install Docker on Linux,
- Run commands below:
curl -fsSL get.docker.com -o get-docker.sh sudo sh get-docker.sh sudo usermod -aG docker $USER
- Log out and log back in (no need to reboot computer)
To install it on MacOS, visit https://www.docker.com/products/docker-desktop and download & install the Docker Desktop installer.