🛠️Installation Instructions

Start by cloning the CryoBench git repository; note that we also have to fetch the codebases for some of the example methods through their git submodules:

git clone --recurse-submodules [e-mail]:ml-struct-bio/CryoBench.git

You will also have to install ChimeraX, which can be done by downloading the correct version for your operating system from their website. For CryoBench to find your installation, create an environment variable pointing to its location:

export CHIMERAX_PATH="/myhome/software/chimerax-1.6.1/bin/ChimeraX"

To avoid having to specify this environment variable every time you load these environments, save the above export expression in your .bashrc file! More info on .bashrc files can be found here.

Method-specific instructions

We developed CryoBench as a tool for quantitatively evaluating a set of reconstruction methods. Shown here are instructions for creating a conda environment for running each of these example methods, as well as for measuring the quality of the reconstructions.

For each reconstruction method, we recommend using a separate conda environment for running the method itself and for using CryoBench on its outputs. This is because many methods — especially those related to cryoDRGN — have package dependencies whose required versions conflict with one another, or with those required by CryoBench!

Installing each method generally means following the instructions provided by the developers, while creating the corresponding CryoBench environment entails repeating the steps of the method installation as necessary and then installing CryoBench dependencies using the repository's requirements.txt file along with a compatible version of cryoDRGN.

The scripts we have provided for each method are generally designed to work with the output of a range of versions of the method, and to have a specific version of the method installed for CryoBench:

CryoDRGN

Create an environment for running cryoDRGN models. Here we specify a recent version; CryoBench is compatible with cryoDRGN v3.x outputs:

$ conda create --name cryodrgn_model python=3.10
$ conda activate cryodrgn_model
(cryodrgn_model)$ pip install 'cryodrgn==3.4.2'

Next, create an environment for running CryoBench analyses on cryoDRGN output using the requirements.txt and an older version of cryoDRGN compatible with CryoBench code:

$ conda create --name cryodrgn_bench python=3.10
$ conda activate cryodrgn_bench
(cryodrgn_bench)$ pip install -r CryoBench/requirements.txt
(cryodrgn_bench)$ pip install git+https://github.com/ml-struct-bio/cryodrgn.git@2.0.0-beta

DRGN-AI

For running DRGN-AI reconstruction we use the latest version available through Git. CryoBench supports handling DRGN-AI output up to v0.3.1:

$ conda create --name drgnai_model python=3.10
$ conda activate drgnai_model
(drgnai_model)$ pip install git+https://github.com/ml-struct-bio/drgnai.git

For running CryoBench analyses on DRGN-AI output we use a specific version of DRGN-AI, along with the necessary CryoBench dependencies as described above:

$ conda create --name drgnai_bench python=3.10
$ conda activate drgnai_bench
(drgnai_bench)$ pip install git+https://github.com/ml-struct-bio/drgnai.git@0.3.1
(drgnai_bench)$ pip install -r CryoBench/requirements.txt
(drgnai_bench)$ pip install git+https://github.com/ml-struct-bio/cryodrgn.git@2.0.0-beta

OPUS-DSD

For this package we have to use their custom dependency list, and then install the package from the cloned submodule using pip install -e (non-developer mode does not work). CryoBench has been tested with OPUS-DSD v1.0.0; this is the version that is attached as a submodule under metrics/methods/opusDSD:

$ conda env create --name opusdsd_model -f CryoBench/metrics/methods/opusDSD/environmentcu11torch11.yml
$ conda activate opusdsd_model
(opusdsd_model)$ pip install -e CryoBench/metrics/methods/opusDSD/

We repeat the same steps for CryoBench analyses, also installing CryoBench dependencies at the end. In this case we can use the version of cryoDRGN installed with OPUS-DSD itself for CryoBench as well:

$ conda env create --name opusdsd_bench -f CryoBench/metrics/methods/opusDSD/environmentcu11torch11.yml
$ conda activate opusdsd_bench
(opusdsd_bench)$ pip install -e CryoBench/metrics/methods/opusDSD/
(opusdsd_bench)$ pip install -r CryoBench/requirements.txt

RECOVAR

In the case of RECOVAR, as per their instructions, we have to install a specific version of jax on top of their custom requirements list, along with setting some bash environment variables and creating a special ipykernel installation. You may find it necessary to install a different version of jax that is compatible with your version of CUDA:

$ conda create --name recovar_model python=3.11
$ conda activate recovar_model
(recovar_model)$ pip install -U "jax[cuda12_pip]"==0.4.23 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
(recovar_model)$ pip install --no-deps -r CryoBench/metrics/methods/recovar/recovar_install_requirements.txt
(recovar_model)$ python -m ipykernel install --user --name=recovar

# can also be added to .bashrc — you may have to change the CUDA version path as necessary for your system
(recovar_model)$ export PATH=/usr/local/cuda-12.6/bin:$PATH
(recovar_model)$ export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:$LD_LIBRARY_PATH

These environment variables have to be redefined every time the environment is loaded if they are not specified in e.g. your ~/.bashrc file!

CryoBench has been tested with a specific commit of the RECOVAR repository, which is also the commit used by the RECOVAR submodule within the CryoBench repo.

For running CryoBench we can avoid redefining the environment variables if creating the environment in the same session. Note that for installation to work correctly, the CryoBench/requirements.txt dependencies must be installed before the remaining RECOVAR dependencies:

$ conda create --name recovar_bench python=3.11
$ conda activate recovar_bench
(recovar_bench)$ pip install -r CryoBench/requirements.txt
(recovar_bench)$ pip install -U "jax[cuda12_pip]"==0.4.23 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
(recovar_bench)$ pip install --no-deps -r CryoBench/metrics/methods/recovar/recovar_install_requirements.txt
(recovar_bench)$ python -m ipykernel install --user --name=recovar

conda install "jaxlib=*=cuda" jax cuda-nvcc -c conda-forge -c nvidia is an alternative method of installing jax that should correctly detect the version compatible with your CUDA installation

CryoSPARC

CryoSPARC is typically accessed through a graphical interface installed by your system administrator, and not through a command-line interface as with the other example methods. We hence assume that such a GUI will be used to generate reconstruction output for subsequent examples.

For running CryoBench on outputs generated by CryoSparc, we create an environment similar to cryodrgn_benchabove — you can also instead use the same environment for both sets of analyses:

$ conda create --name csparc_bench python=3.10
$ conda activate csparc_bench
(cryodrgn_bench)$ pip install -r CryoBench/requirements.txt
(cryodrgn_bench)$ pip install git+https://github.com/ml-struct-bio/cryodrgn.git@2.0.0-beta
(cryodrgn_bench)$ pip install cryosparc-tools

You will also need to replace lines in cryosparc_3dflex.pyif doing analyses on 3D Flex outputs:

$ vim ~/dev/CryoBench/metrics/fsc/old/per_conf/cryosparc_3dflex.py

    23 # replace these as necessary with your CryoSPARC credentials
    24 license_id = None
    25 email = None
    26 password = None
    27 host = None
    28 run_lane = None

CryoBench has been tested using the output of CryoSPARC v4.6.0.