CCPEM-pipeliner

Installation

Once the package has been downloaded navigate into the ccpem-pipeliner directory and install the pipeliner with the command:

pip install -e .

Check the installation

Once the pipeliner is installed use the command check_setup.py to check that the setup is complete and the pipeliner can find the Relion programs it needs to run.

Documentation

Documentation is available online at: ccpem-pipeliner.readthedocs.io/en/latest/

To build the documentation yourself, install the documentation build requirements as follows:

pip install -r requirements-docs.txt

Then navigate to the ccpem-pipeliner/docs directory and run the command:

make html

Then open the file ccpem-pipeliner/docs/_build/html/index.html in a web browser to access the documentation.

Adding plugins

To add additional job plugins the plugin must first be written as a PipelinerJob object.
See the pipeliner documentation for a description of the format.

The plugin must then be placed in the Pipeliner/jobs/ folder or one of its subfolders, or create a new subfolder.

Once the plugin file is in place add it to the list of entry points in setup.cfg.

If setup.cfg has been modified update the pipeliner by running pip install -e . for the changes to take effect.

For Developers

It’s a good idea to work in a virtual environment for this project. Set one up as follows:

python3 -m venv venv/
source venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
pre-commit install

This project uses pre-commit to run Black for code formatting, flake8 for linting and some other simple checks, and mypy for type checks.

You might want to install Black separately yourself too, so you can run it from your IDE.

According to the flake8 documentation, flake8 should not stop a git commit from going ahead unless flake8.strict is set in the git config. That doesn’t actually seem to work: with the current configuration, commits fail if flake8 finds any problems. There are some flake8 warnings that have not been fixed yet, so to get around this, flake8 checks can be disabled: SKIP=flake8 git commit ...

Unit tests

Run the tests with pytest.

Some of the tests are quite slow or require some interaction so these tests are skipped by default. Set the environment variable PIPELINER_TEST_SLOW to a non-empty string to run the slower tests as well. Set the environment variable PIPELINER_TEST_INTERACTIVE to a non-empty string to also run the tests that require interaction.