Command Line Tools

The pipeliner command allows pipeliner functions to be run from the command line.

CCPEM Pipeliner command line utility

usage: pipeliner [-h] [--new_project [project name]]
                 [--available_jobs [search string]]
                 [--check_jobs [search string]]
                 [--run_job [run.job or job.star file]]
                 [--overwrite [job name]] [--continue_job [job name]]
                 [--abort_job [job name]]
                 [--print_command [run.job or job.star file]]
                 [--schedule_job [run.job, job.star file or job name]]
                 [--run_schedule] [--name [NAME]]
                 [--jobs [job name [job name ...]]] [--min_between [n]]
                 [--nr_repeats [1]] [--wait_sec_after [2]]
                 [--wait_min_before [0]] [--stop_schedule [STOP_SCHEDULE]]
                 [--delete_job [job name]] [--undelete_job [job name]]
                 [--set_alias [job name] [new alias]]
                 [--clear_alias [job name]] [--set_status [job name]
                 {finished, failed, aborted}]
                 [--cleanup [job name [job name ...]]] [--harsh]
                 [--validate_starfile [star file]]
                 [--convert_pipeline_file [_pipeline.star file]]
                 [--default_jobstar [job type]] [--relionstyle]
                 [--default_runjob [job type]] [--job_info [job type]]
                 [--empty_trash] [--job_references [job type]]
                 [--references_report [job name]]
                 [--metadata_report [terminal job]]
                 [--full_archive [terminal job name]]
                 [--simple_archive [terminal job name]]
                 [--empiar_deposition [terminal job name]]
                 [--onedep_deposition [terminal job name]]
                 [--deposition_id [Deposition id Assigned by the database]]
                 [--empiar_jobstar_file [Template file to update]]
                 [--empiar_movies] [--empiar_mics] [--empiar_parts]
                 [--empiar_corr_parts]

Arguments

--new_project

Initialize a new project in this directory. Project name is optional, if none is specified the project will be called ‘default’ (recommended)

--available_jobs

Show a list of all available jobs, optionally add a search string to limit the search, Leave blank to show all available job types

--check_jobs

Get a list of available and unavailable jobs with detailed info onthe programs they use and why they are unavailable (if applicable)

Running jobs

--run_job

Create a job using a run.job or job.star file to get the parameters

--overwrite

Use with –run_job to overwrite a current job rather than making a new job. Jobs can only be overwritten with the same job type

--continue_job

Continue a job that has been previously run. Edit the job’s continue_job.star file to modify parameters for the continuation

--abort_job

Abort a running job

--print_command

Read a job.star or run.job file and print the command(s) it would produce

--schedule_job

Add a job to list of scheduled jobs using a run.job or job.star file to get the parameters. If followed by a job.star file it will create a new job if followed by a job name it will schedule a continuation of that job

Executing Schedules

--run_schedule

Create a schedule choosing from the currently scheduled jobs and run it

Default: False

--name

(required) Enter a name for the new schedule

--jobs

(required) Enter the jobs that will be run. Make sure to list the jobs in the order which they should be run

--min_between

(optional) Wait at least this many minutes between jobs

Default: 0

--nr_repeats

(optional) Repeat the schedule this many times

Default: 1

--wait_sec_after

(optional) Wait this many seconds after finishing before starting the next job

Default: 2

--wait_min_before

(optional) Wait this many minutes before starting the schedule

Default: 0

--stop_schedule

(required) Enter a name for the schedule to be stopped

Deleting jobs

--delete_job

Remove job(s) and put in the trash, deletes the job and all of its child processes

Undeleting jobs

--undelete_job

Restore a deleted job and any of its deleted parent processes from the trash

Modifying jobs

--set_alias

Set the alias of a job

--clear_alias

Clear the alias of a job

--set_status

Set the status of a job; choose from ‘finished, failed, or aborted

Cleaning Up Job(s)

--cleanup

Delete intermediate files from these job(s) to save disk space; enter ALL to clean up all jobs

--harsh

Add this argument to –cleanup to delete even more files

Default: False

Utilities

--validate_starfile

Check a star file and make sure it is written in the correct format. If errors are found will attempt to fix them

--convert_pipeline_file

Convert a pipeline file from Relion 3.1 to the CCPEM pipeliner format

--default_jobstar

Make a job.star file with the default values for a specific job type for use with the ccpem-pipeliner

--relionstyle

OPTIONAL: Add this argument to –default_jobstar to write job.star files which are compatible with RELION 4.0. RELIONstyle job.star files are fully compatible with the pipeliner, pipeliner job.star files may havedifferences that cause bugs in RELION

Default: False

--default_runjob

Make a _run.job file with the default values for a specific job type. These files can also be used to run jobs and a more human readable

--job_info

Get info about a specific job type, including any reference(s)

--empty_trash

Delete the files in the trash. THIS CANNOT BE UNDONE!

Default: False

--job_references

Get literature reference(s) for a specific job type

--references_report

Get all the literature references for a project with the selected job as the termial job

Project Analysis

--metadata_report

Prepares a report in .json format for the terminal job and allof its parent jobs

Project Archiving

--full_archive

Create a full archive for a project. This will contain the entire job dirs for the stated terminal job and all of its parents

--simple_archive

Create a simple archive for a project. This will contain just the directory structure and parameter files for the stated terminal job andall of it’s parents along with a script to automatically re-run the project through the terminal job

Prepare EMDB/EMPIAR/PDB depositions

--empiar_deposition

Prepare a deposition for upload to the EMPIAR database

--onedep_deposition

Prepare a deposition for upload to the pdb/emdb databases

--deposition_id

[Optional] The deposition ID. This must be assigned by EBI. If not specified, a draft deposition will be created

--empiar_jobstar_file

(optional) A jobstar file from a pipeliner.deposition.empiar job. This contains data necessary from the deposition that cannot be automatically found. If not included, the deposition will only contain the data that can be found and will need to be manually completed

--empiar_movies

(optional) Add this argument to deposition to include unprocessed movies in the EMPIAR deposition

Default: False

--empiar_mics

(optional) Add this argument to deposition to include corrected micrographs in the EMPIAR deposition

Default: False

--empiar_parts

(optional) Add this argument to deposition to include particles in the EMPIAR deposition

Default: False

--empiar_corr_parts

(optional) Add this argument to deposition to include corrected (polished) particles in the EMPIAR deposition

Default: False