Command Line Tools

CL_pipeline allows pipeliner functions to be run from the UNIX command line

CCPEM Pipeliner command line utility

usage: CL_pipeline [-h] [--new_project [project name]]
                   [--available_jobs [search string]]
                   [--run_job [run.job or job.star file]]
                   [--overwrite [job name]] [--continue_job [job name]]
                   [--create_interactive_job]
                   [--print_command [run.job or job.star file]]
                   [--schedule_job [run.job, job.star file or job name]]
                   [--run_schedule] [--name [NAME]]
                   [--jobs [job name [job name ...]]] [--min_between [n]]
                   [--nr_repeats [1]] [--wait_sec_after [2]]
                   [--wait_min_before [0]] [--stop_schedule [STOP_SCHEDULE]]
                   [--delete_job [job name]] [--undelete_job [job name]]
                   [--set_alias [job name] [new alias]]
                   [--clear_alias [job name]] [--set_status [job name]
                   {finished, failed, aborted}]
                   [--cleanup [job name [job name ...]]] [--harsh]
                   [--validate_starfile [star file]]
                   [--convert_pipeline_file [_pipeline.star file]]
                   [--default_jobstar [job type]] [--relionstyle]
                   [--default_runjob [job type]] [--job_info [job type]]
                   [--empty_trash] [--job_runtime [job name]]
                   [--metadata_report [terminal job]]
                   [--draw_flowchart [job name]] [--upstream] [--downstream]
                   [--interactive] [--full_archive [terminal job name]]
                   [--simple_archive [terminal job name]]

Arguments

--new_project

Initialize a new project in this directory. Project name is optional, if none is specified the project will be called ‘default’ (recommended)

--available_jobs

Show a list of all available jobs, optionally add a search string to limit the search, Leave blank to show all available job types

Running jobs

--run_job

Create a job using a run.job or job.star file to get the parameters

--overwrite

Use with –run_job to overwrite a current job rather than making a new job. Jobs can only be overwritten with the same job type

--continue_job

Continue a job that has been previously run.Edit the job’s continue_job.star file to modify parameters for the continuation

--create_interactive_job

Create a job.star file for any job type with an interactive dialog. Useful for situations where the GUI cannot open

Default: False

--print_command

Read a job.star or run.job file and print the command it would produce

--schedule_job

Add a job to list of scheduled jobs using a run.job or job.star file to get the parameters. If followed by a job.star file it will create a new job if followed by a job name it will schedule a continuation of that job

Executing Schedules

--run_schedule

Create a schedule choosing from the currently scheduled jobs and run it

Default: False

--name

(required) Enter a name for the new schedule

--jobs

(required) Enter the jobs that will be run. Make sure to list the jobs in the order which they should be run

--min_between

(optional) Wait at least this many minutes between jobs

Default: 0

--nr_repeats

(optional) Repeat the schedule this many times

Default: 1

--wait_sec_after

(optional) Wait this many seconds after finishing before starting the next job

Default: 2

--wait_min_before

(optional) Wait this many minutes before starting the schedule

Default: 0

--stop_schedule

(required) Enter a name for the schedule to be stopped

Deleting jobs

--delete_job

Remove job(s) and put in the trash, deletes the job and all of its child processes

Undeleting jobs

--undelete_job

Restore a deleted job and any of its deleted parent processes from the trash

Modifying jobs

--set_alias

Set the alias of a job

--clear_alias

Clear the alias of a job

--set_status

Set the status of a job; choose from ‘finished, failed, or aborted

Cleaning Up Job(s)

--cleanup

Delete intermediate files from these job(s) to save disk space; enter ALL to clean up all jobs

--harsh

Add this argument to –cleanup to delete even more files

Default: False

Utilities

--validate_starfile

Check a star file and make sure it is written in the correct format. If errors are found will attempt to fix them

--convert_pipeline_file

Convert a pipeline file from Relion 3.1 to the CCPEM pipeliner format

--default_jobstar

Make a job.star file with the default values for a specific job type for use with the ccpem-pipeliner

--relionstyle

OPTIONAL: Add this argument to –default_jobstar to write job.star files which are compatible with RELION 4.0. RELIONstyle job.star files are fully compatible with the pipeliner, pipeliner job.star files may havedifferences that cause bugs in RELION

Default: False

--default_runjob

Make a _run.job file with the default values for a specific job type. These files can also be used to run jobs and a more human readable

--job_info

Get info about a specific job type, including any reference(s)

--empty_trash

Delete the files in the trash. THIS CANNOT BE UNDONE!

Default: False

Project Analysis

--job_runtime

Measure how long the steps in a job took to complete

--metadata_report

Prepares a report in .json format for the terminal job and allof its parent jobs

--draw_flowchart

Draw a flowcharts for the pipeline. If used alone; draws the entire pipeline, if followed by a job name; draws upstream and downstream flowcharts for that job. Saves the output as jobxxx_upstream.png unless the –interactive argument is also included

--upstream

[optional] Add this argument to –draw_flowchart to only draw the upstream flowchart for the specified job

Default: False

--downstream

[optional] Add this argument to –draw_flowchart to only draw the downstream flowchart for the specified job

Default: False

--interactive

[optional] Add this argument to –draw_flowchart to show interactive flowcharts

Default: False

Project Archiving

--full_archive

Create a full archive for a project. This will contain the entire job dirs for the stated terminal job and all of its parents

--simple_archive

Create a simple archive for a project. This will contain just the directory structure and parameter files for the stated terminal job andall of it’s parents along with a script to automatically re-run the project through the terminal job