Command Line Tools
The pipeliner
command allows pipeliner functions to be run from the command line.
CCPEM Pipeliner command line utility
usage: pipeliner [-h] [--new_project [project name]]
[--available_jobs [search string]]
[--check_jobs [search string]]
[--run_job [run.job or job.star file]]
[--overwrite [job name]] [--continue_job [job name]]
[--abort_job [job name]]
[--print_command [run.job or job.star file]]
[--schedule_job [run.job, job.star file or job name]]
[--run_schedule] [--name [NAME]]
[--jobs [job name [job name ...]]] [--min_between [n]]
[--nr_repeats [1]] [--wait_sec_after [2]]
[--wait_min_before [0]] [--stop_schedule [STOP_SCHEDULE]]
[--delete_job [job name]] [--undelete_job [job name]]
[--set_alias [job name] [new alias]]
[--clear_alias [job name]] [--set_status [job name]
{finished, failed, aborted}]
[--cleanup [job name [job name ...]]] [--harsh]
[--validate_starfile [star file]]
[--convert_pipeline_file [_pipeline.star file]]
[--default_jobstar [job type]] [--relionstyle]
[--default_runjob [job type]] [--job_info [job type]]
[--empty_trash] [--job_references [job type]]
[--references_report [job name]]
[--metadata_report [terminal job]]
[--full_archive [terminal job name]]
[--simple_archive [terminal job name]]
[--empiar_deposition [terminal job name]]
[--onedep_deposition [terminal job name]]
[--deposition_id [Deposition id Assigned by the database]]
[--empiar_jobstar_file [Template file to update]]
[--empiar_movies] [--empiar_mics] [--empiar_parts]
[--empiar_corr_parts]
Arguments
- --new_project
Initialize a new project in this directory. Project name is optional, if none is specified the project will be called ‘default’ (recommended)
- --available_jobs
Show a list of all available jobs, optionally add a search string to limit the search, Leave blank to show all available job types
- --check_jobs
Get a list of available and unavailable jobs with detailed info onthe programs they use and why they are unavailable (if applicable)
Running jobs
- --run_job
Create a job using a run.job or job.star file to get the parameters
- --overwrite
Use with –run_job to overwrite a current job rather than making a new job. Jobs can only be overwritten with the same job type
- --continue_job
Continue a job that has been previously run. Edit the job’s continue_job.star file to modify parameters for the continuation
- --abort_job
Abort a running job
- --print_command
Read a job.star or run.job file and print the command(s) it would produce
- --schedule_job
Add a job to list of scheduled jobs using a run.job or job.star file to get the parameters. If followed by a job.star file it will create a new job if followed by a job name it will schedule a continuation of that job
Executing Schedules
- --run_schedule
Create a schedule choosing from the currently scheduled jobs and run it
Default: False
- --name
(required) Enter a name for the new schedule
- --jobs
(required) Enter the jobs that will be run. Make sure to list the jobs in the order which they should be run
- --min_between
(optional) Wait at least this many minutes between jobs
Default: 0
- --nr_repeats
(optional) Repeat the schedule this many times
Default: 1
- --wait_sec_after
(optional) Wait this many seconds after finishing before starting the next job
Default: 2
- --wait_min_before
(optional) Wait this many minutes before starting the schedule
Default: 0
- --stop_schedule
(required) Enter a name for the schedule to be stopped
Deleting jobs
- --delete_job
Remove job(s) and put in the trash, deletes the job and all of its child processes
Undeleting jobs
- --undelete_job
Restore a deleted job and any of its deleted parent processes from the trash
Modifying jobs
- --set_alias
Set the alias of a job
- --clear_alias
Clear the alias of a job
- --set_status
Set the status of a job; choose from ‘finished, failed, or aborted
Cleaning Up Job(s)
- --cleanup
Delete intermediate files from these job(s) to save disk space; enter ALL to clean up all jobs
- --harsh
Add this argument to –cleanup to delete even more files
Default: False
Utilities
- --validate_starfile
Check a star file and make sure it is written in the correct format. If errors are found will attempt to fix them
- --convert_pipeline_file
Convert a pipeline file from Relion 3.1 to the CCPEM pipeliner format
- --default_jobstar
Make a job.star file with the default values for a specific job type for use with the ccpem-pipeliner
- --relionstyle
OPTIONAL: Add this argument to –default_jobstar to write job.star files which are compatible with RELION 4.0. RELIONstyle job.star files are fully compatible with the pipeliner, pipeliner job.star files may havedifferences that cause bugs in RELION
Default: False
- --default_runjob
Make a _run.job file with the default values for a specific job type. These files can also be used to run jobs and a more human readable
- --job_info
Get info about a specific job type, including any reference(s)
- --empty_trash
Delete the files in the trash. THIS CANNOT BE UNDONE!
Default: False
- --job_references
Get literature reference(s) for a specific job type
- --references_report
Get all the literature references for a project with the selected job as the termial job
Project Analysis
- --metadata_report
Prepares a report in .json format for the terminal job and allof its parent jobs
Project Archiving
- --full_archive
Create a full archive for a project. This will contain the entire job dirs for the stated terminal job and all of its parents
- --simple_archive
Create a simple archive for a project. This will contain just the directory structure and parameter files for the stated terminal job andall of it’s parents along with a script to automatically re-run the project through the terminal job
Prepare EMDB/EMPIAR/PDB depositions
- --empiar_deposition
Prepare a deposition for upload to the EMPIAR database
- --onedep_deposition
Prepare a deposition for upload to the pdb/emdb databases
- --deposition_id
[Optional] The deposition ID. This must be assigned by EBI. If not specified, a draft deposition will be created
- --empiar_jobstar_file
(optional) A jobstar file from a pipeliner.deposition.empiar job. This contains data necessary from the deposition that cannot be automatically found. If not included, the deposition will only contain the data that can be found and will need to be manually completed
- --empiar_movies
(optional) Add this argument to deposition to include unprocessed movies in the EMPIAR deposition
Default: False
- --empiar_mics
(optional) Add this argument to deposition to include corrected micrographs in the EMPIAR deposition
Default: False
- --empiar_parts
(optional) Add this argument to deposition to include particles in the EMPIAR deposition
Default: False
- --empiar_corr_parts
(optional) Add this argument to deposition to include corrected (polished) particles in the EMPIAR deposition
Default: False