queuechunk(1)               General Commands Manual              queuechunk(1)



NAME
       queuechunk - Place a csh or python script file on a queue or kill a job

SYNOPSIS
       queuechunk [options]  root_name_of_script_file


DESCRIPTION
       Queuechunk will take a csh or python script file and submit it to a
       cluster queue, and also allows such jobs to be removed from the queue.
       It currently supports PBS and PBS-maui, SGE, and slurm queues, and
       would need to be modified to support other types.  It is designed to
       perform these actions when it is called by Processchunks with Python
       files produced from IMOD command files with Vmstopy.  However, csh
       files produced by Vmstocsh could be used instead.  The script file
       will be deleted when the operation finishes.

       The filesystem on which Queuechunk is run must be accessible from the
       cluster.

       The script file must be named with the extension .py or .csh, depending
       on the script type.  The last argument of the command to Queuechunk
       must be the root name of the script file, excluding the extension but
       including everything before the extension (-001, etc).  If the action
       requested with -a is L to report a load average or C:<cmd> to execute a
       user-defined command, this argument may be omitted.


OPTIONS
       -t type
              Type of queue.  This option must be entered with the value of
              either pbs or pbs-maui (for a PBS queue), sge (for a Grid Engine
              queue) or slurm (for the Slurm workload manager).  Type pbs-maui
              can be used if the queue software has been configured so that
                 showq -p queuename
              gives output for the cpus assigned to that queue of the form
                   0 Active Jobs       0 of   40 Processors Active (0.00%)

       -q name
              Name of queue to submit to or desired quality of service (qos)
              on a Slurm queue which uses qos for resource specification. (For
              Slurm queues which use partition names or other means of select-
              ing resources, see the -l option below). This option is required
              if the queue type is pbs-maui.

       -h name
              Name of the head node.  When this option is entered, commands
              (e.g. to queue a chunk, request status, or kill a running chunk)
              are run via ssh to the head node.  It is necessary to have pass-
              wordless access between machines for this to work.

       -w dir Name of the current working directory, as it must be used by the
              machine running the job.  If this name contains spaces, they
              should be escaped with backslashes.  The default is the current
              directory name.

       -a action
              The action to be performed:
                 R       - Run the job by placing it on the queue
                 S       - Run the job synchronously on the queue,
                           waiting until it is done
                 K       - Kill the job unconditionally
                 P       - Delete the job if it has not started yet (pause)
                 L       - Report a load average
                 C:<cmd> - Execute command <cmd> on the queue.
                           Separate optional arguments with spaces and enclose
                           entire string in double quotes. The base command
                           name may not contain spaces.

       -l resources
              A resources entry to make with the "-l" option when running qsub
              on a PBS queue, or additional option(s) to srun or sbatch when
              running under Slurm. Defaults to "nodes=1" under PBS and
              "-n1,-c1" under Slurm. In both cases, entries made here will
              replace, rather than be appended to the default. Spaces are not
              allowed in resources; use commas to separate multiple entries.
              For use with Slurm, commas will be translated back to spaces by
              queuechunk.


   Specifications
       Here are the essential features of this script, which would have to
       replicated by any program to be run from Processchunks.  In order to
       work from Etomo as well, the script mush be in bash.

       The program must accept the -w option with a working directory name.

       The program must accept the -a option with at least the five actions,
       R, S, K, P, and C:<cmd>.  For running from Etomo, the program must also
       respond to the L action with either NA for no information, or one or
       more values separated by commas.

       A secondary script file should be created to submit to the queue, with
       the name rootname.job.  It should be prepared to deal with a script
       file named either rootname.csh or rootname.py.  It should run root-
       name.csh with "csh -ef" or rootname.py with "python -u" and end by
       deleting the rootname.csh or rootname.py file.  The .job file will be
       cleaned up by Processchunks.

       A string containing an identifier for the job or job step resulting
       from chunk submission must be written to a file rootname.qid.  Often,
       suitable information will be printed by the queue when a chunk is sub-
       mitted. This file will also be cleaned up by Processchunks.

       When a job is run with the S action, the program must not return until
       the job is finished, and should exit with an error status if the job
       failed.  Queuechunk does this by submitting to the queue, but as an
       alternative, the .csh or .py file could be run on the local host with
       "csh -ef" or "python -u".


FILES
       The program creates a script file named rootname.job and places queue
       submission output into a file named rootname.qid.  The job script
       deletes the rootname.py or rootname.csh file upon completion.


AUTHORS
       David Mastronarde  <mast at colorado dot edu>
       John Heumann  <john.heumann at colorado dot edu>

SEE ALSO
       processchunks, vmstocsh




IMOD                                4.12.61                      queuechunk(1)