batchruntomo(1) General Commands Manual batchruntomo(1)
NAME
batchruntomo - Align tilt series and generate tomograms automatically
SYNOPSIS
batchruntomo [options] directive_files...
DESCRIPTION
Batchruntomo runs all the operations required to align a tilt series
and build a tomogram, following a set of instructions called "direc-
tives" provided in a text file. These directives indicate either
parameter settings or which pathway to follow at various points in the
processing sequence. The program will perform operations in a fixed
order determined by these choices; the order of the directives does not
affect this order. The program can work on more than one data set and
can also move the tilt series from their current location into a sepa-
rate directory for each data set, which is created if necessary. It
accepts tilt series with extensions ".mrc", ".st", ".hdf", ".tif", and
".tiff". Depending on the image file naming style specified, it may
change the extension to ".st" so that other programs will work cor-
rectly.
Directives
Directives consist of key-value pairs separated by equals signs, where
the key has 2 to 4 components separated by periods. The first compo-
nent is a prefix that indicates the type of directive or the stage at
which it is applied. The value may be blank if appropriate but the
equals sign must be present. There are currently three prefixes:
"setupset", for directives involved when setting up a data set; "com-
param", for directives used to set options in command files; and "run-
time", for directives interpreted by Batchruntomo or Etomo. A descrip-
tion of all directives is contained in the file
"IMOD/com/directives.csv", which is in comma-separated values format.
It is accessible in a more readable form from the IMOD help index via
the link "Directives for batch processing and Etomo templates" (avail-
able from "imodhelp" or when accessing the html package of IMOD docu-
mentation with a browser).
"setupset" directives have two forms. The majority have the form
setupset.copyarg.<copytomocoms short option name>=value
These directives will be processed mostly generically by Etomo and sent
to Copytomocoms as command line arguments. The short option name
must be given in full. Unlike for the other two kinds of directives,
whenever there is a pair of directives corresponding to a pair of
options for each axis of a dual axis set, a directive for the first or
only axis does not apply to the second axis. A separate "b" directive
must be supplied.
Other "setupset" directives are interpreted exclusively by Batchruntomo
or Etomo and have the form
setupset.<parameter or option>=value
"comparam" directives are interpreted generically when a command file
is created either by Copytomocoms or by Makecomfile. They have
the form
comparam.<root name of file>.<program>.<Long option name>=value
The root name of the command file may include an axis letter ("a" or
"b"). For a single axis data set, directives without a letter and with
an "a" are treated as equivalent and ones with "b" are ignored. For a
dual axis data set, a directive without a letter can apply to either
axis, but will be overridden by a directive with a letter matching the
specific axis, regardless of which one occurs later in the processing
sequence. The long option name must be given in full. Options to
Tilt must match the case shown in the Tilt man page, even though
Tilt accepts case-insensitive options.
Because "comparam" directives are treated generically, they are not
restricted to the ones listed in the table of directives. Any valid
option can be used, although some would be meaningless in the context
of automation and some would conflict with the management provided by
Copytomocoms or be overridden by Makecomfile.
"runtime" directives have the form
runtime.<Processing step>.<axis>.<parameter or option>=value
Here the processing step and parameter have names that have been made
up just for the directive. The convention is for the processing step
to be capitalized and the parameter to start with lower case. The axis
must be either "a", "b", or "any". For a single axis data set, direc-
tives with "any" or "a" are treated as equivalent and ones with "b" are
ignored. For a dual axis data set, a directive with "any" can apply to
either axis, but will be overridden by a directive with a letter match-
ing the specific axis, regardless of which one occurs later in the pro-
cessing sequence.
Boolean directives must have a value of either 0 or 1: 1 to turn the
option on or 0 to turn it off; no other values (including blank) are
allowed. If a boolean directive is not present then the option is off,
unless it has been turned on by an entry in a template file (see
below). For any other kind of directive, the value can be blank, which
causes the option in question to be removed from consideration.
Specifically, for a "comparam" directive, a blank value will cause the
option to be removed from the command file if present. For other kinds
of directives, a blank value removes the setting of the parameter by a
previous directive in a template file.
If a processing step is being skipped or bypassed by an alternative
method of processing, then all the directives related to it have no
effect on the processing that is done. It is thus generally safe to
leave unused directives in a file. Such directives with prefix "com-
param" will be used to set parameters in command files produced by
Copytomocoms, but directives for parameters in other command files
will have no effect because those files will not be produced. Direc-
tives with prefix "runtime" are simply ignored if they are not relevant
to the chosen processing sequence.
Template Files
Template files can contain parameters that can be used meaningfully for
multiple, similar data sets. Directives exist to define a microscope
template file, a system template file, and a user template file. The
idea is for the microscope file to have parameters specific to a micro-
scope or microscope/camera combination, and to have the other template
files contain other parameters that would not be dataset-specific.
Each file consists of a subset of directives just like those described
for defining options for the batch processing. There is a heirarchy
among these directives. They are always processed in the order scope
template, system template, user template, and batch directive file, and
entries later in this sequence override ones that appeared earlier.
Batchruntomo reads the template files and uses their contents along
with those of the directive file to determine the values for parame-
ters. It also passes whatever template files exist, plus the batch
file, to Etomo for it to use when setting arguments for calling Copyto-
mocoms(1), and Etomo passes the files on to Copytomocoms to use for
setting parameters. Batchruntomo passes them to Makecomfile to use
similarly when making optional command files, as well as to Setupcom-
bine(1).
Another directive file, "batchDefaults.adoc" from the IMOD "com" direc-
tory, provides some default values needed when running with batch pro-
cessing. As of IMOD 4.10.15, these are read in to Batchruntomo and
processed prior to the templates. Prior to that, Etomo appended these
directives to the batch directive file unless there was an overriding
value in a template. Unfortunately, if such a file is used as a start-
ing batch file in the Etomo interface, these directives are indistin-
guishable from others and override corresponding template values, if
any, instead of being overridden. Batchruntomo now recognizes this
situation and will issue a warning if any such directives do override a
template value.
The program checks all directives after it reads them, including the
ones from a template file, against the master list in "directives.csv".
All "setupset" and "runtime" directives must match ones in this file.
It is possible to enter "comparam" directives not listed in this file,
provided that the command file name and program name match those of
other directives in the master file. The program will consult the
autodoc file for the program to determine whether the option is valid.
It will issue a warning if an autodoc cannot be found; otherwise, it
will be able to determine validity definitively.
Processing Steps Available
This section describes some of the capabilities and current limitations
of various steps.
X-ray removal is optional; if selected, automatic X-ray removal will be
done, and "clip stats" will be run on the fixed stack. There is also a
directive to run Archiveorig to make a compressed difference between
corrected and original stacks, then delete the original stack. A man-
ual replacement model can be specified. It is suggested that object 1
contain patches specified in the default way, by a point on each pixel;
object 2 contain lines, if any; object 3 contain patches specified by
boundary contours, if any; and that directives be added to specify
these choices and that objects 1-3 are all-section objects. These sug-
gestions will be enforced when there is a GUI for setting up batch
directives. If the manual replacement model is located elsewhere, it
will be copied to the dataset and renamed to the standard name for such
models, so that the model can be opened and added to when running in
Etomo.
Detection of blank images and images with large dark regions at the
highest tilt angles can be done before further processing by including
the preprocessing directive "endExcludeCriterion" with a fractional
value such as 0.5. (The directive is ignored for montaged tilt
series.) The program will look at the standard deviation (SD) of
images at each end of the tilt series and find ones whose SD is less
that the criterion times the mean SD of the five adjacent less-tilted
images. Up to three images can be excluded on each end of the series
by this analysis. The program will then do a histogram analysis with
Clip to see whether the intensity distribution has two peaks and how
much of the area falls in the lower peak. A view is excluded if inten-
sity at the lower peak is less than 0.17 times that at the upper peak
and more than 0.33 of the pixels are in the lower peak (these defaults
can be modified with "darkExcludeRatio" and "darkExcludeFraction"
directives). Views can be eliminated up to 6 tilts from either end in
this way, but with either analysis, only consecutive views will be
eliminated. View skipping/exclusion lists will be added or adjusted in
xcorr.com, track.com, align.com, and tilt.com.
Coarse alignment is always performed. The binning of a prealigned
stack can be specified.
Fiducialless alignment can be chosen, in which case tomogram thickness
must be specified. The tomogram will not be reoriented at all (no
angle offset or X-axis tilt applied).
Patch tracking can be chosen. Again, tomogram thickness must be speci-
fied, but in this case the tomogram will be reoriented based on angles
found by Tiltalign. Also, if you specify gold erasing after 3D bead
finding, the thickness must be specified for the tomogram used to find
the beads.
Gold fiducial tracking can be done either with automatic seed-finding
then bead tracking, or with RAPTOR followed by optional bead tracking.
For a second axis, Transferfid can be run, followed by automatic
seed-finding to fill in the resulting model. RAPTOR can be run instead
for the second axis (unlike in Etomo), although there will then be no
list of corresponding fiducials. Beadtrack can be run more than
once, because this completes a model more reliably than selecting fur-
ther rounds of tracking in a single run.
Fine alignment with Tiltalign can include local alignments in all
cases. There is a directive to enable the stretching (distortion)
solution. When enabled, stretching will be solved for only if there
are gold fiducials on two surfaces, and if there are enough on the
minority surface overall and relative to the total number. Stretching
may be solved for in the global solution but not in the local one if
there are too few fiducials on the minority surface per local area. In
addition to making these decisions, Batchruntomo also runs the program
Restrictalign, which will use cross-validation to test whether the
number of variables being solved for should be reduced. This poten-
tially lengthy process will be run once with global variables, then
once again if either stretching or local alignments are added.
Restrictalign will turn off robust fitting if it gives no benefit,
but if it does so when evaluating global variables and local alignments
are being done, Batchruntomo will turn robust fitting on again for the
local tests. The fine alignment produces an angle offset and an X-axis
tilt that is applied when building the tomogram, unless there are too
few gold. In addition, if there is gold on two surfaces, the distance
between the gold particles provides an estimate of thickness that can
be used as one basis for the tomogram thickness. Otherwise, if gold is
to be erased after 3D bead finding, the thickness for the required
tomogram must be specified.
Tomogram positioning can be done for both plastic sections and cryosam-
ples by setting the "Positioning.sampleType" directive to 1 or 2,
respectively. Plastic section positioning is done with a binned-down
whole tomogram for now. If there is no "Positioning.binByFactor"
directive, the program will pick a binning up to 4 that brings the
binned size under 512 in its largest dimension. If there is no "Posi-
tioning.thickness" directive, the program will pick a thickness based
on the largest dimension: 250 if size under 512, 400 if size under
1024, 500 if size under 2048, and 600 otherwise. The program will run
Findsection and try to make a model for Tomopitch with 5 pairs of
sample lines. Cryosample positioning is done with the script Cryoposi-
tion(1). In this case, the "Positioning.thickness" directive is
required. Cryoposition picks an appropriate binning, but if you do
want to set this binning, it can be done with a "cryoposition.Binning-
ToApply" directive instead of the runtime "Positioning.binByFactor"
directive. By default, 25 unbinned pixels will be added on each sur-
face by Tomopitch; this can be changed with a "tomopitch.ExtraThick-
ness" directive.
When there is no positioning, angles are set from the Tiltalign out-
put. The Z height of the specimen in the tomogram will be whatever
location results from the coarse alignment; thus the specimen will be
centered in Z if features contributing to the cross-correlations are
evenly distributed in depth. If the correlations are dominated by fea-
tures near one surface, the specimen will not be centered in Z. If
there are fiducials on two surfaces, one can use a "Position.centerOn-
Gold" directive to apply the Z shift computed by Tiltalign for cen-
tering on the range of fiducials in Z. Another side-effect of using
this directive is that the estimated thickness from alignment will be
larger so that the tomogram will then contain all of the gold included
as fiducials.
The aligned stack can have specified binning and size. Optional modi-
fications of the aligned stack are performed in the standard order, or
with gold erasing first if the -erase option is entered. CTF correc-
tion can be done with a single nominal defocus value. Alternatively,
Ctfplotter can be used to find the defocus automatically if the
"autofitRangeAndStep" directive is present. When Ctfphaseflip is
run, it will use a "setname.defocus" file if it is present; otherwise
it will use a "setname_simple.defocus" file created with the nominal
defocus from the "setupset.copyarg.defocus" directive. (The "set-
name.defocus" file could derive from manual or automatic running of
Ctfplotter, or could be created by other means). Gold erasing can
be done with automatic finding of beads in a tomogram, or by transform-
ing the fiducial model (or the model put out by Tiltalign that has
gaps filled in, if available). For finding beads in the tomogram, the
"GoldErasing.thickness" directive must be supplied unless there are
beads on two surfaces. The binning is set as the unbinned bead size
divided by 5 and rounded to an integer, as in Etomo; thus, the "Gold-
Erasing.binning" directive is no longer required. Both the automatic
finding of beads in 3D, and the detection of defocus, are steps that
one might want to check before proceeding. To allow intervention at a
single stopping place, CTF correction and gold erasing steps are inter-
leaved: defocus is detected, beads are found, CTF is corrected, then
beads are erased. 2D filtering and dose weighting can be done by
adding the directive "AlignedStack.filterStack". There are various
directives to enter the Mtffilter parameters for dose weighting. To
use an .mdoc file or the information inside an HDF file from SerialEM,
add the directive "mtffilter.TypeOfDoseFile" with value 4. The entry
for the dose information file in the command file is managed appropri-
ately by Copytomocoms except in one case: using the old naming style
with descriptive extensions when the input stack is an MRC file with an
extension other than ".mrc". In this case you need the "mtffil-
ter.DoseWeightingFile" directive with either the name of the .mdoc file
or a generic entry that works for multiple data sets: the extension
before the ".mdoc" in the name, typically ".st".
The tomogram can be computed with regular weighted back-projection,
with SIRT, or with a back-projection using a SIRT-like filter. The
iterations to leave can be specified for SIRT. The regular back-pro-
jection can be computed in the same run as a SIRT or SIRT-like one by
adding the directive "Reconstruction.doBackprojAlso". The thickness
can be chosen automatically if there are gold beads on both surfaces or
if tomogram positioning was done, otherwise it must be specified with a
directive. The "Reconstruction.fallbackThickness" directive is the
most useful one because it can be used to specify a thickness to be
used when there is no thickness available from either fine alignment or
positioning. The fallback thickness will also be used if the thickness
from either is less than 40% of the fallback thickness, which would be
the case if there are gold on only one surface and no positioning was
done.
A 3D CTF-corrected tomogram can be computed by adding the directive
"ctf3dsetup.SlabThicknessInNm"; "ctf3dsetup.RunSlabsInParallel" can be
added to make slabs in parallel and "ctf3dsetup.UseUnalignedImages" can
be added to reconstruct directly from the unaligned images (the raw
stack). The result will be given the name of the regular weighted
back-projection tomogram. You can make a SIRT or SIRT-like filtered
tomogram in addition to the 3D CTF-corrected one. Steps for modifying
the aligned stack are skipped when making just the 3D CTF-corrected
tomogram, and they are applied during the reconstruction step.
Tomogram combination for a dual-axis dataset is possible with all kinds
of alignments and will proceed automatically. The program first uses
Findsection to find the Z limits, and decides whether to get the
initial registration between volumes with Solvematch or Dualvol-
match(1). The patch correlations and fitting are done with Autopatch-
fit(1); by default, it will try medium to extra-large patches and add
target residual values of 0.4,0.45 for Findwarp on the final trial.
There are "runtime.Combine" directives to control this behavior and to
set a few other parameters for tomogram combination; also, "comparam"
directives can be used to modify parameters in solvematch.com, patch-
corr.com, and matchorwarp.com. If both SIRT or SIRT-like and regular
back-projection reconstructions are available, the latter will be com-
bined unless the combine "doSIRTifBoth" directive specifies otherwise.
The decision of whether to match A to B instead of B to A is made based
on the ratio of the B to A thicknesses determined from their Z limits.
This ratio is compared to a criterion that can be set with the "matchA-
toBThickRatio" directive and whose default is 0.9. With this crite-
rion, A will be matched to B if B is thinner than A by more than ~10%.
Lowering the criterion would make it match A to B only when the thick-
ness difference is more extreme. Raising the criterion toward 1 would
make it match A to B with less thickness difference. It can be set to
a very low number (0.2) to guarantee that B is always matched to A, or
a very high number (5) to have A always matched to B.
Postprocessing can be done with Trimvol, and should be selected with
the directive "runtime.Postprocess.any.doTrimvol". For compatibility
with old behavior, Trimvol will still be run if any of the "run-
time.Trimvol" directives "findSecAddThickness", "reorient", "thick-
ness", "sizeInX", "sizeInY", "scaleFromX", "scaleFromY", or "scale-
FromZ" are present. Findsection will be used to find the section
limits if the "findSecAddThickness" directive is present. The output
file for a final reconstruction will have the usual name, setname.rec.
For a dual-axis data set, it is possible to trim one or both of the
single axis reconstructions; here the output file will be named with
"_trim.rec". If both SIRT or SIRT-like and regular back-projection
reconstructions were made, the postprocessing "doSIRTifBoth" directive
controls whether to trim one or both reconstructions; in the latter
case the file for the back-projection will be named "_BP.rec". When-
ever a SIRT reconstruction is trimmed, the result on the final itera-
tion will be used. By default, the trimmed volume will be reoriented
by rotating around X. The entire volume will be used unless the "find-
SecAddThickness", "thickness", "sizeInX", or "sizeInY" directive is
present. If one of the "scaleFrom" directives or the "scaleToMeanSD"
directive is present, the volume will be scaled to bytes. If "scale-
FromX" or "scaleFromY" is not present, the area used for scaling in X
or Y will be the Trimvol default of 80% of the range; if "scaleFromZ"
is not present, the central one-third of slices will be used. All of
these size and scaling directives may be specified either as a number
of pixels, or as a fraction between 0.02 and 1.
Nonlinear anisotropic diffusion (NAD) with Nad_eed_3d can be run if
both "runtime.NAD" directives are present for specifying the number of
iterations and the K value. Trimvol must be run at least to create
the input file for the operation.
Reduction and Filtering The final volume can be reduced and/or filtered
by Reducefiltvol, where the reduction is done with antialiased fil-
tering and can be done by a non-integer amount. The directive "run-
time.Postprocess.any.doReduceFilt" must given, along with at least one
of the directives for "ReductionFactor", "LowPassRadiusSigma", or
"DeconvolutionStrength". Trimvol must be run at least to create the
input file for the operation.
Data set cleanup can be done with Tomocleanup by adding the "run-
time.Cleanup.doCleanup" directive. There are several "run-
time.Cleanup.keep..." directives to retain particular intermediate data
set files. The default is for Batchruntomo to include the Tomo-
cleanup(1) option -sirt to retain the output from running a SIRT or
SIRT-like reconstruction as well as a back-projection, but this can be
overridden with a directive.
Stopping and Restarting
The -start and -end options allow datasets to be started and stopped at
chosen steps in the sequence. The most useful step to stop at would be
10, to check CTF plotting and/or gold detection, then restart at 11.
To check fine alignment before going on, stop and restart at 6.
If you are setting up a situation where parameters are being changed
before restarting in an existing dataset, you should experiment to make
sure the changes have the desired effect and are not overridden. Here
are some general guidelines on that point:
1) Changes in "comparam" directives will generally have no effect once
a command file is created, because they are processed generically dur-
ing command file creation. In most cases, creation occurs during tomo-
gram setup, but some files are created with Makecomfile just when
needed, so it would be possible to change "comparam" directives for a
file still to be created. Some exceptions to this are "align.tiltal-
ign.SurfacesToAnalyze" and "tilt.tilt.THICKNESS", for which the current
directive value will be used instead of the value in the command file.
2) Changes in "setupset" directives will also generally have no effect
because they are interpreted during setup; the use of the defocus value
mentioned above is an exception.
3) Changes in "runtime" directives will generally have the desired
effect.
4) Changes in parameters in the command files will generally have the
desired effect, unless that parameter is managed by Batchruntomo on the
basis of "runtime" directives and other information. The biggest exam-
ple of this is the treatment of "align.com" when Tiltalign is rerun;
the -use option is provided specifically to avoid rerunning Tiltal-
ign(1).
Resource Usage
The CPU list provided with the -cpus option is passed directly to Pro-
cesschunks(1) for running operations that are divided into multiple
chunks. However, other operations that run from a single command file
will generally be run on the local machine, unless the -single option
is used to direct them to the first machine in the CPU list. The
exceptions are two time-consuming steps in dual-axis combination,
Matchvol and Autopatchfit, which benefit from using multiple
threads and are sent to the machine with the highest number of CPUs.
When running single command files, the program sets a limit on the num-
ber of threads equal to the number of CPUs designated in the CPU list
for the machine being used, if it is known (-limit can be entered if
this is not otherwise known for the local machine).
Batchruntomo can be run in parallel on multiple data sets, and in this
case it can also be run on a cluster queue. See the section OPTIONS
FOR PROCESSING DATA SETS IN PARALLEL for the options and environment
variables involved; see the Splitbatch(1) man page for descriptions of
resource usage and how to use a cluster.
Running External Commands
It is possible to specify an arbitrary external command to run either
instead of a step or after one has run, with "runtime.ReplaceS-
tep.any.#" or "runtime.RunAfterStep.any.#", respectively, where "#" is
the step number given below for the -end option. So far this is possi-
ble only for CTF determination (step 9), aligned stack filtering (step
13), and post-processing (step 20), but more steps can be added easily
if needed. The directive value must consist of a single line to run
one command-line operation without piping or redirection. The program
will place this command after a "$" in a command file named "replaceS-
tep.#.com" or "runAfterStep.#.com", and a log will be created when it
is run. The command can contain %{setname}, which will be substituted
with the root name of the data set plus the axis letter, if any. A
replacement command will be run even if there are no directives to
activate the normal running of that step. However, a command to be run
after a step will be run only if the normal step is run.
OPTIONS
Batchruntomo uses the PIP package for input (see the manual page for
pip). Options can be specified either as command line arguments
(with the -) or one per line in a command file (without the -).
Options can be abbreviated to unique letters; the currently valid
abbreviations for short names are shown in parentheses.
FILE, DIRECTORY, AND DATA SET OPTIONS
These options give information about files, directories, and the data
set name
-directive (-di) OR -DirectiveFile File name
Input file with directives for automating one or more data sets.
The filename must end in .adoc. All non-option arguments will
be taken as such input files. (Successive entries accumulate)
-root (-ro) OR -RootName File name
A data set name can be provided with this option to override the
name contained in the directive file, thus allowing directive
files to be reused without editing. If this option is used, the
number of directive files must be either one or the same as the
number of root names. The current location of the tilt series
must be indicated with the -current option if only one directive
file is entered with multiple root names. (Successive entries
accumulate)
-current (-cu) OR -CurrentLocation File name
Directory where tilt series is currently located, or where it
was originally located if it has already been delivered. This
entry overrides the data set directory contained in the direc-
tive file. It can be used in two different ways. If the
-deliver or -make option is entered, this option specifies the
directory where a tilt series will be moved from, and it must be
entered either once (if all data sets are in the same place) or
once for each data set. If neither of those options are entered,
then this option specifies the directory where the data set will
be processed and must be entered once for each data set or not
at all. If the program is being rerun with the -deliver option
and the data set has already been delivered on a previous run,
the entry here can be either the original location of the stack
or its current location; the program will recognize that its
current location is under the delivery directory. However, if
the -make option was used and the data set was already deliv-
ered, this entry must be the original location of the stack
rather than its new location; otherwise the program will move it
into a subdirectory of its data set directory. (Successive
entries accumulate)
-deliver (-de) OR -DeliverToDirectory File name
Make a subdirectory of the specified directory for each data
set, named with the root name of the data set, and move the tilt
series into the subdirectory. This option can be entered either
once, or once for each data set to deliver to more than one
location. Several other files with the same root name as the
tilt series will be moved from the current location of the data
set: an associated metadata file with extension ".mdoc", a log
of acquisition with extension ".log", and a file of tilt angles
with extension ".rawtlt". A raw boundary model for patch track-
ing or autoseeding will also be delivered. The -current option
must be entered either once, or once for each data set. (Suc-
cessive entries accumulate)
-make (-mak) OR -MakeSubDirectory
Make a subdirectory for each data set at its current location,
named with the root name of the data set, and move the tilt
series into the subdirectory. Other files will be moved there
as well, just as for the -deliver option. With this option, you
can sort data sets into multiple top-level directories, and the
program will take care of putting each one into its own subdi-
rectory. This option cannot be entered with the -deliver
option. The -current option must be entered either once, or
once for each data set, and must specify the original location
of the stack if the program is being rerun and the stack has
already been delivered.
-check (-ch) OR -CheckFile File name
File to check for quit and finish signals. The default is
batchruntomo.###.input in the directory where this program is
started, where ### is the process ID. This file should not have
the same name as the file used by Batchruntomo to control Pro-
cesschunks when it runs jobs, which is "processchunks.cmds" in
the data set directory. If a Q is placed in this file, it will
terminate any running jobs and stop all processing. If an F is
placed in the file, the program will stop after finishing the
current data set. Termination will usually be quicker if a Q is
placed into "processchunks.cmds" as well, with no delay between
that action and writing Q to this check file. Thus, it should
be done by a program or by entering a single command line with
two commands on it.
-frompath (-fr) OR -TranslatePathsFrom Text string
Prefix from which to translate directive, location, and delivery
paths. When this program is being run in parallel, the path for
accessing the directories on the current machine may be at a
different mount point than on the machine where the runs were
set up. When there is a RemoteDirectory entry, Splitbatch(1)
will add this option and the next one to indicate how to trans-
late the paths for directive files, current locations, and
delivery locations to work on any machine. However, it will
retain any existing entries and avoid duplicating any. These
entries should be equivalent to a mountrule defined in cpu.adoc
(see cpuadoc). Batchruntomo will use these translated paths
in all runs (hence they must work on all machines, even if they
are accessed through links on some machines). However, it will
translate paths in messages to the log back to having the origi-
nal prefixes so that Etomo can track file changes. If multiple
translations are entered (equivalent to having multiple global
mountrules in cpu.adoc), they must all work on all machines,
such as by all translating to the same mount point. (Successive
entries accumulate)
-topath (-t) OR -TranslatePathsTo Text string
Prefix to which directive, location, and delivery paths will be
translated. The number of entries must be the same for this
option as for -frompath (Successive entries accumulate)
-remote (-re) OR -RemoteDirectory File name
Path on remote machines to directory from which this program is
started.
BASIC PROCESSING RESOURCE OPTIONS
These options give information about the processing resources and
their limits in the simplest case of not running data sets in parallel
-cpus (-cp) OR -CPUMachineList Text string
Machines to use in parallel processing, or number of local
cores. Each machine name can be followed by a ":" (colon), fol-
lowed by the number of CPUs to use in that machine; e.g.,
druid:4,tubule:4. Prior to IMOD 4.8.25, a "#" sign was used
instead of ":". Using the "#" sign will now result in a warning
when entering options on the command line and an error when
entering options through standard input, because in the latter
case it cannot be distinguished from the start of a comment. If
the environment variable MULTI_PROC_CPU_LIST is set, it over-
rides this entry to the program.
-single (-si) OR -SingleOnFirstCPU
Run single command files on the first machine in the CPU list,
instead of on the local machine. The matchvol and autopatchfit
steps in combine are not affected by this entry; they are always
run on the machine with the most CPUs.
-gpus (-gp) OR -GPUMachineList Text string
Machines to use for parallel processing with a GPU, or 1 for
local GPU. As when running processchunks with the -G option,
multiple or specific GPUs can be specified by colon-separated
numbers after a machine name, e.g., druid:2,tubule:1:2. If the
environment variable MULTI_PROC_GPU_POOL is set, it overrides
this entry to the program.
-nice (-n) OR -NiceValue Integer
Priority level or niceness for running jobs. Note that this
applies to all computations, not just those run with parallel
processing. The default is 15.
-limit (-l) OR -LimitLocalThreads Integer
Limit on the number of threads when running a single command
file on the local machine. By default, this limit is set to the
number of CPUs on the local machine indicated by the entry to
-cpus. If that entry consists only of remote machines, then the
program has no information about this limit, and single command
files containing programs parallelized with OpenMP will use all
available cores. This option allows one to enter a limit in
this case, as well as to set a different limit than the number
of CPUs in the CPU list.
OPTIONS FOR PROCESSING DATA SETS IN PARALLEL
These options control the processing of multiple data sets in paral-
lel, including on a cluster
-parallel (-pa) OR -ParallelBatchRootName Text string
Root name that will be sent to Gpuallocator when this program
is being run in parallel with other instances and needs to use
one or more GPUs. In this case, the entry for -gpus needs to
have the full list of GPUs available for all of the instances.
This entry needs to be included to indicate batch is being run
in parallel even when there are no GPUs.
-maxGPUs (-max) OR -MaxGPUsInParallelBatch Integer
Maximum number of GPUs to use when running this program in par-
allel. The default is 4. See Splitbatch(1) and Gpualloca-
tor(1).
-queue (-q) OR -QueueCommand Text string
Command for running processes on a cluster queue when running in
parallel, typically a Queuechunk command. This entry would
be the same as the "command" entry for a queue in a cpu.adoc
(see cpuadoc and "IMOD/autodoc/cpu.adoc" for examples). If
the environment variable MULTI_PROC_QUEUE_COMMAND is set, it
overrides this entry to the program. With this entry, the pro-
gram runs all operations that can be parallelized in chunks on
the queue. The treatment of single command files for other
operations depends on whether MULTI_PROC_QUEUE_COMMAND was set:
if it is, then Batchruntomo assumes it is itself running on a
queue and runs each command file normally; otherwise it runs
each command file on the queue. With this entry, -cpus and
-gpus entries are not allowed.
-jobs (-jo) OR -MaxJobsOnQueue Integer
Maximum number of jobs to submit to the cluster queue, which
influences how many chunks parallel operations will be divided
into. If the environment variable MULTI_PROC_MAX_QUEUE_JOBS is
set, it overrides this entry to the program.
-jcores (-jc) OR -CoresPerClusterJob Integer
Number of cores available to each job when running on a cluster;
these cores will be used directly instead of submitting command
files to the cluster queue. Thus, this entry and -queue cannot
be entered together, nor can -cpus and -gpus be entered. This
entry also sets the thread limit for processes whose command
files are not split into chunks. If the environment variable
MULTI_PROC_JOB_CORES is set, it overrides this entry to the pro-
gram.
-jgpus (-jg) OR -GPUsPerClusterJob Integer
Number of GPUs available to each job when running on a cluster;
these GPUs will be used directly instead of submitting command
files to the cluster queue. This option cannot be entered
unless -jcores is. If the environment variable
MULTI_PROC_JOB_GPUS is set, it overrides this entry to the pro-
gram.
-gqueue (-gq) OR -GPUQueueCommand Text string
Command for running GPU-using processes on a secondary cluster
queue, typically another Queuechunk command. For such a
process, either a single command file or separate chunks will be
submitted to this queue, depending on how many jobs are allowed
for this queue (the -gjobs entry). Ideally, this queue should
provide access to just one GPU, because there is no mechanism
for the submitted jobs to make use of multiple GPUs. If the
environment variable MULTI_PROC_GPU_QUEUE is set, it overrides
this entry to the program. The option can entered only if
-queue or -jcores is entered.
-gjobs (-gj) OR -MaxGPUJobsOnQueue Integer
Maximum number of jobs to submit to the secondary cluster queue
specified with -gqueue; this corresponds to the "number" entry
in a cpu.adoc section for that queue. The maximum jobs will
also be limited by the entry for -maxGPUs. If the environment
variable MULTI_PROC_MAX_GPU_JOBS is set, it overrides this entry
to the program.
PROCESSING FLOW CONTROL
These options allow doing a subset of processing operations
-one (-o) OR -ProcessOneAxis Integer
Enter 1 to do setup and process A axis, or 2 to skip setup and
process only B axis. This entry applies to all dual-axis data
sets being run.
-end (-en) OR -EndingStep Floating point
Step to end processing with in each set and axis. Steps are
numbered as follows:
0: Setup
1: Preprocessing
2: Cross-correlation alignment
3: Prealigned stack
4: Patch tracking, autoseeding, or RAPTOR
5: Bead tracking
6: Alignment
7: Positioning
8: Aligned stack generation
9: CTF plotting
10: 3D gold detection
11: CTF correction
12: Gold erasing after transforming fiducial model or
projecting 3D model
13: 2D filtering
14: Reconstruction
14.5: Postprocessing on a/b axis reconstruction
15: Combine setup
16: Solvematch
17: Initial matchvol
18: Autopatchfit
19: Volcombine
20: Postprocessing with Trimvol
21: NAD (Nonlinear anistropic diffusion)
21.5: Reducefiltvol
22: Cleanup
These numbers can be relied on not to change again since this
is now a floating point entry; fractional values will be used if
necessary to insert more steps without renumbering.
-start (-sta) OR -StartingStep Floating point
Step to start processing with. Steps are numbered as described
for -end. Each set will start at this step unless the -first
option is entered, in which case only the first set will start
with this step and others will start from the beginning. In the
latter case, add 100 to start with this step in the second axis
of a dual-axis data set.
-first (-fi) OR -StartForFirstSetOnly
Skip to starting step only for the first set; the following sets
will be run from the beginning.
-use OR -UseExistingAlignment
This option can be used to rerun Tiltalign instead of using the
existing fine alignment when the -start option is used with a
value over 6 and the program will be running any steps that
involve the aligned stack or reconstruction. With an entry of 0
in this situation, the program reruns Tiltalign iteratively
in the usual fashion before proceeding (this can only be done
with input on sequential lines, not from command line). The
default is not to do this, to keep the program from overwriting
an alignment based on manually adjusted parameters. Rerunning
alignment in this situation was a necessity in early versions of
the program but should no longer be needed; this option is being
kept just for backwards compatibility.
-exit (-exi) OR -ExitOnError
Exit on error instead of going on to next axis or data set
EMAIL, VALIDATION, RENAMING, AND TEST OPTIONS
Aside from the email options, these are miscellaneous options for
testing or for use with Etomo.
-email (-em) OR -EmailAddress Text string
Address for sending email messages either when a set is aborted
or when the run completes.
-SMTP (-SM) OR -SMTPserver Text string
An SMTP server for sending email messages. The default is
"localhost", but if the local host cannot send email, a server
can be entered with this option.
-validation (-v) OR -ValidationType Integer
By default, the program will check all directives in the direc-
tive and template files against the master list of directives in
IMOD_DIR/com/directives.csv. Enter 1 to run the program just to
check a directive file and exit, 2 to check a template file and
exit, or -1 to skip this step. When checking a directive file,
the directive file is first processed to find certain required
values, and an error in that step will prevent the checking of
directives. Template files referenced by the directive file
will be checked as well. With an entry of 2, the file will be
checked right after reading. A batch directive file is required
to contain only directives with a Y in the "Batch" column of the
master directives file, and a template file must contain only
directives with a Y in the "Tmplt" column.
-style (-sty) OR -NamingStyle Integer
0 for output image files to have descriptive extensions like
".preali", 1 for extension ".mrc", or 2 for extension ".hdf".
In the latter two cases the usual decriptive text is put before
the extension and generated image files will have the corre-
sponding type.
-pcm (-pc) OR -MakeComExtensionPcm Integer
0 or 1 to name output command files with extension ".com" or
".pcm". The latter is not yet supported by Etomo.
-ext OR -StackExtension Text string
Extension of an existing raw stack, used when just renaming
stacks(s), omitting the period. With this entry, the program
will perform the same renaming operations on one or two raw
stacks in the current directory as when doing a reconstruction.
The -root option must be entered once with the root name of the
file, and the -style option must also be entered.
-axis (-a) OR -AxisOfExtension Integer
When just renaming stacks, enter 0 for a single-axis dataset, or
1 or 2 for whether the extension provided with -ext applies to
the a or b stack of a dual-axis set. With a non-zero naming
style, the stack that this extension applies to will keep its
name, and the other axis of a dual axis set will have its exten-
sion changed to match if necessary.
-etomo (-et) OR -EtomoDebug Integer
Run Etomo with the given debug level
-bypass (-b) OR -BypassEtomo
Run Copytomocoms directly instead of Etomo for the setup step.
The program will add options to extract tilt angles if direc-
tives for using a raw tilt file are not present, and to specify
axis rotation angles if those directives are not present.
Setupset directives in the batch file only are passed as options
to Copytomocoms, but it may still need other directives there.
Excluded views are not removed since Etomo does that. An "edf"
file is made with just the items that would be added by
Batchruntomo.
-PID Print process ID
-help (-h) OR -usage
Print help output
-StandardInput
Read parameter entries from standard input
FILES
Batch and template files must have the extension ".adoc". The tilt
series file must have an extension ".st" or ".mrc". When Etomo is run
to set up the data set, it copies the batch directive file to the
dataset directory with the name "batchDirective.adoc". If there are
any template files, they are copied to the dataset with the the names
"scopeTemplate.adoc", "systemTemplate.adoc", and "userTemplate.adoc".
However, if any feature is used that requires modification of the batch
directive file, namely use of the -root or -current option or template
files listed without paths, Batchruntomo writes the modified file to
the dataset directory as "batchDirective.adoc" and runs Etomo with this
file.
AUTHOR
David Mastronarde
BUGS
Email bug reports to mast at colorado dot edu
SEE ALSO
etomo
HISTORY
IMOD 4.12.61 batchruntomo(1)