chunksetup(1) General Commands Manual chunksetup(1)
NAME
chunksetup - Set up command files to process a volume in chunks
SYNOPSIS
chunksetup [options] [command_file] input_file output_file
DESCRIPTION
chunksetup will produce a set of command files to process a 3D image
file in chunks and reassemble the chunks into a single image file.
This procedure can be used when the processing operation is memory
intensive, when it is more efficiently done in smaller chunks, or when
one wishes to run the operation on multiple processors for speed.
A simple way to use this procedure is to enter a one-line, non-compound
command that takes an input file and produces an output file. Other-
wise, you must supply a master command file that performs the needed
operation, in the format required by Vmstopy, as ordinarily run with
subm. Instead of using the actual filenames, the input file for the
operation in this command file should be entered as INPUTFILE, and the
final output of the operation as OUTPUTFILE. (Note that OUTPUTFILE
should not be used for the name of the final output file given on the
chunksetup command line.) The command file may perform multiple opera-
tions and may even contain INPUTFILE more than once. If the procedure
needs to produce temporary intermediate files, they should be named
ending in ".$$" (e.g., chunktemp.$$). At the end of the command file,
include lines to remove the specific intermediate files, e.g.
$if (-e chunktemp.$$) rm -f chunktemp.$$
If you are going to run the operation on more than one machine, it is
probably best to put the files in a machine-specific location such as
/usr/tmp. Alternatively, you can use a more elaborate method for nam-
ing temporary files. Define a variable at the top of your command
file:
$set tmpext `hostname`.$$
and end your temporary file names with ".$tmpext". (Note that "host-
name" must be enclosed in back-quotes not regular quotes - this charac-
ter may not appear properly here.) In this case, you should remove the
files at the end of the command file with statements like:
$if (-e chunktemp.$tmpext) rm -f chunktemp.$tmpext
The input and output files need not be in the same directory as the
command files; a path specification can precede each filename. Also,
the master command file need not be in the current directory; the chunk
command files will be placed in the directory where it is located. In
this case, however, you must enter the path to the input and output
files relative to the command file's directory, not relative to the
current directory.
If none of the above complexity is required, and the operation can be
run with a single command line entry, you can use the -command option
to provide the program name and its options, instead of using a master
command file.
When Chunksetup is run, it first calls Tomopieces, which analyzes
the dimensions of the input file to determine how to chop it up opti-
mally. It then produces command files for extracting each chunk with
Taperoutvol and processing them, numbered sequentially with 3-digit
numbers. If the name of the master command file is "comfile.com", the
first command file is comfile-001.com, INPUTFILE is replaced by com-
file-001.in, and OUTPUTFILE is replaced by comfile-001.out, and simi-
larly for the following files. If a single-line command is entered
instead of a master command file, the files are named with the program
name followed by "-cs" instead. Each file contains a command to remove
comfile-nnn.in, so only the output chunks accumulate as the files are
run. Chunksetup also produces a command file to reassemble the pieces,
comfile-finish.com, which will also delete the output chunks and the
command and log files for each of the chunks. Finally, it produces a
command file, comfile-all.com, to run all of the chunks in sequence and
run the finishing command file at the end.
You have the option of running the top-level command file using "subm",
or running the individual command files and running the finishing com-
mand file at the end. If you have multiple processors available with
access to the current directory, you can use the generic parallel pro-
cessing interface in Etomo or the Processchunks script to run the
jobs in parallel on the various processors or machines, such as with:
processchunks machine_list comfile
where machine_list is a list of the machines to use, or just the number
of processor to use on the local machine.
OPTIONS
Chunksetup uses the PIP package for input (see the manual page for
pip). Options can be specified either as command line arguments
(with the -) or one per line in a command file (without the -).
Options can be abbreviated to unique letters; the currently valid
abbreviations for short names are shown in parentheses.
-master (-ma) OR -MasterComFile File name
Name of master command file. If neither this option nor -com-
mand is entered, the first non-option argument will be taken as
this name. Either a command file or -command must be entered.
-command (-c) OR -OneLineCommand Text string
Full command to run on the input file, including all options
except ones specifying the input and output files. To use this
option, the program being run by the command must accept input
and output file names as the last non-option arguments on its
command line. If there are any options to the command and
Chunksetup is run with command line arguments, the whole entry
must be quoted.
-input (-i) OR -InputImageFile File name
Name of input image file. If this option is not entered, the
first non-option argument will be taken as this name, unless
neither -command nor -master is entered as an option, in which
case the second non-option argument is used.
-output (-ou) OR -OutputImageFile File name
Name of output image file. If this option is not entered, the
last non-option argument will be taken as this name unless -suf-
fix is entered. Either this entry or -suffix is required.
-suffix (-s) OR -SuffixForOutputName Text string
String to add to the root of the input image file name, before
extension, to produce the output image file name. For example,
if the input name is "setname_rec.mrc", and "filt" is entered
for this option, the default output file name will be "set-
name_rec_filt.mrc". If the input file does not have an exten-
sion corresponding to an allowed file format, the extension will
be placed before the suffix. In the example, an input of "set-
name.rec" will also result in an output name "set-
name_rec_filt.mrc". If the environment variable IMOD_OUT-
PUT_FORMAT is set or the option -format is entered, the output
will have the indicated file format and have the appropriate
extension; otherwise it will be MRC format with extension
".mrc".
-format (-f) OR -FormatOfOutputFile Text string
Set output file format to MRC, TIFF, or HDF by entering MRC,
TIF, TIFF, HDF, or any lower case form of these. This entry
overrides a default format set with the environment variable
IMOD_OUTPUT_FORMAT.
-p OR -PaddingPixels Integer
Number of pixels of padding on each edge of each subvolume. The
default is 8.
-o OR -OverlapPixels Integer
Minimum number of pixels of overlap between the subvolumes. The
default is 8.
-m OR -MegavoxelMaximum Integer
Limit each subvolume to the given number of megavoxels. The
default is whatever the default is for Tomopieces (80
megaVoxels).
-xm (-x) OR -XMaximumPieces Integer
The maximum number of chunks in the X direction. The default is
-1, which is effectively no limit. See the man page for Tomo-
pieces for more details on the choices for maximum number of
pieces.
-ym (-y) OR -YMaximumPieces Integer
The maximum number of chunks in the Y direction. The default is
-1, which is effectively no limit.
-no (-n) OR -NoFFTSizes
Do not increase padding to make suitable sizes for an FFT. Use
this option if no FFTs will be taken of the volumes. It causes
the "-nofft" option to be sent to Tomopieces and to be added
to the input to Taperoutvol that is placed in the command
files.
-param (-pa) OR -ParameterFile Parameter file
Read parameter entries as keyword-value pairs from a parameter
file.
-help (-h) OR -usage
Print help output
-StandardInput
Read parameter entries from standard input
AUTHOR
David Mastronarde <mast at colorado dot edu>
SEE ALSO
tomopieces, taperoutvol, assemblevol, processchunks
IMOD 4.12.61 chunksetup(1)