chunksetup(1) General Commands Manual chunksetup(1)
NAME
chunksetup - Set up command files to process a volume in chunks
SYNOPSIS
chunksetup [options] command_file input_file output_file
DESCRIPTION
chunksetup will produce a set of command files to process a 3D image
file in chunks and reassemble the chunks into a single image file.
This procedure can be used when the processing operation is memory
intensive, when it is more efficiently done in smaller chunks, or when
one wishes to run the operation on multiple processors.
To use this procedure, first make a master command file (the first
required argument) that performs the needed operation, in the format
required by Vmstocsh, as ordinarily run with subm. Instead of using
the actual filenames, the input file for the operation in this command
file should be entered as INPUTFILE, and the final output of the opera-
tion as OUTPUTFILE. (Note that OUTPUTFILE should not be used for the
name of the final output file given on the chunksetup command line.)
The command file may perform multiple operations and may even contain
INPUTFILE more than once. If the procedure needs to produce temporary
intermediate files, they should be named ending in ".$$" (e.g., chunk-
temp.$$). At the end of the command file, include lines to remove the
specific intermediate files, e.g.
$if (-e chunktemp.$$) rm -f chunktemp.$$
If you are going to run the operation on more than one machine, it is
probably best to put the files in a machine-specific location such as
/usr/tmp. Alternatively, you can use a more elaborate method for nam-
ing temporary files. Define a variable at the top of your command
file:
$set tmpext `hostname`.$$
and end your temporary file names with ".$tmpext". (Note that "host-
name" must be enclosed in back-quotes not regular quotes - this charac-
ter may not appear properly here.) In this case, you should remove the
files at the end of the command file with statements like:
$if (-e chunktemp.$tmpext) rm -f chunktemp.$tmpext
The input and output files need not be in the same directory as the
command files; a path specification can precede each filename. Also,
the master command file need not be in the current directory; the chunk
command files will be placed in the directory where it is located. In
this case, however, you must enter the path to the input and output
files relative to the command file's directory, not relative to the
current directory.
When Chunksetup is run, it first calls Tomopieces, which analyzes
the dimensions of the input file to determine how to chop it up opti-
mally. It then produces command files for extracting each chunk with
Taperoutvol and processing them, numbered sequentially with 3-digit
numbers. If the name of the master command file is "comfile.com", the
first command file is comfile-001.com, INPUTFILE is replaced by com-
file-001.in, and OUTPUTFILE is replaced by comfile-001.out, and simi-
larly for the following files. Each file contains a command to remove
comfile-nnn.in, so only the output chunks accumulate as the files are
run. Chunksetup also produces a command file to reassemble the pieces,
comfile-finish.com, which will also delete the output chunks and the
command and log files for each of the chunks. Finally, it produces a
command file, comfile-all.com, to run all of the chunks in sequence and
run the finishing command file at the end.
You have the option of running the top-level command file using "subm",
or running the individual command files and running the finishing com-
mand file at the end. If you have multiple processors available with
access to the current directory, you can use the Processchunks
script to run the jobs in parallel on the various processors or
machines, such as with:
processchunks machine_list comfile
where machine_list is a list of the machines to use, or just the number
of processor to use on the local machine.
OPTIONS
Chunksetup uses the PIP package for input (see the manual page for
pip). Options can be specified either as command line arguments
(with the -) or one per line in a command file (without the -).
Options can be abbreviated to unique letters; the currently valid
abbreviations for short names are shown in parentheses.
-master (-ma) OR -MasterComFile File name
Name of master command file. If this option is not entered, the
first non-option argument will be taken as this name.
-input (-i) OR -InputImageFile File name
Name of input image file. If this option is not entered, the
second non-option argument will be taken as this name.
-output (-ou) OR -OutputImageFile File name
Name of output image file. If this option is not entered, the
third non-option argument will be taken as this name.
-p OR -PaddingPixels Integer
Number of pixels of padding on each edge of each subvolume. The
default is 8.
-o OR -OverlapPixels Integer
Minimum number of pixels of overlap between the subvolumes. The
default is 8.
-m OR -MegavoxelMaximum Integer
Limit each subvolume to the given number of megavoxels. The
default is whatever the default is for Tomopieces (80
megaVoxels).
-xm (-x) OR -XMaximumPieces Integer
The maximum number of chunks in the X direction. The default is
-1, which is effectively no limit. See the man page for Tomo-
pieces for more details on the choices for maximum number of
pieces.
-ym (-y) OR -YMaximumPieces Integer
The maximum number of chunks in the Y direction. The default is
-1, which is effectively no limit.
-no (-n) OR -NoFFTSizes
Do not increase padding to make suitable sizes for an FFT. Use
this option if no FFTs will be taken of the volumes. It causes
the "-nofft" option to be sent to Tomopieces and to be added
to the input to Taperoutvol that is placed in the command
files.
-param (-pa) OR -ParameterFile Parameter file
Read parameter entries as keyword-value pairs from a parameter
file.
-help (-h) OR -usage
Print help output
-StandardInput
Read parameter entries from standard input
AUTHOR
David Mastronarde <mast at colorado dot edu>
SEE ALSO
tomopieces, taperoutvol, assemblevol, processchunks
IMOD 4.11.0 chunksetup(1)