chunksetup(1)               General Commands Manual              chunksetup(1)

       chunksetup - Set up command files to process a volume in chunks

       chunksetup  [options]  [command_file] input_file  output_file

       chunksetup will produce a set of command files to process a 3D image
       file in chunks and reassemble the chunks into a single image file.
       This procedure can be used when the processing operation is memory
       intensive, when it is more efficiently done in smaller chunks, or when
       one wishes to run the operation on multiple processors for speed.

       A simple way to use this procedure is to enter a one-line, non-compound
       command that takes an input file and produces an output file.  Other-
       wise, you must supply a master command file that performs the needed
       operation, in the format required by Vmstopy, as ordinarily run with
       subm.  Instead of using the actual filenames, the input file for the
       operation in this command file should be entered as INPUTFILE, and the
       final output of the operation as OUTPUTFILE.  (Note that OUTPUTFILE
       should not be used for the name of the final output file given on the
       chunksetup command line.)  The command file may perform multiple opera-
       tions and may even contain INPUTFILE more than once.  If the procedure
       needs to produce temporary intermediate files, they should be named
       ending in ".$$" (e.g., chunktemp.$$).  At the end of the command file,
       include lines to remove the specific intermediate files, e.g.
       $if (-e chunktemp.$$) rm -f chunktemp.$$
       If you are going to run the operation on more than one machine, it is
       probably best to put the files in a machine-specific location such as
       /usr/tmp.  Alternatively, you can use a more elaborate method for nam-
       ing temporary files.  Define a variable at the top of your command
       $set tmpext `hostname`.$$
       and end your temporary file names with ".$tmpext".  (Note that "host-
       name" must be enclosed in back-quotes not regular quotes - this charac-
       ter may not appear properly here.)  In this case, you should remove the
       files at the end of the command file with statements like:
       $if (-e chunktemp.$tmpext) rm -f chunktemp.$tmpext

       The input and output files need not be in the same directory as the
       command files; a path specification can precede each filename.  Also,
       the master command file need not be in the current directory; the chunk
       command files will be placed in the directory where it is located.  In
       this case, however, you must enter the path to the input and output
       files relative to the command file's directory, not relative to the
       current directory.

       If none of the above complexity is required, and the operation can be
       run with a single command line entry, you can use the -command option
       to provide the program name and its options, instead of using a master
       command file.

       When Chunksetup is run, it first calls Tomopieces, which analyzes
       the dimensions of the input file to determine how to chop it up opti-
       mally.  It then produces command files for extracting each chunk with
       Taperoutvol and processing them, numbered sequentially with 3-digit
       numbers.  If the name of the master command file is "", the
       first command file is, INPUTFILE is replaced by com-, and OUTPUTFILE is replaced by comfile-001.out, and simi-
       larly for the following files.  If a single-line command is entered
       instead of a master command file, the files are named with the program
       name followed by "-cs" instead.  Each file contains a command to remove, so only the output chunks accumulate as the files are
       run.  Chunksetup also produces a command file to reassemble the pieces,, which will also delete the output chunks and the
       command and log files for each of the chunks.  Finally, it produces a
       command file,, to run all of the chunks in sequence and
       run the finishing command file at the end.

       You have the option of running the top-level command file using "subm",
       or running the individual command files and running the finishing com-
       mand file at the end.  If you have multiple processors available with
       access to the current directory, you can use the generic parallel pro-
       cessing interface in Etomo or the Processchunks script to run the
       jobs in parallel on the various processors or machines, such as with:
       processchunks machine_list comfile
       where machine_list is a list of the machines to use, or just the number
       of processor to use on the local machine.

       Chunksetup uses the PIP package for input (see the manual page for
       pip).  Options can be specified either as command line arguments
       (with the -) or one per line in a command file (without the -).
       Options can be abbreviated to unique letters; the currently valid
       abbreviations for short names are shown in parentheses.

       -master (-ma) OR -MasterComFile     File name
              Name of master command file.  If neither this option nor -com-
              mand is entered, the first non-option argument will be taken as
              this name.  Either a command file or -command must be entered.

       -command (-c) OR -OneLineCommand    Text string
              Full command to run on the input file, including all options
              except ones specifying the input and output files.  To use this
              option, the program being run by the command must accept input
              and output file names as the last non-option arguments on its
              command line.  If there are any options to the command and
              Chunksetup is run with command line arguments, the whole entry
              must be quoted.

       -input (-i) OR -InputImageFile      File name
              Name of input image file.  If this option is not entered, the
              first non-option argument will be taken as this name, unless
              neither -command nor -master is entered as an option, in which
              case the second non-option argument is used.

       -output (-ou) OR -OutputImageFile   File name
              Name of output image file.  If this option is not entered, the
              last non-option argument will be taken as this name unless -suf-
              fix is entered.  Either this entry or -suffix is required.

       -suffix (-s) OR -SuffixForOutputName     Text string
              String to add to the root of the input image file name, before
              extension, to produce the output image file name.  For example,
              if the input name is "setname_rec.mrc", and "filt" is entered
              for this option, the default output file name will be "set-
              name_rec_filt.mrc".  If the input file does not have an exten-
              sion corresponding to an allowed file format, the extension will
              be placed before the suffix.  In the example, an input of "set-
              name.rec" will also result in an output name "set-
              name_rec_filt.mrc".  If the environment variable IMOD_OUT-
              PUT_FORMAT is set or the option -format is entered, the output
              will have the indicated file format and have the appropriate
              extension; otherwise it will be MRC format with extension

       -format (-f) OR -FormatOfOutputFile      Text string
              Set output file format to MRC, TIFF, or HDF by entering MRC,
              TIF, TIFF, HDF, or any lower case form of these.  This entry
              overrides a default format set with the environment variable

       -p OR -PaddingPixels      Integer
              Number of pixels of padding on each edge of each subvolume.  The
              default is 8.

       -o OR -OverlapPixels      Integer
              Minimum number of pixels of overlap between the subvolumes.  The
              default is 8.

       -m OR -MegavoxelMaximum   Integer
              Limit each subvolume to the given number of megavoxels.  The
              default is whatever the default is for Tomopieces (80

       -xm (-x) OR -XMaximumPieces    Integer
              The maximum number of chunks in the X direction.  The default is
              -1, which is effectively no limit.  See the man page for Tomo-
              pieces for more details on the choices for maximum number of

       -ym (-y) OR -YMaximumPieces    Integer
              The maximum number of chunks in the Y direction.  The default is
              -1, which is effectively no limit.

       -no (-n) OR -NoFFTSizes
              Do not increase padding to make suitable sizes for an FFT.  Use
              this option if no FFTs will be taken of the volumes.  It causes
              the "-nofft" option to be sent to Tomopieces and to be added
              to the input to Taperoutvol that is placed in the command

       -param (-pa) OR -ParameterFile      Parameter file
              Read parameter entries as keyword-value pairs from a parameter

       -help (-h) OR -usage
              Print help output

              Read parameter entries from standard input

       David Mastronarde  <mast at colorado dot edu>

       tomopieces, taperoutvol, assemblevol, processchunks

IMOD                                4.12.62                      chunksetup(1)