cpuadoc(1)                  General Commands Manual                 cpuadoc(1)

       cpu.adoc - Description of computers for parallel processing in Etomo


       The cpu.adoc file contains descriptions of the computers you wish to
       use when running parallel processes through Etomo with
       processchunks.  It is used by Etomo to build the parallel processing
       table and to run processchunks.  If you are just using a single
       computer with multiple processors, there are two simple alternatives to
       using the cpu.adoc file: 1) enable parallel processing and enter the
       number of processors in the Settings dialog opened from the Etomo
       Options menu; or 2) define the environment variable IMOD_PROCESSORS to
       be the number of processors (see imodenv).

       The cpu.adoc file must be placed in the IMOD calibration directory.
       The location of this directory is defined by the environment variable
       IMOD_CALIB_DIR, which has a default value of /usr/local/ImodCalib with
       a default installation of IMOD.  You will probably need to create this
       directory.  Then, prepare a cpu.adoc file and place it there.  There is
       an example cpu.adoc file in the autodoc directory of the IMOD
       distribution; you should be able to use this as a starting point.

   Computer Sections and Attributes
       [Computer = computer_name]
              Each section describes one computer.  Computer_name will be used
              by Etomo and IMOD to contact the computer via ssh.  As an
              alternative to listing an actual computer name, the name can
              simply be "localhost", in which case programs will be run
              directly rather through ssh.  A computer should be listed only
              once; do not have one section with an actual name and one
              section with "localhost".

              Computer attributes are name/value pairs found under a computer
              section, they have the following syntax:
              name = value
              The order of the attributes is not important.  All attributes
              are optional; some are entered in order to change the default
              value of an attribute, while others are purely descriptive.  For
              the descriptive attributes, you can put anything in them that
              you want.  However, any attribute you do not use will be omitted
              from the table, so not using attributes can save screen space in
              Etomo.  If all of the users are familiar with the attributes of
              the computers or you are planning to use all of the computers
              every time you run a parallel process, then you do not need to
              use anything but the number attribute.

       number = number_of_CPUs
              The number attribute is necessary if you have a computer with
              multiple processors.  If you do not enter this attribute, a
              single processor is assumed.

       gpu[.local] = 1
              This attribute means that the GPU (graphics processing unit) of
              a video card installed in this computer is available for running
              compatible IMOD processes.  In fact, any value besides 1 also
              indicates that there is a GPU, so the entry needs to be
              commented out or removed to make a GPU unavailable.  If the
              local modifier is used, then the GPU is only available to an
              Etomo that is being run on this computer.  Only video cards with
              CUDA can be used in this way.

       gpu.device = GPU#[,GPU#...]
              Use this attribute instead of gpu = 1 when multiple GPUs are
              available.  GPUs will be added to the processchunks call in the
              same order as they are listed in this attribute.  GPUs are
              numbered from one in this entry, as in the entries to various
              programs, because "0" refers to the best available GPU rather
              than a specific one.  This differs from the numbering (from 0)
              in the output from nvidia-smi.

       type = CPU_type
              This attribute goes into the table under the heading CPU Type
              and can contain a description of the computer's processor.

       exclude-interface =
              Excludes a computer from the Parallel Processing table depending
              on the interface in use.  Only one interface can be excluded per
              computer.  "batchRunTomo" refers to the Batch Tomograms
              interface.  "pp" refers to the Nonlinear Anisotropic Diffusion
              interface, and the Generic Parallel Process interface.  "recon"
              refers to the Build Tomogram interface.  "tools" refers to the
              Flatten Value interface, and the Test GPU interface.

       users = user_name[,user_name...]
              A list of the users who have login priviledges on the computer.
              If a user is not on this list, the computer will be excluded
              from the Parallel Processing table when the user runs Etomo.
              This attribute is optional.  If it is omitted, the computer will
              not be excluded based on user name.

       speed = CPU_speed
              This attribute goes into the table under the heading Speed and
              can contain the speed of the computer.  If you want to put units
              for this attribute in the column header, see the Global
              Attributes section below.

       memory = amount_of_RAM
              This attribute goes into the table under the heading RAM and can
              contain the amount of memory the computer has.  If you want to
              put units for this attribute in the column header, see the
              Global Attributes section below.

       os = operating_system
              This attribute goes into the table under the heading OS and can
              contain a description of the operating system the computer is

       mountname = name_for_mounting_this_computers_directories_remotely
              This attribute specifies the name substituted for %mountname in
              a mount rule (see below) when running on this current computer
              as the
              mountrule.#.local = local_directory_path
              mountrule.#.local = remote_directory_path
              See the section below on Mount Rules.  Mount rules can be
              entered as computer attributes in order to specify a rule that
              applies only for a specific machine or to override a global
              mount rule.

   Queue Sections and Attributes
       [Queue = queue_name]
              Each section describes one queue in a cluster.  Queue_name will
              be displayed in the parallel processing table and sent to
              processchunks with the -Q option, but it does not need to match
              the actual name of a queue.  Computer section attributes are all
              valid in the Queue sections.

       command = command_to_run_queuechunk
              Required.  Queuechunk is a script in IMOD that allows Etomo to
              work with specific types of cluster software, but can be
              modified or replaced to work with any cluster software.  Se the
              man page for Queuechunk for the specifications that such a
              script must meet.  If the version in IMOD is modified or
              replaced, it should be given a different name, and placed
              somewhere on the IMOD user's path other than IMOD/bin.  That
              name would be used in place of "queuechunk" in the command
              defined in this section.

              A PBS example where there is only one queue:
              command = queuechunk -t pbs

              A PBS example where there are multiple queues:
              command = queuechunk -t pbs -q queue_name

              This command will be passed in quotes to processchunks.  It will
              also be used to get the load from the queue.

       initialize = command_to_send_queuechuck_at_start
              Required for a slurm queue set up in exclusive allocation mode.
              Queuechunk will run this command on the cluster machine at
              the beginning of a Processchunks run.  For a slurm queue,
              this is needed to allocate the desired number of nodes.  To use
              the script slurmInit.sh distributed with IMOD, enter:
              initialize = slurmInit.sh %{nodes} QOS

              where QOS is the quality of service, which is used like a queue
              name in slurm.

       deinitialize = command_to_send_queuechuck_at_end
              Required for a slurm queue set up in exclusive allocation mode.
              Queuechunk will run this command on the cluster machine at
              the end of a Processchunks run.  For a slurm queue, this is
              needed to release the allocation.  To use the script
              slurmCleanup.sh distributed with IMOD, enter:
              initialize = slurmCleanup.sh

       coresPerNode = number_of_cores_per_allocated_unit
              This entry is used to compute the number of nodes to request in
              the initialization command for a slurm queue from the number of
              cores that the user requests.  The computed number is
              substituted for the %{nodes} variable.

   Global Attributes
       Global attributes are name/value pairs found at the top of the file,
       before any computer sections.  They typically have the following
       nameA.nameB = value
       The order of the attributes is not important.

       Version = 1.2
              The Version attribute is usually required.  It refers to the
              syntax used in .adoc files and should be set to 1.2.

       units.speed = units_for_the_speed_attributes
              The value appears in sub-heading row in the Etomo parallel
              processing table under the heading Speed.

       units.memory = units_for_the_memory_attributes
              The value appears in sub-heading row in the Etomo parallel
              processing table under the heading RAM.

       units.load = queue_load_header[,queue_load_header...]
              Shows how many comma-separated values will be returned by
              queuechunk when it returns a queue's load and provides header(s)
              for them in the Etomo parallel processing table.  The default is

       max.tilt = maximum_number_of_CPUs_users_should_use_for_tilt
              When this attribute is in use, a reminder not to use too many
              CPUs will appear next to the tilt parallel processing checkbox.
              This number must be a whole number.  The purpose of this
              attribute is to discourage users from using too many CPUs and
              from experiencing diminishing returns because of bottlenecks.
              We are currently setting max.tilt to 12, but your mileage will

       max.volcombine = maximum_number_of_CPUs_users_should_use_for_volcombine
              Similarly to max.tilt, this attribute will cause a
              recommendation on the maximum number of CPUs to appear next to
              the volcombine parallel processing checkbox.  This step is much
              more susceptible than tomogram generation to diminishing returns
              from I/O bottlenecks.  We are currently setting max.volcombine
              to 8.

       separate-chunks = value
              This attribute can be used to force programs to write chunks
              into separate files that will be reassembled at the end of
              processing.  Without this setting, different instances of the
              Tilt program will all write to one output file simultaneously,
              which has given problems in one Mac installation.  Any value
              other than 0 activates the feature.

       min.nice = minimum_nice_value
              The minimum value of the Nice spinner.  The default is 0.

       users-column = [0|1]
              When this attribute is present and not set to 0, the Users
              column will be included in the Parallel Processing table.

       mountrule.#.local = local_directory_path|remote_directory_path
              See the section below on Mount Rules.

   Mount Rules for Local to Remote Path Translations
       In order to use parallel processing in IMOD, all computers must be able
       to access the directory where the data and command files are located.
       However, it is not necessary that the directory be referred to by the
       same name on the different computers.  When these names differ, you
       must provide Etomo with information about how to translate the current
       working directory path on the local computer into a path that can be
       used to access the directory on the remote computers.  This gets tricky
       because the true path of a directory, as revealed by a pwd command, may
       not be the same as the path that the user enters to get there.  Thus,
       in setting up path translations, you need to change to a typical
       directory and then use pwd to find out what the official path to the
       directory is.  This is the path that Etomo will see on the local
       machine, so you need to work out how this needs to be translated so
       that it can be accessed on the remote machines.

       As a simple example, each Linux machine in our laboratory used to have
       a directory named /localscratch which was accessed from any machine as
       /scratch/computer_name (where computer_name is the name of a machine,
       without any domain).  The required mount rules were entered as:

       mountrule.1.local = /localscratch
       mountrule.1.remote = /scratch/%mountname

       Where %mountname is entered exactly as written and will be substituted
       for the appropriate mount name.  In our example, the mount name is just
       the computer name, but a mount name different from the computer name
       can be entered for an individual computer using the mountname

       For a complicated example, we had a Macintosh running OSX 10.4, and it
       mounted our Linux home directories (/home, /home1, /home2) under the
       same names.  It mounted the Linux machine scratch directories under
       /scratch/computer_name.  However, when we were running on the Mac and
       cd'd to a user's home directory and entered pwd, we got, e.g.,
       /private/var/automount/home1/username.  When we cd'd to a Linux scratch
       directory and entered pwd, we got /private/var/automount/computer_name.

       The correct translations can be accomplished with:

       mountrule.2.local = /private/var/automount/home
       mountrule.2.remote = /home
       mountrule.3.local = /private/var/automount
       mountrule.3.remote = /scratch

       The numbers specify the order in which the rules are applied.  Note
       that it is important to apply the rule for home first to avoid having
       /private/var/automount/home get translated to  /scratch/home.  Also
       note that this one rule works for /home, /home1, and /home2.  The
       automount names no longer do this on OSX 10.5 and higher, but the
       example is still good for illustrating how to deal with complex

       Our Linux machines also used to access the home directories under
       /Users on the Mac, by mounting these directories as
       /computer_name/username.  So we had another mount rule:

       mountrule.4.local = /Users
       mountrule.4.remote = /%mountname

       All of the rules in our two examples are compatible, so they could all
       be listed as global mountrules in the same cpu.adoc.  If this were not
       the case, we could still maintain one file by listing some rules as
       local rules, inside the section for a particular computer.

       Here are some other facts about mount rules.  The current directory is
       checked for substitution against one rule at a time, and if it matches
       a rule then the substitution is made and no other rule is checked.
       Local rules for the current host machine, if any, are checked before
       the global rules.

       It is required to have a local rule and a remote rule with the same
       number and in the same area (global attributes area or Computer
       section).  Each mount rule attribute must have a value.

       When %mountname is used, then a Computer section for the current host
       computer must exist, or there must be a Computer section called
       localhost.  In the latter case, a mountname attribute is required for
       that section.

       A cpu.adoc for a standalone two-processor system would be just:
       Version = 1.0
       [Computer = localhost]
       number = 2

       See $IMOD_DIR/autodoc/cpu.adoc for further examples.

       Windows computers may not be placed in the same cpu.adoc parallel
       processing table as Linux and Macintosh computers.

       All computers in the cpu.adoc will be loaded into a scrollable table in
       Etomo and ssh connections will be opened to each one to monitor its
       load.  A cpu.adoc with many tens of computers may slow down Etomo too


IMOD                                4.9.10                          cpuadoc(1)