cpuadoc(1) General Commands Manual cpuadoc(1)
NAME
cpu.adoc - Description of computers for parallel processing in Etomo
SYNOPSIS
/usr/local/ImodCalib/cpu.adoc
DESCRIPTION
The cpu.adoc file contains descriptions of the computers you wish to
use when running parallel processes through Etomo with
processchunks. It is used by Etomo to build the parallel processing
table and to run processchunks. If you are just using a single
computer with multiple processors, there are two simple alternatives to
using the cpu.adoc file: 1) enable parallel processing and enter the
number of processors in the Settings dialog opened from the Etomo
Options menu; or 2) define the environment variable IMOD_PROCESSORS to
be the number of processors (see imodenv). This file also describes
the GPUs available for processing in parallel with multiple graphics
cards. If you have a single computer with multiple GPUs, you need to
use a cpu.adoc; see EXAMPLES below.
The cpu.adoc file must be placed in the IMOD calibration directory.
The location of this directory is defined by the environment variable
IMOD_CALIB_DIR, which has a default value of /usr/local/ImodCalib with
a default installation of IMOD. You will probably need to create this
directory. Then, prepare a cpu.adoc file and place it there. There is
an example cpu.adoc file in the autodoc directory of the IMOD
distribution; you should be able to use this as a starting point.
Computer Sections and Attributes
[Computer = computer_name]
Each section describes one computer. Computer_name will be used
by Etomo and IMOD to contact the computer via ssh. As an
alternative to listing an actual computer name, the name can
simply be "localhost", in which case programs will be run
directly rather through ssh. A computer should be listed only
once; do not have one section with an actual name and one
section with "localhost".
Computer attributes are name/value pairs found under a computer
section, they have the following syntax:
name = value
The order of the attributes is not important. All attributes
are optional; some are entered in order to change the default
value of an attribute, while others are purely descriptive. For
the descriptive attributes, you can put anything in them that
you want. However, any attribute you do not use will be omitted
from the table, so not using attributes can save screen space in
Etomo. If all of the users are familiar with the attributes of
the computers or you are planning to use all of the computers
every time you run a parallel process, then you do not need to
use anything but the number attribute.
number = number_of_cores
The maximum number of processor cores available for simultaneous
use on this system. If you do not enter this attribute a single
core is assumed. Hyperthreading is often not very effective for
computational processes, so this number should generally be
limited to the number of true cores. For a report of the number
of physical cores, enter
imodqtassist -t
(For use on clusters / queue servers, please see Queue Sections
and Attributes, below, for a different interpretation applicable
to those cases).
gpu[.local] = 1
This attribute means that the GPU (graphics processing unit) of
a video card installed in this computer is available for running
compatible IMOD processes. In fact, any value besides 1 also
indicates that there is a GPU, so the entry needs to be
commented out or removed to make a GPU unavailable. If the
local modifier is used, then the GPU is only available to an
Etomo that is being run on this computer. Only video cards with
CUDA can be used in this way.
gpu.device = GPU#[,GPU#...]
Use this attribute instead of gpu = 1 when multiple GPUs are
available. GPUs will be added to the processchunks call in the
same order as they are listed in this attribute. GPUs are
numbered from one in this entry, as in the entries to various
programs, because "0" refers to the best available GPU rather
than a specific one. This differs from the numbering (from 0)
in the output from nvidia-smi.
type = CPU_type
This attribute goes into the table under the heading CPU Type
and can contain a description of the computer's processor.
exclude-interface =
batchRunTomo|join|peet|pp|recon|serialSections|tools
Excludes a computer from the Parallel Processing table depending
on the interface in use. Only one interface can be excluded per
computer. "batchRunTomo" refers to the Batch Tomograms
interface. "pp" refers to the Nonlinear Anisotropic Diffusion
interface, and the Generic Parallel Process interface. "recon"
refers to the Build Tomogram interface. "tools" refers to the
Flatten Value interface, and the Test GPU interface.
users = user_name[,user_name...]
A list of the users who have login priviledges on the computer.
If a user is not on this list, the computer will be excluded
from the Parallel Processing table when the user runs Etomo.
This attribute is optional. If it is omitted, the computer will
not be excluded based on user name.
speed = CPU_speed
This attribute goes into the table under the heading Speed and
can contain the speed of the computer. If you want to put units
for this attribute in the column header, see the Global
Attributes section below.
memory = amount_of_RAM
This attribute goes into the table under the heading RAM and can
contain the amount of memory the computer has. If you want to
put units for this attribute in the column header, see the
Global Attributes section below.
os = operating_system
This attribute goes into the table under the heading OS and can
contain a description of the operating system the computer is
running.
gpu.type = GPU_model
This attribute goes into the GPU table under the heading Type
and can contain a model number for the graphics card.
gpu.ncores = Number_of_GPU_cores
This attribute goes into the GPU table under the heading Cores
and can contain the number of CUDA cores.
gpu.speed = GPU_clock_rate
This attribute goes into the GPU table under the heading Speed
and can contain the clock rate of the GPU. If you want to put
units for this attribute in the column header, see the Global
Attributes section below.
gpu.memory = amount_of_GPU_RAM
This attribute goes into the GPU table under the heading RAM and
can contain the amount of memory the graphics card has. If you
want to put units for this attribute in the column header, see
the Global Attributes section below.
mountname = name_for_mounting_this_computers_directories_remotely
This attribute specifies the name substituted for %mountname in
a mount rule (see below) when running on this current computer
as the
mountrule.#.local = local_directory_path
mountrule.#.local = remote_directory_path
See the section below on Mount Rules. Mount rules can be
entered as computer attributes in order to specify a rule that
applies only for a specific machine or to override a global
mount rule.
Queue Sections and Attributes
[Queue = queue_name]
Each section describes one queue in a cluster. Queue_name will
be displayed in the parallel processing table and sent to
processchunks with the -Q option, but it does not need to match
the actual name of a queue. Some Computer section attributes
(e.g. number) are also valid in Queue but sections can have
altered meaning; most will have no effect. The gpu attribute can
be used to indicate that a queue will assign one GPU per job,
which is suitable for use with the processing steps that can use
a GPU in the Tomogram Reconstruction interface of Etomo. The
gpu.local and gpu.device attributes will have no effect, and
various descriptions for the table will have not show up
anywhere.
number = number_of_processing_units
The maximum number of jobs to be run simultaneously on the queue
(required).
Jobs run from the Etomo Tomogram Reconstruction or PEET
interfaces are single-threaded and will each use only a single
core. In these cases, the maximum number of jobs and the maximum
number of cores are identical, and cores may be scattered across
multiple nodes.
Conversely, jobs created with Batchruntomo can be multi-
threaded and can each use multiple cores if coresPerClusterJob
is defined. In this case, the maximum total number of cores used
will be number times coresPerClusterJob.
For a slurm queue running in exclusive access mode, entire nodes
must be allocated, so number must be evenly divisible by
coresPerNode, and the resulting quotient will be the number of
nodes requested at initialization.
command = command_to_run_queuechunk
Required. Queuechunk is a script in IMOD that allows Etomo to
work with specific types of cluster software, but can be
modified or replaced to work with any cluster software. See the
man page for Queuechunk for the list of supported cluster
types and the specifications that a modified script must meet.
If the version in IMOD is modified or replaced, it should be
given a different name, and placed somewhere on the IMOD user's
path other than IMOD/bin. That name would be used in place of
"queuechunk" in the command defined in this section.
A PBS example where there is only one queue:
command = queuechunk -t pbs
A PBS example where there are multiple queues:
command = queuechunk -t pbs -q queue_name
This command will be passed in quotes to processchunks. It will
also be used to get the load from the queue.
initialize = command_to_send_queuechuck_at_start
Required for a slurm queue set up in exclusive allocation mode.
Queuechunk will run this command on the cluster machine at
the beginning of a Processchunks run. For a slurm queue,
this is needed to allocate the desired number of nodes. To use
the script slurmInit.sh distributed with IMOD, enter:
initialize = slurmInit.sh %{nodes} QOS
where QOS is the quality of service, which is used like a queue
name in slurm, and %{nodes} is substituted with a value computed
as explained below for the coresPerNode entry.
deinitialize = command_to_send_queuechuck_at_end
Required for a slurm queue set up in exclusive allocation mode.
Queuechunk will run this command on the cluster machine at
the end of a Processchunks run. For a slurm queue, this is
needed to release the allocation. To use the script
slurmCleanup.sh distributed with IMOD, enter:
initialize = slurmCleanup.sh
coresPerNode = #_of_cores_per_allocated_unit
When using exclusive allocation mode, this entry is used to
compute the number of nodes to request in the initialization
command for a slurm queue from the number of cores that the user
requests. The computed number is substituted for the %{nodes}
variable in the initialize command.
coresPerClusterJob #_of_cores_available_to_job
This entry is used to indicate the number of cores available to
a job running on a particular queue; it is used when running
Batchruntomo in parallel. Do not use this entry with a slurm
queue running in exclusive allocation mode.
gpusPerClusterJob = #_of_GPUs_available_to_job
This entry is used to indicate the number of GPUs available to a
job running on a particular queue; this is used when running
Batchruntomo in parallel. The attribute gpu = 1 can be used
instead when there is only one GPU. Also, this attribute was
originally named gpusPerNode prior to IMOD 4.12.40, and that
name is still accepted. Do not use this entry with a slurm
queue running in exclusive allocation mode.
pc-option.option = value
This entry will pass the specified option to Processchunks
with the given value only when running on this queue. See the
pc-option.option description in the the Global Attributes
section below. An entry here will override a value specified in
the Global section.
Global Attributes
Global attributes are name/value pairs found at the top of the file,
before any computer sections. They typically have the following
syntax:
nameA.nameB = value
The order of the attributes is not important.
Version = 1.2
The Version attribute is usually required. It refers to the
syntax used in .adoc files and should be set to 1.2.
units.speed = units_for_the_speed_attributes
The value appears in sub-heading row in the Etomo parallel
processing table under the heading Speed.
units.memory = units_for_the_memory_attributes
The value appears in sub-heading row in the Etomo parallel
processing table under the heading RAM.
units.gpu.speed = units_for_the_GPU_speed_attributes
The value appears in sub-heading row in the Etomo GPU parallel
processing table under the heading Speed.
units.gpu.memory = units_for_the_GPU_memory_attributes
The value appears in sub-heading row in the Etomo GPU parallel
processing table under the heading RAM.
units.load = queue_load_header[,queue_load_header...]
Shows how many comma-separated values will be returned by
queuechunk when it returns a queue's load and provides header(s)
for them in the Etomo parallel processing table. The default is
"Load".
max.tilt = maximum_number_of_cores_users_should_use_for_tilt
When this attribute is in use, a reminder not to use too many
cores will appear next to the tilt parallel processing checkbox.
This number must be a whole number. The purpose of this
attribute is to discourage users from using too many cores and
from experiencing diminishing returns because of bottlenecks.
We are currently setting max.tilt to 12, but this is an old
limit.
max.volcombine =
maximum_number_of_cores_users_should_use_for_volcombine
Similarly to max.tilt, this attribute will cause a
recommendation on the maximum number of cores to appear next to
the volcombine parallel processing checkbox. This step is much
more susceptible than tomogram generation to diminishing returns
from I/O bottlenecks. We are currently setting max.volcombine
to 8.
separate-chunks = value
This attribute can be used to force programs to write chunks
into separate files that will be reassembled at the end of
processing. Without this setting, different instances of the
Tilt program will all write to one output file simultaneously,
which has given problems in one Mac installation. Any value
other than 0 activates the feature.
min.nice = minimum_nice_value
The minimum value of the Nice spinner. The default is 0.
users-column = [0|1]
When this attribute is present and not set to 0, the Users
column will be included in the Parallel Processing table.
mountrule.#.local = local_directory_path|remote_directory_path
See the section below on Mount Rules.
pc-option.option = value
This and other pc-option attributes allow the specified option
to be passed to Processchunks with the given value. Such
options are placed at the end of the argument list so that they
can override the value passed by Etomo, if any. The option
would be specied without a leading dash. Meaningful options
that would not conflict with the ones managed by Etomo would be
d, e, C, T, L, v, V, and possibly n and O. An option specified
with this entry would be passed on all Processchunks runs,
although the value may be overridden by other more specific
entries.
pc-option.computer.option = value
An option specified with this entry will be passed only on
Processchunks runs with computers, not queues. The value
would override a value specified with a global pc-option.option
entry.
pc-option.queue.option = value
An option specified with this entry will be passed only when
running on queue. The value would override a value specified
with a global pc-option.option entry.
Mount Rules for Local to Remote Path Translations
In order to use parallel processing in IMOD, all computers must be able
to access the directory where the data and command files are located.
However, it is not necessary that the directory be referred to by the
same name on the different computers. When these names differ, you
must provide Etomo with information about how to translate the current
working directory path on the local computer into a path that can be
used to access the directory on the remote computers. This gets tricky
because the true path of a directory, as revealed by a pwd command, may
not be the same as the path that the user enters to get there. Thus,
in setting up path translations, you need to change to a typical
directory and then use pwd to find out what the official path to the
directory is. This is the path that Etomo will see on the local
machine, so you need to work out how this needs to be translated so
that it can be accessed on the remote machines.
As a simple example, each Linux machine in our laboratory used to have
a directory named /localscratch which was accessed from any machine as
/scratch/computer_name (where computer_name is the name of a machine,
without any domain). The required mount rules were entered as:
mountrule.1.local = /localscratch
mountrule.1.remote = /scratch/%mountname
Where %mountname is entered exactly as written and will be substituted
for the appropriate mount name. In our example, the mount name is just
the computer name, but a mount name different from the computer name
can be entered for an individual computer using the mountname
attribute.
For a complicated example, we had a Macintosh running OSX 10.4, and it
mounted our Linux home directories (/home, /home1, /home2) under the
same names. It mounted the Linux machine scratch directories under
/scratch/computer_name. However, when we were running on the Mac and
cd'd to a user's home directory and entered pwd, we got, e.g.,
/private/var/automount/home1/username. When we cd'd to a Linux scratch
directory and entered pwd, we got /private/var/automount/computer_name.
The correct translations can be accomplished with:
mountrule.2.local = /private/var/automount/home
mountrule.2.remote = /home
mountrule.3.local = /private/var/automount
mountrule.3.remote = /scratch
The numbers specify the order in which the rules are applied. Note
that it is important to apply the rule for home first to avoid having
/private/var/automount/home get translated to /scratch/home. Also
note that this one rule works for /home, /home1, and /home2. The
automount names no longer do this on OSX 10.5 and higher, but the
example is still good for illustrating how to deal with complex
situations.
Our Linux machines also used to access the home directories under
/Users on the Mac, by mounting these directories as
/computer_name/username. So we had another mount rule:
mountrule.4.local = /Users
mountrule.4.remote = /%mountname
All of the rules in our two examples are compatible, so they could all
be listed as global mountrules in the same cpu.adoc. If this were not
the case, we could still maintain one file by listing some rules as
local rules, inside the section for a particular computer.
Here are some other facts about mount rules. The current directory is
checked for substitution against one rule at a time, and if it matches
a rule then the substitution is made and no other rule is checked.
Local rules for the current host machine, if any, are checked before
the global rules.
It is required to have a local rule and a remote rule with the same
number and in the same area (global attributes area or Computer
section). Each mount rule attribute must have a value.
When %mountname is used, then a Computer section for the current host
computer must exist, or there must be a Computer section called
localhost. In the latter case, a mountname attribute is required for
that section.
EXAMPLES
A cpu.adoc for a standalone four-core system would be just:
Version = 1.2
[Computer = localhost]
number = 4
A cpu.adoc for a standalone 16-core system with 4 equivalent GPUs would
be:
Version = 1.2
[Computer = localhost]
number = 16
gpu.device=1,2,3,4
Both of these files are in fact unnecessary because they can be
configured in the Etomo Options-Settings dialog. See
$IMOD_DIR/autodoc/cpu.adoc and Splitbatch(1) for further examples.
LIMITATIONS
Windows computers may not be placed in the same cpu.adoc parallel
processing table as Linux and Macintosh computers.
All computers in the cpu.adoc will be loaded into a scrollable table in
Etomo and ssh connections will be opened to each one to monitor its
load. A cpu.adoc with many tens of computers may slow down Etomo too
much.
SEE ALSO
queuechunk(1)
IMOD 4.12.61 cpuadoc(1)