cpuadoc(1) cpuadoc(1)
NAME
cpu.adoc - Description of computers for parallel processing in eTomo
SYNOPSIS
/usr/local/ImodCalib/cpu.adoc
DESCRIPTION
The cpu.adoc file contains descriptions of the computers you wish to
use when running parallel processes through eTomo with
processchunks. It is used by eTomo to build the parallel processing
table and to run processchunks. If you are just using a single
computer with multiple processors, there are two simple alternatives to
using the cpu.adoc file: 1) enable parallel processing and enter the
number of processors in the Settings dialog opened from the eTomo
Options menu; or 2) define the environment variable IMOD_PROCESSORS to
be the number of processors (see imodenv).
The cpu.adoc file must be placed in the IMOD calibration directory.
The location of this directory is defined by the environment variable
IMOD_CALIB_DIR, which has a default value of /usr/local/ImodCalib with
a default installation of IMOD. You will probably need to create this
directory. Then, prepare a cpu.adoc file and place it there. There is
an example cpu.adoc file in the autodoc directory of the IMOD
distribution; you should be able to use this as a starting point.
Computer Sections and Attributes
[Computer = computer_name]
Each section describes one computer. Computer_name will be used
by eTomo and IMOD to contact the computer via ssh. As an
alternative to listing an actual computer name, the name can
simply be "localhost", in which case programs will be run
directly rather through ssh. A computer should be listed only
once; do not have one section with an actual name and one
section with "localhost".
Computer attributes are name/value pairs found under a computer
section, they have the following syntax:
name = value
The order of the attributes is not important. All attributes
are optional; some are entered in order to change the default
value of an attribute, while others are purely descriptive. For
the descriptive attributes, you can put anything in them that
you want. However, any attribute you do not use will be omitted
from the table, so not using attributes can save screen space in
Etomo. If all of the users are familiar with the attributes of
the computers or you are planning to use all of the computers
every time you run a parallel process, then you do not need to
use anything but the number attribute.
number = number_of_CPUs
The number attribute is necessary if you have a computer with
multiple processors. If you do not enter this attribute, a
single processor is assumed.
gpu[.local] = 1
This attribute means that the GPU (graphics processing unit) of
a video card installed in this computer is available for running
compatible IMOD processes. If the local modifier is used, then
the GPU is only available to an eTomo that is being run on this
computer. Only video cards with CUDA can be used in this way.
type = CPU_type
This attribute goes into the table under the heading CPU Type
and can contain a description of the computer's processor.
exclude-interface = recon|join|pp|peet
Excludes a computer from the Parallel Processing table depending
on the interface in use. Interfaces: Tomogram Reconstruction -
recon, Join - join, Parallel Processing - pp, PEET - peet. Only
one interface can be excluded per computer.
users = user_name[,user_name...]
A list of the users who have login priviledges on the computer.
If a user is not on this list, the computer will be excluded
from the Parallel Processing table when the user runs Etomo.
This attribute is optional. If it is omitted, the computer will
not be excluded based on user name.
speed = CPU_speed
This attribute goes into the table under the heading Speed and
can contain the speed of the computer. If you want to put units
for this attribute in the column header, see the Global
Attributes section below.
memory = amount_of_RAM
This attribute goes into the table under the heading RAM and can
contain the amount of memory the computer has. If you want to
put units for this attribute in the column header, see the
Global Attributes section below.
os = operating_system
This attribute goes into the table under the heading OS and can
contain a description of the operating system the computer is
running.
mountname = name_for_mounting_this_computers_directories_remotely
This attribute specifies the name substituted for %mountname in
a mount rule (see below) when running on this current computer
as the
mountrule.#.local = local_directory_path
mountrule.#.local = remote_directory_path
See the section below on Mount Rules. Mount rules can be
entered as computer attributes in order to specify a rule that
applies only for a specific machine or to override a global
mount rule.
Queue Sections and Attributes
[Queue = queue_name]
Each section describes one queue in a cluster. Queue_name will
be displayed in the parallel processing table and sent to
processchunks with the -Q option, but it does not need to match
the actual name of a queue. Computer section attributes are all
valid in the Queue sections.
command = command_to_run_queuechunk
Required. Queuechunk is a script in IMOD that allows eTomo to
work with specific types of cluster software, but can be
modified or replaced to work with any cluster software. Se the
man page for Queuechunk for the specifications that such a
script must meet. If the version in IMOD is modified or
replaced, it should be given a different name, and placed
somewhere on the IMOD user's path other than IMOD/bin. That
name would be used in place of "queuechunk" in the command
defined in this section.
A PBS example where there is only one queue:
command = queuechunk -t pbs
A PBS example where there are multiple queues:
command = queuechunk -t pbs -q queue_name
This command will be passed in quotes to processchunks. It will
also be used to get the load from the queue.
Global Attributes
Global attributes are name/value pairs found at the top of the file,
before any computer sections. They typically have the following
syntax:
nameA.nameB = value
The order of the attributes is not important.
Version = 1.2
The Version attribute is usually required. It refers to the
syntax used in .adoc files and should be set to 1.2.
units.speed = units_for_the_speed_attributes
The value appears in sub-heading row in the Etomo parallel
processing table under the heading Speed.
units.memory = units_for_the_memory_attributes
The value appears in sub-heading row in the Etomo parallel
processing table under the heading RAM.
units.load = queue_load_header[,queue_load_header...]
Shows how many comma-separated values will be returned by
queuechunk when it returns a queue's load and provides header(s)
for them in the Etomo parallel processing table. The default is
"Load".
max.tilt = maximum_number_of_CPUs_users_should_use_for_tilt
When this attribute is in use, a reminder not to use too many
CPUs will appear next to the tilt parallel processing checkbox.
This number must be a whole number. The purpose of this
attribute is to discourage users from using too many CPUs and
from experiencing diminishing returns because of bottlenecks.
We are currently setting max.tilt to 12, but your mileage will
vary.
max.volcombine = maximum_number_of_CPUs_users_should_use_for_volcombine
Similarly to max.tilt, this attribute will cause a
recommendation on the maximum number of CPUs to appear next to
the volcombine parallel processing checkbox. This step is much
more susceptible than tomogram generation to diminishing returns
from I/O bottlenecks. We are currently setting max.volcombine
to 8.
separate-chunks = value
This attribute can be used to force programs to write chunks
into separate files that will be reassembled at the end of
processing. Without this setting, different instances of the
Tilt program will all write to one output file simultaneously,
which has given problems in one Mac installation. Any value
other than 0 activates the feature.
min.nice = minimum_nice_value
The minimum value of the Nice spinner. The default is 0.
users-column = [0|1]
When this attribute is present and not set to 0, the Users
column will be included in the Parallel Processing table.
mountrule.#.local = local_directory_path|remote_directory_path
See the section below on Mount Rules.
Mount Rules for Local to Remote Path Translations
In order to use parallel processing in IMOD, all computers must be able
to access the directory where the data and command files are located.
However, it is not necessary that the directory be referred to by the
same name on the different computers. When these names differ, you
must provide eTomo with information about how to translate the current
working directory path on the local computer into a path that can be
used to access the directory on the remote computers. This gets tricky
because the true path of a directory, as revealed by a pwd command, may
not be the same as the path that the user enters to get there. Thus,
in setting up path translations, you need to change to a typical
directory and then use pwd to find out what the official path to the
directory is. This is the path that eTomo will see on the local
machine, so you need to work out how this needs to be translated so
that it can be accessed on the remote machines.
As a simple example, each Linux machine in the BL3DEMC used to have a
directory named /localscratch which was accessed from any machine as
/scratch/computer_name (where computer_name is the name of a machine,
without any domain). The required mount rules were entered as:
mountrule.1.local = /localscratch
mountrule.1.remote = /scratch/%mountname
Where %mountname is entered exactly as written and will be substituted
for the appropriate mount name. In our example, the mount name is just
the computer name, but a mount name different from the computer name
can be entered for an individual computer using the mountname
attribute.
For a complicated example, we had a Macintosh running OSX 10.4, and it
mounted our Linux home directories (/home, /home1, /home2) under the
same names. It mounted the Linux machine scratch directories under
/scratch/computer_name. However, when we were running on the Mac and
cd'd to a user's home directory and entered pwd, we got, e.g.,
/private/var/automount/home1/username. When we cd'd to a Linux scratch
directory and entered pwd, we got /private/var/automount/computer_name.
The correct translations can be accomplished with:
mountrule.2.local = /private/var/automount/home
mountrule.2.remote = /home
mountrule.3.local = /private/var/automount
mountrule.3.remote = /scratch
The numbers specify the order in which the rules are applied. Note
that it is important to apply the rule for home first to avoid having
/private/var/automount/home get translated to /scratch/home. Also
note that this one rule works for /home, /home1, and /home2. The
automount names no longer do this on OSX 10.5 and higher, but the
example is still good for illustrating how to deal with complex
situations.
Our Linux machines also used to access the home directories under
/Users on the Mac, by mounting these directories as
/computer_name/username. So we had another mount rule:
mountrule.4.local = /Users
mountrule.4.remote = /%mountname
All of the rules in our two examples are compatible, so they could all
be listed as global mountrules in the same cpu.adoc. If this were not
the case, we could still maintain one file by listing some rules as
local rules, inside the section for a particular computer.
Here are some other facts about mount rules. The current directory is
checked for substitution against one rule at a time, and if it matches
a rule then the substitution is made and no other rule is checked.
Local rules for the current host machine, if any, are checked before
the global rules.
It is required to have a local rule and a remote rule with the same
number and in the same area (global attributes area or Computer
section). Each mount rule attribute must have a value.
When %mountname is used, then a Computer section for the current host
computer must exist, or there must be a Computer section called
localhost. In the latter case, a mountname attribute is required for
that section.
EXAMPLES
A cpu.adoc for a standalone two-processor system would be just:
Version = 1.0
[Computer = localhost]
number = 2
See $IMOD_DIR/autodoc/cpu.adoc for further examples.
LIMITATIONS
Windows computers may not be placed in the same cpu.adoc parallel
processing table as Linux and Macintosh computers.
All computers in the cpu.adoc will be loaded into a scrollable table in
Etomo and ssh connections will be opened to each one to monitor its
load. A cpu.adoc with many tens of computers may slow down Etomo too
much.
SEE ALSO
queuechunk(1)
BL3DEMC 4.5.6 cpuadoc(1)