Rick Gaudette and David Mastronarde
University of Colorado, Boulder
Using Parallel Processing
Other Interfaces in eTomo
eTomo provides a graphical user interface (GUI) to the IMOD Tomography package. The general user interface design idea on which eTomo is built is to present a separate panel for each distinct step in computing a tomogram with the IMOD package. This allows the user to focus on the current step in the processing sequence, presenting only the parameters and controls required for the current step.
eTomo can be started from the command line with the command
It may take a little while to come up; Java applications have a slow startup time. You can also open up a previous dataset by specifying the relative path to the .edf file on the command line, such as
When eTomo is first started, it shows its Front Page, which contains a set of buttons for starting different kinds of interfaces (see below). At this point you can either open a dataset that was previously being worked on by selecting it from the recent file list in the File menu, or you can use one of the buttons to open an interface for working on a new dataset.
Selecting New Tomogram will bring up the tomogram Setup Panel. To start working on a new dataset you need to fill out the fields in the Setup Panel.
The Dataset Name is the name of the file containing the projection sequence (or the name of one of these files in the case of a dual axis tomogram). You can enter the Dataset Name by opening the file selection dialog window using the folder labeled button associated with the Dataset Name field. Alternatively, you may enter the relative path to the projection sequence file directly into the edit box.
The Backup Directory is optionally available to save small working files every time you run a procedure. You can leave this field empty if you do not wish to use a backup directory.
Specify the pixel size, fiducial size and image rotation in
the appropriate fields. If the header of your image file includes pixel
size and/or image rotation information, press the
button to extract these values.
Specify the source for the tilt angles for either one or both axes, as appropriate. You can also optionally specify individual projections to exclude from the processing. The syntax for this exclude list is a comma separated list of ranges (i.e. 1,4-5,60-70)
Press the Create com scripts button to move on to generating the tomogram. You can also use an existing set of com scripts by pressing the Use existing coms button; just be sure that the setup parameters you have entered exactly match those that were entered using copytomocoms.
To return to a previous tomogram, select the .edf (eTomo Data File) file from the most recently used list on the bottom of the File menu, or the Open... item on the File menu if the .edf is not listed in the most recently used list. The file dialog will let you navigate around to select the .edf file associated with the tomogram you wish to work on.
The main window consists of several areas: on the left is a column of buttons (Process Control Buttons) that allow you to select the particular step of the tomogram computation to work on, the top center is the Process Monitor that will inform you of the status of the current process or the last process completed and a button to kill the current process, the bottom status bar presents some basic information about the dataset you are working on. If you are working on a dual axis tomogram this layout will be repeated for both the A and B axes.
The Process Control Buttons are arranged in the
suggested order of processing from top to bottom and are color coded to signify
the state of the process. Red indicates that the process has not been started,
magenta indicates that the process is in progress, and green indicates that the
process has been completed. The state of the process is also indicated in text
on the button. When one of the buttons is selected the right side of the window
will fill in with information and fields associated with that process; these
forms are called Process Panels. They allow you to modify the necessary
parameters and execute the programs required by that processing step. The
parameters and buttons on each Process Panel are typically laid out from
top to bottom in the order they should be executed. When you execute a process
(by pressing a button on one of the Process Panels) the Process Monitor
will indicate what the process is doing and when it is complete. In many
cases, the button that you pressed will turn dark to indicate that the
operation has been completed.
The text output of a particular process is written to a log file named for the particular process; this log file can be viewed by right clicking the mouse pointer over the window region associated with the process. This will open a menu that is split into three or four sections: the entries in the first section will open up the log files associated with the current process; some menus will have a second section with choices for plotting various kinds of information; the next section will allow you to open up man pages associated with the current process; the final menu section opens the general help documents. There is more information regarding online help resources in the section Accessing Help.
When you are done with all of the processes for a given step press the Done button; this will inform eTomo that the step is complete and will also move the GUI forward to the next step in the tomogram processing. Pressing the Postpone button will save all of the current parameters for that step and pressing Cancel will abort all of the changes made since the last process was run for that step. If you move between process panels by pressing the buttons in the left column, this will save the current parameters just as a Postpone does. The final button of interest at the bottom of most processing panels is the Advanced button. This will present a much more complete list of parameters for each step, which are mostly needed for unique or extreme cases.
Since the number of parameters that a particular step requires
can vary significantly,
the amount of
screen space the GUI needs to present that information also changes. The
current version of the program will adjust the window size automatically to fit
tightly around the Process Panel whenever you change screens or change
the amount of information displayed (such as by changing to advanced
mode). You make this fitting occur automatically by selecting the Options/Fit
Window menu item or using Ctrl-F hot key. Whenever the window is
too small to display the information, scroll bars will appear so that you can
pan across the Process Panel. To reduce the need for scrolling,
several of the large panels are divided into sections that can be opened or
closed independently with a button in the upper-left corner. This button
is initially "-" when the section is
open, and becomes "+" when the section
is closed. In addition, a
button may appear in the upper right corner to allow a section to be switched
to advanced or basic mode independently of the others; this button will be
in basic mode and
If you are working on a dual axis tomogram, each tomogram is
processed in a separate window. The upper left corner of the
window (or the
window, if it is the only one open) will have buttons to allow you to switch
between displaying one or the other axis or both axes. There are also hot
keys, Ctrl-A, Ctrl-B, and Ctrl-2, to display axis A, axis B, or both.
Every process panel has one or more buttons for starting 3dmod
to display an image file. To make it easier to deal with very large
datasets and problems that require more control over the way that 3dmod is run,
there are two ways to make 3dmod load in images binned by 2 or to make it open
with its startup window, so that options can be selected. One way is to
right-click on one of these buttons, then select
by 2. The other way is to select the corresponding items in the
menu, which will make 3dmod open that way every time.
For many processing steps, it is also possible to run the
operation and have the resulting images loaded into 3dmod automatically.
Right click on the button that starts the action (e.g., Generate Tomogram) and
menu will pop up with the choices
And open 3dmod,
And open 3dmod with startup window,
And open 3dmod binned by 2. Select
the desired option to start the operation.
It is possible to load more than one dataset into eTomo.
menu there are entries to open an existing dataset, open a new Setup window for
tomogram processing, or open a new window to setup joining serial
sections. If you start working on more than one dataset by any of these
routes, eTomo will place each dataset in a separate tab so that you can go
between them easily.
dialog presents several configuration options, such as one to change the font
size being used, and one to switch to a compact display in which the buttons on
the left are very small. It may be necessary to close and reopen the
program to see options like these take full effect.
When you start processing a tomogram, a separate window will be opened called a project log. This window will receive selected output from various operations, and error messages that occur. You can also type your own notes into the window. There is an entry under the View menu, and a hot key Ctrl-L, for hiding or making it visible again. Even when it is hidden, output will be saved to it. The contents are automatically saved to a file named with the dataset name and "_project.log".
One of the significant features of the eTomo GUI is the immediate access to context sensitive online help. By context sensitive we mean that the help being presented is specific to the current processing problem at hand. Access to the online help is available through two methods. The first, commonly know as Tooltips, is accessed by leaving the mouse stationary over the parameter or other GUI object of interest. This will pop up a short text description of the parameter or function of the object. The object being described can be changed by moving the mouse around or the description can be removed by moving the mouse over a blank window region. The description will also be dismissed (disappear) after a short time if the mouse is not moved. Both the amount of time to delay displaying the Tooltip and the amount of time to keep the Tooltip visible can be set in the Options/Settings dialog.
The second method to access context sensitive help is by right
clicking the mouse over the processing region of interest. This will open up a
pop-up menu containing three sections. The top section presents menu items that
will open up a new window containing a log file appropriate for the current
processing step. The second section presents menu items that will open up the
appropriate man pages for the current processing step. The final section
provides access to the Tomography Guide, Serial Section Joining Guide, IMOD
Guide and the 3dmod Introduction. If the Tomography Guide is selected the Guide
will be opened up to the section that is appropriate for the current processing
step. All documentation is opened in the Qt Assistant, so that you can
follow hyperlinks and return from them, as well as have access to the full IMOD
documentation using the sidebar on the left.
The current state of the processing and the value of any
parameters you entered or changed (that are not saved in the underlying com
scripts) are saved automatically to a file with the dataset name and the
extension .edf. If you are working on joining serial sections, the state
is saved instead to a file with extension .ejf. The state file is updated
whenever you run a process, change process panels, or exit the program.
If you wish to save this file with a different name, use the File/Save as
menu item. Be aware that if you reopen an earlier version of the .edf
file you may get unpredictable results.
Default settings for many tilt series processing parameters can be customized in files called template files. To provide flexibility, eTomo is set up to work with three different levels of template files, although there is no need to use all three. The three levels are:
Scope template: These files are intended to be used for parameters specific to a microscope, a combination of microscope and camera, or perhaps a combination of microscope and some imaging condition, such as voltage and spherical aberration for CTF correction, or image distortion correction files. These files should be placed in a subdirectory ScopeTemplate of the directory pointed to by the environment variable IMOD_CALIB_DIR, which is /usr/local/ImodCalib by default.
System template: These files are meant to hold settings that are available to all users on the system. eTomo will look for these files first in a subdirectory SystemTemplate under the installed IMOD directory, then in a subdirectory SystemTemplate under the directory pointed to by IMOD_CALIB_DIR. When eTomo sees a file with the same name in latter location as in the IMOD directory, it ignores the one from the IMOD distribution.Thus, if you want to customize a template distributed with IMOD, you can modify it and save it with the same name in the $IMOD_CALIB_DIR/SystemTemplate directory.
User template: These files hold settings for an individual user. eTomo will look for these files in a directory .etomotemplate under your home directory, and save them there from its template editor, but a different location can be chosen in the Options/Settings dialog.
All template files must have the extension .adoc. Two system templates are distributed with IMOD. One is plasticSection.adoc, which just turns on the use of a Sobel filter to achieve better centering when tracking gold markers in the bead tracker. The other is cryoSample.adoc, which also turns on the Sobel filter centering but with an appropriate smoothing parameter, and has some other settings that we have found appropriate for cryo-ET. One way to see the settings in a template is to start a data set, select the template, and create the com scripts. At this point you can select Templates/Save System Template in the File menu (to see a system template). The items in the existing template are all checked for inclusion in a new template. You may have to check Show only included to see just the items in the template.
The template files form a hierarchy and are considered in the order Scope - System - User, with later ones overriding earlier ones. In other words, if the same parameter is found in more than one template, the setting in the last template is the one that is used.
The template editor can be used to save template files containing any of the parameters supported in the editor, but currently templates can be saved only from an actual data set. The parameters are organized into sections corresponding to different processing steps; there is a set of checkboxes that control whether eahc section is open or closed. Whether a parameter is saved is determined by the state of the Include checkbox to the left of each entry. You can set these however you want, but the state of the Includes when you first open it depends on which kind of template you select to save. If you select to save a Scope or System template, items will be checked for inclusion only if the data set was started with a Scope or System template, respectively, and the items checked will be the ones in that template. Other items shown grayed out will be ones that eTomo thinks have changed since the data set was started. The same applies if you go to save a User template and started the data set with a User template. However, if no User template was used, the program will shows items it thinks have changed and select some items for inclusion based on how they are coded in a master file of parameters, excluding ones in Scope or System templates. Unfortunately, there are still numerous problems with the program's ability to tell whether an item has changed. In particular, it will think almost everything has changed for a data set started with a version of IMOD before 4.6.20, so it is best to open this editor only with a recently started data set.
The template editor is still under construction; we plan changes that will make it clear whether a parameter is contained in a lower-level template than the one being viewed, and we may change the logic by which items are selected for inclusion. For now, it is important to look through all the sections of the editor and be sure you intend to save what is being saved.
Several steps in the processing can now be split into chunks
and run in parallel on multiple machines or processors, including CTF
correction, tomogram generation, and the final stage of combining volumes in a
dual-axis dataset. This capability is enabled if the machine on which you
are running eTomo has a configuration file called cpu.adoc located in the IMOD
calibration directory, which is /usr/local/ImodCalib by default. Run "man
cpuadoc" for a full description of how to set up such a file. A two-line
file is sufficient to use a single multiprocessor system. There are two
other alternatives when using a single system. First, you can open the
dialog, turn on
Enable Parallel processing, and then
fill in the number of CPUs. Second, you can enable parallel processing by
setting an environment variable IMOD_PROCESSORS to the number of processors in
the computer. For example, if you have 4 processors and use tcsh for your
setenv IMOD_PROCESSORS 4
in your .cshrc file, or if you use the bash shell, place
in your .bashrc file. If you are using multiple machines, you will also need to be able to log in via ssh without entering a password (see below) and run IMOD on all of the other computers that you want to use.
When parallel processing is available, you will be able to
select whether to use it with checkboxes on panels where it can be used.
With it selected, a parallel processing table and controls will appear in the
Process Monitor area of the window. The table will show computers, their
properties, and their current loads. When you are ready to run the given
operation, select the computers and number of CPUs that you want to use, then
press the appropriate action button (e.g.,
Generate Tomogram). The table will
show how many chunks are completed on each machine. The parallel
processing area includes several controls:
The Kill button, as usual, will kill all running processes
The Pause button will stop the processing more gracefully by allowing all currently running chunks to finish.
The Resume button can be used after either a Kill or a Pause to resume processing without having to redo any chunks that were already completed. If you press Pause in order to select a different set of CPUs, you can change your CPU selection then press Resume without waiting for running chunks to finish; processing will resume with the new set of CPUs after those chunks finish.
The Save as Defaults button will save your current selection of computers as the default for future sessions.
The Nice control lets you select the "niceness" of your jobs; the higher the value, the lower the priority. The highest value is 19, which will compete with screen savers, while a value of 0 will compete equally with normal interactive usage of the machine.
The +/- button in the upper left can be used to minimize the table.
button in the upper right can be used to make the table smaller, so that it
shows only the selected computers and information needed while running.
To be able to log in via ssh without a password, you need to generate ssh keys
that are present on each machine. First run
ssh-keygen -t dsa
and type Enter for all of the queries. Then do
cp $HOME/.ssh/id_dsa.pub $HOME/.ssh/authorized_keys2
If necessary, distribute this authorized_keys2 file to the .ssh directories on other machines that do not have the same home directory. The three different kinds of platforms (Linux/Unix, Mac OSX, Windows) may require separately generated keys. If this seems to be the case, run ssh-keygen once on each type of machine then combine the .ssh/id_dsa.pub files from each into one authorized_keys2 file. Distribute this combined authorized_keys2 file to the .ssh directories.
In addition to being done on multiple processors, tomogram generation can now be done on the graphics processing unit (GPU) of an NVIDIA graphics card. See the IMOD User's Guide for instructions on setting up a machine to use the GPU. Reconstructions can also be done in parallel using GPUs on several computers, as well as with more than one GPU on a single computer.
eTomo provides a good framework for developing new interfaces
for managing parameter files and running processes, so we continue to add
interfaces for operations other than generating a tomogram. Interfaces
for different kinds of projects are available on the Front Page and through the
File menu via the various selections starting with
New. They include interfaces for:
Joining together tomograms from serial sections, opened by selecting File/New Join. See the Guide to Joining Tomograms for instructions. The processing state is saved to a file with extension .ejf, which you can reopen to resume your work.
Aligning serial images from conventional serial sections or block face imaging, or simply blending montaged images, opened by selecting File/New/Align Serial Sections - Blend Montages. See Aligning Serial Sections and Blending Montages for instructions. The processing state is saved to a file with extension .ess, which you can reopen to resume your work.
Running the PEET volume alignment and averaging software, opened by selecting File/New/Subvolume Averaging (PEET). See Guide to Particle Estimate for Electron Tomography (PEET) for instructions. The processing state is saved to a file with extension .epe, which you can reopen to resume working on a dataset.
Running parallel processing on an appropriate set of command files, such as the ones created by the programs chunksetup, sirtsetup, and splitcorrection. This interface is opened by selecting File/New/Generic Parallel Process . The program state is saved in a file with extension .epp, which you can reopen to process files with the same name as previously.
Filtering by nonlinear anisotropic diffusion. This procedure often requires finding the right parameter settings with a small test volume before running on the full volume, so the interface provides tools for doing this conveniently. All of the temporary and working files are kept in a subdirectory that you can remove at the end. The interface is opened by selecting File/New/ Nonlinear Anisotropic Diffusion. The program state is saved in a file with extension .epp. See the nad_eed_3d man page for more details.
In addition to these kinds of projects, which have a saved state and can be resumed, interfaces for simpler operations can be started from the Tools menu as well as from the Front Page of eTomo. Currently there are two tools, for flattening a tomogram without opening up its original dataset and for testing the GPU.
We very much appreciate feedback from our users; this includes bug reports, suggestions for enhancements and general comments on what ideas or use models do or do not work. It is often not initially obvious to either you or us whether a problem is occurring because of bug in eTomo, a bug in the rest of IMOD, or other difficulties with a particular dataset. To help diagnose problems, we have developed a script, tomosnapshot, that will collect system information and package up many of the small files that reflect the state of the processing. This script is now able to collect files for all of the kinds of projects that have a saved state (i.e, joining, PEET, NAD as well as tomogram generation). The best way to run this is to cd to the directory where the data are, start eTomo with the project file for the given data set, repeat the series of actions that give the error (if this is reasonably fast), then select the menu item File/Run Tomosnapshot. You can then send the resulting tar file (setname-snapshot), along with a description of what you did and the undesired result, to David Mastronarde or Sue Held (see web site for addresses). You can also report bugs and make suggestions or comments by posting a message on the IMOD mailing list at the University of Colorado. This allows others to observe the discussion and see if they are encountering similar problems; but please don't post your snapshot file to the list. To subscribe to the IMOD mailing list send an email to listproc@lists.Colorado.edu with the contents of the message being
subscribe IMOD your-full-name
replacing your-full-name with your name.