Using Etomo
Rick Gaudette and David Mastronarde
University of Colorado, Boulder
Usage
Using Templates
Using Parallel Processing
Other Interfaces in Etomo
Reporting bugs
Etomo provides a graphical user interface (GUI) to the IMOD Tomography package. The general user interface design idea on which Etomo is built is to present a separate panel for each distinct step in computing a tomogram with the IMOD package. This allows the user to focus on the current step in the processing sequence, presenting only the parameters and controls required for the current step.
Etomo can be started from the command line with the command
etomo
It may take a little while to come up; Java applications have a slow startup time. You can also open up a previous data set by specifying the relative path to the .edf file on the command line, such as
etomo path/to/dataset.edf
When Etomo is first started, it shows its Front Page, which contains a set of buttons for starting different kinds of interfaces (see below). At this point you can either open a data set that was previously being worked on by selecting it from the recent file list in the File menu, or you can use one of the buttons to open an interface for working on a new data set.
Selecting New Tomogram will bring up the tomogram Setup Panel. To start working on a new data set you need to fill out the fields in the Setup Panel.
The Dataset Name is the name of the file containing the tilt series (or the name of one of these files in the case of a dual axis tomogram). You can enter the Dataset Name by opening the file selection dialog window using the folder labeled button associated with the Dataset Name field.
The Backup Directory is optionally available to save small working files every time you run a procedure. You can leave this field empty if you do not wish to use a backup directory.
Specify the pixel size, fiducial size and image rotation in
the appropriate fields. If the header of your image file includes pixel
size and/or image rotation information, press the
Scan Header
button to extract these values.
Specify the source for the tilt angles for either one or both axes, as appropriate. You can also optionally specify individual projections to exclude from the processing. The syntax for this exclude list is a comma separated list of ranges (i.e. 1,4-5,60-70)
See the Tomography guide. for complete instructions on filling in the Setup Panel.
Press the Create com scripts button to move on to generating the tomogram.
To return to a previous tomogram, select the .edf (Etomo Data File) file from the most recently used list on the bottom of the File menu, or the Open... item on the File menu if the .edf is not listed in the most recently used list. The file dialog will let you navigate around to select the .edf file associated with the tomogram you wish to work on.
The main window consists of several areas: on the left is a column of buttons (Process Control Buttons) that allow you to select the particular step of the tomogram computation to work on, the top center is the Process Monitor that will inform you of the status of the current process or the last process completed and a button to kill the current process, the bottom status bar presents some basic information about the data set you are working on. If you are working on a dual axis tomogram this layout will be repeated for both the A and B axes.
The Process Control Buttons are arranged in the suggested order of processing from top to bottom and are color coded to signify the state of the process. Red indicates that the process has not been started, magenta indicates that the process is in progress, and green indicates that the process has been completed. The state of the process is also indicated in text on the button. When one of the buttons is selected the right side of the window will fill in with information and fields associated with that process; these forms are called Process Panels. They allow you to modify the necessary parameters and execute the programs required by that processing step. The parameters and buttons on each Process Panel are typically laid out from top to bottom in the order they should be executed. When you execute a process (by pressing a button on one of the Process Panels) the Process Monitor will indicate what the process is doing and when it is complete. In many cases, the button that you pressed will turn dark to indicate that the operation has been completed.
The text output of a particular process is written to a log file named for the particular process; this log file can be viewed by right clicking the mouse pointer over the window region associated with the process. This will open a menu that is split into three or four sections: the entries in the first section will open up the log files associated with the current process; some menus will have a second section with choices for plotting various kinds of information; the next section will allow you to open up man pages associated with the current process; the final menu section opens the general help documents. There is more information regarding online help resources in the section Accessing Help.
When you are done with all of the processes for a given step press the Done button; this will inform Etomo that the step is complete and will also move the GUI forward to the next step in the tomogram processing. Pressing the Postpone button will save all of the current parameters for that step and pressing Cancel will abort all of the changes made since the last process was run for that step. If you move between process panels by pressing the buttons in the left column, this will save the current parameters just as a Postpone does. The final button of interest at the bottom of most processing panels is the Advanced button. This will present a much more complete list of parameters for each step, which are mostly needed for unique or extreme cases.
Since the number of parameters that a particular step requires can vary significantly, the amount of screen space the GUI needs to present that information also changes. The current version of the program will adjust the window size automatically to fit tightly around the Process Panel whenever you change screens or change the amount of information displayed (such as by changing to advanced mode). You make this fitting occur automatically by selecting the Options/Fit Window menu item or using Ctrl-F hot key. Whenever the window is too small to display the information, scroll bars will appear so that you can pan across the Process Panel. To reduce the need for scrolling, several of the large panels are divided into sections that can be opened or closed independently with a button in the upper-left corner. This button is initially "-" when the section is open, and becomes "+" when the section is closed. In addition, a button may appear in the upper right corner to allow a section to be switched to advanced or basic mode independently of the others; this button will be labeled A in basic mode and B in advanced.
If you are working on a dual axis tomogram, each tomogram is processed in a separate window. The upper left corner of the Axis A window (or the Axis B window, if it is the only one open) will have buttons to allow you to switch between displaying one or the other axis or both axes. There are also hot keys, Ctrl-A, Ctrl-B, and Ctrl-2, to display axis A, axis B, or both.
Every process panel has one or more buttons for starting 3dmod to display an image file. To make it easier to deal with very large data sets and problems that require more control over the way that 3dmod is run, there are two ways to make 3dmod load in images binned by 2 or to make it open with its startup window, so that options can be selected. One way is to right-click on one of these buttons, then select Open with startup window or Open binned by 2. The other way is to select the corresponding items in the Options menu, which will make 3dmod open that way every time.
For many processing steps, it is also possible to run the operation and have the resulting images loaded into 3dmod automatically. Right click on the button that starts the action (e.g., Generate Tomogram) and menu will pop up with the choices And open 3dmod, And open 3dmod with startup window, And open 3dmod binned by 2. Select the desired option to start the operation.
It is possible to load more than one data set into Etomo. Under the File menu there are entries to open an existing data set, open a new Setup window for tomogram processing, or open a new window to setup joining serial sections. If you start working on more than one data set by any of these routes, Etomo will place each data set in a separate tab so that you can go between them easily.
The Options/Settings dialog presents several configuration options, such as one to change the font size being used, and one to switch to a compact display in which the buttons on the left are very small. It may be necessary to close and reopen the program to see options like these take full effect. In addition, this dialog allows you to set a number of defaults for the program, so that you can tailor the initial parameters to the kinds of data sets that you usually work with.
When you start processing a tomogram, a separate window will be opened called a project log. This window will receive selected output from various operations, and error and warning messages that occur. You can also type your own notes into the window. There is an entry under the View menu, and a hot key Ctrl-L, for hiding or making it visible again. Even when it is hidden, output will be saved to it. The contents are automatically saved to a file named with the data set name and "_project.log".
One of the significant features of the Etomo GUI is the immediate access to context sensitive online help. By context sensitive we mean that the help being presented is specific to the current processing problem at hand. Access to the online help is available through two methods. The first, commonly know as Tooltips, is accessed by leaving the mouse stationary over the parameter or other GUI object of interest. This will pop up a short text description of the parameter or function of the object. The object being described can be changed by moving the mouse around or the description can be removed by moving the mouse over a blank window region. The description will also be dismissed (disappear) after a short time if the mouse is not moved. Both the amount of time to delay displaying the Tooltip and the amount of time to keep the Tooltip visible can be set in the Options/Settings dialog.
The second method to access context sensitive help is by right clicking the mouse over the processing region of interest. This will open up a pop-up menu containing three sections. The top section presents menu items that will open up a new window containing a log file appropriate for the current processing step. The second section presents menu items that will open up the appropriate man pages for the current processing step. The final section provides access to the Tomography Guide, Serial Section Joining Guide, IMOD Guide and the 3dmod Introduction. If the Tomography Guide is selected the Guide will be opened up to the section that is appropriate for the current processing step. All documentation is opened in the Qt Assistant, so that you can follow hyperlinks and return from them, as well as have access to the full IMOD documentation using the sidebar on the left.
The current state of the processing and the value of any parameters you entered or changed (that are not saved in the underlying com scripts) are saved automatically to a file with the data set name and the extension .edf. If you are working on joining serial sections, the state is saved instead to a file with extension .ejf. The state file is updated whenever you run a process, change process panels, or exit the program. If you wish to save this file with a different name, use the File/Save as menu item. Be aware that if you reopen an earlier version of the .edf file you may get unpredictable results.
Default settings for many tilt series processing parameters can be customized in files called template files. To provide flexibility, Etomo is set up to work with three different levels of template files, although there is no need to use all three. The three levels are:
Scope template: These files are intended to be used for parameters specific to a microscope, a combination of microscope and camera, or perhaps a combination of microscope and some imaging condition, such as voltage and spherical aberration for CTF correction, or image distortion correction files. These files should be placed in a subdirectory ScopeTemplate of the directory pointed to by the environment variable IMOD_CALIB_DIR, which is /usr/local/ImodCalib by default.
System template: These files are meant to hold settings that are available to all users on the system. Etomo will look for these files first in a subdirectory SystemTemplate under the installed IMOD directory, then in a subdirectory SystemTemplate under the directory pointed to by IMOD_CALIB_DIR. When Etomo sees a file with the same name in latter location as in the IMOD directory, it ignores the one from the IMOD distribution.Thus, if you want to customize a template distributed with IMOD, you can modify it and save it with the same name in the $IMOD_CALIB_DIR/SystemTemplate directory.
User template: These files hold settings for an individual user. Etomo will look for these files in a directory .etomotemplate under your home directory, and save them there from its template editor, but a different location can be chosen in the Options/Settings dialog.
All template files must have the extension .adoc. Two system templates are distributed with IMOD. One is plasticSection.adoc, which turns on the use of a Sobel filter to achieve better centering when tracking gold markers in the bead tracker and sets a default size of patch tracking. The other is cryoSample.adoc, which also turns on the Sobel filter centering but with an appropriate smoothing parameter, and has some other settings that we have found appropriate for cryo-ET. One way to see the settings in a template is to start a data set, select the template, and create the com scripts. At this point you can select Templates/Save System Template in the File menu (to see a system template). The items in the existing template are all checked for inclusion in a new template. You have to check Show only included to see just the items in the template and not all the items in each section that opened up. (Note: If you have included a higher-level template that overrides some of the values, you will see the values imposed by that template, not the values in the template that you are interested in.)
The template files form a hierarchy and are considered in the order Scope - System - User, with later ones overriding earlier ones. In other words, if the same parameter is found in more than one template, the setting in the last template is the one that is used.
The template editor can be used to save template files containing any of the parameters supported in the editor, but currently templates can be saved only from an actual data set. The parameters are organized into sections corresponding to different processing steps; there is a set of checkboxes that control whether eahc section is open or closed. Whether a parameter is saved is determined by the state of the Include checkbox to the left of each entry. You can set these however you want, but the state of the Includes when you first open it depends on which kind of template you select to save. If you select to save a Scope or System template, items will be checked for inclusion only if the data set was started with a Scope or System template, respectively, and the items checked will be the ones in that template. Other items shown grayed out will be ones that Etomo thinks have changed since the data set was started. The same applies if you go to save a User template and started the data set with a User template. However, if no User template was used, the program will show items it thinks have changed and select some items for inclusion based on how they are coded in a master file of parameters, excluding ones in Scope or System templates. Unfortunately, there are still numerous problems with the program's ability to tell whether an item has changed.
To save a template with an arbitrary set of parameters that have nothing to do with a particular data set, just open the editor with File/Templates/Save Scope Template or File/Templates/Save System Template. In the Show Directives Box, check Show unchanged and Show Hidden, and make sure Show only included is unchecked. Now you will see all possible parameters when you open any section with its open/close checkbox. Turn on the Include checkbox for any parameter that you want to save, and then you will be allowed to set its value. When done, save to a file. Copy to one of the locations described above; in particular, if you want to use this file as a user template, copy it to your own template directory (.etomotemplate under your home directory by default).
You can also create a template manually, with a text editor. In a terminal, run
imodhelp -d directives
or run imodhelp and select
Directives for batch processing and Etomo
templates (or just click on the preceding link). For each line in your
file, copy and paste a directive from the Directive column of the
directives table, and add "=" and the value, in the format indicated by the
entry in the Type
column. Specifically, a "Bool" should be 0 or 1, and an "Int" should have
no decimal point.
Once you have made a template, you can make it be the default that appears on the Tomogram Setup page by opening the Options/Settings dialog and making a selection in the Default Templates section.
The template editor is not well-designed, but further work was deferred until the batch tomogram interface was finished. We plan changes that will make it clear whether a parameter is contained in a lower-level template than the one being viewed, and we may change the logic by which items are selected for inclusion. For now, it is important to look through all the sections of the editor and be sure you intend to save what is being saved.
Several steps in the processing can be split into chunks and run in parallel on multiple machines or processors, including CTF correction, tomogram generation, and the final stage of combining volumes in a dual-axis data set. This capability is enabled if the machine on which you are running Etomo has a configuration file called cpu.adoc located in the IMOD calibration directory, which is /usr/local/ImodCalib by default. Run "man cpuadoc" for a full description of how to set up such a file. A two-line file is sufficient to use a single multiprocessor system. There is a simple alternative when using a single system. Open the Options/Settings dialog, turn on Enable Parallel processing, and then fill in the # CPUs box with the true number of cores available. If you are using multiple machines, you will also need to be able to log in via ssh without entering a password and run IMOD on all of the other computers that you want to use. See the IMOD User's Guide for complete details on setting up your systems for parallel processing.
When parallel processing is available, you will be able to select whether to use it with checkboxes on panels where it can be used. With it selected, a parallel processing table and controls will appear in the Process Monitor area of the window. The table will show computers, their properties, and their current loads. When you are ready to run the given operation, select the computers and number of CPUs that you want to use, then press the appropriate action button (e.g., Generate Tomogram). The table will show how many chunks are completed on each machine. The parallel processing area includes several controls:
The Kill button, as usual, will kill all running processes
The Pause button will stop the processing more gracefully by allowing all currently running chunks to finish.
The Resume button can be used after either a Kill or a Pause to resume processing without having to redo any chunks that were already completed. If you press Pause in order to select a different set of CPUs, you can change your CPU selection then press Resume without waiting for running chunks to finish; processing will resume with the new set of CPUs after those chunks finish.
The Save as Defaults button will save your current selection of computers as the default for future sessions.
The Nice control lets you select the "niceness" of your jobs; the higher the value, the lower the priority. The highest value is 19, which will compete with screen savers, while a value of 0 will compete equally with normal interactive usage of the machine.
The +/- button in the upper left can be used to minimize the table.
The </> button in the upper right can be used to make the table smaller, so that it shows only the selected computers and information needed while running.
In addition to being done on multiple processors, tomogram generation can be done on the graphics processing unit (GPU) of an NVIDIA graphics card. See the IMOD User's Guide for instructions on setting up a machine to use the GPU. Reconstructions can also be done in parallel using GPUs on several computers, as well as with more than one GPU on a single computer. When using a single system with a single GPU, it may be as simple as turning on Enable graphics processing in the Options/Settings dialog.
Etomo provides a good framework for developing new interfaces for managing parameter files and running processes, so we continue to add interfaces for operations other than generating a tomogram. Interfaces for different kinds of projects are available on the Front Page and through the File menu via the various selections starting with New. They include interfaces for:
Joining together tomograms from serial sections, opened by selecting File/New Join. See the Guide to Joining Tomograms for instructions. The processing state is saved to a file with extension .ejf, which you can reopen to resume your work.
Aligning serial images from conventional serial sections or block face imaging, or simply blending montaged images, opened by selecting File/New/Align Serial Sections - Blend Montages. See Aligning Serial Sections and Blending Montages for instructions. The processing state is saved to a file with extension .ess, which you can reopen to resume your work.
Doing batch tomogaphic reconstruction for a set of similar data sets, opened by selecting File/New/Batch Tomograms. See Batch Tomogram Reconstruction with Batchruntomo in IMOD for instructions. Parameters are saved to files with extension .adoc and the processing state is saved to a file with extension .ebt, which you can reopen to resume your work.
Running the PEET volume alignment and averaging software, opened by selecting File/New/Subvolume Averaging (PEET). See Guide to Particle Estimate for Electron Tomography (PEET) for instructions. The processing state is saved to a file with extension .epe, which you can reopen to resume working on a data set.
Running parallel processing on an appropriate set of command files, such as the ones created by the programs chunksetup, sirtsetup, and splitcorrection. This interface is opened by selecting File/New/Generic Parallel Process . The program state is saved in a file with extension .epp, which you can reopen to process files with the same name as previously.
Filtering by nonlinear anisotropic diffusion. This procedure often requires finding the right parameter settings with a small test volume before running on the full volume, so the interface provides tools for doing this conveniently. All of the temporary and working files are kept in a subdirectory that you can remove at the end. The interface is opened by selecting File/New/ Nonlinear Anisotropic Diffusion. The program state is saved in a file with extension .epp. See the nad_eed_3d man page for more details.
In addition to these kinds of projects, which have a saved state and can be resumed, interfaces for simpler operations can be started from the Tools menu as well as from the Front Page of Etomo. Currently there are two tools, for flattening a tomogram without opening up its original data set and for testing the GPU.
We very much appreciate feedback from our users; this includes bug reports, suggestions for enhancements and general comments on what ideas or use models do or do not work. It is often not initially obvious to either you or us whether a problem is occurring because of bug in Etomo, a bug in the rest of IMOD, or other difficulties with a particular data set. To help diagnose problems, we have developed a script, tomosnapshot, that will collect system information and package up many of the small files that reflect the state of the processing. This script is now able to collect files for all of the kinds of projects that have a saved state (i.e, joining, PEET, NAD as well as tomogram generation). The best way to run this is to cd to the directory where the data are, start Etomo with the project file for the given data set, repeat the series of actions that give the error (if this is reasonably fast), then select the menu item File/Run Tomosnapshot. You can then send the resulting tar file (setname-snapshot), along with a description of what you did and the undesired result, to David Mastronarde (mast at colorado.dot edu). You can also report bugs and make suggestions or comments by posting a message on the IMOD mailing list, which is now a Google group. This allows others to observe the discussion and see if they are encountering similar problems; but please don't post your snapshot file to the list. To subscribe to the IMOD group, see the instructions on our web site.