This document provides a step-by-step guide for using the interface in eTomo for aligning images from serial sections. It also presents some background information on the concepts and tools involved in this process, and describes some of the other operations that can be run outside of the eTomo interface. This interface is also useful if you have some montaged images that need to be blended into single frames without any alignment; in that case you just need to go through Getting Started and Blending Montaged Images.
Most images can be aligned adequately by applying general linear transformations (also known as affine transformations). These transformations can include a shift, a rotation, a magnification (size change), and a stretch, all of which occur uniformly across an image. See Background on Linear Transformations for more details about the properties of these transforms and the terminology that we use. When the changes between two images are nonlinear (e.g., if distortions occurs differently in different areas), then it is necessary to use a nonlinear or warping transformation to align them. Warping transformations in IMOD are specified by a set of shifts between images at a collection of points. The interface supports two different methods of aligning images: automatically by image correlations with Xfalign, or manually by displaying pairs of images in Midas. A third method of aligning with linear transformations, fiducial points marked between images, is not supported by the interface and is described in Generating or Editing an Alignment with Fiducial Points. Other operations not supported by the interface, refining the alignment after you have created a model on the aligned stack and transforming the model to that new alignment, are described in Realigning Images and Models.
If you are working with serial sections, you should also consider the issues discussed in Stacking Order and Handedness of Structures.
The interface has several features to alleviate the difficulties of dealing with very large images. For any button that opens an image file in 3dmod, you can right click and select either Open binned by 2 or Open with startup window. The latter allows you to select a higher binning in X and Y, and even binning in Z if that would be appropriate. You can make one of these two choices be the default when opening 3dmod by selecting the same option in the eTomo Options menu. There is also an option on the Align tab to load data into Midas with binning, which will save memory, make the program run faster, and may make the images look better than full-sized images zoomed down.
Your unaligned image stack may have any extension: ".st", ".mrc", or anything else. The root name of this stack (before the extension) is used as the dataset name (referred to as "setname") from which other files are named.
Only one serial section data set can reside in a directory, so generally you should start by creating a directory and moving the raw stack into it. If you are just blending montages and want to blend one after another in the same directory, you need to remove the ".ess" file that eTomo uses to keep track of serial section data sets.
Start eTomo. The Front Page will appear; press the button Align Serial Sections / Blend Montages. This will open both the main eTomo window and a starting dialog titled "Starting Serial Sections" where you select the unaligned image stack and indicate whether it consists of Single frames or Montages. The Log window also appears; messages are copied there and you can also enter notes there.
This dialog also has a place
to select an image distortion field file, which can be helpful when
blending montages taken with side-entry (wide-angle) cameras. There are
tools in IMOD for computing such image distortion fields from a set of
overlapping images at a particular magnification. The file
chooser will open in the directory where such files should be kept
/usr/local/ImodCalib/Distortion by default). If you select a
distortion file, be sure to set the binning on the camera at which the
images were acquired.
If you have a montaged image stack, the interface opens with the Initial Blend tab after you finish with the starting dialog. It will first extract the piece list file from the image file if one is not already present (named "setname.pl"). The progress area at the top of the window will indicate that this has happened. The alignment tools work only with single-frame images, so it is necessary to run Blendmont to create the needed images in a file named "setname_blend.mrc". This program uses correlations in overlap zones to determine the shift in X and Y between each pair of horizontally or vertically adjacent montage pieces. It takes all of these shifts into account to solve for a shift to be applied to each individual piece when placing it in the blended image.
Select Treat as very sloppy montage if the montage was acquired with stage movement. With this option, the program will correlate larger portions of the overlap zones and be able to find the shifts when the overlap varies substantially from the nominal amount.
The option for Robust fitting is useful if a minor fraction of the correlations in the overlap zones fail because of grid bars, insufficient image features, or inadequate overlap. This is rare with serial section data but common when blending montages used for mapping in SerialEM. With robust fitting, the program can give less weight to correlation shifts that have larger errors when solving for the overall shifts of the individual pieces. In the best case, the erroneous correlation shifts may be completely discounted and the correct solution obtained from the rest of the data. The criterion entered in the text box controls the sensitivity for considering a shift to be an outlier; a value smaller than 1 will down-weight more shifts.
Press Make Blended Stack to run Blendmont. Press Open Blended Stack to see the resulting blended file. If the shifts between pieces were not all determined accurately, you may be able to see this in the blended file. However, it can be difficult to see this problem in the unaligned images, so you may not see a problem until you make an aligned stack. The log file from the blending run does have useful information about the errors in aligning the pieces, unless you have a montage that consists of just one row or one column of pieces. Except in that case, when the blending finishes, eTomo will pop up a graph showing the remaining error for each section after shifting pieces into optimal alignment. If some of these are unusually high, you should open the "preblend.log" to determine what section numbers they are. (Right-click in the panel and select this file from the popup menu.) For each section, you will see a line like
8 edges, mean&max error before: 25.22 88.08, after by edges: 1.24 3.06The two numbers on the right are the ones plotted in the graph. If these numbers are large (more than a few pixels), it is likely that one or more of the correlations used to align adjacent pieces in that section has failed. It is hard to give a fixed rule for when the numbers are too large, because when there are no errors in the correlations, the errors shown will depend on the amount and kind of distortion in your images. Until you have some experience about what size of errors to expect, you should open the data set in Midas and verify the alignment of pieces in a few of the sections with the biggest errors. Beware: sections are numbered from zero in this log file but from 1 in 3dmod and Midas.
If there is a problem with the alignment of pieces, you can use Midas to adjust the shifts between pieces ("Fix edges") manually until the error in solving for the overall shifts of the pieces is minimal. Midas will display an overlap zone between two pieces. Here and elsewhere, the overlap zones are referred to as "edges"; edges between horizontally or vertically adjacent pieces are referred to as edges in X or Y, respectively. You can use the spin boxes or the hot keys to go between one edge and the next or one section and the next (the hot keys are "a" and "b" to move between sections and "A" and "B" to move between edges). Basically, you need to find each bad displacement and adjust it by shifting with the left mouse button. The buttons in the lower left showing the largest errors will assist in finding the bad shifts on a section. If the shift is bad, you can try the Apply Leave-out Error button near the bottom, as this will often place the piece correctly based on its overlap with other pieces. There are several other features to assist in this task when there are many errors (which can be the case when fixing shifts for a low-magnification map). See Controls when Fixing Montages in the Midas online help and the section on Fixing Montage Overlaps in the Midas man page.
After fixing edges, you need to save the shifts, exit Midas, and remake the blended stack.
The automatic alignment should work better if images are processed to emphasize corresponding features. The processing and area selection options available through the interface consist of the Fraction to ignore on edges, the image Binning in search (which is 2 by default) and low and high pass filtering by Gaussian functions. The binning selection here applies only to the search in Xfsimplex and should be set to bring the images down to about 1K in size or less. This binning is actually done with antialiased image reduction to avoid having high-frequency information aliased into the reduced image in ways that might degrade the signal for aligning the images. Thus, there is no need to use the Gaussian filtering just to avoid the effects of aliasing when using a high binning. You can avoid the filtering by blanking out each of the three parameter fields.
The three filter parameters, the sigma for a low-frequency filter and the cutoff radius and sigma for a high-frequency filter, have the same meaning as the parameters used in many other IMOD programs. This filter is applied to the reduced image. You can experiment with these filter settings by loading the stack into 3dmod with a binning corresponding to the Binning in search (right click on the Open Stack button and select Open with startup window, and enter the appropriate binning in X/Y). Then open the Fourier Filtering panel of the 3dmod Image Processing dialog by selecting Image-Process from the Edit menu and clicking on Fourier filter. Xfalign has several other options for image filtering that are not available through the interface, and you would have to run the program outside of eTomo to access those options. See the man pages for Xfalign and Xfsimplex for more guidance on processing options; let us know if you think other options should be added to the interface.
You can skip over sections with features that will impair alignment by listing them in Sections to skip, with sections numbered from 1. The programs will align the section after a skipped one to a section before a skipped one. Note that if you then align the skipped sections manually, you will need to align both the skipped section to the previous one and the next one to the skipped one.
To run automatic alignment when there is no preliminary alignment of images, press Initial Auto Autoalignment. You can then examine how well the alignment worked in Midas, as described in the next section.
Here are a few key points about using Midas to align images with a linear transformation:
The program keeps track of the last alignment done and uses that as the basis for the next operation. However, if you run an initial autoalignment and the results are bad enough that you want to start from scratch and align just with Midas, you can press Revert to No Transforms then Midas. If you do some alignment in Midas then do a refining autoalignment that produces results not worth working from, you can press Revert Auto Alignment to Midas to restore the transforms that were saved from Midas.
A warping (nonlinear) transformation is specified by a set of shifts between the images at particular points, which are commonly referred to as control points. Warping becomes important for very large images where nonlinearities make it difficult to line them up everywhere with a single linear transformation. Although a warping can be set up with as few as 4 points, in practice you will generally need more than this minimum in order to align two images properly over their whole area. Thus, warping requires more image features that obviously correspond between images than does a linear transformation. Warping can be introduced either manually or automatically. Because of the requirement for well-distributed image features suitable for correlation, the automatic warping will work on an even smaller subset of data sets than automatic searches for linear transformations. However, it is likely to work with data sets based on serial block face imaging.
To compute a warping automatically, select Find warping transformations, which will enable all of the related fields.
After computing warping transformations, you can open the result in
Midas for examination and adjustment, just
as you can for linear transformations. If there are some aberrant shifts that
need fixing, you can navigate to them with the Biggest Warp button at
the bottom of the controls, and select points with the next higher or lower
shift with the adjacent up and down arrow buttons. As mentioned above,
clicking with the left mouse button selects the nearest point and dragging
with it adjusts the shift for the current point. There is a hot key, "D", for
deleting the current point.
To add warping in Midas, select the option Add/edit warp points after aligning as much you wish with a linear transformation. Then add a series of points at locations where there are features that can be aligned, using the middle mouse button. The new point will be the active one, shown in yellow, and the yellow box around it shows the area that would be used if you press Cross-correlate. At each location, you shift the image locally into alignment with the left mouse button. Start by adding points where the images are already aligned, then add points where shifts are needed. See the section Warping Images in the Midas man page for more details.
You can find warping automatically in conjunction with either an initial alignment search or a search to refine an existing alignment. However, you cannot use Refine with Auto Alignment once you have a warping alignment. If you run an initial autoalignment with warping and it runs into problems with getting the initial shifts or linear transformations, then the best approach is to rerun the initial search without the warping. Use Midas to fix any problems from that search. Then use Refine with Auto Alignment to add the warping, omitting the linear search by turning off Search For.
Once you have satisfactory transforms for aligning the images, go to the Make Stack tab to make a stack of aligned images. Up until now, you have been working with transforms that align an image to the previous one, and these need to be converted to transforms that will bring every section into a common, global alignment (see Background on F and G Transforms for more explanation of this point). Pressing Make Aligned Stack runs two programs, first Xftoxg to convert the transforms, then either Newstack or Blendmont to apply the transforms to the images. The options at the top, in the Stack alignment transforms box, control how Xftoxg works. Most of the time, the default choice of Local fitting (retain trends) is appropriate. What "retain trends" means is that if the structures dominating the alignment persistently shift or rotate in one direction, they will still do so in the aligned stack, only more smoothly. See the section on >Converting from an F List to a G List for a detailed explanation of the effects of these different choices and when you might need them. Global alignments are generally not suitable except for relatively small numbers of sections, especially if there are warping transformations. If you do use this choice, there is an option to pick one section as a reference; it will not be modified, and all other images will be aligned to it.
One consequence of the linear fitting is that the originally captured image area will be nearly all present in the output images. If for some reason you do want to remove the trends with one of the other choices, and the trends include shifts, then you would need a larger size to retain all of the aligned image. Thus there are two sets of fields to control the output, ones for the size in X and Y, and ones for a shift in X and Y, where positive numbers shift images up and to the right. You can enter a number in one field and leave the other blank. These entries are all in unbinned coordinates. Thus, if you make a stack with binning in order to work out the size and shift needed, you need to multiply the numbers that you see in 3dmod by the binning, but once you have the right numbers, they can be used to make an unbinned stack.
If you have a very large aligned image file, there are tools in IMOD that make it easier to deal with such files in 3dmod. Specifically, you can make an "image pyramid", a set of files at different resolutions, using the command-line program Makepyramid, which allows relatively fast access to the image data in 3dmod without needing huge amounts of memory.
clip flipz input_file output_file
For serial block-face imaging with scanning EM, the handedness depends on two factors:
Two images can be aligned by applying a coordinate transformation to one image to make the transformed image match the other image as well as possible. The transformations used for alignment are general linear transformations. This means that the same amount of translation, rotation or distortion happens everywhere in an image. Our programs use three different ways of representing transformations. The parameters in these different representations are referred to as "formal", "semi-natural", or "natural".
The formal parameters are the matrix coefficients that are actually used to transform image coordinates. There are six formal parameters in a linear transformation: the new X-coordinate of a point is a linear combination of its old X and Y coordinates, plus a constant offset, so three parameters determine the new X-coordinate; similarly, the other 3 parameters determine the new Y-coordinate. The equations are
X' = A11 * (X - Xcen) + A12 * (Y - Ycen) + DX + Xcen Y' = A21 * (X - Xcen) + A22 * (Y - Ycen) + DY + Ycenwhere
Ycenare the center coordinates of the image, so that rotation and scaling occurs about this center.
For example, if the image is rotated by some angle theta, A11 and A22 are cos(theta), and A21 and A12 are sin(theta) and -sin(theta). In general, the DX and DY correspond to X and Y displacements of the image after the rotations and distortions specified by the A matrix. The disadvantage of the formal parameters is that the geometrical changes in the image (rotation, size change, and stretch) are all mixed together in the four matrix parameters. This makes the transformation hard to interpret intuitively, and also prevents programs from treating different kinds of changes in different ways.
The semi-natural parameters partly alleviate these difficulties by expressing the A matrix in terms of a global rotation, a global magnification, a difference between the rotation of the X and Y axes, and a difference between the stretches along the X and Y axes. This is done quite easily because two of the parameters (A11 and A21) specify how a unit vector along the X-axis is transformed, which directly indicates how much the X-axis is rotated and stretched by the transformation. Similarly, the other two matrix parameters (A12 and A22) indicate how much the Y-axis is rotated and stretched. The global rotation is the average of the two axis rotations, and the global magnification is the average of the two stretches. Although the stretch is not expressed very intuitively, the effects of the different changes have been separated out in the semi-natural parameters, so they are used by programs that need to treat the different changes differently.
The natural parameters are the most intuitive ones: global rotation, global magnification, and stretch along an axis at a particular angle. These still constitute four parameters because stretch amount and stretch axis angle are two separate parameters. One can directly convert these four natural parameters into an A matrix with equations that involve lots of sines and cosines. Given an A matrix, it is also possible (but even more complex) to solve for the corresponding natural parameters. Most of the programs used for alignment deal only with the formal transform parameters; a few programs work with the semi-natural parameters; and the program that provides interactive adjustment of alignment allows one to specify changes in the fully natural parameters.
Transforms stored in a file are represented by formal parameters. Each line of the file specifies:
A11 A12 A21 A22 DX DYThe command-line program Xf2rotmagstr can be used to convert the transforms in a file into natural parameters.
2 ---+------+----- 1 -+------+------- 0 ------+------+--The pairwise comparison of sections indicates that 1 must be shifted 5 units to the right to align it to 0, and 2 must be shifted 2 units to the left to align it to 1. Performing those operations would give:
2 -+------+-----__ f: DX = -2 1 _____-+------+-- f: DX = +5 0 ------+------+-- f: DX = 0(Here, the underscores _ represent empty or non-existent image.) The misalignment here indicates that a second stage is necessary: one must obtain the transforms for each image that will produce an aligned stack when actually applied to the images. This list is consistently referred to as a list of "g" or global transforms. In our example, the appropriate g transforms produce a proper alignment:
2 __---+------+--- g: DX = +2 1 ____-+------+--- g: DX = +4 0 -----+------+--_ g: DX = -1In summary, f transforms align each section to the preceding one; g transforms align each section to the whole stack.
3 --+------+-------- 2 ---+------+------- 1 -------+------+--- 0 --------+------+--To get g transforms that will align all of the images to a single average position, and eliminate the progressive shift, you can choose Global alignments (remove all trends), which will make Xftoxg be run with the option "-n 0". The resulting g transforms, when applied to the images, would give:
3 ___--+------+----- 2 __---+------+----- 1 ------+------+---__ 0 -----+------+--___This result is generally usable if there are relatively few sections and if you do not care about the image that is lost off the edges for sections farther away from the center of the stack. However, if there are a large number of sections (say, > 30), the shifts can accumulate unacceptably. Also, there may be cases in which the cues that are used for alignment actually should progress across the image area through the sections. To avoid these problems, it is possible to have each section aligned to a "local" average alignment based on a linear fit to trends in the transformations for the nearby sections. This procedure will retain the trends but align adjacent sections as well as possible by eliminating deviations from the trends. This is the default mode of operation of Xftoxg, invoked by the Local fitting (retain trends) choice. The resulting transforms would give:
3 --+------+-------- 2 _---+------+------ 1 ------+------+---_ 0 --------+------+--The situation is more complex if there are trends in rotation or size. Rotation trends can occur because of curvature of the ribbon of sections, or because of progressive rotation of the features used for alignment. Trends in size can easily occur when using Xfalign, if the features that govern the alignment change size progressively through the series. Here is a series of sections with such a size change:
3 --+--x--+----------- 2 --+---x---+--------- 1 -----+----x----+---- 0 -----+-----x-----+--Aligning to a single central position is clearly inappropriate here because it squeezes the images at one end of the stack and expands them at the other:
3 _---+----x----+----- 2 __--+----x----+----- 1 ----+----x----+----_ 0 _---+----x----+--___Aligning to a single central position tends to convert features of interest into a right cylinder, regardless of their true geometry. Again, we can avoid this problem and retain all trends with the default choice, with the following result:
3 --+--x--+----------- 2 _--+---x---+-------- 1 ----+----x----+----_ 0 -----+-----x-----+--This may be an acceptable alignment, especially for very large numbers of sections. However, it is frequently desirable to eliminate the trend in lateral displacements, particularly for images acquired on a CCD camera on the microscope or digitized by a method that does not provide a good preliminary alignment. To handle these situation, there is a hybrid or mixed alignment method that is invoked with the choice Remove trends in translation. The resulting transforms applied to the sections give the desired result and avoid distorting the sizes of the objects:
3 ____--+--x--+------- 2 ___--+---x---+------ 1 ----+----x----+----_ 0 ---+-----x-----+--__If you use hybrid alignments, you need to decide whether to eliminate or retain any trends in rotation. Rotations are probably artifactual and should be eliminated if they are simply due to rotation of images during acquisition or digitization. However, automatic alignment with Xfalign may be dominated by features that actually rotate within the volume, in which case rotation trends should be preserved. For example, a tightly packed bundle of microtubules may have a twist, and if these microtubules dominate the alignment, the alignment will contain a progressive rotation that should be preserved. Choose Remove trends in translation & rotation to eliminate rotation trends.
xfmodel setname.fid setname.xfto compute a list of f transforms. By default, it will solve for the full linear transformation, but there are also options to solve for translations only, translations and rotations, or translations, rotations, and magnifications. Although it is most common to use this method to align an entire image stack, one can also use Xfmodel to "edit" an existing list of f transforms. That is, if one enters fiducial points for a subset of troublesome sections, the program will solve for the transforms for those sections only, and replace those transforms in the existing list. Xfmodel can also be used to transform a model to a new alignment (see below).
One limitation is that a warping transform file cannot be used as an existing transform file to be edited by Xfmodel.
cp setname.xg setname_orig.xgsince this will be needed and is going to be overwritten.
1) You can reopen eTomo and modify the f transforms or the choice of how Xftoxg is run, then make a new aligned stack. To transform a model "setname.mod" into alignment with these images, run
xfmodel -pre setname_orig.xg -xf setname.xg setname.mod setname.newmodThe "-pre" argument tells Xfmodel to back-transform the model by the old g transforms, which aligns it to the raw images, before transforming it by the new transforms.
2) Suppose you work from the already aligned images or a model on them, and derive a set of f transforms that would improve their alignment, in the file "setname_refine.xf". These need to be converted to g transforms, but then transforming the model is easy:
xftoxg setname_refine.xf setname_refine.xg xfmodel -xf setname_refine.xg setname.mod setname.newmodHowever, to get a new aligned stack, you should go back to the raw stack instead of applying a second transformation to the aligned stack, in order to avoid interpolating the data any more times than you have to (each interpolation step potentially degrades the images). To do this, you need to multiply the first and second set of g transforms and apply the product:
xfproduct setname_orig.xg setname_refine.xg setname.xgThen run:
subm newstfor single-frame images or
subm blendfor montages. The existing aligned stack, if still present in the directory, will be renamed with a "~" on the end, but you could rename this first to a preferred name if you want to keep it.
There are several ways that you could work from an existing model to get a refined alignment. One way is to take Tiff snapshots of the Zap window in 3dmod through the entire set of sections, with either the model displayed on the images or just the model showing on a black background. (You can take such snapshots automatically with the dialog box opened with File-Movie/Montage.) Stack the Tiff files into a single color MRC file:
tif2mrc zap*.tif zapstack.mrcMidas can read this file and display the colors in gray scale, allowing you to adjust the alignment based on model features:
midas zapstack.mrc setname_refine.xf
An existing model can also be used to provide a new fiducial alignment, provided that it has features that have been modeled across the sections. For example,
xfmodel setname.mod setname_refine.xfThere is a script, Selfalign, that will do this operation as well as realign the model, producing new f transforms in a ".selfxf" file, new g transforms in a ".selfxg" file, and a new model ending in ".selfmod". Aside from convenience, the script has the advantage that it can be given a limited number of sections over which to find the alignment, and it will properly handle the transitions between the regions being realigned and those being left as they were. Use the ".selfxg" file like "setname_refine.xg" in the xfproduct command above to get new g transforms for building a new aligned stack.