This document gives an overview and some background information on the tools available in IMOD for aligning images from serial sections. First there is a description of the transformations that are used for aligning images and the ways that their parameters are represented. There follows an explanation of why it is necessary to process transformations before they can be used to align serial images. Three ways of obtaining transformations are then described, followed by more detailed explanations of the options for processing transformations for particular kinds of data sets.
Two images can be aligned by applying a coordinate transformation to one image to make the transformed image match the other image as well as possible. The transformations used for alignment are general linear transformations. This means that the same amount of translation, rotation or distortion happens everywhere in an image. Our programs use three different ways of representing transformations. The parameters in these different representations are referred to as "formal", "semi-natural", or "natural".
The formal parameters are the matrix coefficients that are actually used to transform image coordinates. There are six formal parameters in a linear transformation: the new X-coordinate of a point is a linear combination of its old X and Y coordinates, plus a constant offset, so three parameters determine the new X-coordinate; similarly, the other 3 parameters determine the new Y-coordinate. The equations are
X' = A11 * (X - Xcen) + A12 * (Y - Ycen) + DX + Xcen Y' = A21 * (X - Xcen) + A22 * (Y - Ycen) + DY + Ycenwhere
Xcen
and Ycen
are the center coordinates of
the image, so that rotation and scaling occurs about this center.
For example, if the image is rotated by some angle theta, A11 and A22 are cos(theta), and A21 and A12 are sin(theta) and -sin(theta). In general, the DX and DY correspond to X and Y displacements of the image after the rotations and distortions specified by the A matrix. The disadvantage of the formal parameters is that the geometrical changes in the image (rotation, size change, and stretch) are all mixed together in the four matrix parameters. This makes the transformation hard to interpret intuitively, and also prevents programs from treating different kinds of changes in different ways.
The semi-natural parameters partly alleviate these difficulties by expressing the A matrix in terms of a global rotation, a global magnification, a difference between the rotation of the X and Y axes, and a difference between the stretches along the X and Y axes. This is done quite easily because two of the parameters (A11 and A21) specify how a unit vector along the X-axis is transformed, which directly indicates how much the X-axis is rotated and stretched by the transformation. Similarly, the other two matrix parameters (A12 and A22) indicate how much the Y-axis is rotated and stretched. The global rotation is the average of the two axis rotations, and the global magnification is the average of the two stretches. Although the stretch is not expressed very intuitively, the effects of the different changes have been separated out in the semi-natural parameters, so they are used by programs that need to treat the different changes differently.
The natural parameters are the most intuitive ones: global rotation, global magnification, and stretch along an axis at a particular angle. These still constitute four parameters because stretch amount and stretch axis angle are two separate parameters. One can directly convert these four natural parameters into an A matrix with equations that involve lots of sines and cosines. Given an A matrix, it is also possible (but even more complex) to solve for the corresponding natural parameters. Most of the programs used for alignment deal only with the formal transform parameters; a few programs work with the semi-natural parameters; and the program that provides interactive adjustment of alignment allows one to specify changes in the fully natural parameters.
Transforms stored in a file are represented by formal parameters. Each line of the file specifies:
A11 A12 A21 A22 DX DYThe program Xf2rotmagstr can be used to convert the transforms in a file into natural parameters.
2 ---+------+----- 1 -+------+------- 0 ------+------+--The pairwise comparison of sections indicates that 1 must be shifted 5 units to the right to align it to 0, and 2 must be shifted 2 units to the left to align it to 1. Performing those operations would give:
2 -+------+-----__ f: DX = -2 1 _____-+------+-- f: DX = +5 0 ------+------+-- f: DX = 0(Here, the underscores _ represent empty or non-existent image.) The misalignment here indicates that a second stage is necessary: one must obtain the transforms for each image that will produce an aligned stack when actually applied to the images. This list is consistently referred to as a list of "g" transforms. In our example, the appropriate g transforms produce a proper alignment:
2 __---+------+--- g: DX = +2 1 ____-+------+--- g: DX = +4 0 -----+------+--_ g: DX = -1In summary, f transforms align each section to the preceding one; g transforms align each section to the whole stack.
There are three general methods for obtaining alignment transforms: automated search for the best fit between two images, solving for a transform from a set of fiducial points that correspond between two sections, and manual adjustment of transformation parameters while viewing the two sections. The latter methods were developed because the automated method is not usually sufficient.
Xfsimplex will normally search for the values of the six formal transformation parameters, but one can select a mode in which it will search for the six semi-natural parameters or selected subsets of them. Unless one wishes to vary only a subset of parameters (e.g. displacement only or displacement, rotation, and uniform magnification only), the semi-natural parameters seem to present no advantage.
In the simplest case, one would invoke Xfalign with
xfalign sections.st sections.xfwhich would find search for alignments between the images in "sections.st" and output the f transforms into the file "sections.xf".
Because Xfsimplex does an iterative search for a local optimum in the fitting measure, it cannot find the right alignment if there is a big initial displacement between the images. Xfalign has an option to do an initial cross-correlationusing Tiltxcorr, which will find the displacement that gives a global optimum in the correlation between the images. These displacements are then passed to Xfsimplex, which searches with them as a starting point. This option can be invoked with
xfalign -pr sections.st sections.xfAnother alternative, if cross-correlations do not work well, is to run Midas initially (see below) and use it to quickly shift the images into approximate alignment. Xfalign can then be told to use these initial alignments, rather than trying to find initial alignments by correlation. If the f transforms produced by Midas are in the file "sections.prexf", then this option can be invoked with
xfalign -in sections.prexf sections.st sections.xf
xfmodel sections.fid sections.xfto compute a list of f transforms. By default, it will solve for the full linear transformation, but there are also options to solve for translations only, translations and rotations, or translations, rotations, and magnifications. Although it is most common to use this method to align an entire image stack, one can also use Xfmodel to "edit" an existing list of f transforms. That is, if one enters fiducial points for a subset of troublesome sections, the program will solve for the transforms for those sections only, and replace those transforms in the existing list. Xfmodel can also be used to transform a model to a new alignment (see below).
midas sections.st sections.xfwhich will load in images from "sections.st". If "sections.xf" is an already-existing file of f transforms, you can then edit the transforms in that file; if the file does not exist, it will be defined as the output file into which transforms will be saved.
Midas allows one to use a nonlinear transformation, or warping, to align any pair of sections. Warping becomes important for very large images where nonlinearities make it difficult to line them up everywhere with a single linear transformation. Essentially, after aligning as much you wish with a linear transformation, you select the option to add warping points. You then add a series of points at locations where there are features that can be aligned. At each location, you shift the image locally into alignment. See the section Warping Images in the Midas man page for more details.
Once you have introduced warping in Midas, the transformation files become much more complex than the single line per section described above. However, all of the programs described below will read these warping transform files interchangeably with simple linear transform files and operate appropriately, which means that all of the operations described below work with warping and require no special options. Two limitations are that a warping transform file cannot be used with the "-initial" option to Xfalign or as an existing transform file to be edited by Xfmodel.
3 --+------+-------- 2 ---+------+------- 1 -------+------+--- 0 --------+------+--To get g transforms that will align all of the images to a single average position, and eliminate the progressive shift, run Xftoxg with the option "-n 0" as follows:
xftoxg -n 0 sections.xfThis entry takes advantage of a feature of Xftoxg: you do not have to enter an output file name if the input file name ends in "f". This command produces a file "sections.xg" of g transforms, which when applied to the images, would give:
3 ___--+------+----- 2 __---+------+----- 1 ------+------+---__ 0 -----+------+--___This result is generally usable if there are relatively few sections and if you do not care about the image that is lost off the edges for sections farther away from the center of the stack. However, if there are a large number of sections (say, > 50), the shifts can accumulate unacceptably. Also, there may be cases in which the cues that are used for alignment actually should progress across the image area through the sections. To avoid these problems, it is possible to have each section aligned to a "local" average alignment based on a linear fit to trends in the transformations for the nearby sections. This procedure will retain the trends but align adjacent sections as well as possible by eliminating deviations from the trends. This is the default mode of operation of Xftoxg, invoked by
xftoxg sections.xfand the resulting transforms in sections.xg would give:
3 --+------+-------- 2 _---+------+------ 1 ------+------+---_ 0 --------+------+--The situation is more complex if there are trends in rotation or size. Rotation trends can occur because of curvature of the ribbon of sections, or because of progressive rotation of the features used for alignment. Trends in size can easily occur when using Xfalign, if the features that govern the alignment change size progressively through the series. Here is a series of sections with such a size change:
3 --+--x--+----------- 2 --+---x---+--------- 1 -----+----x----+---- 0 -----+-----x-----+--Aligning to a single central position with the "-n 0" option is clearly inappropriate here because it squeezes the images at one end of the stack and expands them at the other:
3 _---+----x----+----- 2 __--+----x----+----- 1 ----+----x----+----_ 0 _---+----x----+--___Aligning to a single central position tends to convert features of interest into a right cylinder, regardless of their true geometry. Again, we can avoid this problem and retain all trends with the default entry to Xftoxg, with the following result:
3 --+--x--+----------- 2 _--+---x---+-------- 1 ----+----x----+----_ 0 -----+-----x-----+--This may be an acceptable alignment, especially for very large numbers of sections. However, it is frequently desirable to eliminate the trend in lateral displacements, particularly for images acquired on a CCD camera on the microscope or digitized by a method that does not provide a good preliminary alignment. To handle these situation, there is a hybrid or mixed alignment method that will eliminate trends in position but retain trends in size. It is invoked by
xftoxg -m 2 sections.xfThe resulting transforms applied to the sections give the desired result and avoid distorting the sizes of the objects:
3 ____--+--x--+------- 2 ___--+---x---+------ 1 ----+----x----+----_ 0 ---+-----x-----+--__If you use hybrid alignments, you need to decide whether to eliminate or retain any trends in rotation. Rotations are probably artifactual and should be eliminated if they are simply due to rotation of images during acquisition or digitization. However, automatic alignment with Xfalign may be dominated by features that actually rotate within the volume, in which case rotation trends should be preserved. For example, a tightly packed bundle of microtubules may have a twist, and if these microtubules dominate the alignment, the alignment will contain a progressive rotation that should be preserved. Use the "-m 2" option to retain rotation trends or "-m 3" to eliminate them.
newstack -xf sections.xg -fl 2 sections.st sections.aliHere, "sections.xg" is your file of g transforms produced by Xftoxg, "-fl 2" specifies floating to mean, "sections.st" is your input stack of unaligned images, and "sections.ali" is your output stack of aligned images.
extractpieces sections.st sections.pl
Next, get a copy of a command file for running the program Blendmont, which can analyze the overlapping image area between adjacent frames and compose a single image for each section. Do:
cp $IMOD_DIR/com/sloppyblend.com .
Edit this file, replacing "g5a" with the root name of your data file ("sections", in our example here). If you acquired the montage with stage shifts instead of image shift, you should change the entry "SloppyMontage" to "VerySloppyMontage". Now run the program by entering:
subm sloppyblend
Blendmont will run in the background, and a message will be printed when it finishes. The output file is named sections.bl. Open this in 3dmod and see if the frames appear to have fused properly. Also, unless you have a 1 by n or n by 1 montage, examine the log from Blendmont, in the file sloppyblend.log:
less sloppyblend.logFor each section, you will see a line like
8 edges, mean&max error before: 25.22 88.08, after by edges: 1.24 3.06The two numbers on the right are the ones that indicate whether the pieces line up well after being shifting into registration as well as possible. If these numbers are large (more than a few pixels), it is likely that one or more of the correlations used to align adjacent pieces in that section has failed. If this seems to have happened, then open the file in Midas in order to fix the alignment between pieces:
midas -b 0 -p sections.pl sections.st sections.ecd
In Midas, you can go to each troublesome section and examine each of the overlap zones, adjusting the ones where the pieces do not line up. See the section "Fixing Montage Overlaps" in the Midas man page. (Beware: sections are numbered from zero in sloppyblend.log but from 1 in Midas.) After fixing overlaps, save the shifts (this saves a new copy of sections.ecd). Then edit sloppyblend.com and change the 0 to a 1 on the line "ReadInXcorrs". Rerun the program with subm.
Once you have a blended stack, you can use the same procedures on it that were described above with sections.st to align the stack. Once you have a set of g transforms, edit slopyblend.com again to apply these transforms to the original stack in the blending operations. To do this,
1) You can go back to the original data and modify the f transforms, or rerun Xtftoxg with different parameters to get different g transforms. First make a copy of the xf file:
cp sections.xf sections.newxfIf you want to fix f transforms, you would probably do this by running Midas on the raw stack:
midas sections.st sections.newxfAfter adjusting the alignment or not, you would rerun Xftoxg (using whatever parameters you used before, or new ones if appropriate), for example:
xftoxg sections.newxfThis will give you a new file of g transforms, "sections.newxg". Making a new aligned stack it easy; just rerun the newstack command with this new file instead of the old "xg" file. To transform a model into alignment with these images, run
xfmodel -pre sections.xg -xf sections.newxg sections.mod sections.newmodThe "-pre" argument tells Xfmodel to back-transform the model by the old g transforms, which aligns it to the raw images, before transforming it by the new transforms.
2) Suppose you work from the already aligned images or a model on them, and derive a set of f transforms that would improve their alignment, in the file "sections.xf2". These need to be converted to g transforms, but then transforming the model is easy:
xftoxg sections.xf2 sections.xg2 xfmodel -xf sections.xg2 sections.mod sections.newmodHowever, to get a new aligned stack, you should go back to the raw stack instead of applying a second transformation to the aligned stack, in order to avoid interpolating the data any more times than you have to (each interpolation step potentially degrades the images). To do this, you need to multiply the first and second set of g transforms and apply the product:
xfproduct sections.xg sections.xg2 sections.prod newstack -xf sections.prod sections.st sections.newaliIf you have a montage, instead of running Newstack, edit sloppyblend.com to change "sections.xg" to "sections.prod" and rerun Blendmont.
There are several ways that you could work from an existing model to get a refined alignment. One way is to take Tiff snapshots of the Zap window in 3dmod through the entire set of sections, with either the model displayed on the images or just the model showing on a black background. (You can take such snapshots automatically with the Edit-Movies dialog box.) Stack the Tiff files into a single color MRC file:
tif2mrc zap*.tif zapstack.mrcMidas can read this file and display the colors in gray scale, allowing you to adjust the alignment based on model features:
midas zapstack.mrc sections.xf2
An existing model can also be used to provide a new fiducial alignment, provided that it has features that have been modeled across the sections. For example,
xfmodel sections.mod sections.xf2There is a script, Selfalign, that will do this operation as well as realign the model, producing new f transforms in a ".selfxf" file, new g transforms in a ".selfxg" file, and a new model ending in ".selfmod". Aside from convenience, the script has the advantage that it can be given a limited number of sections over which to find the alignment, and it will properly handle the transitions between the regions being realigned and those being left as they were.
clip flipz input_file output_file