Serial Section Alignment with IMOD



Linear Transformations
F and G Transforms
Automatic alignment
Generating or Editing an Alignment with Fiducial Points
Manually Editing an Alignment and Adding Warping with Midas
Converting from an F List to a G List
Creating Transformed Images
Working with Montaged Images
Realigning Images and Models
Stacking Order and Handedness of Structures

This document gives an overview and some background information on the tools available in IMOD for aligning images from serial sections. First there is a description of the transformations that are used for aligning images and the ways that their parameters are represented. There follows an explanation of why it is necessary to process transformations before they can be used to align serial images. Three ways of obtaining transformations are then described, followed by more detailed explanations of the options for processing transformations for particular kinds of data sets.

Linear Transformations

Two images can be aligned by applying a coordinate transformation to one image to make the transformed image match the other image as well as possible. The transformations used for alignment are general linear transformations. This means that the same amount of translation, rotation or distortion happens everywhere in an image. Our programs use three different ways of representing transformations. The parameters in these different representations are referred to as "formal", "semi-natural", or "natural".

The formal parameters are the matrix coefficients that are actually used to transform image coordinates. There are six formal parameters in a linear transformation: the new X-coordinate of a point is a linear combination of its old X and Y coordinates, plus a constant offset, so three parameters determine the new X-coordinate; similarly, the other 3 parameters determine the new Y-coordinate. The equations are

     X' = A11 * (X - Xcen) + A12 * (Y - Ycen) + DX + Xcen
     Y' = A21 * (X - Xcen) + A22 * (Y - Ycen) + DY + Ycen
where Xcen and Ycen are the center coordinates of the image, so that rotation and scaling occurs about this center.

For example, if the image is rotated by some angle theta, A11 and A22 are cos(theta), and A21 and A12 are sin(theta) and -sin(theta). In general, the DX and DY correspond to X and Y displacements of the image after the rotations and distortions specified by the A matrix. The disadvantage of the formal parameters is that the geometrical changes in the image (rotation, size change, and stretch) are all mixed together in the four matrix parameters. This makes the transformation hard to interpret intuitively, and also prevents programs from treating different kinds of changes in different ways.

The semi-natural parameters partly alleviate these difficulties by expressing the A matrix in terms of a global rotation, a global magnification, a difference between the rotation of the X and Y axes, and a difference between the stretches along the X and Y axes. This is done quite easily because two of the parameters (A11 and A21) specify how a unit vector along the X-axis is transformed, which directly indicates how much the X-axis is rotated and stretched by the transformation. Similarly, the other two matrix parameters (A12 and A22) indicate how much the Y-axis is rotated and stretched. The global rotation is the average of the two axis rotations, and the global magnification is the average of the two stretches. Although the stretch is not expressed very intuitively, the effects of the different changes have been separated out in the semi-natural parameters, so they are used by programs that need to treat the different changes differently.

The natural parameters are the most intuitive ones: global rotation, global magnification, and stretch along an axis at a particular angle. These still constitute four parameters because stretch amount and stretch axis angle are two separate parameters. One can directly convert these four natural parameters into an A matrix with equations that involve lots of sines and cosines. Given an A matrix, it is also possible (but even more complex) to solve for the corresponding natural parameters. Most of the programs used for alignment deal only with the formal transform parameters; a few programs work with the semi-natural parameters; and the program that provides interactive adjustment of alignment allows one to specify changes in the fully natural parameters.

Transforms stored in a file are represented by formal parameters. Each line of the file specifies:

   A11   A12   A21   A22   DX  DY
The program Xf2rotmagstr can be used to convert the transforms in a file into natural parameters.

F and G Transforms

Whenever there are more than two images, a series of transforms will be needed to align the whole stack of images. This series is obtained in two stages. First, each successive pair of images is considered separately, and a transform is found that aligns the images when applied to the second image in the pair. This procedure yields a list of transforms that align each image to the previous image; the programs consistently refer to this as a list of "f" transforms. Simply applying these f transforms to all of the images in the stack will not help, because each section would be aligned to the previous one only if that previous one were not itself transformed. For example, consider the following example of 3 sections that need to be shifted into alignment in one dimension. Here is a side view of the sections, where we need to align the + marks in each section.
 2  ---+------+-----
 1  -+------+-------
 0  ------+------+--
The pairwise comparison of sections indicates that 1 must be shifted 5 units to the right to align it to 0, and 2 must be shifted 2 units to the left to align it to 1. Performing those operations would give:
 2 -+------+-----__    f: DX = -2
 1 _____-+------+--    f: DX = +5
 0 ------+------+--    f: DX =  0
(Here, the underscores _ represent empty or non-existent image.) The misalignment here indicates that a second stage is necessary: one must obtain the transforms for each image that will produce an aligned stack when actually applied to the images. This list is consistently referred to as a list of "g" transforms. In our example, the appropriate g transforms produce a proper alignment:
 2 __---+------+---    g: DX = +2
 1 ____-+------+---    g: DX = +4
 0 -----+------+--_    g: DX = -1
In summary, f transforms align each section to the preceding one; g transforms align each section to the whole stack.

There are three general methods for obtaining alignment transforms: automated search for the best fit between two images, solving for a transform from a set of fiducial points that correspond between two sections, and manual adjustment of transformation parameters while viewing the two sections. The latter methods were developed because the automated method is not usually sufficient.

Automatic alignment

The Python script Xfalign will do automated alignment of a set of serial sections. The main tool that it runs is the program Xfsimplex, which varies the six transformation parameters until it finds the best "fit" between two images. Its primary method of measuring fit is with a difference measure, which is just the sum of the absolute values of pixel-by-pixel differences between the two images. This measure can lead to success if the features that correspond between the two images are relatively large or comprise a substantial fraction of the image area. Two important factors governing the success of Xfsimplex are: 1) appropriate pre-processing of the images to emphasize features that correspond; and 2) proper choice of the parameters for Xfsimplex, particularly how much to reduce the size of images in X and Y. See the man pages for Xfalign and Xfsimplex for more guidance on these points. Xfsimplex provides low-pass and high-pass Fourier filtering, controlled by four parameters specifying the location and width of Gaussian functions. These are the same parameters used by numerous other IMOD programs. You can experiment with simple low- and high-pass filters in the image processing window of 3dmod; the low-frequency sigma, high-frequency cutoff and high-frequency falloff there correspond to the sigma1, radius2, and sigma2 entries to the "-filter" option of Xfalign. You can visualize the filter functions for the full range of possible filters with Filterplot; see that man page for a description of all the effects of the parameters. You can use the same four parameters with Mtffilter to see how more complicated filters treat your data.

Xfsimplex will normally search for the values of the six formal transformation parameters, but one can select a mode in which it will search for the six semi-natural parameters or selected subsets of them. Unless one wishes to vary only a subset of parameters (e.g. displacement only or displacement, rotation, and uniform magnification only), the semi-natural parameters seem to present no advantage.

In the simplest case, one would invoke Xfalign with

    xfalign sections.st sections.xf
which would find search for alignments between the images in "sections.st" and output the f transforms into the file "sections.xf".

Because Xfsimplex does an iterative search for a local optimum in the fitting measure, it cannot find the right alignment if there is a big initial displacement between the images. Xfalign has an option to do an initial cross-correlationusing Tiltxcorr, which will find the displacement that gives a global optimum in the correlation between the images. These displacements are then passed to Xfsimplex, which searches with them as a starting point. This option can be invoked with

     xfalign -pr sections.st sections.xf
Another alternative, if cross-correlations do not work well, is to run Midas initially (see below) and use it to quickly shift the images into approximate alignment. Xfalign can then be told to use these initial alignments, rather than trying to find initial alignments by correlation. If the f transforms produced by Midas are in the file "sections.prexf", then this option can be invoked with
     xfalign -in sections.prexf sections.st sections.xf

Generating or Editing an Alignment with Fiducial Points

The program Xfmodel can solve for the parameters of a transformation between sections from the locations of points that are known or assumed to correspond between the sections. Xfmodel works with a model of connected fiducial points built in 3dmod. In the simplest case, if you built a model of fiducial points named "sections.fid", you would just enter
     xfmodel sections.fid sections.xf
to compute a list of f transforms. By default, it will solve for the full linear transformation, but there are also options to solve for translations only, translations and rotations, or translations, rotations, and magnifications. Although it is most common to use this method to align an entire image stack, one can also use Xfmodel to "edit" an existing list of f transforms. That is, if one enters fiducial points for a subset of troublesome sections, the program will solve for the transforms for those sections only, and replace those transforms in the existing list. Xfmodel can also be used to transform a model to a new alignment (see below).

Manually Editing an Alignment and Adding Warping with Midas

The program Midas provides for interactive visualization and manual adjustment of the alignment between two images. It can be used to edit a list of f transforms. By using Midas, it is not necessary to build an aligned stack to assess the quality of transforms produced by automatic alignment. The program will take the unaligned images, apply any existing transforms, such as those generated by automatic or fiducial alignment, and allow the user to manipulate each transform. It works with the fully natural transform parameters. Midas can be invoked with
     midas sections.st sections.xf
which will load in images from "sections.st". If "sections.xf" is an already-existing file of f transforms, you can then edit the transforms in that file; if the file does not exist, it will be defined as the output file into which transforms will be saved.

Midas allows one to use a nonlinear transformation, or warping, to align any pair of sections. Warping becomes important for very large images where nonlinearities make it difficult to line them up everywhere with a single linear transformation. Essentially, after aligning as much you wish with a linear transformation, you select the option to add warping points. You then add a series of points at locations where there are features that can be aligned. At each location, you shift the image locally into alignment. See the section Warping Images in the Midas man page for more details.

Once you have introduced warping in Midas, the transformation files become much more complex than the single line per section described above. However, all of the programs described below will read these warping transform files interchangeably with simple linear transform files and operate appropriately, which means that all of the operations described below work with warping and require no special options. Two limitations are that a warping transform file cannot be used with the "-initial" option to Xfalign or as an existing transform file to be edited by Xfmodel.

Converting from an F List to a G List

As discussed above, to get from the list of f transforms to a useful stack of aligned images, one must first convert them to g transforms using the program Xftoxg. This will produce transforms that can be used to transform each section into a common alignment. Xftoxg has several different ways of doing this, depending on whether you want to preserve or eliminate trends in the data. The following series of examples will illustrate the treatment of trends. Suppose we have sections with a progressive shift of positions:
 3  --+------+--------
 2  ---+------+-------
 1  -------+------+---
 0  --------+------+--
To get g transforms that will align all of the images to a single average position, and eliminate the progressive shift, run Xftoxg with the option "-n 0" as follows:
   xftoxg -n 0 sections.xf
This entry takes advantage of a feature of Xftoxg: you do not have to enter an output file name if the input file name ends in "f". This command produces a file "sections.xg" of g transforms, which when applied to the images, would give:
 3  ___--+------+-----
 2  __---+------+-----
 1 ------+------+---__
 0  -----+------+--___
This result is generally usable if there are relatively few sections and if you do not care about the image that is lost off the edges for sections farther away from the center of the stack. However, if there are a large number of sections (say, > 50), the shifts can accumulate unacceptably. Also, there may be cases in which the cues that are used for alignment actually should progress across the image area through the sections. To avoid these problems, it is possible to have each section aligned to a "local" average alignment based on a linear fit to trends in the transformations for the nearby sections. This procedure will retain the trends but align adjacent sections as well as possible by eliminating deviations from the trends. This is the default mode of operation of Xftoxg, invoked by
   xftoxg sections.xf
and the resulting transforms in sections.xg would give:
 3  --+------+--------
 2  _---+------+------
 1  ------+------+---_
 0  --------+------+--
The situation is more complex if there are trends in rotation or size. Rotation trends can occur because of curvature of the ribbon of sections, or because of progressive rotation of the features used for alignment. Trends in size can easily occur when using Xfalign, if the features that govern the alignment change size progressively through the series. Here is a series of sections with such a size change:
 3  --+--x--+-----------
 2  --+---x---+---------
 1  -----+----x----+----
 0  -----+-----x-----+--
Aligning to a single central position with the "-n 0" option is clearly inappropriate here because it squeezes the images at one end of the stack and expands them at the other:
 3  _---+----x----+-----
 2  __--+----x----+-----
 1  ----+----x----+----_
 0  _---+----x----+--___
Aligning to a single central position tends to convert features of interest into a right cylinder, regardless of their true geometry. Again, we can avoid this problem and retain all trends with the default entry to Xftoxg, with the following result:
 3  --+--x--+-----------
 2  _--+---x---+--------
 1  ----+----x----+----_
 0  -----+-----x-----+--
This may be an acceptable alignment, especially for very large numbers of sections. However, it is frequently desirable to eliminate the trend in lateral displacements, particularly for images acquired on a CCD camera on the microscope or digitized by a method that does not provide a good preliminary alignment. To handle these situation, there is a hybrid or mixed alignment method that will eliminate trends in position but retain trends in size. It is invoked by
    xftoxg -m 2 sections.xf
The resulting transforms applied to the sections give the desired result and avoid distorting the sizes of the objects:
 3  ____--+--x--+-------
 2  ___--+---x---+------
 1  ----+----x----+----_
 0  ---+-----x-----+--__
If you use hybrid alignments, you need to decide whether to eliminate or retain any trends in rotation. Rotations are probably artifactual and should be eliminated if they are simply due to rotation of images during acquisition or digitization. However, automatic alignment with Xfalign may be dominated by features that actually rotate within the volume, in which case rotation trends should be preserved. For example, a tightly packed bundle of microtubules may have a twist, and if these microtubules dominate the alignment, the alignment will contain a progressive rotation that should be preserved. Use the "-m 2" option to retain rotation trends or "-m 3" to eliminate them.

Creating Transformed Images

Given a list of g transforms, the program Newstack will transform the images to create an aligned stack. This program can also "float" the images to have a common range or mean of intensity, change the size of the images, or extract a portion of the images. See the man page for Newstack for more information or enter "newstack -h". A simple example, which will float the images to the same mean intensity and contrast, is
    newstack -xf sections.xg -fl 2 sections.st sections.ali
Here, "sections.xg" is your file of g transforms produced by Xftoxg, "-fl 2" specifies floating to mean, "sections.st" is your input stack of unaligned images, and "sections.ali" is your output stack of aligned images.

Working with Montaged Images

If you have serial images each consisting of a montage of overlapping frames, as can be acquired with SerialEM, then you need to follow different procedures for getting an initial stack to align, and for getting an aligned stack. The first step is to extract a text file with a list of the X, Y, and Z coordinates for each frame, called a piece list. Do this with
    extractpieces sections.st sections.pl

Next, get a copy of a command file for running the program Blendmont, which can analyze the overlapping image area between adjacent frames and compose a single image for each section. Do:

    cp $IMOD_DIR/com/sloppyblend.com .

Edit this file, replacing "g5a" with the root name of your data file ("sections", in our example here). If you acquired the montage with stage shifts instead of image shift, you should change the entry "SloppyMontage" to "VerySloppyMontage". Now run the program by entering:

    subm sloppyblend

Blendmont will run in the background, and a message will be printed when it finishes. The output file is named sections.bl. Open this in 3dmod and see if the frames appear to have fused properly. Also, unless you have a 1 by n or n by 1 montage, examine the log from Blendmont, in the file sloppyblend.log:

    less sloppyblend.log
For each section, you will see a line like
 8 edges, mean&max error before:  25.22  88.08, after by edges:   1.24   3.06
The two numbers on the right are the ones that indicate whether the pieces line up well after being shifting into registration as well as possible. If these numbers are large (more than a few pixels), it is likely that one or more of the correlations used to align adjacent pieces in that section has failed. If this seems to have happened, then open the file in Midas in order to fix the alignment between pieces:
    midas -b 0  -p sections.pl sections.st sections.ecd 

In Midas, you can go to each troublesome section and examine each of the overlap zones, adjusting the ones where the pieces do not line up. See the section "Fixing Montage Overlaps" in the Midas man page. (Beware: sections are numbered from zero in sloppyblend.log but from 1 in Midas.) After fixing overlaps, save the shifts (this saves a new copy of sections.ecd). Then edit sloppyblend.com and change the 0 to a 1 on the line "ReadInXcorrs". Rerun the program with subm.

Once you have a blended stack, you can use the same procedures on it that were described above with sections.st to align the stack. Once you have a set of g transforms, edit slopyblend.com again to apply these transforms to the original stack in the blending operations. To do this,

  1. change the name on the ImageOutputFile line from ".bl" to ".ali"
  2. remove the "#" before "TransformFile"
  3. change "g5a.xf" on that line to the name of your g transform file ("sections.xg"). Then rerun sloppyblend, and you will get an aligned stack.

Realigning Images and Models

Sometimes one will find after modeling on a set of images that the alignment needs to be improved. There are several tools that can be used to get a better alignment. The operations to run depend first on whether you are going to go back to the original data or operate on the aligned data.

1) You can go back to the original data and modify the f transforms, or rerun Xtftoxg with different parameters to get different g transforms. First make a copy of the xf file:

    cp sections.xf sections.newxf
If you want to fix f transforms, you would probably do this by running Midas on the raw stack:
    midas sections.st sections.newxf
After adjusting the alignment or not, you would rerun Xftoxg (using whatever parameters you used before, or new ones if appropriate), for example:
    xftoxg sections.newxf
This will give you a new file of g transforms, "sections.newxg". Making a new aligned stack it easy; just rerun the newstack command with this new file instead of the old "xg" file. To transform a model into alignment with these images, run
    xfmodel -pre sections.xg -xf sections.newxg sections.mod sections.newmod
The "-pre" argument tells Xfmodel to back-transform the model by the old g transforms, which aligns it to the raw images, before transforming it by the new transforms.

2) Suppose you work from the already aligned images or a model on them, and derive a set of f transforms that would improve their alignment, in the file "sections.xf2". These need to be converted to g transforms, but then transforming the model is easy:

    xftoxg sections.xf2 sections.xg2
    xfmodel -xf sections.xg2 sections.mod sections.newmod
However, to get a new aligned stack, you should go back to the raw stack instead of applying a second transformation to the aligned stack, in order to avoid interpolating the data any more times than you have to (each interpolation step potentially degrades the images). To do this, you need to multiply the first and second set of g transforms and apply the product:
    xfproduct sections.xg sections.xg2 sections.prod
    newstack -xf sections.prod sections.st sections.newali
If you have a montage, instead of running Newstack, edit sloppyblend.com to change "sections.xg" to "sections.prod" and rerun Blendmont.

There are several ways that you could work from an existing model to get a refined alignment. One way is to take Tiff snapshots of the Zap window in 3dmod through the entire set of sections, with either the model displayed on the images or just the model showing on a black background. (You can take such snapshots automatically with the Edit-Movies dialog box.) Stack the Tiff files into a single color MRC file:

    tif2mrc zap*.tif zapstack.mrc
Midas can read this file and display the colors in gray scale, allowing you to adjust the alignment based on model features:
    midas zapstack.mrc sections.xf2

An existing model can also be used to provide a new fiducial alignment, provided that it has features that have been modeled across the sections. For example,

    xfmodel sections.mod sections.xf2
There is a script, Selfalign, that will do this operation as well as realign the model, producing new f transforms in a ".selfxf" file, new g transforms in a ".selfxg" file, and a new model ending in ".selfmod". Aside from convenience, the script has the advantage that it can be given a limited number of sections over which to find the alignment, and it will properly handle the transitions between the regions being realigned and those being left as they were.

Stacking Order and Handedness of Structures

The order in which your images are stacked affects whether structures in your reconstruction have the correct handedness. Handedness matters if you are reconstructing structures such as microtubule bundles that may twist, axonemes, or basal bodies. Note that as sections come off of the block, they represent slices from the top down, when viewing the block face from the outside. Assuming that sections do not get flipped before being loaded onto a grid, the top side of each section faces up and the bottom is against the grid. Handedness will be preserved if the images of the sections represent views from the top of the sections, and if the volume is stacked in order from bottom (deeper in the block) to top. More generally, the handedness of a serial section reconstruction depends on the product of a sequence of factors, each of which you should be able to control or assess: