Image Acquisition from CCD Cameras and Direct Electron Detectors
Sections of interest below:
The first part of this page explains some features of CCD cameras. Direct electron detectors have significantly different behavior and some of their features are explained in Direct Electron Detectors, Especially the Gatan K2/K3. Newer CMOS-based cameras also have different behavior, so this first section mostly applies to traditional, high-quality CCD cameras. The principal differences for CMOS cameras are that readout happens continuously during the exposure instead of at the end (thus readout time is not an issue), and that binning makes little difference in acquisition time except for the time savings of moving smaller amounts of data around in software.
An image is captured from the CCD camera with a sequence of 3 steps: clearing the CCD chip, exposing the camera to the beam and integrating charges for some period of time, then reading the image off of the chip. The readout is done line by line, so it is often very time-consuming. The clearing time is also substantial because the chip is cleared by shifting accumulated charges to one edge. Clearing times are about 0.7 to 0.9 second for 2K Megascan or 4K single-port readout Ultrascan cameras, 0.25 second for a 4-port readout Ultrascan, or very short for some current 1K cameras. It is important not to have a beam on the camera during the clearing time, because then charges will build up as they are being removed, adding a ramp of intensity to the image that is subsequently acquired.
Many plastic-embedded specimens show a transient shift of the image when the beam is turned on to take an exposure; this drift produces a smeared image. To take good images, you need to use drift settling, which provides an initial exposure of the specimen to the beam just before the image is acquired. This initial shift is worse on slot grids than on mesh grids and tends to be worse at high tilts. It seems to depend on the total beam hitting the sample rather than the brightness of the beam in the area of interest. This means you should work with the largest spot size that gives you enough beam.
Exposing the specimen to the beam before the exposure without exposing the camera during its clearing time requires the use of at least two shutters, one below the specimen and one above. The traces below show how DigitalMicrograph manages the two shutters in standard shuttering mode with the alternate (beam) shutter normally closed. A trace goes high to indicate when a shutter is open or beam is on the camera. Although this shuttering provides some pre-exposure, there are two problems here. First, the amount of pre-exposure is fixed at the clear time, which could be too short or longer than needed. Second, the beam is left on the specimen during camera readout. (Note that this occurs only when using the shuttering mode that provides pre-exposure and does not happen if you are just using the beam shutter for low-dose exposures.)
DigitalMicrograph does provide a way to increase the drift settling, but this has problems of its own. As shown in the following traces, the extra settling is implemented by keeping the beam on the CCD during the clear time. This introduces an intensity ramp in the image, which is eliminated by having the beam on for the equivalent time during the 'dark' reference. These ramps use up some of the dynamic range of the camera, but worse than that, the dark reference contains image features and will become invalid when exposure or specimen location changes.
To deal with these problems, SerialEM has implemented a shuttering method using the scripting beam-blanker, which operates separately from the camera shuttering. As indicated in the two sets of traces below, this extra shuttering can provide a flexible range of drift settling times and also eliminates the many seconds of exposure during camera readout.
Low-noise CCD cameras are inherently slow, so you should be aware of what governs image capture time and how to speed it up. The total capture time is the sum of: any drift settling time above the clearing time, the clearing time, the actual exposure time, time to read out the image, and time to process and display it. Drift settling of less than the clearing time is subsumed in the clearing time. Processing time is minimal, so you should always use gain normalization to remove intensity variations due to imperfections in the phosphor screen. Read-out time is proportional to the number of pixels of data being read out of the CCD chip. It is about 12 seconds for the 2K x 2K pixels through a single readout port, so binning, which is done on the chip before read-out, can speed up the read-out tremendously. The read-out time is actually dominated by the number of lines being read, so for a given number of pixels being read, the read-out is fastest if those pixels are in the fewest number of lines. This is the reason for the Wide Quarter and Wide Half areas provided by the buttons in the Camera Setup dialog. Their readouts are only somewhat longer than the readout for square images that are half as big, not twice as long.
Direct Electron Detectors, Especially the Gatan K2/K3
The first part of this section describes features common to direct detector cameras, but most of it is specific to the Gatan K2 or K3 camera. If you are familiar with the K2, see the section Principal Differences between K2 and K3.
Binning and summing of signals. Binning and integration over time behave quite differently for direct detectors and other CMOS-based cameras than for CCD cameras. With a high-end cooled CCD, an image is accumulated on the chip for the whole exposure time, and pixels are binned before readout, so each binned pixel of the final image is read out once. This means that a very weak beam can be integrated for a long time and binned to make the readout noise small compared to the signal. With the CMOS-based cameras, the amount of charge that can be accumulated is limited so the chip has to be read out frequently, and readout happens at each pixel on the chip, so one unit of read-out noise is added to each pixel during each frame. Integrating frames over time and binning the pixels after readout will average out this noise but cannot reduce the amount of noise that has to get averaged, unlike with a CCD. With the K2 and K3 cameras, the readout is particularly fast to allow counting mode to be used, but the result is that linear mode images are quite impaired by noise at low dose rates.
Binning and anti-aliased image reduction with the K2 and K3 Cameras. The K2 camera is intrinsically a 3.8K x 3.7K camera but can produce 7.6K x 7.4K images in Super-Resolution mode. Similarly, the K3 camera has 5.8K x 4.1K sensor but its electron counting mode always produces super-resolution 11.5K x 8.2K images. SerialEM resolves this by treating the cameras internally as 7.6K and 11.5K in all modes of operation, but maintaining binning values that are twice those actually applied to acquire images. However, the user interface presents each as a camera with the size of the sensor and with the binning values that one is accustomed to seeing, with one exception: a 7.6K super-resolution image is referred to as having binning 0.5.
With a K2, Super-resolution mode has an advantage over counting mode only with no binning in DM, which yields a 7.6K image. For example, a super-resolution image taken with binning by 2 in DM will produce a 3.8K image that is identical to the 3.8K image that could have been obtained with counting mode and no binning. To provide a useful alternative, SerialEM uses image reduction by antialias filtering, to remove the high-frequency noise that would be introduced by aliasing when images are binned. This method is used for super-resolution mode whenever a binning above 0.5 is selected. With a K3, antialiased reduction is used for all images returned to SerialEM with binning above 0.5, except in continuous mode.
Antialiased reduction has a measurable advantage over binning with electron counting images. If pairs of super-resolution images are taken at a variety of doses and analyzed by Fourier ring correlation, the FRC past half-Nyquist is about 35% higher with antialiased reduction by 2 than with binning by 2, and about 50% higher with reduction by 4 rather than with binning by 4.
Antialiased reduction is used by default instead of true binning for all "binned" images in K2 counting and super-resolution modes and for K3 counting images, and the reduction is done by the SerialEMCCD plugin on the K2 or K3 computer, where it can run quickly. At the bottom of the Camera Setup dialog, there is a checkbox to turn off this behavior, followed by an indicator in bold about whether a reduction will use Anti-aliasing or Binning.
Scaling of electron counts. In counting or super-resolution mode with a K2, DM produces images with the number of electrons, which is very different from the number of digitized counts produced in linear mode. Since many parts of SerialEM rely on there being a fixed relationship between beam intensity and image counts, the plugin scales electron counts up by a factor close to the number of counts per electron in linear mode, so that all modes produce approximately the same number of counts per physical pixel for a given exposure time and beam intensity.
In contrast, the K3 in electron counting mode produces 32 counts per detected electron, while the linear mode in DM produces huge numbers of counts. In this case, SerialEM scales linear mode to match the counting mode output, using that a factor that has to be set in the SerialEM properties file.
Dose rate for effective electron counting. Electron counting is effective only over a limited range of dose rates. For the K2, it is best to keep the dose rate between 2 and 10 electrons per physical pixel per second; at some point (roughly around 20 e/physical pixel/sec) counting becomes less efficient than integrating in linear mode. The K3 has 3.75 times faster readout rate, which means all these dose rates would be that much higher, so the best operating range would be ~7 to 40 e/physical pixel/sec. There is a dose rate readout of the actual mean e/physical pixel/sec in an image at the upper left of the display window, plus you can use Min/Max/Mean in the Process menu to obtain the dose rate. There is also a dose rate estimate in the Camera Setup dialog, which predicts the dose rate under the current conditions based on the change in beam intensity since the last acquired image.
For the K2, the number of electron counts produced per true electron drops by a factor of ~0.97 at 5 e/physical pixel/sec, ~0.94 at 7.5 e/physical pixel/sec, ~0.87 at 10 e/physical pixel/sec, and ~0.6 at 20 e/physical pixel/sec. Unfortunately, even at low dose rates, the conversion efficiency is apparently only 0.87 counts per true electron (Li et al., 2013). Similar values apply for the K3 at the correspondingly higher dose rates. As of SerialEM 3.6, the dose rate outputs are all adjusted to compensate for both the nonlinearity and the loss of signal; they should match the dose rate readout in DigitalMicrograph.
Dose calibration. Because of the adjustments just described, dose calibration with a counting mode image should be accurate for a fairly wide dose rate range. Dose calibration with linear mode images from a K2 would also be accurate, as long as the property for counts per electron is set correctly. There should be no need to use linear mode images for this purpose with a K3 camera. Measuring Camera Gain describes a procedure for setting the counts per electron and for setting a separate property to equalize the counts between linear and counting mode images at a particular dose rate for K2 camera.
Camera timing. Camera timing is also unlike that of a CCD and is mostly independent of binning. For the K2, linear and counting mode shots take ~3 sec plus the exposure time. A super-resolution shot takes ~6 sec plus the exposure time, plus the time required to return the 7.6K image to SerialEM, which is generally at least 1.3 sec over a fast gigabit network. For a K3, the basic time is shorter (~2 sec) and is no penalty for getting the super-resolution images.
Dark and gain correction. Because of the complexity of the dark and gain corrections in DM, all gain normalization is done in DM instead of SerialEM, and it is not possible to take a gain reference in SerialEM. For the K2, DM will keep one dark reference for each binning and camera mode combination, so when a new exposure time is used, DM has to prepare or obtain a new dark reference for that mode and binning. For linear mode, this takes ~2 sec plus the exposure time, whereas for counting and super-resolution mode it seems to take just ~0.6 sec regardless of exposure time. Thus, when setting up your different camera parameter sets for a tilt series, it is not very important to avoid lots of dark reference events if you are not using linear mode at all (which is advisable for low dose work). However, for higher dose work where linear mode is used, try to set up Record, Focus, and Trial so that they have different binnings to avoid frequent dark references when going between these image types. For the K3, a dark reference is apparently taken for each shot and these considerations do not apply.
Limitations in Dose Fractionation mode. Before GMS 2.3.1, Dose Fractionation mode could produce only unbinned, full frame images, 3.8K in linear or counting mode and 7.6K in super-resolution mode. Although these limitations were removed, SerialEM was never modified to take advantage of the change. For the summed image returned to SerialEM, antialiased image reduction is applied to achieve whatever reduction is selected (i.e., with binning higher than 1 for linear or counting mode, or binning higher than 0.5 for super-resolution mode). For the K3, SerialEM enforces the same restrictions on images in Dose Fractionation mode, with one exception: it allows electron counting frames being saved or aligned to be taken with fast hardware binning by 2, thus providing the same kind of frames as counting mode gives on the K2. When unnormalized frames are saved, the plugin creates and saves a binned reference appropriate for correcting these frames from the Dm super-resolution reference.
Saving frames from the plugin. The ability to save frames in Dose Fractionation mode is implemented by having SerialEM's plugin to DM save frames directly rather than passing them back to SerialEM. Only the final summed image produced by DM is passed back. The frames for one image acquisition are saved either in a single MRC or TIFF stack or as a numbered set of single-image files inside of a newly created directory. The stack or directory is given a unique name based on the date and time or on choices in the K2 Frame File Options dialog; this name will be printed to the log window along with the number of frames saved. This information is also saved in a '.mdoc' metadata file if the summed image is saved, or in a '.mdoc' file opened with Open .mdoc for Frames in the File menu. The frames can be saved only on the computer hosting the K2 or K3 camera, or in file systems accessible to that computer.
The scaling and mode of the saved frames depends on whether gain normalization is selected in SerialEM, which invokes software gain correction in DM for the individual frames, and also differs between K2 and K3 cameras. For a K2 camera, with the gain correction (which does some useful scaling to remove artifacts), DM passes the plugin floating point frames with fractional values in both modes. The plugin scales a super-resolution mode frame by 16 and stores it as bytes, or a counting mode frame by the regular scaling applied to summed images and stores it as integers. The title in the header of the MRC file includes the scaling applied to the images. Without the gain correction, DM passes the plugin frames with electron counts in integer or byte format in counting or super-resolution mode, respectively, and the plugin saves these data without scaling. The state of the Pack unnormalized data option in the Frame File Options dialog determines whether these raw counts are saved in a packed form. If this option is selected, the plugin truncates the integers in counting mode as bytes and packs the bytes in super-resolution mode so that two 4-bit numbers are stored in each byte. Otherwise, the plugin saves the integers or bytes that it received, respectively.
For K3 cameras, normalized data are passed to the plugin as bytes with the fixed scaling of 32, so they are ready to save as bytes. This is still the case if the frames are taken with binning by 2 (to mimic K2 counting mode). Unnormalized frames are always bytes as well, so they can be saved as is or packed two per byte.
By default, the packed 4-bit data files will look like ordinary byte files, but with half the true size in X; thus other programs will be able to open them just like any other byte file but they will appear squeezed by a factor of 2 in X. For MRC files, the Frame File Options dialog allows you to save 4-bit data as mode 101, which IMOD and MotionCor2 support but other programs might not. That dialog also allows you to save the data as TIFF files with LZW or ZIP compression; LZW is best for unnormalized data and is also much faster. If you save data as compressed TIFF files, the program 'tif2mrc' in IMOD can be used to convert each file to MRC format before processing it; or, in fact, any IMOD program can be run directly on the TIFF files.
To take advantage of this option, you can select unnormalized instead of gain normalized images in the Camera Setup dialog. A more convenient approach is to leave the selection as normalized in that dialog, but turn on Save unnormalized frames even if Gain Normalized is selected in the Frame File Options dialog. There is actually no dark-subtraction involved, and these two selections now produce the same result. When frames are saved unnormalized, the plugin will return a gain-normalized sum of the frames even though the normalization is not selected, so that this image will not have artifacts that could impair cross-correlation alignment.
Signal loss from saving normalized frames as integers. With the K2, originally there was some concern that rounding the floating point frame values to integers will lose signal and reduce the efficiency of the camera. To explore this point, the image formation was modeled by 1) generating a Poisson deviate to represent a true electron count as the input signal; 2) adding a Gaussian deviate with a sigma of 0.05 times the electron count, which reproduces the histogram of spreads around integer counts in a normalized frame; 3) setting the output to zero for 0.13 of trials, to reproduce the typical signal loss in electron counting; 4) multiplying by 16, the smallest multiplier applied when saving frames; 5) taking this value as a floating point output, or rounding to the nearest integer for integer output. DQE was computed as the square of the ratio of the output SNR to the input SNR. The loss in DQE from rounding to integers depends on the number of electrons per saved pixel (not per physical pixel for super-resolution mode):
Electrons per pixel Loss in DQE
0.1
0.0005
0.5
0.0004
1.0
0.0003
2.0
0.0002
The reasons the loss is so small are that 1) the vast majority of normalized pixel values are close to integer values anyway; 2) the SNR is so low in the images. To put this loss in perspective, with a loss of 0.0005, you would need 10005 particles instead of 10000 particles to reach a certain resolution.
Gain-normalizing K2/K3 frames during post-processing. If you choose to save frames without software gain normalizing, the plugin will make sure that there is a valid copy of the appropriate gain reference in the folder where the frames are saved, provided that the plugin can find the gain reference. This requires a correct entry for the camera property DMGainReferenceName. In addition, since there is no defect correction in K2 images with GMS 2.3.1 or higher, the plugin will save a file 'defects...txt' describing the defects that need to be corrected in post-processing. For K3, DM does do defect correction, so no 'defects...txt' file is produced unless there are addition bad columns or rows identified in the SerialEMproperties. You can convert the gain reference to an MRC file with the IMOD program 'dm2mrc' or, again, use it directly in any program. The gain reference may need to be rotated and/or flipped by the same operations described in the camera configuration in DM, unless the frames are saved without rotation and flipping as provided by an option in the Frame File Options dialog.
If reoriented frames are saved, the header of the frame stacks will have a title line with the text 'r/f' followed by a value corresponding to a RotationAndFlip entry in the SerialEM properties file. This value is the rotation counterclockwise divided by 90 (i.e., 1, 2, or 3 for angles of 90, 180 or 270) plus 4 for a flip around the Y axis before the rotation. This operation is what needs to be applied to the gain reference. If the frames were saved without rotation and flip, the title line will have the text 'r/f 0 need' followed by the rotation/flip value that was not applied to the frames. The gain reference does not need to be rotated, but the final aligned sums do need to be rotated by this operation to match images returned to SerialEM and restore handedness if a flip is needed. With 'alignframes' in IMOD 4.12.18 or higher, the option '-rotation -2' or '-rfsum -1' can be used to apply the operation specified in the 'need' value.
There are several ways to handle the rotation of the gain reference:
A 'clip' command that will normalize all kinds of saved frames (4-bit packed, 8-bit, 16-bit, mode 101) is:
clip norm -n scalingFactor -m 1 -R -1 -D defects....txt fileWithFrames gainReferenceFile normalizedFrames.mrc
Here, 'scalingFactor' is the regular scaling factor applied to summed images, 'Defects....txt' is the defects file, 'fileWithFrames' is the data file to normalize, 'gainReferenceFile' is the gain reference, and 'normalizedFrames.mrc' is the desired output file. The default scaling factor is 16, but you probably want to apply the appropriate value for counts per electron, which is 32 for K3. The '-D' option would be omitted if there is no defects file. Extreme values can be removed by adding '-l' and '-h' options.
Conversion to floating point can be used instead of, or in addition to, the scaling to guarantee absolutely no information loss from rounding to integers. To convert to float, use "-m 2" instead of "-m 1". The extra storage required will not be an issue if the normalized data are only being used temporarily for alignment and averaging of frames. Enter '-n 1' to prevent the scaling. With IMOD 4.12.17 or higher, you can use '-m 16' to produce 16-bit floating point values.
Finally, if you want to run MotionCor2 directly on the unnormalized data, you
should give it a defect map file as well as the gain reference file. You
can make a defect map from the text file with 'clip defect' in IMOD 4.10.7 or
higher::
clip defect -D defects...txt fileWithFrames defects...mrc
where the fileWithFrames is used only to set the size of the output and can be
any file of the right X and Y size. To make a compressed TIFF file, which
will be much smaller, use:
clip defect -D defects...txt -f tif fileWithFrames
defects...tif
Drift settling Drift settling is implemented not by having two shutters but by having a longer exposure in which readout frames at the beginning are discarded. This is known to work correctly when there is only one shutter connected. If there are two shutters, it may work correctly only if the second shuttering option is selected, as was found to be the case for the OneView.
Constraints on timing. There are several constraints on timing that are enforced both for the entries in the Camera Setup dialog
It is possible to overcome the constraints on exposure time in K2 counting and super-resolution modes by turning on Dose Fractionation mode and setting the frame time equal to the exposure time, with no aligning or saving selected. Doing so will cost an extra 4 sec in counting mode or 5.5 sec in super-resolution mode. For K3, the extra time will be much less but there is much less need to do so. This equality between frame and exposure time will be retained when you use either of the two methods of changing exposure time during a tilt series, i.e., changing exposure time instead of intensity or changing exposure time at scheduled tilt angles. See the description of these options in Beam Intensity Control and Tilt Series Scheduled Changes Dialog for more details.
Principal Differences between K2 and K3
Handling Falcon EER files in IMOD
The support for EER files in IMOD is designed to alleviate the problem that single 4x4 super-resolution frames are unwieldy for many purposes, particularly for tilt series data. EER files are handled directly in the TIFF-reading module, which can return images that are:
Three programs, 3dmod, Alignframes, and Clip, have specific options controlling these features, with defaults that can be overridden by the environment variables IMOD_DFLT_EER_SUPER_RES and IMOD_DFLT_EER_Z_SUMMING . All other programs will see the files by default as 16K and containing all the frames, but the frame-summing and reduction can be controlled by setting other environment variables. Run 'imodhelp imodenv' and search for 'EER' to find descriptions of these environment variables.
You should have IMOD 4.12.13 or higher to have all the capabilities described here.
3dmod will read EER files by default with antialiased reduction by 2 to 8K and Z-summing by 10; these defaults can be set in its Preferences dialog and also overridden with options at the command line and in its startup window. Simple 3dmod packages for Windows are available and suitable for use on microscope and camera computers.
Alignframes also reads EER files by default with antialiased reduction by 2 and Z-summing by 10, and there are options to override these defaults. For example, to reduce by 4 and sum by 16, enter the option '-es 0,16,-1' or set environment variables IMOD_DFLT_EER_SUPER_RES to 0 and IMOD_DFLT_EER_Z_SUMMING to 15. The gain reference needs to supplied with the '-gain' option, and then defect correction will be done based on the defect list in the gain reference.
Clip applies no Z summing or reduction by default and has options '-es' to specify reduction, '-ez' to specify summing, and '-ea' to activate antialiasing. With antialiasing, the default scaling is by 100 and the output is to 16-bit integers by default. For example, to produce normalized, defect-corrected frames based on sums of 8 frames and reduced by 2 with antialiasing:
clip norm -es 1 -ez 8 -ea 1 fileWithFrames.eer gainReferenceFile.tiff normalizedFrames.mrc
Where 'fileWithFrames.eer' is the frame file, 'gainReferenceFile.tif' is the gain reference, and 'normalizedFrames.mrc' is the output. Without antialiasing, you would need to add '-n scalingFactor' to scale by other than 16 and '-m 1' to get integer output. In either case, '-m 2' can be used to get 32-bit floating point output and '-m 16' to get 16-bit floats.