Image Acquisition from CCD Cameras and Direct Electron Detectors

Sections of interest below:

The first part of this page explains some features of CCD cameras. Direct electron detectors have significantly different behavior and some of their features are explained in Direct Electron Detectors, Especially the Gatan K2/K3.  Newer CMOS-based cameras also have different behavior, so this first section mostly applies to traditional, high-quality CCD cameras.  The principal differences for CMOS cameras are that readout happens continuously during the exposure instead of at the end (thus readout time is not an issue), and that binning makes little difference in acquisition time except for the time savings of moving smaller amounts of data around in software.

An image is captured from the CCD camera with a sequence of 3 steps: clearing the CCD chip, exposing the camera to the beam and integrating charges for some period of time, then reading the image off of the chip. The readout is done line by line, so it is often very time-consuming. The clearing time is also substantial because the chip is cleared by shifting accumulated charges to one edge. Clearing times are about 0.7 to 0.9 second for 2K Megascan or 4K single-port readout Ultrascan cameras, 0.25 second for a 4-port readout Ultrascan, or very short for some current 1K cameras. It is important not to have a beam on the camera during the clearing time, because then charges will build up as they are being removed, adding a ramp of intensity to the image that is subsequently acquired.

Many plastic-embedded specimens show a transient shift of the image when the beam is turned on to take an exposure; this drift produces a smeared image. To take good images, you need to use drift settling, which provides an initial exposure of the specimen to the beam just before the image is acquired. This initial shift is worse on slot grids than on mesh grids and tends to be worse at high tilts. It seems to depend on the total beam hitting the sample rather than the brightness of the beam in the area of interest. This means you should work with the largest spot size that gives you enough beam.

Exposing the specimen to the beam before the exposure without exposing the camera during its clearing time requires the use of at least two shutters, one below the specimen and one above. The traces below show how DigitalMicrograph manages the two shutters in standard shuttering mode with the alternate (beam) shutter normally closed. A trace goes high to indicate when a shutter is open or beam is on the camera. Although this shuttering provides some pre-exposure, there are two problems here. First, the amount of pre-exposure is fixed at the clear time, which could be too short or longer than needed. Second, the beam is left on the specimen during camera readout. (Note that this occurs only when using the shuttering mode that provides pre-exposure and does not happen if you are just using the beam shutter for low-dose exposures.)

DigitalMicrograph does provide a way to increase the drift settling, but this has problems of its own. As shown in the following traces, the extra settling is implemented by keeping the beam on the CCD during the clear time. This introduces an intensity ramp in the image, which is eliminated by having the beam on for the equivalent time during the 'dark' reference. These ramps use up some of the dynamic range of the camera, but worse than that, the dark reference contains image features and will become invalid when exposure or specimen location changes.

To deal with these problems, SerialEM has implemented a shuttering method using the scripting beam-blanker, which operates separately from the camera shuttering. As indicated in the two sets of traces below, this extra shuttering can provide a flexible range of drift settling times and also eliminates the many seconds of exposure during camera readout.

Low-noise CCD cameras are inherently slow, so you should be aware of what governs image capture time and how to speed it up. The total capture time is the sum of: any drift settling time above the clearing time, the clearing time, the actual exposure time, time to read out the image, and time to process and display it. Drift settling of less than the clearing time is subsumed in the clearing time. Processing time is minimal, so you should always use gain normalization to remove intensity variations due to imperfections in the phosphor screen. Read-out time is proportional to the number of pixels of data being read out of the CCD chip. It is about 12 seconds for the 2K x 2K pixels through a single readout port, so binning, which is done on the chip before read-out, can speed up the read-out tremendously. The read-out time is actually dominated by the number of lines being read, so for a given number of pixels being read, the read-out is fastest if those pixels are in the fewest number of lines. This is the reason for the Wide Quarter and Wide Half areas provided by the buttons in the Camera Setup dialog. Their readouts are only somewhat longer than the readout for square images that are half as big, not twice as long.

Direct Electron Detectors, Especially the Gatan K2/K3

The first part of this section describes features common to direct detector cameras, but most of it is specific to the Gatan K2 or K3 camera.  If you are familiar with the K2, see the section Principal Differences between K2 and K3.

Binning and summing of signals.  Binning and integration over time behave quite differently for direct detectors and other CMOS-based cameras than for CCD cameras.  With a high-end cooled CCD, an image is accumulated on the chip for the whole exposure time, and pixels are binned before readout, so each binned pixel of the final image is read out once.  This means that a very weak beam can be integrated for a long time and binned to make the readout noise small compared to the signal.  With the CMOS-based cameras, the amount of charge that can be accumulated is limited so the chip has to be read out frequently, and readout happens at each pixel on the chip, so one unit of read-out noise is added to each pixel during each frame.  Integrating frames over time and binning the pixels after readout will average out this noise but cannot reduce the amount of noise that has to get averaged, unlike with a CCD.  With the K2 and K3 cameras, the readout is particularly fast to allow counting mode to be used, but the result is that linear mode images are quite impaired by noise at low dose rates.

Binning and anti-aliased image reduction with the K2 and K3 Cameras.  The K2 camera is intrinsically a 3.8K x 3.7K camera but can produce 7.6K x 7.4K images in Super-Resolution mode.  Similarly, the K3 camera has 5.8K x 4.1K sensor but its electron counting mode always produces super-resolution 11.5K x 8.2K images. SerialEM resolves this by treating the cameras internally as 7.6K and 11.5K in all modes of operation, but maintaining binning values that are twice those actually applied to acquire images.  However, the user interface presents each as a camera with the size of the sensor and with the binning values that one is accustomed to seeing, with one exception: a 7.6K super-resolution  image is referred to as having binning 0.5.

With a K2, Super-resolution mode has an advantage over counting mode only with no binning in DM, which yields a 7.6K image.  For example, a super-resolution image taken with binning by 2 in DM will produce a 3.8K image that is identical to the 3.8K image that could have been obtained with counting mode and no binning.  To provide a useful alternative, SerialEM uses image reduction by antialias filtering, to remove the high-frequency noise that would be introduced by aliasing when images are binned.  This method is used for super-resolution mode whenever a binning above 0.5 is selected.  With a K3, antialiased reduction is used for all images returned to SerialEM with binning above 0.5, except in continuous mode.

Antialiased reduction has a measurable advantage over binning with electron counting images.  If pairs of super-resolution images are taken at a variety of doses and analyzed by Fourier ring correlation, the FRC past half-Nyquist is about 35% higher with antialiased reduction by 2 than with binning by 2, and about 50% higher with reduction by 4 rather than with binning by 4.

Antialiased reduction is used by default instead of true binning for all "binned" images in K2 counting and super-resolution modes and for K3 counting images, and the reduction is done by the SerialEMCCD plugin on the K2 or K3 computer, where it can run quickly.  At the bottom of the Camera Setup dialog, there is a checkbox to turn off this behavior, followed by an indicator in bold about whether a reduction will use Anti-aliasing or Binning.

Scaling of electron counts.  In counting or super-resolution mode with a K2, DM produces images with the number of electrons, which is very different from the number of digitized counts produced in linear mode.  Since many parts of SerialEM rely on there being a fixed relationship between beam intensity and image counts, the plugin scales electron counts up by a factor close to the number of counts per electron in linear mode, so that all modes produce approximately the same number of counts per physical pixel for a given exposure time and beam intensity.

In contrast, the K3 in electron counting mode produces 32 counts per detected electron, while the linear mode in DM produces huge numbers of counts.  In this case, SerialEM scales linear mode to match the counting mode output, using that a factor that has to be set in the SerialEM properties file.

Dose rate for effective electron counting.  Electron counting is effective only over a limited range of dose rates.  For the K2, it is best to keep the dose rate between 2 and 10 electrons per physical pixel per second; at some point (roughly around 20 e/physical pixel/sec) counting becomes less efficient than integrating in linear mode.  The K3 has 3.75 times faster readout rate, which means all these dose rates would be that much higher, so the best operating range would be ~7 to 40 e/physical pixel/sec.  There is a dose rate readout of the actual mean e/physical pixel/sec in an image at the upper left of the display window, plus you can use Min/Max/Mean in the Process menu to obtain the dose rate.  There is also a dose rate estimate in the Camera Setup dialog, which predicts the dose rate under the current conditions based on the change in beam intensity since the last acquired image.

For the K2, the number of electron counts produced per true electron drops by a factor of ~0.97 at 5 e/physical pixel/sec, ~0.94 at 7.5 e/physical pixel/sec, ~0.87 at 10 e/physical pixel/sec, and ~0.6 at 20 e/physical pixel/sec.  Unfortunately, even at low dose rates, the conversion efficiency is apparently only 0.87 counts per true electron (Li et al., 2013).  Similar values apply for the K3 at the correspondingly higher dose rates.  As of SerialEM 3.6, the dose rate outputs are all adjusted to compensate for both the nonlinearity and the loss of signal; they should match the dose rate readout in DigitalMicrograph.

Dose calibration.  Because of the adjustments just described, dose calibration with a counting mode image should be accurate for a fairly wide dose rate range.  Dose calibration with linear mode images from a K2 would also be accurate, as long as the property for counts per electron is set correctly.  There should be no need to use linear mode images for this purpose with a K3 camera.  Measuring Camera Gain describes a procedure for setting the counts per electron and for setting a separate property to equalize the counts between linear and counting mode images at a particular dose rate for K2 camera.

Camera timing. Camera timing is also unlike that of a CCD and is mostly independent of binning.  For the K2, linear and counting mode shots take ~3 sec plus the exposure time.  A super-resolution shot takes ~6 sec plus the exposure time, plus the time required to return the 7.6K image to SerialEM, which is generally at least 1.3 sec over a fast gigabit network.  For a K3, the basic time is shorter (~2 sec) and is no penalty for getting the super-resolution images.

Dark and gain correction.  Because of the complexity of the dark and gain corrections in DM, all gain normalization is done in DM instead of SerialEM, and it is not possible to take a gain reference in SerialEM.  For the K2, DM will keep one dark reference for each binning and camera mode combination, so when a new exposure time is used, DM has to prepare or obtain a new dark reference for that mode and binning.  For linear mode, this takes ~2 sec plus the exposure time, whereas for counting and super-resolution mode it seems to take just ~0.6 sec regardless of exposure time.  Thus, when setting up your different camera parameter sets for a tilt series, it is not very important to avoid lots of dark reference events if you are not using linear mode at all (which is advisable for low dose work).  However, for higher dose work where linear mode is used, try to set up Record, Focus, and Trial so that they have different binnings to avoid frequent dark references when going between these image types.  For the K3, a dark reference is apparently taken for each shot and these considerations do not apply.

Limitations in Dose Fractionation mode.  Before GMS 2.3.1, Dose Fractionation mode could produce only unbinned, full frame images, 3.8K in linear or counting mode and 7.6K in super-resolution mode.  Although these limitations were removed, SerialEM was never modified to take advantage of the change.  For the summed image returned to SerialEM, antialiased image reduction is applied to achieve whatever reduction is selected (i.e., with binning higher than 1 for linear or counting mode, or binning higher than 0.5 for super-resolution mode).  For the K3, SerialEM enforces the same restrictions on images in Dose Fractionation mode, with one exception: it allows electron counting frames being saved or aligned to be taken with fast hardware binning by 2, thus providing the same kind of frames as counting mode gives on the K2.  When unnormalized frames are saved, the plugin creates and saves a binned reference appropriate for correcting these frames from the Dm super-resolution reference.

Saving frames from the plugin.  The ability to save frames in Dose Fractionation mode is implemented by having SerialEM's plugin to DM save frames directly rather than passing them back to SerialEM.  Only the final summed image produced by DM is passed back.  The frames for one image acquisition are saved either in a single MRC or TIFF stack or as a numbered set of single-image files inside of a newly created directory. The stack or directory is given a unique name based on the date and time or on choices in the  K2 Frame File Options dialog; this name will be printed to the log window along with the number of frames saved. This information is also saved in a '.mdoc' metadata file if the summed image is saved, or in a '.mdoc' file opened with Open .mdoc for Frames in the File menu.  The frames can be saved only on the computer hosting the K2 or K3 camera, or in file systems accessible to that computer.

The scaling and mode of the saved frames depends on whether gain normalization is selected in SerialEM, which invokes software gain correction in DM for the individual frames, and also differs between K2 and K3 cameras.  For a K2 camera, with the gain correction (which does some useful scaling to remove artifacts), DM passes the plugin floating point frames with fractional values in both modes.  The plugin scales a super-resolution mode frame by 16 and stores it as bytes, or a counting mode frame by the regular scaling applied to summed images and stores it as integers.  The title in the header of the MRC file includes the scaling applied to the images. Without the gain correction, DM passes the plugin frames with electron counts in integer or byte format in counting or super-resolution mode, respectively, and the plugin saves these data without scaling.  The state of the Pack unnormalized data option in the Frame File Options dialog determines whether these raw counts are saved in a packed form.  If this option is selected, the plugin truncates the integers in counting mode as bytes and packs the bytes in super-resolution mode so that two 4-bit numbers are stored in each byte.  Otherwise, the plugin saves the integers or bytes that it received, respectively. 

For K3 cameras, normalized data are passed to the plugin as bytes with the fixed scaling of 32, so they are ready to save as bytes.  This is still the case if the frames are taken with binning by 2 (to mimic K2 counting mode).  Unnormalized frames are always bytes as well, so they can be saved as is or packed two per byte.

By default, the packed 4-bit data files will look like ordinary byte files, but with half the true size in X; thus other programs will be able to open them just like any other byte file but they will appear squeezed by a factor of 2 in X.  For MRC files, the Frame File Options dialog allows you to save 4-bit data as mode 101, which IMOD and MotionCor2 support but other programs might not.  That dialog also allows you to save the data as TIFF files with LZW or ZIP compression; LZW is best for unnormalized data and is also much faster.  If you save data as compressed TIFF files, the program 'tif2mrc' in IMOD can be used to convert each file to MRC format before processing it; or, in fact, any IMOD program can be run directly on the TIFF files.

To take advantage of this option, you can select unnormalized instead of gain normalized images in the Camera Setup dialog.  A more convenient approach is to leave the selection as normalized in that dialog, but turn on Save unnormalized frames even if Gain Normalized is selected in the  Frame File Options dialog.  There is actually no dark-subtraction involved, and these two selections now produce the same result.  When frames are saved unnormalized, the plugin will return a gain-normalized sum of the frames even though the normalization is not selected, so that this image will not have artifacts that could impair cross-correlation alignment.

Signal loss from saving normalized frames as integers.  With the K2, originally there was some concern that rounding the floating point frame values to integers will lose signal and reduce the efficiency of the camera.  To explore this point, the image formation was modeled by 1) generating a Poisson deviate to represent a true electron count as the input signal; 2) adding a Gaussian deviate with a sigma of 0.05 times the electron count, which reproduces the histogram of spreads around integer counts in a normalized frame; 3) setting the output to zero for 0.13 of trials, to reproduce the typical signal loss in electron counting; 4) multiplying by 16, the smallest multiplier applied when saving frames; 5) taking this value as a floating point output, or rounding to the nearest integer for integer output.  DQE was computed as the square of the ratio of the output SNR to the input SNR.  The loss in DQE from rounding to integers depends on the number of electrons per saved pixel (not per physical pixel for super-resolution mode):

   Electrons per pixel    Loss in DQE
         0.1                         0.0005
         0.5                         0.0004
         1.0                         0.0003
         2.0                         0.0002

The reasons the loss is so small are that 1) the vast majority of normalized pixel values are close to integer values anyway; 2) the SNR is so low in the images.  To put this loss in perspective, with a loss of 0.0005, you would need 10005 particles instead of 10000 particles to reach a certain resolution.

Gain-normalizing frames during post-processing.  If you choose to save frames without software gain normalizing, the plugin will make sure that there is a valid copy of the appropriate gain reference in the folder where the frames are saved, provided that the plugin can find the gain reference.  (This used to be done by specifying the full paths to the gain references in the SerialEM property file, but now a correct entry for the camera property DMGainReferenceName is sufficient.)   In addition, since there is no defect correction in these images with GMS 2.3.1 or higher, the plugin will save a file 'defects...txt' describing the defects that need to be corrected in post-processing. You can convert the gain reference to an MRC file with the IMOD program 'dm2mrc' or, again, use it directly in any program.  The gain reference may need to be rotated and/or flipped by the same operations described in the camera configuration in DM, unless the frames are saved without rotation and flipping as provided by an option in the Frame File Options dialog.

The header of the frame stacks will have a title line with the text 'r/f' followed by a value corresponding to a RotationAndFlip entry in the SerialEM properties file.  This value is the rotation counterclockwise divided by 90 (i.e., 1, 2, or 3 for angles of 90, 180 or 270) plus 4 for a flip around the Y axis before the rotation.  This operation is what needs to be applied to the gain reference.  If the frames were saved without rotation and flip, the title line will have the text 'r/f 0 need' followed by the rotation/flip value that was not applied to the frames. The gain reference does not need to be rotated, but the final aligned sums do need to be rotated by this operation to match images returned to SerialEM.  With 'alignframes' in IMOD 4.12.18 or higher, the option '-rotation -2' or '-rfsum -1' can be used to apply the operation specified in the 'need' value.

There are several ways to handle the rotation of the gain reference:

  1. With 'alignframes' in IMOD, there is no need to normalize the data separately; the gain reference and defect file can be given to the program and the option '-rotation -1' can be used to read the r/f value from the frame stack header.
  2. With IMOD 4.9.2 or higher, or 4.10.1 or higher, the 'clip' program can do the rotation with the universal 'norm' command given below in the section starting 'In IMOD version 4.8.41 or higher.'  It must be given the '-R' option with either the SerialEM rotation/flip value applied to the frames, or -1 to read the r/f value from the frame stack header.
  3. In the general case, use the IMOD commands
        clip flipy reference.dm4 flipped.mrc
    first for a r/f value of 4-7, followed by
        newstack -near -rot angle flipped.mrc rotFlipped.mrc

Once you have the reference in the right orientation, you can use the program 'clip' in IMOD to apply gain normalization (and defect correction with version 4.8.6 or higher).  In the following, 'scalingFactor' is the regular scaling factor applied to summed images, 'fileWithFrames' is the data file to normalize, 'gainReference.mrc' is the reoriented gain reference, and 'normalizedFrames.mrc' is the desired output file. The alternatives for GMS 2.3.0 or lower are:

To apply defect correction to files from GMS 2.3.1 or higher, add '-D defects...txt' before 'fileWithFrames' in the appropriate command, where 'defects...txt' is the file saved by the plugin.

In IMOD version 4.8.41 or higher, all programs can read 4-bit files directly.  The 'clip unpack' command has thus been changed so that it can be used for normalizing any kind of data, and it can also be invoked as either 'clip unpack' or 'clip norm'.  A command that works for all of the above cases is
    clip norm -n scalingFactor  -m 1  fileWithFrames  gainReference.mrc normalizedFrames.mrc
where the default scaling factor is 16, extreme values can be removed with '-l' and '-h' options, and '-D defects...txt' would be added for files from GMS 2.3.1 or higher.  With IMOD 4.9.2/4.10.1 or higher, you can add add '-R -1' and use the DM reference directly instead of a rotated reference.

For K3 frames, you should specify a scaling factor of 32.

In all of these situations, conversion to floating point can be used instead of, or in addition to, the scaling to guarantee absolutely no information loss from rounding to integers.  To convert to float, use "-m 2" instead of "-m 1" or add "-m 2" to a command when the data are already integers.  The extra storage required will not be an issue if the normalized data are only being used temporarily for alignment and averaging of frames.

Finally, if you want to run MotionCor2 directly on the unnormalized data, you should give it a defect map file as well as the gain reference file.  You can make a defect map from the text file with 'clip defect' in IMOD 4.10.7 or higher::
    clip defect -D defects...txt  fileWithFrames  defects...mrc
where the fileWithFrames is used only to set the size of the output and can be any file of the right X and Y size.  To make a compressed TIFF file, which will be much smaller, use:
    clip defect -D defects...txt  -f tif  fileWithFrames  defects...tif

Drift settling  Drift settling is implemented not by having two shutters but by having a longer exposure in which readout frames at the beginning are discarded.  This is known to work correctly when there is only one shutter connected.  If there are two shutters, it may work correctly only if the second shuttering option is selected, as was found to be the case for the OneView.

Constraints on timing.  There are several constraints on timing that are enforced both for the entries in the Camera Setup dialog

  1. All times (exposure, drift settling, frame time in Dose Fractionation mode) must be multiples of the readout interval, 2.5 ms for K2 and 0.666 ms for K3.
  2. When not doing Dose Fractionation, exposure time for K2 must be a multiple of 0.1 sec in linear or counting mode and a multiple of 0.5 sec in super-resolution mode (yes, 0.5 sec!).  For K3, exposure time must be a multiple of 0.013 sec in linear mode  or 0.011 sec in electron counting mode.
  3. In Dose Fractionation mode, exposure time must be a multiple of the frame time, and it must be at least 0.025 sec for K2 or 0.013 sec for K3.

It is possible to overcome the constraints on exposure time in K2 counting and super-resolution modes by turning on Dose Fractionation mode and setting the frame time equal to the exposure time, with no aligning or saving selected.  Doing so will cost an extra 4 sec in counting mode or 5.5 sec in super-resolution mode.  For K3, the extra time will be much less but there is much less need to do so.  This equality between frame and exposure time will be retained when you use either of the two methods of changing exposure time during a tilt series, i.e., changing exposure time instead of intensity or changing exposure time at scheduled tilt angles.  See the description of these options in Beam Intensity Control and Tilt Series Scheduled Changes Dialog for more details.

Principal Differences between K2 and K3

  1. The K3 has been optimized to produce super-resolution images, does so intinsically when electron counting, and does not have a distinction between counting and super-resolution modes.  The user interface in DM simply offers linear and counting modes and adopts the SerialEM convention of using binning 0.5 to refer to super-resolution images.  To maintain consistency with DM, SerialEM also provides just two modes and refers to them and linear and counting.
  2. The K3 has only two gain references, linear and super-resolution.
  3. The K3 does allow fast binning by 2 for electron counting images, thus replicating the counting mode of the K2 camera, but this capability was added late in the development of SerialEM support for K3.  I decided not to recreate a distinction between counting and super-resolution modes, which would have been largely artificial.  Instead, SerialEM provides an option to acquire frames with 'hardware' binning by 2 for either saving or aligning.
  4. SerialEM includes another option for getting frames the same size as the sensor: saving super-resolution frames with antialiased reduction by 2, which takes about 0.1 sec per frame.  This option is available for K2 as well, but there it costs even more time because super-resolution mode itself takes longer than counting.
  5. K3 electron counting images are scaled to 32 counts per detected electron and gain normalization results in byte values.  Each super-resolution pixel can thus hold up to 7 electrons.
  6. The readout rate of the K3 is 3.75 times higher than the K2 and it can thus count electrons arriving at dose rates per pixel that are higher by this amount.  The optimal dose rate is around 30 electrons per physical pixel per second instead of 8.  (Sensor pixels are referred to as physical pixels.)
  7. Linear mode is even less useful for the K3 than for the K2 because of this higher dose rate range for electron counting.  It may still be useful for continuous mode.  It consists of very high counts in DM; these get scaled to match the 32 counts per electron of counting mode.
  8. The processors in the K3 computer are much less occupied during the time when it is feeding frames to SerialEM's plugin to DM, so parallelization of various frame processing steps in the plugin is more effective.  The plugin has thus gotten faster just as the camera has.

Remaining Problems with the K3

This list will be continually updated and eventually removed as Gatan either solves the problem or clarifies the situation.  (Last update January 29, 2021)

  1. Linear mode images during continuous acquisition have an incorrect offset which would give a misleading dose rate estimate.  This is currently adjusted for in SerialEM.  If the problem is ever resolved, there will probably be a property setting to prevent the adjustment.
  2. Hardware binning in counting mode is constrained to producing normalized data in bytes.  This could potentially saturate frames taken with the option for hardware binning by 2 with long frame times.  Also, continuous mode for counting images with a binning selection of 1 or higher uses hardware binning by 2 and can saturate with high dose rates or with longer exposures, and hardware binning by 4 is simply not used for these images because it saturates so easily.
  3. SerialEM has no control over the alignment of frames by MotionCor2 available in DM as an option, and apparently cannot trigger this alignment or access the result.