File Formats

Image Metadata Files

SerialEM stores metadata about images in text files in the IMOD "autodoc" format.  This format consists of keyword-value pairs organized into blocks called sections.  These sections can be of different types with different kinds of data in them, although SerialEM's files use essentially only one kind of section.  A section begins with a bracketed key-value pair:
      [sectionType = name]
where the section "name" or value will typically be unique but is not required to be so by the IMOD autodoc reader.  Lines below a section header of the form
      key = value
provide data associated with that section.  In addition, key-value pairs can occur at the beginning of the file, before any section header, and these are referred to as global values.

Although all values are stored as text strings in the autodoc file, software that expects information of a particular kind will often convert these strings to or from one or more floating point or integer values.  The autodoc module in the IMOD library libcfshr contains functions for storing or retrieving one, two, three, or many floating point or integer values in specified sections of the file.  The autodoc format was chosen for SerialEM because of the ready availability of this library, the convenience of using such functions, and compatibility with the autodoc-reading capability in IMOD.

There are two kinds of image-associated autodoc files produced by SerialEM:

1) The ".mdoc" file provides data about an MRC file and has the same name as the image file, with the additional extension ".mdoc".  In these files, the section type is "ZValue" and the name for each section is the Z value of the image in the file, numbered from 0.

2) The ".idoc" file provides information that allows a series of single-image TIFF files to be treated interchangeably with MRC files.  The section type here is "Image" and the name for each section is the name of the TIFF file.

Global data replicate some of the information that is in an MRC file header.  In addition, the titles for an MRC file are stored in section headers of the type "T".

Global Data
DataMode MRC file mode
ImageSize X and Y size of images
Montage 1 if the file is a montage
ImageSeries 1 for an .idoc file describing a series of single-image files
ImageFile In a .mdoc file, contains the name of the image file
PixelSpacing Pixel spacing in Angstroms that would be in MRC header
Section Data
TiltAngle Tilt angle in degrees
PieceCoordinates Pixel coordinates in X and Y and section Z value for montage piece
StagePosition X and Y stage position in microns
StageZ Z stage position in microns
Magnification "Film" magnification value
MagIndex Magnification index
Intensity Raw intensity value between 0 and 1
SuperMontCoords X and Y pixel coordinates of frame in a supermontage
PixelSpacing Pixel spacing in Angstroms for individual image
ExposureDose Dose on specimen during camera exposure
SpotSize Microscope spot size
Defocus Relative defocus readout from microscope (microns)
TargetDefocus Current target for autofocus (microns)
ImageShift X and Y image shift in basic units (close to microns)
RotationAngle Rotation of image from having tilt axis along X axis (CCW +)
ExposureTime Image exposure time
Binning Image binning on the camera
CameraIndex Index of the CameraProperties section for the camera used
DividedBy2 1 if image was divided by 2
MinMaxMean Minimum, maximum, and mean value for this image
PriorRecordDose For an image in a tilt series taken in Low Dose mode, the cumulative dose in the Record area prior to this image, including Record and Preview images and View images taken in tasks
NavigatorLabel Label of Navigator item, added if Acquire at Items is being run and a tilt series is not
XedgeDxy Edge displacement in X and Y for montage piece to the right of this piece
YedgeDxy Edge displacement in X and Y for piece above this piece
XedgeDxyVS Edge displacement for piece to right, computed with very sloppy option
YedgeDxyVS Edge displacement for piece above, computed with very sloppy option
StageOffsets When aligning montage pieces with image shift, this is the effective stage position (actual position plus the stage equivalent to the image shift adjustment) minus the nominal stage position of the piece.
AlignedPieceCoords Piece coordinates adjusted by the solved shifts for each piece when pieces are aligned in overview with 'Very sloppy' option off.
AlignedPieceCoordsVS Piece coordinates adjusted by the solved shifts for each piece when pieces are aligned in overview with 'Very sloppy' option on.
SubFramePath Directory or file in which subframes of exposure were stored
NumSubFrames Number of subframes stored
FrameDosesAndNumbers Dose per frame in electrons per square Angstrom followed by number of frames at that dose; variable-sized frame sums will have multiple pairs of such values
DateTime Time and date of image acquisition
Direct Electron Specific Data
DE12-ServerSoftwareVersion Server software version
DE12-PreexposureTime(s) Pre-exposure time
DE12-TotalNumberOfFrames Total number of frames in acquisition
DE12-FramesPerSecond Frame rate in frames per second
DE12-CameraPosition Whether camera was retracted or inserted
DE12-ProtectionCoverMode Whether protection cover is kept open or open and closed for this shot
DE12-ProtectionCoverOpenDelay(ms) Delay after opening prtection cover
DE12-TemperatureDetector(C) Detector temperature
DE12-FaradayPlatePeakReading(pA/cm2) Current reading from Faraday plate during exposure
DE12-SensorModuleSerialNumber Serial number of sensor
DE12-SensorReadoutDelay(ms) Delay before starting to read out sensor during exposure
DE12-IgnoredFramesInSummedImage Number of frames ignored in a summed image

 

If an ".idoc" file is being constructed for a set of TIFF files, the essential elements are the "DataMode", "ImageSize", and "ImageSeries" entries, as well as a section header for each image.  The global "PixelSpacing" may be helpful in some contexts.  For a montage, the "Montage" entry must be present with the value 1, and there must be a "PieceCoordinates" entry in each section.  The X and Y values must fall on a regularly spaced grid, so that the coordinates can be analyzed to deduce the spacing and overlap between pieces unambiguously.  Not all pieces in a rectangle need to be present, and there need not be pieces at 0.  Pixel coordinates are in a right-handed system despite the ubiquitous inversion of images in Y.

If the alignment between montage pieces is known, this information can be placed in both the XedgeDxy/YedgeDxy and the XedgeDxyVS/YedgeDxyVS entries, so that the alignment can be used by SerialEM regardless of whether the Very Sloppy option is checked.  The entries are the amount that an "upper" piece, i.e., one to the right (for X) or above (for Y), is displaced from being aligned in the overlap zone.  This is the negative of the amount that the upper piece needs to be shifted to align the two pieces in the overlap zone.  The program uses the IMOD library routine findPieceShifts to solve for the best shifts to be applied to each piece to bring all of them into best alignment.  There is a fixed (but easily raised) limit to the size of montage for which this solution will be found, based on the nominal number of pieces in X and Y rather than the actual number.

Navigator Files

In SerialEM 3.6, the Navigator output was switched from one tab-delimited line per item to an autodoc format, with an option to output as XML instead.  XML or autodoc files can be read back in, as well as the tab-delimited files from earlier versions.  There is one line of global data,
  AdocVersion = x.xx
then every item is in a section of type 'Item' and with its label string as the value.  The following table lists all of the keys that can occur in a section, the kind of value that is expected, and whether it is required, or if it is not required, what default value is set.

Key Type of value Default value, or Required CMapDrawItem member(s) Description
Color Integer Req mColor Color index (0-5)
StageXYZ Three Floats Req  (unless external) mStageX/Y/Z Stage position adjusted for current mag
NumPts Integer Req mNumPoints Number of points
Corner Integer 0 mCorner Flag for corner point
Draw Integer 1 mDraw Flag to draw
RegPt Integer 0 mRegPoint Registration point number
Regis Integer Req mRegistration Registration at which item exists
Type Integer Req mType Type: 0-2 for point, polygon, map
Note String blank mNote Note string
GroupID Integer 0 mGroupID ID of group thatitem belongs to
PolyID Integer 0 mPolygonID ID of polygon used to define supermon
Imported Integer 0 mImported Indicator of an imported map or point drawn on one
SuperMontXY Two Integers -1,-1 SuperMontX/Y Camera coordinate of a montage in supermontage
OrigReg Integer Regis mOriginalReg Original registration (default is Registration)
DrawnID Integer 0 mDrawnOnMapID ID of map point/polygon was drawn on
BklshXY Two Floats 0,0 mBacklashX/Y Backlash when a montage was taken or associated with item
SamePosId Integer 0 mAtSamePosID Items with this matching were taken at same raw stage position
RawStageXY Two Floats -10000,-10000 mRawStageX/Y Raw stage position before adjustments
Acquire Integer 0 mAcquire Flag for acquiring
PieceOn Integer -1 mPieceDrawnOn Index of montage piece item was drawn on (piece_#_in_X * #_of_pieces_in_Y + piece_#_in_Y)
MapFile String Req if Map mMapFile Full path of map file
MapID Integer Req if Map mMapID Unique ID (all items get one now)
MapMontage Integer Req if Map mMapMontage Flag that map is a montage
MapSection Integer Req if Map mMapSection Section number in file
MapBinning Integer Req if Map mMapBinning Binning at which map was taken or of initial map image
MapMagInd Integer Req if Map mMapMagInd Magnification index
MapCamera Integer Req if Map mMapCamera Camera index
MapScaleMat Four Floats Req if Map mMapScaleMat Scale matrix for drawing
MapWidthHeight Two Integers Req if Map mMapWidth/Height Size of image at which scale matrix was defined
MapMinMaxScale Two Floats 0,0 mMapMin/MaxScale Min and max scale values of image when map was defined
MapFramesXY Two Integers 0,0 mMapFramesX/Y Number of montage frames when acquired
MontBinning Integer 0 mMontBinning Actual binning of montage used to make the map
MapExposure Float 0. mMapExposure Exposure time for original map images
MapSettling Float 0. mMapSettling Drift settling
ShutterMode Integer -1 mShutterMode Shutter mode
MapSpotSize Integer 0 mMapSpotSize Read mode for K2 camera
MapIntensity Double 0 mMapIntensity Intensity at which map was taken
MapSlitIn Integer 0 mMapSlitIn Filter slit state when map was taken
MapSlitWidth Float -1. mMapSlitWidth Filter slit width when map was taken
RotOnLoad Integer 0 mRotOnLoad Flag to rotate when load
RealignedID Integer 0 mRealignedID ID of lower mag map with nearby realign error
RealignErrXY Two Floats 0,0 mRealignErrX/Y Final stage error of that realign operation
LocalErrXY Two Floats 0,0 mLocalRealiErrX/Y Error in second round of realign operation
RealignReg Integer 0 mRealignReg Original registration of that realign
ImageType Integer 0 mImageType Map file type 0-3 for MRC, IMOD, TIFF, IDOC
MontUseStage Integer -1 mMontUseStage 1 if montage was taken with stage, 0 if not, -1 if unknown
DefocusOffset Float 0. mDefocusOffset Defocus offset for map based on View images
K2ReadMode Integer 0 mK2ReadMode Read mode for K2 camera
NetViewShiftXY Two Floats 0,0 mNetViewShiftX/Y Net IS offset for a View image map
MapAlpha Integer -999 mMapAlpha JEOL alpha value
ViewBeamShiftXY Two Floats 0,0 mViewBeamShiftX/Y Incremental beam shift to apply for a View map
ViewBeamTiltXY Two Floats 0,0 mViewBeamTiltX/Y Incremental beam tilt to apply for a View map
MapProbeMode Integer -1 mMapProbeMode Probe mode (0 for nano, 1 for micro)
MapLDConSet Integer -1 mMapLowDoseConSet Set # (0-4) used for map taken in low dose
MapTiltAngle Float -10000. mMapTiltAngle Tilt angle at which map was taken
PtsX NumPts Floats Req mPtX X stage cordinates of points
PtsY NumPts Floats Req mPtY Y stage cordinates of points

Importing Externally Defined Navigator Items

It is possible to add items to a Navigator file whose positions are specified by image coordinates in a map and have SerialEM do the conversion to stage coordinates.  Instead of the StageXYZ entry, the item can have one of these entries:

The following points apply to externally defined items:

Detailed error reporting is available for items that have been read in successfully from the Navigator file.  A summary of the types of errors when processing the items is given in the log, and if Debug Output is set to 'n', then the type of error will be reported for each item, identified by its label.  However, there is no detail when the item is initially rejected either because it lacked a required entry or because there was an error (such as too few values on a line) reading data from the autodoc.