File Formats and Other Documentation

Image Metadata Files
Navigator Files
Importing Externally Defined Navigator Items
Settings and Calibration File Entries
Other Notes

Image Metadata Files

SerialEM stores metadata about images in text files in the IMOD "autodoc" format.  This format consists of keyword-value pairs organized into blocks called sections.  These sections can be of different types with different kinds of data in them, although SerialEM's files use essentially only one kind of section.  A section begins with a bracketed key-value pair:
      [sectionType = name]
where the section "name" or value will typically be unique but is not required to be so by the IMOD autodoc reader.  Lines below a section header of the form
      key = value
provide data associated with that section.  In addition, key-value pairs can occur at the beginning of the file, before any section header, and these are referred to as global values.

Although all values are stored as text strings in the autodoc file, software that expects information of a particular kind will often convert these strings to or from one or more floating point or integer values.  The autodoc module in the IMOD library libcfshr contains functions for storing or retrieving one, two, three, or many floating point or integer values in specified sections of the file.  The autodoc format was chosen for SerialEM because of the ready availability of this library, the convenience of using such functions, and compatibility with the autodoc-reading capability in IMOD.  IMOD also has a simple Python module, autodoc.py, with a function to take a set of autodoc lines and produce a top-level dictionary of the different types of sections, keyed by 'sectionType', with each entry containing an array of dictionaries, one for each section.  Other functions will return a list of integer or floating point values from the value in a key-value pair.

There are two kinds of image-associated autodoc files produced by SerialEM:

1) The ".mdoc" file provides data about an MRC file and has the same name as the image file, with the additional extension ".mdoc".  In these files, the section type is "ZValue" and the name for each section is the Z value of the image in the file, numbered from 0.

2) The ".idoc" file provides information that allows a series of single-image TIFF files to be treated interchangeably with MRC files.  The section type here is "Image" and the name for each section is the name of the TIFF file.

Global data replicate some of the information that is in an MRC file header.  In addition, the titles for an MRC file are stored in section headers of the type "T".

It is important to realize that no steps in IMOD tilt series processing rely on information directly from an .mdoc file or require the presence of one.  An .mdoc file is used for dose weighting, but only if the user specifies that as the source of dose information.  It can be used with Alignframes to create a tilt series from frame files.  Extracttilts and Extractpieces will fall back to looking for an .mdoc file if tilt angle or piece list values, respectively, are not in the image file header.

 The IMOD library used by SerialEM uses the same internal autodoc structure to access the metadata in HDF files.  It is thus simple to extract all of these metadata information into a text file in the autodoc format, with a command like:

    extracttilts -attr filename.hdf  filename.hdf.mdoc 

If an ".idoc" file is being constructed for a set of TIFF files, the essential elements are the "DataMode", "ImageSize", and "ImageSeries" entries, as well as a section header for each image.  The global "PixelSpacing" may be helpful in some contexts.  For a montage, the "Montage" entry must be present with the value 1, and there must be a "PieceCoordinates" entry in each section.  The X and Y values must fall on a regularly spaced grid, so that the coordinates can be analyzed to deduce the spacing and overlap between pieces unambiguously.  Not all pieces in a rectangle need to be present, and there need not be pieces at 0.  Pixel coordinates are in a right-handed system despite the ubiquitous inversion of images in Y.

If the alignment between montage pieces is known, this information can be placed in both the XedgeDxy/YedgeDxy and the XedgeDxyVS/YedgeDxyVS entries, so that the alignment can be used by SerialEM regardless of whether the Very Sloppy option is checked.  The entries are the amount that an "upper" piece, i.e., one to the right (for X) or above (for Y), is displaced from being aligned in the overlap zone.  This is the negative of the amount that the upper piece needs to be shifted to align the two pieces in the overlap zone.  The program uses the IMOD library routine findPieceShifts to solve for the best shifts to be applied to each piece to bring all of them into best alignment.  There is a fixed (but easily raised) limit to the size of montage for which this solution will be found, based on the nominal number of pieces in X and Y rather than the actual number.

Navigator Files

In SerialEM 3.6, the Navigator output was switched from one tab-delimited line per item to an autodoc format, with an option to output as XML instead.  XML or autodoc files can be read back in, as well as the tab-delimited files from earlier versions.  There are two lines of global data,
  AdocVersion = x.xx
  LastSavedAs = x\y.nav
where the second is the full absolute path and name of the file as it was last saved.  Then every item is in a section of type 'Item' and with its label string as the value.  The following table lists all of the keys that can occur in a section, the kind of value that is expected, and whether it is required, or if it is not required, what default value is set.

Key	Type of value	Default value, or Required	CMapDrawItem member(s)	Description
Color	Integer	Req	mColor	Color index (0-5)
StageXYZ	Three Floats	Req  (unless external)	mStageX/Y/Z	Stage position adjusted for current mag
NumPts	Integer	Req	mNumPoints	Number of points
Corner	Integer	0	mCorner	Flag for corner point
Draw	Integer	1	mDraw	Flag to draw
RegPt	Integer	0	mRegPoint	Registration point number
Regis	Integer	Req	mRegistration	Registration at which item exists
Type	Integer	Req	mType	Type: 0-2 for point, polygon, map
Note	String	blank	mNote	Note string
GroupID	Integer	0	mGroupID	ID of group thatitem belongs to
PolyID	Integer	0	mPolygonID	ID of polygon used to define supermontage
FitToPolygonID	Integer	0	mFitToPolygonID	ID of polygon that this montage map was fit to
Imported	Integer	0	mImported	Indicator of an imported map or point drawn on one
RegisteredToID	Integer	0	mRegisteredToID	For an imported map, ID of map that it was registered to for transforming
SuperMontXY	Two Integers	-1,-1	SuperMontX/Y	Camera coordinate of a montage in supermontage
OrigReg	Integer	Regis	mOriginalReg	Original registration (default is Registration)
DrawnID	Integer	0	mDrawnOnMapID	ID of map point/polygon was drawn on
Flags	Integer	0	mFlags	Bit flags: bit 1 on if item was drawn on an FFT
BklshXY	Two Floats	0,0	mBacklashX/Y	Backlash when a montage was taken or associated with item
SamePosId	Integer	0	mAtSamePosID	Items with this matching were taken at same raw stage position
RawStageXY	Two Floats	-10000,-10000	mRawStageX/Y	Raw stage position before adjustments
Acquire	Integer	0	mAcquire	Flag for acquiring
PieceOn	Integer	-1	mPieceDrawnOn	Index of montage piece item was drawn on (piece_#_in_X * #_of_pieces_in_Y + piece_#_in_Y), or index of nearest piece if item is on a missing piece, in which case XYinPc will not be written to file.
XYinPc	Two Floats	-1,-1	mX/YinPiece	X,Y coordinates in the montage piece, in right-handed pixels of stored montage
MapFile	String	Req if Map	mMapFile	Full or relative path of map file (see Read & Open)
MapID	Integer	Req if Map	mMapID	Unique ID (all items get one, not just maps)
FocusAxisPos	Float	-1.e8	mFocusAxisPos	Stored position of Focus area on inter-area axis
LDAxisAngle	Two Integers	0,0	mRotateFocusAxis, mFocusAxisAngle	First value: 1 to rotate inter-area axis from tilt axis, 0 not to; second value: angle of rotation
FocusOffsets	Two Integers	0,0	mFocusX/Yoffset	Offset of focus subarea in X and Y in unbinned pixels (Y inverted)
HoleArray	Two Integers	0,0	mNumX/Yholes	# of positions in X and Y for multiple Record
SkipHoles	List of integers	none	mSkipHolePos	Pairs of X,Y indexes for positions to skip from  that array specified in HoleArray.  Positions are numbered from 0 at lower-left in stage coordinates
HoleISXspacing	Three Floats	0,0,0	mXHoleISSpacing	X component of vectors from hole finding converted to image shift for use with multiple Records
HoleISYspacing	Three Floats	0,0,0	mYHoleISSpacing	Y component of hole vectors converted to image shift
TSstartEndAngles	Two Floats	-1.e8,-1.e8	mTSstartAngle, mTSendAngle	Starting and ending angles of range for  tilt series item
TSbidirAngle	Float	-1.e8	mTSbidirAngle	Bidirectional starting angle for tilt series item
TargetDefocus	Float	-1.e8	mTargetDefocus	Target defocus for Acquire or tilt series item
FileToOpen	String	none	mFileToOpen	Name of file to open for Acquire or tilt series item
TSParamIndex	Integer	-1	mTSParamIndex	Index of into parameter array for tilt series
MontParamIndex	Integer	-1	mMontParamIndex	Index into array of parameters for montages to open
FilePropIndex	Integer	-1	mFilePropIndex	Index into array of file options for file to open
MapMontage	Integer	Req if Map	mMapMontage	Flag that map is a montage
MapSection	Integer	Req if Map	mMapSection	Section number in file
MapBinning	Integer	Req if Map	mMapBinning	Binning at which map was taken, or of initial overview map image for montage
MapMagInd	Integer	Req if Map	mMapMagInd	Magnification index of map, or of non-map image a point or polygon was drawn on
MapCamera	Integer	Req if Map	mMapCamera	Camera index
MapScaleMat	Four Floats	Req if Map	mMapScaleMat	Stage to pixel scale matrix for drawing, based on pixels of initial map image.  For montage, needs to be adjusted by ratio of currently loaded map width/height to initial map width/height
GridMapXform	Six Floats	none	mGridMapXform	Transformation matrix and shift found by the last run of the routine to realign a reloaded grid that transformed items
MapWidthHeight	Two Integers	Req if Map	mMapWidth/Height	Size of initial map image at which scale matrix was defined
MapMinMaxScale	Two Floats	0,0	mMapMin/MaxScale	Min and max scale values of image when map was defined
MapFramesXY	Two Integers	0,0	mMapFramesX/Y	Number of montage frames when acquired
MontBinning	Integer	0	mMontBinning	Actual binning of montage used to make the map
MapExposure	Float	0.	mMapExposure	Exposure time for original map images
MapSettling	Float	0.	mMapSettling	Drift settling
ShutterMode	Integer	-1	mShutterMode	Shutter mode
K2ReadMode	Integer	0	mK2ReadMode	Read mode for K2/K3 camera
MapSpotSize	Integer	0	mMapSpotSize	Spot size at which map was taken
MapIntensity	Double	0	mMapIntensity	Intensity at which map was taken
MapSlitIn	Integer	0	mMapSlitIn	Filter slit state when map was taken
MapSlitWidth	Float	-1.	mMapSlitWidth	Filter slit width when map was taken
RotOnLoad	Integer	0	mRotOnLoad	Flag to rotate when load
RealignedID	Integer	0	mRealignedID	ID of lower mag map with nearby realign error
RealignErrXY	Two Floats	0,0	mRealignErrX/Y	Final stage error of that realign operation
LocalErrXY	Two Floats	0,0	mLocalRealiErrX/Y	Error in second round of realign operation
RealignReg	Integer	0	mRealignReg	Original registration of that realign
ImageType	Integer	0	mImageType	Map file type 0-3 for MRC, IMOD, TIFF, IDOC
MontUseStage	Integer	-1	mMontUseStage	1 if montage was taken with stage, 0 if not, -1 if unknown
DefocusOffset	Float	0.	mDefocusOffset	Defocus offset for map based on View images
K2ReadMode	Integer	0	mK2ReadMode	Read mode for K2/K3 camera
NetViewShiftXY	Two Floats	0,0	mNetViewShiftX/Y	Net IS offset for a View image map
MapAlpha	Integer	-999	mMapAlpha	JEOL alpha value
ViewBeamShiftXY	Two Floats	0,0	mViewBeamShiftX/Y	Incremental beam shift to apply for a View map
ViewBeamTiltXY	Two Floats	0,0	mViewBeamTiltX/Y	Incremental beam tilt to apply for a View map
MapProbeMode	Integer	-1	mMapProbeMode	Probe mode (0 for nano, 1 for micro)
MapLDConSet	Integer	-1	mMapLowDoseConSet	Set # (0-4) used for map taken in low dose
MapTiltAngle	Float	-10000.	mMapTiltAngle	Tilt angle at which map was taken
PtsX	NumPts Floats	Req	mPtX	X stage cordinates of points
PtsY	NumPts Floats	Req	mPtY	Y stage cordinates of points
UserValueN	String	none	mUserValueMap	Arbitrary value set by script; N is # (currently1-8)
TSParam - a section containing tilt series parameters.  See TiltSeriesParam.h in the source code for a description of each item.
MontParam - a section containg montage parameters.  See MontageParam.h in the source code for a description of each item.
FileOptions - a section containing properties for opening a file.  See FileOptions.h in the source code in case this has descriptions.
StateParam - a section containing an imaging state, which has a State entry matching the StateParameters line in settings, and may have a LowDose entry matching LowDoseParameters in settings; see Settings and Calibration File Entries.

Importing Externally Defined Navigator Items

It is possible to add items to a Navigator file whose positions are specified by image coordinates in a map and have SerialEM do the conversion to stage coordinates.  Instead of the StageXYZ entry, the item can have one of these entries:

• CoordsInMap - Pixel coordinate in the map image, which can be either a single-frame image or a montage with pieces at their regularly spaced, nominal piece coordinates.  If entire rows or columns were skipped on the bottom or left side of a montage, coordinates starting at (0,0) in an analyzed image must be offset by the nominal piece coordinates of the first column and row included in the montage.  The same offset should be applied for the following two entries.
• CoordsInAliMont - Pixel coordinates in a montage map where the pieces are aligned without the Very Sloppy option, i.e., placed at the coordinates specified by AlignedPieceCoords in the .mdoc file.
• CoordsInAliMontVS - Pixel coordinates in a montage map where the pieces are aligned with the Very Sloppy option, i.e., placed at the coordinates specified by AlignedPieceCoordsVS.
• CoordsInPiece - Pixels coordinates within one piece of a montage, which must be specified with a PieceOn entry.  Pieces are numbered from 0 by their positions in a complete array; i.e., piece index = piece_#_in_X * #_of_pieces_in_Y + piece_#_in_Y, all numbered from 0.  The number of pieces in Y must be the value when the montage was first defined, including all skipped rows.  This value is available in the Navigator MapFramesXY entry and in the FullMontSize entry in any MontSection of the montage metadata.  Similarly, the piece number must be the value it would have if skipped rows or columns were included.

The following points apply to externally defined items:

• Only one of the four entries or StageXYZ may be entered for an item.
• Each entry takes 3 numbers, the third being a stage Z coordinate, which will be used without modification.
• Items must have a DrawnID entry specifying a map that appears before the external item in the Navigator file, or is already in the Navigator table merging in a file containing only externally defined items.
• Pixel coordinates are right handed, with 0,0 at the lower left of the map.
• Montage overview images are always displayed with the size in X rounded up to a multiple of 4.
• With CoordsInAliMont or CoordsInAliMontVS, the map file must have an mdoc file with AlignedPieceCoords or AlignedPieceCoordsVS entries, respectively.
• Either points or polygons may be entered externally.
• Coordinates of the points, given by PtsX and PtsY, must also be pixel coordinates and will be converted in the same way as the main item stage position is.
• Point coordinates in PtsX and PtsY are optional for a point item having a NumPts entry of 0; if missing, they will be made identical to the main stage coordinates.
• In addition to the entries mentioned already, every item must have a Color, Regis, and Type entry.
• If an externally defined item does not have a MapID, a unique ID will be assigned.  If you do assign IDs, it is important to make sure that they will not duplicate those of map items.  The easiest way to do this is to assign IDs under 100000, since the program now generates IDs of >= 100000.
• Group IDs will not be assigned if they are missing.

Detailed error reporting is available for items that have been read in successfully from the Navigator file.  A summary of the types of errors when processing the items is given in the log, and if Debug Output is set to 'n', then the type of error will be reported for each item, identified by its label.  However, there is no detail when the item is initially rejected either because it lacked a required entry or because there was an error (such as too few values on a line) reading data from the autodoc.

Settings and Calibration File Entries

This section is for lengthy entries to the settings or calibration file that sometimes need to be interpreted.  Entries will be added upon need or request.  All alpha values are numbered from 0, not 1.

Settings file entries:

StateParameters:
   0 for non-low dose, -area # - 1 for low dose, 1 for old LD Record
   camera index
   mag index
   spot size
   intensity
   filter slit in
   filter energy loss
   filter slit width
   zero loss flag
   binning
   frame size in X
   frame size in Y
   exposure time
   drift settling
   shuttering
   read mode
   probe mode
   dose fractionation
   save frames
   processing/normalization
   align frames
   1 to use framealign, 2 to align in IMOD
   frame alignment parameter set index
   View read mode
   Focus read mode
   Trial read mode
   Preview read mode
   focus axis position
   alpha on JEOL, -999 otherwise
   defocus target
   low dose focus offset
   low dose shift offset X
   low dose shift offset Y
   flag that it is Mont-map

AutocenterParams:

   camera
  mag index
  spot size
  intensity
  binning
  exposure
  1 to use centroid
  probe mode
  1 to shift beam for centering
  beam shift in microns
  added X beam shift
  added Y beam shift

MultishotParams:

   Beam diameter for display
   Distance to off-center shots in main ring
   Number of shots in main ring
   Do center shot: -1 before, 0 none, 1 after
   Early return: 1 for last only, 2 for all, 3 for full sum on first
   Number of frames in early return
   Whether to save Record shot
   Additional delay after image shift
   Whether to use illuminated area for drawing diameter
   Whether to adjust beam tilt if possible
   1 to do pattern in hole, 2 to do multi-hole, 3 to do both
   Use the list of custom holes
   Factor by which to increase regular IS delay between holes
   X component of image shift vector in 'X' direction
   Y component of image shift vector in 'X' direction
   X component of image shift vector in 'Y' direction
   Y component of image shift vector in 'Y' direction
   Number of holes in 'X' direction
   Number of holes in 'Y' direction
   Mag at which regular pattern hole positions are measured
   Mag at which custom holes are defined
   Take cross pattern when it is 3x3
   Do a second ring of shots
   Number of shots in second ring
   Distance to off-center shots in second ring
   Tilt angle at which regular pattern was defined
   Tilt angle at which custom holes were defined
   Angle found in hole finder associated with regular array
 

Calibration file entries:

StandardLMFocus: This entry contains the recorded standard focus in LM or the eucentric focus is nonLM.
   Magnification index
   Focus value in microprobe, -999 if none
   Focus value in nanoprobe, -999 if none
   Magnification
Focus values are either absolute values between 0 and 1, or on JEOL, a large value that encodes coarse and fine focus components

FocusCalibration:
   mag index
   camera number
   Slope in X:   shift in X in unbinned pixels per micron defocus per unit of beam tilt
   Slope in Y:   shift in Y per micron per unit of beam tilt
   beam tilt:   Beam tilt in mrad or % at which calibration was done
   number of points
   direction index:   0 for +X tilt; 1 for +X, +Y; 2 for +Y; 3 for -X, +Y
   probe mode
   alpha
   magnification
Each line then has the adjusted defocus value (where 0 would be 0 defocus) and the unbinned shifts in X and Y

SpotIntensities:  There may be two tables if either nanoprobe or microprobe has two calibrations, one below crossover and one above crossover.  However, a table can have one condition done below crossover and the other above.  It all gets sorted out into separate tables internally.  Each row contains:
   Spot size
   Microprobe intensity ratio, intensity at which acquired, and crossover intensity
   Nanoprobe intensity ratio, intensity at which acquired, and crossover intensity

where a ratio of 0 indicates no measurement for that spot size.

BeamShiftCalibration: This calibration is a 2x2 transformation matrix from image shift to beam shift at the given magnification:
   mag index
   X change in beam shift per X change in image shift
   X change in beam shift per Y change in image shift
   Y change in beam shift per X change in image shift
   Y change in beam shift per Y change in image shift
   alpha
   probe mode
   1 to retain the calibration when a new one is done in the same mag range and beam conditions, 0 not to
   magnification

HighFocusMagCal and HighFocusISCal:
   spot size
   probe mode
   defocus
   intensity
   mag change
   rotation
   crossover
   aperture size
   Mag index (0 for mag cals)

ComaVsISCal:
   Mag index
   spot size
   probe mode
   alpha
   aperture size
   intensity
   X change in beam tilt needed per X change in image shift
   X change in beam tilt needed per Y change in image shift
   Y change in beam tilt needed per X change in image shift
   Y change in beam tilt needed per Y change in image shift
   X change in astigmatism needed per X change in image shift
   X change in astigmatism needed per Y change in image shift
   Y change in astigmatism needed per X change in image shift
   Y change in astigmatism needed per Y change in image shift

Stage Calibration and 2x2 transformation matrices

The stage calibration consists of 4 matrix terms, and like all such 2x2 transformation matrices, they are expressed as a change in output X value per change in X input value (xpx), change in X per change in Y (xpy), change in Y per change in X (ypyx), and change in Y per change in Y (ypy).  They are written to file in this order.   In this case the input value is the change in stage position in microns, and the output value is the negative of the amount that the image shifts in unbinned pixels, or the position of the new centered coordinate in the image relative to the original center.  This negative value is unfortunate in that it inverts the typical stage coordinate system relative to the underlying "specimen" coordinate system that has its X along the tilt axis.

The StageCalibration line has
   Mag index
   Camera number
   xpx
   xpy
   ypx
   ypy
   Absolute focus at which calibration taken (compound of coarse and fine values for JEOL)
   Magnification

Other Notes

Image Rotation

The 'specimen' coordinate system is a right-handed system with its X axis along the positive tilt axis.  (Looking toward the origin along the positive tilt axis, a positive tilt rotates the stage counterclockwise).  Image rotation is defined as the amount that the camera image has rotated CCW from the image at the specimen plane, i.e., the angle (CCW positive) from the positive X axis of the image to the positive tilt axis in the image.

If there is only one camera and there is no RotationAndPixel line with a calibrated absolute rotation, the rotation at one magnification is estimated from a fallback value, the sum of that mag's rotation value in the MagnificationTable, the GlobalExtraRotation, and the ExtraRotation for that camera.  If there are relative rotations in the table, the fallback is assigned to one mag as the absolute rotation, and this allows a rotation to be derived for the others.  This is an ambiguous situation because there is no way to know just which mag that fallback angle should apply to. 

Once there is a calibrated absolute angle, the MagnificationTable rotation, GlobalExtraRotation, and ExtraRotation become irrelevant for all the mags that are listed in the RotationAndPixel table and linked by relative rotations to a mag with an absolute rotation.  The relative rotation for a mag equals the rotation angle of that mag minus the rotation of the next lower mag; this relationship allows a rotation angle to be propagated outward from the mag with the calibrated absolute angle. If any of these mags have an image shift calibration, other mags in the same mag range that have an image shift calibration can also be assigned a rotation by assuming that image shift is invariant at the specimen within that mag range.  On a Thermo/FEI scope, the mag ranges are LM and nonLM; on a JEOL using image shift deflections there are other boundaries across which rotation cannot be transferred; on a JEOL scope using PLA, rotation cannot be transferred at all. The MagnificationTable rotations retain a fallback role for other mags that have not been assigned a rotation by one of these mechanisms; the rotation for such a mag is derived from the nearest mag with an assigned rotation by adding the difference between the MagnificationTable rotation of the mag in question and that of the nearby mag.

The situation is even more complicated when there is more than one camera at the same level, i.e., more than one non-energy filter camera or more than one energy filter camera, and one lacks a calibrated absolute angle.  The program will use image shift calibrations if possible to get an absolute angle from another camera.  All of the assignments described here are listed when the program is started with DebugOutput set to 'c'.  See also ShiftManager::PropagateRotations(). 

The image rotation value is intimately related to the stage calibration.  For a Thermo/FEI scope or JEOL scope without omega filter, an image rotation of 0 produces this pattern of image movement

  ^  image movement for positive Y stage move
  |
  |
  |
  |
  --------->   image movement for positive X stage move

           ^  image movement for positive X stage move
           |
           |
           |
           |
  <---------   image movement for positive Y stage move

A typical stage calibration will have values like: -54   261   -262   -55, which indicates that an X movement produces predominantly a negative Y change in camera coordinate, while a Y movement produces a positive change in X coordinate.  This is inverted from the diagram because a positive shift of the image corresponds to a negative change in the (right-handed) camera coordinates of the centered point.  Similarly, a Krios or a JEOL F200 has an axis angle near 180 with a pattern

  <---------   image movement for positive X stage move
           |
           |
           |
           |
           V  image movement for positive Y stage move

For a JEOL with omega filter with the tilt axis near horizontal (.e.g, 2200FS), the pattern is this:

  --------->   image movement for positive X stage move
  |
  |
  |
  |
  V  image movement for positive Y stage move