newstack(1) General Commands Manual newstack(1)
NAME
newstack - Make a new mrc image stack.
SYNOPSIS
newstack [options] input_file(s) output_file
DESCRIPTION
Newstack is a general stack editor to move images into, out of, or
between stacks. It can float the images to a common range or mean of
density. It can bin images and apply a general linear transformations
as well as a specified rotation or expansion. It can put the output
into a smaller or larger array and independently recenter each image
separately from the transformation. Images can be taken from multiple
input files and placed into multiple output files.
OPTIONS
Newstack uses the PIP package for input (see the manual page for
pip) and can take input interactively only for options available
before conversion to PIP, to maintain compatibility with old command
files. The options accepted by the old newst script are all accepted
by newstack. Almost all options can be abbreviated to one or two let-
ters. The following options can be specified either as command line
arguments (with the -) or one per line in a command file or parameter
file (without the -). Options can be abbreviated to unique letters;
the currently valid abbreviations for short names are shown in paren-
theses. However, if you are using short names in a script or command
file, always use at least 3 letters, because new options might use the
same one or two initial letters.
In Windows, Newstack expands wildcards ('*' and '?') for input file-
names internally, which is particularly useful without Cygwin. It can
also be used in Cygwin when the command line is too long after expan-
sion; enclose the names with wildcards in quotes to keep the shell from
expanding then. On other systems, when there are too many input files,
use the script Expandargs to produce a file that can be used with
the -filein option. That script can also handle ranges enclosed in
square brackets.
INPUT AND OUTPUT FILE OPTIONS
These options are involved in specifying input and output files.
-input (-in) OR -InputFile File name
Input image file. Input files may also be entered after all
arguments on the command line, as long as an output file is the
last name on the command line. Files entered with this option
will be processed before files at the end of the command line,
if any. (Successive entries accumulate)
-output (-ou) OR -OutputFile File name
Output image file. The last filename on the command line will
also be treated as an output file, following any specified by
this option. (Successive entries accumulate)
-fileinlist (-filei) OR -FileOfInputs File name
Name of file with list of input files and sections to read. The
format of the file is:
number of input files
name of first file to read
list of sections to read from first file
name of second file to read
list of sections to read from second file
...
If this option is used, no input filenames may be specified with
the -input option or as non-option arguments on the command
line. The section lists are allowed to have 8 times as many
characters as the maximum number of sections, which is 1,000,000
by default but can be changed with the -megasec option. A file
list can be constructed for a set of input files with the
Expandargs script.
-fileoutlist (-fileo) OR -FileOfOutputs File name
Name of file with list of output files and number of sections to
place in each. The format of the file is the number of output
files on the first line, the first file to write on the next
line, number of sections to write on next line, next filename on
next line, etc. If this option is used, no output filenames may
be specified with the -output option or as the last file on the
command line.
-reverse (-rev) OR -ReverseInputFileOrder Integer
Process some or all of the input files in reverse order. Enter
0 to reverse the order for all files, a number N greater than 0
to process only the first N files in reverse order, or a number
N less than 0 to reverse only the last N files. This option
allows you to stack sequentially numbered files in a desired
order and still enter them with wild cards on a command line.
No other entries, such as multiple section lists, are reversed;
they all apply to the files in the order that they are pro-
cessed. There must be at least the given number of input files,
entered with -input or as non-option arguments.
-split (-sp) OR -SplitStartingNumber Integer
Starting number for a series of numbered output files, where one
section will be placed in each file. There must be only one
output file name entered, and it will be taken as the root name
for the output files. A dot and the number will be appended to
the root name for each file unless an extension is entered with
the -append option. The number of digits used for the number
will be the same for all files so that they will list in order.
-append (-appe) OR -AppendExtension Text string
Extension to append to the filenames in a series of numbered
files. With this option, files will be named like "rootnnn.ext"
instead of "root.nnn", where "root" is the root name entered as
the output file name, and "ext" is the extension entered with
this option.
-format (-fo) OR -FormatOfOutputFile Text string
Set output file format to MRC, TIFF, HDF (if the package sup-
ports HDF), or JPEG (if the package supports JPEG) by entering
MRC, TIF, TIFF, HDF, JPG, JPEG, or any lower case form of these.
Only a single image can be output to a JPEG file. This entry
overrides a default format set with the environment variable
IMOD_OUTPUT_FORMAT.
-compression (-com) OR -HDFCompressionIndex Integer
Sets the level of ZIP compression when writing to an HDF file.
Compression values range from 1 (fastest, least compressed) to 9
(slowest, most compressed), or 0 for no compression. This entry
overrides a value for compression set by the IMOD_HDF_COMPRES-
SION environment variable.
-volumes (-vo) OR -VolumesToRead List of integer ranges
If any HDF files containing multiple volumes are included in the
input file list, this option must be entered to specify which
volume to read from such a file. Enter a volume number for each
input file, numbered from 1, at least up until the last multi-
volume HDF file in the input file list. Enter 1 for an HDF file
with only one volume or a stack of 2D images, or for other file
types.
-3d (-3) OR -Store3DVolumes Integer
This option controls whether data are stored in a 3D volume in
an HDF file. If 1 is entered, a new HDF file will be created
(overriding the default output type) and data will be stored as
3D volume rather than a stack of 2D images. If 2 or 3 are
entered, the output file must be an existing HDF file with data
stored as volumes, and data will be stored in a new 3D volume.
A value of 3 is used by Makepyramid to cause Newstack to
insert the attribute "image_pyramid". By default, if an HDF
output file is being created, it will be stored as a volume if
the first input file is an HDF file with volume storage. A
value of -1 will cause an output HDF file to be organized as a
stack even if the input file has volumes.
-chunk (-ch) OR -ChunkSizesInXYZ Three integers
Set target chunk sizes in X, Y, and Z for 3D volumes stored in
HDF files. Based on the target, an actual chunk size will be
chosen in each dimension that minimizes the amount of extra
image area created by making a set of full tiles. When the tar-
get value already evenly divides the image size in a dimension,
that target will be used. If this option is entered, the output
file will be an HDF file with data stored as a 3D volume, making
it unnecessary to enter -3d or -format. The default is for 3D
volumes to be stored with no tiling in X and Y and a chunk size
of 1 in Z, or with the tiling and chunk size of the first input
file if it is an HDF file stored as a 3D volume.
-mdoc (-md) OR -UseMdocFiles
This option allows data about each image section in metadata
autodoc (.mdoc) files to be transferred and managed much as data
in the extended header of an MRC file are. With it selected,
the program will search for a matching .mdoc file for each input
file that is not HDF, and create a matching .mdoc file for each
output file that is not HDF. (A matching file is one with .mdoc
appended to the image filename, as SerialEM creates). Metadata
about each image slice in ZValue sections will be transferred
between autodocs and the ZValue will be renumbered appropri-
ately. Thus, data can be transferred from one .mdoc to another,
if input and output files are MRC; from an .mdoc into the
attributes of an HDF file if input is MRC and output is HDF; or
from HDF attributes into an .mdoc if input is HDF and output is
MRC. When input and output are both HDF, attributes are trans-
ferred automatically. As of IMOD 4.11, metadata in autodoc sec-
tions other than ZValue ones will be transferred in their
entirety.^ If there is a single image in the input file, an
.mdoc file produced by SerialEM along with a stack of frames
will also be accepted as a valid source of metadata for the
image. These .mdoc files have a single section named "FrameSet"
and no global information about the image size or mode at the
top. Their contents will be transferred to ZValue sections in
the new .mdoc file
-remove (-rem) OR -RemoveForMdocName Text string
This entry and the next one allow use of .mdoc files whose name
endings do not match those of the corresponding image files.
The two entries specify how to get from the current name of the
input image file to the image file name upon which the .mdoc is
based. For this option, enter the text to be removed from the
end of the image file name. If necessary, use the -addback
option to restore the full original image file name. For exam-
ple, if a stack of frames is saved into "originalName.tif" and
the metadata into "originalName.tif.mdoc", then they are aligned
and summed into a single image file "originalName_ali.mrc", the
entry to this option would be "_ali.mrc" to get to "original-
Name". The entry to -addback would be ".tif" to restore the
original extension and get back to "originalName.tif". The pro-
gram would then look for and find the .mdoc "original-
Name.tif.mdoc".
-addback (-ad) OR -AddBackForMdocName Text string
String to add back to image file name after removing the string
specified by the -remove option (which must also be entered), to
obtain a file name for which there is a matching .mdoc.
-pixel (-pi) OR -PixelSizeFromMdoc
Set the pixel size in the header of each output file from the
value of "PixelSpacing" in an .mdoc file associated with the
input file, or in the metadata of an HDF file. Specifically,
the pixel size for an output file is set from the value for the
first section going into that file. When outputting one image
per file, such as with the -split option, each file will have
the correct pixel size. Output .mdoc files are not created
unless -mdoc is also entered.
-tilt (-ti) OR -TiltAngleFile File name
Name of file with tilt angles, one per line, to substitute or
insert into the metadata of the output file(s). The successive
angles in the file will be placed into the output in order, so
the angles in the file should be in the same order as the
intended order of the output. Angles can subsequently be
retrieved with Extracttilts. If the input file(s) have
extended headers with either the SerialEM or Agard/FEI format,
the new values will be substituted for existing angles and other
extended header items will be preserved. (There must already be
angles in the SerialEM header, and all input files must have the
same type of extended header.) If the input file(s) have no
extended header, angles will be placed into an extended header
in the Agard/FEI format (just 1 floating point number per sec-
tion). (In this case, all input files must have no extended
header.) Tilt angles can also be replaced or inserted if output
file(s) are HDF, with or without an incoming .mdoc file; and
tilt angles in .mdoc files associated with MRC files will be
maintained as well. If just an extension is entered (for exam-
ple, ".tlt"), the program will look for a file with the root
name of the input image file and this extension.
-reorder (-reo) OR -ReorderByTiltAngle Integer
Reorder sections by tilt angle, which are taken either from an
extended header or from a file of current angles specified with
the -angle option. If any input file does not have angles in
its extended header, the angles must be supplied with the -angle
option. Enter 1 or -1 to have sections put in increasing or
decreasing order by angle. If angles are provided with the
-angle option, you can also enter 2 or -2 to have the reordered
angles inserted into the extended header. Reordering is compat-
ible with the section selection options below except -twodir;
the subset of sections specified by them will be output in order
by their angles. The -tilt option cannot be combined with this
one. There must be only one output file. If there is more than
one input file, the sections will be ordered separately for each
one.
-angle (-ang) OR -AngleFileToReorder File name
File with tilt angles to use for reordering sections by angle
with the -reorder option. The file must contain one angle per
output section, each on its own line (not one per section in the
input file, if a subset of sections is being used.) The angles
should be in the same order as the input sections. The angles
in this file override any angles present in the extended header,
but they will replace the angles copied over in the extended
header only if -reorder is entered with 2 or -2.
-newangle (-new) OR -NewAngleOutputFile Integer
File for output of tilt angles after reordering with the
-reorder option. This option can be used regardless of whether
original angles are supplied with the -angle option.
SECTION SELECTION OPTIONS
These options are used to specify which sections are read and written
-secs (-se) OR -SectionsToRead List of integer ranges
List of sections to read from an input file, numbered from 0
unless -fromone is entered. Ranges may be entered (e.g.,
1-3,5,8), and / may be used to specify all sections in the file.
If multiple lists are entered with multiple input files, each
one will be applied to the respective input file. If no list is
entered for a file, all sections will be read. When there is a
single input file, multiple entries will simply be concatenated
to make one list. Such entries may be necessary for entering
very long lists, because each list entry is limited to 100,000
characters. (Successive entries accumulate)
-samesec (-sa) OR -SameSectionsToRead
Read the same set of sections from all input files, without hav-
ing to enter a section list for each file. The -secs options
must be entered only once.
-fromone (-fr) OR -NumberedFromOne
With this option section numbers entered with the -secs,
-replace, and -exclude options, line numbers entered with -use-
lines, and field numbers entered with -fields are numbered from
1 instead of 0.
-exclude (-exc) OR -ExcludeSections List of integer ranges
List of sections to exclude from the input file(s). Exclusion
is applied to the entire section list after it has been composed
from other section-specifying options. Any number in the sec-
tion list contained in this exclusion list is removed from the
section list, no matter how many times that number appears.
Thus, one exclude entry can be used to exclude the same set of
section numbers from multiple input files. The list can consist
of ranges; sections are numbered from 0 unless -fromone is
entered.
-twodir (-tw) OR -TwoDirectionTiltSeries
Combine two files from the two halves of a bidirectional tilt
series, where each half is taken in opposite directions of tilt
from a common starting angle. The sections will be stacked in
inverted order for the first file. There must be two input
files, and section lists cannot be entered.
-skip (-sk) OR -SkipSectionIncrement Integer
Increment for reading a regularly spaced subset of the sections
in the list from the input file. Enter 2 to take every other
section, 3 to take every third section, etc. Note that the
increment is used to select from the input list, not from all
the sections in the file.
-numout (-nu) OR -NumberToOutput Multiple integers
Number of sections to write to each output file, when there are
multiple output files. These numbers may be listed in sequence
in one entry, or in several entries. (Successive entries accu-
mulate)
-replace (-rep) OR -ReplaceSections List of integer ranges
List of section numbers at which to write images into an exist-
ing output file, numbered from 0 unless -fromone is entered.
There must be only a single output file, its size in X and Y
must be appropriate, and the given sections must already exist
in the file. The minimum and maximum density in the file header
will be modified by the min and max of the sections being writ-
ten, so they can become more extreme but not less extreme. The
mean density will not be changed.
-blank (-bl) OR -BlankOutput
Output blank sections for nonexistent section numbers in input
file, namely for section numbers less than 0 or more than the NZ
- 1. The sections are filled with the mean value of the input
file or with a fill value if one is entered with the -fill
option; this value is then scaled in the same way that actual
data are.
OPTIONS FOR OPERATING ON IMAGES
These options specify geometrical operations performed on the images.
-offset (-of) OR -OffsetsInXandY Multiple floats
X and Y center offsets for each section. This is an offset to
the center of the area from which image will be taken, so a pos-
itive offset in X will move images to the left. If images are
being reduced with -bin, -shrink, or -ftreduce, the entry should
be in unbinned pixels. Enter one X,Y pair to apply a single
offset to all sections. If other transformations are being
applied to the sections, these offsets will generally be com-
bined with those operations, and non-integer offsets are mean-
ingful. Otherwise, the nearest integer offset will generally be
applied, except when the -phase, -ftreduce, or -ftexpand option
is used and the offsets are applied in Fourier space. (Succes-
sive entries accumulate)
-applyfirst (-appl) OR -ApplyOffsetsFirst
Apply offsets in X and Y before transforming image, which will
make a rotated image come from a predictable location. The
default is to apply offsets after, which shifts the transformed
image instead. This option is relevant only when offsets are
combined with either general linear transformations or rotation,
expansion, or shrinkage; when shifts are applied in Fourier
space with the -ftreduce option, they are always considered as
unbinned shifts and thus occur before the reduction.
-xform (-x) OR -TransformFile File name
File with one or more linear transformations to apply to images;
if there is only one transform it will be applied to all images.
-uselines (-use) OR -UseTransformLines List of integer ranges
A list of the line numbers of lines of transforms to use in the
transform file (numbered from 0 unless -fromone is entered).
Ranges are allowed. The default is for the line numbers to
match the input section numbers, unless there is just one trans-
form in the file. To have a single transform applied to all of
the sections, just enter a single number. (Successive entries
accumulate)
-onexform (-on) OR -OneTransformPerFile
Apply one transform line for each input file. The default is
for the first transform to be applied to the first file, the
second transform to the second file, etc., in which case the
transform file must have at least one line per input file. Oth-
erwise, -uselines can be used to specify which transform to
apply for each file, in which case at least one line must be
specified per input file.
-phase (-ph) OR -PhaseShiftFFT
Shift images by taking a Fourier transform, changing phases
appropriately, and taking an inverse Fourier transform, rather
than by interpolating in real space. In principle, this method
preserves high frequencies better than interpolation does. The
method can be used only if images are just being shifted or
scaled with the -ftreduce or -ftexpand option, not when they are
rotated, expanded, shrunk, stretched, warped, or undistorted.
The shifts can be specified in two ways: with the -offset
option, or as linear transforms that contain only shifts (i.e.,
transform lines containing "1 0 0 1" followed by the shifts in X
and Y). This option can be used together with either -bin or
-shrink, but not both, because the data are scaled down when
read in. The option need not be entered when doing image reduc-
tion in Fourier space with the -ftreduce option, as any shifts
will then be applied as addition phase shifts in the cropping
operation.
-rotate (-ro) OR -RotateByAngle Floating point
Rotate all images by the given angle in degrees, positive for a
counterclockwise rotation. Rotation is applied after binning
and transformation, if any. It can be combined with linear
transforms but not with warping transforms. If the rotation is
by +90 or -90 degrees and no output size is specified, then the
X and Y sizes of the image will be transposed so the the output
file will show the whole rotated image. For any other rotation
angles, you will get the same output size as the input image
unless you enter a new size explicitly.
-expand (-exp) OR -ExpandByFactor Floating point
Scale all images up in size by the given factor, greater than 1
to expand, or less than 1 to shrink with ordinary interpolation.
Scaling is applied after binning and transformation, if any. It
can be combined with linear transforms but not with warping
transforms. If no output size is specified, the size of the
input image will be scaled appropriately so that the output will
contain the entire image.
-shrink (-sh) OR -ShrinkByFactor Floating point
Scale all images down in size by the given factor, a value
greater than 1, using antialias filtering. This option should
be used for large image reductions, because using ordinary
interpolation will give aliasing artifacts that appear primarily
as noise. Even binning will give some aliasing, especially for
images rich in information above the highest frequency that will
be retained in the reduced image. Shrinking can be applied
either after binning or in place of binning. With no binning,
it can be followed by rotation, general transformation, or dis-
tortion correction, although only an integer shrinkage factor
can be used with warping transformations. With binning, it can-
not be used with these other operations. If no output size is
specified, the size of the input image will be scaled appropri-
ately so that the output will contain the entire image.
-antialias (-ant) OR -AntialiasFilter Integer
Type of antialiasing filter to use when reducing images with the
-shrink option. In addition, if this option is entered along
with -bin and without -shrink, it will cause image reduction to
be done with antialias filtering instead of by simple binning.
The available types of filters are:
1: Box - equivalent to binning
2: Blackman - fast but not as good at antialiasing as slower
filters
3: Triangle - fast but smooths more than Blackman
4: Mitchell - good at antialiasing, smooths a bit
5: Lanczos 2 lobes - good at antialiasing, less smoothing
than Mitchell
6: Lanczos 3 lobes - slower, even less smoothing but more
risk of ringing
The default is 6 for Lanczos 3 as of IMOD 4.7. Although many
people consider Lanczos 2 the best compromise among the various
factors, that sentiment may be based on images of natural scenes
where there are sharp edges. Enter a negative number to select
the default, whatever it is set to.
-bin (-bi) OR -BinByFactor Integer
Use ordinary binning to reduce images in size by the given fac-
tor, or use antialias filtering to reduce images by this factor
if the -antialias option is entered and -shrink is not. The
value of a binned pixel is the average of pixel values in each
block of pixels being binned. Binning is applied before all
other image transformations. If no output size is specified,
the size of the input image will be scaled appropriately so that
the output will contain the entire image. If the -oddeven
option is not entered, the binned size in a dimension will be
made even or odd depending on whether the input file's size is
even or odd, respectively, which can result in an output size
one pixel bigger even when the input size divided by the binning
is an integer.
-oddeven (-od) OR -AllowOddEvenChange
Allow the output size from binning in a dimension to be odd when
the input size is even, or even when the input size is odd, but
only when this will add just 1 pixel to the size that results
from integer division instead of 2.
-ftreduce (-ftr) OR -FourierReduceByFactor Floating point
Scale all images down in size by the given factor by cropping
the Fourier transform. This reduction method provides complete
removal of frequencies above the highest frequency being
retained (i.e., antialiasing) with no attenuation of the
retained frequencies. However, this option is slower than image
reduction in real space and cannot be used with any other image
transformation, reduction, or warping except simple shifting.
Shifts to keep the image centered will be applied in Fourier
space during the cropping operation, so this option implies the
-phase option and any other shifts that you choose to enter will
be applied along with this. Shifts can be entered either with
the -offset option or as linear transformations containing only
shifts. Currently, the reduction factor must be either an inte-
ger, or an integer divided by 2, 3, 4, 5, 6, 8, or 10. Thus,
2.7, 1.375, 3.75, 1.333, and 2.167 are all acceptable factors.
Factors that are integers divided by 3 or 6 must be entered with
three decimal places, as in the last two examples. The actual
factor applied in those cases will be a ratio of integers, not
the entered number, and be within 0.001 of the entered factor.
If the factor is not acceptable, the program exits with an
error.
-ftexpand (-fte) OR -FourierExpandByFactor Floating point
Scale all images up in size by the given factor by padding the
Fourier transform. This expansion method provides complete
preservation of the existing spatial frequencies without intro-
ducing any higher frequencies in the result, as linear or cubic
interpolation would. However, this option is slower than real-
space interpolation and has all the features and limitations
just described for the -ftreduce option.
-noise (-no) OR -NoisePadForFFT
Use tapered noise based on nearby pixels along the edge of an
image to pad an image for reduction and/or shifting in Fourier
space. The default is to pad with a simple taper that makes
streaks outside the edge of the image, which may appear as
structure if shifted into the image area. For low-dose images,
these streaks may look odd or otherwise cause trouble, which can
be avoided by using this option. This is the padding method
used in Alignframes.
-distort (-d) OR -DistortionField File name
Image distortion field file to use for undistorting images. The
undistortion is applied before any transformations.
-imagebinned (-im) OR -ImagesAreBinned Floating point
The current binning of the images, so that the distortion field
can be applied correctly. This entry is required unless the
program can determine the binning unambiguously from the image
size.
-fields (-fie) OR -UseFields List of integer ranges
A list of the distortion fields to apply for each section (num-
bered from 0 unless -fromone is entered). Ranges are allowed.
The default is for the field numbers to match the input section
numbers, unless there is only one field in the file. To have a
single field applied to all of the sections, just enter a single
number. This option was added when the only way to warp images
was with distortion field files but it is not needed with cur-
rent warping files. (Successive entries accumulate)
-subarea (-su) OR -SubareaOffsetsXandY Multiple floats
X and Y offsets between the center of a distortion field and the
center of each section. A subarea from the upper right of the
full image on which the field was based will have a positive
offset in X and Y. If images are being binned, the entry should
be in unbinned pixels. Enter one X,Y pair to use a single off-
set for all sections. (Successive entries accumulate)
-gradient (-g) OR -GradientFile File name
File with magnification gradients to be applied for each image.
This should be a file listing the tilt angle, the percent magni-
fication change per micron of Z height, and the degrees of rota-
tion per micron of Z height for each image, such as is produced
by Extractmaggrad. The mag gradient correction is applied
before a distortion field correction and before any transforma-
tions.
-origin (-or) OR -AdjustOrigin
Adjust the origin values in the image file header for changes in
image size or the centering of a subarea. With this adjustment,
a model built on the input stack should be correctly located
when loaded onto the output stack in 3dmod. Model points
will be correctly located in Z provided that a contiguous set of
sections is output. They will be correct in X and Y provided
that the only operations on the data are a change in size of the
output, binning with -bin, scaling with -expand, or -shrink, and
shifting with -offset.
-linear (-l) OR -LinearInterpolation
Use linear instead of cubic interpolation to transform images.
Linear interpolation is more suitable when images are very
noisy, but cubic interpolation will preserve fine detail better
when noise is not an issue. Images are transformed when the
-xform, -expand, -rotate, -distort, or -gradient option is
entered.
-nearest (-nea) OR -NearestNeighbor
Use nearest neighbor interpolation instead of cubic interpola-
tion to transform images. This method simply picks the nearest
existing pixel value instead of interpolating between surround-
ing values, so it can be used when pixels have discrete or mean-
ingful values that need to be preserved. This option and -lin-
ear are mutually exclusive. Images are transformed when the
-xform, -expand, -rotate, -distort, or -gradient option is
entered.
OUTPUT CONTROL OPTIONS
These options control the size, form, or density scaling of the out-
put.
-size (-si) OR -SizeToOutputInXandY Two integers
The dimensions of the output image in X and Y. The default size
is the same as that of the first input file, except in two situ-
ations where the input sizes are transposed for the output: when
-rotate is entered with an angle within 40 degrees of plus or
minus 90, or when non-warping transforms are entered that all
contain a rotation closer to plus or minus 90 degrees than to 0.
-mode (-mo) OR -ModeToOutput Integer
The storage mode of the output file; 0 for byte, 1 for 16-bit
signed integer, 6 for 16-bit unsigned integer, 2 for 32-bit
floating point, 12 for 16-bit floating point, or 101 for 4-bit
data. The default is the mode of the first input file, with two
exceptions. 1) For a 4-bit input file, the default is to output
as bytes. If you create a file in mode 101, the standard header
listing will show the "Map mode" as 0, but there will be a line
above that indicating that the mode is truly 101. 2) The
default mode of floating point output for MRC files is governed
by the value of environment variable IMOD_WRITE_FLOATS_16BIT.
Without this set non-zero, mode 2 is output by default for any
floating point input. Mode 12 is allowed only if the output for-
mat is MRC. The range for mode 12 is -65504 to 65504; values
are stored with 11-bit precision down to 6.1e-5 and 10-bit pre-
cision down to 6.0e-8.
-bytes (-by) OR -BytesSignedInOutput Integer
This entry controls how bytes are stored in the output file and
overrides both the default for this version of IMOD and the set-
ting of the environment variable WRITE_MODE0_SIGNED. Enter 0
for unsigned numbers or 1 for signed numbers. Unsigned output
is needed to work with versions of IMOD before IMOD 4.2.23;
signed output may be needed for compatibility with external
software that has followed recent documentation of the MRC for-
mat. Regardless of the representation in the file, bytes are
read into IMOD programs as unsigned with a range of 0 to 255.
-strip (-st) OR -StripExtraHeader
Do not transfer extra header information in input file(s) to
output file(s). The default is to transfer this information
whenever possible.
-float (-fl) OR -FloatDensities Integer
Adjust densities of sections individually. Enter 1 for each
section to fill the data range, 2 to scale sections to common
mean and standard deviation, 3 to shift sections to a common
mean without scaling, or 4 to shift sections to a common mean
then rescale the resulting minimum and maximum densities to the
Min and Max values specified with the -scale option (which must
be entered in this case). When floating to mean and SD, the
program will express the minimum and maximum densities for each
image as the number of SDs from the mean and analyze these min-
ima and maxima for extreme outliers. Images with extreme ranges
will be truncated to preserve the dynamic range for the rest of
the images. If no truncation is warranted from this analysis,
the output when transforming images while floating to bytes may
have a variable SD to keep the data after transformation within
the byte range. If this is problematic, float to an integer
mode output and then restack into bytes with the default or a
specified constant scaling.
-meansd (-mea) OR -MeanAndStandardDeviation Two floats
Scale all images to the given mean and standard deviation. This
option implies -float 2 and is incompatible with all other scal-
ing options. There is no check that the scaling is sensible for
the data mode, so be sure to change data modes to prevent exces-
sive truncation or loss of resolution. For example, change from
unsigned bytes to integers if setting the mean to 0 or the SD to
a large number (> 50); change from bytes or integers to floating
point if setting the SD to a small number (< 10).
-contrast (-con) OR -ContrastBlackWhite Two integers
Rescale densities to match the contrast seen in 3dmod with
the given black and white values. This works properly only when
the output file will be bytes. If the data were loaded into
3dmod with intensity scaling, add the -map option with the
intensity range given to 3dmod. If you already have a byte file
and want to reverse the contrast, you can do so with "contrast
0,255 -map 255,0".
-scale (-sc) OR -ScaleMinAndMax Two floats
Without the -float option, this option will rescale the densi-
ties of all sections by the same factors so that the original
minimum and maximum density will be mapped to the Min and Max
values that are entered. The Max can be less than or greater
than the Min. With "-float 4", the minimum and maximum result-
ing from the initial shifting of densities will be mapped to the
entered values. If you want to reverse the contrast of the
file, use Header to determine the existing minimum and maxi-
mum of the file and then use "-scale Max,Min".
-map (-ma) OR -MapFromRange Two floats
Scale from the given range of densities rather than the current
min and max when using -contrast or -scale without "-float 4".
The second number can be either less than or greater than the
first. Combined with -scale, this option allows an arbitrary
mapping of two density values to two other values. Combined
with -contrast and "-mode 0", the option has the same effect as
using mrcbyte with its -s and -c options.
-multadd (-mu) OR -MultiplyAndAdd Two floats
Rescale the densities of all sections by multiplying by the
first entered value then adding the second value. This option
must be entered either once only, or once per input file. (Suc-
cessive entries accumulate)
-fixrange (-fix) OR -FixRangeIfNeeded Two floats
This option controls the conditional scaling and shifting of
data described in detail in the section below on Density Scal-
ing. Such operations will occur only for integer image data
that are being transformed in some way, and only if the data
distribution fits some criteria. The first value entered is the
number of standard deviations from the mean within which data
should be protected from truncation after image transformations;
it must be at least 2. The second value is a scaling factor
that will be applied when the SD is below a criterion value (10
by default); the default scaling is 1. This option cannot be
used with any other scaling specifications.
-rfparam (-rf) OR -RangeFixingParams Two floats
This option sets two parameters relevant to the -fixrange
option: the criterion SD value below which data should be scaled
(default 10), and the factor by which the range of the data
might expand on each end upon interpolation or shifting by phase
shifts in Fourier space (i.e, the default of 1.2 allows for an
increase in total range of 1.4).
-fill OR -FillValue Floating point
Value to fill areas of the output image that have no image data.
The filling is done before intensity scaling, so the value in
the filled areas will be modified by any scaling that is done.
As of IMOD 4.12.7, the default is to fill with the mean at the
edge of the input image if it can be loaded entirely into memory
at once, otherwise to use the image mean.
-taper (-ta) OR -TaperAtFill Two integers
To taper an output image at a border between real image and a
filled area, just as is done with Mrctaper, enter this option
with two values: 1) the distance over which to taper or 1 for a
default distance, and 2) a 0 for tapering outside or 1 for
tapering inside the border. The default distance is 0.5% of the
mean of the X and Y sizes of the output image, but at least 16
pixels and no more than 127. The program will exit with an
error if there is not enough memory to hold the entire output
image.
OTHER OPTIONS
These include options for testing and memory control, and generic PIP
options
-memory (-mem) OR -MemoryLimit Integer
Use this entry to specify the amount of memory used by the pro-
gram for its main array, in megabytes. The use of memory and
the default limits for memory allocation are described above in
the section "Memory Usage and Size Limitations." Limiting the
memory can keep it from using more memory than available but may
result in it operating on the images in chunks. Entering a
value larger than its default limit will allow it to use more
memory and avoid writing to a scratch file. The main array is
used to hold both the input image (in whole or in part) and a
transformed image (in whole or in part).
-test (-te) OR -TestLimits Two integers
To test the code for reading in and binning images in chunks,
enter limits for the total size of the working array, and for
the size of the array used for binning. The sizes are the num-
ber of 4-byte elements.
-megasec (-meg) OR -MaxMegaSections Integer
Maximum number of sections that will be processed, divided by
1,000,000. The default is 1, which means that the program will
set up arrays for a million section numbers. If you get the
error message "ERROR: PARSELIST - TOO MANY LIST VALUES FOR
ARRAY" and are trying to process more than a million sections,
enter this option with a higher value.
-quiet (-q) OR -QuietOutput
Suppress image file opening and min/max/mean output
-print (-pr) OR -PrintXYSizeAndExit
Determine output file size in X and Y, print it, and exit
-verbose (-ve) OR -VerboseOutput Integer
1 for diagnostic output
-param (-pa) OR -ParameterFile Parameter file
Read parameter entries as keyword-value pairs from a parameter
file.
-help (-h) OR -usage
Print help output
-StandardInput
Read parameter entries from standard input.
INTERACTIVE INPUT
If no command line arguments are entered, inputs are prompted interac-
tively. The input is as follows; lines starting with IF are entered
only if a particular option is selected.
Number of input files, or -1 to enter list of files from a file
IF you enter a number >0, then for each input file, enter two lines:
File name
list of section numbers to take from that file. The first
section is numbered 0 and ranges may be entered (e.g. 0-4,5,7-9)
OTHERWISE, IF you enter -1, then enter the name of the file with list
of files and sections. The format of this file should be
Number of input files
File name
list of section numbers (numbered from 0), ranges allowed
next file and list of sections, etc.
Number of output files, or -1 to enter list of files from a file
IF you enter 1, next enter the name of the single output file.
OR IF you enter a number >1, then for each output file, enter two
lines:
File name
Number of sections to put in that file
OR IF you enter -1, then enter the name of the file with the list of
files and numbers of sections. The format of this file is:
Number of output files
File name
number of sections to place in that file
next file and number of sections, etc.
The X and Y dimensions of the output file, or / to take the same
dimensions as the first input file
The data mode of the output file, or / to take the same mode as the
first input file. Mode can be 0 for bytes, 1 for 16-bit integers,
2 for 32-bit real numbers, or 9 to 15 for 9 to 15 bit values.
0 for no offsets, or 1 to apply X and Y offsets separately to each
section, or -1 to apply the same X and Y offset to all sections
IF offsets are selected, next specify the X and Y center offsets of
each of the sections (they need not be on separate lines), or the
single offset to apply to all sections.
Hint: offsets of 50,50 will take the center of the output image
from the upper right quadrant of the input image.
0 for no transforms, or 1 to transform images using cubic
interpolation, or 2 to transform using bilinear interpolation
IF transforming is selected, next specify the file for transforms, and
on a separate line, a list of lines to use in the file (numbered
from 0). Ranges are allowed here, but all must be on one line. To
have a single transform applied to all of the sections, just enter a
single number.
0 for no floating of densities, 1 to float densities so that each
section occupies the same RANGE of densities (the optimum range for
the particular data mode), 2 to float so that all sections have the
same MEAN and STANDARD DEVIATION of density (which requires the
input sections to be read twice), -1 to specify a single scaling
that will be applied equally to all sections, -2 to scale all
sections by the same amount, determined by black and white contrast
settings, in order to convert to bytes with stretched contrast, 3
to SHIFT sections to the same mean value without rescaling, or 4 to
SHIFT sections to the same mean then rescale all sections by the
same specified scaling.
IF single rescaling is selected with "-1", next specify the density
values to which the minimum and maximum in the data file should be
mapped, or just enter "/" to have them mapped to the full-scale
range for the data mode, or enter "1,1" to override any scaling of
data that would otherwise occur because of a change of mode. This
scaling method can be used to invert densities by entering, e.g.,
255,0.
IF scaling by black and white contrast settings is selected with -2,
next enter the black and white settings (see mrcbyte).
OR, IF shifting and rescaling was selected with "4", next specify the
density values to which the new, shifted minimum and maximum values
should be mapped, or just enter "/" to have them mapped to the full-
scale range for the data mode.
Additional Notes
Pixel size. The program will adjust the pixel spacing in the header of
the output file if binning and/or the expansion or shrinkage options
are used. Models built on the input file in 3dmod should display cor-
rectly on a file scaled by these three options. However, the image
origin is not adjusted for images that are a subset of the input image
unless the -origin option is entered as well. Without this option, it
would be necessary to shift a model manually in 3dmod to display it on
a subset.
Density Scaling When there is a change of modes, data are rescaled
optimally in most cases. If floating or other scaling is not specified,
the scaling is by an amount that would map the range for the input data
mode into the range for the output data mode (e.g., going from mode 0
to mode 1, it would scale data up by a factor of 128). Generally, all
sections will be rescaled by the same amount, thus preserving relative
differences between sections. However, if either the input or the out-
put mode is 2 (real), this kind of rescaling will not occur. Data will
also be scaled down by default if being stored in 4-bit mode (101), but
not scaled up when going from 4 bits to bytes. In general, if your
input mode is real and you are outputting to bytes or integers, you
need to specify some kind of floating, shifting, or scaling in order to
get usable values. These same kinds of automatic scalings upon change
of mode will occur with shifting to a common mean (option 3), but here
the program is more careful to apply the same scaling to all sections.
To disable the scaling upon change of mode, use the option "-mult 1,0,"
or the "-1" floating option with interactive input.
At the end, the program will report how many pixels have been truncated
at the low or high limits of the output range. If this happens after
shifting with a specified output range, try again with a slightly
smaller range.
When the image is being transformed, automatic scaling or shifting of
integer values can be selected with the -fixrange option to avoid trun-
cation of values or loss of intensity resolution. Truncation can occur
when densities are at the bottom of the range for the data mode,
because transformations (especially with phase shifting in Fourier
space) can produce values outside the original range. Loss of resolu-
tion can occur if input values with few gray levels (such as from elec-
tron counting cameras) are interpolated but then rounded to the same
few gray levels. When given the -fixrange option, the program will
decide whether the data need to have their range shifted, their values
scaled, or both to avoid these effects. The decision is controlled by
the two values entered with that option. The first is the number of
standard deviations from the mean within which data should be protected
from truncation. (If the actual minimum or maximum of the data are
within this range, they will be used for the decision instead of the
mean minus or plus the given number of SDs.) When the program decides
that a change is needed and can be done without saturating the high end
of the data, it will either shift signed data (mode 1) that starts at
-32768 up by 32768, or change the mode of unsigned data (mode 6) to
signed (mode 1). A value around 5 should be adequate. The second
value is a scaling factor, which can be 1 to have no scaling. Data
will be scaled by this factor if its SD is below a criterion value (10
by default, copntrolled by the -rfparam option), and if that does not
produce truncation on the high end. Scaling in the range of 10-20
should be adequate to preserve resolution in unscaled electron counting
data.
Warping Transformations. The transformation file entered with -xform
can contain warping transformations as well as simple linear trans-
forms. The transforms can consist of either displacements at a set of
control points, as output directly by Midas, or displacements on a
regular grid, as produced by Xftoxg. Such warping files contain
information about the size of the images that were aligned and their
pixel size. The transforms will be adjusted for a difference between
that pixel size and the pixel size of the image file being transformed.
They will also be extrapolated as necessary to cover an image area
larger than that specified in the warping file; however, the program
assumes that the input images are centered on the images with which the
warping file was prepared. The warping will be incorrect if this is
not the case.
Warping cannot be used with distortion corrections. If this is neces-
sary, consider using Blendmont, which can combine such operations.
The -expand and -rotate options also cannot be used with warping trans-
formations because such transformation would occur after the warping
while the linear transform included in a warping file occurs before the
warping. If these operations are necessary, you can avoid transforming
the images twice by preparing a one-line text file with
factor 0. 0. factor 0. 0.
for expansion by the factor, or with
cos(angle) -sin(angle) sin(angle) cos(angle) 0. 0.
for rotation by the angle. Then use Xfproduct to multiply the warp-
ing transforms by the single linear transform, and transform the images
with the product.
Complex and Color Data. Complex data (Fourier transforms) can be
stacked but not otherwise manipulated. Don't try to change the size or
center of the FFT's.
Newstack can read in color data (MRC data mode 16, TIFF RGB or colormap
files) as floats, but it cannot preserve the color channels and output
color data again. If the input data are mode 16 and you want to com-
bine the channels, you must specify a different output mode. If you
want to preserve the color, there is a script, Colornewst, that will
split the color channels into three byte-mode files with "clip
splitrgb", use Newstack to perform identical operations on each of the
three files, and recombine the data into a color file with "clip join-
rgb".
Memory Usage and Size Limitations. By default, the amount of memory
that the program will try to use depends on the operations being done
and on the amount of memory in the system. When data are being trans-
formed in real space, it will seek to hold the entire input image and
the transformed image in memory, at 4 bytes per pixel. For other oper-
ations, including shifting and reduction in Fourier space, the same
memory may be used for input and output, although there are some cases
where it would need separate input and output memory. The memory to be
allocated is limited to the minimum of 3/4 of system memory, system
memory minus 1 GB, and 15 GB when there is up to 30 GB of system mem-
ory, or to half of system memory when there is more than 30 GB. For
large images, it can work with less memory than the ideal amount by
reading, operating on, and writing strips of the image if necessary.
The ability to work on the image in strips is limited when images are
rotated by large angles (30-90 degrees); in such cases the program will
need at least enough memory to hold the whole input images. If there
is image scaling that requires seeing the entire image to be output
before knowing how to scale it (i.e., with the -float option), the pro-
gram may even write these strips to a scratch file then reread them for
final scaling into the output file. On a 32-bit system, if the memory
allocation fails, it will try to work with just under 2 GB of memory.
Memory allocation will probably not fail on a 64-bit system. In either
case, system performance will suffer if the program allocates more than
the amount of memory actually available, and it may be necessary for
you to limit the memory as appropriate with the -memory option. Con-
versely, when you have a lot of memory for handling large images, you
may be able to use that option to allow the program to use more memory
and avoid writing to a scratch file. The sign that a scratch file is
being used is a line of output starting with "SCRATCH image file on
unit 3 :".
The program can manipulate and transform images bigger than 4 gigapix-
els, but distortion correction is currently limited to 2 gigapixel
images.
Format of Linear Transformations. Each linear transformation in a
transform file is specified by a line with six numbers:
A11 A12 A21 A22 DX DY
where the coordinate (X, Y) is transformed to (X', Y') by:
X' = A11 * (X - Xci) + A12 * (Y - Yci) + DX + Xco
Y' = A21 * (X - Xci) + A22 * (Y - Yci) + DY + Yco
where (Xci, Yci) and (Xco, Yco) are the center coordinates of the input
and output images, respectively.
HISTORY
DNM (mast) for VAX 8/88
5/4/89 changed to float to same mean and sd rather than mean and min.
12/19/89 added full control of density scaling; fixed bug in
interpolation that was generating half-pixel error half the time.
4/19/90 added line-by-line prompts for file and section entry and
added time/date stamp
1/23/91: fixed so that it can correctly pad images and extract
subsets of images with either center-shifting or transforms
6/11/91: fixed bug if trying to output scaled real values les than 0
Feb 94 : Ported to SGI by Paul Furcinitti.
Jul 94 : PSF (Furcinitti) Fixed bug when writing byte files.
11/16/94: DNM added shifting to mean, rationalized scaling somewhat
9/24/01: changed to read and write in chunks if images are big
12/24/03: converted to PIP input, added undistorting and binning
June 2011: Added support for warping
SEE ALSO
mrcbyte, tif2mrc, clip
BUGS
Email bug reports to mast at colorado dot edu.
IMOD 5.2.0 newstack(1)