Boulder Laboratory for 3-Dimensional Electron Microscopy of Cells NEWSTACK(1) NEWSTACK(1) NAME newstack - Make a new mrc image stack. SYNOPSIS newstack [options] <input file(s)> <output file> DESCRIPTION Newstack is a general stack editor to move images into, out of, or between stacks. It can float the images to a common range or mean of density. It can bin images and apply a general linear transformations as well as a specified rotation or expansion. It can put the output into a smaller or larger array and independently recenter each image separately from the transformation. Images can be taken from multiple input files and placed into multiple output files. Newstack uses the PIP package for input (see the manual page for pip) and can take input interactively only for options available before conversion to PIP, to maintain compatibility with old command files. The options accepted by the old newst script are all accepted by newstack. Almost all options can be abbreviated to one or two letters. The following options can be specified either as command line arguments (with the -) or one per line in a command file or parameter file (without the -): -input OR -InputFile File name Input image file. Input files may also be entered after all arguments on the command line, as long as an output file is the last name on the command line. Files entered with this option will be processed before files at the end of the command line, if any. (Successive entries accumulate) -output OR -OutputFile File name Output image file. The last filename on the command line will also be treated as an output file, following any specified by this option. (Successive entries accumulate) -fileinlist OR -FileOfInputs File name Name of file with list of input files and sections to read. The format of the file is the number of input files on the first line, the first file to read on the next line, the list of sections to read on next line, next filename on next line, etc. If this option is used, no input filenames may be specified with the -input option or following options on the command line. -fileoutlist OR -FileOfOutputs File name Name of file with list of output files and number of sections to place in each. The format of the file is the number of output files on the first line, the first file to write on the next line, number of sections to write on next line, next filename on next line, etc. If this option is used, no output filenames may be specified with the -output option or as the last file on the command line. -split OR -SplitStartingNumber Integer Starting number for a series of numbered output files, where one section will be placed in each file. There must be only one output file name entered, and it will be taken as the root name for the output files. A dot and the number will be appended to the root name for each file unless an extension is entered with the -append option. The number of digits used for the number will be the same for all files so that they will list in order. -append OR -AppendExtension Text string Extension to append to the filenames in a series of numbered files. With this option, files will be named like "rootnnn.ext" instead of "root.nnn", where "root" is the root name entered as the output file name, and "ext" is the extension entered with this option. -secs OR -SectionsToRead List of integer ranges List of sections to read from an input file, numbered from 0. Ranges may be entered (e.g., 1-3,5,8), and / may be used to specify all sections in the file. If multiple lists are entered, each one will be applied to the respective input file. If no list is entered for a file, all sections will be read. (Successive entries accumulate) -skip OR -SkipSectionIncrement Integer Increment for reading a regularly spaced subset of the sections in the list from the input file. Enter 2 to take every other section, 3 to take every third section, etc. Note that the increment is used to select from the input list, not from all the sections in the file. -numout OR -NumberToOutput Multiple integers Number of sections to write to each output file, when there are multiple output files. These numbers may be listed in sequence in one entry, or in several entries. (Successive entries accumulate) -replace OR -ReplaceSections List of integer ranges List of section numbers at which to write images into an existing output file. There must be only a single output file, its size in X and Y must be appropriate, and the given sections must already exist in the file. The minimum and maximum density in the file header will be modified by the min and max of the sections being written, so they can become more extreme but not less extreme. The mean density will not be changed. -blank OR -BlankOutput Output blank sections for nonexistent section numbers in input file, namely for section numbers less than 0 or more than the NZ - 1. The sections are filled with the mean value of the input file or with a fill value if one is entered with the -fill option; this value is then scaled in the same way that actual data are. -size OR -SizeToOutputInXandY Two integers The dimensions of the output image in X and Y. The default size is the same as that of the first input file. -mode OR -ModeToOutput Integer The storage mode of the output file; 0 for byte, 1 for 16-bit signed integer, 6 for 16-bit unsigned integer, or 2 for 32-bit floating point. The default is the mode of the first input file. -bytes OR -BytesSignedInOutput Integer This entry controls how bytes are stored in the output file and overrides both the default for this version of IMOD and the setting of the environment variable WRITE_MODE0_SIGNED. Enter 0 for unsigned numbers or 1 for signed numbers. Unsigned output is needed to work with versions of IMOD before IMOD 4.2.23; signed output may be needed for compatibility with external software that has followed recent documentation of the MRC format. Regardless of the representation in the file, bytes are read into IMOD programs as unsigned with a range of 0 to 255. -offset OR -OffsetsInXandY Multiple floats X and Y center offsets for each section. This is an offset to the center of the area from which image will be taken, so a positive offset in X will move images to the left. If images are being binned, the entry should be in unbinned pixels. Enter one X,Y pair to apply a single offset to all sections. (Successive entries accumulate) -applyfirst OR -ApplyOffsetsFirst Apply offsets in X and Y before transforming image, which will make a rotated image come from a predictable location. The default is to apply offsets after, which shifts the transformed image instead. -xform OR -TransformFile File name File with one or more linear transformations to apply to images -uselines OR -UseTransformLines List of integer ranges A list of the line numbers of lines of transforms to use in the transform file (numbered from 0). Ranges are allowed. The default is for the line numbers to match the input section numbers. To have a single transform applied to all of the sections, just enter a single number. (Successive entries accumulate) -onexform OR -OneTransformPerFile Apply one transform line for each input file. The default is for the first transform to be applied to the first file, the second transform to the second file, etc., in which case the transform file must have at least one line per input file. Otherwise, -uselines can be used to specify which transform to apply for each file, in which case at least one line must be specified per input file. -rotate OR -RotateByAngle Floating point Rotate all images by the given angle in degrees, positive for a counterclockwise rotation. Rotation is applied after binning and transformation, if any. It can be combined with linear transforms but not with warping transforms. If the rotation is by +90 or -90 degrees and no output size is specified, then the X and Y sizes of the image will be transposed so the the output file will show the whole rotated image. For any other rotation angles, you will get the same output size as the input image unless you enter a new size explicitly. -expand OR -ExpandByFactor Floating point Scale all images up in size by the given factor, greater than 1 to expand, or less than 1 to shrink with ordinary interpolation. Scaling is applied after binning and transformation, if any. It can be combined with linear transforms but not with warping transforms. If no output size is specified, the size of the input image will be scaled appropriately so that the output will contain the entire image. -shrink OR -ShrinkByFactor Floating point Scale all images down in size by the given factor, a value greater than 1, using antialias filtering. This option should be used for large image reductions, because using ordinary interpolation will give aliasing artifacts that appear primarily as noise. Even binning will give some aliasing. Shrinking can be applied after binning but cannot be combined with rotation, general transformation, or distortion correction. If no output size is specified, the size of the input image will be scaled appropriately so that the output will contain the entire image. -antialias OR -AntialiasFilter Integer Type of antialiasing filter to use when reducing images with the -shrink option. The available types of filters are: 1: Box - equivalent to binning 2: Blackman - fast but not as good at antialiasing as slower filters 3: Triangle - fast but smooths more than Blackman 4: Mitchell - good at antialiasing, smooths a bit 5: Lanczos 2 lobes - good at antialiasing, less smoothing than Mitchel 6: Lanczos 3 lobes - slower, even less smoothing but more risk of ringing The default is 5 for the Lanczos 2, which many people consider the best compromise among the various factors. -bin OR -BinByFactor Integer Use binning to reduce images in size by the given factor. Binning is applied before all other image transformations. If no output size is specified, the size of the input image will be scaled appropriately so that the output will contain the entire image. -origin OR -AdjustOrigin Adjust the origin values in the image file header for changes in image size or the centering of a subarea. With this adjustment, a model built on the input stack should be correctly located when loaded onto the output stack in 3dmod. Model points will be correctly located in Z provided that a contiguous set of sections is output. They will be correct in X and Y provided that the only operations on the data are a change in size of the output, binning with -bin, scaling with -expand, and shifting with -offset. -linear OR -LinearInterpolation Use linear instead of cubic interpolation to transform images. Linear interpolation is more suitable when images are very noisy, but cubic interpolation will preserve fine detail better when noise is not an issue. Images are transformed when the -xform, -expand, -rotate, -distort, or -gradient option is entered. -float OR -FloatDensities Integer Adjust densities of sections individually. Enter 1 for each section to fill the data range, 2 to scale sections to common mean and standard deviation, 3 to shift sections to a common mean without scaling, or 4 to shift sections to a common mean then rescale the minimum and maximum densities to the Min and Max values specified with the -scale option. -contrast OR -ContrastBlackWhite Two integers Rescale densities to match the contrast seen in 3dmod with the given black and white values. This works properly only when the output file will be bytes. It will not work if the data were loaded into 3dmod with intensity scaling; use mrcbyte in that case. -scale OR -ScaleMinAndMax Two floats Rescale the densities of all sections by the same factors so that the original minimum and maximum density will be mapped to the Min and Max values that are entered. -fill OR -FillValue Floating point Value to fill areas of the output image that have no image data. The filling is done before intensity scaling, so the value in the filled areas will be modified by any scaling that is done. -multadd OR -MultiplyAndAdd Two floats Rescale the densities of all sections by multiplying by the first entered value then adding the second value. This option must be entered either once only, or once per input file. (Successive entries accumulate) -taper OR -TaperAtFill Two integers To taper an output image at a border between real image and a filled area, just as is done with Mrctaper, enter this option with two values: 1) the distance over which to taper or 1 for a default distance, and 2) a 0 for tapering outside or 1 for tapering inside the border. The default distance is 1% of the mean of the X and Y sizes of the output image, but at least 16 pixels and no more than 127. The program will exit with an error if there is not enough memory to hold the entire output image. -distort OR -DistortionField File name Image distortion field file to use for undistorting images. The undistortion is applied before any transformations. -imagebinned OR -ImagesAreBinned Integer The current binning of the images, so that the distortion field can be applied correctly. This entry is required unless the program can determine the binning unambiguously from the image size. -fields OR -UseFields List of integer ranges A list of the distortion fields to apply for each section (numbered from 0). Ranges are allowed. The default is for the field numbers to match the input section numbers, unless there is only one field in the file. To have a single field applied to all of the sections, just enter a single number. This option was added when the only way to warp images was with distortion field files but it is not needed with current warping files. (Successive entries accumulate) -gradient OR -GradientFile File name File with magnification gradients to be applied for each image. This should be a file listing the tilt angle, the percent magnification change per micron of Z height, and the degrees of rotation per micron of Z height for each image, such as is produced by Extractmaggrad. The mag gradient correction is applied before a distortion field correction and before any transformations. -memory OR -MemoryLimit Integer Use this entry to specify the amount of memory used by the program for its main array, in megabytes. By default, the program will try to use enough memory to hold both an input image and an output image in floating point form (4 bytes per pixel). Limiting the memory can keep it from using more memory than available but may result in it operating on the images in chunks. The main array is used to hold both the input image (in whole or in part) and a transformed image (in whole or in part). -test OR -TestLimits Two integers To test the code for reading in and binning images in chunks, enter limits for the total size of the working array, and for the size of the array used for binning. The sizes are the number of 4-byte elements. -verbose OR -VerboseOutput Integer 1 for diagnostic output -param OR -ParameterFile Parameter file Read parameter entries as keyword-value pairs from a parameter file. -help OR -usage Print help output -StandardInput Read parameter entries from standard input. If no command line arguments are entered, inputs are prompted interactively. The input is as follows; lines starting with IF are entered only if a particular option is selected. Number of input files, or -1 to enter list of files from a file IF you enter a number >0, then for each input file, enter two lines: File name list of section numbers to take from that file. The first section is numbered 0 and ranges may be entered (e.g. 0-4,5,7-9) OTHERWISE, IF you enter -1, then enter the name of the file with list of files and sections. The format of this file should be Number of input files File name list of section numbers (first is 0!!!), ranges allowed (e.g. 0-3,5,7-9) next file and list of sections, etc. Number of output files, or -1 to enter list of files from a file IF you enter 1, next enter the name of the single output file. OR IF you enter a number >1, then for each output file, enter two lines: File name Number of sections to put in that file OR IF you enter -1, then enter the name of the file with the list of files and numbers of of sections. The format of this file is: Number of output files File name number of sections to place in that file next file and number of sections, etc. The X and Y dimensions of the output file, or / to take the same dimensions as the first input file The data mode of the output file, or / to take the same mode as the first input file. Mode can be 0 for bytes, 1 for 16-bit integers, 2 for 32-bit real numbers, or 9 to 15 for 9 to 15 bit values. 0 for no offsets, or 1 to apply X and Y offsets separately to each section, or -1 to apply the same X and Y offset to all sections IF offsets are selected, next specify the X and Y center offsets of each of the sections (they need not be on separate lines), or the single offset to apply to all sections. Hint: offsets of 50,50 will take the center of the output image from the upper right quadrant of the input image. 0 for no transforms, or 1 to transform images using cubic interpolation, or 2 to transform using bilinear interpolation IF transforming is selected, next specify the file for transforms, and on a separate line, a list of lines to use in the file (first is 0!!!). Ranges are allowed here, but all must be on one line. To have a single transform applied to all of the sections, just enter a single number. 0 for no floating of densities, 1 to float densities so that each section occupies the same RANGE of densities (the optimum range for the particular data mode), 2 to float so that all sections have the same MEAN and STANDARD DEVIATION of density (which requires the input sections to be read twice), -1 to specify a single scaling that will be applied equally to all sections, -2 to scale all sections by the same amount, determined by black and white contrast settings, in order to convert to bytes with stretched contrast, 3 to SHIFT sections to the same mean value without rescaling, or 4 to SHIFT sections to the same mean then rescale all sections by the same specified scaling. IF single rescaling is selected with "-1", next specify the density values to which the minimum and maximum in the data file should be mapped, or just enter "/" to have them mapped to the full-scale range for the data mode, or enter "1,1" to override any scaling of data that would otherwise occur because of a change of mode. This scaling method can be used to invert densities by entering, e.g., 255,0. IF scaling by black and white contrast settings is selected with -2, next enter the black and white settings (see mrcbyte). OR, IF shifting and rescaling was selected with "4", next specify the density values to which the new, shifted minimum and maximum values should be mapped, or just enter "/" to have them mapped to the full- scale range for the data mode. ADDITIONAL NOTES The program will adjust the pixel spacing in the header of the output file if binning and/or the expansion or skrinkage options are used. Models built on the input file in 3dmod should display correctly on a file scaled by these three options. However, the image origin is not adjusted for images that are a subset of the input image unless the -origin option is entered as well. Without this option, it would be necessary to shift a model manually in 3dmod to display it on a subset. When there is a change of modes, data are rescaled optimally in most cases. If floating is not selected, the scaling is by an amount that would map the range for the input data mode into the range for the output data mode (e.g., going from mode 1 to mode 2, it would scale data up by a factor of 128). Generally, all sections will be rescaled by the same amount, thus preserving relative differences between sections. However, if the output mode is 2 (reals), this kind of rescaling will not occur. In general, if your input mode is real and you are outputing to bytes or integers, you need to specify some kind of floating, shifting, or scaling in order to get usable values. These same kinds of automatic scalings upon change of mode will occur with shifting to a common mean (option 3), but here the program is more careful to apply the same scaling to all sections. To disable the scaling upon change of mode, the use the option "-scale 1,1," or the "-1" floating option with interactive input. At the end, the program will report how many pixels have been truncated at the low or high limits of the output range. If this happens after shifting with a specified output range, try again with a slightly smaller range. The transformation file entered with -xform can contain warping transformations as well as simple linear transforms. The transforms can consist of either displacements at a set of control points, as output directly by Midas, or displacements on a regular grid, as produced by Xftoxg. Such warping files contain information about the size of the images that were aligned and their pixel size. The transforms will be adjusted for a difference between that pixel size and the pixel size of the image file being transformed. They will also be extrapolated as necessary to cover an image area larger than that specified in the warping file; however, the program assumes that the input images are centered on the images with which the warping file was prepared. The warping will be incorrect if this is not the case. Warping cannot be used with distortion corrections. If this is necessary, consider using Blendmont, which can combine such operations. The -expand and -rotate options also cannot be used with warping transformations because such transformation would occur after the warping while the linear transform included in a warping file occurs before the warping. If these operations are necessary, you can avoid transforming the images twice by preparing a one-line text file with factor 0. 0. factor 0. 0. for expansion by the factor, or with cos(angle) -sin(angle) sin(angle) cos(angle) 0. 0. for rotation by the angle. Then use Xfproduct to multiply the warping transforms by the single linear transform, and transform the images with the product. Complex data (Fourier transforms) can be stacked but not otherwise manipulated. Don't try to change the size or center of the FFT's. Newstack will not work with color data (MRC data mode 16). If your images came from color Tiff files and you do not need the color information, use the "-g" option when converting them with Tif2mrc. If you do need to preserve the color information, then you can split the color channels into three byte-mode files with "clip splitrgb", use Newstack to perform identical operations on each of the three files, and recombine the data into a color file with "clip joinrgb". The program attempts to allocate enough memory to hold an entire input image and output image in 4 bytes per pixel, but it can work on large images with less memory than this by reading, operating on, and writing strips of the image if necessary. On a 32-bit system, if the memory allocation fails, it will try to work with just under 2 GB of memory. Memory allocation will probably not fail on a 64-bit system. In either case, system performance will suffer if the program allocates more than the amount of memory available, and it is up to the user to limit the memory as appropriate with the "-memory" option. The ability to work on the image in strips is limited when images are rotated by large angles (30-90 degrees); in such cases the program will need at least enough memory to hold the whole input images. The program can manipulate and transform images bigger than 4 gigapixels, but distortion correction is currently limited to 2 gigapixel images. Each linear transformation in a transform file is specified by a line with six numbers: A11 A12 A21 A22 DX DY where the coordinate (X, Y) is transformed to (X', Y') by: X' = A11 * (X - Xci) + A12 * (Y - Yci) + DX + Xco X' = A21 * (X - Xci) + A22 * (Y - Yci) + DY + Yco where (Xci, Yci) and (Xco, Yco) are the center coordinates of the input and output images, respectively. HISTORY DNM (mast) for VAX 8/88 5/4/89 changed to float to same mean and sd rather than mean and min. 12/19/89 added full control of density scaling; fixed bug in interpolation that was generating half-pixel error half the time. 4/19/90 added line-by-line prompts for file and section entry and added time/date stamp 1/23/91: fixed so that it can correctly pad images and extract subsets of images with either center-shifting or transforms 6/11/91: fixed bug if trying to output scaled real values les than 0 Feb 94 : Ported to SGI by Paul Furcinitti. Jul 94 : PSF (Furcinitti) Fixed bug when writing byte files. 11/16/94: DNM added shifting to mean, rationalized scaling somewhat 9/24/01: changed to read and write in chunks if images are big 12/24/03: converted to PIP input, added undistorting and binning June 2011: Added support for warping SEE ALSO mrcbyte(1), tif2mrc, clip