framewatcher(1) General Commands Manual framewatcher(1)
NAME
framewatcher - watch for command files or stacks to run with align-
frames
SYNOPSIS
framewatcher [options]
DESCRIPTION
Framewatcher is a Python program that can watch for either command
files or frame stacks appearing in a directory. The command files are
expected to run the program Alignframes and to have extension .pcm,
the extension on files produced by the SerialEMCCD plugin to DigitalMi-
crograph. The program can modify a few of the parameters in such a
file with the options below. The frame stacks must have extension .mrc
or .tif. Output files with aligned sums can be placed in a different
directory. Stacks and command files can be moved to another directory
when they are finished. Whole tilt series can be processed when a com-
mand file specifies the metadata (.mdoc) file for the tilt series
stack.
To process frame stacks, either a master command file for alignment
must be supplied or the -nocom option must be entered; in either case,
the program will watch only for stacks, not for command files. It will
avoid moving an image file that has no images in it yet and a tilt
series file that is not yet finished. The master command file can run
Alignframes, in which case the same parameters can be modified
before running on each stack as when watching for command files.
Alternatively, the command file can run other software, in which case
it should contain the text "INPUTFILE" and "OUTPUTFILE" wherever the
input and output filenames are needed, respectively. The name of the
stack will be substituted for "INPUTFILE", and the name of the output
file for the aligned sum, including the optional output directory, will
be substituted for "OUTPUTFILE". A command file will be written with
the root name of the stack plus the extension ".pcm".
In either case, the command file is run with Vmstopy in the direc-
tory where it is located and produces a log file there. If files are
not being moved elsewhere when done, the command file is renamed with
".done" added at the end, which is how the program keeps from rerunning
stacks or command files even if restarted. If no command file at all
is run, ".done" will be added to the stack name itself unless it is
moved elsewhere.
An arbitrary command can be run before or after the command file is run
(or a stack is moved or renamed, when there is no command file). Items
such as the root name of the data can be substituted into the command,
as described below. This may provide a more flexible way to run other
alignment software than the master command file.
The program works by looking in the directory for eligible files and
running all of the ones that it sees. When it sees no files, it waits
5 seconds before scanning the directory again. When it see stacks, it
runs all but the last one immediately, then waits until the last one
(which typically would be the only one seen) has not changed for at
least 15 seconds.
Entrl Ctrl C to terminate the program. This will kill any running
alignment operation.
OPTIONS
Framewatcher uses the PIP package for input (see the manual page for
pip). Options can be specified either as command line arguments
(with the -) or one per line in a command file (without the -).
Options can be abbreviated to unique letters; the currently valid
abbreviations for short names are shown in parentheses.
-watch (-w) OR -WatchDirectory Text string
Directory to watch for Alignframes command files, whose
extension should be .pcm, or stacks, whose extension should be
.mrc or .tif.
-output (-o) OR -OutputDirectory Text string
Directory in which to write aligned sums and other output. This
directory must already exist. By default, files are written to
the command file directory.
-processed (-pr) OR -ProcessedDirectory Text string
One or more directories into which frame stacks will be moved
when they are finished being processed. If a relative path is
entered, it will be treated as relative to the watched direc-
tory, not the current directory. The directory will be created
if it does not exist. If multiple directories are entered, the
program will cycle through them. When a command file to process
a single frame file is done, the program will move the stack,
command file, and log file into the directory. If this direc-
tory is on another filesystem, the files will be copied there
and deleted from their current location. If the command file
has options to use a gain reference file or a defect correction
file, copies of these files will also be maintained in the pro-
cessed directory. Alternatively, when moving files without a
command file, the program will look for the names of the gain
and defect files in the stack header and make sure there are
copies of those. When a command file to process a whole tilt
series is done, nothing is moved unless the last component of
the path to the frame files is the same as the tilt series root
name. SerialEM will write files in this way if the option to
use the file name of the current open file in the name of the
frame stack folder is selected in the Frame File Options dialog.
In that case, the directory itself is moved to the processed
directory, since it can be assumed not to contain other frame
stacks to be processed. If this -processed option is not
entered, the program will just rename each ".pcm" file used to
align the frames to "...pcm.done". (Successive entries accumu-
late)
-master (-ma) OR -MasterCommandFile File name
Name of a command file to use if watching for frame stacks
instead ".pcm" files. For each stack, the entries will be modi-
fied appropriately, a file will be written with the root name of
the stack plus ".pcm", and that file will be run. This master
file should have either commands to run Alignframes (includ-
ing lines with the options "InputFile" and "OutputImageFile") or
entries INPUTFILE and OUTPUTFILE that will be substituted with
the names of the frame stack and aligned sum file, respectively.
If a file contains Alignframes options to apply a gain refer-
ence or defect file, the specified files will be used without
any checking for whether there is a newer file that applies to
the current frames. Thus, be sure these entries are correct for
the current session before using a command file with such
entries.
-nocom (-noc) OR -NoCommandFile
Just watch for frame stacks without running a command file.
This option cannot be entered with -master and must be accompa-
nied by either -processed, -before, or -after; i.e., there must
be something for the program to do with each stack. If a com-
mand file does exist with the same root name as the stack, it
will be copied also.
-exclude (-e) OR -ExcludeFilesMatching Text string
When watching for stacks, exclude file names containing the
given string. This option can be used to exclude the frame
stacks when processing whole tilt series with an .mdoc file con-
taining frame file names.
-lacks (-l) OR -MdocLacksImageExt
When watching for stacks, .mdoc file names will be derived from
the root name of the stack instead of the full name with exten-
sion (e.g., setname.mdoc instead of setname.mrc.mdoc).
-fmdoc (-f) OR -FrameFilesInMdoc
Use names of frame stacks from an .mdoc file. This option is
useful when watching for tilt series stacks and not doing dose
weighting.
-ddrop (-dd) OR -DropAndReplacementDoses Two floats
Threshold dose for excluding frame set and dose to accumulate
instead. This option is appropriate if Alignframes is being
run and causes the corresponding option to be inserted into the
command file.
-mdrop (-md) OR -DropSetIfMeanBelow Floating point
Exclude frame sets with a mean in an .mdoc file below the given
value. This option is appropriate if Alignframes is being
run and causes the corresponding option to be inserted into the
command file.
-age (-ag) OR -MinimumAgeOfStacks Floating point
When watching for frame stacks, use this option to set the mini-
mum time interval between the last detected change in the modi-
fication time of the stack and starting to process the stack.
The default is 15 seconds.
-gpu (-g) OR -UseGPU Integer
Use the GPU (graphical processing unit) for computations; enter
0 to use the best GPU on the system, or the number of a specific
GPU (numbered from 1). This entry overrides whatever was origi-
nally written into the command file.
-binning (-bi) OR -BinningOfSum Integer
Image reduction to apply to summed frames; this will override
the value in the command files, which was set from the binning
of the image returned to SerialEM.
-sumscale (-su) OR -ScalingOfSum Floating point
Amount to scale summed values before output; this will override
the value in the command files, which was set to match the scal-
ing of images returned to SerialEM. If dose-weighting is being
used without the -dnorm option, additional scaling may be needed
to preserve intensity resolution in highly filtered images.
-dmdoc (-dm) OR -DosesInMetadataFile
Do dose weighting with dose values in metadata file, if it is
present. This entry will lead to an entry "TypeOfDoseFile 4" in
the ".pcm" file and remove entries for FixedTotalDose or Fixed-
FrameDoses. When frames sets are being aligned for a whole tilt
series, the metadata file entry is already present in the ".pcm"
file. In other cases, the program will look for a frame stack
.mdoc file, i.e., a file with ".mdoc" appended to the name of
the input stack. If this file is present, the MetadataFile
option will be added to the ".pcm" file; otherwise nothing will
be added. This option cannot be entered with -dframe or -dto-
tal.
-dtotal (-dt) OR -FixedTotalDose Floating point
Total dose for each set of frames (not for entire tilt series),
in electrons/square Angstrom. This entry will lead to a corre-
sponding entry in the ".pcm" file, replacing an existing one if
any, and remove entries for TypeOfDoseFile or FixedFrameDoses.
This option cannot be entered with -dmdoc or -dframe.
-dframe (-df) OR -FixedFrameDoses Floating point
List of frame doses and numbers that applies to all sets of
frames. The list is a set of paired numbers: a dose in elec-
trons/square Angstrom and the number of frames taken at that
dose (e.g., enter "2,10,4,5" for 10 frames of 2 e/A^2 and 5
frames of 4 e/A^2). This entry will lead to a corresponding
entry in the ".pcm" file, replacing an existing one if any, and
remove entries for TypeOfDoseFile or FixedTotalDose. This
option cannot be entered with -dmdoc or -dtotal.
-dprior (-dp) OR -InitialPriorDose Floating point
Dose applied before any of the images in each frame set were
taken; this value will lead to a corresponding entry in the
".pcm" file, replacing an existing one if any.
-dnorm (-dn) OR -NormalizeDoseWeighting
Normalize the dose weight filters so that they add up to to 1.0
within each set of frames. This entry will lead to a corre-
sponding entry in the ".pcm" file. It is useful for tilt series
where dose weighting of the tilt images is to be done in a later
step.
-unweight (-un) OR -UnweightedFileSuffix Text string
Suffix for name of output file of summed images that have not
been dose weighted, created in addition to the dose-weighted
images placed in the main output file. The file will be named
with _suffix.mrc after the stack root name.
-volt (-v) OR -Voltage Integer
Microscope voltage in kV; this value must be either 200 or 300.
This entry will lead to a corresponding entry in the ".pcm"
file, replacing an existing one if any; if none is entered, the
default in Alignframes is 300 kV.
-drift (-dr) OR -DriftLimitDistAndNumber Two floats
Maximum shift between frames above which to drop initial frames
of the set (in unbinned pixels), and either the maximum fraction
or the maximum number of frames to drop. This entry will lead
to a corresponding entry in the ".pcm" file. It is particularly
useful for single-exposure (frame) tilt series.
-reduce (-r) OR -ReducedSum Integer
Amount to reduce image by, or target size, to get a reduced copy
of the summed image. A value of 32 or less will be taken as the
shrinkage factor, and a value greater than 32 will be taken as
the target size to shrink too. In the latter case, the geomet-
ric mean of the X and Y size will be divided by the target size
and rounded to an integer. The output file will be named as the
stack root name followed by "_redN.mrc", where N is the reduc-
tion factor. The file will be placed in the same directory as
the aligned sum unless a different destination is specified
with the -thumb option. If the summed file is a tilt series,
only the middle image will be used.
-power (-po) OR -PowerSpectrum Integer
Height for a JPEG file with a side-by-side reduced sum and
reduced power spectrum. The summed image will be reduced by
exactly the amount to give the specified height. The power
spectrum will be computed on the full-sized summed image and
reduced to the given height, so it shows all the frequencies in
the summed image. The output file will be named as the stack
root name followed by "_power.jpg". The file will be placed in
the same directory as the aligned sum unless a different desti-
nation is specified with the -thumb option. If the summed file
is a tilt series, only the middle image will be used.
-scale (-sc) OR -ScaleSpectrum Two floats
Background gray level near the edge of the power spectrum, and
the relative diameter of the annulus around the center whose
mean will be scaled to be white. The default gray level is 48,
and the default diameter is 0.02. Making the diameter smaller
will saturate less area at the center of the power spectrum;
making it bigger may give more dynamic range to show Thon rings
away from the center.
-thumb (-thu) OR -ThumbnailDirectory Text string
Directory into which the reduced sums or power spectra will be
written. If this option is not entered, they will be placed in
the same directory as the aligned sums. The directory will be
created if it does not exist.
-before (-be) OR -RunCommandBefore Text string
Command to run before running a command file, if any. The com-
mand can include items enclosed in %{...} to be substituted with
the actual values for the current frame stack. These items
include:
%{frameFile} - Frame stack, including original path when
watching for ".pcm" files
%{outputFile} - Aligned sum output file, including path
%{gainReference} - Gain reference file, including path
%{defectFile} - Defect file, including path
%{rootName} - Root name of command file/stack
%{outputDir} - Directory entered with -output, if any
%{processedDir} - Directory entered with -processed, if any
%{reducedDir} - Directory entered with -thumb, if any
The substituted text will be enclosed in double quotes, unless
it is a root name with no spaces included. Despite the quotes,
it may be possible to concatenate these items with other text
(it works in Linux). The output file will not exist when a com-
mand is run before a command file.
-after (-af) OR -RunCommandAfter Text string
Command to run after running a command file, if any. The com-
mand can include the items enclosed in %{...} just described.
This command will be run after any moving or renaming of pro-
cessed files, so the input stack may no longer exist by the same
name.
-none (-non) OR -TextForNone Text string
Text to substitute in a command entered with -before or -after
if the file or directory is not available for a particular item.
The default is simply to remove the %{...} item if it has no
value. Using a defined text such as "NONE" can make it easier
for a script to parse the command, for example, because then
each option entered to the script would always be followed by
another token instead of a blank.
-keep (-k) OR -KeepCommandLogs
Keep log files with output from running commands before or after
a command file. The default is to write a log then delete it if
there is no error.
-avoid (-av) OR -AvoidWindowsCopy Integer
1 to use Xcopy in Windows instead of Robocopy, or 2 to use nei-
ther Xcopy nor Robocopy. In the latter case, Cygwin commands
will be used when running through Cygwin or Python commands when
running a Windows Python. This option exists because Robocopy
can give obscure failures in some cases. Xcopy can do the same
operations but is "deprecated". Copies with Cygwin commands may
be significantly slower in some cases (e.g., with a 10G net-
work); copies of whole directories through Python may be incred-
ibly slow.
-nice (-ni) OR -NicenessSetting Integer
Priority at which to run jobs, a number between 1 and 19, where
a higher value has lower priority. All values in this range are
meaningfully different on Linux or Mac OS X, but they will be
translated into a smaller selection of priority levels on Linux.
The default is 15.
-thread (-thr) OR -ThreadLimit Integer
Limit on number of threads used for processing. This value will
be used to set OMP_NUM_THREADS, which sets an upper limit to the
number of threads used for operations that are multi-threaded
with OpenMP (image reductions, FFTs). This thread number is
already limited to the number of real cores when hyperthreading
is enabled, but allowing that many threads may be counterproduc-
tive when other multi-threaded operations are running. You can
determine the number of physical and logical cores by entering
"imodqtassist -t" in a terminal.
-help (-h) OR -usage
Print help output
-StandardInput
Read parameter entries from standard input
EXAMPLES
To run command files that appear in the current directory with no
optional modifications, enter
framewatcher
To watch for files in a directory "X:\MyFrames" and move them to a sub-
directory "Processed" when done, enter (in a DOS terminal on Windows)
framewatcher -watch X:\MyFrames -proc Processed
where the directory would be entered as "X:/MyFrames" in a Cygwin win-
dow.
To make sums with image reduction by 2 and run on a specific GPU, say #
2, enter
framewatcher -bin 2 -gpu 2
where you could enter "-gpu 0" to run on the best available GPU even if
the command file does not specify using a GPU.
AUTHOR
David Mastronarde
SEE ALSO
alignframes
Email bug reports to mast at colorado dot edu.
IMOD 5.0.2 framewatcher(1)