About Scripts

Introduction
Using Variables
Array Variables
Arithmetic Expressions
IF, ELSEIF, ELSE, ENDIF, BREAK and CONTINUE Statements
Labels and SKIPTO Statements
Defining and Calling Functions
Error Handling and TRY - CATCH - ENDTRY Blocks
Advanced Examples
Scripting with Python

Introduction

A script is a sequence of commands that SerialEM can execute. A script can perform all of the individual actions involved in acquiring a tilt series, such as tilting, autofocusing, autoaligning, and saving images. In addition, there are many commands for other actions such as moving the stage or changing magnification or defocus. Most of this document describes how to use SerialEM's built-in scripting language.  Scripts can also be written in Python if you have a Python module for SerialEM and a matching Python installation is on the system path.

Scripts were called 'macros' until SerialEM 3.6.  You may still see the term 'macro' used in various places, or in older scripts, or in the source code.

Scripts can automatically repeat themselves and they can contain internal loops that are executed a specified number of times. One script can call another like a subroutine; when the second script finishes, the next command in the calling script will be run.  Functions can also be defined and called, but in a more flexible fashion than calling a script. There can also be IF statements for conditionally executing some statements. Another kind of structure is a TRY-CATCH block for handling errors that occur.  Loops, IFs, and TRYs together may be nested to 40 levels deep and calls may be nested to 40 levels deep. A script can also contain a statement at its end to switch to running another script, but this is not allowed in a script called by another script.

The commands are typed into a Script Editing Window, which can be opened either from the Script menu or by holding down the Ctrl key while pressing an enabled script button in the Camera & Script Control Panel or in the toolbar that can be opened from the Script menu.

One command is entered per line, but a long command can be continued onto multiple lines by ending each line but the last with a backslash (\) after a space. However, the combined line is limited to 60 space-separated items.  (This feature was new in SerialEM 3.7; if there is any chance that someone would try to run your script on an earlier version, you should include the command
   Require 201
somewhere in the script, or replace whatever is listed on an existing 'Require' statement with '201'.)

Blank lines are ignored.  Lines whose first non-blank character is '#' are treated as comments and ignored.  Comments can also usually be placed after the command on a line, starting with # after a space; except when the command treats the remainder of the line past a certain point as a single string.  As of SerialEM 3.8, multiline comments can be enclosed in /* ... */, similar to some programming languages (C/C++), but with more restrictions.  Specifically, 1) the opening '/*' must be the first non-white-space on a line; 2) although the ending '*/' can appear anywhere on a line, all text after it will be lost, so effectively it should be at the end of a line or on a line of its own.  These comments may not be nested.  A single line with /*... */ is not accepted in SerialEM 3.8 but is in 4.0.

Scripts can be run from the Script menu, from the Camera & Script Control Panel, or from the Script toolbar. In addition, CTRL F1 through CTRL F10 are hotkeys for running the first 10 scripts. A script will stop running if various conditions specified in the Script Controls dialog box are not met, such as the minimum number of counts in an image.

When a script is started, the program will first check through the script, and through any other scripts that it calls. It will check that the script(s) contain defined commands, that loops and IF blocks are properly nested and terminated, and that various commands have the required entries after them. If it finds an error at this phase, it will report the error along with a statement that nothing has been executed. Only some types of errors are found at this stage; other errors will not be found until the bad statement is reached during execution.

A script can define a name that will be displayed in several places: on the title bar of the editing window, in the Camera and Script Control Panel when one of the script running buttons is dialed to that script, in the script toolbar, and in the Run submenu of the Script menu. The name should be short to fit into the script buttons. It can contain spaces.  A longer name can be defined to display in the Run submenu. When a name is first added to a script, it will appear after some action is taken with the script.

Scripts will be saved when Settings are saved, and when a settings file is loaded, the scripts in that file will replace all of the scripts already in the program.

Using Variables

Variables can be defined in scripts and some arithmetic can be done in a line that defines a variable. There are a variety of different kinds of variables: regular ones defined on a line with an equals sign; persistent ones defined on a line with ':=' which will persist from one script run to another; loop index variables defined in a LOOP or DOLOOP statement; local ones that are defined only within a function or called script; variables set when values are reported with commands starting with 'Report' and many others; regular or persistent string variables that are defined on a line with '@=' or ':@=', respectively; and variables with one- or two-dimensional arrays. A variable can be used in any command by putting a '$' in front of the variable name; all commands are scanned first to substitute the values of variables.  However, the variable name is used without the '$' in numerous commands that operate o n the variable itself. A variable can be embedded in a text string, such as for opening a file, without needing to be separated by spaces from the rest of the string.  Note that like commands, variables are not case-sensitive and can be referred to as '$Name', '$name', '$NAME', etc.

1. A regular variable can be defined with an expression of the form 'Name = value'. In general, everything in such an expression must be separated by spaces. There must be spaces between the name and the equals, and between the equals and the value. The value can consist of an arithmetic expression, which is processed and reduced to a single value (see below). Only the first value or word of text after the equals sign is stored as the value; additional text is ignored. An existing variable can be given a new value, but it must be assigned to in the same way; i.e., with '=' for a regular variable or ':=' for a persistent variable. If the value is text rather than a number, such as a filename, its case is preserved only as of SerialEM 4.6.0 (Sept. 10, 2016); before that, the text was converted to upper case.  If you rely on case being preserved in variable assignments and there is some chance of your script being shared and tried on earlier versions of SerialEM, you should include a line
      Require keepcase
   somewhere in your script.
2. A persistent variable can be defined with an expression of the form 'Name := value'. The 'ClearPersistentVars' command can be used to remove all persistent variables, or an individual variable can be cleared by leaving out the value, as in 'Name :='.  The 'NewArray' and 'New2DArray' commands also the variable that they create to be defined as persistent.
3. A regular or persistent variable can be defined with an arbitrary string using an expression of the form 'Name @= string including everything else on the line' or 'Name :@= string', respectively.  Variables will be substituted but the text on the right will not be evaluated in any other way, and the entire text after the '@=' will be assigned to the variable.  This would be useful for preserving spaces in a filename that has been defined by the user, or for setting a variable with a message to be printed.
4. A loop index can be defined by adding an unused variable name after the number on a LOOP line, or by using the DOLOOP statement, which always includes an index variable. The variable on a LOOP line will have a value from 1 through the loop count; the variable for a DOLOOP will run from the starting to the ending values specified on the line. This variable is maintained by the program and cannot be assigned to.  It becomes undefined after the loop ends and can then be reused as a loop index or defined variable.
5. Other than loop indexes, all variables are 'global' by default, which means that they can be accessed or reassigned from any script or function, regardless of where they are first created.  Local variables that are accessible only within the script or function where they are defined can be created with the command
      LocalVar varName1 varName2 varName3 ...
   The listed variables will not be seen as defined and cannot be changed in any other functions or scripts, including functions in the same script.  They will become undefined when returning from function or script.  They must not already exist as global variables created in the current function or script.  If the variable name has been defined elsewhere, the local copy will be kept separate and not affect the global value.  Loop indexes can also be made local with the 'LocalLoopIndexes' command.  The 'NewArray' and 'New2DArray' commands also allow the variable that they create to be defined as local.
6. When a 'Report...' command is used, the values reported are generally assigned to variables named 'ReportedValue1', 'ReportedValue2', and so on up to 6, as well as to shorter variables named 'RepVal1', etc.  This assignment can be assumed when the command documentation is silent about it.  Some other commands set these variables as well; in these cases the command documentation should specify what is set.
7. For almost all 'Report...' commands, and a few others that set 'ReportedValue' variables, one or more variable names can be placed at the end of the command, and the reported values will also be assigned directly to those variables.  This particularly useful for saving values that need to be restored later.  There need not be a variable provided for every value being reported.  For example
     CameraProperties sizeX sizeY
   will assign the X and Y size of the current camera to 'sizeX' and 'sizeY' as well as to the regular 'ReportedValue' variable, and assigns a third output, the RotationAndFlip property, only to 'RepVal3' and 'ReportedValue3'.  Note these points:
    *  'Report...' commands will assign to listed variables unless the command documentation says 'No optional assignment to variable.'  These will generally be commands with optional entries.
    *  Some commands that take optional entries will assign to listed variables, but the optional entry is required in order for this to work.  These will generally be commands that take a buffer letter, where it is easy for the program to tell if you have omitted the letter and listed a variable instead.  The command documentation will say 'Values can be assigned to variables after a buffer letter.'
    *   Commands other than 'Report...' commands will not assign to listed variables unless the command documentation says so, such as with 'Values can be assigned to variables at end of command.'
    *   If you use this feature and there is any possibility that the script will be distributed and possibly run on earlier versions of SerialEM, you should include the command
      Require variables1
   somewhere in your script.   Earlier versions will not pay any attention to variables after commands and will just not work correctly, so this command is a good way to catch the problem.  (This feature was added to the 3.6.0 beta version on about March 10, 2016).
8. Variable names should not match the names of scripting control commands (like 'Loop') or arithmetic functions (like 'SQRT').  When such a 'reserved' name is used on the left side of an assignment statement, the script will fail during initial checking.  Otherwise it will run, but if such a variable name is assigned to by a command, such as in the mechanism just described, the program will print a warning in the log.  The list of reserved names can be seen in MacroProcessor.cpp at the top of the class constructor, CMacroProcessor::CMacroProcessor(), in the arrays tmpOp1, tmpOp2, and keywords.

Array Variables

Variables can also be set to an array of values, and individual values can be accessed by their index in the array.  To assign an array to a regular or persistent variable, enclose the value in curly braces, '{' and '}', for example:
    exposures = {0.5 1.0 1.5 2.0}
Space between the '{' or '}' and the values is optional.  Variables can be included in the list of values.  Some arithmetic expressions can also be included, provided that each element requiring evaluation is enclosed in parentheses.  For example:
    exposures = {$base ($base + 0.5) ($base + 1.0)}

These values can then be accessed with '$exposures[index]', where the index is numbered from 1.  The index can be a variable; it can even be an indexed array variable; but no arithmetic can be done inside the brackets before SerialEM 3.8   As of that version, simple arithmetic can be done, with a minor and a major restriction:  1) only parentheses and the operators '+', '-, '*' and '/' are allowed; 2) there can be no spaces within the brackets.  Yes, this is the opposite of arithmetic elsewhere, which requires spaces between all operators and values.  With a space within brackets, the program will split opening and closing brackets between words and complain about unmatched brackets.  If you want the convenience of arithmetic within subscripts, you will have to remember this rule.

The index must be close to an integer and between 1 and the number of elements in the array.  Thus,
   $exposures[2]
   $exposures[$loopInd]
   $exposures[$crossInd[$loopInd]]
   $xShift[($iy-1)*$nx+$ix]
are all legal, as long as 'crossInd' is itself an array with enough elements.  The last expression is the standard way to access element ix,iy of two-dimensional data stored in a one-dimensional array, where nx is the dimension of the data in X.

 The number of elements can be obtained by using '$#' instead of '$' in front of the array name, such as '$#exposures'.

Two dimensional (2D) arrays are also possible; they consist of a set of rows, each row being an independent array with its own size.  Unlike one-dimensional arrays, which can be created by assignment or with the 'NewArray' command, 2D arrays can be created only by the 'New2DArray' command.  This command can be used either to create an empty array, which you would add rows to with the 'AppendToArray' command, or to set up an array with a specified number of rows and values on each row, initialized to 0's.  The number of rows in a 2D array is available by using '$#' in front of the variable name, while the number of elements in a row is available by specifying the row number, as in '$#shifts[2]'.

Individual elements of a 2D array are accessed with two subscripts, as in $xShift[$iy][$ix].  The first subscript specifies the row, and the second the element within the row.  An expression with one subscript refers to the entire row of the 2D array.

A 1D array, or one row of a 2D array, can be made larger with the 'AppendToArray' command, which can either a single element or a set of values to the array.  An alternative method is to use the array or row in an array assignment with additional values.  Thus:
    exposures = {$exposures ($base + 1.5)}
would add one element to a 1D array.  However, this method is not restricted to add one element at the end; it can be used to add multiple elements or add elements at the front of the array.  If the value expression contains other arrays, they will be added in their entirety, thus allowing array to be concatenated.   An expression like '$exposures', i.e., an array variable not followed by an index, should be used only in assignments to new arrays, 'AppendToArray', and in the Echo command and its variants.  The value of this expression actually consists of the elements separated by newlines, which are turned into spaces by Echo but make the variable not useful in other contexts.

Once an array is defined, individual elements can be assigned to by placing an index after the variable name.  This index can be a constant, a variable, or an indexed array variable.  For example
    exposures[2] = 4
    exposures[$loopInd] = $base
    exposures[$crossInd[$loopInd]] = 1.5
    xShift[$ix][$iy] = 0.87
    xShift[$ix] = {-1.5 -0.75 0. 0.75 1.5}
are all legal as long as the index variables have appropriate values.  The last example is of an array being assigned to a row of a 2D array; the array on the right replaces the previou s row entirely.  Finally, the item being assigned to one array element can itself be an array variable or an array expression in '{ }'; the sequence of elements will replace the one being assigned to and the array size will increase accordingly.  To just insert an array or value after an element without replacing that element, use it on both sides as in:
    exposures[6] = {$exposures[6] 0.9 1.1 1.3}

Arrays of strings can be created in several ways.  First, a string assignment can contain embedded newlines ('\n'), as in
    message @= The first line\nThe second line\nA final line
which will become an array of 3 strings (simply by converting the '\n' to the newline character that separates array elements). Second, you can define an array with quoted strings after giving the 'ParseQuotedStrings' command, as in
    messList = { 'The first message' 'The second message' \
       'A third kind of error' 'And the last message' }
Finally, you can start an array and add lines to it one by one with AppendToArray.  SerialEM 4.0 contains several changes to prevent both a string assignment and the assignment of an array of quoted strings from being broken into words and triggering an error at ~55 words; instead, the limit is ~55 strings.

If you use arrays and there is any possibility that the script will be distributed and possibly run on earlier versions of SerialEM, you should include the command
   Require arrays
somewhere in your script to keep it from being used on versions that do not support arrays.  If you use arithmetic in subscripts, you should include
   Require 202
instead. If you use embedded newlines in a string assignment, want to assign a string with more than 55 words, or define an array of quoted strings with more than 55 words (including all continuation lines), you should include
   Require 204
instead.

Arithmetic Expressions

An arithmetic expression can contain numbers, variables that have numeric values, the 4 operators '+', '-', '*', and '/', and some arithmetic function names. Parentheses can also be included for clarity and to control the order of evaluation.  Expressions within parentheses are evaluated separately and replaced by a single number, working outwards from the deepest set of parentheses.  An expression is scanned first from left to right for "*' and '/' together; the number to the left and right of an operator are replaced by the result. Then the expression is scanned from left to right for '+' and '-' together and those operators are applied. Finally, it is scanned for some arithmetic function names.  The table below lists the functions that take one or two arguments, which are just the following one or numbers (each possibly derived from arithmetic operations).  Without  parentheses, this order of evaluation means that functions will only work at the beginning of an expression, and that the function will be taken of the whole remainder of the expression.  Technically, the solution to this restriction is to enclose the function and its arguments in parentheses, such as
   (ATAN2 $x +1 $y / 2)
but the program will also accept the more conventional placement of arguments within parentheses, as in
   ATAN2 ($x + 1 $y / 2)
There must be a space between the function name and the opening parenthesis. RAND is also available to obtain a random number between 0 and 1; it takes no arguments and either has to be assigned to a variable or written as (RAND) to use in any expressions.   Arithmetic can be done in variable assignment statements, in IF or ELSEIF statements, and in some other statements (see below).

Parentheses may be surrounded by spaces or attached to the items that they enclose.  Specifically, one or more left parentheses can be at the beginning of an item, and one or more right parentheses can be at the end of an item.  For example
    SIN_((12_+_$var)_/_2) is equivalent to SIN_(_(_12_+_$var_)_/_2)
where the underscores represent spaces.  Other than this flexibility, there must be a space between each value and each operator.

As of SerialEM 3.7, arithmetic expressions can be used in a wide variety of commands, in addition to variable assignments and conditional expressions.  Each arithmetic expression will evaluate to one numeric argument to the command.  An expression can contain parentheses and be enclosed in parentheses; the latter is a good idea for clarity in this situation.  The spaces around parentheses are optional, as just described, but there must be space between operators and values.  An example of such a command is:
    SetImageShift ($baseX + $deltaX) ($baseY + $deltaY)
Virtually all commands starting with 'Set' allow arithmetic expressions.  All other commands allowing them are listed in Commands Allowing Arithmetic Expressions for Arguments.

Older versions of the program will probably convert most components of the expression to zeros instead of giving an error.  If there is any chance that someone would try to run your script on an earlier version, you should include the command
   Require evalargs
somewhere in the script, or add 'evalargs' to an existing 'Require' statement.

IF, ELSEIF, ELSE, ENDIF, BREAK and CONTINUE Statements

An IF statement is used to start a block ending in ENDIF, in which statements are executed conditionally on the truth of the expression in the IF statement. The IF statement can contain one or more comparisons between two numbers, joined by the logical operators AND or OR. Each numerical comparison is referred to as a 'clause' (including in error messages). The format of the IF statement is thus:

    if clause1 logical_operator clause2 logical_operator clause3 ...

where each clause consists of:

    expression1 comparison_operator expression2

Here, each expression may be an arithmetic expression with multiple components, as long as it evaluates to a single number.  The comparison operator occurs between two values and is one of '<', '>', '<=', '>=', '==', and '!=' (the latter two test for equality and non-equality). If the statement is true, then following lines are executed until an ELSE or ELSEIF is encountered, if any, at which point lines are skipped until the ENDIF. If the statement is false, following lines are skipped until either an ELSEIF, an ELSE, or an ENDIF is encountered.

The format of the logical expression in an ELSEIF statement is the same as that in an IF statement. When an ELSEIF is encountered and no previous test in the IF block has been satisfied, then the expression is evaluated and the same actions are taken as for an IF statement.

Just as for an arithmetic expression, parentheses are allowed for grouping the logical expressions and controlling how they are evaluated.  The deepest set of parentheses containing a logical operator is evaluated first and replaced by a simple true or false clause, and evaluation works outward from there. Within parentheses or in their absence, a logical expression is evaluated from left to right. If there are more than two clauses, a cumulative truth value on the left is combined with the truth value on the right for each logical operator. For example, for
   $A < $B OR $A < 12 AND $I == 5
the truth of whether A < B or A < 12 is determined and ANDed with whether I is 5.  As programmers have learned over the years, relying on the order of evaluation in complex expressions is prone to error, so just use parentheses.
   ($A < $B OR $A < 12) AND $I == 5
makes it clear what is happening, and
   $A < $B OR ($A < 12 AND $I == 5)
gives the different result that a person used to AND having precedence over OR would expect.

A common use of IF statements would be for loop control using the BREAK, CONTINUE, or SKIPTO statements. BREAK causes termination of the innermost loop being executed; the script then continues with statements after the next ENDLOOP. CONTINUE causes statements to be skipped to the end of the innermost loop, but the next iteration of the loop is then run, if any.

Labels and SKIPTO Statements

The SKIPTO statement can be used to jump ahead to a location defined by a label.  A label is the only item on a line and ends in a colon; any character string not starting with $ is allowed. The SKIPTO statement includes the label without the colon.  For example,
    if $error > 0
       SkipTo Cleanup
    endif
    ......
    Cleanup:
    ......

The SKIPTO can be used to jump forward within a loop or IF block, but it cannot be used to jump into a different loop, a different IF block, or even a different ELSE/ELSEIF section of the same IF block.  SKIPTO statements would be most useful for breaking out of multiple loops at once, for skipping over substantial numbers of lines where an IF statement does not seem suitable.  SKIPTO is like GOTO in older programming languages, which is disapproved of for good reasons.  It is thus recommended that you use it only when other control constructs would be clumsy and result in a script that is harder to follow.  For example, if you have commands to restore initial settings that must be run from multiple places before exiting, it is preferable to put those in a function instead of using SKIPTO.

Defining and Calling Functions

It is possible to define named functions within scripts.  These functions must begin with Function, end with EndFunction, and occur after any non-function commands in a script.  (Commands after an EndFunction will not be run).  The Function statement includes the name of the function, which must be all one word without spaces, two optional entries for the number of numeric arguments and whether there is a string argument, and optional variable names for arguments to be assigned to.
     Function FunctionName #_of_numeric_args 1_if_string_arg argName1 argName2 ...
Any integer other than a 0 in the last position indicates that there is a string argument.  If any variable names are entered, they must be preceded by both of the numeric values.  There need not be as many variable names as arguments.

Like script names, function names are case sensitive.  There is no limit to the number of functions in a single script, although they must all have distinct names. The same name can be used for functions in different scripts; however, if this is the case, functions must be called with the script name or number as shown next.

A function is called with CallFunction, followed by the name of the function, the numeric arguments if any, then the string argument if any.  The string argument can include spaces; all text after the numeric argument is used for the string argument.  For example, for a function defined as
    Function AlignWithBuffer 1 1
the call could be
    CallFunction AlignWithBuffer $ifNeedImage $buffer
where the two variables would indicate whether a new image is needed and what buffer letter to align with.  If AlignWithBuffer occurs in more than one script, and the one in script 15 whose name is 'Many Funcs' is intended, the call would need to be either
    CallFunction 15::AlignWithBuffer $ifNeedImage $buffer
or
    CallFunction Many Funcs::AlignWithBuffer $ifNeedImage $buffer
Notice that it can handle spaces in the script name but not the function name.  However, as of SerialEM 3.7, if a function occurs in the current script as well as in other ones, the function in the current script will be used when there is no script number or name in front of the function name.

Inside the function, the arguments are defined as regular variables: either the variables listed at the start of the function, or 'argVal1', 'argVal2', etc.  If you listed fewer variable names than arguments, the last arguments will be assigned to 'argVal' variables with the same numbers as they would have had if no variables were listed.  The arguments are actually optional when calling a function, so if a numeric argument is not supplied in the call, the corresponding variable is 0, and a string variable will be empty if the string argument is not supplied.  The number of arguments actually supplied in the call is assigned to the variable 'numCallArgs'. As of SerialEM 3.7, this variable and the arguments are all created as local variables, which means they are not accessible if this function calls another script or function (which gets its own argument variables), they will not be lost if the function calls another function, and they become undefined when returning from a function.  As mentioned above, all other variables are global by default: variables defined by the calling script are available within a called function or script, and variables defined within a function or script are retained when it returns.

The 'Return' command can be used to return from a function before its end, and also (as of SerialEM 3.9, 4/28/21) can be used to pass up to 6 return values in the reportedValue variables.  A script relying on this feature that might be run on an earlier version should include the the command 'require 203'.

If you list variables to be assigned in function definitions and there is any possibility that the script will be distributed and possibly run on earlier versions of SerialEM, you should include
   Require variables1
somewhere in your script.   Earlier versions will not pay any attention to variables on a Function line and will not work correctly, so this command is good way to catch the problem.

If you have a cleanup function that restores initial settings, you can use the command 'OnStopCallFunc' to have the function called whenever there is an error or the user presses STOP.  Ideally, this should be placed after commands that record the initial state; otherwise, the function should use 'IsVariableDefined' to test if particular variables are defined before using those variables.

The existence of functions presents new possibilities for errors, and some of these errors cannot be detected in the initial checking of scripts.  For example, a function that has already been called cannot be called again before it returns, this error may not be detected until the second call of the function during execution.

Error Handling and TRY - CATCH - ENDTRY Blocks

For automated acquisition from multiple locations, it is usually undesirable for a message box to pop up when an error occurs that stops the script.  The simplest way to avoid this and keep acquisition going is with the command 'NoMessageBoxOnError'.  With this command, the message box will be intercepted, its text printed to the log, and the script stopped.  The Navigator will then go on to the next point.  However, this provides no way to recover from an error in cases where that is possible.

To recover from an error and possibly continue running a script, you can place commands that could result in errors inside TRY blocks, which are a feature of most programming languages.  The structure is:

TRY

   Commands that can give errors from the program

   and/or 

   Tests of results from various operations, using a THROW statement when a test fails

CATCH

   Commands to respond to the error in some way, such as by giving another message, restoring some conditions, exiting, or returning from a function

ENDTRY

An  error generated by a higher-level operation, as well as any THROW statement after the script finds an unsatisfactory result, will make the program jump to the CATCH block and run the commands there.  You can include text after the THROW to get a message specific to the test that failed.  You can add a non-zero number after the TRY to suppress the log output about the error. If no error occurs, the CATCH block is skipped.

However, errors generated by scope operations, which usually come through as being 'COM errors', cannot currently be caught because they cause a general error stop to be issued, which stops the script.  The problem here is that most higher-level operations do not test for an error return from each scope call, but instead rely on this error mechanism to stop them upon scope failure.

These TRY blocks can be nested; there can even be one inside the CATCH portion of a TRY block.  Otherwise, errors generated in the CATCH block will result in a message box, unless this TRY block is nested inside another one.  When TRY blocks are nested, the THROW statement can be used in the CATCH section of an inner TRY block.  If a script calls a function or another script from within a TRY block, errors or THROW statements within the called script or function will be caught by the CATCH section of this TRY block.

A THROW statement outside of any TRY block is not allowed and the initial script checking should object to this.  If not, when the THROW is encountered, the program will stop the script.

The interception of message boxes requires that the message be sent through a special call, and there are some errors that do not use that call.  If you encounter a message box that stops acquisition and think it should not, report the exact text of the message so that this call can be changed.

Advanced Examples

Variables, loops, arithmetic, and a function:  This script will save the current magnification and change to 20000x, then move the stage to a 4 by 4 array of positions centered around the current location and autofocus, take a Record, and save it at each position. Then it will restore the mag and the stage position. This is done in a function that is set up to be called if the user stops the script.  The indentation is for readability only.

# A script to illustrate variables, loops, and arithmetic

MacroName Example
Require variables1
ReportMag oldMag
ReportStageXYZ baseX baseY
OnStopCallFunc RestoreState
SetMag 20000
Loop 4 ix
    x = $baseX + $ix * 3 - 7.5
    Echo Doing column at X = $X
    Loop 4 iy
         y = $baseY + $iy * 2 - 5
         MoveStageTo $x $y
         Autofocus
         Record
         Save
    EndLoop
EndLoop
CallFunction RestoreState

Function RestoreState
MoveStageTo $baseX $baseY
SetMag $oldmag
EndFunction

Waiting for drift to be low enough: This example will take pictures at a certain time interval, measure the displacement between each pair of pictures, and do this repeatedly until the displacement falls below a criterion or the limit on the number of iterations is reached. Note that important parameters are defined as variables at the top of the script, so it would be easy for users to set parameters without having to be adept at writing such a script.

# A script to take a picture after drift falls below a criterion
MacroName Drift
shot = Trial
interval = 5
times = 4
crit = 0.7
SuppressReports
$shot
Delay $interval
Loop $times index
    $shot
    AlignTo B
    ReportAlignShift
    ClearAlignment
    dx = $reportedValue3
    dy = $reportedValue4
    dist = sqrt ($dx * $dx + $dy * $dy)
    echo Distance = $dist nm
    if sqrt ($dx * $dx + $dy * $dy) < $crit
        echo Drift is low enough after shot $index
        break
    endif
    if $index < $times
        Delay $interval
    else
        Pause Drift never got below $crit: Continue anyway?
    endif
EndLoop
Record

Three Choice Box:  This example shows how to use the ThreeChoiceBox command.  This is most easily done by using quoted strings as here:

ParseQuotedStrings 1
header = { 'It is critical that you answer a question about what to do at this point.' \
   ' This way is better than asking several yes-no questions in a row.' }
prompt1 = { 'Press "Fiddle" to experiment with the parameters;' \
   ' this could be rewarding but dangerous' }
prompt2 = 'Press "What to Do?" to find out more about the possibilities'
prompt3 = { 'Press "Proceed" to continue the operation with current parameters;' \
  ' this will give familiar results' }
buttons = { 'Fiddle' 'What to Do?'  'Proceed' }
ThreeChoiceBox header prompt1 prompt2 prompt3 buttons

The other alternative is to to assign each line to a variable with '@=' and make an array from those variables, but without using single quote wrappers as above, leading spaces will be lost.  For example:

p1a @= Press "Fiddle" to experiment with the parameters;
p1b @= this could be rewarding but dangerous
prompt1 = { $p1a $p2a }

Scripting with Python

Scripts can be written in Python as of SerialEM 3.9.0, 3/24/21, and run scripting commands exposed in the module 'serialem'.  Python and regular scripting can even be used together to some extent.

Initial lines and other special features:  A python script must start with '#!Python', plus an optional version specification if there are multiple installed pythons.  You can include lines that are blank or consist of just '#' and/or space before the '#!Python' line.    If you start with '#!Python' it will use the first available version entered with a 'PathToPython' property, or just try to run it as 'python.exe' without a path if you have no property entries. If you start with, e.g., '#!Python3.6', you must have a 'PathToPython' property for that version.  Here are other special features that you need to be aware of:

• The script must import the serialem module in order to call any SerialEM scripting commands.  This can be done in three ways:
    import serialem
    import serialem as sem      (or other alias of your choice)
    from serialem import *
  The recommended programming practice is to use 'import serialem' with a convenient alias like 'sem', although in a complicated script using other modules it might be better not to have an alias.  The third form should probably be reserved for quick, private scripting.
• In order for the script editor to handle Python optimally, you must have either a line importing serialem, or a '#serialemPrefix' line, in the initial lines of the script (see the section on Behavior for Python scripts).  Specifially, the editor scans the initial lines for import lines and special comments; this scanning stops at the first non-blank line that is neither an import nor a comment.  If it encounters the serialem import line, it uses that to determine the prefix required when completing script commands (i.e., 'serialem', the alias such as 'sem', or nothing for the three forms shown above).  If you do not want to import serialem until later, or not at all (for a script with just functions), you specify this prefix in the initial lines instead with a line like
     #serialemPrefix sem
• A script name, if desired or needed, can be entered in a comment anywhere in the script:
    #ScriptName name
  The older 'MacroName' works as well.  One of these entries is needed in a script to be included in another script.  '#LongName' can also be included to specify a long name for the script.
• Other Python scripts within SerialEM can be included at any point in the script, specified by their names:
  #include name
  The other scripts should all start with '#!Python' and a '#serialemPrefix' or serialem import line so that the editor behaves correctly, even if they are never run on their own.
• Python scripts in separate files can be included at any point in the script with:
  #includeFile filename
  thus allowing you to edit scripts in an external editor of your choice and run them immediately within SerialEM.  These points apply:
  • 'includefile' is also recognized
  • 'filename' can be either the full path to the file or just the name of a file in either the current directory or one of the directories on a path list in the property 'PythonIncludePath'.
  • Like a Windows executable path, the list in 'PythonIncludePath' can have multiple directories separated by semicolons (but extra spaces must be avoided). The current directory is searched before the directories on the path list.
  • The filename can include a path relative to the current directory or one of the directories in the path list.  The program simply appends 'filename' after each path in the list to look for the file.
  • Python exception reporting will show the line number within the file but not the text of the line.
  • There is no need for #!Python or a serialem import line unless you plan to edit within SerialEM also.
• The script sent to Python consists of some initial lines, your script indented in a 'try' block, and exception handlers at the end.  The additional lines are referred to as the "script wrapper" below.

You can of course import any other modules available within the Python installation; these imports can be anywhere in the script.

Backslashes and escape characters: Before sending the script to Python, the backslash character, '\', is replaced by '\\' so that filenames are preserved and not misinterpreted as containing escape characters like '\f' and '\x'.  In order to include escape characters like '\n' in your script, you need to prefix them with another character, '|'.  Thus, the line ending to print a multi-line string to the log would be '|\r|\n'.  For convenience, the prefix character is not needed for these strings consisting solely of the line endings surrounded by quotes:
  '\n'
  "\n"
  '\r\n'
  "\r\n"
Thus, it is possible to split a SerialEM array variable on its separator with
  var.split('\n')
or assign the line ending to a Python variable with
  eol = '\r\n'
  Also, the prefix is not needed for an escaped newline '\n' (without the quotes) in a line with the command 'EchoBreakLines(', so this will work the same as in SerialEM scripting.

Calling script commands as functions: About 90% of SerialEM's script commands are accessible from the serialem module by calling them as functions, with the arguments in the same order as used in SerialEM scripting and shown in the help.  The name must be the mixed case command name shown in the help and provided when using completion in the script editor.  Required arguments come first, followed by optional ones.  For example:
  sem.Autoalign()
  sem.AlignTo('P')
  sem.AlignTo('P', 1, 1)
  sem.CropImage('A', 100, 700, 200, 600)
  sem.FocusChangeLimits(-10., 5.)
  (isx, isy) = sem.ReportImageShift()

There is type checking in the function calls: string arguments must be strings, integer arguments must be integers, and floating point arguments must be numeric (with or without a decimal point).  A failure to have the right types will lead to an exception and a traceback in the SerialEM log.  Any argument that the help indicates must be or can be text needs to be passed as a string.  It should be fairly obvious from the help which arguments are integers: they are indexes of various kinds, simple 0 or non-zero entries to specify options, or more elaborate enumerated entries.  If in doubt, consult https://bio3d.colorado.edu/ftp/SerialEM/Scripts/PythonArgTypes.txt.

The return value from a script is either None, if it sets no reported values, a single value if there is one reported value, or a tuple with all of the reported values that were set.  It should be clear from the command documentation how many reported values to expect for given arguments to the command.  Commands that report one value when there is a failure of some kind or more values when there is not will always return a tuple for consistency.  Numeric values are returned as floating point, other values are returned as strings.  If it is not clear from the help which to expect, test before using, such as with 'if isinstance(val[0], str):'.  Reported values are cleared before the next command, unlike in SerialEM scripts, so if you need to fetch one after a command, it can only be done with 'GetVariable' immediately after the command.  Other variables set by commands such as 'ReportNavItem' can also be obtained with 'GetVariable', which will return a floating point value for values designated as numeric when they were stored.

There are two exceptions to the match between arguments: One is 'Exit'.  From SerialEM, this takes an optional string which is printed to the log.  From Python, the first argument is an optional integer which should be positive when calling from an exception, 0 otherwise, and the second argument can be a multi-line text string.  Lines should be separated by '|\r|\n' to print properly in the log.  The second exception is 'CallFunction', where in regular scripting a function call takes set of numeric arguments followed by an optional string argument.  From Python, the string argument must come first, followed by the numeric ones.  The string argument is ignored inside SerialEM if the function does not expect one.

Printing to the log window:  The 'Echo' function in the module can be used to print a single string to the log window.  In addition, the standard output of the script is read through a pipe and written to the log as it becomes available.  Thus you can use the standard Python 'print()' function (or 'print' command in Python 2.7). However, this output will be delayed and may come out later than SerialEM's own output to the log unless the script output is flushed.  All of the 'print()' statements in your script will be converted to 'SEMprint()', a Python function defined in the script wrapper that calls the standard print function with its arguments plus 'flush = True.  This will not help if you call into another module that uses 'print()', do output with 'sys.stdout.write', or are using Python 2.7.  For those cases, the script wrapper defines 'SEMflush = sys.stdout.flush' so you can write 'SEMflush() to do the flushing.  You need not flush after every print; you can flush once just before a command where some SerialEM output is expected.

Accessing SerialEM script variables:  Three module functions allow variables to be set within the SerialEM script processor: 'SetVariable', 'SetPersistentVar', and 'SetFloatVariable'.  The module function 'GetVariable' will return a variable value, which can be either a string or a float.  A one-dimensional array variable is stored in SerialEM as a string with values separate by newlines, which would have to written as '|\n'.  There are three useful Python functions defined in the script wrapper:

• SEMarrayToFloats(var): Takes a variable returned by 'GetVariable' with one or more numeric values and returns a list of float values; thus it should work with any numeric return from 'GetVariable', not just an array.  This will throw an exception if 'var' is neither a string nor a float or if any of the elements in a string do not convert to a float.
  Example: if 'm' has been built in regular scripting by adding a series of mean values 12.3, 13.2, 11.5, 12.7 with 'AppendToArray' use
          ret = SEMarrayToFloats(sem.GetVariable('m'))
  and 'means' will be a list [12.3, 13.2, 11.5, 12.7].
• SEMarrayToInts(var): Takes an array variable returned by 'GetVariable' and returns a list of integer values.  This will throw an exception if 'var' is not a string or if any of the array elements is not an integer, so it is only useful for array variables where you know that integer values have been stored.
  Example: if 'a' is defined with 'a = { 1 2 3 4 }' in a regular script, use
          ret = SEMarrayToInts(sem.GetVariable('a'))
  and 'ret' will be a list [1, 2, 3, 4].
• listToSEMarray(list): Takes a list of numbers and/or strings and packs them a single string separated by newlines, which can be supplied as the argument to one of the 'SetVariable' module functions.
  Example: if 'names' is a list in Python with [0, 'View', 1, 'Focus', 2, 'Trial'], use
          SetPersistentVar('names', listToSEMarray(names))
  to create a persistent 1-D array in regular scripting with those entries.

If you have an array of strings instead of numbers, split the string returned by 'GetVariable' on newlines:  strArr = ret.split('|\n')

Errors and exceptions:  An error within a SerialEM operation will generate an exception of type 'SEMerror'.  The namespace is optional here because the script wrapper imports this name.  You can catch this exception in a 'try - except' block.  The message from the error should be available if you use 'except SEMerror as err' and convert with 'str(err)'. If appropriate, you can deal with the problem and let the script continue, but a message box with SerialEM's text will come up as usual unless you first call 'NoMessageBoxOnError'.  If you do call this, there will be the same kind of 'SCRIPT ERROR' output to the log as when an error occurs within a 'Try - Catch' block in SerialEM scripting.

An error within the serialem module will generate an exception of type 'SEMmoduleError', also imported by this name.  You can catch this and try to interpret the error string if appropriate.

You can also catch Python exceptions and handle them.  If you do not catch these or the SEMerror or SEMmoduleError exceptions, they will be caught by the script wrapper and result in script termination plus a traceback or message in the log.  Parsing errors in the script will also be reported in the log.  If the script with an error is open in an editor window, the program will attempt to show the line there just as when using the To Line button.  The highlighting of the start of the line will not work if the script is run with the Run button; if you run it another way, click on the editor's title bar and the highlighting should show up then.

'sys.exit' can be used if desired instead of serialem.Exit(); it will be intercepted and terminate the script through Exit().

On some systems, starting a Python script will often or always give this message in the log:
   'WARNING: Error 5 occurred assigning python processes to job object'
because of some unfathomable difference in Windows permissions on these systems.  The only consequence is that if SerialEM crashes, the Python process will not be killed and will be left running in the background.  This may interfere with starting a new script, in which case it is necessary to find the process in Task Manager and kill it.

Calls between script types and embedded python scripts:  It is possible to call a Python script from a regular SerialEM script and vice versa.  These are the specific rules:

• A Python script can call a regular script or function provided that the Python script was not itself called from a regular script.  The script or function being called can call functions and other regular scripts.  This script and other ones that it appears to call will be checked before being run.
• A regular script can call a Python script provided that the regular script was not called, directly or indirectly, from a Python script; i.e., the top-level script was not Python.  The script being called cannot call any other scripts.  If 'CallScript' or 'CallMacro' is used with the script number, this number can be followed by arguments that will be available in the Python script as a list of strings named SEMargStrings.  This list will always be defined, but may be empty.  Although arguments cannot be passed when using 'Call' and the script name, you can still use a name by first looking up the script name with 'FindScriptByName' then using 'CallScript -1'.
• A Python script cannot call another Python script; only '#include' can be used to access more than one script.
• The call to a Python script from a regular script can be contained in a 'Try - Catch' block.  Execution errors can be caught this way, but syntax errors will end the script.
• The call to a regular script from a Python script can be contained in a 'try - except' block to catch errors as usual.

Additionally, it is possible to embed a block of Python code within a regular script, starting with the line 'PythonScript' and ending with 'EndPythonScript'.  The 'PythonScript' line can contain optional arguments that will be available in the list of strings 'SEMargStrings', just as for 'CallScript'.  This mechanism is particularly useful if you want to have a single script with numerous Python functions available from regular scripting.  If you have such a script, e.g., 'PyFuncs', you can do this is in a regular script:
     PythonScript 3 $ISX $ISY
     #
     #!Python
     import serialem
     #include PyFuncs
     functionInPyFuncs(SEMargStrings)
     EndPythonScript

You could also make a regular script with wrapper functions to simplify the calling of a function needed from multiple places:
     ScriptName PyWrappers
     #
     Function functionInPyFuncs 3 0
     PythonScript $argVal1 $argVal2 $argVal3
     #
     #!Python
     import serialem
     #include PyFuncs
     functionInPyFuncs(SEMargStrings)
     EndPythonScript
     #
     EndFunction

and from a regular script you would just need
     CallFunction PyWrapper::functionInPyArgs 3 $ISX $ISY

Installation options:  There must be at least one Python installation, but can be multiple ones.

The PythonModules directory is now included in the SerialEM installation package and unpacked in the directory with the SerialEM executable.  To use a command in a SerialEM executable newer than the install package, you can obtain an up-to-date set of modules from the SerialEM download page.  If you unpack this anywhere other than the SerialEM directory, the property 'PythonModulePath' must be defined with the path to this directory of modules.  This package contains modules for 32-bit Python 2.7, 3.4, and 3.5, and 64-bit Python 3.4, 3.5, 3.6, 3.8, 3.9, 3.11, and 3.12.

If you have one Python on the system path that is 3.5 or higher, you are all set.  If you have multiple Pythons, Python is not on the system path, or the version is 3.4 or 2.7, you must have one or more 'PathToPython' property entries.  These should have the following form (changed for your actual path to the Python in each case):
     PathToPython 3.4-64 C:\Program Files\Python34
     PathToPython 3.5-64 C:\Program Files\Python35
     PathToPython 3.6 C:\Program Files\Python36
     PathToPython 3.8 C:\Program Files\Python38
     PathToPython 3.9 C:\Program Files\Python39
     PathToPython 3.11 C:\Program Files\Python311
     PathToPython 3.12 C:\Program Files\Python312
     PathToPython 2.7 C:\Program Files (x86)\Python27
     PathToPython 3.4-32 C:\Program Files (x86)\Python34
     PathToPython 3.5-32 C:\Program Files (x86)\Python35
See the description of PathToPython for more details.  These specifications will allow the modules to be found in the distributed module directory.

Some Python packages are available on the IMOD/SerialEM web site:

• 32-bit Python 2.7
• 32-bit Python 3.4
• 64-bit Python 3.4
• 64-bit Python 3.8

For older Python, click on the downloaded Python file to install it; it installs without using 'Run as Administrator'. For Python 3.8, right click on the file and select 'Run as Administrator'.  Python does not need to install in the default location that it offers; you can put it in locations like the ones above (e.g., set the install location to 'C:\Program Files\Python34').  Check the option to add Python to the PATH if one is present, and you might as well let it disable the path length limit if that is offered.  If you customize the installation to leave out unneeded components (e.g., Documentation, tcl/tk, and the test suite), be sure to keep pip if that is listed.

To use pip and install packages into the Python installation's 'site-packages', you will need to open a command prompt window as administrator.  If you put Python on the path, you can just use the command 'pip'; otherwise you can run it from the Python Scripts directory, e.g., give the command 'C:\Program Files\Python34\Scripts\pip'.

Running scripts in an externally run Python:  An externally run Python script can take control of SerialEM and execute script commands.  This can be done either from an interactive Python window or from a script run at the command line.

• The appropriate serialem module must be on the Python path.  This can be done either by setting the environment variable PYTHONPATH (or adding to it if it is already set) with the path to the module, or by adding it to the system path within the script, such as with
       sys.path.insert(0, 'C:\Program Files\SerialEM\PythonModules')
  where you would add '2.7', '3.4-32', or '3.4-64' for those versions.
• The SerialEM property 'EnableExternalPython' must be set non-zero to allow connections.  This property can be set with the 'SetProperty' script command, so the connections can be enabled or disabled during a session.
• By default, the module will try to connect to SerialEM on port 48888 in the local host (IP 127.0.0.1).  You can modify this is in two different ways: by setting environment variables PY_SERIALEM_PORT and/or PY_SERIALEM_IP with a different port or IP address, respectively; or by calling the module function ConnectToSEM(port, IP_address) as the first function call to the module.  'Port' is an integer and 'IP_address' is a string.  Both arguments are optional, so you can call it with just the port.  Values passed through this function call will override the environment variables if any.
•  If you change the port, define it in the SerialEM properties file with the entry 'SocketServerPort 8' plus the alternative port.  Do not use 48887 or 48889 - they are dedicated for running Python scripts from SerialEM.
• The first actual function call to SerialEM is considered to start a script, and it will leave the program waiting for another command, with 'EXTERNAL CONTROL' in the status bar.  The program stays in this state until an Exit command is received or the external Python exits (i.e., when a script ends or an interactive Python is closed). If the program is busy when the first command it given, it will raise a SEMmoduleError exception with text indicating that SerialEM 'is now busy'.  If you call Exit with no arguments or with a 1, it will not raise a SEMexited exception; call with a 0 to have this exception be raised, such as if you want to catch it from multiple places.
• After the first command is received, the STOP button will be enabled and can be used to regain user control of the program.  If this happens, SerialEM will return an error code on either the command that was being processed during the STOP, or on the next command from Python, if the script was doing something else when the button was pressed.  The module will raise a SEMexited exception with the message 'User STOP' instead of the usual  'Normal exit'.
• Do not try to run two operations at once.  If you are working in multiple panes of a Jupyter notebook, be sure a command is completed before running one in another pane.

For connecting from a Windows machine, you can use the modules in the PythonModules package.  The module will also build and work on Linux and Mac OS.  The source code is in a mercurial repository at https://bio3d.colorado.edu/SerialEM/PythonModule/ which you can clone with

     hg clone https://bio3d.colorado.edu/SerialEM/PythonModule/

The module is built by running the desired Python with this command:

     python setup.py build

and the module is left in a 'lib...' subdirectory of 'build'.  The module needs to be rebuilt to access any new commands added to 'MacroMasterList.h'; you can update it with 'hg pull -u'.  Depending on demand, a system may be set up to provide updated modules automatically for some set of Python versions on Linux and Mac.