Parallel Writing Module

This module contains a set of functions for writing in parallel to a file from multiple processes. The functions shown first apply generally or when a file is opened as an ImodImageFile, such as with iiFOpen; the later functions apply when a file is opened for unit-based I/O. For MRC files, data are written directly to a file from multiple processes with no locking, but also lines near defined boundaries are written to separate boundary files, which must be rewritten at the end with Fixboundaries. For HDF files, a file lock is set up and the image file must be opened and closed for each use, but there are no boundary files written.

Parallel writing to an HDF file requires a specific sequence of operations. When the file is first created, it needs to have at least one dataset (section) defined if not written; the routine iiuWriteDummySecToHDF or hdfWriteDummySection can be used to do this without having to write or supply data. When an existing file needs to be opened initially for writing, these are the steps:

  1. Whether a file is HDF must be tested with iiTestIfHDF.
  2. Initialization with parWrtInitialize or iiuparWrtInitialize must be done with a negative X dimension and any existing file that can be used for locking (the boundary file is fine for this).
  3. The internal index of the lock file must be obtained by calling parWrtProperties.
  4. The file lock must be obtained with b3dLockFile.
  5. The file can then be opened as usual, but then the underlying HDF file should be closed immediately with parWrtRecloseHDF or iiuParWrtRecloseHDF.
  6. Writing proceeds as usual with the section or line routines below, and this module will buffer the data and reopen and close the file as needed.
  7. Before exiting, the program must write all buffered data with parWrtFlushBuffers or iiuParWrtFlushBuffers, then terminate all the locks with parWrtClose.

Header to include: iimage.h or iiunit.h


int parWrtInitialize(const char *filename, int nxin, int nyin)
int parWrtProperties(int *allSec, int *linesBound, int *nfiles)
int parwrtproperties(int *allSec, int *linesBound, int *nfiles)
int parWrtSetCurrent(int index)
int parwrtsetcurrent(int *index)
void parWrtClose()
void parwrtclose()
int parallelWriteSlice(void *buf, FILE *fout, MrcHeader *hdata, int slice)
int parWrtToHDFchunk(ImodImageFile *iiFile, MrcHeader *hdata, void *buf, int section)
int parwrtgetregion(int *regionNum, char *filename, int *sections, int *startLines, fortStrLen_t strlen)
int parWrtRecloseHDF(ImodImageFile *iiFile, MrcHeader *hdata)
int parWrtFlushBuffers(ImodImageFile *iiFile, MrcHeader *hdata)
int iiuParWrtInitialize(const char *filename, int iunitBound, int nxIn, int nyIn, int nzIn)
int parwrtinitialize(char *filename, int *iunitBound, int *nxIn, int *nyIn, int *nzIn, fortStrLen_t namelen)
void parWrtPosn(int iunit, int iz, int iy)
void parwrtposn(int *iunit, int *iz, int *iy)
int parWrtSec(int iunit, void *array)
int parwrtsec(int *iunit, void *array)
int parWrtLin(int iunit, void *array)
int parwrtlin(int *iunit, void *array)
int iiuParWrtSecPart(int iunit, void *array, int nxdim, int ixStart, int indX0, int indX1, int iyStart, int iyEnd)
int iiuParWrtRecloseHDF(int iunit, int writeHeader)
int iiuparwrtreclosehdf(int *iunit, int *writeHeader)
void iiuWriteDummySecToHDF(int iunit)
void iiuwritedummysectohdf(int *iunit)
int iiuParWrtFlushBuffers(int iunit)
int iiuparwrtflushbuffers(int *iunit)

int parWrtInitialize(const char *filename, int nxin, int nyin)

Initialize parallel writing to an image file by reading in the boundary info file whose name is filename, and storing various values in static variables. filename can be NULL or an empty string, in which case the count of parallel writing files is incremented and this info file index is marked as having no info file.  nxin and nyin specify the dimensions of the images.  Pass the negative of nxin to indicate that the file is an HDF file, in which case the file specified by filename will not be read and will simply be used as a lock file.  The format of the info file is to start with a line with these entries:
Version  type  nx  numLines  numFiles  
version = 1 for now  
type = 0 for chunks in Z, 1 for chunks in Y that require boundaries on each section  
nx = X dimension of image file  
numLines = number of lines written at each boundary  
numFiles = number of boundary files  
For each boundary file, there are then two lines:  
Name of boundary file  
Boundary 1 section and starting line, boundary 2 section and starting line
Sections and lines are numbered from 0.  For chunks in Z, the section number should be -1 for no boundary (i.e., for boundary 1 of first chunk and boundary 2 of last chunk).  A line number of -1 indicates that the boundary extends to the end of the section (thus, a setup script does not need to know the size of the images in Y).  This routine will convert a line number of -1 to the appropriate starting line.  
For chunks in Y, the section number is ignored (and should be -1) and the line number should be -1 for no boundary. Returns 1 for failure to open file, 2 for error reading file, 3 for inappropriate values in header line of file, 4 for memory allocation errors, 5 for trying to open too many boundary info files for the array, or 6 for an error opening the given file as a lock file for HDF.

int parWrtProperties(int *allSec, int *linesBound, int *nfiles)

Returns properties read from the current boundary info file.  If the file not HDF, it returns allSec 1 if chunks are in Y, linesBound with the number of lines in each boundary, nfiles with the number of boundary files.  If the file is HDF, the index to the lock file is returned in allSec, X size in linesBound, and Y size in nfiles. These values are all zero if there was no boundary info file at the current index. Returns 1 for parallel writing not initialized or -1 for an HDF file.

int parwrtproperties(int *allSec, int *linesBound, int *nfiles)

Fortran wrapper for parWrtProperties

int parWrtSetCurrent(int index)

Sets the index of the current boundary info file or file for parallel writing to index, numbered from 0.  Nearly every function in this module applies to the current file set by this index, so if there are multiple files open for parallel writing, this function must be called before any operations on a different file from the previous one.  Returns 1 for index out of range.

int parwrtsetcurrent(int *index)

Fortran wrapper for parWrtSetCurrent, with index numbered from 1.

void parWrtClose()

Closes any lock files opened for parallel writing

void parwrtclose()

Fortran wrapper for ParWrtClose

int parallelWriteSlice(void *buf, FILE *fout, MrcHeader *hdata, int slice)

Writes one Z slice of data at Z = slice from the buffer buf to file fout according to the header in hdata.  If parallel writing has been initialized, lines will be written to a boundary file if appropriate. Returns errors from writing the slice with mrc_write_slice and also returns other non-zero values from opening the boundary file, writing its header, or writing to the file.

int parWrtToHDFchunk(ImodImageFile *iiFile, MrcHeader *hdata, void *buf, int section)

Writes data to one chunk of an HDF file Z = section from the buffer buf to file iiFile, accessing the header in hdata, and following the subarea and subarray information in the llx, urx, lly, ury, padLeft, and padRight members of iiFile.  Returns errors from writing the slice with iiWriteSection and also returns other non-zero values from opening the boundary file, writing its header, or writing to the file.

int parwrtgetregion(int *regionNum, char *filename, int *sections, int *startLines, fortStrLen_t strlen)

Fortran-callable function to get the parameters of parallel writing region number regionNum (numbered from 1) and return the boundary file in filename and the sections and starting lines in arrays sections and startLines.  Returns 2 if writing not initialized, or 1 if the region number is out of bounds.  This is used by Fixboundaries.

int parWrtRecloseHDF(ImodImageFile *iiFile, MrcHeader *hdata)

Closes the ImodImageFile iiFile for an HDF file opened with  iiFOpen and with header data in hdata.  The file must be HDF and be the current parallel file.  Returns 1 for error.

int parWrtFlushBuffers(ImodImageFile *iiFile, MrcHeader *hdata)

Writes out data in HDF buffer for the ImodImageFile iiFile opened with iiFOpen with header data in hdata.  The file must be HDF and be the current parallel file.  Returns 1 for error.

int iiuParWrtInitialize(const char *filename, int iunitBound, int nxIn, int nyIn, int nzIn)

Initializes the parallel writing routines for use with unit-based I/O.  The name of the boundary info file should be in filename, which can be empty.  An available unit for opening image files should be in iunitBound, the unit number for the image file to be written in iunitOut and the dimensions of the image file in nxin, nyin, nzin. Returns the return value from the initialization routine parWrtInitialize

int parwrtinitialize(char *filename, int *iunitBound, int *nxIn, int *nyIn, int *nzIn, fortStrLen_t namelen)

Fortran wrapper for iiuParWrtInitialize. Also returns 7 for failure to convert Fortran string.

void parWrtPosn(int iunit, int iz, int iy)

Positions unit iunit for writing at section iz, line iy, numbered from 0.

void parwrtposn(int *iunit, int *iz, int *iy)

Fortran wrapper for parWrtPosn

int parWrtSec(int iunit, void *array)

Writes an entire section from array into unit iunit at the current position and writes boundary lines if appropriate

int parwrtsec(int *iunit, void *array)

Fortran wrapper for parWrtSec

int parWrtLin(int iunit, void *array)

Writes one line from array into unit iunit at the current location and writes it to a boundary file if appropriate

int parwrtlin(int *iunit, void *array)

Fortran wrapper for parWrtLin

int iiuParWrtSecPart(int iunit, void *array, int nxdim, int ixStart, int indX0, int indX1, int iyStart, int iyEnd)

Writes data to one or more chunks of an HDF file from a potential subarray of array at the current line and section of the file open on unit iunit. The X dimension of array is given in nxdim, the starting and ending index at which to write the lines in X in indX0 and indX1, the starting index from which to take data in X in ixStart, and the starting and ending indices from which to take data in Y in iyStart and iyEnd (inclusive, numbered from 0). Returns errors from writing to the slice with iiuWriteSecPart and also returns other non-zero values from storing the data or reopening the file.

int iiuParWrtRecloseHDF(int iunit, int writeHeader)

Closes the ImodImageFile that is open on unit iunit if it is an HDF file, and writes the header first if writeHeader is non-zero.  Returns 1 if a file was closed, 0 if not, and exits on error.

int iiuparwrtreclosehdf(int *iunit, int *writeHeader)

Fortran wrapper for iiuParWrtRecloseHDF

void iiuWriteDummySecToHDF(int iunit)

Adds a dataset for section 0 to the HDF file open on unit iunit without writing any data.  Exits on error.

void iiuwritedummysectohdf(int *iunit)

Fortran wrapper for iiuWriteDummySecToHDF

int iiuParWrtFlushBuffers(int iunit)

Writes output buffers for the the current file open for parallel writing, which must be an HDF file.  Returns 1 if it is not an HDF file or if there is an error writing data

int iiuparwrtflushbuffers(int *iunit)

Fortran wrapper for iiuParWrtFlushBuffers