Package PIRL.Conductor

The PIRL Conductor package provides an application to manage procedure pipelines.

See:
Description

Interface Summary

Management	Management is the Conductor management interface.
Processing_Listener	A Processing_Listener specifies the interface to receive Conductor Processing_Event notifications.

Class Summary

Colors
Conductor	Conductor is a queue management mechanism for the sequential processing of data files.
Evaluate	Evaluates a mathematical expression.
Manager	A Manager is a GUI for managing a Conductor.
Native_Methods	Native methods.
Notify	Notify is a mechanism for delivering messages to destinations.
Pipeline_Source_Manager	The Pipeline_Source_Manager manages unprocessed source records in a Conductor pipeline Sources table.
Procedures_Table	A Procedures_Table is a table view of a Procedures_Table_Model.
Procedures_Table_Model	A Procedures_Table_Model contains a table of Conductor procedure definition records.
Processing_Changes	Processing_Changes contains information provided by a Conductor about changes in its processing conditions.
Processing_Event	A Processing_Event is used to notify a Conductor manager that a Conductor processing event occured.
Reference_Resolver	A Reference_Resolver resolves embedded references in text strings.
Sources_Table	A Sources_Table is a table view of a Sources_Table_Model.
Sources_Table_Model	A Sources_Table_Model contains a table of Conductor Source records.
Stream_Logger	A Stream_Logger is a Thread that is used to forward the lines read from an InputStream to one or more Writers.
String_Vector_Comparator	A String_Vector_Comparator implements the Comparator interface to sort Lists of Objects.
Table_Mouse_Listener	A Table_Mouse_Listener provides deselect capability for table selections.

Exception Summary

Unresolved_Reference

The Exeception that is thrown if a parameter source reference or database field reference can not be resolved and a Default_Value has not been provided.

Package PIRL.Conductor Description

The PIRL Conductor package provides an application to manage procedure pipelines.

A procedure pipeline is defined by a Database table of Procedures definition records paired with a table of file Sources records. Each file specified in a Sources record is processed in sequence by each procedure defined in a Procedures record. Procedure definitions include the relative order of the procedure in the pipeline, its command line to be executed, the successful exits status or message, the maximum amount of time a procedure will be allowed to run, and a command line to be executed if the procedure does not complete successfully. The Sources records include, in addition to the pathname of the file to be processed, a record of the completion status of each pipeline procedure applied to it.

Each source file processed is provided with a log file that is tied by a unique name to is Sources record. The log includes a detailed description of all procedures executed, their stdout and stderr output, and a record of how the procedure completed.

All procedure definitions may include embedded references to Database fields or Conductor configuration parameters. These are effectively variable names that are replaced with the values to which they resolve. A reference to a Database field resolves to the value of the field in the table of a catalog named in the reference for the record selected by a conditional expression. A reference to a configuration parameter resolves to the value of user specified parameter name in the current configuration file or any of the implicit parameters describing the current state of source processing. References may contain nested references and may resolve to values that are themselves references. This allows procedure definitions to be described in terms of variables that are dynamically resolved to specific values at the time of procedure execution.

Conductor manages the acquisition of Sources records for processing to ensure that only one Conductor will process each record, and the source will only be processed once unless specifically reset for reprocessing by some other agent. It is possible to set Sources to be (re)processed starting in the middle of the Procedures sequence. The exclusive control of a Sources record by one Conductor does not preclude multiple entries of the same file into a Sources table, if appropriate. It does enable more than one Conductor, on the same or separate host systems, to process the same pipeline at the same time; for example to safely distribute and parallelize processing.

Conductor offers a monitor mode that includes user controlled starting and stopping of processing and a terminal-like scrolling text display of all log output. Conductor can also be run silently for batch processing by manual initiation, as a cron job, or as a daemon that will constantly poll its pipeline for new Sources entries to be processed.

See Also:

PIRL.Database