public class Merge_Reprocessed_Data
extends java.lang.Object
This program requires specifying one database server, one source catalog, one destination catalog, a list of tables and a list of orbit ranges. Entries will then be copied from one database to the other by querying for the table metadata and stripping away specific excluded fields. See the usage information for what items can be specified in a configuration file and how to specify them.
This program uses the Database Class to obtain information from the source catalog about what needs to be copied and then it uses the Update_DB Class to copy the data into the destination catalog. It does this for all of the tables in the specified list of tables.
When inserting rows into the destination catalog this program will not include data for fields that are blank or NULL. Instead it will skip over that field entirely. So if the data query returns that the "Software_Name" field contains "" it will not include that field in the Insert command sent to the database through Update_DB. Also if the "Software_Name" field contains "NULL" it will not include that field in the Insert command.
This program will only copy over products that have a higher version in their standard table than the version in their proffered table. For instance, a product X in EDR_Products with a version of 2 will only be copied if product X in the Proffered_EDR_Products table has a version lower than 2. Also, when a product is copied it will overwrite what is in the existing catalog if there is already a product there with that version and Product ID. Otherwise it will simply be inserted into the database.
The Observation_Geometry and
A log file for this program can be specified. It can be specified on the
command line with -L or in the Configuration file with the appropriate
parameter. See Usage information for more information.
The command line usage
description provides details
on the application operation.
PIRL.Database
,
Query_DB
,
Update_DB
,
PIRL.Configuration
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
CONFIG_GROUP
The configuration group to pull parameters from
|
static java.lang.String |
CONFIG_OBS_GEO_GROUP
The sub-group to pull observation geometry max change values from
|
static java.lang.String |
CONFIG_PARAM_DESTINATION_CATALOG
The parameter in the configuration file for the
destination database/catalog.
|
static java.lang.String |
CONFIG_PARAM_LOG_PATHNAME
The parameter in the configuration file for the log file name
|
static java.lang.String |
CONFIG_PARAM_SOURCE_CATALOG
The parameter in the configuration file for the
source database/catalog.
|
static java.lang.String |
CONFIG_PARAM_TABLES
The parameter in the configuration file for the
list of tables.
|
static boolean |
DEBUG
Debugging option
|
static java.lang.String |
DEFAULT_ANAGLYPH_PRODUCT_FIELD
The field that should contain the product id to join on
for the Anaglyph products.
|
static java.lang.String |
DEFAULT_ANAGLYPH_PRODUCT_TABLE
The default Anaglyph_Product table.
|
static java.lang.String |
DEFAULT_CONFIGURATION_FILENAME
The default Configuration filename.
|
static java.lang.String |
DEFAULT_EDR_OBSERVATION_FIELD
The field that should contain the observation id to join on
for the EDR products.
|
static java.lang.String |
DEFAULT_EXCLUDED_FIELDS
The default fields to exclude from the copy
|
static java.lang.String |
DEFAULT_LEFT_OBS_ID
The default Left Observation for the Anaglyph
|
static char |
DEFAULT_MAX_CHANGE_DELIMITER
The delimiter to separate max change value field names from
their values on the command line
|
static java.lang.String |
DEFAULT_OBSERVATION_FIELD
The field that should contain the observation id to join on
|
static java.lang.String |
DEFAULT_OBSERVATION_GEOMETRY_TABLE
The default Observation_Geometry table.
|
static char |
DEFAULT_PATH_DELIMITER
The parameter path delimiter
|
static java.lang.String |
DEFAULT_PRODUCT_ID_FIELD
The field that should contain the product id to check
|
static java.lang.String |
DEFAULT_RIGHT_OBS_ID
The default Right Observation for the Anaglyph
|
static char |
DEFAULT_SEPARATING_TOKEN
The orbit range separating token.
|
static java.lang.String |
DEFAULT_SOURCE_ID_FIELD
The field that should contain the source id for the
sources tables.
|
static java.lang.String |
DEFAULT_SOURCES_LOG_PATHNAME_FIELD
The field that should contain the log pathname for the sources table.
|
static java.lang.String |
DEFAULT_SOURCES_TABLE_IDENTIFIER
The string identifier for sources tables.
|
static java.lang.String |
DEFAULT_VERSION_FIELD
The field that should contain the version number to check
|
static int |
EXIT_CONFIGURATION_PROBLEM
Application exit status when there is a Configuration file problem.
|
static int |
EXIT_DATABASE_ERROR
Application exit status when there is a problem accessing the Database.
|
static int |
EXIT_INSERTION_FAILURE
Application exit status when one or more Database insertions failed.
|
static int |
EXIT_INVALID_COMMAND_LINE_SYNTAX
Exit status when invalid command line syntax was encountered.
|
static int |
EXIT_LOGGING_FAILURE
Application exit status when the application fails at logging something.
|
static int |
EXIT_PROCESSING_FAILURE
Application exit status when the application fails at processing something.
|
static int |
EXIT_QUERY_FAILURE
Application exit status when either the query for table metadata or
the query for product information failed.
|
static int |
EXIT_SUCCESS
Success exit status.
|
static java.lang.String |
ID
The Class identification with revision number and date.
|
static java.lang.String |
NL
System new line sequence.
|
static java.lang.String |
SERVER
The parameter name for the default database server: "Server".
|
static java.lang.String |
TYPE
The parameter name for the type of database server which
corresponds to the Data_Port class to use: "Type".
|
Constructor and Description |
---|
Merge_Reprocessed_Data() |
Modifier and Type | Method and Description |
---|---|
static java.lang.String |
Change_Source_Number(java.lang.String pathname,
java.lang.String old,
java.lang.String updated)
modify the source number of a source pathname for en entry in the sources table
|
static java.lang.String |
Create_Key(java.lang.String table,
java.util.Vector<java.lang.String> row,
java.util.Vector<java.lang.String> fields,
java.util.Vector<java.util.Vector<java.lang.String>> max_change_values,
boolean force,
java.lang.String prefix)
Create a mysql where clause that checks specified value ranges
|
static java.lang.String |
Create_Query(java.lang.String table,
java.util.Vector<java.lang.String> fields,
java.util.Vector<java.util.Vector<java.lang.Integer>> orbit_ranges,
java.lang.String key)
build a query based on a fields vector
|
static java.lang.String |
Default_Server_Name(PIRL.Configuration.Configuration The_Configuration)
Gets the default server name from the Database Configuration.
|
static boolean |
isSourcesTable(java.lang.String table)
Check if a table is a sources table
|
static void |
main(java.lang.String[] arguments)
Run the Merge_Reprocessed_Data application.
|
static java.util.Vector<java.lang.Integer> |
parse_orbit(java.lang.String orbit)
Parse an orbit range from a String.
|
static java.util.Vector<java.lang.String> |
Run_Command(java.util.Vector<java.lang.String> args,
boolean verbose)
Runs a command line program from a Vector
|
static java.util.Vector<java.util.Vector<java.lang.String>> |
Run_Query(java.lang.String query,
PIRL.Database.Database database,
boolean verbose)
run a query for data and return the data as a vector
|
static java.util.Vector<java.lang.String> |
String_Split(java.lang.String s,
char delim)
Splits a string into multiple strings on a specified delimiter
|
static java.lang.String |
Table_To_Observation_ID(java.lang.String table)
figure out what the observation field name is
|
static java.lang.String |
Table_To_Proffered(java.lang.String table)
figure out what the proffered table name is
|
static void |
Update_Table(java.lang.String table,
java.util.Vector<java.util.Vector<java.lang.Integer>> orbit_ranges,
java.util.Vector<java.util.Vector<java.lang.String>> data,
java.util.Vector<java.lang.String> fields,
PIRL.Database.Update_DB updater,
PIRL.Database.Database update_db,
java.util.Vector<java.util.Vector<java.lang.String>> max_change_values,
boolean verbose)
Update a table with the information found in a data query
|
static void |
Usage()
Command line usage syntax.
|
public static final java.lang.String ID
public static final boolean DEBUG
public static final java.lang.String SERVER
public static final java.lang.String TYPE
public static final java.lang.String CONFIG_GROUP
public static final java.lang.String CONFIG_OBS_GEO_GROUP
public static final java.lang.String DEFAULT_OBSERVATION_GEOMETRY_TABLE
public static final java.lang.String DEFAULT_ANAGLYPH_PRODUCT_TABLE
public static final java.lang.String DEFAULT_EXCLUDED_FIELDS
public static final java.lang.String DEFAULT_VERSION_FIELD
public static final java.lang.String DEFAULT_PRODUCT_ID_FIELD
public static final java.lang.String DEFAULT_LEFT_OBS_ID
public static final java.lang.String DEFAULT_RIGHT_OBS_ID
public static final java.lang.String DEFAULT_OBSERVATION_FIELD
public static final java.lang.String DEFAULT_EDR_OBSERVATION_FIELD
public static final java.lang.String DEFAULT_ANAGLYPH_PRODUCT_FIELD
public static final java.lang.String DEFAULT_SOURCE_ID_FIELD
public static final java.lang.String DEFAULT_SOURCES_LOG_PATHNAME_FIELD
public static final java.lang.String DEFAULT_CONFIGURATION_FILENAME
public static final char DEFAULT_SEPARATING_TOKEN
public static final char DEFAULT_PATH_DELIMITER
public static final char DEFAULT_MAX_CHANGE_DELIMITER
public static final java.lang.String DEFAULT_SOURCES_TABLE_IDENTIFIER
public static final java.lang.String CONFIG_PARAM_TABLES
public static final java.lang.String CONFIG_PARAM_SOURCE_CATALOG
public static final java.lang.String CONFIG_PARAM_DESTINATION_CATALOG
public static final java.lang.String CONFIG_PARAM_LOG_PATHNAME
public static final int EXIT_SUCCESS
public static final int EXIT_INVALID_COMMAND_LINE_SYNTAX
public static final int EXIT_CONFIGURATION_PROBLEM
public static final int EXIT_DATABASE_ERROR
public static final int EXIT_QUERY_FAILURE
public static final int EXIT_INSERTION_FAILURE
public static final int EXIT_PROCESSING_FAILURE
public static final int EXIT_LOGGING_FAILURE
public static final java.lang.String NL
public static void main(java.lang.String[] arguments)
Exit status values:
EXIT_SUCCESS
EXIT_CONFIGURATION_PROBLEM
EXIT_DATABASE_ERROR
EXIT_QUERY_FAILURE
EXIT_INSERTION_FAILURE
EXIT_PROCESSING_FAILURE
EXIT_LOGGING_FAILURE
arguments
- The command line
arguments.public static java.util.Vector<java.lang.Integer> parse_orbit(java.lang.String orbit)
Orbit ranges should be entered in the format of
If an orbit range is entered backwards where the end is
a smaller number than the start this method will flip
the range. As such the range 100-200 is the same as
the range 200-100.
DEFAULT_SEPARATING_TOKEN
orbit
- The string representation of an orbit range.public static java.lang.String Default_Server_Name(PIRL.Configuration.Configuration The_Configuration)
If a
parameter is not found in
the Configuration of the Database the
Server
parameter is obtained. These
parameters must be at the top level of the Configuration.
Type
The_Configuration
- the configuration to find the
default server in.public static java.util.Vector<java.lang.String> String_Split(java.lang.String s, char delim)
This function loops over a given string looking for the delimiter and upon finding that delimiter cuts out a string from the given string from the last point that it found the delimiter. Then the white space is removed from the cut out string and it is placed in a vector of strings to be returned.
If this method is given an empty string or a null string it will return an empty vector.
s
- The string to splitdelim
- The character delimiter to split onpublic static java.lang.String Table_To_Observation_ID(java.lang.String table)
This function is a result of the fact that EDR_Products holds Observation_IDs in the field Commanded_ID while other tables hold it as Observation_ID. This function will return the name of the field that the given table holds an Observation_ID in.
The Anaglyph table Observation ID equivalent is actually the Anaglyph Product ID field.
table
- The table we want information about.public static java.lang.String Table_To_Proffered(java.lang.String table)
This function takes a table name and returns the name of the proffered_*_products table for that table. If none exists it returns null.
table
- The table we want information about.public static java.util.Vector<java.util.Vector<java.lang.String>> Run_Query(java.lang.String query, PIRL.Database.Database database, boolean verbose)
This function takes a standard mysql query as a string
and returns the requested data as a Vector
query
- The table we want information about.database
- The database to query.verbose
- The level to which information is reportedpublic static java.lang.String Create_Query(java.lang.String table, java.util.Vector<java.lang.String> fields, java.util.Vector<java.util.Vector<java.lang.Integer>> orbit_ranges, java.lang.String key)
This function produces a query for data based on provided information. The query will be for data that exists within the specified orbit range.
The query produced will work for all product tables, but not other tables. There is a separate function for Observation Geometry.
table
- The table to generate the query forfields
- The vector of fields to query fororbit_ranges
- The ranges to constrict the query topublic static boolean isSourcesTable(java.lang.String table)
This method checks if a table is a sources table. Sources tables
are marked with the identifier DEFAULT_SOURCES_TABLE_IDENTIFIER
.
table
- The table to check.public static void Update_Table(java.lang.String table, java.util.Vector<java.util.Vector<java.lang.Integer>> orbit_ranges, java.util.Vector<java.util.Vector<java.lang.String>> data, java.util.Vector<java.lang.String> fields, PIRL.Database.Update_DB updater, PIRL.Database.Database update_db, java.util.Vector<java.util.Vector<java.lang.String>> max_change_values, boolean verbose)
This function takes data and then sends it to the destination database as an update.
Nothing is returned by this function. Instead success and failure are reported only when verbose was specified in the command line to this program. Errors are checked within the function.
table
- The table to update in the databaseorbit_ranges
- The ranges of orbits that are being copieddata
- The data to insert into the tablefields
- The fields that correspond to the dataupdater
- The Update_DB object that contains the databaseupdate_db
- The database object for the updatermax_change_values
- the vector that stores the maximum values
by which each field can change.verbose
- The level to which information is reportedpublic static java.util.Vector<java.lang.String> Run_Command(java.util.Vector<java.lang.String> args, boolean verbose)
This method will run a standard command line program with Runtime.getRuntime().exec(). This will never return a null value unless an error occurs. An error is detected when the command being executed outputs anything to standard error. The parameter this method takes should contain the program name followed by any arguments the program should take. The elements of the returned Vector are the output of the program split up into lines. Empty output with no error will return a Vector with one element which is an empty String.
The first entry in the Vector args should be the name of the process that is being run. All entries that follow that entry should be the arguments for the process and should be in the order they would appear on the command line.
args
- The program name and arguments to supply
to the program being run.verbose
- True prints all output of the command while
false prints nothing.public static java.lang.String Change_Source_Number(java.lang.String pathname, java.lang.String old, java.lang.String updated)
This method takes the pathname to change, the old source number and the new source number and replaces the old number with the new one in the pathname. The log pathname must have
pathname
- The string to update.old
- The source number to look for in the pathname.updated
- The source number to place in the returned string.public static java.lang.String Create_Key(java.lang.String table, java.util.Vector<java.lang.String> row, java.util.Vector<java.lang.String> fields, java.util.Vector<java.util.Vector<java.lang.String>> max_change_values, boolean force, java.lang.String prefix)
This method will create the key that will become the where clause to the updater. The key will only check the Observation_ID and Version fields for a standard products table, it will contain those fields and range checks on all fields listed in max_change_values for Observation_Geometry, or it will be null if force is true or if the table is a sources table.
table
- The table to create the key for.row
- The vector that stores the data for the key.fields
- The vector that stores the field names for the key.max_change_values
- The vector that stores the maximum values
by which each field can change.force
- If force is true the returned key will be null.
if force is false the returned key will not be null.prefix
- The string that will be prefixed to the field names
of the key.public static void Usage()
Usage: Merge_Reprocessed_Data [<Options>] [-Orbit Range] <orbit> [...] orbit ranges are specified as- with no spaces where start and end orbit are integers Options - -Configuration <filename> -Server <server name> -CAtalogs <source> <destination> -Tables <name> [...] -Verbose -Help
These are the orbit ranges which will be copied from the source
table to the destination table. These should be entered in the
form:
More than one orbit range can be specified. There must be at least
one orbit range.
N.B. Only the values from the source table with the latest
If the name of the
All non-database-connection values in the configuration should be
placed in the group
For this program
Only one configuration file may be used.
The Configuration file may contain connection information for more
than one database. The information for each database is organized
by
Only one server may be used. Both Catalogs should reside on the
same server.
Two catalogs are needed for this program. The source is the catalog
that values will be copied from. The destination is the catalog that
values will be copied into. The destination and the sourec catalogs
must have different names.
Both catalogs can be specified in the Configuration file under the
server specified in the Configuration. The source can be specified
with the
Using the command line option will override both values in the
Configuration. This means that either both must be specified in
the configuration or both must be specified on the command line.
Tables are entered on the command line as a list. It is assumed
that the both the source and destination catalogs will contain
each table in the list with the same fields between the two
catalogs.
A list of tables can be specified in the Configuration file
under the server specified in the Configuration. The list of
tables should be comma delimited, white space does not matter.
The parameter to specify a source is
Using the command line option will override values in the
Configuration.
Max Change Values are entered on the command line as a list, or in
the configuration file within the
Example: if IMAGE_CENTER_LATITUDE = 0.1 is specified and a record in
the Observation Geometry table of the source catalog has a value of
12.345 then the value that gets copied over must be between 12.445 and
12.245 or an out of bounds error will be reported for that record and
the copy will not occur.
A log file for this program can be specified either on the command line
or in the configuration file. The parameter for the configuration file
is
Prints verbose information to the command line.
This prints the programs ID and all output from sub-programs and
lists the Observation_ID of each product that it attempts to copy.
// TODO:
Not quite sure yet what this will be.
The command line syntax usage is listed and the program exits.
N.B.: If the -Help option occurs anywhere on the command line
no other command line arguments will be used. If the command line is
empty the usage will be listed.
N.B.The usage is printed to stderr. This method always results
in a System.exit with a status of DEFAULT_SEPARATING_TOKEN
version
will be copied into the
destination table. If there is a product which has not been reprocessed
in the source catalog (ie: the version of the product being copied
is the same in the source catalog as in the destination catalog)
the program will silently not copy the product (unless verbose
is specified and it will echo to the command line that the product
was not copied).
Configuration:
Configuration
file is not specified the DEFAULT_CONFIGURATION_FILENAME
will be used. If the configuration
file is not in the current working directory, it will be looked for
in the user's home directory. The configuration file must contain
the necessary information needed to identify and connect with the
database server (as required by the Database
constructor
and its Connect
method).
These are typically the server "Host" name and database "Type",
"User" and "Password" access values.
CONFIG_GROUP
.
Source_Catalog
,
Destination_Catalog
and
Tables
values may also be expressed
in the configuration file in the same group as the Server access
information. See the options specified below for more information.
Server:
Server
name, which may be specified. If
no server name is provided, the Database will attempt to use the
default (first) Server specified in the Configuration.
Catalogs:
Source_Catalog
parameter. The destination can be specified with the Destination_Catalog
parameter.
Tables:
CONFIG_PARAM_TABLES
Max Change Values
CONFIG_OBS_GEO_GROUP
which
is within the CONFIG_GROUP
group. These are entered as
DEFAULT_MAX_CHANGE_DELIMITER
Log File
CONFIG_PARAM_LOG_PATHNAME
. Most of the information that is
displayed with verbose will be displayed in the log file as well. If a
log file is specified then verbose will be on by default.
Verbose:
Help:
.EXIT_INVALID_COMMAND_LINE_SYNTAX
Copyright \ (C) Arizona Board of Regents on behalf of the \ Planetary Image Research Laboratory, Lunar and \ Planetary Laboratory at the University of Arizona