FTP_Fetch
Fetch selected files from one or more FTP sources.
The names of one or more configuration file groups containing FTP source parameters. Multiple source names may be separated by commas or whitespace. The option may be specified more than once on the command line; all source names will be accumulated. Duplicate source names are ignored. The special source name "DEFAULT" (or anything starting with this name, case insensitive) will cause the source name(s) to be obtained from the FTP_Source_Default configuration parameter; in this case no other source names will be accepted.
If the configuration file does not contain a corresponding FTP source parameter group the source name is expected to be the name of the FTP server and all other parameters will have their default values.
Each FTP source group in the configuration file may contain the following parameters:
The hostname where the FTP server can be contacted (e.g. naif.jpl.nasa.gov). The default is the FTP source name.
The username used to login to the FTP server. The default is "anonymous". If a username has been specified on the command line this parameter is ignored.
The password used to login to the FTP server. The default is "HiROC@HiRISE.LPL.Arizona.edu"; however, the value of the FTP_Password environment variable will be used as the default instead if present. If a password has been specified on the command line this parameter is ignored.
The directory on the FTP server from which to fetch files. The default is to use the login directory. If an FTP directory has been specified on the command line this parameter is ignored.
Specifies which files to fetch. The first value is a regular expression pattern which will be used to select files from the list of filenames in the FTP_Directory; files in subdirectories are not included. The default is to select all files; i.e. ".*".
All files matching the regular expression pattern will be fetched unless an optional subselection criteria is also specified. A subselection criteria may be in one of two forms:
Specifies a subselection criteria based on an ordering of the selected files. This subselection specification is <which>[_<order>], where <which> is either "First" or "Last" and <order> is either "Alphabetic" or "Time". The default <order> is "Alphabetic". For example, "Last_Time" would fetch the file with the most recent modification time from the selected list; "Last" would select the last file from the alphabetically ordered selection list. N.B.: Alphabetic ordering is not the same as numeric ordering; e.g. the name "file10" precedes the name "file9" in an alphabectic ordered list.
Examples:
Selects the file with the most recent modification time from the login directory of the FTP server on the "server" host (assuming there is no configuration file group named "server") from all files with a name that ends with "ext".
As in the previous example, but all files containing "data" in their name are sorted alphabetically and the last one is selected.
Same as the previous example.
As in the previous example, but the first filename in the alphabetically sorted list is selected.
As in the previous example except files from the /pub/data directory on the FTP server are used and the files are sorted by modification time with the oldest one being selected.
Specifies a subselection criteria such that only those files with a modification time that falls within a given time range will be selected. This subselection specification is <start>[, <end>], where <start> is a time value for the start of the time range and <end> is the end of the time range. The time values must specify a date, and can optionally specify a clock time. If no time range end value is specified, then it is assumed to be the same as the start value. If no clock time is specified on the time range end value, then the clock time is assumed to be one second before midnight (23:59:59).
N.B.: All times are assumed to be in reference to local time unless a specific time reference such as GMT, MST, etc. is specified. When determining which files fall within the time range, the time reference of the FTP server is used. Thus in cases where the FTP server is in a different timezone than the client, the timezone of the FTP server needs to be specified if its time reference is to be used. Also, be aware that many computer systems report time values inaccurately, so be prepared to adjust time value specifications according to the specific circumstances of the systems involved; adding -no_op to the command line to test the results before committing to file fetching may be helpful.
There are many possible time value expression formats. Some conform to established standards, and most are intuitively obvious, but some that may seem obvious may not be recognized. <BN.B.>: Time expressions that have spaces must be enclosed in double or single quotes, whether specified on the command line or in the configuration file. The following are examples of acceptable time expressions:
Examples:
Selects files using the configuration source definition(s) named in the FTP_Source_Default parameter list that have "data" in their filenames and also have file modification times between 8:00 am and 5:00 pm Pacific Daylight Time on the 10th of August, 2009.
As in the previous example, but selects files with any modification time on the specified date in the PDT timezone.
As in the previous example, but selects files with modification times for the work week of 10 August, 2009.
As in the previous example, but only selects files with the specific modification time and then writes the fetched files to the local system's /tmp/data directory.
The pathname to the local directory where the fetched files will be copied. A relative pathname is relative to the current working directory. The default local directory is the current working directory (.). If the pathname does not exist an attempt is made to create the directory. If the local directory has been specified on the command line this parameter is ignored.
If the value of this parameter is "Enabled", "True", "Yes" or "Y", "On" or "1" the access and modification time for each fetched file will be preserved in the copy on the local filesystem. Otherwise fetched files will have the current local access and modification time when the file was fetched. The default is to use the local time and not preserve the time from the FTP server.
The username used to login to an FTP server. This value overrides the use of an FTP_Username configuration parameter. The default, if the FTP source group in the configuration does not contain an FTP_Username parameter, is "anonymous".
The password used to login to an FTP server, This value overrides the use of an FTP_Password configuration parameter. The default, if the FTP source group in the configuration does not contain an FTP_Password parameter, is "HiROC@HiRISE.LPL.Arizona.edu"; however, the value of the FTP_Password environment variable will be used as the default instead if present.
The directory pathname on the FTP server from which to fetch files. This value overrides the use of an FTP_Directory configuration parameter. The default, if the FTP source group in the configuration does not contain an FTP_Directory parameter, is the login directory.
The file selection pattern, and optional subselection order criteria, to be used to select one or more files to fetch. These values override the use of a File_Selection_Pattern configuration parameter. The default, if the FTP source group in the configuration does not contain a File_Selection_Pattern parameter, is to select all files.
The selection pattern, if it contains characters special to the command shell, must be quoted (single quotes are best).
Note that the subselection, if provided, must be a separate argument (not comma separated). The value of the subselection argument is described in the File_Selection_Pattern parameter section.
The pathname to the local directory where the fetched files will be copied. A relative pathname is relative to the current working directory. This value overrides the use of a Local_Directory configuration parameter. The default, if the FTP source group in the configuration does not contain a Local_Directory parameter, is the current working directory (.). If the pathname does not exist an attempt is made to create the directory.
Enbale or disable modification time preservation for fetched files. Normally, when a file is written to the local filesystem its modification time is the time at which it was written. When -Preserve is specified the modification time of each fetched file written to the local filesystem is set to its modification time on the source FTP server; i.e. the modification time of fetched files is preserved in the local filesystem. When -No_Preserve is specified the modification times of fetched files will be the current time when they were written to the local filesystem. The default, if the FTP source group in the configuration does not contain a Preserve parameter, is that file modification times will not be preserved.
The configuration file pathname. If the pathname is a simple filename and the file is not present in the current working directory it will be sought in the user's home directory. The default is FTP_Fetch.conf.
Do not fetch the files or create local directories, just list those that would be fetched.
Print this man page.
FTP_Fetch will copy selected files from each FTP source to a local directory. The inability to fetch a selected file - e.g. because a directory was selected instead of a plain file - does not prevent all other files from being fetched, but a partial file is never left in the local directory if it can not be competely copied.
An FTP source is defined by its server hostname (or address), the username and password to login to the server, the directory on the server from which to select files and the filename selection pattern and order or time range subselection criteria for choosing one or more files to copy to a specified local directory. FTP sources are typically defined in a configuration file as groups of parameters providing the source definition with each group having the name of the source. However, an FTP source may also be defined simply as the server hostname with all other information being supplied by default values. Source definition values may be specified on the command line in which case the corresponding configuration parameters are ignored. This enables FTP files to be fetched from an FTP source specified on the command line without the need for any configuration file.
Files are selected from the FTP server directory by applying the file selection pattern to the list of filenames obtained for the directory. The file selection pattern is a regular expression; the default pattern will select all files. By default, all selected files will be fetched. However, if a file subselection criteria has been specified the criteria is applied to the selected files: If an ordering criteria is specified the selected files are sorted according to the order - alphabetically or by modification time - and then either the first or last file from the ordered list is fetched. This enables, for example, the most recent file that matches the selection pattern, or the first file in a numeric sequence, to be fetched. If a time range criteria is specified, then all files matching the regular expression whose modification time is within the time range will be fetched. This enables files to be fetched that have been modified within a narrow or broad time range within an appropriate timezone.
Fetched files are copied in binary mode to a local directory. The local directory is created if it does not exist. Only plain files are fetched; directories that are selected are skipped. N.B.: If the modification time for a selected file on the FTP server can not be obtained - this usually means it is a directory rather than a plain file - then it will be skippped. If files must be fetched from the server in passive mode - perhaps due to a firewall configuration - set the FTP_PASSIVE environment variable.
Note: Configuration parameters may be queried by various utilities. Any utility may used as long as a listing of selected parameters can be obtained with a command line that uses the following syntax:
$Config_Query $parameter_pathname $config_source
where $config_source is the source of the configuration file, $parameter_pathname is the parameter pathname (relative or absolute) that names the parameter of interest, and $Config_Query is the fixed portion of the command line. If a "Config_Query" environment variable is present its value will be used for the fixed portion of the command line. By default the $Config_Query is "Configuration -query". The Configuration utility will accept both file pathnames and URLs for the $config_source and has the advantage of supporing @Include parameters when the source is a locally accessible file, but as a Java application it runs slower than PPVL_report. The PPVL_report utility is an appropriate alternative that is a fast compiled binary application but it will only accept file pathnames for the $config_source and does not support @Include parameters.
All selected files were fetched.
A command line syntax usage message will be provided.
A required configuration parameter could not be found or a parameter value is invalid.
A system operation - such as running a helper utility or creating a directory - failed.
An unrecoverable error condition was encountered while accessing the FTP server.
A time expression specified in a time range subselection criteria could not be parsed.
Bradford Castalia, UA/HiROC
Copyright (C) 2009-2020 Arizona Board of Regents on behalf of the Lunar and Planetary Laboratory at the University of Arizona.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
1.35 2023/06/01 00:02:07