TSTool-Vol2-CommandReference - Laserfiche WebLink

time series must be converted to a daily time series before writing. An option to convert 24-hour values to 1DAY may be added to this command in the future. • HEC-DSS uses times through 2400. TSTool will convert this to 0000 of the next day. Year, month, and day data are not impacted. The internal TSTool values will be converted to hour 2400 when writing. Therefore, reading from a HEC-DSS file and then writing should result in no change in data. • Time series that are written overwrite existing time series, but only for the period that is written. Therefore, previously written values may remain, even if not appropriate. A future enhancement will allow the option of removing the old data before writing new data. The work-around is to write a period that is sufficiently long to guarantee that old data values do not remain in the file, or clear the file out with another tool such as DSSUTL before writing. • Currently the connections to the HEC-DSS file will remain open after the write, in order to minimize performance degradation for multiple write commands. However, this will lock the HEC-DSS file so that other commands or programs cannot perform file manipulation, such as removing the file. The connections will automatically time out after several minutes. A future enhancement will ensure that the file connections can be closed. The A-F parts of the HEC-DSS time series pathname by default are taken from the time series properties, as follows: • The A and B parts are taken from the time series identifier location, where location should be defined as A:B. • The C part is taken from the time series data type. • The D part is taken from the time series period in memory or as defined by the output period. • The E part is taken from the time series interval. • The F part is taken from the time series identifier scenario. These conventions can be overruled by specifying the parts explicitly with command parameters. The parameter values will apply to all time series being written. 509 Command Reference – WriteHecDss() -1 WriteHecDss() Command TSTool Documentation The following dialog is used to edit the command and illustrates the syntax for the command. WriteHecDss WriteHecDss() Command Editor The command syntax is as follows: WriteHecDss(Parameter=Value,…) Command Parameters Parameter Description Default OutputFile The name of the HEC-DSS file to write, surrounded by double quotes to protect whitespace and special characters. If the file does not exist it will be created. None – must be specified. Type The HEC-DSS time series type, indicating whether the time series is instantaneous, mean, or accumulated. None – must be specified. TSList Indicates the list of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards) will be processed. AllTS Command Reference – WriteHecDss() -2 510 TSTool Documentation WriteHecDss() Command Parameter Description Default • AllTS – all time series before the command will be processed. • EnsembleID – all time series in the ensemble will be processed. • FirstMatchingTSID – the first time series that matches the TSID (single TSID or TSID with wildcards) will be processed. • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards) will be processed. • SelectedTS – the time selected with the SelectTimeSeries() command will be processed. TSID The time series identifier or alias for the time series to be processed, using the * wildcard character to match multiple time series. Required if TSList=*TSID. EnsembleID The ensemble to be processed, if processing an ensemble. Required if TSList= EnsembleID. OutputStart The date/time for the start of the output. Use the global output period or write all available data. OutputEnd The date/time for the end of the output. Use the global output period or write all available data. Precision The number of digits after the decimal for numerical output. HEC-DSS default. A The DSS path A-part to use for the time series as written to the HEC-DSS file. Time series identifier location part before the : (if : is present) or the entire location if : is not present. B The DSS path B-part to use for the time series as written to the HEC-DSS file. Time series identifier location part after the : (if : is present) or the blank if : is not present. C The DSS path C-part to use for the time series as written to the HEC-DSS file. Time series identifier data type. E The DSS path E-part to use for the time series as written to the HEC-DSS file. Time series identifier data interval, converted to HECDSS conventions. F The DSS path F-part to use for the time series as written to the HEC-DSS file. Time series identifier scenario. Replace Under development – whether to replace the contents of the previous time series in the HEC-DSS file. Only replace what is actually written. Close Indicate whether to close connections to the HEC-DSS file and allow other processes to move/rename/delete the file. Specifying as True may slow the software as files are repeatedly opened and closed. False – let the HEC-DSS internal software close the connection after timing out. 511 Command Reference – WriteHecDss() -3 WriteHecDss() Command TSTool Documentation A sample command file is as follows: WriteHecDss(OutputFile="sample.dss",TYPE=PER-AVER, OutputStart="1992-01-01",OutputEnd="1992-12-31") Command Reference – WriteHecDss() -4 512 Command Reference – WritePropertiesToFile -1 Command Reference: WritePropertiesToFile() Write one or more time series processor properties to a file Version 10.12.00, 2012-07-27 The WritePropertiesToFile() command writes the value of one or more time series processor properties to a file (this command replaces the older WriteProperty() command, which is being phased out). The ReadPropertiesFromFile() command can be used to read properties from a file. Processor properties include global defaults such as InputStart, InputEnd, OutputStart, OutputEnd, OutputYearType, WorkingDir, and also user-defined properties set with the SetProperty() command. Internally, properties have a name and a value, which is of a certain type (string, integer, date/time, etc.). Examples of using the command include: • creating tests to verify that properties are being set • passing information from TSTool to another program, such as a Python script • storing persistent information for later use, such as the date/time that data were last downloaded from a web service A number of property formats are supported as listed in the following table. Property File Formats Format Description NameValue Simple format, all properties handled as text: PropertyName=PropertyValue PropertyName=”Property value, quoted if necessary” NameTypeValue Same as NameValue format, with non-primitive objects treated as simple constructors: PropertyName=PropertyValue DateTimeProperty=DateTime(“2010-10-01 12:30”) NameTypeValue Python Similar to the NameTypeValue format, however, objects are represented using “Pythonic” notation, to allow the file to be used directly by Python scripts: PropertyName=”PropertyVa lue” DateTimeProperty=DateTime(2010,10,1,12,30) 513 WritePropertiesToFile() Command TSTool Documentation Command Reference – WritePropertiesToFile() -2 The following dialog is used to edit this command and illustrates the syntax of the command. WritePropertiesToFile WritePropertiesToFile() Command Editor The command syntax is as follows: WritePropertiesToFile(Parameter=Value,…) Command Parameters Parameter Description Default OutputFile The property file to write, as an absolute path or relative to the command file. None – must be specified. IncludeProperty The names of properties to write, separated by commas. If not specified, all processor properties will be written. WriteMode Indicates how the file should be written: • Append – append the properties to the file without checking for matches (create the file if it does not exist) • Overwrite – overwrite the properties file • Update – update the properties in the file by first checking for matching property names (which will be updated) and then appending unmatched properties (not yet implemented) Overwrite FileFormat Format of the properties file (see descriptions in the above Property File Formats table): • NameValue • NameTypeValue • NameTypeValuePython NameValue 514 Command Reference – WriteProperty() -1 Command Reference: WriteProperty() Write a time series processor property to a file Version 10.12.00, 2012-07-30 This command has been replaced with the WritePropertiesToFile() command and will be removed from the software in a future release. The WriteProperty() command writes the value of a time series processor property to a file. This is useful for testing that properties are being set. It could also be used to pass information from TSTool to another program. The format of the output is: Property=”Value” The following dialog is used to edit this command and illustrates the syntax of the command. WriteProperty WriteProperty() Command Editor 515 WriteProperty() Command TSTool Documentation Command Reference – WriteProperty() -2 The command syntax is as follows: WriteProperty(Parameter=Value,…) Command Parameters Parameter Description Default OutputFile The property file to write, as an absolute path or relative to the command file. None – must be specified. PropertyName The property name to write None – must be specified. Append Indicates whether the property should be appended to the file (True) or overwrite the file (False). True A sample command file is as follows: WriteProperty(OutputFile="Results/Example_WritePro perty.txt", PropertyName="WorkingDir") 516 Command Reference – WriteReclamationHDB() -1 Command Reference: WriteReclamationHDB() Write a time series to a Reclamation HDB database Version 10.20.00, 2013-04-21 The WriteReclamationHDB() command writes time series to a Reclamation HDB database: • a single time series (which can be part of an ensemble), indicated by the individual time series identifier: o a “real” time series (observations) o a “model” time series (output from a model) • all time series in an ensemble, indicated by the ensemble identifier: o ensemble trace time series are stored as “model” time series ( individual ensemble trace time series can then be read as single time series by specifying the appropriate “hydrologic indicator”, which is set to the ensemble time series sequence number from TSTool time series) See the ReadReclamationHDB() command documentation for information about reading the time series that are written by this command. See the Reclamation HDB Data Store Appendix for more information about the database features and limitations. Command functionality includes: • Time series metadata/new time series: o The command will not define a new time series (site and model data). It is expected that time series previously have been defined in the database. This ensures that TSTool can perform error handling and users do not accidentally load new time series. o The exception is that new ensembles and corresponding trace time series can be defined by specifying ensemble name, trace number, and model run date. • Date/time handling: o TSTool uniformly uses the time at the end of the recorded interval for data values (instantaneous time or end of interval for mean and accumulated values), whereas HDB uses the time at the beginning of the recorded interval for hourly data. See the ReclamationHDB Datastore appendix for more information. o Writing NNour data uses WRITE_TO_HDB procedure where the SAMPLE_END_DATETIME is set to the TSTool date/time and SAMPLE_DATE_TIME is set the TSTool date/time minus NHour. o Writing other than NHour data uses the WRITE_TO_HDB procedure with SAMPLE_DATE_TIME passed as the same value as the TSTool date/time. • Updating time series records: o Time series data records for an existing time series will be updated if previously written. • Missing data: o Missing data currently are not written. By convention missing values in HDB are simply not included in the database. Currently the command will not delete previous records if the new value at a date/time is missing. • Data units: o Data units in the time series are not checked against data units in the database because the units in TSTool data may originally have come from various sources that do not use the same units abbreviations as HDB. It is the user’s responsibility to ensure that time series that are being written have units that are compatible with HDB. • Data flags: 517 WriteReclamationHDB() Command TSTool Documentation Command Reference – WriteReclamationHDB() -2 o Data flags from the time series are not written to the database. The ValidationFlag, OverwriteFlag, and DataFlags parameters are provided to specify HDB flags. Additional capability may be added in the future. • Time zone: o Time zone can be indicated in TSTool time series by including in the start and end date/time information; however, time zones are difficult to standardize when data comes from different sources. The default time zone for HDB is configured for the Reclamation office that uses the database. If the time series time zone is different from the default (displayed in the note for the TimeZone command parameter in the command editor), it can be specified as a command parameter. It is the user’s responsibility to verify that the correct time zone is being used. • HDB data table: o The time series interval is used to determine the HDB time series table to write, with irregular data being written as instantaneous data with date/time precision to minute. o Irregular data also can be written to a specific output table by using the IntervalOverride parameter, for example in cases where a time series was read as irregular but should be treated as hourly in HDB. o TSTool treats year-interval data generically and does not manage water year (or other types of years) in special fashion, other than when processing data into year interval time series. Water year data can be saved in year interval data but currently there is no way to write to the water-year tables in HDB. • HDB database procedure: o The HDB WRITE_TO_HDB stored procedure is used to write individual time series data records:  The time series is written to a model time series table if model parameters are specified.  The model run date, for single time series and ensembles, is truncated to minutes in time series identifiers and for query purposes. o When writing ensembles, the HDB procedure ENSEMBLE.GET_TSTOOL_ENSEMBLE_MRI is used to determine the model run identifier corresponding to model time series and then the WRITE_TO_HDB procedure (above) is used to write data records:  The ensemble name is determined from the EnsembleName parameter – existing names can be selected or a new name can be specified  The trace number is determined from the EnsembleTraceID command parameter, and will result in the trace being taken from specific time series properties.  The model name is determined from the EnsembleModelName parameter. Model names consistent with non-ensemble model time series are used.  The model run date is determined from the EnsembleModelRunDate parameter (if specified then the P_IS_RUNDATE_KEY procedure parameter is set to Y, if not specified N). The following dialog is used to edit the command and illustrates the syntax of the command when writing “real” data, in which case model information is not specified. 518 TSTool Documentation WriteReclamationHDB() Command Command Reference – WriteReclamationHDB() -3 WriteReclamationHDB WriteReclamationHDB() Command Editor for “Real” Time Series 519 WriteReclamationHDB() Command TSTool Documentation Command Reference – WriteReclamationHDB() -4 The following figure illustrates the syntax of the command when writing “model” data for a single time series, in which case the model parameters are specified via the Single model time series tab. WriteReclamationHD_Model WriteReclamationHDB() Command Editor for Single Model Time Series 520 TSTool Documentation WriteReclamationHDB() Command Command Reference – WriteReclamationHDB() -5 The following figure illustrates the syntax of the command when writing an ensemble of “model” time series, in which case ensemble and related model parameters are specified via the Ensemble of model time series tab. The TSTool ensemble is specified with the TSList=EnsembleID and EnsembleID parameters. WriteReclamationHD_Ensemble WriteReclamationHDB() Command Editor for Ensemble of Model Time Series The command syntax is as follows: WriteReclamationHDB(Parameter=Value,…) 521 WriteReclamationHDB() Command TSTool Documentation Command Reference – WriteReclamationHDB() -6 Command Parameters Parameter Description Default DataStore The identifier for the ReclamationHDB data store to use for the database. None – must be specified. TSList Indicates the list of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards) will be processed. • AllTS – all time series before the command. • EnsembleID – all time series in the ensemble will be processed. • FirstMatchingTSID – the first time series that matches the TSID (single TSID or TSID with wildcards) will be processed. • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards) will be processed. • SelectedTS – the time series are those selected with the SelectTimeSeries() command. AllTS TSID The time series identifier or alias for the time series to be processed, using the * wildcard character to match multiple time series. Required if TSList=*TSID. EnsembleID The identifier for the TSTool ensemble to be processed, if processing an ensemble, not to be confused with the Ensemble* parameters below that match HDB data. Required if TSList= EnsembleID. Site CommonName The site common name for the time series location; used with the data type common name to determine the site_datatype_id in the database. None – must be specified unless SiteDataTypeID is specified. DataType CommonName The data type common name for the time series; used with the site common name to determine the site_datatype_id in the database. None – must be specified unless SiteDataTypeID is specified. SiteDataTypeID The site_datatype_id value to match the time series. If specified, the value will be used instead of the site_datatype_id determined from SiteCommonName and DataTypeCommonName. Use the following parameters when reading a single model time series. ModelName The model name for the time series; used with the model run name, hydrologic indicator(s), and model run date to determine the model_run_id in the database. None – must be specified unless ModelRunID is specified. ModelRunName The model run name for the time series; used with the model name, hydrologic indicator(s), and model run date to determine the model_run_id in the database. None – must be specified unless ModelRunID is specified. ModelRunDate The model run date (timestamp) to use for the time series; None – must be 522 TSTool Documentation WriteReclamationHDB() Command Command Reference – WriteReclamationHDB() -7 Parameter Description Default used with the model name, model run name, and hydrologic indicator(s) to determine the model_run_id in the database. The run date should be specified using the format YYYY-MM-DD hh:mm (zero-padded with hour 0-23, minute 0-59, seconds and hundredths of seconds will default to 0). Need to implement tests to make sure this is properly handled, including formatting and listing existing values. specified unless ModelRunID is specified. Hydrologic Indicator The hydrologic indicator(s) to use for the time series; used with the model name, model run name, and model run date to determine the model_run_id in the database. None – must be specified unless ModelRunID is specified. ModelRunID The model_run_id value to match the time series. If specified, the value will be used instead of the model_run_id determined from ModelName, ModelRunName, ModelRunDate, and HydrologicIndicator. Use the following parameters when writing an ensemble of model time series. EnsembleName The name of the ensemble to write. The TSList=EnsembleID and EnsembleID parameters also should be specified. Must be specified if writing an ensemble. EnsembleModelName The model name corresponding to the ensemble. Must be specified if writing an ensemble. EnsembleTraceID Indicate how to identify time series trace identifiers: • %X – use standard time series properties to format the ensemble trace ID (see command editor for format characters) • ${TS:property} – format the trace identifier from time series properties (e.g., properties read from original ensemble data) TSTool and the HDB GET_TSTOOL_ENSEMBLE_MRI procedure currently require the identifier to be an integer – additional options for identifying traces may be added in the future. The time series sequence number (equivalent to the %z formatting string) EnsembleModel RunDate When writing an ensemble, the model run date for the ensemble, specified using format: • YYYY-MM-DD hh:mm (zero-padded with hour 0-23) • ${ts:property} – use a run date from a time series property, truncated to minute Need to implement tests to make sure this is properly handled, including formatting and listing existing values. If not specified, the ensemble identifier in HDB will not include the model run date. The following parameters are always appropriate. Agency The agency abbreviation (e.g., USBR) for data records written to the database. No agency is indicated in database. Validation HDB validation flag. Only uppercase characters are No flag is used. 523 WriteReclamationHDB() Command TSTool Documentation Command Reference – WriteReclamationHDB() -8 Parameter Description Default Flag supported. OverwriteFlag HDB overwrite flag. Overwrite (enforced by HDB stored procedure).. DataFlags User-defined flags, up to 20 characters. No flags are used. TimeZone Three-letter time zone abbreviation for the data records written to the database. Default HDB time zone is assumed. OutputStart The date/time for the start of the output. Use the global output period. OutputEnd The date/time for the end of the output. Use the global output period. 524 Command Reference – WriteRiversideDB() -1 Command Reference: WriteRiversideDB() Write time series to a RiversideDB database Version 10.06.00, 2012-04-15 This command is under development – please provide feedback to developers. The WriteRiversideDB() command writes time series to a RiversideDB database. See the RiversideDB Data Store Appendix for more information about the database features and limitations. The command will not define a new time series but will replace or insert the data records for an existing time series. The current functionality allows a single time series to be written by matching a specific existing time series in the database; however, in the future time series properties may be used to write multiple time series with one command (e.g., use %L to match the location). The following dialog is used to edit the command and illustrates the syntax of the command when writing a single time series to the database. In this case the choices are used to select a matching time series and only one time series can be in the list to process (otherwise a warning will result and nothing is written). WriteRiversideDB WriteRiversideDB() Command Editor when Matching/Writing a Single Time Series 525 WriteRiversideDB() Command TSTool Documentation Command Reference – WriteRiversideDB() -2 The following figure illustrates use of the Match Time Series Using Properties tab, which is envisioned as a future enhancement. In this case, the parameter values do not match a specific time series in the database but instead use the properties from the list of time series being written to indicate how to match a time series in the database. For example, DataType=%T would use the data type from the time series to match the data type in the database and Location=%L would use the location identifier from the time series to match the location identifier in the database. Writing multiple time series requires fewer commands but the command editor will not be able to confirm that the time series are matched in the database (this can only occur when running the commands and data are actually processed). WriteRiversideDB2 WriteRiversideDB() Command Editor when Matching/Writing Multiple Time Series The following technical issues apply when writing time series: • Currently there is no authentication to prevent users from using this command. Options will be explored based on specific system requirements. • Time series being written must have compatible units with the units of the time series in the database. The same units or conversion factor of 1.0 must be detected to allow writing. • Missing values in time series are written as missing values (null) in the database. This allows missing values in the database to have flags. o Regular interval time series could be written in such a way that missing values are simply not written to the database. If this is required, then a new parameter WriteMissingValues could be added. However, tracking revisions might be difficult using this approach because older values would need more complex handling since no newer missing value record would be present. o Irregular interval time series in TSTool do not have missing value records unless they were specifically read from input. • Irregular time series present challenges that may not be fully addressed with the current software features and may require additional enhancements. For example, WriteMethod=TrackChanges may not work properly if updates to irregular data have different date/times than the original values. In this case WriteMethod=DeleteInsert may be more appropriate; however, this does not check the ProtectedFlags parameter. The command syntax is as follows: WriteRiversideDB(Parameter=Value,…) 526 TSTool Documentation WriteRiversideDB() Command Command Reference – WriteRiversideDB() -3 Command Parameters Parameter Description Default DataStore The identifier for the RiversideDB data store to use for the database. None – must be specified. TSList Indicates the list of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards) will be processed. • AllTS – all time series before the command. • EnsembleID – all time series in the ensemble will be processed. • FirstMatchingTSID – the first time series that matches the TSID (single TSID or TSID with wildcards) will be processed. • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards) will be processed. • SelectedTS – the time series are those selected with the SelectTimeSeries() command. AllTS TSID The time series identifier or alias for the time series to be processed, using the * wildcard character to match multiple time series. Required if TSList=*TSID. EnsembleID The ensemble to be processed, if processing an ensemble. Required if TSList= EnsembleID. WriteMode This parameter is envisioned if writing multiple time series is enabled in the future (currently only WriteSingle is enabled). Indicates how time series are being matched, one of: • WriteSingle (the only mode that is enabled), in which case the Match Specified Single Time Series tab is used to enter command parameters. • WriteMultiple (future enhancement), in which case the Match Time Series Using Properties tab is used to enter command parameters. Ensemble time series may fit into either of the above depending on how the sequence numbers are handled. WriteSingle DataType The data type abbreviation in the database to match. None – must be specified. DataSubType The data sub-type in the database to match (may be blank). Not used if blank. Interval The data interval in the database to match. None – must be specified. LocationID The location identifier in the database to match. None – must be specified. DataSource The data source abbreviation in the database to match. Not used if blank. Scenario The scenario in the database to match (may be blank). Not used if blank. SequenceNumber Used for ensembles – the trace number, often the starting year of the trace (may be blank). Not used if blank. WriteDataFlags Indicate whether data flags should be written. True 527 WriteRiversideDB() Command TSTool Documentation Command Reference – WriteRiversideDB() -4 Parameter Description Default OutputStart The date/time for the start of the output. Use the