TSTool-Vol2-CommandReference - Laserfiche WebLink

TSTool Documentation TSTool – Time Series Tool – Command Reference Version 10.21.00, 2013-04-21 1 This page is intentionally blank. This document is formatted for double-sided printing. 2 Table of Contents TSTool – Time Series Tool – Blank Page TSTool Syntax Guide Commands – Basic Syntax Commands – Referring to Parameters Commands – Comments Commands – Time Series Identifiers Processor – Properties Time Series – Properties Time Series – Data Flags Date/Time Regular Expression – Notation Template – Syntax Configuration File – TSTool Configuration File Configuration File – Datastore Properties Configuration File – Time Series Product Files 11 16 17 17 17 18 18 18 19 19 20 21 21 21 21 22 3 Table of Contents Command Glossary Command Reference: # Command Reference: /* Command Reference: */Command Reference: Time Series Identifier (TSID) Command Reference: Add() Command Reference: AddConstant() Command Reference: AdjustExtremes() Command Reference: AnalyzePattern() Command Reference: AnalyzeNetworkPointFlow() Command Reference: AppendFile() Command Reference: AppendTable() Command Reference: ARMA() Command Reference: Blend() Command Reference: CalculateTimeSeriesStatistic() Command Reference: ChangeInterval() Irregular Time Series to Regular Time Series 23 33 35 37 39 41 43 45 47 51 63 65 67 73 75 81 81 4 Table of Contents Regular Time Series to Regular Time Series ACCM (Accumulation) to ACCM (Accumulation) ACCM (Accumulation) to INST (Instantaneous) ACCM (Accumulation) to MEAN INST (Instantaneous) to INST (Instantaneous) INST (Instantaneous) to ACCM (Accumulation) INST (Instantaneous) to MEAN MEAN to MEAN MEAN to ACCM (Accumulation) MEAN to INST (Instantaneous) Command Reference: ChangePeriod() 60_Command_CheckTimeSeries.pdf Command Reference: CheckTimeSeriesStatistic() 60_Command_CompareFiles.pdf 60_Command_CompareTables.pdf Command Reference: CompareTimeSeries() Command Reference: ComputeErrorTimeSeries() 83 83 84 84 85 86 86 88 88 88 97 99 103 107 109 111 115 5 Table of Contents Command Reference: ConvertDataUnits() Command Reference: Copy() Command Reference: CopyEnsemble() Command Reference: CopyTable() Command Reference: CreateEnsembleFromOneTimeSeries() Command Reference: CreateFromList() Command Reference: CreateRegressionTestCommandFile() Command Reference: Cumulate() Command Reference: Delta() Command Reference: DeselectTimeSeries() Command Reference: Disaggregate() Command Reference: Divide() Command Reference: Exit() Command Reference: ExpandTemplateFile() Example Using Simple Variable Assignment Example of Passing Time Series Processor Properties to Templates Example of Protecting TSTool Properties in Template with a Literal FreeMarker String 117 119 121 123 125 129 133 137 141 145 147 151 153 155 157 157 157 6 Table of Contents Example of Using a Comment in the Template, which is Omitted from Expanded Output Example Using Variable Assignment and Loop Using List Example Using a One-Column Table for a List for Looping Example Using a Multiple-Column Table to Loop Through Two Lists Example of Expanding a Template to a Processor Property Example of Using ExpandTemplateFile() in a Loop to Expand Multiple Files Command Reference: FillConstant() Command Reference: FillDayTSFrom2MonthTSAnd1DayTS() 60_Command_FillFromTS.pdf Command Reference: FillHistMonthAverage() Command Reference: FillHistYearAverage() Command Reference: FillInterpolate() Command Reference: FillMixedStation() Implementation in Colorado’s Decision Support Systems Command Reference: fillMOVE1() Command Reference: FillMOVE2() Command Reference: FillPattern() 158 158 159 160 161 161 167 169 173 175 177 179 181 182 187 189 193 7 Table of Contents 60_Command_FillPrincipalComponentAnalysis.pdf Command Reference: FillProrate() Command Reference: FillRegression() Command Reference: FillRepeat() Command Reference: FillUsingDiversionComments() Diversion Comment Not Used Flag Structure Currently in Use Flag Command Reference: FormatDateTimeProperty() Command Reference: FormatTableString() Command Reference: Free() Command Reference: FreeTable() 60_Command_FTPGet.pdf Command Reference: InsertTimeSeriesIntoEnsemble () Command Reference: LagK() 60_Command_LookupTimeSeriesFromTable.pdf Command Reference: ManipulateTableString() Command Reference: Multiply() 195 197 201 211 213 213 213 219 221 223 225 227 229 231 235 239 241 8 Table of Contents Command Reference: NewDayTSFromMonthAndDayTS() Command Reference: NewEndOfMonthTSFromDayTS() Command Reference: NewEnsemble () Examples Command Reference: NewStatisticTimeSeries() Examples Command Reference: NewStatisticTimeSeriesFromEnsemble() Examples Command Reference: NewStatisticYearTS() Example Command Reference: NewTable () Command Reference: NewTimeSeries() Command Reference: NewTreeView() Command Reference: Normalize() Command Reference: OpenHydroBase() Command Reference: PrintTextFile() Command Reference: ProcessTSProduct() 243 247 251 254 255 257 259 262 265 270 271 273 275 277 279 281 285 9 Table of Contents Command Reference: ProfileCommands() Command Reference: ReadDateValue() Command Reference: ReadDelimitedFile() Command Reference: ReadHecDss() Command Reference: ReadHydroBase() Command Reference: ReadMODSIM() Command Reference: ReadNrcsAwdb() Command Reference: ReadPatternFile() Command Reference: ReadPropertiesFromFile() Command Reference: ReadRccAcis() Command Reference: ReadReclamationHDB() 60_Command_ReadRiversideDB.pdf Command Reference: ReadRiverWare() Command Reference: ReadStateCU() Command Reference: ReadStateCUB() Command Reference: ReadStateMod() Command Reference: ReadStateModB() 289 293 295 301 303 309 311 313 315 317 321 329 331 333 335 337 339 10 Table of Contents Command Reference: ReadTableFromDataStore() Command Reference: ReadTableFromDBF() 60_Command_ReadTableFromDelimitedFile.pdf Command Reference: ReadTableFromExcel() Command Reference: ReadTimeSeries() Command Reference: ReadTimeSeriesList() Command Reference: ReadUsgsNwisDaily() Command Reference: ReadUsgsNwisGroundwater() Command Reference: ReadUsgsNwisInstanta neous() 60_Command_ReadUsgsNwisRdb.pdf Command Reference: ReadWaterML() 60_Command_ReadWaterOneFlow.pdf Command Reference: RelativeDiff() Command Reference: RemoveFile() Command Reference: RemoveTableRowsFromDataStore() 60_Command_ReplaceValue.pdf Command Reference: ResequenceTimeSeriesData() 341 345 347 351 355 357 361 365 369 373 375 377 379 383 385 387 391 11 Table of Contents Command Reference: RunCommands() Command Reference: RunDSSUTL() Command Reference: RunningAverage() Command Reference: RunningStatisticTimeSeries() Command Reference: RunProgram() 60_Command_RunPython.pdf Command Reference: Scale() Command Reference: SelectTimeSeries() Command Reference: SetAutoExtendPeriod() Command Reference: SetAveragePeriod() Command Reference: SetConstant() Command Reference: SetDataValue() Command Reference: SetDebugLevel() Command Reference: SetFromTS() Command Reference: SetIgnoreLEZero() Command Reference: SetIncludeMissingTS() Command Reference: SetInputPeriod() 395 397 401 405 411 415 419 421 423 425 427 429 431 433 437 439 441 12 Table of Contents Command Reference: SetOutputPeriod() Command Reference: SetOutputYearType() Command Reference: SetPatternFile() 60_Command_SetProperty.pdf Command Reference: SetTimeSeriesProperties FromTable() Command Reference: SetTimeSeriesProperty() Command Reference: SetToMax() Command Reference: SetToMin() Command Reference: SetWarningLevel() Command Reference: SetWorkingDir() Command Reference: ShiftTimeByInterval() Command Reference: SortTimeSeries() Command Reference: StartLog() Command Reference: StartRegressionTestResultsReport() Command Reference: StateModMax() Command Reference: Subtract() Command Reference: TableMath() 443 445 447 449 451 453 455 457 459 461 463 465 467 469 471 473 475 13 Table of Contents 60_Command_TableTimeSeriesMath.pdf Command Reference: TableToTimeSeries() Command Reference: TimeSeriesToTable() Command Reference: VariableLagK() 60_Command_WebGet.pdf Command Reference: WeightTraces() Command Reference: WriteCheckFile() 60_Command_WriteDateValue.pdf Command Reference: WriteHecDss() Command Reference: WritePropertiesToFile() Command Reference: WriteProperty() Command Reference: WriteReclamationHDB() Command Reference: WriteRiversideDB() Command Reference: WriteRiverWare() Command Reference: WriteStateCU() Command Reference: WriteStateMod() Command Reference: WriteSummary() 477 479 487 493 499 501 505 507 509 513 515 517 525 531 533 535 537 14 Table of Contents Command Reference: WriteTableToDataStore() 60_Command_WriteTableToDelimitedFile.pdf Command Reference: WriteTableToHTML() Command Reference: WriteTimeSeriesProperty() Command Reference: WriteTimeSeriesToDataStore() Command Reference: WriteTimeSeriesToJson() Command Reference: WriteTimeSeriesToKml() 60_Command_WriteWaterML.pdf 99_TSTool_Spine_CDSS_CommandReference. pdf 539 543 545 547 549 553 557 559 561 15 Blank Page This page is intentionally blank. 16 TSTool Syntax Guide -1 TSTool Syntax Guide Version 10.13.00, 2012-10-23 TSTool commands use a number of syntax (notation) conventions that have been implemented over time in response to functionality requirements. This appendix provides a summary of the syntax as a guide for users and future software development. Syntax standards listed here should be used where possible to ensure consistency in software features. Where appropriate, notation has been selected based on other efforts. For example, date/time formatting is patterned after the C language strftime() function, which has been available for over 30 years. In cases where notation is specific to TSTool, an attempt has been made to consider common notation standards that can be adapted for TSTool. In cases where one or more existing standards are in place, the most common or relevant standard for TSTool has been selected, with an option to implement additional standards in the future. Although standard notation is utilized into the software design, support for notation in commands may be incomplete because some commands use older code. For example, the ability to use properties to specify command parameters is implemented only for commands that have specifically required such functionality. Future software enhancements will continue to update code to universally provide standard features. The following sections are ordered roughly in the order that topics are likely to be encountered, with headings grouped according to major TSTool design elements. Commands – Basic Syntax The syntax for commands adheres to the following syntax: CommandName(Parameter1=Value1,Parameter2=Value2) The CommandName matches a command from the TSTool Commands menu and as documented in the Command Reference documentation, which describes command parameters. Any parameter value can be surrounded by double quotes to protect whitespace and other characters (such as characters used in the command itself including equal sign, comma, and parenthesis). However, double quotes typically are used only for parameter values that are text, dates, filenames, etc., and not simple data such as numbers. Commands currently cannot be indented, although this may be enabled in the future. Command names and parameters generally are case-insensitive. However, “camel” notation (mixed upper and lower-case letters) is used to improve readability. In some cases this results in an acronym being converted from uppercase to missed-case (e.g., “USGS” becomes “Usgs”). Commands – Referring to Parameters In some cases it is necessary to set one command parameter using the value of another command parameter. This capability has been implemented for a small number of commands, for example NewStatisticEnsemble(). To reference a command parameter in another parameter, use the notation: CommandName(Parameter1=Value1,Parameter2=”${C:Parameter1}…etc”) 17 Syntax Guide TSTool Documentation TSTool Syntax Guide -2 This notation uses C: to provide a “command scope”, similar to how the TS: notation provides a scope for time series properties (discussed below). Commands – Comments Command files use comments to disable commands without deleting them. A # character at the start of a line indicates a one-line comment. A group of lines that start with /* and end with */indicate a block of comments and all intervening commands will be ignored in processing. Commands – Time Series Identifiers Time series identifiers (TDIDs) uniquely identify time series and are discussed in detail in the Introduction chapter. TSID commands, which match the syntax discussed below, are created when using the data browsing features of the TSTool main interface, are specified by some commands, and can be edited manually if the user edits a command file with a text editor. These commands are essentially “read” commands that use default parameters (e.g., the global input period and do not assign an alias). There are two main forms of TSIDs: Location.DataSource.DataType.Interval[.Scenario]~DataStore[~ FileName] Location.DataSource.DataType.Interval[.Scenario] The first form of the TSID is a unique identifier for a time series, similar to a Universal Resource Indicator (URI) for a web page, and allows software to locate the data for reading. The datastore (or “input type” and corresponding filename) allow the software to find the source of the data. The second form of the TSID is a unique identifier for a time series within TSTool and is used after reading the data. In cases where more than one time series will have the same TSID after reading, an alias can be assigned (see the Introduction chapter and the Time Series – Properties section below). TSIDs may be more complex if, for example, the data type requires the use of multiple parts for uniqueness. In this case, a dash may be used (e.g., Streamflow-Max). The datastore appendices describe how time series properties from the original source are mapped into TSID notation. Processor – Properties TSTool commands are processed, and data managed, by a time series “processor”. The processor interacts with all commands and is controlled with properties that initially have internal defaults (e.g., the default is to read all available data rather than a specified input period). Properties that control the processor are set with specific commands (e.g., SetInputPeriod()) and user-supplied properties can be set with the SetProperty() command (e.g., it is common to manage file locations and dates used in processing). The ReadPropertiesFromFile() and WritePropertiesToFile() commands can be used to save and manage properties outside of TSTool. Processor properties can be used to specify parameters for commands using the following notation: ${PropertyName} 18 TSTool Documentation Syntax Guide TSTool Syntax Guide -3 For example, some commands that operate on files allow the property ${WorkingDir} to be used for the current working directory. Refer to command documentation to determine if properties are supported. Additional support is being phased in as resources allow and to satisfy requirements. Properties internally have a specific data type. For example the input start and end use a “DateTime” object type supported by TSTool. All properties will convert to strings, for example when saved to a properties file. Some care may need to be taken to use properties of an appropriate type but a general rule is that properties used in file names or similar can simply be handled as strings. Time Series – Properties Time series properties are specific to individual time series. Some internal properties are handled as specific data values (e.g., data units are a string associated with a time series) whereas user-assigned properties are assigned to the time series as a list (see the SetTimeSeriesProperty() command). Time series properties are used by some commands to control the command functionality and output. For example, many commands that create time series allow the alias to be assigned using time series properties. The following notation is used when dealing with time series properties: • % formatting – Many commands that create time series allow the Alias or other parameters to be assigned using % formatters. For example, Alias=”%L” indicates that the time series alias should be assigned to the location part of the time series identifier, which for a read command is controlled by the rules of the command. Format specifiers are provided for fundamental time series data properties that are required for each time series (units, location, data type, etc.). • TS:Property reference – Some command parameters need to specify a time series property by reference but the above formatting notation is inappropriate. In this case, the following design is being phased in (under development): o TS:PropertyName o ${TS:PropertyName} The latter notation allows a time series property to be specified using a notation similar to processor properties, but the TS: prefix differentiates the property from the more generic processor notation. Note that using time series properties in commands in some cases must be limited because TSTool uses a “discovery mode” to partially read/create time series so that they can be listed in “downstream” commands. Too much reliance on internal time series data might require reading more time series data, which can greatly decrease software performance in discovery mode. Time Series – Data Flags Time series data values (measurements, observations, etc.) are managed internally as lists of date/time, value, and flag data. A data flag is a string that is assigned a value based on one of the following cases: • missing data value with a flag • non-missing data value and no data flag • non-missing data value with a flag Data flags are useful for indicating the quality of a data value (e.g., E might indicate estimated) and for tracking how specific data values are manipulated (e.g., append to the data flag as specific actions are taken). TSTool generally does not implement a standard for data flags because flags used in input data may vary. However, some commands allow setting flags based on simple rules. For example fill 19 Syntax Guide TSTool Documentation TSTool Syntax Guide -4 commands generally have a FillFlag parameter to set the data flag for filled values. The following table lists notation that is used to provide flexibility in setting data flags. The first notation option is used by most commands and the other options are being phased in (refer to command documentation to confirm available data flag functionality). Command Parameter Notation Used When Setting Data Flag Notation Description x Set the data flag to x regardless of whether it has already been set. +x If the flag has not been set, set to x. If the flag has been set, append x. This notation is useful when there are no concerns about the order of characters in multi-character flags. +,x If the flag has not been set, set to x. If the flag has been set, append ,x. This notation is useful when flags are set for each step in a process. Auto Some commands allow Auto or another string as the flag. In this case, the command will decide the flag value that is assigned, based on some condition. For example, the flag may be assigned based on which time series was used to fill the value. Data points in graphs can be labeled in various ways to facilitate interpretation of the data. For example, each data point can be labeled with the data value, flag, or other information. Similar to time series property formatting, the notation %q in graph data point labels indicates that the points should be labeled with the data flag. Date/Time Date/time notation is ubiquitous when dealing with time series, and includes use for the following: • date/time associated with specific data values • date/time pair that indicates data period or subset of the full data period • date/time pair indicating a window within each year In most cases TSTool will default to displaying date/time using the ISO 8601 specification, which is essentially YYYY-MM-DD hh:mm:ss. Not only does this implement a global standard, but it also ensures that date/times are formatted in a way that allows sequential sorting. The precision of formatted date/times is generally consistent with the time series data interval (e.g., monthly time series will have dates that are by default formatted as YYYY-MM). It may be desirable or necessary to specify the format of date/times, for example to indicate the format for output or parsing. When this is necessary, the notation utilizes an optional format type prefix and the format itself, as follows: • The default is to parse the date/time string by matching ISO or other common formats (this works most of the time). The default output format is the ISO format. • C:%m%d%y – Indicates that a C-style format is being used, where the formats match the UNIX strftime() function syntax. See the FormatDateTimeProperty() command documentation. 20 TSTool Documentation Syntax Guide TSTool Syntax Guide -5 • In the future support for Microsoft Excel or other notation may be added (e.g., MM-YYYY). Regular Expression – Notation Regular expressions are strings that indicate how to match patterns, for example to match file names or time series identifiers (see: http://en.wikipedia.org/wiki/Regular_expression). Many software tools and programming languages implement regular expressions to facilitate efficient data processing; however, the notation can be confusing, especially if not used on a regular basis. Within TSTool the following regular expression notations are used: • “globbing” – This notation was popularized by UNIX and in simple terms relies on the * character to indicate “match zero or more characters”. For example, it can be used to match a list of comma-separated-value files using the expression *.csv. • Regular expression syntax – True regular expression syntax provides much more power than globbing notation, but also introduces complexity in notation. TSTool is written in Java and internally relies on Java’s regular expression syntax. In most cases, TSTool commands and configuration files use the simpler globbing notation because it is easier to use and explain. However, in some cases the more powerful regular expression syntax is needed. Where confusion may result, the command documentation clearly indicates the syntax that is supported, and commands may accept the notation glob:xxxx or regex:xxxx to indicate the type of regular expression that is being specified. Template – Syntax Template files are used when processing is automated to iterate over one or more lists of input data. For example, the same 10 commands may be executed for each of 100 time series. TSTool uses the FreeMarker template library to process templates. See the ExpandTemplateFile() command documentation for an explanation of syntax. Configuration File – TSTool Configuration File The TSTool configuration file uses a simple notation to assign properties: [Section] Property = Value The [Section] notation is internally used as a prefix on the property name (e.g., Section.Property = Value). Comments are lines that start with #. Property values can be surrounded by double quotes. Configuration File – Datastore Properties Datastore property files use the simple notation: Property = Value Comments are lines that start with #. Property values can be surrounded by double quotes. The specific property values are described in TSTool datastore appendices. 21 Syntax Guide TSTool Documentation TSTool Syntax Guide -6 Configuration File – Time Series Product Files Time series product configuration file uses a simple notation to assign properties: [Section] Property = Value The [Section] notation is internally used as a prefix on the property name (e.g., Section.Property = Value). Comments are lines that start with #. Property values can be surrounded by double quotes. See also the TSView Time Series Viewing Tools appendix. 22 Command Glossary Version 07.01.00, 2007-03-02, Acrobat Distiller The following parameter names and terms are used throughout TSTool commands. A term indicated in bold font is a definition. A term indicated in bold courier font is a parameter name. Parameters that are infrequently used are listed with the corresponding commands. Common parameters are defined but long lists of corresponding commands are not provided. a1,… – Used with the ARMA() command. b1,… – Used with the ARMA() command. Alias – A (generally) short identifier for a time series, used in place of the TSID, which simplifies commands. The Alias and TSID values are interchangeable when used as parameters to commands and may both be referred to as TSID in command editors. See also TSID. Alias – A (generally) short identifier for a time series, used in place of the TSID, which simplifies commands. When used to create/read a time series, the syntax of a command is typically similar to: TS Alias = command(…). See also TSID. AddTSID – Time series identifiers for time series to add. See the add() command. AddValue – A numerical value to be added to a time series. See the addConstant() command. AdjustMethod – Indicates the method used when adjusting a time series. See the adjustExtremes() command. AllowMissingCount – Indicate how many missing data values are allowed in an interval, in order to allow processing. See the changeInterval() and newStatisticYearTS() commands. AnalysisEnd – A DateTime that indicates the end of an analysis. AnalysisMonth – One or more months indicating which months should be processed in the analysis. See the fillRegression() command. AnalysisStart – A DateTime that indicates the start of an analysis. ARMAInterval – The data interval used in an ARMA analysis. See the ARMA() command. AutoExtendPeriod – Indicate whether to autoextend the period of all time series to be the output period. See the setAutoExtendPeriod()command. AverageEnd – A DateTime that indicates the end of an averaging analysis. See the setAveragePeriod() command. AverageMethod – Indicate the method to use when averaging data. See the runningAverage() command. 23 Command Glossary -1 Command Glossary TSTool Documentation AverageStart – A DateTime that indicates the start of an averaging analysis. See the setAveragePeriod() command. BlendMethod – The method to use when blending time series. See the blend() command. BlendTSID – Time series identifiers for time series to blend into main time series. See the blend() command. Bracket – The number of days to search forward and back for a non-missing value. See the newEndOfMonthTSFromDayTS() and runningAverage() commands. CalculateFactorHow – Indicate how to calculate the factor used when prorating values. See the fillProrate() command. CommandLine – The command line for a program to run. See the runProgram() command. ConstantValue – A numerical value used for filling, etc. See the fillConstant(), setConstant() and setConstantBefore() command. DatabaseName – The name of a database, when making a database connection. See the openHydroBase() command. DatabaseServer – The name of a database server, when making a database connection. See the openHydroBase() command. DataSource – The data source to use when forming a TSID. See the createFromList() command. DataType – The data source to use when forming a TSID. See the createFromList() command. DateTime – A date/time value, typically represented as a string, which indicates a point in time. Date/time strings have a precision that is interpreted by the software. For example, the date/time string 1990 has a precision of year, whereas the string 1990-01-12 has a precision of day. DateTime – A specific date/time associated with time series data. See the setDataValue() command. DayTSID – Time series identifier for a daily time series. See the newDayTSFromMonthAndDayTS() command. DefaultFlow – Indicate a default flow value to be used if observations or filled values cannot be found. See the lagK() command. Delim – The delimiter character(s) used when processing delimited files. See the createFromList() command. DependentAnalysisEnd – A DateTime that indicates the end of an analysis of dependent time series. See the fillMOVE2() command. Command Glossary -2 24 TSTool Documentation Command Glossary DependentAnalysisStart – A DateTime that indicates the start of an analysis of dependent time series. See the fillMOVE2() command. Description – The description (name) for a time series. See the newTimeSeries() command. DeselectAllFirst – Indicate whether to deselect all time series before processing the command. See the selectTimeSeries()com mand. DiffFlag – A character flag used to indicate when time series values are different. See the compareTimeSeries() command. Divisor – Indicate which time series is the divisor. See