TSTool-Vol2-CommandReference - Laserfiche WebLink

series are those selected with the SelectTimeSeries() command. AllTS TSID The time series identifier or alias for the time series to be modified, using the * wildcard character to match multiple time series. Required for TSList=*TSID. EnsembleID The ensemble to be modified, if processing an ensemble. Required for TSList=EnsembleID. HandleMissingHow Indicate how to handle missing data, one of: • CarryForwardIfMissing –carry forward the last non-missing value • SetMissingIfMissing – set the result to missing if the original value is missing. The only difference in output is that the period of missing data will either be blank or a horizontal line in graphs. SetMissingIfMissing Reset A date to the precision of the time series (e.g., 01-01 for January 1 in a daily time series) that indicates when to reset the cumulative value to the initial value, before beginning to cumulate again. Specifying the reset effectively defines the first timestep in a new year, whether calendar or some other year is being used for the cumulative values. Use the format MM-DD, MM-DD hh, or MM-DD hh:ss. Do not reset. ResetValue When Reset is specified: the value to initialize the total at the Reset date/time, one of: • DataValue – the data value from the original time series • Number – a number to use for the reset 0 (zero) 138 TSTool Documentation Cumulate() Command Command Reference – Cumulate() -3 Parameter Description Default AllowMissingCount When Reset is specified: the number of values allowed to be missing in a year. If more values are missing, the entire year is set to missing. The missing value count for the first year includes the period from analysis start to Reset. A partial year at the end of the analysis period will not count as missing beyond the analysis end. No limit on the number of missing values. MinimumSampleSize When Reset is specified: the minimum number of non-missing values required in a year to perform the computation. If fewer values are in the sample, the entire year is set to missing. The missing value count for the first year includes the period from analysis start to Reset. A partial year at the end of the analysis period will result in the sample size being less than the full year. No minimum sample size is required. A sample command file to cumulate times from the State of Colorado’s HydroBase is as follows: # 1458 -CENTER 4 SSW 1458.NOAA.Precip.Month~HydroBase # 2184 -DEL NORTE 2 E 2184.NOAA.Precip.Month~HydroBase Cumulate(TSList=AllTS,HandleMissingHow=CarryForwardIfMissing) The following graph illustrates cumulative data for two precipitation gages in the same region, where missing data results in carrying forward the last known value. cumulate_graph Example Graph Showing Results of cumulate() Command 139 Cumulate() Command TSTool Documentation Command Reference – Cumulate() -4 This page is intentionally blank. 140 Command Reference: Delta() Create new time series where values are the difference between each value in original time series Version 9.07.00, 2010-08-05 The Delta() command creates a new time series from an input time series. The resulting values are computed as the difference between each value and the previous value. Consequently, the delta result is the change from the previous value. The CheckTimeSeries() command can be used to check time series for changes that exceed a threshold; however, the Delta() command handles the complexity of time series that reset to a new starting value – the output can be used in conjunction with CheckTimeSeries(). The Delta() command will create as many output time series as there are input time series. The output value is simply the current value minus the previous value. The result is set to missing if this value cannot be computed due to missing values, or in cases where a transition across a reset has errors. If the data do reset, then the expected trend should be specified to allow the ResetMin and ResetMax parameters to be properly interpreted. For example, if Trend=Increasing and a decrease is detected, it is assumed that the values have circled past the reset values. In this case the command will attempt to compute the change across the reset values. If this is not possible, then warnings will be generated and the result will be set to missing. Specific cases that are handled are: • The previous value is out of range – in this case the contribution from the out of range previous value is added to the delta and default flag value is assigned (see Flag parameter description). A warning will be generated. • The current value is out of range – in this case the difference will be decreased because the reset value has not be achieved. A warning will be generated. The above special cases result in somewhat arbitrary difference values because the inputs do not conform to expected values. Out of range values indicate erroneous data that should be corrected before being used in further analysis. Irregular-interval time series that result in differences not being computed will have missing values inserted at appropriate locations to maintain consistent data point spacing with the original data. 141 Command Reference – Delta () -1 Delta() Command TSTool Documentation The following dialog is used to edit the command and illustrates the command syntax. Delta Delta() Command Editor The command syntax is as follows: Delta(Parameter=Value,…) Command Parameters Parameter Description Default TSList Indicates the list of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards). • AllTS – all time series before the command. • EnsembleID – all time series in the ensemble specified by TSID. • FirstMatchingTSID – the first time series that matches the TSID (single TSID or TSID with wildcards). • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards). • SelectedTS – the time series are those selected with the SelectTimeSeries() command. AllTS TSID The time series identifier or alias for the time series to be modified, using the * wildcard character to match multiple time series. Must be specified if TSList=*TSID. EnsembleID The ensemble to be modified, if processing an ensemble. Must be specified Command Reference – Delta () -2 142 TSTool Documentation Delta() Command Parameter Description Default if TSList= EnsembleID. ResetMin The minimum expected data value, used when data are expected to increase (or decrease) to a threshold and then reset, for example raw precipitation values that reset to zero when a container fills. Data are not expected to reset. ResetMax The maximum expected data value, used when data are expected to increase (or decrease) to a threshold and then reset, for example raw precipitation values that reset to zero when a container fills. Data are not expected to reset. ExpectedTrend Indicates trend of data, used when values can reset: • Decreasing – values should decrease and then reset • Increasing – values should increase and then reset Data are variable and don’t reset at fixed thresholds. AnalysisStart The date/time to start analyzing data. Full period is analyzed. AnalysisEnd The date/time to end analyzing data. Full period is analyzed. Flag A string to flag problem values, or Auto for default flags: • R – indicates reset transition out of range > ResetMax • r – indicates reset transition out of range < ResetMin • V – indicates value out of range > ResetMax • v – indicates value out of range < ResetMin Do not flag problem values. Alias Alias to assign to created time series. A literal string can be specified or use %-specifiers to set the alias dynamically (e.g., %L) to use the location part of the identifier. None (but is highly recommended). 143 Command Reference – Delta() -3 Delta() Command TSTool Documentation This page is intentionally blank. Command Reference – Delta () -4 144 Command Reference – DeselectTimeSeries() -1 Command Reference: DeselectTimeSeries() Deselect time series Version 10.13.00, 2012-10-25 The DeselectTimeSeries() command deselects output time series, as if done interactively, to indicate which time series SHOULD NOT be operated on by following commands. The command minimizes the need for the free() command when used in conjunction with other commands that use a time series list based on selected time series (TSList=SelectedTS). See also the SelectTimeSeries() command. The following dialog is used to edit the command and illustrates the command syntax. DeselectTimeSeries DeselectTimeSeries() Command Editor 145 DeselectTimeSeries() Command TSTool Documentation Command Reference – DeselectTimeSeries() -2 The command syntax is as follows: DeselectTimeSeries(Parameter=Value,…) Command Parameters Parameter Description Default TSList Indicates the list of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards) will be modified. • AllTS – all time series before the command. • EnsembleID – all time series in the ensemble will be modified. • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards) will be modified. • TSPosition – time series specified by position in the results list (see TSPosition parameter below). AllTS TSID The time series identifier or alias for the time series to be modified, using the * wildcard character to match multiple time series. TSID or EnsembleID must be specified if identifiers are being matched. EnsembleID The ensemble to be modified, if processing an ensemble. TSID or EnsembleID must be specified if identifiers are being matched. TSPosition A list of time series positions in output (1+), separated by commas. Required if TSList=TSPosition. SelectAllFirst Indicates whether all time series should be selected before deselecting the specified time series: True or False. False A sample command file is as follows: NewPatternTimeSeries(Alias="401234",NewTSID="401234..Precip.Day", Description="Example data",SetStart="2000-01-01",SetEnd="2000- 12-31", Units="IN",PatternValues="0,1,3,0,0,0") DeselectTimeSeries(TSList=AllMatchingTSID,TSID="40*",SelectAllFirst=True) 146 Command Reference: Disaggregate() Create a new time series with shorter interval Version 10.00.01, 2011-05-12 The Disaggregate() command creates a new time series by disaggregating a time series with a longer data interval into a time series with a shorter data interval. The resulting time series will have the same metadata and identifier as the original time series, with a different data interval. See also the general ChangeInterval() command. Converting longer-interval data may cause a perceived shift in the time. For example, 1Day data shifted to 24Hour data will result in the daily values being set at hour zero of the following day. This shift is necessary to generically represent different time precision. Plots will also reflect the shift because hours are not considered when computing plot positions for daily data. It is important to understand how disaggregated data is treated with respect to time when using with other applications. If necessary, use the ShiftTimeByInterval() command to manipulate the resulting output time series. The following dialog is used to edit the command and illustrates the syntax for the command. Disaggregate_Alias Disaggregate() Command Editor The command syntax is as follows: Disaggregate(Parameter=Value,…) The following older command syntax is updated to the above syntax when a command file is read: 147 Command Reference – Disaggregate() -1 Disaggregate() Command TSTool Documentation TS Alias = Disaggregate(Parameter=Value,…) Command Parameters Parameter Description Default TSID The time series identifier or alias for the time series to be disaggregated. None – must be specified. Alias The alias to assign to the time series, as a literal string or using the special formatting characters listed by the command editor. The alias is a short identifier used by other commands to locate time series for processing, as an alternative to the time series identifier (TSID). None – must be specified. Method The method used to perform the disaggregation, one of the following: Orsmbee – this method was presented in “Rainfall Disaggregation Model for Continuous Hydrologic Modeling,” Ormsbee, Lindell E., Journal of Hydraulic Engineering, ASCE, April, 1989. Currently the method has only been enabled for disaggregating 1Day (not 24Hour) data to 6Hour data. SameValue – this simple method causes the resulting time series to have the same value as the original. For example, a monthly time series that is disaggregated to a daily time series will result in each daily value being the same as for the corresponding value in the original monthly time series. Currently the following disaggregations are supported: • Year to Month • Month to Day • Day to NHour (including 24Hour) • Hour to NMinute (including 60Minute) None – must be specified. NewInterval The data interval for the disaggregated time series (NHour, NDay, etc.). None – must be specified. NewDataType The data type for the disaggregated time series, if different from the original. Same data type as the original time series. NewUnits The units for the disaggregated time series, if different from the original. Same units as the original time series. An example command file to process data from the State of Colorado’s HydroBase is as follows: # 08223000 -RIO GRANDE RIVER AT ALAMOSA ReadTimeSeries(TSID="08223000.DWR.Streamflow.Day~HydroBase",Alias=”DayTS”) Disaggregate(TSID="DayTS",Alias=”HourTS”,Method=Ormsbee,New Interval=6Hour) Command Reference – Disaggregate() -2 148 TSTool Documentation Disaggregate() Command Examples of graphs for the original and disaggregated data are shown below, for the two disaggregation methods: disaggregate_SameValue_Graph Daily Input Time Series and 6-Hour Disaggregated Time Series using SameValue Method disaggregate_SameValue_Graph Daily Input Time Series and 6-Hour Disaggregated Time Series using Ormsbee Method 149 Command Reference – Disaggregate() -3 Disaggregate() Command TSTool Documentation This page is intentionally blank. Command Reference – Disaggregate() -4 150 Command Reference: Divide() Divide the data values in one time series by data values in another time series Version 08.16.04, 2008-09-24 The Divide()command divides one time series by another. This is useful for comparing the relative size of time series values (see also RelativeDiff()). If the divisor is zero or missing, the result is set to missing. Use the Scale() command to divide by a numerical value. The following dialog is used to edit the command and illustrates the syntax of the command. Divide Divide() Command Editor The command syntax is as follows: Divide(Parameter=Value,…) Command Parameters Parameter Description Default TSID The time series identifier or alias for the time series to be modified. None – must be specified. DivisorTSID The time series identifier or alias for the time series that is the divisor. None – must be specified. 151 Command Reference – Divide() -1 Divide() Command TSTool Documentation A sample command file to process data from the State of Colorado’s HydroBase database is as follows: # 2184 -DEL NORTE 2 E 2184.NOAA.TempMean.Month~HydroBase # 5706 -MONTE VISTA 2 W 5706.NOAA.TempMean.Month~HydroBase Divide(TSID="2184.NOAA.TempMean.Month", DivisorTSID="5706.NOAA.TempMean.Month") The resulting graph is as follows: divide_graph Results from Divide() Command Command Reference – Divide() -2 152 Command Reference: Exit() Stop processing commands Version 08.16.04, 2008-09-25 The Exit() command can be inserted anywhere in a command file and causes the processing of commands to stop at that line. This is useful for temporarily processing a subset of a long list of commands. Multi-line comments (/* */) can also be used to temporarily disable one or more commands. It may also useful to add an Exit() command at the end of the file so that it is easy to insert commands above this command when the end line is selected (rather than having to deselect all commands when editing). In the future the command may be enhanced to have parameters that more explicitly control processing shut-down. The following dialog is used to edit the command and illustrates the command syntax: Exit Exit() Command Editor The command syntax is as follows: Exit(Parameter=Value,…) Command Parameters Parameter Description Default There are currently no command parameters. A sample command file is as follows: Exit() 153 Command Reference – Exit() -1 Exit() Command TSTool Documentation This page is intentionally blank. Command Reference – Exit() -2 154 Command Reference – ExpandTemplateFile() -1 Command Reference: ExpandTemplateFile() Process a template file to create a fully-expanded file Version 10.21.00, 2013-06-21 The ExpandTemplateFile() command processes a template file (such as a command file, time series product file, or HTML but can be any text file) to create a fully-expanded file and/or processor property. Templates facilitate utilizing conditional logic, loops, and other dynamic processing functionality that is not provided directly by TSTool. For example, a template can be used to repeat commands for multiple location identifiers. One advantage of using the template approach is that problems in the expanded file are clearly indicated, whereas a problem in logic that is represented as a loop might be difficult to diagnose. The FreeMarker software (http://freemarker.org) is used to implement templates (support for other templating engines such as Apache Velocity could be added if needed). Refer to the online Freemarker documentation for information about the markup language used to create templates. Because TSTool checks commands for errors and does not itself understand FreeMarker syntax, template files must be edited with a text editor outside the normal TSTool editing. Attempts to edit a template command file in TSTool may result in error indicators and some command editors may not allow changes to be saved, such as when template notation is used for a filename and the command expects a parent folder name to exist. TSTool may be enhanced in the future to provide template editing features. Examples below illustrate how to use common FreeMarker features. The FreeMarker built-in normalizeNewlines user directive is automatically used to ensure that expanded files use newline characters appropriate for the operating system. Otherwise the results may have all lines merged together (not an issue for HTML used by web browsers but a big issue with TSTool). The normalizeNewlines directive leads to temporary extra first and last lines in the template during processing, which need to be accounted for when interpreting FreeMarker warning messages. For example, a FreeMarker warning about line 21 would actually be line 20 in the original template file. FreeMarker messages may be difficult to interpret. In general errors are because variable names are not spelled correctly or there is an error specifying FreeMarker syntax. The following information is automatically passed from TSTool to the ExpandTemplateFile() command: • Properties set with the SetProperty() command are passed to the template processor. Consequently, the property names can be referenced with ${Property} in the template without using a FreeMarker assign command. • One-column tables are passed as FreeMarker lists, using the table identifier (TableID) as the list property name. Null values in the table are passed as an empty string so that list have the correct number of items for iteration. Use the CopyTable() command to create a one-column table that can be used as a list for template expansion. The UseTables command parameter can be used to turn off this transfer, for example in cases where an ExpandTemplateFile() command is being repeated many times, does not use the tables, and is slowed down by converting the tables to FreeMarker lists. 155 ExpandTemplateFile() Command TSTool Documentation Command Reference – ExpandTemplateFile() -2 The following dialog is used to edit the command and illustrates the syntax for the command. ExpandTemplateFile ExpandTemplateFile() Command Editor The command syntax is as follows: ExpandTemplateFile(Parameter=Value,…) Command Parameters Parameter Description Default InputFile The name of the template file to process. It is recommended that the filename include “template” and that a comment in the file include @readOnly, which will cause TSTool to warn users when saving the expanded result. None – must be specified. OutputFile The name of the expanded output file. None – must be specified. OutputProperty The name of a property to receive the results of the template expansion. This is appropriate when templates are used to expand single-line text, for example. No property value will be set. UseTables Indicate whether 1-column tables should be passed to the template expander. Doing so is a performance hit and should be avoided if tables are not used in the template. True ListInResults Indicate whether the results of the expansion should be listed in the TSTool results area. This may be undesirable for “worker” files that users will normally not view. True 156 TSTool Documentation ExpandTemplateFile() Command Command Reference – ExpandTemplateFile() -3 Example Using Simple Variable Assignment The following example illustrates a simple template command file and expanded result. # Simple test to expand a text file using FreeMarker #@readOnly <#assign message="Hello World"> ${message} # Simple test to expand a text file using FreeMarker #@readOnly Hello World Example of Passing Time Series Processor Properties to Templates TSTool uses the ${Property} notation to dynamically replace the string with the corresponding property value (as a string). FreeMarker uses the ${Variable} notation to dynamically replace the string with the corresponding variable (as a string). Because the same notation is used by both software components, care must be taken to ensure that values are properly interpreted. TSTool automatically passes all TSTool properties to the ExpandTemplateFile() command. Consequently, one of the main ways to avoid conflicts is to ensure that template command files do not use any of the properties defined in TSTool. One way to check property names is to insert a WritePropertyToFile() command at the appropriate line in a command file and review the list of properties that are shown when editing the command. To utilize TSTool processor properties in a template, do not use the FreeMarker assign command and instead reference the property directly. The following TSTool command file illustrates how to define a property that is used by the ExpandTemplateFile() command: # Simple test to expand a text file using FreeMarker SetProperty(PropertyName="HelloWorldProp",PropertyType=String, PropertyValue="Hello World") ExpandTemplateFile(InputFile="Data\ProcessorStringProperty.txt", OutputFile="Results/Test_ExpandTemplateFile_HelloWorld_out.txt") The corresponding template command file is as follows: # Simple test to expand a text file using FreeMarker ${HelloWorldProp} The variables also can be used in assignment, similar to the following: # Simple test to expand a text file using FreeMarker <#assign message="${HelloWordProp}"> ${message} Example of Protecting TSTool Properties in Template with a Literal FreeMarker String It is possible to use TSTool property notation in a template and cause FreeMarker to ignore the property. This will ensure that the property notation is present in the output, for TSTool to interpret at run-time. 157 ExpandTemplateFile() Command TSTool Documentation Command Reference – ExpandTemplateFile() -4 The following example illustrates an input file that uses the FreeMarker ${r"…"} raw literal string notation: # Simple test to expand a text file using FreeMarker # and also escape text so that it passes through to the expanded file # @readOnly <#assign message="Hello World"> ${r"SomeCommand($SomeProperty)"} The corresponding output file is as follows: # Simple test to expand a text file using FreeMarker # and also escape text so that it passes through to the expanded file # @readOnly Hello World SomeCommand($SomeProperty) Example of Using a Comment in the Template, which is Omitted from Expanded Output It often is desirable to have comments in the template file to explain the template, but not have the comments propagated to the expanded output. The following example illustrates an input file that uses the FreeMarker $<#--… --> notation for comments: # Simple test to expand a text file using FreeMarker # There should be no comment in the expanded output below this line <#--This is a comment in the template --> <#assign message="Hello World"> ${message} The corresponding output file is as follows: # Simple test to expand a text file using FreeMarker # There should be no comment in the expanded output below this line Hello World Example Using Variable Assignment and Loop Using List The following example illustrates a template command file repeat a command for a list of location identifiers. A block of multiple commands can be repeated, as appropriate. Long lines are indented for illustration but would exist on a single line without indentation in the template file. Note that the loc_index FreeMarker syntax allows the loop counter to be used. # Simple template to illustrate how to repeat commands with a list of # location identifiers # Create a time series for each location # The following ensures that the created template is read-only, so users # modify the template instead: #@readOnly <#assign setStart = "2000-01-01"> <#assign setEnd = "2000-03-15"> <#assign units = "CFS"> <#assign locList = ["loc1", "loc2", "loc3", "loc4"]> <#list locList as loc> 158 TSTool Documentation ExpandTemplateFile() Command Command Reference – ExpandTemplateFile() -5 NewPatternTimeSeries(Alias="${loc}",NewTSID="${loc}..Streamflow.Day", SetStart="${setStart}",SetEnd="${se tEnd}",Units="${units}", PatternValues="${loc_index + 1},0") </#list> The expanded command file is as follows: # Simple template to illustrate how to repeat commands with a list of # location identifiers # Create a time series for each location # The following ensures that the created template is read-only, so users # modify the template instead: #@readOnly NewPatternTimeSeries(A lias="loc1",NewTSID="loc1..Streamflow.Day", SetStart="2000-01-01",SetEnd="2000-03-15",Units="CFS",PatternValues="1,0") NewPatternTimeSeries(Alias="loc2",NewTSID="loc2..Streamflow.Day", SetStart="2000-01-01",SetEnd="2000-03-15",Units="CFS",PatternValues="2,0") NewPatternTimeSeries(Alias="loc3",NewTSID="loc3..Streamflow.Day", SetStart="2000-01-01",SetEnd="2000-03-15",Units="CFS",Patt ernValues="3,0") NewPatternTimeSeries(Alias="loc4",NewTSID="loc4..Streamflow.Day", SetStart="2000-01-01",SetEnd="2000-03-15",Units="CFS",PatternValues="4,0") Example Using a One-Column Table for a List for Looping The following example illustrates a template command file that reads the location list from a table. Note that the list must be a one-column table. If the original table has more than one column, read the original file and then use the CopyTable() command to create a new one-column table. A comma-separatedvalue (CSV) file is used for the list: # Simple list to use during template expansion "Location" loc1 loc2 loc3 loc4 The template file is similar to the previous example; however, the list of locations is now provided via the table (no <#assign> element for the list) rather than having to hard-code in the template, which separates data from the processing logic: # Simple template to illustrate how to repeat commands with a list of # location identifiers # Create a time series for each location # The following ensures that the created template is read-only, # so users modify the template instead: # The list is provided by the processor as a one-column table with ID # matching the list name #@readOnly <#assign setStart = "2000-01-01"> <#assign setEnd = "2000-03-15"> <#assign units = "CFS"> <#list locList as loc> NewPatternTimeSeries(Alias="${loc}",NewTSID="${loc}..Streamflow.Day", SetStart="${setStart}",SetEnd="${setEnd}",Units="${units}", PatternValues="${loc_i ndex + 1},0") </#list> 159 ExpandTemplateFile() Command TSTool Documentation Command Reference – ExpandTemplateFile() -6 The following command file reads the list of locations from the table and then expands the template file. Note that the TableID must match the list name in the <#list…> element in the template. # Test expanding a FreeMarker template for a list of time series, using a # one-column