TSTool-Vol2-CommandReference

• Title: TSTool - CommandReference - 10.21.00
• Description: Command Reference for Version 10.21.00
• Decision Support - Doc Type: Software Documentation
• Date: 7/14/2013
• DSS Category: DMI Utilities
• Prepared By: Riverside Technology, Inc.

in a time series by interpolating between known values within the same time series. The command currently will not extrapolate past end points. The following dialog is used to edit the
command and illustrates the syntax of the command. FillInterpolate FillInterpolate() Command Editor The command syntax is as follows: FillInterpolate(Parameter=Value,…) Command Parameters
Parameter Description Default TSList Indicates the list of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards)
will be modified. • AllTS – all time series before the AllTS 179 Command Reference – FillInterpolate() -1
FillInterpolate() Command TSTool Documentation Parameter Description Default command. • EnsembleID – all time series in the ensemble will be modified. • FirstMatchingTSID – the first
time series that matches the TSID (single TSID or TSID with wildcards) will be modified. • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards)
will be modified. • SelectedTS – the time series are those selected with the SelectTimeSeries() command. TSID The time series identifier or alias for the time series to be modified,
using the * wildcard character to match multiple time series. Required when TSList=*TSID EnsembleID The ensemble to be modified, if processing an ensemble. Required when TSList=EnsembleID.
FillStart The starting date/time for the fill. Available period. FillEnd The ending date/time for the fill. Available period. MaxIntervals The maximum number of consecutive intervals
to fill (0 indicates no limits on the number of consecutive intervals that can be filled). 0 Transformation Indicate the data transformation to occur for interpolation. Currently, None
is the only option and is the default. Earlier versions used Linear. None (no transformation). FillFlag A string to flag data values that are filled. None – do not flag filled data.
A sample command file using data from the State of Colorado’s HydroBase is as follows: # 06707500 -SOUTH PLATTE RIVER AT SOUTH PLATTE 06707500.DWR.Streamflow.Month~HydroBase FillInterpolate(TSList=Al
lMatchingTSID,TSID="06707500.DWR.Streamflow.Month", MaxIntervals=3,Transformation=None) Command Reference – FillInterpolate() -2 180
Command Reference – FillMixedStation() -1 Command Reference: FillMixedStation() Fill missing data in dependent time series using the best fit from 1+ independent time series, using OLS
regression or MOVE2, data transforms, one/monthly equations Version 10.20.00, 2013-04-21 This command is under development. It is envisioned that the FillRegression() command enhancements
will be completed first. Then the FillMixedStation() analysis command will utilize much of the same logic, using the output statistics table to eliminate candidate relationships and
use the remaining relationships to calculate estimated values to check against standard error of prediction, etc. The FillMixedStation() command fills missing data in a time series where
one or more independent time series is used to sequentially fill missing data. This approach has been developed to automate analysis of regression filling (see Mixed Station Analysis
Tool below) and to facilitate batch filling of many related time series. This implementation is based on the Mixed Station Model implemented for Colorado’s Decision Support Systems (Ayres
Associates, 2000), which was based on the similarly named approach implemented by the USGS (Alley and Burns, 1981). The time series involved in the analysis are typically related, such
as being from nearby locations in a region. The main uses of the command are: 1. To automatically fill every time series in a data set, using other time series in the data set. For example,
for hydrologic modeling natural flow time series may have been estimated by processing measured streamflow, diversion, and reservoir time series. The natural flow time series can be
filled for use in modeling. 2. To generate a report on relationships, so that the user can configure individual FillRegression() and FillMOVE2() commands in TSTool. This may be appropriate
when using FillMixedStation() on a list of time series is inappropriate. Important: TSTool does not automatically exclude time series that have been filled in previous steps. Consequently,
care must be taken when specifying the list of independent time series to NOT use time series that were filled in a previous step. For each dependent time series being filled, the Mixed
Station Analysis (MSA) selects the independent time series and parameters that result in the best filling results, considering combinations of the following: • The list of independent
time series being considered can be constrained to a subset of available time series. • Filling methods include ordinary least squares (OLS) regression (see the FillRegression() command
for details) and MOVE2 (see the FillMOVE2() command for details). • One equation or monthly equations can be used. However, both options cannot be evaluated together due to the complexity
of ranking and reporting results. • The data can be transformed using log10, or no transformation can be applied. • A minimum number of overlapping data points (sample size N1) can be
specified to indicate a valid relationship. • A minimum correlation coefficient r can be specified to indicate a valid relationship. 181
FillMixedStation() Command TSTool Documentation Command Reference – FillMixedStation() -2 • A minimum confidence level for the slope of the regression line can be specified (see T-Test
discussion below). • The best fit indicator can be the correlation coefficient (R), or the standard error of prediction (SEP, described below). Because extensive analysis may be necessary
to evaluate all the combinations of parameters, the FillMixedStation() command will be slower than other commands that specifically indicate how to perform the filling. The number of
combinations can also be limited by reducing the number of parameter options and using stricter limitations on the number of overlapping points and correlation coefficients that are
required for a good regression result. The full MSA process is as follows: 1. For each dependent time series, perform a regression analysis using a unique combination of parameters (e.g.,
use an independent time series, OLS regression with one equation, no data transform). This results in 1+ regression results for each dependent time series. 2. Qualifying results (those
that meet the requirements of minimum number of overlapping points and correlation coefficient) are retained in a list for the dependent time series, for processing in the next step.
3. The qualifying results are used to estimate each missing value. Typically, the SEP is used to select the relationship to use (the one that has lowest SEP). 4. Missing data in the
dependent time series are filled using the regression results for he selected relationship. If missing values remain, the next highest ranking regression result is used until all missing
values are filled (or no additional qualifying regression results are available). Monthly filling occurs on each of the 12 months. This approach may use different stations because of
the goodness of fit of the relationship and because different stations may or may not have data that overlap the period to be filled. Implementation in Colorado’s Decision Support Systems
The Mixed Station Model implemented for the State of Colorado typically used the following input: • Log transform • Monthly relationships • Rank on SEP • Ordinary lease squares regression
• Minimum concurrent values = 5 • Confidence level = 95% • Fill all time series in data set 182
TSTool Documentation FillMixedStation() Command Command Reference – FillMixedStation() -3 The following dialog is used to edit the FillMixedStation() command and illustrates the syntax
of the command. Note that this interface will be updated to be similar to that of the FillRegression() command. FillMixedStation FillMixedStation() Command Editor The command syntax
is as follows: FillMixedStation(Parameter=value,…) Command Parameters Parameter Description Default DependentTSList Indicates the list of independent time series to be processed, one
of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards) will be processed. • AllTS – all time series before the command will be processed. •
EnsembleID – all time series in the ensemble will be processed. • FirstMatchingTSID – the first time series that matches the TSID (single TSID or TSID with wildcards) will be processed.
None – must be specified. 183
FillMixedStation() Command TSTool Documentation Command Reference – FillMixedStation() -4 Parameter Description Default • LastMatchingTSID – the last time series that matches the TSID
(single TSID or TSID with wildcards) will be processed.  SelectedTS – the time series selected with the SelectTimeSeries() command will be processed. DependentTSID The time series identifier
or alias for the dependent time series to be processed, using the * wildcard character to match multiple time series. Required if DependentTSList= *TSID. IndependentTSList Indicates
the list of independent time series to be considered for each dependent time series, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards)
will be processed. • AllTS – all time series before the command will be processed. • EnsembleID – all time series in the ensemble will be processed. • FirstMatchingTSID – the first time
series that matches the TSID (single TSID or TSID with wildcards) will be processed. • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards)
will be processed. • SelectedTS – the time series selected with the SelectTimeSeries() command will be processed. None – must be specified. IndependentTSID The time series identifier
or alias for the independent time series to be compared, using the * wildcard character to match multiple time series. Required if IndependentTSList= *TSID. 184
TSTool Documentation FillMixedStation() Command Command Reference – FillMixedStation() -5 Parameter Description Default BestFitIndicator Specifies the indicator to use when determining
the best fit, one of:  R – correlation coefficient  SEP – Standard Error of Prediction, defined as the square root of the sum of differences between the known dependent value, and
the value determined from the equation of best fit at the same point.  SEPTransformedpoin – same as SEP; however the data values have first been transformed as per the Transformation
parameter.  SEPTotal, when used with one equation, it is the same as SEP. When used with monthly equations, it is the SEP considering all months.  SEPTransformedTotal, when used with
one equation, it is the same as SEPTransformed. When used with monthly equations, it is the SEPTransformed considering all months. SEP AnalysisMethod Specify the method(s) to analyze
the data, in order to determine the best fit, including OLSRegression and/or MOVE2. If multiple methods are specified, separate with commas and surround with double quotes. OLSRegression
NumberOfEquations The number of equations to use for the analysis: OneEquation or MonthlyEquations. Only one may be chosen. If necessary, use more than one command to use different parameter
combinations for different groups of time series. None – must be specified. Transformation Indicates how to transform the data before analyzing. Specify as None (no transformation) or
Log (for Log10). If the Log option is used, zero and negative values in data are set to .001. Missing data are ignored. If multiple values are selected, separate with a comma and surround
with double quotes. None (no transformation) Intercept Specify as 0 to force the intercept of the best-fit line through the origin. This is made available only for OLS regression analysis
on untransformed data, to be consistent with the FillRegression() command. Do not force the intercept through zero. ConfidenceLevel Required confidence level for the T-Test on the regression
slope. Relationships not passing the test are not allowed for filling. No limit on confidence level. 185
FillMixedStation() Command TSTool Documentation Command Reference – FillMixedStation() -6 Parameter Description Default AnalysisStart The date/time to start the analysis, to focus on
a period appropriate for analysis. For example, specify the unregulated period for streamflow. If blank, analyze the full period. AnalysisEnd The date/time to end the analysis. If blank,
analyze the full period. FillStart The date/time to start filling, if other than the full time series period. If blank, fill the full period. FillEnd The date/time to end filling, if
other than the full time series period. If blank, fill the full period. MinimumDataCount The minimum number of overlapping data points that are required for a valid analysis (N1 in FillRegression()
and FillMOVE2() documentation). If the minimum count is not met, then the independent time series is ignored for the specific combination of parameters. For example, if monthly equations
are used, the independent time series may be ignored for the specific month; however, it may still be analyzed for other months. 10 MinimumR The minimum correlation coefficient required
for a best fit. If the minimum is not met, then the results are not considered in the best fit ranking or filling. 0.5 OutputFile Output file for the results, either as a file name to
be written to the working directory, or a full path. If not specified, partial results of the analysis may be available in the log file. The following example command file fills natural
flow time series from a StaeMod file using one equation (not monthly): # Test filling the gunnison monthly baseflow time series with # Mixed Station Analysis (all combinations for one
equation) StartLog(LogFile="fill-baseflow.log") ReadStateMod(InputFile="gunnv.xbg") FillMixedStation(BestFitIndicator=SEP,AnalysisMethod="MOVE2,OLSRegression", NumberOfEquations=OneEquation,
Transformation="Log,None",OutputFile="Results.txt") # Check for missing data -all should be filled CheckTimeSeries(CheckCriteria="Missing",MaxWarnings=10) # Check for negative flows
-should not be any CheckTimeSeries(CheckCriteria="<",Value1=0,MaxWarnings=10) 186
Command Reference: fillMOVE1() Fill Missing Time Series Data Using MOVE1 Procedure Version 06.08.02, 2004-08-02, Color, Acrobat Distiller The fillMOVE1() command has not been enabled.
This documentation serves as a reference for the MOVE1 procedure. Refer to the fillMOVE2() command. The fillMOVE1() command is more sophisticated than the fillRegression() command. Maintenance
of variance extension (MOVE) procedures are methods of fitting straight lines to data. The slope and intercept of the MOVE equations are computed differently than in ordinary least squares
(OLS) regression (see the fillRegression() command for a discussion of OLS regression). As shown below, an area of a triangle is minimized in the MOVE procedures rather than a vertical
distance as in OLS regression. The MOVE procedures do not provide the minimum-variance estimate of a single value but an ensemble of points estimated by the MOVE procedures will have
the same variability as the true values. MOVE procedures are useful in extending record at gaging stations where the extended record will be subsequently used in another analysis such
as frequency analysis. MOVE procedures will provide about the same estimates as OLS regression near the mean of the data but will provide smaller and larger estimates at the extremes
of the data set. The slope of the MOVE relation is steeper than OLS regression. The MOVE procedures are based on only one independent variable and the assumption is that there is a linear
relation between the dependent and independent variables. If the untransformed data are not linearly related, then it is common to transform the data using a logarithmic transformation.
The MOVE.1 procedure uses just the data from the N1 years of concurrent data. The MOVE.2 procedure (see the fillMOVE2() command) uses the Two-Station Comparison procedure described in
Appendix 7 of Bulletin 17B, Guidelines for Determining Flood Flow Frequency, USGS, to compute improved estimates of the mean and variance for the dependent time series and uses all the
data at the dependent time series to estimate the mean and variance of the dependent time series. The MOVE.2 procedure has been shown to be marginally better than MOVE.1. 187 Command
Reference – fillMOVE1() -1
fillMOVE1() Command TSTool Documentation Maintenance of Variance Extension (MOVE) (Xi, Yi) Minimize area of triangle Y X The MOVE.1 equation is used to estimate values for the dependent
time series from the independent time series: ⎥ ⎥⎦ ⎤ ⎢ ⎢⎣ ⎡ = 1 + − 1 ___ 11 ___ Y Y S X i X S i xy or i i Y = a + bX where = 1 N concurrent or overlapping period of record = 1 X mean
for independent variable for 1 years N Y 1 = mean for dependent variable for 1 years N = y1 S standard deviation for 1 years N = x1 S standard deviation for 1 years N 11 xy SS b = a
= Y 1 -bX 1 Note that the slope of the line does not include the correlation coefficient. This is the only difference between OLS regression and MOVE.1. Command Reference – fillMOVE1()
-2 188
Command Reference: FillMOVE2() Fill missing data in time series using the Maintenance of Variance Extension (MOVE.2) procedure Version 08.15.00, 2008-05-04 The FillMOVE2() command fills
missing data in a time series using the MOVE.2 procedure (see the FillMOVE1() command for background information). The MOVE.2 procedure uses the Two-Station Comparison procedure described
in Appendix 7 of Bulletin 17B, Guidelines for Determining Flood Flow Frequency, USGS, to compute improved estimates of the mean and variance at the dependent or short-term station and
uses all the data at the dependent time series to estimate the mean and variance of the dependent time series. The MOVE.2 procedure has been shown to be marginally better than MOVE.1.
The following MOVE.2 equation is used to estimate values for the dependent time series from the independent time series: ⎥⎦ ⎤ ⎢⎣ ⎡ Y = Y + X i − X SS i xy ___ ___ where = i Y discharge
for dependent time series = i X discharge for independent time series X = mean for independent time series for 1 2 N + N years ( N 2 is the additional years in the longterm time series)
= x S standard deviation for independent time series for 1 2 N + N years [ ( )] 2 1 1 2 2 1 b X X N N N Y Y − + = + (Equation 7-5a for Two-Station Comparison in Appendix 7 of Bulletin
17B) [ 2 ] 2 1 2 1 2 2 1 2 y1 2 1 1 2 2 1 1 x2 2 2 2 1 y1 1 2 2 y b (X X ) N N N N (1 r )S (N 3)(N 2)N (N 4)(N 1) (N 1)S (N 1) b S (N N 1) S 1 − + − + − − − − − + − + + − = (Equation
7-10 for Two-Station Comparison in Appendix 7 of Bulletin 17B) where = r = SS b r xy , 11 correlation coefficient (Note that b is the slope of the ordinary least squares regression line.)
= 1 N concurrent or overlapping period of record = 2 N additional years available at long-term site = 1 X mean of independent time series for 1 years N = 2 X mean of independent time
series for 2 years N = y1 S standard deviation of dependent time series for 1 years N = x1 S standard deviation of independent time series for 1 years N 189 Command Reference – FillMOVE2()
-1
FillMOVE2 () Command TSTool Documentation The following dialog is used to edit the command and illustrates the command syntax. FillMOVE2 FillMOVE2() Command Editor Command Reference
– FillMOVE2() -2 190
TSTool Documentation FillMOVE2() Command The command syntax is as follows: FillMOVE2(Parameter=Value,…) Command Parameters Parameter Description Default TSID The time series identifier
or alias for the time series to be filled (dependent time series). None – must be specified. IndependentTSID The time series identifier or alias for the independent time series, to supply
data. None – must be specified. NumberOf Equations OneEquation or MonthlyEquations, indicating how many relationships are to be determined. OneEquation Transformation Log or None, indicating
the type of data transformation. If the Log option is used, zero and negative values are set to .001 (-999 values are treated as missing data and are ignored), and the data values are
transformed using log10. None Dependent Analysis Start/End The period for N1 (overlapping data) that is used to analyze the dependent time series. For example, this may be the unregulated
period for streamflow data. Typically, this is longer than the independent analysis period. Analyze the full period. Independent Analysis Start/End The period for N2 (non-overlapping
data) that is used to analyze the independent time series. For example, this may be the unregulated period for streamflow data. Analyze the full period. FillStart The date/time to start
filling. Fill the full period. FillEnd The date/time to end filling. Fill the full period. FillFlag A single character to be used to flag filled points on graphs and other output. Do
not flag filled data. 191 Command Reference – FillMOVE2() -3
FillMOVE2 () Command TSTool Documentation A sample command file illustrating how to fill time series from the State of Colorado’s HydroBase is as follows (MOVE2 and ordinary least squares
regression are used to allow comparing the results): StartLog(LogFile="Results/commands.TSTool.log",Suffix="Date") SetOutputPeriod(OutputStart="1901-01",OutputEnd="2004-12") # 06758500
-SOUTH PLATTE RIVER NEAR WELDONA 06758500.DWR.Streamflow.Month~HydroBase # 06754000 -SOUTH PLATTE RIVER NEAR KERSEY 06754000.DWR.Streamflow.Month~HydroBase FillMOVE2(TSID="06758500.DWR.Streamflow.Mon
th", IndependentTSID="06754000.DWR.Streamflow.Month", NumberOfEquations=MonthlyEquations,DependentAnalysisStart="1952-10", DependentAnalysisEnd="2004-09",IndependentAnalysisStart="1901-01",
IndependentAnalysisEnd="1950-12",FillStart="1930-01", FillEnd="1940-12",FillFlag="m") # 06758500 -SOUTH PLATTE RIVER NEAR WELDONA 06758500.DWR.Streamflow.Month~HydroBase # 06754000 -SOUTH
PLATTE RIVER NEAR KERSEY 06754000.DWR.Streamflow.Month~HydroBase FillRegression(TSID="06758500.DWR.Streamflow.Month", IndependentTSID="06754000.DWR.Streamflow.Month") Command Reference
– FillMOVE2() -4 192
Command Reference: FillPattern() Fill missing time series data using historical average patterns Version 08.16.04, 2008-09-19 The FillPattern()command fills missing data in a time series
using historic averages based on a pattern file. For example, if May 1910 is missing and the pattern indicates that May 1910 is a WET month, then the average of all WET Mays is used
to fill the time series. The pattern file indicates the WET/DRY/AVG patterns and the time series to be filled supplies data to compute averages, for use in filling. This feature is enabled
for monthly data only. Averages are computed as described for the FillHistMonthAverage() command. There is currently no way to limit the fill operation to a period (the entire time series
is filled). The pattern file is created with the AnalyzePattern() command and a saved file must be read with a ReadPatternFile() command. See below for an example of a fill pattern file.
One or more patterns can be included in each pattern file, similar to StateMod time series files (see the StateMod Input Type appendix), and multiple pattern files can be used, if appropriate.
# Years Shown = Water Years # Missing monthly data filled by the Mixed Station Method, USGS 1989 # Time series identifier = 09034500.CRDSS_USGS.QME.MONTH.1 # Description = COLORADO RIVER
AT HOT SULPHUR SPRINGS, CO. # -e-b----------eb------eb------eb------eb------eb------eb------eb------eb------eb------eb------eb------eb------eb--------e 10/1908 -9/1996 ACFT WYR 1909
09034500 AVG AVG AVG WET WET AVG AVG AVG WET WET WET WET 1910 09034500 WET WET WET WET WET WET AVG AVG AVG AVG AVG AVG 1911 09034500 AVG AVG WET AVG AVG AVG AVG WET WET WET AVG WET 1912
09034500 WET WET WET WET WET AVG AVG WET WET WET WET WET ...ommitted... The following dialog is used to edit the FillPattern() command and illustrates the syntax of the command. FillPattern
FillPattern() Command Editor 193 Command Reference – FillPattern() -1
FillPattern() Command TSTool Documentation The command syntax is as follows: FillPattern(Parameter=Value,…) Command Parameters Parameter Description Default TSList Indicates the list
of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards) will be modified. • AllTS – all time series before
the command. • EnsembleID – all time series in the ensemble will be modified. • FirstMatchingTSID – the first time series that matches the TSID (single TSID or TSID with wildcards) will
be modified. • LastMatchingTSID – the last time series that matches the TSID (single TSID or TSID with wildcards) will be modified. • SelectedTS – the time series are those selected
with the SelectTimeSeries() command. AllTS TSID The time series identifier or alias for the time series to be modified, using the * wildcard character to match multiple time series.
Required for TSList=*TSID. EnsembleID The ensemble to be modified, if processing an ensemble. Required for TSList=EnsembleID. PatternID The pattern identifier, matching a pattern read
with ReadPatternFile() commands. None – must be specified. A sample command file to process data from the State of Colorado’s StateMod model is as follows: # Read StateMod time series
to fill ReadStateMod(InputFile="..\StateMod\sjm_prelim.ddh") # Read the file containing the patterns ReadPatternFile(PatternFile="fill.pat") # Fill time series having identifiers that
start with "30" FillPattern(TSList=AllMatchingTSID,TSID="30*",PatternID="09034500") # Write the results WriteStateMod(TSList=AllTS,OutputFile="..\StateMod\sjm.ddh") The above example
fills all diversion time series with identifier starting with 30, using the pattern 09034500 (a stream gage for the region). Command Reference – FillPattern() -2 194
Command Reference: FillPrincipalComponent Analysis() Fill missing time series data using principal component analysis (PCA) Version 09.04.00, 2009-06-11 This command is under development.
19 5 Command Reference – FillPrincipalComponentAnalysis() -1
FillPrincipalComponentAnalysis() Command TSTool Documentation This page is intentionally blank. Command Reference – FillPrincipalComponentAnalysis() -2 196
Command Reference: FillProrate() Fill missing time series data by prorating values in another time series Version 08.16.04, 2008-09-30 The FillProrate() command fills missing data in
time series by prorating values from another time series. This fill technique is useful, for example, where two time series are likely to have the same general trend and ratio of data
values. The ratio can be computed two ways, as specified by the FactorMethod parameter: • NearestPoint – causes the ratio to be recomputed each time that a non-missing value is found
in both time series. The ratio computed from the nearest points in each time series is used for filling until another value can be computed. • AnalyzeAverage – computes the ratio as
the average ratio of the time series (numerator) and the independent time series (divisor). This was implemented to match an existing fill procedure but can lead to some bias in the
results. A different overall average will be obtained depending on whether ratios are computed first and then averaged than if the sum of the numerators are added and divided by the
sum of the denominators. In the former, the choice of which time series is in the denominator could impact results. More parameters may need to be added in the future to implement an
analysis different from the current defaults. The initial computation of the ratio may require specifying an initial value due to missing data on the endpoints of the time series (see
the InitialValue parameter). Alternatively, the time series can be filled in one direction first and then filled in the other direction with a second command. 197 Command Reference –
FillProrate() -1
FillProrate() Command TSTool Documentation The following dialog is used to edit the command and illustrates the syntax of the command: FillProrate FillProrate() Command Editor Command
Reference – FillProrate() -2 198
TSTool Documentation FillProrate() Command The command syntax is as follows: FillProrate(Parameter=Value,…) Command Parameters Parameter Description Default TSList Indicates the list
of time series to be processed, one of: • AllMatchingTSID – all time series that match the TSID (single TSID or TSID with wildcards) will be modified. • AllTS – all time series before