Order Online Power Walk Forward
Optimizer
Walk Forward
Metric Explorer
Walk Forward
Input Explorer
Walk Forward
Surface Explorer
Key Daily & Intraday
Trading Strategies
Nth Order Fixed Memory
Polynomial Strategy
Nth Order Fading Memory
Polynomial Strategy
End Point Fast Fourier
Transform Strategy
Goertzel DFT
Strategy
Five Parameter
Parabolic Strategy
Dennis Meyers
Working Papers

 The Walk Forward Out-Of-Sample Performance Metric Explorer v7

Click Here For Video: How To Use The Walk Forward/Out-Of-Sample PWFO Metric Explorer

The Walk Forward Out-Of-Sample Performance Metric Explorer v7

The Walk Forward Metric Explorer (WFME) reads all files generated by the PWFO and searches each PWFO file for the in-sample Metric filters that generate the statistically best average out-of-sample performance. The Top N Metric filter chooses the in-sample rows that contain then Top N (N=10 or 50 or your choice) values of a PWFO Performance Metric after in optional in-sample maximum consecutive losers(LR) in a row, minimum number of in-sample trades(nT) and maximum Profit Factor(PF) elimination screen. From the N rows chosen, the WFME chooses the maximum of another in-sample PWFO metric. This allows two performance metrics in the in-sample section to be used to find the strategy inputs that give the best out-of-sample returns. Experience has shown that the use of only one in-sample performance metric to choose strategy inputs from the in-sample section does not produce good out-of-sample results. The Top N criteria ranges are user selectable generating many filter searches in one run. The WFME v7 is a stand alone 64bit exe program that is super fast and automatically displays it's extensive statistical results in Excel. In addition, using modern "Bootstrap" techniques, the WFME calculates the probability of whether or not each filter's out-of-sample results were due to chance. WFMEv7
Click here to see the WFME Input Form and a description of the inputs

Data Mining and Curve Fitting.

Data mining or curve fitting a price series will always produce the best performance results. If you look hard enough using optimization you will always find patterns in the price data by mistaking chance fluctuations for genuine effects. But they are not real. Why use the PWFO walk forward technique? Why not just perform a TradeStation optimization on the whole price series and choose the input parameters that give the best total net profits or profit factor? Whenever you run a optimization (combinatorial search} over many different combinations of input parameters on noisy data on a fixed number of prices, no matter how many, the best performance parameters found are guaranteed to be due to "curve fitting" the noise and signal. When we run, say, 5000 different input parameter combinations , the best performance parameters will be from those system input variables that are able to produce profits from the price pattern and the random spurious price movements While the price patterns, if there, will repeat, the same spurious price movements will not. If the spurious movements that were captured by a certain set of input parameters were a large part of the total net profits, then choosing these input parameters will produce losses when traded on future data. These losses occur because the spurious movements will not be repeated in the same way. This is why system combinatorial searches with no out-of-sample testing cause loses when traded in real time from something that looked great in the test section.

In order to gain confidence that our system input parameter selection procedure on test data will produce profits "on average" in the future we must perform the walk forward out-of-sample analysis many times. Why not just do the analysis once? Well just as in poker, where there is considerable vagaries in hand to hand luck, walk forward out-of-sample analysis give considerable vagaries in week to week out-of-sample profit "luck". That is, by pure chance we may have chosen some input parameters that did well in the test section data and the out-of-sample section data. In order to minimize this type of "luck", statistically, we must repeat the walk forward out-of-sample (oos) analysis over many test/oos sections and take the average of our weekly results over all out-of-sample sections (we need at least 50 oos sections for statistically significant results). This average gives us an expected weekly return and a standard deviation of weekly returns which allows us to statistically estimate the expected equity and it's range for N weeks in the future.

Walk Forward Performance Metric Explorer (WFME) Description.

Here is an example of just one of the many filters the WFME can generate in a run and the logic behind this particular filter. Very few strategies can sustain Profit Factors (PF) above 3 or 4 over time. High profit factors (PF) in the in-sample section usually mean poor performance in the out-of-sample-section. This is a kind of reversion to the mean. Therefore if we eliminate from the in-sample optimization results, all rows that have profit factors greater than 3, we will likely eliminate many of the cases that are due to curve fitting the noise. Also In addition, because in real time it is difficult to sustain more than seven losses in a row and still keep trading, we will eliminate all those in-sample rows that have more than 7 losses in a row(LR). After using a PF and LR filter as above there can still be 100's of rows left in the in-sample section of the PWFO file. The PWFO generates the metric eqR2 which is the Trade Equity Regression Trend Line Coefficient of Correlation in the test section. As an example, let us choose the rows that contained the Top 50 eqR2 values from the rows that are left from the PF-LR screen. That is the WFME sorts eqR2 from high to low, eliminates the rows that have PF>3 and LR>7 and then chooses the Top 50 Rows of whatever is left. This particular filter will now leave 50 cases or rows in the PWFO file that satisfy these filter conditions. We call this filter t50eqR2|p<3|lr7 where t50eqR2 means the Top 50 eqR2 Rows left after the PF-LR filter. p<3 means PF<=3, "lr7" means LR<=7. Suppose for this filter, within the 50 PWFO rows that are left, we want the row that has the maximum PWFO mTrd metric (the maximum median trade profit) in the in-sample section. This would produce a filter named t50eqR2|p<3|lr7-mTrd. This filter leaves only one row in the PWFO in-sample section with it's associated strategy inputs and out-of-sample net profit in the out-of-sample section. This particular t50eqR2|p<3|lr7-mTrd filter is then calculated for the in-sample section of each of the PWFO files that are run by the WFME and the average out-of-sample performance from this in-sample filter, plus many other important statistics ( as shown here) is summarized along with the hundreds of other filter combinations that are constructed in a similar manner. This filter summary is sorted by total out-of-sample net profits after costs for all the various filters and written to a comma delimited Excel file by the WFME. When The WFME run is done just click on the Excel Icon next to the Run button and the WFME output file will appear in Excel or your spreadsheet

Bootstrap Probability of Filter Results.

Using modern "Bootstrap" techniques, we can calculate the probability of obtaining each filter's total out-of-sample net profits by chance. By net we mean subtracting the cost and slippage of all round trip trades from the total out-of-sample profits. Here is how the bootstrap technique is applied. Suppose we have 300 out-of-sample PWFO files with 8000 different strategy input combination rows. Note each of the 8000 different strategy inputs rows have an in-sample section and an out-of-sample section in each of the 300 PWFO files. A given filter on the in-sample section chooses a set of strategy inputs based upon the filter parameters and those strategy inputs chosen generate net profits or losses when applied to the out-of-sample section. Applying this filter to each of the 300 in-sample sections gives in-sample strategy inputs that generate net profits or losses in each of the 300 out-of-sample sections. The sum of all the net profits and losses in the 300 out-of-sample sections generate the total net profits for that particular filter. Each of the different filters total net profits are calculated in the same way. Let us create a mirror random filter. However with the random filter, the random filter picks a random row's out-of-sample net profits in each of the 300 PWFO files (there are 8000 different rows from the 8000 different strategy inputs and the random filter chooses one of those 8000 rows at random). Suppose we repeat this random row section 5000 times. Each of the 5000 mirror filters will choose a random row's out-of-sample net profit(OSNP) of their own in each of the 300 PWFO files. At the end, each of the 5000 mirror filters will have 300 random OSNP's picked from the rows of the 300 PWFO files. The sum of the 300 random OSNP picks for each random filter will generate a random total out-of-sample net profit(tOnpNet). The average and standard deviation of the 5000 mirror filter's different random tOnpNets will allow us to calculate the chance probability for each our filter's tOnpNet. Thus given the randon filter's bootstrap random tOnpNet average and standard deviation, we can calculate the probability of obtaining our filter's tOnpNet by pure chance alone.

The WFME Ave File Definition

After the WFME has completed it's run click in the Excel icon above Ave File on the WFME form and the Ave File automatically loads into Excel. Below is a snippet of the Ave File output from a WFME run sorted by the total out-of-sample net profit statistic (tOnpNet). This example shows the partial output file from a WFMEx64v7 run on the PWFOv5t files generated with the End Point Fast Fourier Transform Strategy that was run on 1 contract of the Crude Light(CL) 5 minute bar futures for the 378 days from 8/11/2014 to 02/12/2016. The in-sample(IS) period is 5 trading weekdays and the out-of-sample(OOS) period is the next trading weekday following. The days are weekdays only. Weekdays where the OOS falls on a exchange holiday or partial days are eliminated. Holidays that fall on a weekday create a 4day IS. All other IS periods consist of 5 trading weekdays. Sundays were skipped because this strategy only traded between 8am to 14:30pm CST


WFME Example a WFME Example b

The WFME Columns are defined as follows

  • Row 1, Columns: A=The PWFO Stub, B=File Start Date, C=File End Date, D= Number of oos periods(in this example weeks), M= Bootstrap average, N= Bootstrap Standard Deviation, O=Number of filters run, U= Cost/trade.
  • Row 2, Columns:
  • Filter-Metric = The filter that was run on the 378 PWFO in-sample sections. For example on Row 4, t50mWTr|p<1.5|l3-tWb is: top 50 median of winning trades in the in-sample sections after eliminating the in-sample rows that don't meet the following criteria: Profit Factor <=1.5. and consecutive losers in a row <=3. Out of the 50 rows left the WFME chooses the in-sample row with the highest tWb(total winning bars) metric.
  • toGP = Total out-of-sample(oos) gross profit for these 378 oos periods(for this run periods = weeks).
  • aoGP = Average oss gross profit for the 378 oos periods
  • aoTr = Average oos trade gross profit
  • aoW = Average of the out-of-sample profits of winning trades
  • aoL = Average of the out-of-sample profits of losing trades
  • oW|oL = Ratio of average oos winning trades(aoW) divided by average oos losing trades(aoL) .
  • PFtr = Ratio of total winning trades divided by total losing trades
  • %Wtr = The percentage if oos winning trades
  • t = The student t statistic for the 378 oos periods. The higher the t statistic the higher the probability that this result was not due to pure chance
  • std = The standard deviation of the 378 oos periods
  • LLtr = The largest losing oos trade in all oos periods
  • LLp = The largest losing oos period
  • eqDD = The oos equity drawdown
  • LRp = The largest number of losing oos periods in a row
  • # = The number of oos periods this filter produced any profit or loss. Note for some oos periods there can be no strategy inputs that satisfy a given filters criteria and no trades will be made during that period.
  • eqTrn = The straight line trend of the oos equity curve in $/oos period.
  • eqV^2 = The velocity of a 2nd order polynomial that is fit to the equity curve.
  • eqR2 = The correlation coefficient(R^2) of a straight line fit to the equity curve.
  • Dev^2 = A measure of equity curve smoothness. The square root of the average (equity curve minus a straight line)^2)
  • Blw = The maximum number of oos periods the oos equity curve failed to make a new high.
  • BE = Break even in oos periods. Assuming the average and standard deviation are from a normal distribution, this is the number of oos periods you would have to trade to have a 98% probability that your oos equity is above zero.
  • tOnpNet = Total out-of-sample gross profit(toGP) minus the total trade cost. tOnpNet=toGP - (#)*ao#T*Cost.
  • Prob = The probability that the filters tOnpNet was due to pure chance. For example, for row 4 where tOnpNet=69,180 there is a 1 in 2079 (4.81E-04) chance that the tOnpNet obtained by this filter was due to chance. Note for our filter, in row 4, the expected number of cases that we could obtain by pure chance that would match or exceed the $69,180 is 14415x 4.81x10-4 = 6.9 where 14415 is the total number of different filters we looked at in this run. This number basically says this if we repeated the random selection 14415 times we would find approximately 7 cases on average that would produce better profits than our filter. I usually like to see this number much less than 1. However the 14415 bootstrap random average was $21,351 which meant that 50% of the random picks were at or above $21,351. This indicates that this strategy is in general a profitable strategy no matter what the inputs. Thus in my opinion the 7 random cases that could beat our filter does not negate the validity of the out-of-sample filter chosen.

How To generate an Equity Chart plus a Table with the Equity and Strategy Inputs from the WFME Ave File

Shown below again is a snippet of the WFME Ave File discussed above. To generate an Equity plot of any given Filter we would select the Filter from Column A as shown in the snippet and then click on the Excel Add-Ins and then click on the black circled Meyers Analytics Macro Icon shown in the snippet. The Excel Equity plot and Table would then immediately appear in Excel as shown below. Note the equity plot shows the equity with costs (brown line, new equity highs are shown as green dots) and without costs(blue line, new equity highs are shown as red dots).Any number of Filters could be selected, one at a time by highlighting the column A filter and clicking on the Add-In macro shown , and an Equity chart plus Table would be produced. This Meyers Analytics Excel Equity Chart Add-In macro is supplied with the Walk Forward Metric Explorer v7 and all the other explorers.


WFME v7 Ave File Plot Chart Example

Equity Plot Generated By The Meyers Analytics Excel Add-In Macro Shown Above


WFME v7 Equity Plot
Brown line plot = Net Equity, Equity minus trade costs. New equity highs are shown as green dots.
Blue line plot = Equity without costs. New equity highs are shown as green dots.

Excel Table Generated By The Meyers Analytics Excel Add-In Macro Shown Above


WFPE v7 Equity Table

The Excel Table Columns are defined as follows

  • Column A The Filter-Metric. Example t50mWTr|p<1.5|l3-tWb
  • Column B The PWFO File dates
  • Column C = The out-of-sample(oos) gross profit for each date using the in-sample strategy inputs.
  • Column D = The out-of-sample(oos) number of trades for each date using the in-sample strategy inputs.
  • Column E = The out-of-sample(oos) Equity running total.
  • Column C = The out-of-sample(oos) net profit (Gross Profit-OnT*Cost) for each date using the in-sample strategy inputs.
  • Column G = The out-of-sample(oos) Net Equity (equity minus costs) running total.
  • Columns H-L = Strategy Inputs found by the Filter in the in-sample section that gave the out-of-sample results on this same line.

Please see video for a visual explanation. Click Here For Video: How To Use The Walk Forward PWFO Metric Explorer v3>

The Walk Forward PWFO Metric Explorer comes with a detailed manual explaining:
  • How to setup, install and run the Walk Forward Performance Metric Explorer. The Walk Forward PWFO Metric Explorer is a stand alone exe file that can be executed directly from your desktop icon or from the Windows Start Program menu.
  • How to use the WFME with your PWFO files.
  • An explanation of each of the performance statistics columns.
  • How to select the best WFME filter to use on real time runs (Past performance is no guarantee of future results).

The WFME Input Finder Excel Add-In
Supplied with the WFME is an Excel Add-In Filter. For the Excel Add-In, as shown below, you just click on a special Add-In icon on the Excel Toolbar and a pop up window displays. Fill in the WFME parameters and click on the Run button. The Excel Add-In macro will filter any number of PWFO files loaded in Excel, and will display the PWFO Excel file row that satisfies the typed in criteria.

WFME v7 XLA                  

The The Walk Forward PWFO Metric Explorer package consisting of Manual, WFME EXE file, and WFME Excel Add-Ins is being offered, for $395. Please note that the WFME will only read files generated by the PWFO product. The WFME has a "Key Licence" that only allows it to be installed on three computers.



How To Order
To order online click Order Online. If you would like to talk to me about the product, please call me at (312) 280-1687 M-F 12pm to 5pm CST. All E-mail queries can be sent to support@meyersanalytics.com.

Thank you for your Interest....Dennis Meyers



top Top | Home