The Walk Forward/Out-Of-Sample Performance Explorer v5

Click Here For Video: How To Use The Walk Forward/Out-Of-Sample Performance Explorer

The Walk Forward/Out-Of-Sample Performance Explorer (WFPE) reads all files generated by the PWFO and searches the PWFO performance metric variables in each of those files for those performance metrics that generate the statistically best average out-of-sample returns. The WFPE eliminates all cases in the PWFO test files that do not meet certain Profit Factor (PF), Losers in a row (LR), Number of trades (NT) and Performance Metrics criteria . The PF, LR, Metric, and NT criteria ranges are user selectable generating many filter searches in one run. The WFPE is a stand alone exe program that is super fast. It can run 100 PWFO files in less that 2 minutes and automatically display the results in Excel. In addition, using modern "Bootstrap" techniques, the WFPE calculates the probability of whether or not each filter's out-of-sample results were due to chance. No other Walk Forward software offers this capability.

Click here to see the WFPE Input Form and a description of the inputs

Data Mining and Curve Fitting. Data mining or curve fitting a price series will always produce the best performance results. If you look hard enough using optimization you will always find patterns in the price data. But they are not real. Why use the PWFO walk forward technique? Why not just perform a TradeStation optimization on the whole price series and choose the input parameters that give the best total net profits or profit factor? Whenever you run a optimization (combinatorial search} over many different combinations of input parameters on noisy data on a fixed number of prices, no matter how many, the best performance parameters found are guaranteed to be due to "curve fitting" the noise and signal. When we run, say, 5000 different input parameter combinations , the best performance parameters will be from those system input variables that are able to produce profits from the price pattern and the random spurious price movements While the price patterns, if there, will repeat, the same spurious price movements will not. If the spurious movements that were captured by a certain set of input parameters were a large part of the total net profits, then choosing these input parameters will produce losses when traded on future data. These losses occur because the spurious movements will not be repeated in the same way. This is why system combinatorial searches with no out-of-sample testing cause loses when traded in real time from something that looked great in the test section.

In order to gain confidence that our strategy input selection procedure on the in-sample(test) data will produce profits "on average" in the future we must perform a walk forward out-of-sample analysis many times. Why not just do the analysis once? Well just as in poker, where there is considerable vagaries in hand to hand luck, walk forward out-of-sample analysis give considerable vagaries in week to week out-of-sample profit "luck". That is, by pure chance we may have chosen some input parameters that did well in the in-sample(test) section data and the out-of-sample section data. In order to minimize this type of "luck", statistically, we must repeat the walk forward out-of-sample (oos) analysis over many test/oos sections and take the average of our weekly results over all out-of-sample sections (we need at least 30 oos sections for statistically significant results). This average gives us an expected weekly return and a standard deviation of weekly returns which allows us to statistically estimate the expected equity and it's range for N weeks in the future.

Walk Forward Performance Explorer (WFPE) Description. As stated above, curve fitting or over fitting the price series with strategy inputs by optimization will, as a rule of thumb, always produce the best in-sample(test) performance results. If we eliminate the optimization cases with the best performance results, we are sure to eliminate many of the curve fitting system input parameters that fitted the past spurious noise movements of the price series. Here is an example of one of the filters that the WFPE can produce: Very few systems can sustain Profit Factors (PF) above 3 over time. Here is an explanation of a result(Row 3) shown in the output snippet below. If we eliminate from the in-sample(test sample) optimization results, all cases that have profit factors greater than 2, we will likely eliminate most of the cases that are due to curve fitting the noise. Also, for this example, we like to filter for only those cases in the test section that had a positive net profit. In addition, because in real time it is difficult to sustain more than five losses in a row and still keep trading, we will eliminate all those in-sample cases that have more than 5 losses in a row. The PWFO generates the metric R22 which is the Trade Equity 2nd Order Polynomial Trend Line Coefficient of Correlation in the in-sample section. We would like to exclude any cases in the in-sample section that have an R22<70. This particular filter might leave anywhere from 10 to 200 cases or rows in the PWFO file that satisfy all these filter conditions. What is left in the PWFO file after applying the above filter are 10 to 200 rows with 31 columns of PWFO metric statistics If we look for the maximum value in each one of the 31 PWFO columns in the rows that are left we have up to 31 different filters for this PF, LR, R2 screen. We call this filter PF<20|5|r22>70|12 where PF<20 means PF<=2.0, "|5" means LR<=5 and r22>70 means r22≥70 and |12 means number of trades in the in-sample section ≥12. Suppose for this filter, for the PWFO rows that are left we want the row that has the maximum PWFO wlb metric. The wlb metric is the Total Bars in Winning Trades divided by the Total Bars In Losing Trades. This would produce a filter named PF<20|5|r22>70|12-wlb. This particular PF<20|5|r22>70|12-wlb is calculated for each of the PWFO files that are run by the WFPE and the average out-of-sample performance, plus other important statistics for this filter is summarized along with the hundreds of other filter combinations that are constructed in a similar manner. This filter summary is sorted by total out-of-sample Net profits after costs for all the various filters and written to a comma delimited file by the WFPE. When The WFPE run is done just click on the "Av" Excel Icon and the WFPE output file will appear in Excel as shown below

Bootstrap Probability of Filter Results. Using modern "Bootstrap" techniques, the WFPE calculates the probability of obtaining each PF-LR-Metric-NT filter's total out-of-sample net profits by chance. Here is how the bootstrap technique is applied. Suppose, for this example, the WFPE calculates the total out-of-sample net profits(tONet) of 700 different PF-LR-Metric-NT filters. A mirror filter is created for each of the 700 WFPE filters. However, instead of picking an out-sample net profit(OSNP) from a filtered row, the mirror filter picks a random row's OSNP in each PWFO file. Each of the 700 mirror filters will choose a random row's OSNP of their own in each of the PWFO files. Thus if there are 50 PWFO files, each of the 700 mirror filters will pick a random row's OSNP in each of the 50 PWFO files. At the end, each mirror filter will have 50 random OSNP's picked from the rows of the 50 PWFO files. The sum of the 50 random OSNP picks for each mirror filter will generate a random total out-of-sample net profit(tONet) for each of the 700 mirror filters. The average and standard deviation of the 700 mirror filter's random tONets will allow us to calculate the chance probability of each WFPE filter's tONet. Thus given the mirror filter's bootstrap random tONet average and standard deviation, we can calculate the probability of obtaining the PF-LR-Metric-NT filter's tONet by pure chance alone.

The WFPE v5 WFR File Output

Below is a snippet of the output from a WFPE run sorted by the total out-of-sample net profit statistic (tONet). This example shows the partial output file from a WFPE run on the PWFO files generated by the Repeated Median Velocity Strategy that was run on 1 contract of the Russell 2000 Index 1 minute bar futures for the 156 weeks ending 1/23/09 to 01/13/12.

---

---

The WFPE Columns are defined as follows

• Row 1, Columns: A=The PWFO Stub, B=File Start Date, C=File End Date, D= Number of oos periods(in this example weeks), E=tnp>0, J= Bootstrap average, K= Bootstrap Standard Deviation, L=Number of filters run, T= Cost/trade.
• Row 2, Columns:
• Filter = The filter that was run. For example on Row 3, PF<20|5|r22>70|12-wlb is: PF<=2.0, LR<=5, R22>=70, Number of trades/in-sample section>=12. Another example: pfxx|4|r22>60|10 is: any PF, LR<=4, R2>=60, Number of trades/in-sample section>=10
• Metric = The PWFO performance metric (defined on the PWFO page). For this WFPE filter, pf<20|5|r22>70:12-wlb the Metric=wlb (Total winning bars/total losing bars in in-sample section), this PWFO metric produced the following average 156 week statistics on this line.
• tOnp = Total out-of-sample(oos) net profit for these 156 oos periods(for this run periods = weeks).
• mOsp = median oos net profit for the 156 weeks
• aOnp = Average oss net profit for the 156 weeks
• aOTrd = Average oos profit per trade
• aO#T = Average number of oos trades per week
• B0 = The 156 week trend of the out-of-sample weekly profits
• %P = The percentage of oos weeks that were profitable
• t = The student t statistic for the 156 weekly oos profits. The higher the t statistic the higher the probability that this result was not due to pure chance
• std = The standard deviation of the 156 weekly oos profits
• llosp = The largest losing oos period
• eqDD = The oos equity drawdown
• lr = The largest number of losing oos weeks in a row
• # = The number of weeks this filter produced a weekly result. Note for some weeks there can be no strategy inputs that satisfy a given filter's criteria.
• eqTrn = The straight line trend of the oos equity curve in $/week.
• eqR2 = The R squared of the straight line equity fit.
• Blw = The maximum number of weeks the oos equity curve failed to make a new high.
• BE = Break even weeks. Assuming the average and standard deviation are from a normal distribution, this is the number of weeks you would have to trade to have a 99% probability that your oos equity is above zero.
• ndd = The normalized equity drawdown = 100*eqDD/tONet
• tOnpNet = Total out-of-sample net profit(tOnp) minus the total trade cost. tOnpNet=tOnp - #*aOnT*Cost.
• Prob = The probability that the filter's tOnpNet was due to pure chance. For example, for row 3 where tOnpNet=104750 there is a 1 in 107875 (9.27E-06) chance that the tOnpNet obtained by this filter was due to chance.
• Notice: Past performance is no guarantee of future results

How To generate an Equity Chart plus a Table with the Equity and Strategy Inputs from the WFR File

Shown below again is a snippet of the WFR File discussed above. To generate an Equity plot of any given Filter we would select the Filter from Column A as shown in the snippet and then click on the Excel Add-Ins and then click on the black circled Meyers Analytics Macro Icon shown in the snippet. The Excel Equity plot and Table would then immediately appear in Excel as shown below. Any number of Filters could be selected, one at a time by highlighting the column A filter and clicking on the Add-In macro shown , and an Equity chart plus Table would be produced. This Meyers Analytics Excel Equity Chart Add-In macro is supplied with the the Walk Forward performance Explorer v5.

---

---

Equity Plot Generated By The Meyers Analytics Excel Add-In Macro Shown Above

---

---

Excel Table Generated By The Meyers Analytics Excel Add-In Macro Shown Above

---

The Excel Table Columns are defined as follows

• Row 1 is the Date, Filter-Metric, Osnp, Ont and Strategy Inputs found in the in-sample section by the Filter..
• Column A The PWFO File dates
• Column B The Filter-Metric. Example PF<20|5|r22>70|12-wlb
• Column C = The out-of-sample(oos) net profit for each date for these in-sample strategy inputs.
• Column D = The out-of-sample(oos) number of trades for each date for these in-sample strategy inputs.
• Column E = The out-of-sample(oos) Equity running total.
• Column F = The out-of-sample(oos) Net Equity (equity minus costs) running total.
• Columns G-K = Strategy Inputs found by the Filter in the in-sample section that gave the out-of-sample results on this same line.

Please see video for a visual explanation. Click Here For Video: How To Use The Walk Forward Performance Explorer v4>

The Walk Forward Performance Explorer comes with a detailed manual explaining:

• How to setup and install the Walk Forward Performance Explorer. The Walk Forward Performance Explorer is a stand alone exe file that can be executed directly from your desktop icon or from the Windows Start Program menu.
• How to use the WFPE with your PWFO files.
• An explanation of each of the performance statistics columns.
• Walk Forward Optimization tutorial.

The WFPE Input Finder Excel Add-In and WFPE Strategy Input finder EXE.

Supplied with the WFPE is an Excel Add-In Filter and a separate WFPE Input Finder EXE. For the Excel Add-In, as shown below, you just click on a special Add-In icon on the Excel Toolbar and a popup window displays. Fill in the WFPE parameters and click on the Run button. The Excel Add-In macro will filter any number of PWFO files loaded in Excel, and will display the PWFO Excel file row that satisfies the typed in criteria. For the WFSE Input finder, as shown below, fill in the WFPE parameters click on Run and the Strategy Inputs are displayed as shown below.

   

The The Walk Forward Performance Explorer package consisting of manual, WFPE EXE file, WFPE Excel Add-In, and WFPE Input finder EXE is being offered, for $395. Please note that the WFPE will only read files generated by the PWFO product. The WFPE has a "Key Licence" that only allows it to be installed on three computers.

How To Order
To order online click Order Online. If you would like to talk to me about the product, please call me at (305) 549-5843 M-F 12pm to 5pm CST. All E-mail queries can be sent to support@meyersanalytics.com.

Thank you for your Interest....Dennis Meyers

---