The purpose of walk forward testing is to give confidence that the system that was developed using historical data will be profitable when traded. That is, to give confidence that what worked in the past will continue to work in the future.
"Worked" means is profitable with acceptable risk.
"In the past" means in the in-sample period used to develop the system and select the parameters.
"In the future" means in the out-of-sample period that follows the in-sample period. And eventually in the out-of-sample period that is real trading.
The walk forward process performs several sets of "learn in-sample" then "test out-of-sample," stepping forward in time with each new learn / test.
As you describe, the parameters are chosen using a process of generating many alternative models -- each of the alternatives is different than the others and unique in some way -- different issues, or rules, or parameters. Each is scored using the objective function of your choice -- the higher the score, the better that alternative meets your preferences of trading accuracy, holding period, profitability, and risk.
The walk forward process helps the developer understand how the system changes over time.
We need the system to be stationary over the length of the in-sample period plus out-of-sample period. The in-sample results will almost always be good. When the out-of-sample results are also good, the system is stationary. When the out-of-sample results are not good, that is an indication that the model was over-fit to the data in-sample and has not learned a persistent profitable pattern. That system must be returned to development -- it is not tradable.
There is no way to tell how long the system will remain stationary, except by making some test runs.
As the walk forward process continues, the values of the parameters usually change. Well behaved systems change by small amounts. It is possible that the parameters are nearly the same over several learn / test sequences. If so, this is an indication that the system is stationary over long periods -- it is a good thing. However the system changes, that is an indication of a regime change. Watching the change from one learn / test period to the next suggests that the system will need periodic retuning to keep the model synchronized with the data.
At the end of the walk forward runs, there is a set of trades from each of the out-of-sample periods. That entire set is the best estimate of future performance of the system. Use only the out-of-sample trades. The in-sample trades have no value in estimating future performance.
Questions often arise --
Is walk forward optimizing? yes, definitely.
Is it data mining? Yes, definitely.
Is it over fitting? It is learning through fitting a model to some data. It is possible that the model is overfit. The resulting model could be underfit, well fit, or overfit. The test is whether the out-of-sample results are good. If they are good, the model is well fit. If they are poor, the model is poorly fit -- perhaps overfit -- perhaps underfit.
Why do the parameters change as the walk forward process moves through the data? Because the data changes. The model must change to stay synchronized with the data.
How long should the in-sample period be? There is no way to tell without doing some test runs. No rule of thumb will help. No single answer fits all systems.
How long should the out-of-sample period be? At a minimum, it must be long enough to generate enough trades to give data for analysis and to provide confidence. There is no maximum length. As long as the trades continue to meet your criteria, continue to use the system. When the trade performance deteriorates, reoptimize -- just as if you were making one of the walk forward tests.
The parameters change throughout the test runs. What values should be used? Each out-of-sample period uses the values of the parameters chosen using the previous in-sample period.
Should I average the values a parameter takes on over all of the out-of-sample periods? No. You cannot expect the average to be best for any of the periods, or for the future. As the data changes, the mmodel changes. Use the most recent model and its parameters.
What will happen when I trade this system? Nothing can be known for certain. The best estimate of future performance is the out-of-sample results from the walk forward run.
Best,
Howard
http://www.blueowlpress.com/trading-system-development
Hi Howard.
I was going to ask something similar to this on your website, think you have opened up a place to discuss system development topics.
Just finished reading your "Quantitative Trading Systems" , lots of great info, actually about to re-read the last few chapters, Walk Forward Testing, Statistical Test and Monte Carlo especially, a lot to take in for someone with no formal education in quantitative analysis. Actually got a few more topics I'd like to ask but save it for later after I give those chapters a good read again. Then can't wait to get into "Mean Reversion Systems".
I think I know what tim_testpat is trying to ask.
From my very basic knowledge and lack of great terminology, my understanding of working with price data, and trying to optimize a trading system, you need a in sample series of data and out of sample series of data. The goal is to develop a profitable system in the in sample series that will duplicate itself in the out of sample series.
I myself have used the optimize features a bit, I can see the troubles of easily over fitting the data depending on what variable is being optimized.
But from what I know, the idea is to take the optimized parameters that best fit your objective function and to use this in the system and forward test it on the out of sample data.
Example: In Sample period is 2008-01-01 to 2012-01-01 and Out of Sample is 2012-01-01 to present.
When I get confused is the idea of walk forward testing. In this case your using some sort of variable that can be optimized and your looking for a parameter to best fit your objective function, and producing a series of In Sample and Out Of Sample results to observe if the parameter being optimized has discovered a pattern in the price series that proves to be a profitable model.
Now to me in my head this is over fitting as I understand it but as I said, I'm very novice at this quantitative trading approach.
The reason I say that is because for example in your book, page 292, figure 20.26, you show a table of a walk forward result list. The in sample is in steps of 2 years and the out of sample is the following year. The next step is a year after the beginning point and continues in increments of 2 years of in sample and 1 year of out of sample.
Each in sample optimize run provides different results for your optimization of "Periods" and "AMAAvg".
What I don't get and I believe the original poster is how does one use these changing optimizations or validate the results.
What I've always done prior to learning about walk forward analysis is backtest in sample data which is optimized and built to provide a profitable model, then to backtest the out of sample, the variables don't change between the two.
Using the sample above, the system was optmizing the two previous years and using those optimized parameters for the system the following year. So going live, say this year with such a system, would one optimize 2014-01-01 to 2016-01-01 and use these parameters in the system for 2016?
I know this is probably super complicated with my terminology and wording. If this is too much I can maybe switch it over to your new section on your website but figured maybe you could help myself and the original posters, or maybe even anyone else chime in here. Lots of great minds in this group.
Posted by: Howard B <howardbandy@gmail.com>
| Reply via web post | • | Reply to sender | • | Reply to group | • | Start a New Topic | • | Messages in this topic (5) |
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
EmoticonEmoticon