So in the first post about FAA, I was requested to make a walk-forward test of FAA. While the results here aren’t good, I’d like to share the general process anyway.

Here’s the additional code I wrote, assuming the first post‘s code is still in your environment (the demo will have the function in the namespace as well):

weightMom <- seq(0, 1, by=.5) weightVol <- c(0, .5, 1) weightCor <- c(0, .5, 1) monthLookback=c(3, 4, 6, 10) permutations <- expand.grid(weightMom, weightVol, weightCor, monthLookback) colnames(permutations) <- c("wMom", "wVol", "wCor", "monthLookback") require(doMC) registerDoMC(detectCores()) t1 <- Sys.time() out <- foreach(i = 1:nrow(permutations), .combine = cbind) %dopar% { FAAreturns(prices=adPrices, monthLookback = permutations$monthLookback[i], weightMom = permutations$wMom[i], weightCor = permutations$wCor[i], weightVol=permutations$wVol[i]) } t2 <- Sys.time() print(t2-t1) out <- out["1998-10::"] #start at 1999 due to NAs with data FAAwalkForward <- function(portfolios, applySubset = apply.quarterly, applyFUN = Return.cumulative) { metrics <- applySubset(portfolios, applyFUN) weights <- list() for(i in 1:nrow(metrics)) { row <- metrics[i,] winners <- row==max(row) weight <- winners/rowSums(winners) #equal weight all top performers weights[[i]] <- weight } weights <- do.call(rbind, weights) returns <- Return.rebalancing(portfolios, weights) return(returns) } WFQtrRets <- FAAwalkForward(portfolios = out, applySubset = apply.quarterly, applyFUN = Return.cumulative) WFYrRets <- FAAwalkForward(portfolios = out, applySubset = apply.yearly, applyFUN = Return.cumulative) WFMoRets <- FAAwalkForward(portfolios = out, applySubset = apply.monthly, applyFUN = Return.cumulative) WF <- cbind(WFQtrRets, WFYrRets, WFMoRets) colnames(WF) <- c("quarterly", "annually", "monthly") WF <- WF["1999::"] original <- FAAreturns(adPrices) original <- original["1999::"] WF <- cbind(WF, original) colnames(WF)[4] <- "original" charts.PerformanceSummary(WF) Return.annualized(WF) maxDrawdown(WF) SharpeRatio.annualized(WF)

So what I did was take about a hundred permutations, and compute them all in parallel using the doMC package (Windows uses a different parallel architecture, but this post explains a more OS-agnostic method). Next, I wrote a small function that would compute a metric for all of the permutations for some period (monthly, quarterly, annually), and equal-weight all of the maximum configurations, which were in some cases, more than one. This would be the strategy’s holdings over the next period, and repeat this iteration to the end. I started in late 1998, as I used a 10-month lookback period for one of the monthly lookback settings, while the data starts in mid 1997.

Here are the results:

> Return.annualized(WF) quarterly annually monthly original Annualized Return 0.1303674 0.1164749 0.1204314 0.1516936 > maxDrawdown(WF) quarterly annually monthly original Worst Drawdown 0.1666639 0.1790334 0.1649651 0.1315625 > SharpeRatio.annualized(WF) quarterly annually monthly original Annualized Sharpe Ratio (Rf=0%) 1.257753 1.025863 1.194066 1.519912

In short, using a walk-forward with FAA seriously harmed the performance.

After seeing these disappointing results, I mulled over as to the why, and here’s my intuition:

A) FAA uses a ranking algorithm, which loses the nuance of slightly changing this weight or that, leading to many identical configurations for a given time period. That some of these or all of these configurations are only the best for that period is a very real possibility.

B) Given that there are so few securities, it’s quite possible that the walk-forward process is simply chasing performance–that is, switching into a configuration right after it had its one moment in the sun. Considering that the original strategy is fairly solid to begin with, it certainly seems to be better to pick a decent configuration and stick with it.

C) One last thought that stuck with me is that the original FAA strategy meets all the qualifications *for* a walk-forward strategy already. It is a monthly-rebalanced algorithm that seeks to maximize an objective function (rank of the weighted sum of ranks of momentum, volatility, and correlation) in a robust fashion–it’s simply that instead of strategy configurations, the inputs were seven mutual funds. To me, in fact, the more I think about it, the more FAA looks like an extremely solid walk-forward framework, simply presented as a strategy in and of itself.

In any case, while the results were far from spectacular, I’m hoping that the code has given others some ideas about how to conduct some generalized returns-based walk-forward testing on their own strategies.

On one last note, if any readers have ideas that they’d like me to investigate, I’m always open to input and new ideas.

Thanks for reading.

Amazing work Ilya! You have advanced the FAA concept considerably. I very much appreciate your efforts (and speed!) in doing a WFA. The fact that the results were poor is a red flag and warrants further thought. I agree that WFA appears to be an embedded part of the FAA concept but the parameters for Look-back, Volatility and Correlation were selected by the authors of FAA after optimizing across the entire data set. That’s a curve fit and obviously there is no guarantee that those parameters will work in markets going forward. Admittedly, it’s going to take me some time to digest your WFA research here. When I get some time (in a few weeks from now due to travel), I will study this in more detail and get back to you. Again, really great work! Thank you!

I wouldn’t say it’s a curve fit so much as the momentum factor is immense. For instance, try a 1.5-.5-.5 weighting. I’ll be publishing a post soon incorporating stepwise correlation into the analysis, but I wouldn’t say it’s a curve fit so much as the fact that the momentum factor weight is rather strong, and I didn’t ever try going above the 1 for the momentum weight, but only lowered it.

Pingback: An Example of Walking Forward in R - Matlab, R project and Python | Big Mike Trading

Pingback: Combining FAA and Stepwise Correlation | QuantStrat TradeR