December 1, 2002

THE RISKS IN MODELS
History Tells Us A Lot, But Not Necessarily What's Ahead

By Sam Savage

Uncertain markets tend to cause knee-jerk urges to rebalance one's portfolio. But before doing so, it is important to understand how the risk environment has really changed. This cannot be done without some sort of probabilistic risk modeling technique, such as Monte Carlo simulation. Then, if the shape of the efficient frontier has indeed changed, rebalancing usually makes sense.

Of course, uncertain markets highlight the over-reliance of our risk models on historical data. Like classical music, these models are built on rigid and elaborate rules. Although mathematically aesthetic, they cannot always be expected to keep up with the changing rhythms of the market. Mike Thompson of RiskMetrics correctly refers to these history-based models as the first step. "You still need people to sit there and come up with scenarios for events that haven't happened, could happen, and could become a shock event," he notes.

An example of this on a grand and successful scale was Shell Oil's scenario analysis of the late 1970s, which foresaw and planned for the collapse of the Soviet Union. But dreaming up scenarios is analogous to musical improvisation. And just as training in classical music is no guarantee that you can sit down without sheet music and belt it out with a jazz band, those trained in classical statistics may find it difficult to stray from the straight and narrow path of historical data. Today's technology does make it easy to create a fusion of classical statistics and scenario planning, based on historical volatilities and future potential 9/11's. I predict that this sort of modeling will be used by investors to create "Jazzier Efficient Frontiers."

But let's be clear about what a good risk model is. If a model tells you there is a 5% chance of some bad thing happening, and you can live with this risk, then the bad thing had better not have a 10% chance of happening, or you will be subject to undisclosed risk. Nor should the bad thing have a 1% chance of happening, because then you would be misadvised to invest too conservatively. If over the long run, a bad thing that was supposed to happen only 5% of the time actually occurs one time in 20, then your risk model is batting 1.000. If you want a system that is never wrong, you will need a clairvoyant model. They are much better than probabilistic models, with the one shortcoming that they don't exist.

Sam Savage, Ph.D., is a consulting professor at Stanford University and president of AnalyCorp, a business-analysis software firm in Palo Alto, Calif.