Leland Teschler's Editorial: When You Can’t Believe the Model

February 17, 2009

Amid all the hand-wringing about financial systems in meltdown mode, the subject of modeling hasn’t gotten a lot of notice. Banks and other financial institutions employed legions of Ph.D. mathematicians and statistics specialists to model the risks those firms were assuming under a variety of scenarios. The point was to avoid taking on obligations that could put the company under.

Judging by the calamity we are now living through, one would have to say those models failed miserably. They did so despite the best efforts of numerous professionals, all highly paid and with a lot of intellectual horsepower, employed specifically to head off such catastrophes.

What went wrong with the modeling? That’s a subject of keen interest to engineers who must model the behavior and risks of their own complicated systems. Insights about problems with the mathematics behind financial systems come from Huybert Groenendaal, whose Ph.D. is in modeling the spread of diseases. Groenendaal is a partner and senior risk analyst with Vose Consulting LLC in Boulder, a firm that works with a wide variety of banks and other companies trying to mitigate risks.

“In risk modeling, you use a lot of statistics because you want to learn from the past,” says Groenendaal. “That’s good if the past is like the future, but in that sense you could be getting a false sense of security.”

That sense of security plays directly into what happened with banks and financial instruments based on mortgages. “It gets back to the use of historical data,” says Groenendaal. “One critical assumption people had to make was that the past could predict the future. I believe in the case of mortgage products, there was too much faith in the idea that past trends would hold.”

Therein lies a lesson. “In our experience, people have excessive confidence in their historical data. That problem isn’t unique to the financial area,” says Groenendaal. “You must be cynical and open to the idea that this time, the world could change. When we work with people on models, we warn them that models are just tools. You have to think about the assumptions you make. Models can help you make better decisions, but you must remain skeptical.”

Did the quantitative analysts who came up with ineffective financial models lose their jobs in the aftermath? Groenendaal just laughs at this idea. “I have a feeling they will do fine. If you are a bank and you fire your whole risk-analysis department, I don’t think that would be viewed positively,” he says.

Interestingly enough, Groenendaal suggests skepticism is also in order for an equally controversial area of modeling: climate change.

“Climate change is similar to financial markets in that you can’t run experiments with it as you might when you are formulating theories in physics. That means your skepticism should go up,” he says.

We might add there is one other similarity he didn’t mention: It is doubtful anyone was ever fired for screwing up a climate model.

— Leland Teschler, Editor

Comments

"Garbage in - garbage out" doesn't tell the whole story

On February 27th, 2009 Mike (not verified) says:

While the results of simulations are only as good as the model itself, the other obvious fact that many people miss is simulations results only apply to the conditions simulated. I wonder if anyone's simulations of the mortgage markets took into account the impact of record high gas prices and their ripple effect on higher costs for food, utilities and other items that people would consider more vital than making their mortgage payment.

Much of what drives the ups and downs of the stock market and other securities is simply emotions. It would be very difficult for these models to take into account emotion impacts on the market. If your model assumes you can sell at any time but in reality nobody wants to buy then you have a huge disconnect.

The formula that killed Wall Street

On March 4th, 2009 lteschler says:

Wired Magazine had a pretty good article on the subject of bad financail modeling:

http://www.wired.com/techbiz/it/magazine/17-03/wp_quant

As for climate modeling, a thousand years is an eye blink in terms of geological time frames. Models based on a thousand years of data have their fair share of assumptions.

Models

On February 27th, 2009 WShaw (not verified) says:

Hold on, check the data first. Banks used 20-25 years of historical data, whereas global weather scientists have used several thousand years of historical data.

May I suggest that one model may be more robust than the other?

Perfect!

On February 27th, 2009 Eduardo (not verified) says:

Congratulations!

Mr. Groenendaal got the core point on this matter.
Too much credit and overestimate models and tools.

Regards,
Eduardo

AS GREENSPAN DISCOVERED - MODELS ARE FALLIBLE

On February 20th, 2009 Oliver K. Manuel (not verified) says:

Leland is right on target!

Computer output depends on computer input.

I remember two astrophysicists at the National AGU meeting in Washington, DC in April of 1976 telling me that:

1. Stellar explosions are always symmetric, completely mixing up material from the various stellar layers.

2. Material ejected from a supernova cannot form a planetary system orbiting a supernova remnant.

When I inquired how they knew that stars alwys explode symmetrically, they replied that they had modeled stellar explosions in computers.

I asked if they considered the spin of the star before it explosed.

They replied, "No, our computer can't handle spin so we threw it out."

The Hubble Space telescope later later found that:

1. Stars routinely explode axially, like SN 1987A, and

2. The first planetary system found outside the solar system was three Earth-like planets orbiting a pulsar - the core of a supernova.

We now know that the astrophysicists themselves were riding on an iron-rich object that orbits near the core of a supernova that exploded ~5 billion years ago and ejected all of the material that now orbits the Sun.

With kind regards,
Oliver K. Manuel
http://myprofile.cos.com/manuelo09

Global Warming

On February 20th, 2009 Dave (not verified) says:

When they want to add a carbon tax amidst financial meltdown, remember this article. Climate models are based on very little historical data

mathematical modeling

On February 19th, 2009 Wesley Prais (not verified) says:

Dear editor:
Good editorial, I offer one comment. Often mathematical models are built with some tolerance for change. But in the case of financial prediction on mortgages I suggest that the models used were too optimistic because they greatly undersestimated the fact that banks were being forced (by the same government thugs who are now "rescuing" the nation) to make loans to people who were clearly too unstable to pay them back. In the past, there was much more realism to the mortgage screening process and that is the past trend that didn't hold.

Extinction is Cheap

On February 19th, 2009 David Mathews (not verified) says:

* “Climate change is similar to financial markets in that you can’t run experiments with it as you might when you are formulating theories in physics. That means your skepticism should go up,” he says.

When you don't know the outcome of your experiment, particularly when the experiment impacts the entire planet and there doesn't happen to be another hospitable planet within easy walking distance, it is wise to not behave in a reckless manner.

Glenn Beck, Rush Limbaugh and the Republicans might all believe that scientific warnings against pollution's impact upon the climate amount to nothing more than wild speculation (incidentally, they have a similar opinion regard evolution), but the history of Earth's climate indicate that the planet has experiences phases in which it was particularly inhospitable to both civilization and human life.

In other words: In the last 10,000 years humankind has been especially blessed with a stable and hospitable climate. If this was not the case, it is extremely unllikely that civilization would have survived.

As the human population increases and as humankind piles up by the millions along the shoreline the more dangerous climate change becomes. A small change in sea level or slight change in precipitation patterns is sufficient to displace millions of humans. On an overpopulated planet such massive displacements of humankind poses a mortal threat to civilization.

Unfortunately, the amount of pollution humankind has already dumped into the biosphere is sufficient to generate large changes in the climate. Under such circumstances, the threat isn't so much the displacement of millions of humans but rather the death of billions of humans.

Nor should anyone dismiss the threat of the Homo sapiens going extinct. Humankind's survival isn't guaranteed by God, Nature, nor the laws of physics. All of our ancestors are already extinct and this would serve as a warning to a wiser animal.

At this time in which the global economy is collapsing I encourage everyone to take the threat of climate change seriously. Otherwise the biosphere will collapse and your grandchildren could very well die in a horrendous manner. Not that you would value the survival of your grandchildren more than you value your jobs, comforts, and investments today.

When you can't believe a model

On February 19th, 2009 Paul Nevins (not verified) says:

Wonderful, simply wonderful.

I always have trouble getting young physics students to understand both the power and the limitations of models. For that matter the power and limitations of scientific method is tough to learn other than by experience as well. Very nice Mr. Teschler