By Margaret Elliott

For the top managers at NatWest Capital Markets and Bank of Tokyo/Mitsubishi, it was egg-on-the-face time. In March, Bank of Tokyo/Mitsubishi announced that its New York-based derivatives unit was taking an $83 million after-tax write-off because a computer model overvalued a portfolio of swaps and options on U.S. interest rates. A few weeks later, NatWest Capital Markets announced that it was taking a £50 million hit because of a mispriced portfolio of German and U.K. interest rate options and swaptions run by a single derivatives trader in London. After further investigation, the hole grew to £90 million. And then all at once, rumors about multimillion dollar losses related to derivatives book mispricings at UBS, BZW and elsewhere were flying around the market.

Don't jeer, and try not to point fingers. Given the range of models, assumptions and data used in every bank and every dealer each day around the world, it's likely that most firms are bound to have an "unfortunate profit/loss event" lurking somewhere in their books.

Everyone from risk management consultants to newspapers calls this problem "model risk," an imprecise term that belies the multifaceted nature of the problem or problems. But model risk covers at least three distinct areas: the choice, testing and safekeeping of the mathematics and computer code that form a model; the choice of inputs and calibration of the model; and the management issues associated with these activities.

Each of these presents a huge challenge, and few observers can point to financial institutions that have all these areas under control. "Maybe one or two institutions that have been in the derivatives business from the beginning are up to speed on the model risk issue," says one former head of a swaps group. "But certainly no banks that have gotten in the business recently." "Very few do it well," concurs Michael Haubenstock, a partner at Price Waterhouse.

Ordering the code

It isn't a surprise that the use of computer models should pose a health risk to banks. The derivatives business would be unthinkable without the elaborate models used to price its enormously complicated instruments. It's easy to forget that the whole system is built on the intellectual labor-and assumptions-of a few key people. "The actual analytics in a model come from the quants on the trading desk, the front office," says Zahid Ullah, head of derivatives investment management at Gotham Derivatives Corp. and a former NatWest risk manager. "It could be a new version of an old model, for a similar product, or a new spreadsheet model. They give this to the back office for pricing-with no checks. The traders run the show, and if they're challenged, they simply say it is proprietary."

Companies that use a model developed by a trader to value positions for both the front and back office are asking for a rerun of the Nick Leeson problem-a trader assigned to watch himself. The solution? The back office develops or buys its own models to serve as an independent check on the front office, usually with the aid of a high-priced independent risk management group or outside consulting firm. Although that sounds easy, developing a system to manage and test old and new models involves a number of difficult steps.

Model tracking

The first step is pure common sense: to understand how models affect your business, you must keep careful track of all the models you use. That means keeping records of which models are used, who uses them and how they're used. It also means keeping track of who keeps the code and who is allowed to change the code.

Once that inventory is complete, it's important to control access to those models. "We suggest locking up approved versions of the models to facilitate control," says David Lukach, a partner in the capital markets practice at Coopers & Lybrand. "Some of the problems we see involve derivations of models-someone changes the logic in the model and an inappropriate result occurs. Or a new product is developed and an older model is used to price it with or without authorized modifications."

Models, of course, shouldn't be kept under lock and key. They should evolve and improve with time. That means establishing a procedure for changing the code. "Markets change and models need to reflect these changes," says Jitendra Sharma, director of derivatives and risk management at Arthur Andersen in New York. "Making sure that each model change is verified and implemented across all appropriate models keeps the bank up-to-date."

Vetting challenge

One important step-perhaps the most important-is developing a procedure to test each of the models in use, preferably before they're used.

This is a complicated procedure, but one that pays off in spades. For simpler, more liquid instruments, the model produces a price that can be immediately checked against the market. Indeed many traders don't even bother to use models for many simple options.

But as the instruments become more complicated, longer in term or less liquid, traders may develop pricing models that reflect their own thoughts about how these products will trade. The back office, however, may use different models, either older or, in some cases, one that is more in line with market standards. "For more liquid instruments, there may be a market standard model-say Black-Scholes for options-that is routinely used for pricing," says PW's Haubenstock. "But while the back office uses that to price the book, the traders may use a different model, one that makes adjustments for the shortcomings of Black-Scholes. The two should use the same, approved model, or else there will be an internal control problem and a potential for unexplained profit-and-loss differences between front and back office."

For more complicated instruments, the starting point is the code and the math. Institutions increasingly are turning to the Big Six or to independent consultants for risk management checks that include model audits. And not surprisingly, demand has soared since the NatWest announcement.

Outsiders who come in often try first to rebuild the model from scratch. If that is successful, they then run a series of bench-marking tests that measure the dealer model against their own market models. "We see a broader range of market models than a single bank, so our models may be more up-to-date in terms of quant theory," says Andersen's Sharma. "This can tell us whether the model the bank proposes to use is simply wrong-that is, if it does not reflect or replicate the behavior of the underlying instrument and therefore is inappropriate to the product."

Dealers that assign their internal risk management groups to vet their models often purchase externally-developed software to check on their internally-developed models. But if derivatives software doesn't offer the ability to bench-test using the same inputs, it runs the risk of being just like the trader's proprietary spreadsheet-unverifiable. "It is important that system vendors give customers the option of dynamically inserting new models into those systems," says Mark Garman of Financial Engineering Associates, a top derivatives software house. "This allows institutions the opportunity of using and comparing three different sources of models, namely the vendor models, the customer models and the third-party vendor models." Some vendor models do not offer this flexibility.

"A few years ago we saw mostly external auditors interested in purchasing our models as a check on the front office," says Emmanuel Mond, managing director at Monis Software. "In last few months, we've had more purchases by internal auditors who perform spot-check audits rather than trying to running every trade though the model."

When auditors find disparities, they go back and check the numbers more carefully. "We're not seeing very large differences, which indicates that models are standardizing to some extent," notes Mond. "But you've got to compare oranges with oranges. If you're pricing off a Black-Scholes and they're pricing off of a Heath-Jarrow-Morton, you're bound to have disparities."

In the inevitable push me­pull you that occurs in the process of vetting models, one issue stands out-simplicity. "If you make the model overcomplex, you embed risk that you often cannot easily verify," says Simon Moss, head of the trading and risk management practice for IBM North America. "If I assume constant volatility, for example, that says to me that I am modeling a term structure. For instruments with optionality, the common assumptions should be stochastic in nature."

In designing and vetting models, it's important to remember that models should serve as guides for fast-thinking traders in dynamic markets. No model ever completely replicates the dynamic changes in an instrument. In modeling a basket option on the OEX, for example, IBM's Moss says, "When seeking volatility for the basket, it would be inappropriate to supply the volatility for each element." Andersen's Sharma puts it this way: "In the model-development process there is necessarily a trade-off between replicating the market exactly and over modeling. It's important to find a simple and elegant way of accurately reflecting the market with the minimum number of inputs and iterations." Don't overcomplicate unnecessarily seems to be a rule of thumb.

Input error

Most problems with models do not occur in the code, but in the data, inputs or calibration of the model in use. Do not underestimate the complexity of the problem. For longer-dated illiquid instruments, it may be quite some time before an input problem can be identified. And Value-at-Risk calculations won't pick it up.

Take NatWest's much-publicized £90 million ($138 million) loss. One popular theory is that the bank was using a single volatility for all strike prices in its sterling/Deutsche mark over-the-counter swaptions book. If the bank had used different volatilities for each strike, a so-called volatility smile, the book would have been priced more closely with the market and NatWest wouldn't have incurred the mispricing loss. "Particularly for out-of-the-money deals, the smile effect gives a higher implied volatility," says Leon Borodovsky, Ph.D., executive director of the Global Association of Risk Professionals and a risk manager at Credit Suisse First Boston in New York.

The magnitude of the loss may have been a reflection of the length of time it took the bank to pick up the inappropriate calibration. Observers suggest that the incorrect volatility may have been used on the book since late 1994, or about when the bank entered this particular business. "Though I have no independent knowledge of the NatWest situation, an incorrectly calibrated model will have far less effect when the book of business held by the institution is either completely long or completely short," says Anthony Neuberger, a professor at London Business School. "In my experience, the problems occur when a bank or dealer makes the business decision to make a market in a particular instrument, going both long and short."

The other factor that shines out of the NatWest affair is the seeming lack of consistency in the inputs used by the bank. "If flat volatility was used in the sterling/Deutsche mark book, why wasn't it used in the sterling/dollar book or the Deutsche mark/dollar book? What did those traders know that that the sterling/Deutsche mark traders didn't?" asks Gotham Derivatives' Ullah.

Stress check

The best way to check model calibration is through stress testing. Running simulations with data that reflect different market scenarios should help determine what inputs are appropriate for which book of business.

But like code testing, it is imperative to use more than one model while stress testing. "By using an independent model or a market-standard model, you are essentially benchmarking the results for the same inputs to the models and evaluating the potential differences in valuations," says Dilip Kumar, senior manager at KPMG. "Of course, you expect that the results are quite close. But the interesting part occurs when there are differences. To the extent that there are appreciable differences, you are looking to interrogate the differences and fully understand the sources of the differences so that you can form an opinion of the model being tested. This is what educates you as to the strengths and weaknesses of the models, so that you are aware of the sensitivity of the valuation to each of the inputs. This becomes an important exercise when you are dealing with illiquid transactions and not easily verifiable inputs."

Stress testing often forces certain peculiarities of models into high relief-making them appear like calibration errors. "If a structured derivatives desk is using a lattice pricing model, certain calibration values may imply a relatively low hedging rate. That is because that value corresponds to a point on the lattice itself. But another quite close value that falls between the lattice points may imply a huge hedge," says Dave Penny, vice president of software development at Algorithmics. "If you interrogate the results you'll see that the huge implied hedge occurs in between the lattice points. In all likelihood, it can be ignored." This concept is very important to Tanya Styblo Beder, principal of Capital Market Risk Advisors Inc. "We look for drivers of difference-it's an early warning system that tells you when a model might be misleading," she says.

Where's the data?

It may be common sense, but consultants say few organizations make sure that all their traders work off the same data. "We suggest that the firm set up a centralized data capture mechanism," suggests Algorithmics' Penny. "At the end of the day, for instance, external data come into one place and then are fed into the necessary models to allow for an independent double-check on the data the traders were using."

At CIBC in Toronto, explains Michael Crouhy, vice president for global analytics in the market risk management group, implementing a first-class financial rates database was one of the bank's key decisions in entering risk management. "We use our own database to calibrate pricing models and to feed our VAR system. We spent considerable of effort ensuring the integrity and accuracy of our data. Using poor data can ruin the benefit of sophisticated analytical models-garbage in, garbage out. We also insist on full consistency between our pricing models in the front and middle offices, and for profit-or-loss calculation, as well as in the VAR system. When we develop risk management calculators at the desk level, the risk assessment is consistent with what is produced by the VAR model. This greatly contributes to the credibility and the acceptance by the traders of our VAR system, especially when it comes to the calculation of capital charges. It was easier for us to implement this approach as we started with a blank page two years ago."

Few banks have that luxury and may be using not only a wide variety of legacy systems, but also many data sources and incompatible historical information. It's almost axiomatic that banks bring in data from manysources, and that testing is done using historical data that may not match the detail of data feeds available today. "It is imperative to stay up-to-date with best practice as far as data and quantitative methods are concerned," says Beder. "That means determining which data, which sources and which models are used. We've only been technologically advanced enough to short-circuit keypunch error for a few years; often older data are full of errors or no longer relevant to the new realities of the marketplace."

At Bank of Tokyo/Mitsubishi, at least some of the losses resulted from models and procedures that weren't kept up to date. "Take the swaption market as an early example," says Beder. "When they started, swaptions were too small to have their own volatility, so capital markets volatilities were used instead. Eventually swaption volatilities evolved. If your model was still using capital markets volatility when the market moved on, then it would give you the wrong answer."

Going deeper

Many model risk issues become amplified when dealers extend existing businesses or enter new ones. Dealers that trade or make a market in a product for the first time should make a special effort to reassess models, procedures, data and best practice before they jump in. In fact, any significant change in the way you do business should trigger a "rethink." "In our experience, it is valuable to reverify models on a periodic basis given the dynamic nature of the derivatives business, and advisable to verify model imputs by comparison with independent sources, on a frequent basis (daily, weekly or monthly)," says Coopers & Lybrand partner Mark Casella. But that may just be routine housekeeping.

A change in the direction of the model data should encourage a more in-depth investigation. After all, models that are required to support a market-making operation are several degrees more complex than those required for a simple presence in the market. Understanding how to make that leap requires conceptual as well as technological support.

Here's where external consultants can help. CMRA's Beder outlines the process she puts new models in new businesses through. "First we compare the math engine and results of the client's new model with models we build at CMRA. Once we've determined that the model is robust, we further evaluate it to answer a variety of questions," she explains. These questions include what variables, given a small move, cause a large move in price or risk valuation of the position or portfolio? What variables or exposures are considered to offset each other? By how much? What is this model's acceptance in the marketplace? Does the majority of the market use the same data inputs and modeling assumptions (for example, raw data, curve building, interpolation or extrapolation techniques, bid/ask levels, volatility assumptions, mapping or other simplifications)?

Beder suggests that these questions should be answered on a periodic basis for all models. "For example, a classic problem that arises is the use of at-the-money volatilities to perform pricing on options portfolios that may have moved to in-the-money or out-of-the-money status (or will move to this status under stress-testing assumptions). Answering the questions above often prevents unexpected model losses by revealing the degree to which the use of at-the-money volatilities-or a lack of a term structure to volatilities-matters." Beder says that to the degree that it is significant in the valuations or risk views created by the models, changes can be made. To the degree that the error is insignificant or within a palatable range, the firm may be able to avoid expensive technological or operational changes.

Top view

The establishment of independent risk management groups (which report directly to the board) to evaluate models, rather than doing so through the trading profit centers, has been a widely applauded effort. But internal risk management groups will always trail traders. "The expertise resides on the trading floor," says KPMG's Kumar. "Few firms have equally talented people on the control side. Risk management needs to stand apart from the traders. Models must be validated independently."

The answer, it seems, involves strong checks and balances. "We work with the traders, but we have the right to challenge them as well," says CIBC's Crouhy, who heads this effort at the Canadian bank. Just how much of a right depends on the power invested by the board in this effort. CIBC uses external consultants as well. "Risk management comes from the top of organizations," says Ken Young, vice president of Technological Solutions Co. "There needs to be an incentive for middle management to make a profit and control the losses, not just to make a profit."

Not all banks have been successful in empowering their middle offices, and thus rely more heavily on the external consultants. What this implies, according to one consultant, "is that senior management doesn't know who to listen to within their own organizations."

Intensive model auditing, stress testing and smart risk managers are all necessary-but they aren't enough. All the math geniuses in the world don't help if management either neglects to implement the procedures necessary to produce accurate calculations of risk or ignores those outputs. In all the recent derivatives losses, management can be faulted for a lack of understanding of the problem.

An earlier conceptual step is necessary: understanding the nature of the businesses in which the risk is incurred. Observers of the recent travails at NatWest Markets and Bank of Tokyo/Mitsubishi suggest that while the actual culprit in these losses is indeed a form of model risk, the management at these institutions-and many others besides-may not have understood the full range of risks associated with their businesses. "Every time you make a mathematical assumption, input data from an external feed or estimate the value of an illiquid private placement, you are incurring potential model risk," says CMRA's Beder. "That said, three-quarters of the situations we've seen recently that led to derivatives losses were attributable to model risk." Beder estimates that $1.9 billion in recent publicly declared derivatives losses by financial institutions are model-related.

Beder makes this point to emphasize that not all model risk problems are associated with the longer-dated, highly complex and illiquid instruments that were present in the NatWest and Mitsubishi cases, or the earlier revelations of smaller losses by BZW and Nationsbank on currency trades. Although the risk of mispricing may be greatest on these products, they typically form less than 10 percent of the derivatives book. Problems with data or assumptions could have a much greater impact if they appear in association with a more mainstream form of derivatives business.

Model risk can't be eliminated entirely. It's a problem that mutates as markets change. What's required is a true understanding of the breadth and depth of the problem-it affects every nook and cranny of an institution's derivatives business. Eternal vigilance is necessary as well. Beyond the mechanical procedures of testing models and calibrations and vetting the accuracy of data inputs, banks need to find a happy tension between the honest and intellectually curious risk manager and the profit-seeking trader.

The Cost Of Fudging It As the NatWest and Bank of Tokyo Mitsubishi losses make clear, it's critically important to check and update the input variables going into a model. Before pricing a portfolio, for example, it's usually necessary to recalculate the zero curve, and enter the most current foreign exchange rates, as well as the prices of all the underlying instruments. Sometimes, when all the data are not available for the type of instrument priced, a trader may decide to input data relevant to a similar, but not exactly the same, instrument. Here's what would happen if you used the volatility of the three-month forward rate to price a semiannual resetting cap. Table 1 shows the resulting errors based on a six-month caplet starting six months from today. The notional amount is $1,000,000, the strike rate is 6.6 percent (making the cap slightly in-the-money) and the volatility is assumed as 10 percent. If instead, a volatility of 12 percent is input into the valuation model (a 20 percent error, which is not that uncommon), the error in the estimation of the caplet's price is 21.23 percent. The impact of the input error would be amplified if the option was far out-of-the-money. This is shown in Table 2, where everything is kept the same as before, except the strike rate is now 7.5 percent. Under this scenario a 20 percent error in the input (volatility) causes a 155 percent error in the valuation. Taking into account that we are only considering one caplet in this example and that portfolios usually comprise hundreds or thousands of caps, swaps, bonds and other instruments, one can readily see that the overall error could be substantial. -Frank Fronzo, INSSINC Software

Table 1. Errors in the caplet's premium when the option is slightly in-the-money.

Volatility(%) 10.00 10.10 10.20 10.30 10.40 10.50 10.60 10.70 10.80 10.90 11.00 11.50 12.00

Premium($) 831.27 840.09 848.92 857.74 866.56 875.39 884.21 893.03 901.85 910.68 919.50 963.61 1007.73

Volatility Error 0.00% 1.00% 2.00% 3.00% 4.00% 5.00% 6.00% 7.00% 8.00% 9.00% 10.00% 15.00% 20.00%

Premium Error 0.00% 1.06% 2.12% 3.18% 4.25% 5.31% 6.37% 7.43% 8.49% 9.55% 10.61% 15.92% 21.23%

Table 2. Errors in the caplet's premium when the option is far out-of-the-money.

Volatility(%) 10.00 10.10 10.20 10.30 10.40 10.50 10.60 10.70 10.80 10.90 11.00 11.50 12.00

Premium($) 29.41 31.13 32.91 34.75 36.64 38.59 40.60 42.67 44.80 46.99 49.24 61.35 74.91

Volatility Error 0.00% 1.00% 2.00% 3.00% 4.00% 5.00% 6.00% 7.00% 8.00% 9.00% 10.00% 15.00% 20.00%

Premium Error 0.00% 5.85% 11.90% 18.16% 24.58% 31.21% 38.05% 45.09% 52.33% 59.78% 67.43% 108.60% 154.71%