Wall Street is littered with poorly designed in-house systems projects that have been tossed into the digital dust heap.

By Tom Groenfeldt

It's a story repeated over and over again on Wall Street. A derivatives trading group flush with either ambition or cash decides to junk its outmoded trading system and install the system of its dreams.

A committee of traders and in-house systems people is convened, an initial wish list is drawn up and a search of existing systems vendors is assigned. After a period of research, it's determined that none of the existing vendors can meet the desk's particular needs. The only reasonable alternative, it seems, is to develop an in-house system custom-made to meet the company's unique requirements.

A budget is established, people are hired, the clock starts ticking and money flows freely. A year later, the group reports that several components are nearing completion and promises that delivery is just a year away. At least for beta.

Two years later, the system is mired in development hell. Initial designs did not provide support for later developments and had to be done over. Traders objected when the look and feel changed from what they had been using, while risk managers pointed out that the system neglected to provide them useful information fast enough. Even the graphical user interface has been changed several times. Meanwhile, several key software engineers have quit in disgust and gone to vendors or to more technically advanced firms where they can improve their skills on hot new projects. The team left behind is decidedly second-tier in both development and project management skills. To make matters worse, no one on the business side understands software well enough to evaluate progress, much less direct the program.

Eventually, a consultant is called in to look over the mess. He decides it's time to junk all or most of the system and adapt an existing system sold by a vendor. The IT project manager screams about wasting the bank's huge investment, which he insists is just about to pay off. Afraid of getting fired, he soon jumps to another firm, and a new cycle begins.

This time, if the firm is lucky and has learned its lesson, it buys a package to meet 70 percent to 85 percent of its needs, adjusts some internal processes to accommodate the vendor's design, and then builds the added functionality it requires in a joint effort with specialists from the vendor That, anyway, is the picture drawn from interviews with more than two-dozen vendors and independent consultants, who are called in periodically to clean up messes like these. The history of Wall Street's in-house derivatives systems projects, in fact, resembles the repetitive cycles of the movie Groundhog Day.

At any moment on Wall Street, you can bet that several companies are wrestling with grossly underestimated time and budget projections, incompetent systems design, mismanaged resources, unqualified programmers, and constantly rewritten base code-all jealously managed by competing internal political groups that are unable to make, or stick with, decisions.

Design myopia

The critical problem is vision. Effective systems development requires a single vision translated into an architecture. But instead of beginning with a vision, banks begin with some bullet points. And instead of assigning responsibility to a benevolent dictator, they settle for a cross between an American participatory democracy and a Japanese seniority system. To get anything accomplished requires the skills of Machiavelli.

Most internally developed systems, in fact, start with traders. And that, say consultants, is where the problems begin. "Almost every in-house system originates with a bunch of traders getting together and writing a quickie," says Jos Stoop, president of Intuitive Products International Corp. "If you build that out, you will run into something that needs replacement. Typically, the heart of the system is seen as the stuff the traders and analysts come up with. Unless you pull that apart completely and reassemble it in a modular fashion, it is difficult to get away from the structure."

In most cases, traders want systems that meet their immediate needs, and they want delivery yesterday, at the latest. Their self-confidence is rarely tempered by any trace of patience. When IT doesn't deliver, traders are apt to spend a weekend putting together a simple program themselves and loading it Monday, without telling the wet blankets from IT. "In every bank," recalls one consultant, "traders take pride in what they have accomplished. The systems people say it is awful and create their own. Then traders say they can't work with what systems people built."

"It's not that systems people simply don't get along with traders and the business side, it's that they hate each other," explains another former trader who now works for a derivatives systems vendor. "They don't talk, they don't cooperate, they aren't even organized along similar lines."

Missing framework

One thing traders don't understand is systems architecture-the idea that front, middle and back offices have different functions and different requirements. A trader-developed trading system, for example, might have good front-office analytics running in real time but may not be able to perform credit risk management because the middle and back offices use batch processing. Risk management, after all, is nothing without linking current trade inputs with data from legacy back-office systems to provide up-to-date exposure information. Traders are notorious for losing interest in a trade after its input, but if the information can't flow smoothly from front office to back, risk management is impossible.

Fixing these shortcomings inevitably leads to patches, add-ons and workarounds, but never to elegant design. Although banks boast about their global trading books and comprehensive risk management, many of them still transfer their order information by e-mail. "Financial institutions developed their systems in step with their activities in the market," says Simon Moss, head of the Americas trading and risk management practice at IBM. "Many of the solutions were robustly built but were often designed for specific functional coverage. They rarely took into consideration demands for such things as integrated, consistent firm-wide risk management or horizontal client-rather than product-based structures."

Consultants say that one Swiss bank spent $3 million to $4 million trying to build an in-house swaps system running on two IBM 3090 mainframes before it finally relented and bought SunGard's Devon. "The problem arose from starting to build a piece of the system without designing an architecture," says one consultant. "When banks decide to build a system in-house, they start talking to users, but the system is always driven by the needs coming from the front office, which worries about pricing. Only later do designers begin to think about risk management and credit management."

Equally disastrous is asking everyone what they want from a system without having a method to cut the project down to the essentials. "Firms fall into the toy store syndrome," explains a consultant. "Everyone wants something, so they decide to write specs." One Swiss bank did just that, and $35 million later, developed the specs for a great system but nothing else. Toronto Dominion, for example, also reportedly spent $4 million to $5 million by first asking everyone for a wish list. It quickly had a massive wish list, and three years later there was still no software to show for its ambitious plans.

Bulking up

Although traders get their fair share of blame, the blame falls equally on in-house systems managers, who often push big projects because they provide them the opportunity to build up their fiefdoms. "These projects take on a life of their own, and banks throw good money after bad," said one consultant who used to work in a bank. "Software engineers will build a word processor from scratch, just to learn how to do it, rather than just going over to Comp USA and buying one for $99," said another.

This do-it-yourself-at-all-costs mentality is exacerbated by basic inexperience with the way systems components must work together in a larger architecture-a lesson that vendors often learn by painful experience. One Japanese firm, for example, recently completed a high-tech global risk management system whose performance is bogged down by its graphical user interface. "It's very difficult to change anything on the screen," said a consultant. The GUI, chosen because it is multiplatform, turned out to offer more flexibility than the bank needs, since it is running just one type of workstation.

Another common failing is a basic failure to understand the limitations of their own designs. The New York office of a French bank has a derivatives system that was perfectly suited for a boutique business. But now that volumes have grown, a database that crashes somewhere beyond 5,000 deals is proving to be a disaster. When it takes hours at the end of the day to do revaluation, the system becomes completely useless for intraday risk management.

Info mismanagement

These design mistakes are often compounded by poor project management decisions. The very top firms, such as Citibank and JP Morgan, have IT staff capable of undertaking complex development programs, says one London consultant. "But the next tier down, that's not necessarily the case. It is getting to be much more complex than many banks have the resources for. In many cases, the staff at the bank has no project management experience or no experience with the technologies involved."

And when banks do manage to hire a competent systems team, they typically underestimate the complexity of a large development project. "They think that throwing 100 people at a problem should take care of it," says a software expert at a small shop, "but running a large software project requires extraordinary skills that only a few companies such as IBM, Lucent and Martin-Marietta have really mastered."

IT departments taking on development projects to keep abreast of new technology often do it backwards. They start writing data classes and lower-level parts of the system while they are learning the technology. By the time they move to higher-level development, their early building blocks are outdated, so they have to go back and modify them. The result is kludge piled upon kludge. "This type of project doesn't lend itself to teams doing it the first time around," noted a consultant. "It would be more useful to buy something and work from there, but that is not in the IT interest."

The project management problems are compounded by an ugly truth: most top banks can't attract enough top programmers. One highly skilled programmer is more productive than several moderately skilled ones, and there just aren't enough great programmers to meet the demands within the financial industry. "It's hard to find C++ programmers on Wall Street, and even harder to find quants," says a vendor who has a few of both on staff. Software experts who understand the business challenges of derivatives are extremely rare. Perhaps it shouldn't be surprising that once a system is built, the owners often aren't sure what they have.

When they do get good talent, it's hard to keep them focused on long-term development projects. Software engineers at banks don't want to spend years on a single project. It gets boring, they risk abrupt termination if the project isn't accepted, and their marketability decreases as they limit their work to the narrow scope of a single derivatives project. The best engineers want to work on advanced projects with other experts. Consultants say one Canadian bank saw its derivatives development grind to a halt as its IT staff drifted off to other jobs. Threats can come from unexpected directions. At another Canadian bank, a software team was working long hours on development. During those long hours, two key team members became romantically involved and then left to get married. The project died.

Another common project management failing is mismanaging the work assigned to different teams of outside vendors. One bank, for example, decided to bring in two vendors so it could develop both a middle office and a back office simultaneously. Not surprisingly, the solution was very slow. Equally predictably, each vendor blamed the other.

One outside consultant found the bank couldn't even cooperate at the database level but was using intermediary views. "The bank had no control over what these people were doing," explains Intuitive's Jos Stoop. "After more than two years of work, no workable solution had been produced, so they called us in." Stoop found the bank was blaming the wrong vendor and suggested improvements that made an unusable result into a crippled, but working, system. "A lot of it was forcing the vendors to work together because they had both been hiding by blaming each other. You cannot leave it up to vendors to do the right thing for you."

Fear of commitment

Dealers shouldn't be surprised if their systems groups underperform or if morale is low. Many banks don't have the patience to invest in long development programs with no visible returns. In many cases, the higher-ups call out for human sacrifices, although programmers will often suffice. Basically, says the vice president of a large financial technology firm, banks have begun to realize that software development is not their core business. "At some point, even big banks like JP Morgan will realize that they have staffs costing millions to support proprietary software. They have to maintain it and staff a help desk, and that isn't their business." Eventually, even banks with tens of millions invested in a functional home-built derivatives system will find the courage to shut it down.

When a bank decides to cut its development budget and has to decide between a small project producing usable programs and a three-year development project that won't produce a useful program for another 12 months, there's no question of what is going to get axed first. All this is unlikely to inspire great loyalty among software staff who know their careers depend on their ability to remain current with the latest programs and tools.

Among consultants and vendors, the most talked-about recent development failure is at a large British bank. Details vary widely from one account to another, but depending on whose tale you believe, the bank spent somewhere between $10 million and $80 million on a system that "ultimately doesn't do anything," according to one former bank employee. For several years, a British executive in the New York office of this bank drove development, but years of development weren't enough to overcome an extremely convoluted design. "Everything got broken down into metaphysical components," explains the former employee. "They abstracted too far from the nuts and bolts of what they were doing."

In typical bank fashion, the development continued with little or no accountability as long as the program director remained in political favor. Eventually, however, senior management changed its opinion of him. He left, and the development project has died, leaving the bank with a ticket-writing system and some word processing capability in return for its multimillion dollar (or pound) investment. Some consultants say the bank is about to buy Summit Systems. More recent accounts say the bank plans to try again with in-house development. The bank, several consultants agree, has gone through a series of management changes, each of which has changed its development direction.

Giving up

At the end of many exhaustive development projects, it's not even possible to determine whether the project was a success or a flop. Citibank, which spent upward of $30 million on its Global Derivatives System, had to hire outside consultants to tell the bank whether it was a success of failure. The project was over budget and past deadline-not an unusual result in a complicated software scheme-so critics were calling it a disaster. The consultants' verdict? The project was officially designated a success. Give that team a bonus-if they haven't all been fired.

Other times, even when a system works, it is at least partially a failure, says one consultant who worked on a Swiss bank's project. Originally planned for 12­18 months, the system took three-and-a-half years and was multimillions of dollars over budget. "Size alone can be a big problem in a huge system. Time eats up enthusiasm, [and] it's difficult to hold onto the professionals and to maintain the budget."

The ultimate problem, say consultants, is that banks simply try to do more than they can deliver. Banks commonly bite off more than they can chew, says Josie Palazzolo at Sailfish, a risk management software developer. "They are looking for a big bang approach, and in risk management you have to do it in small stages. There is often no right or wrong answer-there is a best answer for the amount you are willing to spend in the time allotted for the problem."

By Tom Groenfeldt

Vendors will eagerly tell you they can provide better software than banks can develop for themselves in-house. They also claim they can deliver it faster, and often even provide documentation-something often overlooked during in-house system development. What vendors are less likely to tell you is that they are more likely to do this because they have learned from their own repeated mistakes and failures. Their lead over mere banks is that, having been through the error cycle several times, they have already learned what not to do.

When banks try to build software, they are systematic and rigid, says Gabriel Bousbib, vice president of the risk management unit at Reuters. They start with a clean slate, plan to build the best system money can buy, hire lots of people and try to meet all the desires presented by all business units. What you need, he explains, are just a few key objectives, and a small team of smart people commanded by a dictator. Then lock them in a room and let them work.

One key thing vendors have learned is the importance of getting the architecture done first. Again, that's experience speaking-most software shops were founded by developers who left banks in frustration and set out on their own with strong beliefs about how a system should be built and little appetite for taking orders from someone who doesn't understand technology but has greater rank in the hierarchy.

Vendors manage to avoid long drawn-out projects is pure self-interest; they'd go broke. Vendors that manage to survive do so by developing an architecture that usually reflects the vision of just one or two people-the firm's founders. Then the company tries to build modular and reusable pieces it can sell many times.

"Design everything and implement a little bit," advises John Fry at Principia Partners LLC, a risk management software house in Jersey City, N.J. "Banks are not in the business of writing software," says a consultant who gets called in when banks or software houses get bogged down. "They should let vendors develop the basics-GUIs, replication and data integrity. Where they get the most bang for the buck is adding an API or using a toolkit approach."

The movement toward open systems strengthens the case to buy rather than build. Individual software companies can develop specific products in-depth on a standard platform. Then the products can be linked by systems integrators or on Windows NT by Microsoft's Object Linking and Embedding (OLE) technology. "Having a common infrastructure will change the dynamics of how banks mix software," says Reuters' Bousbib. "They will buy components and plug them into a common infrastructure." Then bank IT staffs can add proprietary models to achieve customized systems.

"Banks want a system that can handle the entire universe of products or provide a handoff to a specialized system like Sailfish or Algorithmics," says another vendor. Banks shouldn't expect to develop a system for less than vendors spend on their development programs, said one vendor. In fact, since they are starting fresh, it may cost them more. Infinity, which went public last year, calculated its research and development costs at $6.1 million in 1995 and $4.1 million for just the first six months of 1996.

When the margins earned by derivatives groups were several hundred points higher than they are today, it was relatively easy for a bank IT department to make a case for building an expensive system. Now, however, derivatives often generate profits comparable to fixed-income desks-but the supporting systems can cost 10 or 20 times as much. In this new environment, the business case for building expensive in-house derivatives systems becomes difficult indeed.

Despite all the advantages third-party vendors possess, systems consultants caution that third parties can rarely provide a completely successful front-to-back-office solution. "The success of any project involves synthesizing the internal and external components," says IBM's Moss. "If you run everything in-house, the project could become unfocused as internal pressures affect the project. It could also become undisciplined as people ignore the project's original methodology. But if you give too much to the vendor, the client's original requirements may get lost or misinterpreted. The ideal balance involves close control and scrutiny of the project's direction by both users and the IT group."