New York Times "A Smarter Computer to Pick Stocks" Published: November 24, 2006

So-called “neural networks” and “genetic algorithms” have become common in higher-level computer science. Neural networks permit computers to create new rules and automatically change underlying assumptions by experimenting with thousands of random sequences and processes. Genetic algorithms encourage software to “evolve” by letting different rules compete, and combining the most successful outcomes.

Wall Street has rushed to mimic the techniques. Because arbitrage opportunities disappear so quickly now, neural networks have emerged that can consider thousands of scenarios at once. It is unlikely, for instance, that Microsoft will begin selling ice-cream or I.B.M. will declare bankruptcy, but a nonlinear system can consider such possibilities, and thousands of others, without overtaxing computers that must be ready to react in milliseconds.

“Most software fails in pattern recognition because there aren’t enough sequential rules in the world to teach a computer to discern between two faces, or to find almost imperceptible relationships between stocks,” said Orhan Karaali, a computer scientist and director at Advanced Investment Partners, a $1.7 billion hedge fund. “But a machine that can generate complicated rules a person would never have thought of, and that can learn from past mistakes is a powerful tool.”

Last year, the funds using Mr. Karaali’s model returned in excess of 20 percent by using nonlinear techniques, according to his company. Whereas older methods of stock analysis rely on certain assumptions — for instance, that market volatility always reverts to the mean — Mr. Karaali’s model calculates probabilities and generates assumptions on the fly, and might predict that during a panic, investors will sell Microsoft but, for seemingly irrational reasons, hold onto I.B.M.

“Only an elite group of people are using these ideas, but a lot of people are thinking about them,” said Stacy Williams, director of quantitative strategies at HSBC Global Markets. HSBC is working with Cambridge University in using models based on how viruses spread to forecast foreign currency markets.

“The downside with these systems is their black box-ness,” Mr. Williams said. “Traders have intuitive senses of how the world works. But with these systems you pour in a bunch of numbers, and something comes out the other end, and it’s not always intuitive or clear why the black box latched onto certain data or relationships.”

Such qualms, however, have not stopped Wall Street from scouring university doctoral programs or listening to people like Mr. Kurzweil.

In the pursuit of previously undetectable patterns, hedge funds are racing to quantify things — like newspaper headlines — that were previously immune from number-crunching.

Both Dow Jones Newswires and Reuters have transformed decades of news archives into numerical data for use in designing and testing algorithmic systems. The companies are beginning to structure news so it can be absorbed by quantitative models within milliseconds of release.

Moreover, companies like Progress Software are working with news agencies to create computer programs that instantly translate news — for example, a headline regarding Microsoft’s earnings — into data. M.I.T. is examining, among other things, evaluating companies by seeing how many positive versus negative words are used in a newspaper article.

Software in development could potentially respond automatically to almost anything; changes in weather forecasts on television news, shifting analyst sentiments or what a particular movie critic said about the new blockbuster.

“Right now, everyone basically has access to the same data,” said John Bates, a Progress Software executive. “To get an edge, we want to give investors the ability to immediately turn news into numbers. We want to automate what before required human analysis.”

But as these new techniques proliferate, some worry that promotion is outpacing reality. These techniques may be better for marketing than stock picking.

“Investment firms fall over themselves advertising their latest, most esoteric systems,” said Mr. Lo of M.I.T., who was asked by a $20 billion pension fund to design a neural network. He declined after discovering the investors had no real idea how such networks work.

“There are some pretty substantial misconceptions about what these things can and cannot do,” he said. “As with any black box, if you don’t know why it works, you won’t realize when it’s stopped working. Even a broken watch is right twice a day.”