Hedge funds face new era as AI becomes a master of the universe

It was AI versus Warren Buffett.

The artificial intelligence was unleashed by Winton, the London hedge fund, to test an old principle of the Berkshire Hathaway chairman: that major acquisitions usually hurt the buyers' shareholders. Researchers collected and analysed data on almost 9,000 US deals back to the 1960s.

The result? Winton says Buffett's thesis doesn't hold up – big acquisitions don't inherently destroy value.

"It prevented us from trading on a false signal and potentially losing money," said Daniel Mitchell, who runs a team of data scientists at the $US30 billion hedge fund. Buffett didn't respond to a request for comment sent to an assistant.

Bit by bit, AI is laying a claim to the future of investing after many false dawns going back decades. Giant money managers like Two Sigma and Goldman Sachs Group and smaller players like Schonfeld Strategic Advisors have adopted it as a cornerstone strategy or research tool.

From this foothold, how far will AI go?

Man Group Luke Ellis sees a slow takeover coming. The $US103.5 billion firm in London already devotes about $US13 billion to several hedge funds using machine learning. In 10 years, it will play a role in everything Man does, from executing trades to helping pick securities at the firm's discretionary unit, Ellis, the chief executive officer, said in an interview.

"If computing power and data generation keep growing at the current rate, then machine learning could be involved in 99 per cent of investment management in 25 years," Ellis said. "It will become ubiquitous in our lives. I don't think that machine learning is the answer to everything we do. It just can make us better at a lot of things that we do."

The human toll could be severe: 90,000 jobs in asset management, including fund managers, analysts and back-office staff, out of 300,000 worldwide will go poof by 2025 because of AI, according to estimates by consultancy Opimas from a survey of financial firms.

Quant pioneers like Man Group and Winton have a head start in their AI revamp. The obstacles are daunting for almost everyone else.

There's a paucity of scientists who can create profitable strategies. The wizardry is hard for investors to grasp, keeping some on the sidelines. And the high costs of the technology and data are a burden to firms already suffering fee pressure from the flow of assets to passive funds.

But machine learning's prowess in finding investing opportunities beyond the reach of humans makes the technology too alluring to ignore. Firms now use AI to prep reams of messy social media and smartphone data, forecast company earnings and sales faster than analysts, decipher the sentiment of executives from documents and create entire strategies.

"Machines will be doing more of the grunt work of discovering opportunities," said Vasant Dhar, who 20 years ago founded one of the first machine-learning hedge funds, the $US350 million Adaptive Quant Trading program at SCT Capital Management. "They can generate hypotheses, test them, and then tell humans, 'This is interesting, go dig deeper.' As machines add more value, it changes the nature of work humans do."

AI strategies also have to wrestle with the assault from passive investing as BlackRock and Vanguard Group gobble up assets on their way to potentially managing $US20 trillion. Index and smart-beta funds threaten to arbitrage away AI's edge in picking value or growth stocks. But machine learning is showing it can get ahead of the passive wave and exploit patterns in markets that haven't been discovered, almost becoming a superior version of smart beta.

Investors, fed up with years of lackluster performance by discretionary firms, are buying in. Assets in quant funds, many of which use AI, have surged by 86 per cent to $US940 billion since 2010. Last year, when fundamental hedge funds suffered $US83 billion in outflows, quants took in $US13 billion, according to Hedge Fund Research. The trend continued this year through September.

For all of AI's power with data, its limitations are just as profound. AI lacks imagination, or the human ability to anticipate events - from political to macroeconomic - if such occurrences haven't happened in the same way many times before. While hedge fund manager John Paulson saw the subprime mortgage meltdown coming, AI would have had no clue, because it wouldn't have had enough relevant historical data to make comparisons and form an opinion.

"A machine would have no basis for predicting a crisis since each one is unique," said Dhar, who's also a professor of data science and business at NYU. "Humans are good at reasoning about things like a crisis and can sometimes predict it, but we are often wrong. Look at the predictions about interest rates over the last few years."

Right or wrong, fund managers and their market views will play a major role in the era of AI. Fundamental analysts face a bigger threat.

Firms are sometimes paying almost $US1 million in annual compensation for experienced machine learning specialists who can exploit big data. That leaves less money for analysts who research company fundamentals. They may have to learn to code to save their jobs.

"As active managers are forced to spend more money on engineers as their revenues fall, they are going to be forced to slash spending on human equity analysts to protect margins," said Martin Taylor, who shut down his discretionary hedge fund Nevsky Capital last year in the face of competition from quants. "It's very depressing for humans."

Quant firm Acadian Asset Management, where assets soared 79 per cent to $US93 billion in the past five years, offers a clue to how roles may change in the future.

Managers' intuition about economic trends are the foundation of Acadian's long-short and other strategies. Quants then deploy machine learning to refine and improve the 20 most influential factors, from cash flow to unusual events like fraud, that fuel those economic themes to make better predictions. The factors are then plugged into an automated system that takes positions on about 10,000 different stocks across several months or quarters.

Acadian managers and analysts are polymaths: They all have a sophisticated understanding of statistics, and almost everyone writes code and has market experience, said Ryan Stever, director of quantitative global macro research.

The Boston-based firm is investing in AI and big data to better forecast metrics, such as sales, that are key to a company's performance. If Acadian could wager on sales data before it's publicly released, the firm would gain an edge.

"You could use machine learning to get the metric earlier, faster and more accurately," said Wes Chan, director of stock selection research. "If it works, that's pretty significant."

An even bigger ambition for some firms is mastery of deep learning, a smarter AI that powers Google's search and Tesla Inc.'s self-driving cars. Deep learning machines, which loosely mimic activity in the multiple layers of neurons in our brains, require fewer instructions from humans. They make discoveries without being told what to find.

"You will see neural networks become better predictors and better tools for all kinds of trades," said Juergen Schmidhuber, who helped lay the groundwork for modern AI systems and is a consultant to hedge funds. "Many trades will be executed by self-learning algorithms, with a few high-level guys occasionally injecting human decisions. That's near-term future."

Ultimately, the future of AI will depend on its ability to make money. Today's small group of fully automated AI strategies are off to a middling start. Their performance beats the broader hedge fund industry but not the stock market. Thirteen AI funds gained an average of 10.6 per cent annually in six years through 2016, and rose 8.5 per cent through October, according to an Eurekahedge index.

The same is true for old-school stock pickers, who will always have a job as long as they produce healthy returns for investors.

AI may have toppled one of Buffett's pillars. But with Berkshire returning 12.5 per cent annually from 2011 through 2016, machines have yet to beat the legendary investor.

Artificial Intelligence Gets Real: A Timeline

AI's recent feats – beating a Go champion, navigating driverless cars, making money for investors –were many decades in the making.

1950: Alan Turing developed the Turing Test for recognising machine intelligence

1956: John McCarthy coins "artificial intelligence" at Dartmouth College conference

1957: Invention of Perceptron, an algorithm that could be trained to classify images

1964: Computers understand natural language enough to solve algebra word problems

1968: Stanley Kubrick's 2001: A Space Odyssey features intelligent computer HAL 9000

1979: The Stanford Cart, an autonomous vehicle, navigates across a room full of obstacles

1982: James Simons starts quant investment firm Renaissance Technologies

1988: David Shaw founds D.E. Shaw, an early AI adopter among hedge funds

1990s: AI advances in machine learning, case-based reasoning, data mining, virtual reality

1997: IBM computer Deep Blue beats world chess champion Garry Kasparov

1990s: Web crawlers, other AI-based information programs, become Internet mainstays

1999: Sony AIBO, a robotic pet dog, understands 100 voice commands, learns and matures

2005: Sebastian Thrun's Stanford team wins DARPA's 132-mile driverless car race

2011: IBM Watson, a system capable of answering questions, wins quiz show Jeopardy

2012: Google's self-driving car gets license in Nevada

2014: Man Group starts using machine learning algorithms to manage client money

2016: Alphabet's DeepMind AlphaGo computer program beats Go champion

2017: AlphaGo Zero learns by playing against itself, beats AlphaGo by 100 games to 0

2017: Facebook switches entirely to neural networks for 4.5 billion translations a day

2017: First AI Powered Equity ETF driven by IBM's Watson computer starts trading

2017: Two Sigma, a hedge fund that deploys machine learning, crosses $US50 billion in assets under management

2040s: AI could be involved in 99 per cent of investment management, according to Man Group.

Sources: Bloomberg, Man Group, Superintelligence by Nick Bostrom, Winton