As the director of a PhD program in data science, I give all my students the same advice regardless of what they major in. – Data science and computing skills are the “new math.” If you don’t have a good analytical grounding in these areas, then you will be at a severe disadvantage.
For those aspiring to join a quantitative hedge fund, the same recommendations apply, but with an additional prerequisite: knowledge about financial markets. I’ve seen some very smart people flounder in finance because they don’t know markets and waste their time doing great science that is of limited value because they’re asking the wrong questions or making dubious assumptions about how their results will be implemented in practice.
Where finance quants go wrong
Financial services is very different from other domains, especially academia, because finance problems often do not have a “ground truth” on which a machine-learning algorithm can be based – identical situations can be associated with very different outcomes. You need lots of data, because then you can make models complex and the algos work well. In finace, complexity either pays off or it kills you – it is essential to find the right balance between simple and complex.
Everyone in this business makes the mistake of over-fitting at one time or another and they don’t realize it. I learned the many ways one can over-fit models to data in finance and how to address this issue.
Risk management is equally important. Often people underestimate the risk they’re taking and get an unpleasant surprise. The future is not like the past, so it is important to craft your “objective function” clearly: What types of risks are you willing to take? Are you designing your program for a certain type of market environment, or do you want it to hold its own in many different types of markets? Often people are not aware of their objective functions until the market shows it to them.
My objective is to design a program that will perform within a range of expectations regardless of how the future unfolds. This is extremely challenging.
Between 1994 and 1997 I worked as a prop trader at Morgan Stanley, where I designed and implemented trading strategies, data mining customer data. Between 2002 and 2009, I ran a program called DB Radix, which was part of Deutsche Bank’s fund-of-funds. This was a machine-learning program, but it was human-curated. In 2009, I launched the Adaptive Quantitative Trading (AQT) program, where the machine adapts its strategy on its own over time. It took me more than a decade of trading AI-based models before I trusted a scientific process for model discovery more than I trusted myself.
Why did I trust the AQT program that followed a process more than I trusted my own intuition? My colleagues and I had done several experiments where we could override the machine if we didn’t like its actions. We invariably did worse.
My hedge fund has eliminated humans from the process of model discovery and selection, and the returns of AQT over almost 10 years aren’t hypothetical, but rather realized performance.
We will see more AI in trading programs given the gold rush in this space. The trouble with a gold rush is hope and greed, and people should be wary about that. When people ask me whether my program “shoots the lights out,” they’re missing the point. It is humans who shoot the lights out or get shot out of existence. They tend to have very high variance in performance. The whole point of a systematic program is a low-variance product. That’s what we aim to deliver. This is what all quant funds should aspire to.
Vasant Dhar is the director of the Ph.D. program in data science at New York University, a professor of information systems at NYU’s Stern School of Business, the founder of quantitative hedge fund SCT Capital Management and co-founder of Deep Blue Analytics, a data-analysis consulting company.
Related articles:
12 top AI professionals to know in hedge fundsThese are a dozen names of AI and machine learning experts who are shaking up the hedge fund space.
How to get a job at finance AI firm Kensho TechnologiesKensho Technologies, the data analytics and machine learning firm backed by Goldman Sachs, only hires certain types of people.
Why the buy-side is winning the battle for the hottest machine learning talentThere's a new hot job in town, and so far hedge funds and other buy-side firms are beating investment banks to the punch.
Photo credit: PeteSherrard/Getty
““