This article is the last part of a series. In Part 1, I outlined the premise: ML/AI shops can borrow tips and best practices from how algorithmic trading (“algo trading”) shops operate. The rest of the articles explore those ideas in more detail.
Over the last several posts, I’ve shared best practices from the world of algorithmic trading and how to apply them to the world of ML/AI. These should help you to address problems you face today, and shield you from problems you may not have known were coming down the road. Such is the benefit of borrowing from fields that have had more time to hone their craft: they learned lessons the hard way, so you don’t have to.
Still, just following best practices isn’t enough. The dirty secret of ML/AI is that you can take all of the “right” steps in preparation and you’re still not guaranteed to win. You’ll have to measure the distance between what you set out to do and what you’ve actually accomplished. You hope that distance is small, but you won’t know unless you check.
That’s why this last post of the series will cover the toughest practice of all: analyze your performance. It’s time to close that feedback loop on your ML/AI efforts.
Breaking it down
Understanding why you chose to do something is the first step in evaluating your performance. You may recall from the earlier post, Know Your Objective, that the purpose of algo trading is:
“… to make money through buying and selling market instruments, such as shares of stocks. We analyze various data sources in an attempt to develop investment algorithms (to determine what kinds of trades we’d want to place) and execution algorithms (to place those trades in response to market activity). We evaluate the quality of our work, and our benefit to the company, based on how much money those trades generate for us.”
Algo trading shops therefore analyze their performance by asking: “did we make money?” And with a little more digging, the traders can see the roles the different components played in the outcome. A very experienced colleague in the algo trading space said it best:
It’s important to decompose your overall effort into investment (choosing which assets to invest in) and execution (how you trade those assets) so you can see which one needs tuning.
You then get to ask yourself: “Did I lose money because the investment algo was wrong? or because the execution algo was wrong?”
Having determined the reason(s) behind a loss, the traders can modify the responsible component(s) – investment, execution, or both – and try again.
That last point relates to something we covered earlier, the importance of experiments. Algo traders try an idea, compare the outcome to their success criteria, learn from the experience, tune the operation, and then run the experiment again. (This assumes that the experiment is still viable. It’s possible the algo’s poor performance is deemed too risky to move forward to the next round.)
The key is to start small: they first give the algo very narrow scope in which to trade, and widen that scope only as the experiment proves successful. This practice limits the traders' risk, which means they’re always in a position to try another experiment tomorrow.
Also true in the data world
Your company’s data strategy should state your purpose and how you evaluate your performance of your ML/AI efforts. You can explore any unpleasant results by asking: “Did I lose out because the model made too many bad decisions? Or because I gave the model too much leeway to make decisions? Or because a model was a poor fit for this work, and I should have instead employed human decision-making?”
If your ML/AI operation is meant to be a series of experiments … then consider that ML/AI is also an experiment in your company. Not everything will go according to plan, and that’s alright. The best approach is to take the time to sort out what went wrong and adjust accordingly.
This is why it’s so important to invest wisely in ML/AI. Your money, time, effort, and even professional reputation are on the line. An algo trader knows to not put all of their money in a single trade. Similarly, anyone implementing ML/AI in a company should know to not put all of their budget on a single project. If things don’t go well that one time – especially if it’s your company’s first time running data projects – you may not have an opportunity to try again.
Running down the list
You can further decompose your ML/AI efforts according to the practices we’ve explored in this series:
- Know your objective
- Think in terms of experiments
- Develop a solid data infrastructure
- Monitor your data feeds
- Monitor your models
- Keep an eye on your risk
- Develop controls around data
When you think about what went well and what didn’t: how did each piece contribute? And when did a given piece fail, but luck swooped in to save the day? There’s nothing wrong with luck, but there’s a lot wrong with relying on it. At some point, luck runs out.
Since doing all the right things doesn’t guarantee success, you may wonder why you should bother to do any of them. That’s a fair question, and – like so many things in trading – the answer goes right back to managing risk: If you’re going to invest this much time, effort, and money into ML/AI, you can’t rely on just luck. Following these best practices will give you the best chance of winning. This is solid risk mitigation around your ML/AI investment, which reduces the chances of losing money and having to shrink (or even dismantle) your data team later.
Standing on the shoulders of giants
There’s nothing quite like learning from established players to shorten your learning curve. While your company may be a far cry from a trading floor, algo trading’s data practices can inform your ML/AI work and save you a lot of trouble.
Many thanks to Joshua Ulrich and other colleagues for reviewing this post and providing feedback. Any errors that remain are mine.