Holy grail or poisoned chalice?
TL;DR Nobody has cracked it. Period.*
Machine Learning has always fueled the fantasies of Wall Street. After all, AI detects faces, drives cars, beats the World Champions at Chess, Go, and now Starcraft 2. Its application to trading seems natural, doesnât it?
Totally! What if we could throw a treasure trove of data at an AI and let it predict future market prices?
Donât bother with thatâŚ
Prices cannot be predicted, they are mostly random. They are not predictable on average, only on occasions but nobody knows when.
The era of the magic âblack boxâ automating alpha hasnât come yet.
Nobody has cracked automated trading using Machine Learning-based predictions*.
Thatâs right. Sorry to disappoint. Nobody knows whether their bot will behave well next month.
Wait a minute⌠there must be someone Einsteinly smart out there who has figured it out.
How about the Renaissance and Two Sigmasâ of the world? With their Billions of AUM and their armies of PhDs, they must have sealed the deal.
There is no real evidence they have. Those guys have made a habit of keeping things secret, letting outsiders speculate. That is their brand after all, and in the game of FOMO, thatâs a pretty lucrative move.
Donât get me wrong, quant trading strategies exist, ranging from arbitrage to high-frequency trading, they manage north of $1.5 trillion worth of assets according to Morgan Stanley. But they remain relatively simple in the grand scheme of things. They exploit sequences of predictable behaviors and biases.
Thereâs still no evidence that weâve reached the algorithmic trading singularity, with an end-to-end automated black-box consistently generating alpha.
And this is why funds experimenting with more complex strategies spend fortunes on execution and safety measures to protect their back: cross-signal confirmations, alerts, stop-losses, crash-recoveries, roll-backsâŚ
Dad, I have a good feeling about that one.
Youâre telling me that this thing is the ultimate home-run, and nobody has ever cracked it⌠Iâm in!
I know what youâre thinking. What if you were the exception, the missing link, the chosen one?
Right! Someone must fulfill the prophecy, and bring the balance in the force.
Easy, Anakin⌠Before you get all carried away, letâs talk about the most common traps we all fall into.
1âThe answer is 42
Have you ever read Hitchhikerâs Guide to the Galaxy?
So good!
The âAnswer to the Ultimate Question of Life, the Universe, and Everythingâ, calculated by an enormous supercomputer named Deep Thought over a period of 7.5 million years is⌠42.
Hilarious! :â)
Finally, the one and only truth.
Right, and Deep Thought points out that the answer is meaningless because the beings who instructed it never knew what the Question actually was.
The funny part is that quants do it all the time. They set up GPUs and train complex algorithms for days to ask what the price will be tomorrow.
Whatâs wrong with that? You canât feed-in historical prices to predict future ones?
Donât forget, markets are unpredictable and youâre breaking one of the most essential rules of trading:
Past Performances are not indicative of Future Results.
Not convinced yet?
Here is a great reality-check for you with some price predictions of Bitcoin using an LSTM. The web is full of disillusioned traders attempting ML-based price predictions.
Ouch! Party with backtest, hangover with live tradingâŚ
When something looks too good to be true, it probably is.
Now, letâs discuss the second most frequent mistake quants make.
2-The Rube Goldberg Syndrome
Have you heard of the Rube Goldberg Machine?
The machine that uses a complex chain reaction of events to achieve a final, trivial goal?
Correct. Or metaphorically, the tale of an over-engineered useless machine.
Quant: âI think weâre getting really close!â
Come here⌠Can you read me out loud my notes from that year-long experiment I told you about?
Sure. Letâs seeâŚ
âNormalizing data. Check.
Eliminating noise. Done.
Reducing overfitting⌠I almost lost my sleep over it, but I think we are in a pretty good shape.
Using gradient descent to find global minima. Eh eh! thatâs it⌠Iâm a genius.
âŚ
Wait a minute⌠Why is the live agent still not behaving like in backtest?
Either Iâm going mad, or Iâm missing something.â
We dug really deep but never ended up finding the gold mine.
Could you have used more features, run a better data cleaning, or further tweaked hyper-parameters?
Smart suggestions⌠but youâre now catching the Rube Goldberg syndrome!
Eventually, it became clear that we were approaching it from the wrong angle.
Your machine will deliver on what it is designed to do. Youâre not trying to land a rover on the moon, so no need to build a rocket ship.
As often in life, less is more.
Last but not least, letâs unwrap the sneakiest misunderstanding that tricks us all.
3-âBuy low, Sell highâ they say
Aaah, I know that one! Must be the single most famous adage in investing.
Well, this law is misleading at best in algorithmic terms. Lows and highs only become clear in retrospect, and what looks high one day may look low another day.
And we, humans, are good at exercising âcommon senseâ. We employ our judgment in universal ways without thinking expansively or requiring large data sets. Machines are in their relative infancy in this field.
Take a market price for example.
Right, itâs just a number.
The number is a shell. But underneath the shell is a complex derivative of the underlying business, its capital structure, its macro-economic fundamentals, as well as human emotions and buyers/sellersâ intentions.
So the only way for a machine to precisely predict the market price, you would need to feed all those elements that could potentially affect the price. Which is practically impossible to obtain and train an algorithm on.
Gotcha! And if you only input price history, there is a whole lot of information your algo is missing about the underlying factors that affect the price.
Price is the consequence of everything that is happening on the market, not the cause.
Alright, Morpheus, youâve enlightened me. I wonât take the blue pill.
But what are your suggestions to trade using AI?
1-Ask The Right Questions
It becomes important to ask questions differently. Instead of asking âwhat will the price beâ knowing that price movements are random walks, ask questions like:
- What other sources of independent information evidently drive market dynamics?
- What mix of indicators, trends, and past events most accurately generate the right signals?
I see! You need to operate a mental shift and focus on the factors that drive proven wins.
Exactly. The key resides in developing empirical evidence from correlations between data events and the corresponding market responses, then ask the machine learning model to find patterns in the data that precede that trade.
Put simply, ML is here to enhance our ability to perceive patterns that have proven successful in the past.
You have to set the game as a âwinnableâ game by taking advantage of demonstrated correlations.
To detect real correlations you need to be proficient in many independent disciplines. Letâs uncover how you can develop these skills.
2-Develop Transverse Thinking
Have you ever watched a decathlon race?
Run, Throw, JumpâââDecathletes are considered the âWorldâs greatest athletesâ.
From what I read, decathletes donât have to be the Worldâs bests at any particular discipline but consistent across ten of them.
Precisely.
In algorithmic trading, you need to be technically savvy, develop a battle-tested understanding of Machine Learning, decipher the odd nature of time series data, and grasp the fundamentals of markets, finance and trading.
Makes sense. The well-balanced quants with the ability to think transversally will take the prize home.
And until AI reaches higher-intelligence and comprehends every humansâ intricacies, you will have to meet it half-way and understand the machinesâ logic.
Thatâs good news! That combination of skills must be really scarce. I can see how itâs only developed organically.
With a brand new discipline come brand new opportunities.
Unfortunately, developing transverse thinking to spot the right market correlations isnât enough. Thereâs an external factor that is unavoidable in the world of innovation and that you need on your side: timing.
3-Timing is Everything
Did you hear about the recently published Five stages of Autonomy of Self-driving cars?
Fascinating stuff! Level 5 autonomy is where drivers simply plug in their destination and leave the rest up to the vehicle itself.
In the realm of autonomous trading, we can realistically estimate that trade execution has reached Level 3 to 4 while generating profitable and reliable trade signals remains between Level 1 and 2.
Letâs face it, the fully automated trading engine is still far off. We still donât have a clue when the inflection point will be?
At the current point in time, Humans and machines complement each other, excelling at different types of skills. Human Augmentation is whatâs here and now.
âSomeday, yes, we will have flying cars. It does not mean that we need to be making flying parking lots at this moment.ââââScott Belsky
Conclusion
Did you notice anything peculiar about our dialogue?
HmmâŚ
I used a series of metaphors, illustrating complex concepts with simple analogies. Why is that?
Easier to read? :)
Because it talks directly to the human brainâs core strength. Building bridges between the known and the unknown through associations. Learning something new rarely starts from a blank sheet. Neurons have to unwire in order to re-wire to update our brainâs chart. Our very own neural network is a living map of experience-based rules (be it conscious or unconscious).
Same goes for ML-based trading, I assume.
You cannot start from a blank sheet, hoping the neural net will come up with something logical.
Spot on!
You have to start from a map of simple, evidence-based rules that have been carefully crafted and have withstood the test of time.
Thatâs 90% of the work, right there.
And then chug-in massive amounts of data into the ML system and let it evaluate, weigh, and toss out information to come up with Alpha!
*âNobody has cracked AI-based tradingâ: by now youâll have understood that this initial assertion was purposely provocative. The reality is always more contrasted. There are lots of brilliant people exploring this field and their work shouldnât be under-appreciated. Some projects reveal promising results, but quants remain the #1 responsible for profitable signals. AI comes second.
I hope this article contributed to demystifying AI-based trading and re-aligning our short to mid-term expectations with the brutal and unpredictable reality of markets.
If you found it helpful, please clap (up to 50 times) and share to get it in front of smart people like yourself.
If you are developing your own models, seek external expertise to guide you in the right direction, or simply want to nerd-it-out on quant strategies, come say hi! [email protected].
At Supernova, we also offer the following Service Packages to actively managed crypto funds: automated rebalancing, backtesting, alpha creation, and automated execution. Get in touch at [email protected].