Clients often ask for strategies that trade on very short time frames. Some are possibly inspired by “I just made $2000 in 5 minutes” stories on trader forums. Others have heard of High Frequency Trading: the higher the frequency, the better must be the trading! The Zorro developers had been pestered for years until they finally implemented tick histories and millisecond time frames. Totally useless features? Or has short term algo trading indeed some quantifiable advantages? An experiment for looking into that matter produced a surprising result.
It is certainly tempting to earn profits within minutes. Additionally, short time frames produce more bars and trades – a great advantage for strategy development. The quality of test and training depends on the amount of data, and timely price data is always in short supply. Still, scalping – opening and closing trades in minutes or seconds – is largely considered nonsense and irrational by algo traders. Four main reasons are given:
Higher costs, less profit, more risk, feed dependency, no working strategies – seemingly good arguments against scalping (HFT is a very different matter). But never trust common wisdom, especially not in trading. That’s why I had not yet added scalping to my list of irrational trade methods. I can confirm reasons number 3 and 4 from my own experiences: Below bar periods of about 10 minutes, backtests with price histories from different brokers began to produce noticeably different results. And I never managed to develop a strategy with a significantly positive walk-forward test on bar periods less than 30 minutes. But this does not mean that such a strategy does not exist. Maybe short time frames just need special trade methods?
So I’ve programmed an experiment for finding out once and for all if scalping is really as bad as it’s rumored to be. Then I can at least give some reasoned advice to the next client who desires a tick-triggered short-term trading strategy.
The first part of the experiment is easily done: a statistic of the impact of trading costs. Higher costs obviously require more profits for compensation. How many trades must you win for overcoming the trading costs at different time frames? Here’s a short script (in C, for Zorro) for answering this question:
This script calculates the minimum win rate to compensate the trade costs for different trade durations. We assumed here a spread of 0.5 pips and a round turn commission of 60 cents per 10,000 contracts – that’s average costs of a Forex trade. PIPCost/PIP in the above script is the conversion factor from a price difference to a win or loss on the account. We’re also assuming no win/loss bias: Trades shall win or lose on average the same amount. This allows us to split the Return of any trade in a win and a loss, determined by WinRate. The win is WinRate * Return and the loss is (1-WinRate) * Return. For breaking even, the win minus the loss must cover the cost. The required win rate for this is
WinRate = Cost/(2*Return) + 0.5
The win rate is averaged over all bars and plotted in a histogram of trade durations from 1 minute up to 1 day. The duration is varied in steps of 1, 5, 30, and 60 minutes. We’re entering a trade for any duration every 101 minutes (Bar += 100 in the script is a hack for running the simulation in steps of 101 minutes, while still maintaining the 1-minute bar period).
The script needs a few seconds to run, then produces this histogram (for EUR/USD and 2015):
You need about 53% win rate for covering the costs of 1-day trades (rightmost bar), but 90% win rate for 1-minute trades! Or alternatively, a 9:1 reward to risk ratio at 50% win rate. This exceeds the best performances of real trading systems by a large amount, and seems to confirm convincingly the first reason why you better take tales by scalping heroes on trader forums with a grain of salt.
But what about reason number two – that short time frames are plagued with ‘noise’ and ‘randomness’? Or is it maybe the other way around and some effect makes short time frames even more predictable? That’s a little harder to test.
‘Noise’ is often identified with the high-frequency components of a signal. Naturally, short time frames produce more high-frequency components than long time frames. They could be detected with a highpass filter, or eliminated with a lowpass filter. Only problem: Price curve noise is not always related to high frequencies. Noise is just the part of the curve that does not carry information about the trading signal. For cycle trading, high frequencies are the signal and low-frequency trend is the noise. So the jaggies and ripples of a short time frame price curve might be just the very inefficiencies that you want to exploit. It depends on the strategy what noise is; there is no ‘general price noise’.
Thus we need a better criteria for determining the tradeability of a price curve. That criteria is randomness. You can not trade a random market, but you can potentially trade anything that deviates from randomness. Randomness can be measured through the information content of the price curve. A good measure of information content is the Shannon Entropy. It is defined this way:
This formula basically measures disorder. A very ordered, predictable signal has low entropy. A random, unpredictable signal has high entropy. In the formula, P(si) is the relative frequency of a certain pattern si in the signal S. The entropy is at maximum when all patterns are evenly distributed and all P(si) have about the same value. If some patterns appear more frequently than other patterns, the entropy goes down. The signal is then less random and more predictable. The Shannon Entropy is measured in bit.
The problem: Zorro has tons of indicators, even the Shannon Gain, but not the Shannon Entropy! So I have no choice but to write a new indicator, which fortunately is my job anyway. This is the source code of the Shannon Entropy of a char string:
A char has 8 bit, so 28 = 256 different chars can appear in a string. The frequency of each char is counted and stored in the Hist array. So this array contains the P(si) of the above entropy formula. They are multiplied with their binary logarithm and summed up; the result is H(S), the Shannon Entropy.
In the above code, a char is a pattern of the signal. So we need to convert our price curve into char patterns. This is done by a second ShannonEntropy function that calls the first one:
PatternSize determines the partitioning of the price curve. A pattern is defined by a number of price changes. Each price is either higher than the previous price, or it is not; this is a binary information and constitutes one bit of the pattern. A pattern can consist of up to 8 bits, equivalent to 256 combinations of price changes. The patterns are stored in a char string. Their entropy is then determined by calling the first ShannonEntropy function with that string (both functions have the same name, but the compiler can distinguish them from their different parameters). Patterns are generated from any price and the subsequent PatternSize prices; then the procedure is repeated with the next price. So the patterns overlap.
Now we only need to produce a histogram of the Shannon Entropy, similar to the win rate in our first script:
The entropy is calculated for all time frames at every 101th bar, determined with the modulo function. (Why 101? In such cases I’m using odd numbers for preventing synchronization effects). I cannot use here the hack with skipping the next 100 bars as in the previous script, as skipping bars would prevent proper shifting of the price series. That’s why this script must really grind through any minute of 3 years, and needs several minutes to complete.
Two code lines should be explained because they are critical for measuring the entropy of daily candles using less-than-a-day bar periods:
StartWeek = 10000;
This starts the week at Monday midnight (1 = Monday, 00 00 = midnight) instead of Sunday 11 pm. This line was missing at first and I wondered why the entropy of daily candles was higher than I expected. Reason: The single Sunday hour at 11 pm counted as a full day and noticeably increased the randomness of daily candles.
TimeFrame = frameSync(Duration);
This synchronizes the time frame to full hours respectively days. If this is missing, the Shannon Entropy of daily candles gets again a too high value since the candles are not in sync with a day anymore. A day has often less than 1440 one-minute bars due to weekends and irregularities in the historical data.
The Shannon Entropy is calculated with a pattern size of 3 price changes, resulting in 8 different patterns. 3 bit is the maximum entropy for 8 patterns. As price changes are not completely random, I expected an entropy value slightly smaller than 3, steadily increasing when time frames are decreasing. However I got this interesting histogram (EUR/USD, 2013-2015, FXCM price data):
The entropy is almost, but not quite 3 bit. This confirms that price patterns are not absolutely random. We can see that the 1440 minutes time frame has the lowest Shannon Entropy at about 2.9 bit. This was expected, as the daily cycle has a strong effect on the price curve, and daily candles are thus more regular than candles of other time frames. For this reason price action or price pattern algorithms often use daily candles. The entropy increases with decreasing time frames, but only down to time frames of about ten minutes. Even lower time frames are actually less random!
This is an unexpected result. The lower the time frame, the less price quotes does it contain, so the impact of chance should be in fact higher. But the opposite is the case. I could produce similar results with other patterns of 4 and 5 bit, and also with other assets. For making sure I continued the experiment with a different, tick-based price history and even shorter time frames of 2, 5, 10, 15, 30, 45, and 60 seconds (Zorro’s “useless” micro time frames now came in handy, after all):
The x axis is now in second units instead of minutes. We see that price randomness continues to drop with the time frame.
There are several possible explanations. Price granularity is higher at low time frames due to the smaller number of ticks. High-volume trades are often split into many small parts (‘iceberg trades‘) and may cause a sequence of similar price quotes in short intervals. All this reduces the price entropy of short time frames. But it does not necessarily increase trade opportunities: A series of identical quotes has zero entropy and is 100% predictable, but can not be traded. Of course, iceberg trades are still an interesting inefficiency that could theoretically be exploited – if it weren’t for the high trading costs. So that’s something to look further into only when you have direct market access and no broker fees.
I have again uploaded the scripts to the 2015 scripts collection. You’ll need Zorro Beta 1.36.4 or above for reproducing the results. For the seconds time frames the tick data plugin is needed.
Shannon Entropy: Lecture
联系客服
