The evolution of technical indicator system in ftap

In my ftap algorithm trading system, indicator system is the most revised module among all of the other modules. This may be out of your surprising, but indeed I myself didn't expect that too. The other modules, such as the back testing system, the robot system, work quite stable and don't need very much revision. But indicator system is different. After more indicators are added, there are more needs and requirement and the system needs to change over time to get better abstraction and encapsulation.

The ultimate goals of the indicator system

The most important requirement of the indicator system is the performance, especially in back testing. Recalculating an indicator on each new bar is definitely not acceptable.

Performance is one of the core criteria in the whole ftap system. Indicators performance is more important because usually an indicator requires a lot of CPU time.

Let it calculate all data in a single shot

Ideally in back testing an indicator should calculate only once on all historical data because the data is already available. For example, if current bar under testing is in 2011, the value of a moving average indicator in 2015 is already calculated because we have the data until 2015. In brief, before the testing starts, the indicator already contains all values up to now.

The old version indicator system accesses the bar database directly for historical data. It takes all bar data and calculates the values. That works for both back testing and live trading.

The problems in the old indicator system

Now let's see the problems in the old version indicator system

  • The indicator calculates on all bar data. Most my bar data are from 2005 or even 2003, but I usually test from 2010 to now. Calculating on extra 5 years data is not a good idea.
  • Depends on bar database directly. For performance purpose, the indicator system bypasses the data accessor wrapper and access the bar database directly.
  • The indicators user, usually it's the robot, needs to actively update the indicator. If the user forgets to do so (I did it several times), the indicator refuses to work.

Evolution: only calculate on the period we need

It's easy to make the indicator system only calculate on the period that we need. The back testing system knows the period to test on, so it only needs to tell the indicators about the period. The result is, instead of calculating from bar 0 to bar count, now the indicators accept a range parameter, which contain the necessary bar begin and end indexes.

Evolution: stay away from bar database

The bar database system is a very core module in ftap. It wraps the access to the bar data files on disk. However, it's not intended to be used everywhere because it's one of the lowest level module.

The robot system uses a virtual data accessor, which allows bar data is required. With the data accessor, the bar data can be in the database, or in the memory, or even created by the accessor according to certain algorithm. The data accessor uses an indexing scheme similar as the one in MetaTrader, which 0 is current un-closed bar, 1 is the previous bar, barCount - 1 is the oldest bar, etc.

To strip the indicator system off bar database, just use data accessor instead. One problem to do so is that in back testing the indicators need to access “future” data to calculate. To do so, data accessor allows negative index, which -1 is tomorrow bar, -2 is the day after tomorrow bar, etc.

Evolution: data accessor emits data change event

It's very painful that the indicators user needs to update the indicators actively. To change that, data accessor emits event whenever the underlying data changed (such as new bar is added). Then the indicators subscribe to the event and calculate on any data change event. This works perfectly on both back testing and live trade. In back testing, the bar data never changes, thus only one event is emitted so the indicator is guaranteed to only calculate once. In live trade, whenever a new bar is constructed, the event is emit so the indicator can recalculate base on the new bar.

Another evolution: uniform value getter of all indicators

All indicators have to expose value accessing function to the user. However, each indicator has its own meaning of the value and has different value count. For example, both EMA and RSI have only one value, but the meaning is quite different. Value of EMA is usually for trend, while value of RSI is for over bought/sold. And MACD has 3 values, which is different with EMA and RSI.

The problem of the different value accessing is that I can write one robot to test all similar indicators. For example, if I want to test over bought/sold, I can't have one robot for over b/s, but I have to write a robot for RSI, another for CCI, which is not productive.

To solve the problem, I add a value getter function to let the indicators return a uniform GCallback object. So both RSI and CCI can override the function and return a GCallback object which contains access to their own function. Then I can make uniform robot to utilize the GCallback object, instead of access the indicators' own function!

Final words

I bet you won't fully understand what I'm talking about in this blog because the description is too simple and the whole ftap system is far complicated which can't be explained in several blogs. Any way, it doesn't matter. If you get any inspiration from this blog, I will be very happy. :-)


Enter your comment. Wiki syntax is allowed: