F# – A trading strategy backtester #2

In the previous post of this series we’ve seen how to download market data from Alpha Vantage. Today I’m gonna show a way to put this data into charts. In doing so, we will see some of the nicest features of F# and how to leverage on them to write powerful succinct  generic code.

The next post will dive into technical strategy backtesting.

Charting in F#

There are several options available to the F# developer to create charts based on numerical data. We can simply rely on the Operating System (OS) graphics primitives, but there are libraries and frameworks publicly available that offer higher level abstractions over those primitives. Often they adhere to the Open Source Software (OSS) principles. OSS, for obvious reasons, is not always well documented or supported, but it offers undoubted advantages to the developer (beside the fact that it is free!):

  • We can start using the pre-built binaries directly as with any library/framework and quickly implement our first-versions. This is important because often (if not always) being able to create live prototypes and not only written drawings and descriptions greatly streamlines the design phase of any software project.
  • Having the source available allows us to “learn by example” much quicker that we would do using just the reference documents. An OSS graphics software based on the GDI or OpenGL primitives, for instance, is a live documentation on how to effectively use those primitives. It is often more productive, by a learning point of view, to start looking at an OSS implementation and from there spot-jump to the reference documentation and not viceversa.
  • Very often the licensing scheme for OSS software allows the user to grab, fork, embed, basically “do what you want” with the source code. So, when the OSS software is not usable as-is in our project, it still constitutes a golden basis where to start-from-and-tweak/personalize, giving the opportunity to significantly shorten our project’s time-to-market/production.

For our necessities here I am going to use XPlot. XPlot is an F# OSS graphics library that encapsulates popular and battle-proof graphics frameworks (Google.Charts and Plotly) and makes them easily usable from an F# codebase. XPlot is part of the excellent FsLab project, whose main purpose is to ease data manipulation in F#. Another component of the FsLab project, Deedle, will be used in the strategy tester that we’ll see in the next post. The Plotly flavor of XPlot will be used here.

Understanding XPlot.Plotly

After reading the web documentation, another thing I usually find useful to understand a library is to look at how it appears in the Visual Studio Object Browser (VSOB), which might seem baffling considering that VSOB is an Object Oriented tool and we are talking of an F# library. But we must remember that

  • F# is a .NET (firstly functional) language, and .NET is inherently object oriented. In other words, F# allows us to write functional source code, and it then translates it to  an object oriented format compatible with CLI and executable by a CLR capable agent (someone argues that F# is not really a functional language because of this, but in my opinion this is a silly argument because, at the end of the day, everything we write must be reduced to a bunch of numbers a processor can execute with its finite instruction set). Therefore, F# modules become static .NET classes and F# functions are translated to .NET static methods inside the module (static class) they belong to. Hence, VSOB shows them all.
  • F# is not purely functional, it offers full OOP weaponry. It is perfectly normal to create class types in an F# component, and even encouraged under well defined guidelines (“Good F# code makes use of object programming, not object orientation”). All classes in a F# compiled binary appear as we would expect in the VS Object Browser.

In this specific case, looking at XPlot.Plotly in the VSOB after installing it trough NuGet, turns out to be very useful and helps understanding how to use it.

VSOB1

We see that there are many “nested classes” Graph.*, this tells us that Graph is much probably a module defined into the XPlot.Plotly namespace and containing several class types (we can always confirm our hypothesis looking at the source in github). Central class is Trace, and we can see that it is a base class (its base is object) and it constitutes a base for many other classes (that derive from it)

VSOB2

A quick look into the plotly documentation tells us that those derived classes correspond to plotly chart traces. And in fact, if we look at the right panel in VSOB we can see that they define properties corresponding to the attributes of the original plotly attributes

vsob3.png

Putting this knowledge together to the XPlot documentation we conclude that to show a chart we need at least to:

  • Construct one or more instances of some of the Trace derived classes
  • Put them all in a PlotlyChart instance using the Chart.Plot overloaded static method. Optionally we can ask Chart.Plot to use a non standard Layout (instance of the Layout class in the Layout module)
  • Finally, simply pass this PlotlyChart instance to the Chart.Show static method

MarketData charts

From the previous post, we have seen that our domain types for market data are


type PriceBarPart = | Open | High | Low | Close
type OHLCV = { O:float; H:float; L:float; C:float; V:int }
type InstrumentMarketDataElement = DateTime * OHLCV
type InstrumentMarketData = InstrumentMarketDataElement seq
type IndicatorMarketDataElement = DateTime * float
type IndicatorMarketData = IndicatorMarketDataElement seq

type IndicatorDefinitions =
    | SMA of period:int * price:PriceBarPart
with
    member this.Id =
        match this with
        | SMA (period, price) -> sprintf "SMA%d%A" period price

type Instrument = {
    Symbol : string
    MarketData : InstrumentMarketData
}

type Indicator = {
    Instrument : string
    Definition : IndicatorDefinitions
    MarketData : IndicatorMarketData
}
with
    member this.Id = sprintf "%s(%s)" this.Definition.Id this.Instrument

type MarketData =
    | Instrument of Instrument
    | Indicator of Indicator

Financial charts use to display instrument OHLC timeframed data as bars or candlesticks. Depending on the case, though, it may be useful to only display a line trace with one of the components (typically the close price in the period). Indicators have only one value per time period, so they are always line traces. To convert our domain types we write this function

    let private marketToTrace withCandlestick marketData =
        match marketData with
        | Instrument {Symbol = symbol; MarketData = series} ->
            if withCandlestick then
                Candlestick(
                    name = symbol
                    ,x = (series |> Seq.map (fun (dt, _) -> dt))
                    ,``open`` = (series |> Seq.map (fun (_, ohlcv) -> ohlcv.O))
                    ,high = (series |> Seq.map (fun (_, ohlcv) -> ohlcv.H))
                    ,low = (series |> Seq.map (fun (_, ohlcv) -> ohlcv.L))
                    ,close = (series |> Seq.map (fun (_, ohlcv) -> ohlcv.C))
                ) :> Trace
            else
                Scatter(
                    name = symbol
                    ,x = (series |> Seq.map (fun (dt, _) -> dt))
                    ,y = (series |> Seq.map (fun (_, ohlcv) -> ohlcv.C))
                ) :> Trace
        | Indicator {Definition = def; MarketData = series} ->
            Scatter(
                name = def.Id
                ,x = (series |> Seq.map (fun (dt, v) -> dt))
                ,y = (series |> Seq.map (fun (dt, v) -> v))
            ) :> Trace

Instruments are converted to Candlestick or Scatter (line) traces, x axis labels are always the date-time of the sample, y axis values are the value(s) required by the specific trace.

This is not all though. We want a more powerful API to be exposed at user level (this is why the function is private). This function (marketToTrace) is the building block we need to get a trace from an instrument or indicator, but we want to put more traces together in the same chart, usually an instrument and one or more overlaid indicators.

let private marketsToTraces marketsData =
    marketsData |> Seq.map (marketToTrace (marketsData |> Seq.length = 1))

This function takes a collection of MarketData and gives back a collection of Traces. It also chooses whether to display an instrument as candlestick or line, depending on the fact that the instrument is the only data to be displayed or there are also indicators to be overlaid.
But, again, we can do more with this API, we would like to give to the user a function capable of receiving whatever we can wish to display in a chart, and be able to discern what to do. In other words, a generic function.

Generic show function

We want to create a generic function that accepts

  • A collection of collections of market data, showing an array of charts. For example, if we pass it something like [[MSFT; SMA20(MSFT)]; [AAPL; SMA50(AAPL)]] we will get a page with two charts, one for Microsoft and one for Apple
  • A backtest report, showing a chart with the operations carried out during a backtest. This will be clarified in the next post
  • A tuple containing a backtest report and a collection of market data. In this case we will get an unique chart with the market traces and the backtest operations overlaid as markers

Here an example of how we want to use it

let gets symbol = getStock symbol DAY
let geti symbol = getIndi symbol DAY

let msft = gets "MSFT"
let aapl = gets "AAPL" 

let pickLast n m =
    let last n seq' = seq' |> Seq.skip ((Seq.length seq' - n))
    match m with
    | Instrument ins -> Instrument { ins with MarketData = last n ins.MarketData }
    | Indicator ind -> Indicator { ind with MarketData = last n ind.MarketData }

let sma8Msft = geti "MSFT" (SMA (8, Close))
let sma20Msft = geti "MSFT" (SMA (20, Close))
let sma50Msft = geti "MSFT" (SMA (50, Close))

let sma100Aapl = geti "AAPL" (SMA (100, Close))
let sma200Aapl = geti "AAPL" (SMA (200, Close))

show [[pickLast 50 msft]; [pickLast 100 aapl]]
show [[msft; sma8Msft; sma20Msft; sma50Msft]; [aapl; sma100Aapl; sma200Aapl]]

And this is what we want as result

XPlotChart2

XPlotChart1We want this function to be “statically generic”, or in other words, we want all the work of deciding how to behave in relation to the various types of inputs to be done by the compiler, not by our code.

The following block gives some clarification of what I mean with this.

Generic programming is a quite wide topic and way out of the target of this post. In essence, though, it is one of the ways programming language creators have come out with in the effort to give developers tools to improve the process of developing software.
I want to emphasize that “one of the ways”. At the end of the days, in fact, every and each of the efforts spent in programming languages creation, in the whole history of computer programming, has always had one principal objective, that I would synthesize as “writing less more powerful and possibly beautiful code“. It is always difficult to condense something big in few words so I will expand a little

  • less: even today, microprocessors have a finite and small set (few dozens) of basic instructions they can execute. It would be non-economic to write any significant software if we had to write it using those basic instructions. Programming languages solve this impasse because they allow to write few humanly meaningful idiomatic words and translate them into millions or even billions of machine instructions that complete the task. Of course, the less we have to write, the more attractive and affordable the process becomes
  • powerful: less is necessary, but it must also be able to articulate whatever we want to achieve. We want to do complex things so the language constructs must give us the tools to do whatever we want in as few instructions as possible
  • beautiful: once power and shortness are achieved, we clearly opt for something we also like. Expressiveness and intuitiveness are very important

Even the absolute first symbolic language ever created (assembly) had an huge impact in programming. Who would prefer to write “1000 1001 1101 1000” or “89D8h” instead of “mov bx,ax”?
And even back then, before the first high level compiler or interpreter were ever written, generic programming as a concept was already invented, indeed embedded in the hardware, as a couple of machine instructions baptized CALL and RET in assembly. The seed of abstraction in programming was there since the beginning, and generic programming is a (much more powerful) expression of that very same concept.

Coming back from this prehistoric (in computer programming terms) rewind, F# offers several tools to generalize code. Some of them are inherited from the .NET environment, others are brand new and baked in the language.

Among the inherited are .NET generics, interfaces, polymorphism and reflection. F# goes a little further and gives us features like Type Inference, Statically Resolved Type Parameters and Generic Type Constraints.

An appropriate use of these features (.NET inherited and F# enhanced) gives us the ability to create very powerful APIs with relatively smaller development and testing effort, as we can leverage on the compiler features.

We can write as in a dynamic language (where most is done at runtime so the language is dry and concise but you pay errors with runtime exceptions) and get the benefits of a static one (where the compiler resolves more but also expects us to give much more information in our source code)

We want to write our generic show function as in

let inline show o = ...

A unique function with a unique parameter that we can call passing actual things of different types. The user just needs to write show whatever. If whatever is supported, the program will compile and there will be no runtime type resolution (no runtime exceptions caused by wrong type use); if it is not supported, the user will get a nice alert at compilation time (even earlier directly in the code editor is using an intellisense IDE); no need to try, no need to test.

Note that we are talking about a single function, not an overloaded method. Even if (as we’ll see) the final implementation relies on overloading, this is not exposed to the user, but hidden in the implementation. the user doesn’t write ObjectOrClass.Show whatever, but only show whatever. Functions in F# cannot be overloaded, yet we can make a function that behaves “polymorphically”.

We could use reflection. Box our argument into an object and then pattern match on the type, but this means writing explicit code in the function implementation. This code then needs to be tested and maintained. Also, it will be executed every time we call show slowing down our application.

The execution time burden would be relieved if we use interfaces, but this doesn’t improve the amount of code written, including the testing and maintenance burdens.

Turns out that F# allows us to do better than this. We define the following helper type and implement our show function as a surprisingly short line of code

    type Shower =
        static member ($) (_:Shower, m:seq) = m |> Seq.map marketsToTraces |> Seq.map Chart.Plot |> Chart.ShowAll
        static member ($) (_:Shower, r:BacktestReport) = reportToTrace r
        static member ($) (_:Shower, (r:BacktestReport, m:seq)) = seq { yield! marketsToTraces m; yield reportToTrace r } |> Chart.Plot |> Chart.Show

    let inline show o = Unchecked.defaultof $ o

Let’s see how this works:

  • When the compiler encounters the definition of our show function, it infers a generic type for the o argument and also it doesn’t generate any code as the function is marked inline. It defers code generation to the use site (the point where show is used by the user code).
  • When a use site is reached, the compiler will recall show‘s definition. And it finds that the argument is used in an operator ($). The compiler then will search for an operator $ globally available in the current scope, but not only: it will also look in the actual types of the arguments involved in the $ operation. Notice that the compiler does that only with operators, not with methods. And overloaded method would lead the compiler to chose the first overload and infer o to be of the type in that first overload, failing to generalize.
  • For how we defined the helper type Shower, the compiler will find an overloaded implementation of the $ operator; if the actual argument passed in the call site matches one of the overloads of this operator, the loop closes and a call to the chosen operator overload will be generated at the call site (inline)

That’s it, we didn’t create any interfaces, we didn’t write a bit of reflection code. The F# compiler did all the work for us. We got both the benefits of an early safety check on our code and a fast exception-free runtime execution.

A word on typeclasses

There is abundant debate in the community around the typeclass construct pioneered by Haskell and how other functional languages are good in mimicking it.

Haskell and F# are both functional languages, but they are based on different premises and have significantly different objectives.

In particular, F# was created primarily as a .NET language, while Haskell main principle is purity in its respect to the exquisite mathematical abstraction axioms of category theory.

The choices made by the F# creators may be not completely adherent to the purity Haskell is the beacon of. However, the trade offs that have conducted to those choices look entirely justified (to the writer at least) once we consider the massive advantage F# gives in applying functional programming principles to an echosystem  (.NET) that offers countless resources, even more nowadays with the ongoing effort to spread it to a multiplatform universe (.NET Core)

Conclusion

We are now able to download market data from a free online datasource and show them in graphics charts. One thing we could want to add is the capability to embed the charts in a custom web page instead than in the default web page provided by XPlot.

Plotly already offers nice built-in interactivity to work with the generated chart and data. Once displayed in the browser, our charts are equipped with a toolbar that we can use to operate on the chart in different ways

XPlotChart3

 

XPlotChart4.png

However, if we want we can use the XPlot.Plotly Html APIs

VSOB5.png

These functions return HTML or javascript markup that render the PlotlyChart instance they are called upon. We can then embed it in our custom page and serve it as we like.

The code in this post can be seen in github.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s