This example shows techniques for managing, processing, and visualizing large amounts of financial data in MATLAB®. It is part of a series of related examples on machine learning for statistical arbitrage (see Machine Learning Applications).
Financial markets, with electronic exchanges such as NASDAQ executing orders on a timescale of milliseconds, generate vast amounts of data. Data streams can be mined for statistical arbitrage opportunities, but traditional methods for processing and storing dynamic analytic information can be overwhelmed by big data. Fortunately, new computational approaches have emerged, and MATLAB has an array of tools for implementing them.
Main computer memory provides high-speed access but limited capacity, whereas external storage offers low-speed access but potentially unlimited capacity. Computation takes place in memory. The computer recalls data and results from external storage.
This example uses one trading day of NASDAQ exchange data  on one security (INTC) in a sample provided by LOBSTER  and included with Financial Toolbox™ in the zip file
LOBSTER_SampleFile_INTC_2012-06-21_5.zip. Extract the contents of the zip file into your current folder. The expanded files, including two CSV files of data and the text file
LOBSTER_SampleFiles_ReadMe.txt, consume 93.7 MB of memory.
The data describes the intraday evolution of the limit order book (LOB), which is the record of market orders (best price), limit orders (designated price), and resulting buys and sells. The data includes the precise time of these events, with orders tracked from arrival until cancellation or execution. At each moment in the trading day, orders on both the buy and sell side of the LOB exist at various levels away from the midprice between the lowest ask (order to sell) and the highest bid (order to buy).
Level 5 data (five levels away from the midprice on either side) is contained in two CSV files. Extract the trading date from the message file name.
MSGFileName = "INTC_2012-06-21_34200000_57600000_message_5.csv"; % Message file (description of data) LOBFileName = "INTC_2012-06-21_34200000_57600000_orderbook_5.csv"; % Data file [ticker,rem] = strtok(MSGFileName,'_'); date = strtok(rem,'_');
Daily data streams accumulate and need to be stored. A datastore is a repository for collections of data that are too big to fit in memory.
tabularTextDatastore to create datastores for the message and data files. Because the files contain data with different formats, create the datastores separately. Ignore generic column headers (for example,
VarName1) by setting the
'ReadVariableNames' name-value pair argument to
false. Replace the headers with descriptive variable names obtained from
LOBSTER_SampleFiles_ReadMe.txt. Set the
'ReadSize' name-value pair argument to
'file' to allow similarly formatted files to be appended to existing datastores at the end of each trading day.
DSMSG = tabularTextDatastore(MSGFileName,'ReadVariableNames',false,'ReadSize','file'); DSMSG.VariableNames = ["Time","Type","OrderID","Size","Price","Direction"]; DSLOB = tabularTextDatastore(LOBFileName,'ReadVariableNames',false,'ReadSize','file'); DSLOB.VariableNames = ["AskPrice1","AskSize1","BidPrice1","BidSize1",... "AskPrice2","AskSize2","BidPrice2","BidSize2",... "AskPrice3","AskSize3","BidPrice3","BidSize3",... "AskPrice4","AskSize4","BidPrice4","BidSize4",... "AskPrice5","AskSize5","BidPrice5","BidSize5"];
Create a combined datastore by selecting
Time and the level 3 data.
TimeVariable = "Time"; DSMSG.SelectedVariableNames = TimeVariable; LOB3Variables = ["AskPrice1","AskSize1","BidPrice1","BidSize1",... "AskPrice2","AskSize2","BidPrice2","BidSize2",... "AskPrice3","AskSize3","BidPrice3","BidSize3"]; DSLOB.SelectedVariableNames = LOB3Variables; DS = combine(DSMSG,DSLOB);
You can preview the first few rows in the combined datastore without loading data into memory.
DSPreview = preview(DS); LOBPreview = DSPreview(:,1:5)
LOBPreview=8×5 table Time AskPrice1 AskSize1 BidPrice1 BidSize1 _____ _________ ________ _________ ________ 34200 2.752e+05 66 2.751e+05 400 34200 2.752e+05 166 2.751e+05 400 34200 2.752e+05 166 2.751e+05 400 34200 2.752e+05 166 2.751e+05 400 34200 2.752e+05 166 2.751e+05 300 34200 2.752e+05 166 2.751e+05 300 34200 2.752e+05 166 2.751e+05 300 34200 2.752e+05 166 2.751e+05 300
The preview shows asks and bids at the touch, meaning the level 1 data, which is closest to the midprice. Time units are seconds after midnight, price units are dollar amounts times 10,000, and size units are the number of shares (see
Tall arrays work with out-of-memory data backed by a datastore using the MapReduce technique (see Tall Arrays for Out-of-Memory Data). When you use MapReduce, tall arrays remain unevaluated until you execute specific computations that use the data.
Set the execution environment for MapReduce to the local MATLAB session, instead of using Parallel Computing Toolbox™, by calling
mapreducer(0). Then, create a tall array from the datastore
DS by using
tall. Preview the data in the tall array.
mapreducer(0) DT = tall(DS); DTPreview = DT(:,1:5)
DTPreview = Mx5 tall table Time AskPrice1 AskSize1 BidPrice1 BidSize1 _____ _________ ________ _________ ________ 34200 2.752e+05 66 2.751e+05 400 34200 2.752e+05 166 2.751e+05 400 34200 2.752e+05 166 2.751e+05 400 34200 2.752e+05 166 2.751e+05 400 34200 2.752e+05 166 2.751e+05 300 34200 2.752e+05 166 2.751e+05 300 34200 2.752e+05 166 2.751e+05 300 34200 2.752e+05 166 2.751e+05 300 : : : : : : : : : :
Timetables allow you to perform operations specific to time series (see Create Timetables). Because the LOB data consists of concurrent time series, convert
DT to a tall timetable.
DT.Time = seconds(DT.Time); % Cast time as a duration from midnight. DTT = table2timetable(DT); DTTPreview = DTT(:,1:4)
DTTPreview = Mx4 tall timetable Time AskPrice1 AskSize1 BidPrice1 BidSize1 _________ _________ ________ _________ ________ 34200 sec 2.752e+05 66 2.751e+05 400 34200 sec 2.752e+05 166 2.751e+05 400 34200 sec 2.752e+05 166 2.751e+05 400 34200 sec 2.752e+05 166 2.751e+05 400 34200 sec 2.752e+05 166 2.751e+05 300 34200 sec 2.752e+05 166 2.751e+05 300 34200 sec 2.752e+05 166 2.751e+05 300 34200 sec 2.752e+05 166 2.751e+05 300 : : : : : : : : : :
Display all variables in the MATLAB workspace.
Name Size Bytes Class Attributes DS 1x1 8 matlab.io.datastore.CombinedDatastore DSLOB 1x1 8 matlab.io.datastore.TabularTextDatastore DSMSG 1x1 8 matlab.io.datastore.TabularTextDatastore DSPreview 8x13 4516 table DT Mx13 4950 tall DTPreview Mx5 2840 tall DTT Mx12 4746 tall DTTPreview Mx4 2650 tall LOB3Variables 1x12 780 string LOBFileName 1x1 234 string LOBPreview 8x5 2204 table MSGFileName 1x1 230 string TimeVariable 1x1 150 string date 1x1 156 string rem 1x1 222 string ticker 1x1 150 string
Because all the data is in the datastore, the workspace uses little memory.
Tall arrays allow preprocessing, or queuing, of computations before they are evaluated, which improves memory management in the workspace.
S and imbalance index
I are used to model LOB dynamics. To queue their computations, define them, and the time base, in terms of
timeBase = DTT.Time; MidPrice = (DTT.BidPrice1 + DTT.AskPrice1)/2; % LOB level 3 imbalance index: lambda = 0.5; % Hyperparameter weights = exp(-(lambda)*[0 1 2]); VAsk = weights(1)*DTT.AskSize1 + weights(2)*DTT.AskSize2 + weights(3)*DTT.AskSize3; VBid = weights(1)*DTT.BidSize1 + weights(2)*DTT.BidSize2 + weights(3)*DTT.BidSize3; ImbalanceIndex = (VBid-VAsk)./(VBid+VAsk);
The imbalance index is a weighted average of ask and bid volumes on either side of the midprice . The imbalance index is a potential indicator of future price movements. The variable
lambda is a hyperparameter, which is a parameter specified before training rather than estimated by the machine learning algorithm. A hyperparameter can influence the performance of the model. Feature engineering is the process of choosing domain-specific hyperparameters to use in machine learning algorithms. You can tune hyperparameters to optimize a trading strategy.
To bring preprocessed expressions into memory and evaluate them, use the
gather function. This process is called deferred evaluation.
[t,S,I] = gather(timeBase,MidPrice,ImbalanceIndex);
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: Completed in 2.4 sec Evaluation completed in 2.6 sec
A single call to
gather evaluates multiple preprocessed expressions with a single pass through the datastore.
Determine the sample size, which is the number of ticks, or updates, in the data.
numTicks = length(t)
numTicks = 581030
The daily LOB data contains 581,030 ticks.
You can save both unevaluated and evaluated data to external storage for later use.
Prepend the time base with the date, and cast the result as a datetime array. Save the resulting datetime array,
ImbalanceIndex to a MAT-file in a specified location.
dateTimeBase = datetime(date) + timeBase; Today = timetable(dateTimeBase,MidPrice,ImbalanceIndex)
Today = 581,030x2 tall timetable dateTimeBase MidPrice ImbalanceIndex ____________________ __________ ______________ 21-Jun-2012 09:30:00 2.7515e+05 -0.205 21-Jun-2012 09:30:00 2.7515e+05 -0.26006 21-Jun-2012 09:30:00 2.7515e+05 -0.26006 21-Jun-2012 09:30:00 2.7515e+05 -0.086772 21-Jun-2012 09:30:00 2.7515e+05 -0.15581 21-Jun-2012 09:30:00 2.7515e+05 -0.35382 21-Jun-2012 09:30:00 2.7515e+05 -0.19084 21-Jun-2012 09:30:00 2.7515e+05 -0.19084 : : : : : :
location = fullfile(pwd,"ExchangeData",ticker,date); write(location,Today,'FileType','mat')
Writing tall data to folder /tmp/Bdoc20b_1465442_39282/tp0e7983a3/finance-ex97702880/ExchangeData/INTC/2012-06-21 Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: Completed in 3.6 sec Evaluation completed in 4.1 sec
The file is written once, at the end of each trading day. The code saves the data to a file in a date-stamped folder. The series of
ExchangeData subfolders serves as a historical data repository.
Alternatively, you can save workspace variables evaluated with
gather directly to a MAT-file in the current folder.
In preparation for model validation later on, evaluate and add market order prices to the same file.
[MOBid,MOAsk] = gather(DTT.BidPrice1,DTT.AskPrice1);
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: Completed in 2.3 sec Evaluation completed in 2.4 sec
The remainder of this example uses only the unevaluated tall timetable DTT. Clear other variables from the workspace.
clearvars -except DTT whos
Name Size Bytes Class Attributes DTT 581,030x12 4746 tall
To visualize large amounts of data, you must summarize, bin, or sample the data in some way to reduce the number of points plotted on the screen.
One method of visualization is to evaluate only a selected subsample of the data. Create a snapshot of the LOB at a specific time of day (11 AM).
sampleTimeTarget = seconds(11*60*60); % Seconds after midnight sampleTimes = withtol(sampleTimeTarget,seconds(1)); % 1 second tolerance sampleLOB = DTT(sampleTimes,:); numTimes = gather(size(sampleLOB,1))
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: Completed in 2.6 sec Evaluation completed in 2.8 sec
numTimes = 23
There are 23 ticks within one second of 11 AM. For the snapshot, use the tick closest to the midtime.
sampleLOB = sampleLOB(round(numTimes/2),:); sampleTime = sampleLOB.Time; sampleBidPrices = [sampleLOB.BidPrice1,sampleLOB.BidPrice2,sampleLOB.BidPrice3]; sampleBidSizes = [sampleLOB.BidSize1,sampleLOB.BidSize2,sampleLOB.BidSize3]; sampleAskPrices = [sampleLOB.AskPrice1,sampleLOB.AskPrice2,sampleLOB.AskPrice3]; sampleAskSizes = [sampleLOB.AskSize1,sampleLOB.AskSize2,sampleLOB.AskSize3]; [sampleTime,sampleBidPrices,sampleBidSizes,sampleAskPrices,sampleAskSizes] = ... gather(sampleTime,sampleBidPrices,sampleBidSizes,sampleAskPrices,sampleAskSizes);
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 2: Completed in 2.2 sec - Pass 2 of 2: Completed in 2.6 sec Evaluation completed in 5.2 sec
Visualize the limited data sample returned by
gather by using
figure hold on bar((sampleBidPrices/10000),sampleBidSizes,'r') bar((sampleAskPrices/10000),sampleAskSizes,'g') hold off xlabel("Price (Dollars)") ylabel("Number of Shares") legend(["Bid","Ask"],'Location','North') title(strcat("Level 3 Limit Order Book: ",datestr(sampleTime,"HH:MM:SS")))
Some visualization functions work directly with tall arrays and do not require the use of
(see Visualization of Tall Arrays). The functions automatically sample data to decrease pixel density. Visualize the level 3 intraday depth of market, which shows the time evolution of liquidity, by using
plot with the tall timetable
figure hold on plot(DTT.Time,-DTT.BidSize1,'Color',[1.0 0 0],'LineWidth',2) plot(DTT.Time,-DTT.BidSize2,'Color',[0.8 0 0],'LineWidth',2) plot(DTT.Time,-DTT.BidSize3,'Color',[0.6 0 0],'LineWidth',2) plot(DTT.Time,DTT.AskSize1,'Color',[0 1.0 0],'LineWidth',2) plot(DTT.Time,DTT.AskSize2,'Color',[0 0.8 0],'LineWidth',2) plot(DTT.Time,DTT.AskSize3,'Color',[0 0.6 0],'LineWidth',2) hold off xlabel("Time") ylabel("Number of Shares") title("Depth of Market: Intraday Evolution") legend(["Bid1","Bid2","Bid3","Ask1","Ask2","Ask3"],'Location','NorthOutside','Orientation','Horizontal');
To display details, limit the time interval.
xlim(seconds([45000 45060])) ylim([-35000 35000]) title("Depth of Market: One Minute")
This example introduces the basics of working with big data, both in and out of memory. It shows how to set up, combine, and update external datastores, then create tall arrays for preprocessing data without allocating variables in the MATLAB workspace. The
gather function transfers data into the workspace for computation and further analysis. The example shows how to visualize the data through data sampling or by MATLAB plotting functions that work directly with out-of-memory data.
 LOBSTER Limit Order Book Data. Berlin: frischedaten UG (haftungsbeschränkt).
 NASDAQ Historical TotalView-ITCH Data. New York: The Nasdaq, Inc.
 Rubisov, Anton D. "Statistical Arbitrage Using Limit Order Book Imbalance." Master's thesis, University of Toronto, 2015.