Incrementally Read Data Too Large to Fit into Memory from MLDATX File to Workspace

This example shows how to load simulation data for analysis when the amount of stored data is too large to fit into memory. MLDATX files can store data such as Signal Analyzer sessions, logged Simulink® Test™ assessment data, or Simulation Data Inspector sessions. You can also configure a model to log simulation data directly to an MLDATX using the Log data to file configuration parameter. Such data can sometimes be too large to load into memory for post-processing and analysis. Instead, load the data incrementally using an MLDATXSignalDatastore object.

The example uses the MLDATX file TwoRuns, which contains data from a Simulation Data Inspector session of two runs. In the first run, data from a Ramp block is logged using an Outport block. In the second run, a Sine Wave block is added to the model and logged using a Record block. For more information about saving Simulation Data Inspector sessions, see Save and Share Simulation Data Inspector Data and Views.

Create an mldatx.io.File object that references the TwoRuns MLDATX file.

myFileObj = mldatxfile("TwoRuns.mldatx");
listSignals(myFileObj)

ans=3×6 table
      Name       RunIndex         Domain            BlockPath        BlockSubPath    PortIndex
    _________    ________    ________________    ________________    ____________    _________

    "rampSig"       1        "Outports"          "myModel/Out1"           ""             1    
    "rampSig"       2        "Outports"          "myModel/Out1"           ""             1    
    "sineSig"       2        "myModel/Record"    "myModel/Record"         ""             1    

Create Datastore for Signal

A datastore is an interface to data stored on a disk or at a remote location. Using a datastore allows you to read and process the data incrementally to fit into memory. Create a datastore for interacting with the rampSig signal logged in the first run in the TwoRuns MLDATX file. To do so, use the getAsDatastore function. getAsDatastore returns a Simulink.SimulationData.Signal object whose Values property contains the MLDATXSignalDatastore object. You can examine a subset of data from the datastore using preview.

run1Sig = getAsDatastore(myFileObj,RunIndex=1);
run1Dstore = run1Sig.Values;
preview(run1Dstore)

ans=8×1 timetable
         Time          rampSig  
    ______________    __________

    0 sec                      0
    3.1554e-30 sec    3.1554e-30
    0.2 sec                  0.2
    0.4 sec                  0.4
    0.6 sec                  0.6
    0.8 sec                  0.8
    1 sec                      1
    1.2 sec                  1.2

Set the ReadSize property of the datastore such that the read function loads five data points at a time. Read the first chunk of data from the rampSig signal.

run1Dstore.ReadSize = 5;
read(run1Dstore)

ans=5×1 timetable
         Time          rampSig  
    ______________    __________

    0 sec                      0
    3.1554e-30 sec    3.1554e-30
    0.2 sec                  0.2
    0.4 sec                  0.4
    0.6 sec                  0.6

To see the percentage of the data read from the datastore after reading the first chunk, use progress.

progress(run1Dstore)

ans = 
0.0962

To return to the initial read position, use reset. The hasdata function checks whether unread data remains in the datastore. To incrementally read and process the data in chunks of 5 samples, use the hasdata function as the condition for a while loop. For this example, for each chunk of data, find the maximum data value. Because rampSig is monotonically increasing, the maximum value increases with each chunk of data that is read.

reset(run1Dstore);
maxVal = [];
n = 1;
while hasdata(run1Dstore)
    tt = read(run1Dstore);
    dataChunk = tt.rampSig;
    maxVal(n) = max(dataChunk);
    n = n+1;
end
maxVal

maxVal = 1×11

    0.6000    1.6000    2.6000    3.6000    4.6000    5.6000    6.6000    7.6000    8.6000    9.6000   10.0000

Create Array of Datastores for Multiple Signals

When you choose a filtering option that specifies a set of signals, getAsDatastore returns an array of Simulink.SimulationData.Signal objects. Create an array of datastores for the signals in the second run saved in the MLDATX file TwoRuns. The MLDATXSignalDatastore objects are stored in the Values property of each of the Signal objects. Set the datastore for rampSig to read 10 samples at a time and the datastore for sineSig to read five samples at a time.

run2Sigs = getAsDatastore(myFileObj,RunIndex=2);
run2RampDstore = run2Sigs(2).Values;
run2RampDstore.ReadSize = 10;
run2SineDstore = run2Sigs(1).Values;
run2SineDstore.ReadSize = 5;

Read the first three chunks of data from the sineSig signal. Use the reset function first to ensure that data is read from the beginning.

reset(run2SineDstore)
begSineSigData = [];
for i = 1:3
    tt = read(run2SineDstore);
    begSineSigData = [begSineSigData;tt];
end
begSineSigData

begSineSigData=15×1 timetable
         Time          rampSig  
    ______________    __________

    0 sec                      0
    3.1554e-30 sec    3.1554e-30
    0.2 sec                  0.2
    0.4 sec                  0.4
    0.6 sec                  0.6
    0.8 sec                  0.8
    1 sec                      1
    1.2 sec                  1.2
    1.4 sec                  1.4
    1.6 sec                  1.6
    1.8 sec                  1.8
    2 sec                      2
    2.2 sec                  2.2
    2.4 sec                  2.4
    2.6 sec                  2.6

When you are sure that your signal can fit into memory, use readall to read all the data in the datastore.

allSineSig = readall(run2SineDstore);