Horizontal concatenating table with matrix/array in loop

Hi All,

I am running a loop and from each iteration in the loop, I am saving a couple of datasets separately. Essentially, these datasets are either a table (contain values and strings) or matrix/array. In case if there was no dataset, I synthetically created a zeros matrix. At the end of the loop, I want a table or matrix (I don't mind) where all values from each iteration of the loop are horizontally concatenated in their respective matrices/tables.

Is there any way to concatenating table with matrix or other way around and get the output at the end of the loop? Thank you in advance.

Below is my poorly written code:

RD = []; RD_AV = []; RD_FV_TL = []; RD_LV_CL = []; RD_LV_TL = []; 
load('LC_duration_start_end_centre_Edge.mat');
P = 'F:\Data folders\Waymo_LC_Data\New_data_29-3-23\LCdata_correct_longitudinal\LCdata_correct_longitudinal';
S = dir(fullfile(P,'*.csv')); 
d = table();
fileNames = dir('*.csv');
for i = 1:numel(S)
    F = fullfile(P,S(i).name);
    d = readtable(F);
    v_id = unique(d.VehID(d.vehtype=="av",:));
    dat = LCstartendDuration([LCstartendDuration.veh_id(:)==v_id],:);
    d_AV = d(d.vehtype=="av",:); d_FV_TL = d(d.vehtype=="tgtFollower",:); 
    d_LV_TL = d(d.vehtype=="tgtLeader",:); d_LV_CL = d(d.vehtype=="OrgLeader",:);
    d_AV = d_AV(1:dat.start_centre,:);
    d_AV.acc = (gradient(d_AV.speed))./0.1;
    d_AV.dec = zeros(height(d_AV),1); d_AV.dec (end) = 1;
    no = height(d_AV); rows = sort([no:-20:1, 1]); d_AV = d_AV(rows,:);
    if isempty (d_FV_TL)
        d_FV_TL = zeros(size(d_AV));
    else
        d_FV_TL = d_FV_TL(1:dat.start_centre,:);
        d_FV_TL.acc = (gradient(d_FV_TL.speed))./0.1;
        d_FV_TL = d_FV_TL(rows,:);
    end
    if isempty (d_LV_TL)
        d_LV_TL = zeros(size(d_AV));
    else
        d_LV_TL = d_LV_TL(1:dat.start_centre,:);
        d_LV_TL.acc = (gradient(d_LV_TL.speed))./0.1;
        d_LV_TL = d_LV_TL(rows,:);
    end
    if isempty (d_LV_CL)
        d_LV_CL = zeros(size(d_AV));
    else
        d_LV_CL = d_LV_CL(1:dat.start_centre,:);
        d_LV_CL.acc = (gradient(d_LV_CL.speed))./0.1;
        d_LV_CL = d_LV_CL(rows,:);
    end
end
RD_AV = [RD_AV; d_AV]; RD_FV_TL = [RD_FV_TL; d_FV_TL]; 
RD_LV_CL = [RD_LV_CL; d_LV_CL]; RD_LV_TL = [RD_LV_TL; d_LV_TL]; 

Answer 1

[Animesh]

Hey @Yasir

You can use horzcat function which concatenates arrays and tables together.

You can do something like this:

for i = 1:numel(S)
    % your code to read data and process it
    
    % concatenate the processed data to the existing matrices/tables
    RD_AV = horzcat(RD_AV, d_AV);
    RD_FV_TL = horzcat(RD_FV_TL, d_FV_TL);
    RD_LV_CL = horzcat(RD_LV_CL, d_LV_CL);
    RD_LV_TL = horzcat(RD_LV_TL, d_LV_TL);
end

% you can now use RD_AV, RD_FV_TL, RD_LV_CL, RD_LV_TL as concatenated matrices/tables

You may refer to the following documentation for more information:

horzcat: https://in.mathworks.com/help/matlab/ref/double.horzcat.html

[Yasir]

Thanks @Animesh, this helped a lot.

Answer 2

[Seth Furman]

Sounds like you might want to use a tall table.

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

% Make some fake data in multiple *.csv files inside a folder named
% "my_files"
mkdir("my_files");
for i = 1:3
    t = array2table(rand(2))
    writetable(t, fullfile("my_files", i + ".csv"));
end

t = 2×2 table
     Var1       Var2  
    _______    _______

    0.32173    0.70811
    0.30252      0.836

t = 2×2 table
     Var1       Var2  
    _______    _______

    0.55296    0.68461
    0.70207    0.43517

t = 2×2 table
     Var1       Var2  
    _______    _______

    0.52096    0.59888
     0.2055     0.3628

dir("my_files");

.      ..     1.csv  2.csv  3.csv  

% Create a tall table from the "my_files" folder
parpool("Threads");

Starting parallel pool (parpool) using the 'Threads' profile ...
Connected to parallel pool with 2 workers.

tallTbl = tall(datastore("my_files"))

tallTbl =

  6×2 tall table

     Var1       Var2  
    _______    _______

    0.32173    0.70811
    0.30252      0.836
    0.55296    0.68461
    0.70207    0.43517
    0.52096    0.59888
     0.2055     0.3628