How to place multiple csv files along coloum, side by side, in a single file.

19 views (last 30 days)
hosse
hosse on 18 Dec 2015
Commented: Afshin Goodarzi on 14 Sep 2020
Hi I want to place all csv files, under a folder, into a single sheet as single csv file alone coloum direction, side by side.
For example, first file- col A to C, second file- col D to F, and so on, according to file hierarchy (order) of the input folder. I Have studied some existing tutorial in mathwork but still can not solve my problem. All files have equal col and row dimension. I have attached 2 sample file here out of 4000 files. Note that all csv files have single sheet only.
Thanks in advance for solution.

Accepted Answer

Guillaume
Guillaume on 18 Dec 2015
Edited: Guillaume on 18 Dec 2015
csv is a text format, and as all text formats it is written to file line by line. Therefore to do what you want, you have no choice but to hold in memory all 4000x3 columns of at least the row you are writting. So, depending on how much memory you have available you have two choices.
a) To hold 4000 files made of 3 columns by 47772 rows in memory, you need around 4.3 GB of memory. If you have the memory, the simplest thing is to read all 4000 files into a cell array of matrices, concatenate those matrices into one matrix and write that matrice in one go. Something similar to
%warning code is completely untested, there may be bugs
folder = 'somefolder';
files = dir(fullfile(folder, '*.csv')); %assuming you want the files in dir order
filedata = cell(1, numel(files));
for filecount = 1:numel(files)
filedata{filecount} = csvread(fullfile(folder, files(filecount).name));
end
mergeddata = [filedata{:}];
csvwrite(fullfile(folder, 'somename.csv'), mergeddata);
b) You don't have enough memory to hold all the files at once in memory. You would have to read the first few rows of the 4000 files, merge them and write it to your destination, then read the next few rows, merge them, and append that to the file, and so on.
Option b) is bound to be much slower than a), but even a) is not going to be fast, since matlab will have to parse 4000 files.
  6 Comments
Afshin Goodarzi
Afshin Goodarzi on 14 Sep 2020
hi Guillaume
in my case the first rows in csv files are names, not numbers. when trying to run the "Option c1" code, i get the error below, could you help me please?
Error using dlmread (line 147)
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1, field number 1) ==>
lat,lan,pre,pre2\n
Error in csvread (line 48)
m=dlmread(filename, ',', r, c);
Error in Untitled6 (line 7)
filedata{filecount} = csvread(fullfile(folder, files(filecount).name));

Sign in to comment.

More Answers (1)

Ingrid
Ingrid on 18 Dec 2015
this can be easily achieved with the following code
listing = dir(nameFolder);
N = numel(listing);
data = [];
for ii = 3:N
fid = fopen(listing{ii}));
newData = textscan(fid,'%f%f%f);
data = [data, newData];
fclose(fid);
end
  1 Comment
Guillaume
Guillaume on 18 Dec 2015
I'd qualify that easily. Doing 4000 reallocations of data due to the resizing on each file, to end up with 4.3 GB matrix is going to take a long time.

Sign in to comment.

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!