How to Create txt output from multiselect file by using readtable?

Hi everyone, i have this following file :

 Format                 IAGA-2002                                    |
 Source of Data         Istituto Nazionale Geofisica e Vulcanologia  |
 Station Name           Castello Tesino                              |
 IAGA Code              CTS                                          |
 Geodetic Latitude      46.047                                       |
 Geodetic Longitude     11.649                                       |
 Elevation              1175                                         |
 Reported               HDZF                                         |
 Sensor Orientation     HDZ                                          |
 Digital Sampling       1 second                                     |
 Data Interval Type     Filtered 1-minute (00:15-01:45)              |
 Data Type              variation                                    |
 #  MAGO 4.13.4                                                      |
 #  SyS 1                                                            |
 #  H, Z and F in nT, D in minutes of arc                            |
 #  F Digital Sampling 5 seconds                                     |
 #  Approx H = 0                                                     |
 #  Approx D = 0                                                     |
 #  Approx Z = 0                                                     |
 #  Vectorial Lemi 18 (text) - 115200                                |
 #  Scalar GEM GSM 90 1 (048183.03 09) - 9600                        |
 #  Scalar Tuning = 47                                               |
DATE       TIME         DOY     CTSH      CTSD      CTSZ      CTSF   |
2021-07-05 00:00:00.000 186     22495.32      6.92  42278.17  47880.43
2021-07-05 00:01:00.000 186     22494.96      6.88  42278.31  47880.39
2021-07-05 00:02:00.000 186     22495.40      6.91  42278.35  47880.62
2021-07-05 00:03:00.000 186     22494.58      6.97  42278.27  47880.19
2021-07-05 00:04:00.000 186     22494.71      6.90  42278.35  47880.30
2021-07-05 00:05:00.000 186     22495.76      6.83  42278.47  47880.94
2021-07-05 00:06:00.000 186     22495.51      6.84  42278.35  47880.79
2021-07-05 00:07:00.000 186     22495.11      6.98  42278.14  47880.40
2021-07-05 00:08:00.000 186     22495.22      6.99  42278.19  47880.45

And by using a single file input (uigetfile) without multiselect command, iam trying to create a script to get data column of each of those header (DATE, TIME, DOY, CTSH, CTSD, CTSD, CTSZ, and CTSF). And this is my successful script to run :

[namafile,direct1]=uigetfile({'*.txt', 'Text-files (*.txt)'},'Load Data Magnet Format IAGA (Definitif / Variatif)', 'Multiselect','on');
         cd(direct1);
         full = fullfile(direct1, namafile);
         fulls = string(full);
         nfiles = size(full,2);
         f = cellstr(fullfile(direct1,namafile));
         file = length(f); 
                
         if file == 1 % Because the file input from Uigetfile is just one.
    
         [~, base, ~] = fileparts(full);
         judul = length(base);
         nb = [upper(base(1)), lower(base(2:judul))];
         defm = contains(nb, 'dmin');
         varm = contains(nb, 'vmin');
         hord = contains(nb, 'dhor');
      
         if judul == 15 || varm == 1 
            
         stasiun = upper(extractBetween(base,1,3));
         stx = upper(extractBetween(base,1,3))+"X";
         sth = upper(extractBetween(base,1,3))+"H";
         std = upper(extractBetween(base,1,3))+"D";  
         stz = upper(extractBetween(base,1,3))+"Z";
         stf = upper(extractBetween(base,1,3))+"F";
         stt = upper(extractBetween(base,4,11));
                 
         opts = detectImportOptions(full,"NumHeaderLines",22);
         opts.SelectedVariableNames = ["DATE","TIME",sth,std,stz,stf];
         T = readtable(full,opts);
         DATE = T{:,1};
         TIME = T{:,2};
         TIME.Format = 'hh:mm:ss';
         DOH = T{:,3};
         DOZ = T{:,5};
         DOF = T{:,6};
         per = strcat(string(DATE(1,1)));
         tahun = str2double(extractBetween(per,1,4));
         bulan = str2double(extractBetween(per,6,7));
         hari = str2double(extractBetween(per,9,10));
         w = datetime(tahun,bulan,hari,0,0,0);
         waktu = string(w);         
         periode = DATE + TIME;
         sti = "Extraction File" + " " + stasiun + " " + "Periode" + " " + "(" +waktu+ ")" ;
         th = "Variatif Data Of H Magnet Component Curve (Minutes Interval)" + " " + stasiun + " " + "Periode" + " " + "(" +waktu+ ")" ;             
         tf = "Variatif Data Of F Magnet Component Curve (Minutes Interval)" + " " + stasiun + " " + "Periode" + " " + "(" +waktu+ ")" ;             
         
         [savefile, direct, ~] = uiputfile('*.txt', 'Save As', sti);
         eval(['cd ''' direct ''';']);
         fout=fopen(savefile,'w');
         for i=1:length(TIME)
         DOFS = sqrt((DOH.^2)+(DOZ.^2));
         fprintf(fout,'%s %s %.2f %.2f %.2f\n', char(DATE(i)), char(TIME(i)), DOH(i), DOFS(i), DOF(i));               
         end
         fclose(fout);
         
         figure('Name','Plot Magnetic Component H Versus Time')     
         plot(periode, DOH, 'Color', 'r', 'LineWidth',1.1);
         title(th, 'fontsize', 14, 'fontweight','bold');
         xlabel('Time Series','fontweight','bold','fontsize',10);
         ylabel('Magnetic Component H IAGA Format (nT)','fontweight','bold','fontsize',10);
         legend('off');
    
               
         else
            
         if judul == 13 || defm == 1 .......... still more depend on the file title

Alright, the above script was already good. However, i just wonder, how to manage a new script by using that readtable concept so that i can get the output file just like the single file script one. Iam trying to understand the document and write down the multiselect script, but it still doesnt success.... Here is my trial and error script :

[namafile,direct1]=uigetfile({'*.txt', 'Text-files (*.txt)'},'Load Data Magnet Format IAGA (Definitif / Variatif)', 'Multiselect','on');
         cd(direct1);
         full = fullfile(direct1, namafile);
         fulls = string(full);
         nfiles = size(full,2);
         f = cellstr(fullfile(direct1,namafile));
         file = length(f); 
         
             if file > 1
           
        S = dir(fullfile(direct1,'*.txt'));
               
        for k = 1 : nfiles
               
          base = S(k).name;
          folder = S(k).folder;
          ff = fullfile(direct1, base);
          F = fullfile(folder,namafile); 
          xx = string(F);         
          [~, nama] = fileparts(xx{k});          
          judul2 = length(nama);
          %Capitalize first character and lower case remaining characters.
          nb2 = [upper(nama(1)), lower(nama(2:judul2))];
          defm = contains(nb2, 'dmin');
          varm = contains(nb2, 'vmin');
          hord = contains(nb2, 'dhor');
                  
        end
                
         if judul2 == 15 || varm == 1 
            
         stasiun = upper(extractBetween(nama,1,3));
         sth = upper(extractBetween(nama,1,3))+"H";
         std = upper(extractBetween(nama,1,3))+"D";  
         stz = upper(extractBetween(nama,1,3))+"Z";
         stf = upper(extractBetween(nama,1,3))+"F";
         stt = upper(extractBetween(nama,4,11));
         opts = detectImportOptions(F{k},"NumHeaderLines",12);
         opts.SelectedVariableNames = ["DATE","TIME",sth,std,stz,stf];
        
         x{k} = readtable(ff, opts);
         T2 = x{k};
         DATE = T2{:,1};
         TIME = T2{:,2};
         TIME.Format = 'hh:mm:ss';
         DOH = T2{:,3};
         DOD = T2{:,4};
         DOZ = T2{:,5};
         DOF = T2{:,6};
         per = strcat(string(DATE(1,1)));
         tahun = str2double(extractBetween(per,1,4));
         bulan = str2double(extractBetween(per,6,7));
         hari = str2double(extractBetween(per,9,10));
         w = datetime(tahun,bulan,hari,0,0,0);
         waktu = string(w);         
         periode = DATE + TIME;
         sti = "Extraction File" + " " + stasiun + " " + "Periode" + " " + "(" +waktu+ ")" ;
         th = "Variatif Data Of H Magnet Component Curve (Minutes Interval)" + " " + stasiun + " " + "Periode" + " " + "(" +waktu+ ")" ;             
         tf = "Variatif Data Of F Magnet Component Curve (Minutes Interval)" + " " + stasiun + " " + "Periode" + " " + "(" +waktu+ ")" ;             
         
         [savefile, direct, ~] = uiputfile('*.txt', 'Save As', sti);
         eval(['cd ''' direct ''';']);
         fout=fopen(savefile,'w');
         for i=1:length(TIME)
         DOFS = sqrt((DOH.^2)+(DOZ.^2));
         fprintf(fout,'%s %s %.2f %.2f %.2f\n', char(DATE(i)), char(TIME(i)), DOH(i), DOFS(i), DOF(i));               
         end
         fclose(fout);

So... Would anyone lend me a hand to fix the error of the above script i made, or maybe just fix the entire concept of my script? and yes, i just wonder how it could be fixed by another function besides of readtable, maybe like readmatrix or readcell? Thank you everyone..... (i already attached my data)

Answer 1

[Walter Roberson]

        for k = 1 : nfiles

You end that loop without having stored any of the results. Your readtable() is after the loop, at a time when ff is referring to the last value it was ever assigned -- the last file.

         cd(direct1);

You do not take advantage of the fact that you have cd'd, so you probably should refrain from using cd().

         full = fullfile(direct1, namafile);
         fulls = string(full);
         nfiles = size(full,2);

Okay, you construct the file names qualified with the directory, and build them as strings, and count them.

         f = cellstr(fullfile(direct1,namafile));

and then you construct the file names qualified with the directory, which will be a cellstr already if more than one was chosen, but will be a character vector if only one is chosen. Then you cellstr() that, which will return the existing cellstr if more than one was chosen but will wrap the single one into a cell if only one is chosen. It is not at all obvious why you are reconstructing the file names that way? If you needed the names in cellstr() form, then why construct string form? If you need string form then you could get the cellstr form just by using cellstr(fulls) .

         file = length(f); 

The number you are going to get in f is going to be the same as nfiles, so not clear why you are doing that.

         
             if file > 1
           
        S = dir(fullfile(direct1,'*.txt'));

You are taking the directory name and finding all .txt files in that directory, including the ones that were individually selected in the uigetfile(), that are represented in fulls and f

       for k = 1 : nfiles
               
          base = S(k).name;
          folder = S(k).folder;

You used fullfile(): you know the folder is going to be what is in direct1 . But in itself using the folder field does not hurt.

          ff = fullfile(direct1, base);

but then you reconstruct a filename using the known directory name together with the file name. What is the point of that?? You just pulled out the folder information but you are not using it. That is confusing, hinting that you are looking up the file in a different directory. If you are going to use the known directory direct1, then use the known directory instead of extracting the folder. When you do the same thing two different ways, it implies to readers that you expect that the results might be different.

You do not use ff after that; you might as well not calculate it. But if you do not calculate ff, then you can go back step by step and show that you do not use S at all and so might as well not do that dir()

          F = fullfile(folder,namafile); 

namafile is the complete list of selected files.

          xx = string(F); 

but string() of that information is already stored in fulls . Why are you recreating it?

[Tyann Hardyn]

Yes Sir, Im just trying to make the difference among some functions i created, i often to run the script by using the functions. But i have no good skills to built the true script which able to process my selected data files to reach the output just like my single file script one...

[Tyann Hardyn]

when i used readmatrix, it would says an error like this :

Selected variables in the import options must have the same data type.

To import heterogenous data, use readtable.

So thats why im using readtable, but i just dont know how to handle cell data from uigetfile multiselect to be readed by readtable function.

Answer 2

[Peter Perkins]

This code

         DATE = T{:,1};
         TIME = T{:,2};
         TIME.Format = 'hh:mm:ss';
         DOH = T{:,3};
         DOZ = T{:,5};
         DOF = T{:,6};
         per = strcat(string(DATE(1,1)));
         tahun = str2double(extractBetween(per,1,4));
         bulan = str2double(extractBetween(per,6,7));
         hari = str2double(extractBetween(per,9,10));
         w = datetime(tahun,bulan,hari,0,0,0);
         waktu = string(w);         
         periode = DATE + TIME;

seems almost completely unnecessary. You appear to be taking a datetime, converting to string, parsing that string to get the y/m/d components, putting those back together to get another (the same, AFAICT) datetime, then converting that to text. None of that should be necessary! Take a look at the methods you can apply to datetime by typing "methods datetime". I would think that

         periode = T.DATE + T.TIME

would do the same thing.

If you really do need to extract things from the table, you may find that things like

         DOH = T.DOY

are more readable than things like

         DOH = T{:,3}

In fact, I think that all your indices on those lines are off by one, which you would have avoided by using

         DOH = T.CTSH

etc.

[Tyann Hardyn]

Alright, Sir. Thank you very much for the fixing explanation, i ll try that

[Tyann Hardyn]

I cannot do dot indexing like DOH = T.DOY because im using

opts.SelectedVariableNames = ["DATE","TIME",sth,std,stz,stf];

as my table variables because the sth, std, stz, and stf are my variables which re get from :

sth = upper(extractBetween(nama,1,3))+"H";
          std = upper(extractBetween(nama,1,3))+"D";  
          stz = upper(extractBetween(nama,1,3))+"Z";
          stf = upper(extractBetween(nama,1,3))+"F";
          stt = upper(extractBetween(nama,4,11));

Please share your other solution, Sir. Thanks

[Walter Roberson]

Your nama is going to be a character vector.

n3 = upper(nama(1:3));
sth = n3 + "H";
std = n3 + "D";  %careful, conflicts with std() function
stz = n3 + "Z";
stf = n3 + "F";
stt = n3 + upper(nama(4:11));
opts.SelectedVariableNames = ["DATE","TIME",sth,std,stz,stf];
DOH = T.(std);
DOZ = T.(stz);
DOF = T.(stf);