Removing the line number from a numerical matrix

I was using the below code a few months ago to separate a file into smaller matrices, while removing the line number on each row. Now that I have gone back to use it again, It has stopped removing the line number. I'm not sure what could have occurred in the intermediate time to cause the code to stop preforming as it had been. Any ideas?

myFile = uigetdir('C:\Users\c13459232\Documents\MATLAB'); % Generate command window to choose a folder
if ~isdir(myFile) % if the directory is not a valid path
errorMessage = sprintf('Error: the following file does not exist: \n%s', myFile); % print this error message
uiwait(warndlg(errorMessage)); % block the execution of program and wait to resume
return;
end
f2d = fopen(fullfile(myFile,'temp_01.asc'),'w'); % open a file to have data written into (fullfile builds full file name from parts)
f1d = fopen(fullfile(myFile,'TEST_A.asc'),'r'); % open the file to me looked at ('r' means read file)
k = 0;
while ~feof(f1d) % while  find end of file, the TEST_A file
    str = fgetl(f1d); %fgetl reads TEST_A line by line and this is identified by str
    [row, ~, ~, idx] = sscanf(str, '%d'); % Read formatted data from string ignoring the first element of each row
    if sscanf(str, '%d') ==1 % if the data (read line by line) from a string is equal to one
        k = k+1; % k should increase by one with every block
        fclose(f2d); % close the file that is being written to
        fnm = fullfile(myFile,sprintf('temp_%02d.asc',k)); %build a full file from parts that are being formatted into strings
        f2d = fopen(fnm,'w'); %open the parts to be written
    end
    fprintf(f2d,'%s\n',str); % writes the blocks to the file 
end
fclose(f1d); %close the original TEST_A file
fclose(f2d); %close the new file
message = sprintf( 'Task Complete' );
uiwait(msgbox(message));

[Stephen23]

@Aaron Smith : Please upload the file 'TEST_A.asc' (or something that has exactly the same properties as it) by clicking the paperclip button.

[Aaron Smith]

Hey Stephen. If I remember correctly, you helped me with this back when i was first trying to write the code. The file is too large to attach here

[dpb]

Don't need the whole file, only a record or two...

[Aaron Smith]

it's a collection of what should be 50 1025x1024 matrices. The aim is to remove the line number and make the matrices 1024x1024

[dpb]

Surely much betters ways to do that than to process line-by-line in text mode...see ADDENDUM to ANSWER.

[dpb]

I went and looked at the previous thread back in February where you had code supplied to build these arrays.

Why don't you fix that code to correct the size issue there instead of propagating it downstream?

[Stephen23]

"Surely much betters ways to do that than to process line-by-line in text mode"

You are obviously much better than I am if you can read 50x1024x1025 values into memory without any "out of memory" error. The data, which was eventually delivered to me via dropbox, is large. My code, which was developed after attempting many different solutions, including reading the data into numeric arrays, is clearly that of an incompetent code monkey. Yep, I certainly look forward to seeing your "much betters ways" (you use the plural, so please provide multiple much better ways of copying a large amount of text data from one text file to some others).

https://www.mathworks.com/matlabcentral/answers/323811-creating-multiple-equally-sized-matrices-from-a-single-numerical-cell

https://www.mathworks.com/matlabcentral/answers/324347-data-is-not-saving-to-the-workspace

https://www.mathworks.com/matlabcentral/answers/325155-large-format-spec-statement-in-textscan

Answer 1

[Stephen23]

The answer to your question is that you need to use the code that I gave you, and not remove that parts that removes the line number, in particular this line:

fprintf(f2d,'%s\n',str(idx+1:end)); % this is the correct line.

This was clearly shown in my comment to your original request for this feature:

https://www.mathworks.com/matlabcentral/answers/324347-data-is-not-saving-to-the-workspace#comment_430364

In that comment I also explained how it works, and showed a small working example. Simply clicking the idx variable would have shown you that it was unused in your code (the editor shows where variables are used).

Answer 2

[dpb]

The obvious answer would be this isn't the same code as you ran previously. There's nothing in this code that would do anything except echo what was read to the new file...

str = fgetl(f1d);
...[some code that doesn't do anything to _str_ elided for clarity]...
fprintf(f2d,'%s\n',str); % writes the block

That's the guts of the code you have here--it reads a line and then writes it to a second file verbatim. Whatever processing you did to remove something from the record before rewriting it is missing.

ADDENDUM

Sans the user interface stuff, the following should work quite a lot quicker...

data=dlmread(fullfile(myFile,inFileName),';');  % read the input file
data=data(:,2:end);                             % eliminate first column
nCol=size(data,2);                              % column size
data=reshape(data.',nCol,nCol,[]);  % reshape to 3D to write planes
nBlk=size(data,3);                  % find out how many there are
for k=1:nBlk                        % and loop over them
  fnm=fullfile(myFile,sprintf('temp_%02d.asc',k));
  dlmwrite(fnm,data(:,:,k),';');    % and write it out
end

ADDENDUM 2:

"... dlmread results in an out of memory error"

OK, can't read whole file at once...the initial code of just a fgetl loop is simple; don't see what the issue must be that's so baffling, but an alternative that might be a little quicker still processing only a single block at a time...

N=50;         % block size, records
k=0;          % block counter
fid=fopen(fullfile(myFile,inFileName),'r');  % input file
while ~feof(fid)
  k=k+1;              % increment block counter
  fnm=fullfile(myFile,sprintf('temp_%02d.asc',k)); % output file name
  data=textscan(fid,'',N,'delimiter',';');   % read block
  dlmwrite(fnm,cell2mat(data(2:end)),';');   % and write it out less first cell content (line numbers)
end
fid=fclose(fid);

I still wonder why don't do this initially, though, when created these files instead of needing second pass...

[Aaron Smith]

Using dlmread results in an out of memory error that did not arise with fopen

[Stephen23]

@Aaron Smith: are you still trying to resolve this question? May I be of assistance?

[Aaron Smith]

Yeah. I was trying to rearrange and alter some of the things in dpb's code or combine anything i could with what i already had, adapting it for my needs but i thus far have not been able to. That's the reason behind my lateness in replying. Right now i am looking up how can use the ignore (~) to skip the line number

[Stephen23]

@Aaron Smith: first please give some background:

1. Are you still trying to read that 50x1024x1025 file? Is the data different in any way?
2. What is the new task? How is it different to that of the earlier (working) code that I came up with? (the original task: split a large text file of numbers into 50 smaller files).
3. If you are trying to remove the line number (as your question above states), why did you remove the part of my code that does exactly that? You specifically asked how to do this, and in the following comment I showed you how.

[Aaron Smith]

The code you had written before was not ignoring the line number. I was using it intact

[Stephen23]

@Aaron Smith: "I was using it intact" The code in your question above uses this line:

fprintf(f2d,'%s\n',str); %

and the code that I gave you in February uses this line:

fprintf(f2d,'%s\n',str(idx+1:end));

You can find my original code (including the code that removes the line number) here:

https://www.mathworks.com/matlabcentral/answers/324347-data-is-not-saving-to-the-workspace#comment_430364