import data function breaks at NaN or not a number

I am trying to use import_data function to import dataset from the data file. This data set is generated from iterations of an experiment based upon the response from the subjects. This kind of data set is bound to end up have a Not a Number or a missing response and not extract anything after that. I am trying to figure out a way to import data and not have the function break at Not A Number or a missing response. The function extracts the row before the Not a Number and not after the NaN row. I am attaching the data file that i am interested in. Any help would be greatly appreciated.

P.S. The experiment running the experiment generates a .txt file everytime and i am not allowed to mess with that program.I am including the .txt file as well

Answer 1

[Guillaume]

With format like this you have no choice but to write your own parser. Here is a draft:

function testresult = import_testresult(filename)
    %import_testresult imports an individual test results stored in a text file
    %
    % filename:  full path of the file to import  
    % tesresult: a structure containing the test results
    filecontent =  fileread(filename);  %read whole file at once
    sections = strsplit(filecontent, ''''''''''''''''''''''''''''''''''''''''''''''''''''''''); %split into individual sections
    %the use of quotes for delimiting section is unfortunate as each quote needs to be escaped in matlab
    sections = cellfun(@(s) strsplit(s, {'\n', '\r'}), sections, 'UniformOutput', false); %split each sections into lines
    %at this point, you have 4 sections (+1 empty one for the last line of the file) split into individual lines
    %parse each section.
    %
    %section 1: assume all values are text. Note that the last cell is empty (due to the way the file is parsed earlier)
    propvalues = regexp(sections{1}(1:end-1), '\[([^\]]+)\] (.+)', 'tokens', 'once'); %extract property names and values
    propvalues = cellfun(@(pv) {matlab.lang.makeValidName(pv{1}), pv{2}}, propvalues, 'UniformOutput', false); %convert property names into valid matlab variable names
    propvalues = vertcat(propvalues{:})'; %concatenate into a single cell array
    testdescription = struct(propvalues{:}); %convert into structure
    %
    %section 2: assume all values are numbers. Note that the first and last cell are empty (due to the way the file is parsed earlier)
    propvalues = regexp(sections{2}(2:end-1), '\[([^\]]+)\] (.+)', 'tokens', 'once'); %extract property names and values
    propvalues = cellfun(@(pv) {matlab.lang.makeValidName(pv{1}), str2double(pv{2})}, propvalues, 'UniformOutput', false); %convert property names into valid matlab variable names
    propvalues = vertcat(propvalues{:})'; %concatenate into a single cell array
    testsummary = struct(propvalues{:}); %convert into structure
    %
    %section 3: assume it's a table of numbers with two header lines. missed and no response are converted to NAN
    %Note that the first and last cell are empty (due to the way the file is parsed earlier)
    results = cellfun(@(l) strsplit(l, '\t'), sections{3}(4:end-1)', 'UniformOutput', false); %split each line at the tab character
    results = str2double(vertcat(results{:})); %convert to a matrix      
    %
    %section 4: assume it's a column of number with a header.
    %Note that the first and last cell are empty (due to the way the file is parsed earlier)
    rawtimes = str2double(sections{4}(3:end-1)');
    %put everything together into one structure
    testresults = struct('description', testdescription, 'summary', testsummary, 'results', results, 'rawtimes', rawtimes);
end

There's no validation anywhere that the file conforms to the expectations, you would have to add that to make the code robust.

Answer 2

[Walter Roberson]

I have no problem with that file even on a Mac (which cannot use Excel to open the file.)

[num, txt, raw] = xlsread('time-react.xlsx');

The num array has NaN for the missing or string responses. txt or raw can be used to get to the corresponding strings.

[Sanwal Yousaf]

i should have added that the program running the experiment generates a .txt file and i am not allowed to mess with that program. This data was just a copy from that .dat file.

[Walter Roberson]

import_data() is completely unsuited for importing files with multiple structures. If you are required to use import_data() for that USE_THIS.txt file then there is no way you are going to be able to do it.

[Sanwal Yousaf]

any suggestions?

[Walter Roberson]

My suggestion would be to report back to whoever imposed the restriction of using import_data() to tell them that it is not at all suitable for importing files that look anything like that.

If it was you who choose to use import_data() then "Don't do that!". Guillaume made a start at code that could be used.

[Guillaume]

I assume the question has been edited at some point since I don't see where the OP says he has to use import_data, nor do I see the excel file you reference in your initial answer.

Certainly, there's no single one-liner operation that can import the file neatly.

[Walter Roberson]

There was originally an excel file that would have been imported without difficulty; that was replaced with the tricky text file.

[Sanwal Yousaf]

The excel file was made by me for ease. The text file is something that the function would have to deal with. There is a program that runs the experiment and then generates the data in a text file and i am not allowed to mess with that program. As a result, the code that i write must be adept at dealing with .txt files for the forseeable future