How to filter certain parts of a string.

Hello,

So I have a string file which has a ton of data in it.

Let's say it looks like this:

"Introduction:

x1x1x1x1x1x1x1x1x1x1x1x1x1

Summary:

x2x2x2x2x2x2x2x2x22x2x2x2

Conclusion:"

What I would want to do is setup a filter so that it reads the keyword Introduction and Summary and captures everything in between. While ignoring everything else.

In this case it should be

"Introduction:

x1x1x1x1x1x1x1x1x1x1x1x1x1"

If it helps, I am running a script with this code to generate this string file after reading the contents from a word document:

word = actxserver('Word.Application');
word.visible = 0;
wdoc = word.Documents.Open('wordfile.docx');
x = splitlines(string(wdoc.Content.Text));  
wdoc.Close;
word.Quit; 

Answer 1

[Walter Roberson]

S = char(wdoc.Content.Text);
x = regexp(S, '^Introduction:.*?(?=^\nSummary:)', 'match', 'lineanchors');

[Nom]

Thank you for such a quick response,

Unfortunately, this yields a blank 0x0 cell to me.

[Walter Roberson]

Please use

S_wdoc = wdoc.Content.Text;
S = char(S_wdoc);
save('480826.mat', 'S', 'S_wdoc')

and attach 480826.mat here.

[Nom]

Hello,

I've attached the file below

[Walter Roberson]

x = regexp(S, '^Introduction:.*?(?=^\rSummary:)', 'match', 'lineanchors');

Your data is unusual: the lines are separated with carriage return instead of newline or cr+nl .

If you plan to do more processing with pattern matching, it is often easier to work with newlines. I would suggest

S = regexprep(wdoc.Content.Text, '\r', '\n');

x = regexp(S, '^Introduction:.*?(?=\n^Summary:)', 'match', 'lineanchors','once');
x_lines = regexp(x, '\n', 'split');

[Nom]

Thank you Walter, it works perfectly!

Just going to sit down and read up on regexp so I may actually understand what is going on.

Where would I be able to find a documentation of all the symbols you used in this line?

'^Introduction:.*?(?=\n^Summary:)'

Thank you,

[Nom]

Also on a quick note, using regexp is really funky

I attached my matlab workplace again,

S is the the whole word document it captures.

But running through all the variables you gave me do not yield the proper

result for some reason. I think it has a lot to do with how the document is

formatted but not sure.

[Walter Roberson]

The regexp documentation has it all... but it can take a lot of readings to really understand it. Even today I struggle with matching multiple patterns into named tokens when there are parts of the patterns that I do not want to capture.

'^Introduction:.*?(?=\n^Summary:)'

The ^ in connection with 'lineanchors' means "beginning of line". That can be right at the very beginning of the input, or it can be immediately after a newline character.

Introduction: is literal characters to be matched.

. is any one character, possibly including newlines, unless the option 'dotexceptnewline' is present, in which case . will not cross newlines (and so becomes restricted on to the same line)

* generally means that the pattern before it can be repeated any number of times including zero times. The pattern immediately before it in this case is . so it would generally mean "any number of any characters, including no characters at all"

? in this context modifies the previous * . Normally * is "greedy", matching as far along the string as possible, so normally .* would mean effectively "move to the end of the input and then only move back as far as is needed to match whatever patterns follow". For example if the input were 'abbacad' and the pattern were 'a.*a' then the leading 'a' in the pattern would match the first a in input, and then the .* would capture as many characters as possible, which is all of the rest of the characters. Then the 'a' after that in the pattern would be checked and found not to match end-of-string so the pointer would move backwards one and would find the 'd' . That would be checked for matching 'a' and no match would be found, so the pointer would move backwards one and would find the 'a' that is second last on the input. That does match the 'a' of the input pattern, so the entire 'abbaca' of input would be matched.

The ? after .* modifies the .* to only go the minimum it needs to find a match. So with input 'abbacad' and pattern 'a.*?a', the initial 'a' would match the 'a' of the pattern, and then the '.*?' of the pattern would take effect. The parser would start examining current input to see if it matched the next pattern, the a after the .*? . It would see the first 'b' and se that did not match, so it would advance the pointer. It would then check the second 'b' and see that it did not match the needed 'a' so it would advance the pointer. It would then see the 'a' after that (the second 'a' of 'abba') and see that it does match the 'a' after the .*? so it would stop matching. The overall pattern would then have matched 'abba' with the 'cad' afterwards not having been examined yet. (After that it would normally start looking for a.*?a patterns again.)

The (?=pattern) portion you will find in the documentation for "look around". It means that the previous pattern is not considered to have matched unless what follows exactly matches the pattern. In this case the pattern is \n^Summary which is a newline (\n) followed by the word Summary at the beginning of a line. That could probably have been written as \nSummary without the ^ .

With the (?=\n^Summary) immediately following the .*? it means that the .*? should match all characters, including crossing line boundaries, until what follows is a newline followed by Summary . The newline is pattern of the "look-around" and so will not be included in what the .*? matches.

So, the effect of this pattern is to start matching at Introduction: at the beginning of a line, and stop matching just before the newline prior to Summary at the beginning of a line.

The .*? does end up crossing one line boundary, so there is a \n embedded in the result. The regexp split after that is to divide line by line.

The option 'match' for regexp says to return the character vector that is matched, instead of one of the other possibilities (the default is to return starting position numbers.)

The option 'lineanchors' for regexp says that the ^ and $ patterns are permitted to match at newline boundaries, start and end of lines, instead of start and end of entire input. Historically ^ and $ for regexp always matched line boundaries unless told to match only beginning and end of input, but for whatever reason MATLAB choose to do the opposite. Using ^ and $ to match line boundaries is, in my experience, used a lot more often, so if you use ^ or $ in your patterns you should probably be using 'lineanchors' unless you have good reason not to.

The 'once' option tells regexp to stop processing as soon as it matches the input pattern once. Otherwise it will go back to the beginning of the pattern and try to match again and again until it either fails or all of the input has been consumed. (There are a lot of situations where that is useful.)

[Nom]

Wow, amazingly written.

I will be reading through this,

Thank you for taking the time to write this out Walter!

[greenyellow22]

Hey Walter! I thought I might just ask you directly. I have a table of which one of the columns includes text. Some cells of this column look like this: 'New Scene was loaded: Castle' or 'New Scene was loaded: World1'.

I want to filter out/split for all cells with the text 'New Scene was loaded:' the last word e.g. 'Castle' or 'World1' and insert it into a new column. Can you tell me how I can split these cells based on this rule? I appreciate any help!! :)

[Walter Roberson]

TableName.NewColumnName = regexp(TableName.ColumnName, '(?<=New Scene was loaded:\s+)\S+', 'match', 'once')

There is a possibility that TableName.ColumnName might need to be {TableName.ColumnName} but I think it unlikely.