Wanted: Examples on how to use "Dynamic Regular Expressions" to debug regular expressions

6 views (last 30 days)
I try to develop a function, is_string_constant, which takes a text string of Matlab code and returns a logical vector, which is true for the positions of string constants, e.g. '%s%f'. (But neither ';c=d' in a=b';c=d'; nor in comments.)
Status
  • I have not found a similar function in the FEX or elsewhere
  • I found % MATLAB Comment Stripping Toolbox by Peter J. Acklam
  • At regex101 I have a working regular expression (in PCRE(php)). Matlabs regular expressions is close to PCRE.
  • At regex101 my PCRE-expression works under Python too. regex101 automatically makes a Python-script based on my test case.
  • So far I failed to port my PCRE-expression to Matlab
  • I'm trying to use "Dynamic Regular Expressions" to understand where it goes wrong. Now, I'm moving around (?@disp($0)) in the expression and see code fragments printed in the Command Window. However, it remains to make something useful out of it.
(?@cmd)
Execute the MATLAB command represented by cmd, but discard any output
the command returns. (Helpful for diagnosing regular expressions.)
Example: '\w*?(\w)(?@disp($1))\1\w*' matches words that include double
letters (such as pp), and displays intermediate results.
Questions:
  • Is there already a is_string_constant to find somewhere
  • Where can I find tutorials and example on how to debug regular expressions in Matlab
  • Would it be crazy to try Java or Python to do the job? (I haven't used either.)
  • Other tips

Accepted Answer

Stephen23
Stephen23 on 14 Oct 2017
Edited: Stephen23 on 17 Oct 2017
As far as I can tell there is no simple "regexp-debug" tool or method, but here are a few tips based on my experience writing my FEX submission words2num. The function words2num is based around several large regular expressions in order to identify numbers written in English words. By default the main regular expression currently has a total of 2803 characters, around 100 groups, and 11 dynamic regular expressions. Obviously I spent a lot of time reading the MATLAB documentation, in particular:
and I also wrote a simple interactive tool iregexp for quickly checking what I was working on.
Your questions:
"Is there already a is_string_constant to find somewhere?"
Not that I have seen.
"Where can I find tutorials and example on how to debug regular expressions in Matlab?"
I have not seen tutorials specifically about debugging regular expressions. The hints given in the dynamic regular expression help are about the closest in the documentation: one of the most useful pieces of info for me was this line: "The operators $& (or the equivalent $0), $`, and $' refer to that part of the input text that is currently a match, all characters that precede the current match, and all characters to follow the current match, respectively." Using these, together with $1, $2, etc., made identifying and debugging the matches easier.
"Would it be crazy to try Java or Python to do the job? (I haven't used either.)"
Python regular expressions have their own quirks, but are generally easy to use and has some nice features (e.g. options DEBUG and VERBOSE).
"Other tips"
  • I broke the regular expression down into very small atomic parts, each of which was tested and matches exactly one particular feature of the words I needed to match. In the code the parts are joined systematically into one expression. This provided a way for me to ensure that the expression matched the required logic.
  • Identifying and clearly specifying the required logic was 90% of the task for me. I spent months with pieces of paper scribbling down ideas and crossing things out, breaking the problem into what could be called a high-level and mid-level description of the regular expression. Once the logic was correct the regular expression itself followed quite naturally.
  • I created a set of around two thousand test cases, some sourced from the internet and some created systematically by hand. These were hugely important to ensuring the correct output for all cases, and identifying cases that I had not considered.
  • The regexp option warnings is useful when debugging.
  • The most common challenge for me was because of greediness and backtracking not always providing the matching that I needed when there are many similar-but-different possible matches. The only solution was to go back to the paper and start again...

More Answers (1)

Cedric Wannaz
Cedric Wannaz on 14 Oct 2017
Edited: Cedric Wannaz on 16 Oct 2017
Regular expressions may not be that appropriate in this context; I used them in the past for doing exactly this, but it was too complicated for being really satisfactory.
I took 10 minutes for building a basic loop (being a regexp evangelist, it was quite painful ;)), which seems to be working on a few test strings:
strs = {
'', ...
'abc', ...
'''abc''', ...
'% ''abc''', ...
's = ''hello'' ; b = c'' ; fprintf([''A''''s content '',''%d : %s''], i, str{:}.'') % ''abc''' ...
} ;
for sId = 1 : numel( strs )
is_string_constant( strs{sId}, true ) ;
fprintf( '\n' ) ;
end
Outputs (skipping the empty string):
>> test
abc
000
'abc'
11111
% 'abc'
0000000
s = 'hello' ; b = c' ; fprintf(['A''s content ','%d : %s'], i, str{:}.') % 'abc'
00001111111000000000000000000000111111111111111011111111100000000000000000000000
EDIT 1: I spent another 20 minutes building a simple debug function (attached). It doesn't do much but avoids the hassle of updating patterns. It seems to be managing well internal levels of parentheses and escaped ones.
PS: .. but of course, I don't see why it is useful unless we can't output the match and/or tokens for a reason, so I may just have wasted 20 minutes (lol) and we are back to "However, it remains to make something useful out of it.".
>> match = regexp_debug( 'hello world', '(ll.).*?(o.l)', 'match', 'once' )
match: 'lo worl'
token_1: 'llo'
token_2: 'orl'
match =
'llo worl'
>> tokens = regexp_debug( 'hello world', '(ll.).*?(o.l)', 'tokens', 'once' )
match: 'lo worl'
token_1: 'llo'
token_2: 'orl'
tokens =
1×2 cell array
{'llo'} {'orl'}
>> [tokens, start] = regexp_debug( 'hello world', '((?<=(l|\(\)))l.).*?(o.l)', 'tokens', 'start', 'once' )
match: 'lo worl'
token_1: 'lo'
token_2: 'orl'
tokens =
1×2 cell array
{'lo'} {'orl'}
start =
4
Further EDITs:
  • 15/10 - Added the match in the output.
  • 16/10 @ 02:34UTC - Corrected bug in tokens count.
  5 Comments
per isakson
per isakson on 19 Oct 2017
Edited: per isakson on 19 Oct 2017
Yes, I've studied the code, which you attach to the comment above.
"the whole stream in one shot I guess" Yes, the seven colored lines are the debug-print-out from one call to regexp
cac = regexp( str, xpr, 'tokenExtents', 'warnings' );
I've edited my description above on my "small test".

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!