Production names can't contain whitespace

[elhabert]

I wanted to have a \/[A-Za-z0-9-]+\/ "unknown directive" lex rule, but this turns out to produce an error. I haven't checked if Lex or Flex accept those inputs, but at least my syntax highlighter does.

IMO, regardless of whether spaces become allowed, it might be a good idea to document the accepted charset for token names?

[ratmice]

It doesn't seem so much like spaces aren't allowed, or the character set only allows certain characters, it appears that grmtools lrlexer allows pretty much any utf-8 name, however how it finds the name is a little bit funky and leads to not allowing spaces.

1. It does a reverse search backwards from the end of the line first space.
2. Checks that the next character after the first space is a " or a '.

I believe the main reason it is doing this is that grmtools has historically used a simple line splitting method
where everything before that space is part of the regex, and everything after that space is part of the token name.
It didn't attempt to parse the regex really at all in any way (until escaping was extended),
and when you consider a regex with embedded spaces \ "foo".

perhaps it could be extended to search for the first space followed by a " or ',
It seems like spaces shouldn't be allowed in a %token directive,
but I think the parse_name function using this RE_TOKEN

grmtools/cfgrammar/src/lib/yacc/parser.rs

Line 292 in 419fa16

LazyLock::new(|| Regex::new("^(?:(\".+?\")|('.+?')|([a-zA-Z_][a-zA-Z_0-9]*))").unwrap());

I'd have to go and read the regex docs to figure out the first half of that regex there is doing (?:(\".+?\")|('.+?'),
but the second half looks like it only accepts a subset of the valid lrlex token names to be used in %token (i.e. alphanumeric + '_' without spaces)

So perhaps there is some wiggle room, for allowing space, when not used in a %token directive?

Edit: I think it'll work anyways, but I'm a little uncertain/skeptical of the behavior when there is an error in the lrlex file like an unquoted name, the "problem regex" for this is one like:

\ " "regex contains a space followed by unescaped quote"

I think the issue is perhaps that then if we have something like that regex but

\ " Unquote_name

In this case we still have an error because of the terminating quote, which brings us to this odd-ball where the quote is somehow "both" part of the regex, and part of the token name depending upon whichever forward/reverse direction you read it

\ "Unquoted_name"

[ltratt]

This is one of those fuzzy things where no-one's quite sure what old tools, or modern people, expect. I think I'd be OK with saying "if the line ends with " then we search backwards for the matching opening quote (allowing for escaping etc.)."

[I'm not sure how other parts of the system will end up dealing with whitespace in names, though. I tend towards the Unix philosophy of "spaces are allowed, but you're better off not using them" unless I can convince myself of the opposite!]

[ratmice]

Yeah I think that "if the line ends with ", search backwards for the matching quote, preceeded by a non-escaped space
might work

%grmtools{allow_wholeline_comments}
%%
// Just to note we can't escape inside the name
\\ "\"
// We can't just do the simple reverse search for ", because of the following
\" """
// We currently accept the following regex
\ " "name"

The "preceeded by a non-escaped space"
specifically allows us to avoid something like, disambiguating the quote at the end of the regex from the first quote of the name.

\ "name", or \ " name"

Edit:
It seems we also need to have some ability to recognize that the following is not an escaped space, but it seems reasonable?

\\ "foo"