[wg-camlp4] nested quotation lexing problem?

[Wojciech Meyer]

Hi,

Bob Zhang <bobzhang1988 at gmail.com> writes:

>    Sample 1:
>    {:q|  "|}" |}
>        Shall we accept this or reject it? the intuition is that this
> is a legal quotation, however, if we treat the contents inside the
> quotation in a verbatim way, this is illegal...

I think quotation engine should lex something in between quotes as a
string, therefore I believe it ought to be accepted. From what I can
think, " pair is already so special that it makes sense to lex it always
as string. It might be a bit more difficult to do, but I think it's
worth.

>   Sample 2:
>   {:q| '"' |}    
>      If we treat " inside the quotation specially, the quotation above
> seems to be illegal (a non-terminated string).

Again pair of ' is also special, and the lexer should be aware always
about this.

>    If we treat both ' and " inside quotation in a special way, which
> is good in most cases, then we lose some use cases, for example, the
> contents inside the quotation have to follow the same string char
> lexical convention as OCaml.. what do you think?

I'd make it following OCaml convention, but also make it plugable so the
user can choose how to lex the contents, because string can contain
arbitrary text, and I've never seen a computer language that had
different delimiters (apart from Python ''') so it all makes sense. On
other hand allowing strings between other delimiters would be good to.

Now it's time to say, you would love how the PEG packrat parses, because
it does not have lexing pass so you would not have this problem. (but
has some slightly different).

Wojciech