[wg-camlp4] Meta Programming from the view of the implementaion

[Alain Frisch]

On 01/30/2013 02:18 PM, Hongbo Zhang wrote:
> Ast Rewriter is *bad *for arbitrary code generation, its neat case is
> something like bisect.

One of the most widely used kind of Camlp4 extensions today is about 
code generation based on type declarations (plus some annotations), à la 
type-conv or deriving.  This seems to be a perfect match for AST 
rewriters plus attributes.  Don't you agree?

> And Ast Rewriter is not composable at  all,
> suppose you have two AST Rewriters A,B ,there are no guarantee that  A.B
> == B.A, it just gives you an illusion that it works, probably it
> silently fails.

I don't agree.  A and B are functions from AST to AST, so there is a 
very nice notion of composition: the composition of functions.  It is 
true that A.B and B.A don't have the same behavior but at least, I can 
understand what A.B and B.A will do as soon as I know how A and B behave 
individually, without having to know anything about their implementation.

This is very different from camlp4's model of letting extensions patch a 
common grammar in place, by removing rules and injecting new ones which 
can (partially) hide existing ones.  The order in which those updates to 
the grammar are applied also matters, and it is impossible to know how 
the combination of two extension behaves without knowing precisely how 
they are implemented.

I know you have projects to provide a cleaner functional API for 
grammars, but I don't see how modifications of the concrete syntax can 
be combined, in full generality, in a predictable way.  It seems you 
also have plans to get rid of the ability to modify the concrete syntax. 
  This sounds very reasonable to me.

>     3. How do you implement the expander?  Somehow, you need to parse
>     the content of the quotation (stored as a string in the AST), which
>     involves non trivial stuff, like a parser being able to parse OCaml
>     code mixed with something else.  Personally, I don't know how to
>     implement the quotation expander with the parsing technologies I'm
>     familiar with.
>
> when you design your DSL, the left part is pretty easy to parse, when
> you see the delimiter "->", invoking an existing parser.

So you need to:

  - Write a custom parser for left parts (regexps), which stops at "->".
To write this parser, you need to use ocamlyacc, menhir, or another 
tool. (And probably plug the official OCaml lexer?)
  - Call the official(?) OCaml parser on its "expr" entry and hope that 
it stops exactly when it encounters the next "|" (does this really work?).
  - Adapt the locations returned by this parser, or be careful to 
initialize the lexbuf with locations that match exactly the concrete 
location of the corresponding part in the original source code.

This seems much more complex than the current implementation of sedlex 
(or a variant of it where regexps would be written with some concrete 
syntax), for no obvious benefit.


Alain