[wg-camlp4] Time for a summary?

[Alain Frisch]

On 2/7/2013 3:56 PM, Hongbo Zhang wrote:
> Not always, as I pointed out, as long as you combine several global Ast
> Rewriter, I am  optimistic that Fan has a great chance to outperform
> ppx.

Oh sorry, I thought you were referring to the compilation of Fan itself.

> Well, things are always possible, but just too difficult in some cases,
> remember that (Camlp4/Camlp5/Fan) preprocess the intermediate Ast, if we
> clutter Parsetree too much, like adding some attributes node per syntax
> category and ignored by the compiler later, which may require all other
> tools to mix Intermediate Ast with Parsetree, it's always possible, just
> too hard.

I don't follow. We are talking, basically, about adding simple 
constructions to the AST, like:

   Pexp_annotation of expression * expression
   Pexp_extension of string * expression

and you were suggesting that this would almost kill Camlp4/Fan. Why?

> I am perfectly fine with ppx so far, and I am happy to see
> bisect switch to ppx, because global Ast Rewriter happen to be very fit
> there, but  if we try to make everything possible in ppx, there's a
> large chance that it will result in a ugly design in the compiler.

Again, we are talking about adding a few constructs to the AST.  The 
only real "consumer" of the AST in the compiler is the type-checker, and 
its behavior is pretty clear for the new nodes:  attributes are ignored, 
extensions raise a type error.  Where is the "ugly design"?

> Let me explain a bit why quosi-quotation is superb here:
> 1. It quickly goes into my muscle memory, because I write ocaml everyday
> (not parsetree)

So do I and thus I'm more familiar with constructors of a standard sum 
types (wrapped into simple functions) than something which requires a 
new syntax.

>, it's just my *intuition* that the Ast should be
> constructed that way, and I don't need to remember those alien function
> names, to be honest, I don't know what's the module E here. If I want to
> make a piece of code into data, I just need to quote them, add
> "{:expr||}" surrounding them, that's all. How would that happen in your
> "E" module?

I don't see where "lifting ASTs" is required in order to implement all 
the interesting tools we are discussing here (except Fan itself).

> 2. The E module seems to be a bit magic here

E is defined in Ast_mapper:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/trunk/parsing/ast_mapper.mli?view=markup

For instance, let_ is defined very cleverly as:

let mk ?(loc = Location.none) x = {pexp_desc = x; pexp_loc = loc}
...
let let_ ?loc a b c = mk ?loc (Pexp_let (a, b, c))

There is absolutely no magic.


> 3. For the parsetree, if you want to do syntactic grep over bigarray,
> you will get into much trouble:
>     ({pexp_desc=Pexp_ident
>                      {txt= Ldot (Ldot (Lident "Bigarray", array),
> ("get"|"set" as gs)) ;_};_}
>     This does not solve the problem at all, since it can not
> discriminate "a.{b,c}" from "Bigarray.get a b c " in Fan, you only need
> to do say
>     "`Bigarray (a,b,c)"
>     There are so many syntax desugaring in the parsetree that I don't
> know where to stop.

This is a valid point, and I believe there would not be any strong 
opposition to move some desugaring logic from parser.mly into the 
typechecker, i.e. make Parsetree look more like a parse tree than an AST.

That said, I've no idea what

  `Bigarray (a,b,c)

means.


>  > Can you describe in more detail your position concerning the
> following points?
>  >
> If it can be implemented by ppx, it can be straightforwardly translated
> into Fan, since ppx is simply a tiny proper subset of Fan(the part of
> global Ast Rewriter).

Will you introduce in Fan the notion of attributes attached to every 
possible category and represented explicitly in the (Fan) AST?  I 
thought you only had the intention to extend the syntax of OCaml with a 
single construct {:xxx| ... |}  with  .... being left unparsed.  I don't 
see how you can simulate "ignorable attributes" with that. Can you 
elaborate?

> Fan provides first class parser/lexer(or in-line parser
> lexer), which is good, and there are people who appreciate
> it(https://sympa.inria.fr/sympa/arc/caml-list/2013-02/msg00020.html),

I'm fine with Camlp4/Fan as a "parsing technology", and I've used it 
quite a lot (e.g. the parser for CDuce was implemented in Camlp4).  But 
this is largely independent of the use of Camlp4 to write "syntax 
extensions".  I believe that most interesting uses of Camlp4 to write 
syntax extensions today don't require any parsing technology at all, 
because they are not about really extending the syntax per so, but 
rather about providing convenient tools to OCaml developpers, and this 
does not require to play with the syntax.


Alain