[wg-camlp4] My no-use of camlp4 syntax extensions for Otags

[Alain Frisch]

On 04/12/2013 11:26 AM, Hendrik Tews wrote:
> and nothing is clear. When I look up mkpat, then all I know is
> that an or-pattern is not a Ppat_or node, it is some record, for
> which I don't even know the type. So I would have to search for
> ppat_desc, and select the right one of the 100 matches that grep
> reports. Note that I have been lucky here, because mkpat is very
> simple and is defined in parser.mly . There are surely more
> complicated functions that are elsewhere defined.

In order to write robust AST rewriters, you'll need to be quite familiar 
with the definition of the Parsetree structure, so reading parsetree.mli 
once will be more or less necessary anyway.

The fact that most fragments are represented with a record containing a 
"location" field, a "description" field (and now, an "attributes" field) 
is pretty uniform.  I agree that the Parsetree definition would deserve 
some documentation and some introductory content will be useful as well, 
but even without any further documentation, finding the information 
relative to "Ppat_or" is not very difficult if you read parsetree.mli.

> There might however be cases with the danger of wrong
> generalization. What I mean with this is the following: The
> ocamlast tool only works on correct OCaml code. I cannot feed
> it with "assert _ " in order to get the most general ast
> tree for assertions. So instead I have to feed it with, for
> instance, "assert true" and get Pexp_assert ... I might now
> believe that "assert false" is also represented with Pexp_assert,
> which is wrong.
>
> So, if there are cases, where similar looking concrete syntax is
> represented differently, then they should be listed somewhere.

It makes sense to list all such weird cases, or, when it makes sense, to 
fix them.  For this specific examples (Pexp_assertfalse), Romain Bardou 
(email on Jan. 28) argued that it should stay as of today.  Do other 
people agree?

> For documentation need (2), ocamlast is not really a help. For
> (2) I would suggest to annotate each constructor in parsetree.mli
> with the concrete syntax. For instance
>
>    | Pexp_while of expression * expression
>       (** while <expression> do <expression> done *)

Good idea. I've started with type expressions:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/parsing/parsetree.mli?r1=13512&r2=13517

Do you feel this brings useful information in this form?

> There are probably a lot of side conditions. For
> instance, "expr_1; expr_2; expr_3" is
> Pexp_sequence(expr_1, Pexp_sequence (expr_2, expr3) and not
> Pexp_sequence( Pexp_sequence(expr_1, expr_2), expr3) (or vice
> versa).

(e1; e2; e3) is indeed parsed as (e1; (e2; e3)), but ((e1; e2); e3) will 
be represented as you expect (confirmed by ocamlast).  I don't think it 
is worth documenting again the definition of the Parsetree 
disambiguation and associativity rules; this is already done in the 
manual, and rules can be discovered with ocamlast.

Moreover, I'd argue that authors of ppx extensions don't really need to 
be familiar with this topic, since they operate on abstract syntax. 
What's important is rather to specify invariants on the Parsetree data 
structure, especially those which are not checked by the type-checker. 
(And we should probably strengthen the type-checker to test more of 
those conditions such as the validity of identifier strings, etc.)

> Another kind of side condition is that certain nodes do not and
> must not occur inside other nodes. For instance, the left
> expression in Pexp_sequence is never a Pexp_sequence. (This is
> probably wrong, but you get the idea.)

Indeed, this is wrong, but I get the idea :-)  Luckily, there aren't 
many such structural invariants, and it should not be too difficult to 
list them.

> You could write the documentation for (1) and (2) in comments or
> ocamldoc comments in the source files. However, I would prefer
> something like a wiki page, where we users can immediately
> contribute.

Authors of ppx extensions are probably going to have parsetree.mli under 
they eyes quite often.  It seems good to me to put some documentation 
directly there.  Which is not to say that an external wiki won't be 
useful as well.


Alain