[wg-camlp4] My uses of syntax extension

[Török Edwin]

On 01/28/2013 02:04 PM, Alain Frisch wrote:
> On 01/25/2013 08:38 PM, Török Edwin wrote:
>> * camlp4o     - for better syntax errors (see http://caml.inria.fr/mantis/view.php?id=5068). This is a time-saver during development
> 
> Interesting.  It would certainly be useful to improve OCaml syntax error messages.  There has been some improvements recently, with the addition of some explicit error rules, but more work (or maybe a
> different approach as the one you suggest in the note of this ticket) would be needed.
> 
> Of course, nothing will prevent you from continuing using camlp4 instead of the official OCaml parser, as a "local convenience" (code you might distribute will not depend on camlp4, at least if you
> don't start relying on differences between camlp4's ocaml parser and the official one).
> 
>> * js_of_ocaml - (blogpost says its not really needed if a different operator would be used)
> 
> Indeed, it was our first use of -ppx here at LexiFi.  Some version of it can be found there:
> 
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/trunk/experimental/frisch/js_syntax.ml?revision=12932&view=markup
> 
> Basically, instead of introducing new syntax (o ## m) to access, say, a Javascript property, we ask the user to write:
> 
>   JVS.(o.m)
> 
> In the scope of a local-open on the pseudo JVS module name, we interpret a few syntactic constructs as meaning something else.  We did the minimum work required to allow us playing with js_of_ocaml as
> part of an internship exploratory project.  Hopefully, the js_of_ocaml folks will pick up on that and provide all the required syntactic support, including for the creation of JS objects and so on. 
> (We could for instance use the following syntax: JVS.(object ... end).)
> 
> Note that we have reused the compact syntax for local opens in order to define a special "scope" where some constructions are interpreted differently.  Maybe we should keep two different kinds of node
> in the Parsetree to distinguish "let open M in e" from "M.(e)".  Or maybe we should introduce explicitly another kind of "named scope" to the OCaml grammar, to be interpreted by -ppx rewriters  (and
> this could be available not only for expressions, but also for other entities like patterns, types, etc).  Or maybe we can find a nice syntax for "attributes" which could be used for this kind of
> purposes.  Or is the solution based on piggy-backing the syntax for local opens just fine? Comments are welcome!
> 
> 
>> * bisect      - haven't figured how to use ppx from oasis/ocamlfind, The camlp4 one is as simple as:
>>     $(SETUP) -build -tag pkg_bisect -tag pkg_str -tag syntax_camlp4o -tag syntax_bisect_pp $(BUILDFLAGS)
> 
> We use the -ppx version of Bisect here.  This is probably not supported by ocamlfind right now.  We simply add the following command-line options:
> 
>  * for compiling:
>     -ppx ..../bisect_ppx$(EXE)   
> 
>  * for linking:
>     -I <path_to_bisect>  bisect.cma (or .cmxa)
> 
> 
>> camlp4 syntax extensions needed to build libraries that I use:
>> lwt, tyxml (needs camlp4of), camlp4.macro
> 
> tyxml: my understanding is that support for quotations (with anti-quotations in them?) would be enough.
> 
> camlp4.macro:  does this refer to pa_macro?  It seems that this is more about conditional compilation than proper macro expansion.  I suspect that conditional compilation is used rarely enough in
> OCaml project, so that a slightly more verbose syntax would not a big deal.  Do you confirm?
> 
> See http://www.lexifi.com/blog/syntax-extensions-without-camlp4-lets-do-it for a -ppx based version of conditional compilation.  The syntax is:
> 
>   include IFDEF(SYMBOL)(struct
>      ... (* structure items in case SYMBOL is defined *)
>   end)(struct
>      ... (* structure items in case SYMBOL is not defined *)
>   end)
> 

This is for ocamlnet and ounit, I don't use it in my code.

Backward compatibility would be appreciated, especially for something as simple as IFDEF.
Otherwise if a library switches to use the new ppx-based tool, it will no longer build with OCaml 3.12.1 + the camlp4 based syntax.
In fact the uses of IFDEF are quite simple, and maybe a tool that uses the lexer from compiler-libs and performs a lex-time ifdef replacement would work.

Alternatively whenever a new ppx-based tool is released that requires a new syntax, its camlp4-based complement should be updated to support the new syntax as well.
But that is probably a lot of work for little benefit.

> 
> 
> For lwt: this project (and similar ones) might justify some special extensions to the concrete grammar.  There are basically two approaches are:  either support "let-fix" operators directly in the
> OCaml or just extend the syntax and let -ppx rewriters to the job.  There are several variations around the first solution, but we could for instance allow defining let-fix operators like:
> 
>   let (let***) x f = ...
> 
> and then use them as:
> 
>  let*** x = e1 in e2
> 
> which would be just sugar for:
> 
>   (let***) e1 (fun x -> e2)
> 
> (here I've assumed that let-fix operators are made of the characters "l", "e", "t" followed by an non-empty sequence of "operator" characters.)
> 
> The second approach is simply to extend the parser to accept
> 
>   let*** x = e1 in e2
> 
> and represent this explicitly in the Parsetree (a -ppx rewriter will then have to be used to rewrite that to something the type-checker accepts).
> 
> Both approaches can be combined: a default behavior for let-fix operators, but with an explicit representation in the Parsetree, allowing -ppx rewriters to provide a different interpretation.
> Typically, lwt has a notion of parallel binding, which would probably require some custom support (-ppx).

Thinking about backward compatibility again perhaps there should be more transformation phases:
 -ppl Parser.token -> Parser.token, this can transform 'lwt' into 'let***' (or whatever ppx expects) for backwards compat
 -ppx Parsetree -> Parsetree as currently

Also it'd be nice to have:
 -ppt Typedtree -> Typedtree for type-level transformations

> 
> 
>> Syntax extensions I considered using but never got around to:
>> pa-monad (didn't really need it), bitstring
> 
> pa-monad: Wojciech Meyer announced omonad, a -ppx version of pa_monad.
> 
> bitstring: see my comment in another post.  Conceptually, this is quite similar to Sedlex.  The question is whether we can come up with a reasonable syntax (either with standard patterns, or maybe
> with pattern quotations?).
> 
>> Also when initially learning OCaml I have used the revised syntax (camlp4r) quite heavily.
>> Nowadays I write everything in the original syntax.
>> While I wouldn't miss the revised syntax, I do think that its much easier to first learn the revised syntax so it'd be nice
>> if the camlp4 replacement provided that as well.
> 
> See the other post.  -ppx is really aimed at "syntax extensions"
> which are more about code generation (type-conv, sedlex, ...) or code manipulation (conditional compilation, ...) than actual changes to the concrete syntax.  When the concrete syntax is the main
> objective, I can see two different cases:
> 
>   - Small local syntactic additions, like the one implemented by pa_lwt: we should think about generic extension points in the official parser to support them (together with -ppx rewriters).  Or even
> direct support in the compiler when it makes sense (e.g. direct support for local opens is much better than pa_openin).
> 
>   - More pervasive changes to the whole language grammar (like the revised syntax):  this is really what -pp is good at, and I personally don't have plans to propose an alternative for these cases. 
> It's good that you can experiment with parsers written with any technology.  That said, while I can admit that alternative syntaxes are interesting for experiments or educational purposes, I believe
> that released source code (libraries, applications) should stick to the official syntax, to avoid fragmentation of the community and issues with editors/IDE.
> 

Will have to think about this more.

--Edwin