[wg-camlp4] My uses of syntax extension

[Alain Frisch]

On 01/25/2013 08:38 PM, Török Edwin wrote:
> * camlp4o     - for better syntax errors (see http://caml.inria.fr/mantis/view.php?id=5068). This is a time-saver during development

Interesting.  It would certainly be useful to improve OCaml syntax error 
messages.  There has been some improvements recently, with the addition 
of some explicit error rules, but more work (or maybe a different 
approach as the one you suggest in the note of this ticket) would be needed.

Of course, nothing will prevent you from continuing using camlp4 instead 
of the official OCaml parser, as a "local convenience" (code you might 
distribute will not depend on camlp4, at least if you don't start 
relying on differences between camlp4's ocaml parser and the official one).

> * js_of_ocaml - (blogpost says its not really needed if a different operator would be used)

Indeed, it was our first use of -ppx here at LexiFi.  Some version of it 
can be found there:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/trunk/experimental/frisch/js_syntax.ml?revision=12932&view=markup

Basically, instead of introducing new syntax (o ## m) to access, say, a 
Javascript property, we ask the user to write:

   JVS.(o.m)

In the scope of a local-open on the pseudo JVS module name, we interpret 
a few syntactic constructs as meaning something else.  We did the 
minimum work required to allow us playing with js_of_ocaml as part of an 
internship exploratory project.  Hopefully, the js_of_ocaml folks will 
pick up on that and provide all the required syntactic support, 
including for the creation of JS objects and so on.  (We could for 
instance use the following syntax: JVS.(object ... end).)

Note that we have reused the compact syntax for local opens in order to 
define a special "scope" where some constructions are interpreted 
differently.  Maybe we should keep two different kinds of node in the 
Parsetree to distinguish "let open M in e" from "M.(e)".  Or maybe we 
should introduce explicitly another kind of "named scope" to the OCaml 
grammar, to be interpreted by -ppx rewriters  (and this could be 
available not only for expressions, but also for other entities like 
patterns, types, etc).  Or maybe we can find a nice syntax for 
"attributes" which could be used for this kind of purposes.  Or is the 
solution based on piggy-backing the syntax for local opens just fine? 
Comments are welcome!


> * bisect      - haven't figured how to use ppx from oasis/ocamlfind, The camlp4 one is as simple as:
> 	$(SETUP) -build -tag pkg_bisect -tag pkg_str -tag syntax_camlp4o -tag syntax_bisect_pp $(BUILDFLAGS)

We use the -ppx version of Bisect here.  This is probably not supported 
by ocamlfind right now.  We simply add the following command-line options:

  * for compiling:
     -ppx ..../bisect_ppx$(EXE)	

  * for linking:
     -I <path_to_bisect>  bisect.cma (or .cmxa)


> camlp4 syntax extensions needed to build libraries that I use:
> lwt, tyxml (needs camlp4of), camlp4.macro

tyxml: my understanding is that support for quotations (with 
anti-quotations in them?) would be enough.

camlp4.macro:  does this refer to pa_macro?  It seems that this is more 
about conditional compilation than proper macro expansion.  I suspect 
that conditional compilation is used rarely enough in OCaml project, so 
that a slightly more verbose syntax would not a big deal.  Do you confirm?

See 
http://www.lexifi.com/blog/syntax-extensions-without-camlp4-lets-do-it 
for a -ppx based version of conditional compilation.  The syntax is:

   include IFDEF(SYMBOL)(struct
      ... (* structure items in case SYMBOL is defined *)
   end)(struct
      ... (* structure items in case SYMBOL is not defined *)
   end)



For lwt: this project (and similar ones) might justify some special 
extensions to the concrete grammar.  There are basically two approaches 
are:  either support "let-fix" operators directly in the OCaml or just 
extend the syntax and let -ppx rewriters to the job.  There are several 
variations around the first solution, but we could for instance allow 
defining let-fix operators like:

   let (let***) x f = ...

and then use them as:

  let*** x = e1 in e2

which would be just sugar for:

   (let***) e1 (fun x -> e2)

(here I've assumed that let-fix operators are made of the characters 
"l", "e", "t" followed by an non-empty sequence of "operator" characters.)

The second approach is simply to extend the parser to accept

   let*** x = e1 in e2

and represent this explicitly in the Parsetree (a -ppx rewriter will 
then have to be used to rewrite that to something the type-checker accepts).

Both approaches can be combined: a default behavior for let-fix 
operators, but with an explicit representation in the Parsetree, 
allowing -ppx rewriters to provide a different interpretation. 
Typically, lwt has a notion of parallel binding, which would probably 
require some custom support (-ppx).


> Syntax extensions I considered using but never got around to:
> pa-monad (didn't really need it), bitstring

pa-monad: Wojciech Meyer announced omonad, a -ppx version of pa_monad.

bitstring: see my comment in another post.  Conceptually, this is quite 
similar to Sedlex.  The question is whether we can come up with a 
reasonable syntax (either with standard patterns, or maybe with pattern 
quotations?).

> Also when initially learning OCaml I have used the revised syntax (camlp4r) quite heavily.
> Nowadays I write everything in the original syntax.
> While I wouldn't miss the revised syntax, I do think that its much easier to first learn the revised syntax so it'd be nice
> if the camlp4 replacement provided that as well.

See the other post.  -ppx is really aimed at "syntax extensions"
which are more about code generation (type-conv, sedlex, ...) or code 
manipulation (conditional compilation, ...) than actual changes to the 
concrete syntax.  When the concrete syntax is the main objective, I can 
see two different cases:

   - Small local syntactic additions, like the one implemented by 
pa_lwt: we should think about generic extension points in the official 
parser to support them (together with -ppx rewriters).  Or even direct 
support in the compiler when it makes sense (e.g. direct support for 
local opens is much better than pa_openin).

   - More pervasive changes to the whole language grammar (like the 
revised syntax):  this is really what -pp is good at, and I personally 
don't have plans to propose an alternative for these cases.  It's good 
that you can experiment with parsers written with any technology.  That 
said, while I can admit that alternative syntaxes are interesting for 
experiments or educational purposes, I believe that released source code 
(libraries, applications) should stick to the official syntax, to avoid 
fragmentation of the community and issues with editors/IDE.



Alain