[wg-camlp4] My uses of syntax extension

[Török Edwin]

On 01/28/2013 04:33 PM, Alain Frisch wrote:
> On 01/28/2013 02:06 PM, Török Edwin wrote:
>> Backward compatibility would be appreciated, especially for something as simple as IFDEF.
>>...
>> Thinking about backward compatibility again perhaps there should be more transformation phases:
> 
> Backward compatibility is an important topic, since the goal of this working group is to propose a migration plan.
> 
> My personal view is that we should not complexify our "target" (-ppx) only to support backward compatibility.  Clearly, -ppx will require some user-visible changes, since it is no longer an option to
> change the concrete syntax.  To take a concrete example, what should a project like js_of_ocaml do, once they have a fully functional -ppx version?
> An option could be to keep maintaining their camlp4 extension in parallel, but extend it to support for the syntax chosen for the -ppx version.  The only benefit of it is that users stuck with older
> versions of ocaml can still move smoothly to the new syntax, so as to be ready to get rid of camlp4 at some point.  Other users don't need that: they can rewrite their project progressively (e.g.
> module per module) using both the camlp4 syntax extension and the -ppx rewriter, and then stop using camlp4 when everything has been rewritten.  Is it worth the effort to adapt the camlp4 extension to
> match the new syntax?  It's probably up to each project maintainer to decide (considering that supporting several versions of camlp4 is also quite challenging).

Maybe the simplest solution to maintaining backwards compatibility is to ship pre-processed source code (using -ppx version) in release tarballs of libraries,
but that isn't without problems either:
+ you don't need to have 4.00.1 to build newer versions of libraries
+ you can switch to the -ppx syntax in your library
+ building the library doesn't require building/running the syntax extension
- you need a simple way to do this, i.e. oasis/OCamlMakefile/etc. should be updated to support this transparently
- ppx's output has to be converted to source code again, and that must be backwards compatible with older OCaml versions
- ppx transformations have to be architecture independent
- without an automated way to test the above it could easily break

>  Who is really stuck with old versions
> of ocaml (and still wanting and able to use new versions of packages)?

My use-case would be: I write an application that I want to be buildable at least with OCaml 3.12.1 and 4.00.1 (3.11.2 support would be nice, but not required).
The end-user doesn't have to be aware of what OCaml is, I just want them to run my application on their OS without having to build a compiler to do that.

Of course I can stick to the last version of a library that supports both (if they introduce an incompatibility) but then I'd have to backport
bugfixes to a private branch on my own. If it comes to that I'd rather figure a way to distribute preprocessed OCaml source code for the libraries that I use.

> 
>> Also it'd be nice to have:
>>   -ppt Typedtree -> Typedtree for type-level transformations
> 
> I don't believe this is a viable option.  Producing Typedtree without breaking invariants of the type system (and the type soundness) is extremely challenging.  I'd be more comfortable with supporting
> a "Typedtree -> Parsetree" pass, where you'd only need to be able to use information from the typedtree so as to produce fragments of code which are then type-checked again.  You don't need to fully
> understand everything in the Typedtree to do that, and the type-checker acts as a safety net.
> OCamlPro worked on a project called "OCaml Templates" doing exactly that, but I don't know the status of this project.  (Someone from OCamlPro to comment?)

That'd work too, although the -ppx based tool could do all that on its own. There wouldn't be any compiler changes required (i.e. no -ppt), just a library
like astmapper to simplify this task.

In fact my -ppl proposal is not really needed either, it could be just a library and the tool: a -pp based tool.

The tool would be invoked by -pp:
load all plugins
read source
use OCaml's lexer from compiler-libs to create tokens
apply any token mappers from the plugins
create the parsedtree using compiler-libs again
apply any astmappers from the plugins
run any ppx astmappers given on cmdline
create the typedtree using compiler-libs
apply and typedtree -> parsetree mappers from the plugins
output the ast

This would then be an almost drop-in replacement for camlp4, except that syntax extensions (plugins here) will depend on compiler-libs instead of camlp4,
and of course you would have similar limitations as -ppx, except you can modify the tokenstream as well.
I haven't checked how ocamlfind invokes camlp4 exactly, but it might be possible to reuse its camlp4 support for this.

Best regards,
--Edwin