[wg-camlp4] Time for a summary?

[Gabriel Scherer]

>> c) Sebastien Mondet suggested that run-time types or dyntype might be
>> sufficient without ppx for many extensions.
>>
>> Markus Mottl pointed out that there are runtime performance
>> implications for using those solutions.
>
> I strongly agree with Markus on this.  Run-time types are lovely, but
> we really need something more efficient for the vast majority of our
> camlp4 needs.  Using a comparison function constructed with run-time
> types, for example, would be a disaster performance wise.  We've done
> experiments internally, and the performance hit is absolutely
> crushing.

Two remarks on this.

First, it is unclear that runtime types (or any form of internalized
type-driven value generation) will scale to higher-kinded settings
(generating a "Functor" instance out of a parametrized datatype), so
there is also an argument of expressivity. In the current state of the
OCaml type system, "a ty" is only robustly defined for closed types or
type schemes (as opposed to parametrized types as type operators, that
is anything of kind different from '*'), and while I don't exclude
impressive workarounds from Alain or other, I wouldn't trust the
dyntype approach to work really well on type operators.

Second, Pierre Chambart and Grégoire Henry are working on a different
take on type-directed code injection that is more general than just
inferred dynamic type representations, and could make much of that
performance overhead you're mentioning go away, at least on paper
(whether the actually generated terms will be actually optimized as
you would hope is another question in which I'm sure they are also
interested). The idea is that instead of having a type-directed
generation of a ('a ty) and then a function ('a ty -> 'a printer), you
could directly generate ('a printer) in a type-directed way. This
generalization allows to express type-directed behavior that are not
defined on all types (eg. comparison rejected statically rather than
at runtime on function types), and could potentially make the
performance issue simpler. Note that in theory clever partial
evaluation could work on the previous dyntype approach (but I think
that's beyond hope for now).

On Thu, Feb 7, 2013 at 4:48 PM, Yaron Minsky <yminsky at janestreet.com> wrote:
> Thanks Leo!  This is very helpful.  Maybe we should throw this up on a
> wiki page somewhere, and keep it up to date as proposals show up.
>
> A few notes of my own below.
>
> On Wed, Feb 6, 2013 at 1:11 PM, Leo White <lpw25 at cam.ac.uk> wrote:
>> c) Sebastien Mondet suggested that run-time types or dyntype might be
>> sufficient without ppx for many extensions.
>>
>> Markus Mottl pointed out that there are runtime performance
>> implications for using those solutions.
>
> I strongly agree with Markus on this.  Run-time types are lovely, but
> we really need something more efficient for the vast majority of our
> camlp4 needs.  Using a comparison function constructed with run-time
> types, for example, would be a disaster performance wise.  We've done
> experiments internally, and the performance hit is absolutely
> crushing.
>
>> d) Jeremy suggested that rather than implementing extensions as AST
>> transformers that operate over the whole AST they might be implemented    as
>> transformers only for the part of the AST that they were to operate    on.
>> So an extension used like this:
>>
>>      (:perform
>>          x <-- m;
>>          y <-- n;
>>          return (x y))
>>
>>    would be implemented as a function with type:
>>
>>      val perform : Parsetree.expression -> Parsetree.expression
>>
>>    He suggests that this may help with scoping, safety and
>> compositionality.
>>
>>    Alain pointed out that some legitimate uses of ppx don't work nicely
>> with the constraint that they can operate only on a marked fragment. He
>> also pointed out that handling extensions as those extensions are found
>> within the source forces a top-down expansion order, which is not
>> necessarily desirable.
>
> I wonder if this is a place where we can have it both ways: have an
> underlying ppx mechanism that is a whole-AST transformation; and
> expose libraries that support more disciplined, scoped
> transformations.  We could strongly encourage the latter, but still
> allow the former when necessary.
>
>> e) I suggested an extension to Jeremy's proposal. This involves giving the
>> expansion functions in their own kind of file (".mlq" - compiled to
>> ".cmq"), referring to them using a namespace mechanism, and then having
>> the compiler itself perform the expansion of the extensions.
>>
>>    There are more details in my blog post:
>>
>>      http://www.lpw25.net/2013/02/05/camlp4-alternative-part-2.html
>>
>>    Since such a solution would take a while to implement, and since moving
>> an extension from ppx to it would be trivial, I proposed using ppx for
>> the short/medium term.
>>
>>    I suggested that such a mechanism might improving tooling and make it
>> easier for average users to use extensions.
>>
>>    Both Gabriel and Alain suggested that the use of namespaces might be
>> unnecessary and too heavyweight. Alain also questioned the need for
>> special ".mlq" files.
>
> I do like the idea of eventually getting to a name-spaced solution.
> In our world, there are really two cases: one, where we really just
> want to extend the language, and we're happy to specify which language
> we're using fairly broadly at the build-system level.  For such
> things, namespaces do not seem helpful.
>
> The other case is where you want to invoke a particular,
> special-purpose syntax extension for just one file.  There, having
> something that would allow the specification of the extension in the
> file, ideally with some integration with namespaces, seems lovely.
> That said, I think it's a less important use-case in the current
> world.
>
> I wonder, does anyone on the list of have much experience with
> Scheme's approach to macros?  There's been a lot of work there,
> covering everything from hygeine to integration with module systems.
> It would be great to leverage off of that experience.
>
> y
> _______________________________________________
> wg-camlp4 mailing list
> wg-camlp4 at lists.ocaml.org
> http://lists.ocaml.org/listinfo/wg-camlp4