[wg-camlp4] benchmarks

[Yaron Minsky]

On Thu, Feb 7, 2013 at 4:22 PM, Gerd Stolpmann <info at gerd-stolpmann.de> wrote:
> Am 07.02.2013 21:39:39 schrieb(en) Thomas Gazagnaire:
>
>> Hi all,
>>
>> In my opinion as a *user* of camlp4, the main pitfall that I can see (and
>> which has not yet been addressed) is that it is really slowing down the
>> compilation process. To improve that, the usual methods are:
>> 1. to use native syntax extensions
>> 2. to statically link the syntax extensions into one preprocessor binary
>
>
> Well, Alain started with the suggestion to run all preprocessors as separate
> binaries. Of course, if you have many preprocessors or many files to
> preprocess, this will become slow. Under the most common assumption, though,
> namely that only a few files need preprocessing at all, this is no problem,
> and certainly the most flexible option.

I'm not sure this should be the basic assumption.  At Jane Street,
essentially all of our files are processed by camlp4.  Extensions like
sexplib and camlp4 are at their best when used fairly pervasively.

That said, I could easily imagine that we would build a single, fast
binary that combined all of our transformations into one, if that
turns out to be a bottleneck.

> Regarding dynloading. First, it's not only that native dynloading does not
> work on all platforms. Worse, not even bytecode dynloading works everywhere
> when it comes to loading external C libraries (which may be required here
> and there). Because of this, we should design without dynloading in mind,
> and at most consider this as an accelarator that is available on certain
> platforms only.
>
> In my opinion, there should be ppx drivers (= something that turns a
> preprocessor into a program suited for -ppx). The simple default driver just
> pipes the parsetree through the preprocessors, and these are running as
> subprocesses. But it should also be possible to link a custom ppx driver for
> the not-so-frequent cases that the preprocessors are needed often. And
> finally, there could also be a ppx driver that dynloads.
>
> I guess the details are not so complicated to figure out (e.g. define a
> module type a ppx preprocessor must satisfy, and have some utilities to
> "make this a program", "make this a dynloadable extension", and "bind
> several ppx processors together").
>
>
>> Currently, 1. is very broken  (natdynlink doesn't work on all platforms
>> and most of the syntax packages do not install their native libraries
>> anyway) and 2. is not very flexible because of lack of tooling, and so very
>> rarely used.
>>
>> So, if we decide to switch to a new preprocessing tool, I guess it's
>> important to get these hings right:
>> * it should be easy enough for external tools to statically build
>> pre-processors.
>> * the new preprocessor should be benchmarked on realistic examples to see
>> how much time we gain vs. camlp4. A good example might be mirage which is
>> using only pa_lwt and cstruct (or xen-api which is using only ocaml-rpc, or
>> core but that maybe too much syntax in one go).
>
>
> Fully agreed to both points.
>
> Gerd
>
>
>>
>> Best,
>> Thomas
>> PS: I'm not speaking about the bootstraping time of camlp4, which is
>> indeed very painful, but which doesn't really affect the end-user once the
>> compiler is installed.
>> _______________________________________________
>> wg-camlp4 mailing list
>> wg-camlp4 at lists.ocaml.org
>> http://lists.ocaml.org/listinfo/wg-camlp4
>>
>>
>
>
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd at gerd-stolpmann.de
> Creator of GODI and camlcity.org.
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------
>
> _______________________________________________
> wg-camlp4 mailing list
> wg-camlp4 at lists.ocaml.org
> http://lists.ocaml.org/listinfo/wg-camlp4