[wg-camlp4] benchmarks

[Török Edwin]

On 02/11/2013 05:44 PM, Gerd Stolpmann wrote:
> Am 11.02.2013 11:45:57 schrieb(en) Gabriel Scherer:
>> On Mon, Feb 11, 2013 at 11:31 AM, Anil Madhavapeddy <anil at recoil.org> wrote:
>> > It should work just fine with ppx, and has the advantage of reducing the number of ocamlfind calls per-file (which is a considerable overhead on large projects, since it has to scan the META
>> directories on every invocation).
>>
>> That's of course rather orthogonal to the present discussion, but have
>> you been in contact with Gerd to see if some caching solution would be
>> possible? Intuitively it feels like ocamlfind could easily maintain a
>> cache in /var/run/blah and only clean it on install/remove commands.
> 
> As a system engineer I think this is overkill. ocamlfind only loads the META files actually needed (it does not "scan"), and loading a file already in the page cache is a matter of microseconds. Even
> when you have 100 packages this should be hardly noticeable. Such a cache can even make things worse, because loading once a large cache can be more costly than a few META files (simply because of the
> data volume).
> 
> What is probably more a problem when you have many packages: This generates many -I paths, and the compiler has to search for modules. I don't know how this is implemented, but if it is the naive way,
> this is an O(n*m) algorithm (n=number of -I directories, m=number of modules to look up).

Is there a reason that ocamlbuild has to spawn the compiler for each file?
Couldn't it use compiler-libs to perform the compilation directly, and re-use some of the cached information (location of modules, parsed interfaces for example)?

Also I've noticed some suboptimal behaviours with oasis+ocamlbuild+ocamlfind, would that new wg-build list be appropriate to discuss such issues (and find solutions)?
Or should I just file bugreports for the individual tools?

For example:
 - ocamlfind returning -I paths in different order (with different -tag), which caused ocamlbuild to think that the "build command" has changed, and thus rebuild files it shouldn't
 - oasis sometimes adds more -I than necessary (if two libraries share a directory)
 - oasis sometimes adds too few -I (causing interface mismatch errors between my src/ and test/ for example)
 - even when there's nothing to rebuild it takes 1-2s to say that


Best regards,
--Edwin

> 
> Gerd
> 
> 
>> (An external process invocation would still make some overhead so in
>> you're case you'd still have to go all the way, but that could improve
>> the threshold at which such ~hacks become interesting.)
>>
>> On Mon, Feb 11, 2013 at 11:31 AM, Anil Madhavapeddy <anil at recoil.org> wrote:
>> > On 11 Feb 2013, at 10:06, Alain Frisch <alain.frisch at lexifi.com> wrote:
>> >>
>> >> For a code base like Jane Street's where many "extensions" have to be used everywhere, I suspect that the "independent" ppx processes might become an issue (or maybe not, if compared to camlp4),
>> but the following solutions would work:
>> >>
>> >> - Statically linking a "big rewriter" called with -ppx.
>> >
>> > This worked well in the case of Mirage; in the monolithic repository, we link together all the camlp4 extensions into a single big one, and invoke that across all files.
>> >
>> > It should work just fine with ppx, and has the advantage of reducing the number of ocamlfind calls per-file (which is a considerable overhead on large projects, since it has to scan the META
>> directories on every invocation).
>> >
>> > There's also the advantage in camlp4 that this checks that syntax extensions do not conflict. This is less of an issue with ppx, thankfully.
>> >
>> > -anil
>> > _______________________________________________
>> > wg-camlp4 mailing list
>> > wg-camlp4 at lists.ocaml.org
>> > http://lists.ocaml.org/listinfo/wg-camlp4
>>
>>
> 
> 
>