[opam-devel] CommonML: An opinionated build/package/develop workflow on top of CommonJS

[Jordo]

I think you've made the right design decision by keeping package management separate from building. CommonJS does the same - it's merely a way to specify dependencies for what should be installed on disk. The npm command line tools merely solve and install those files into local directories. Coincidentally, nodeJS knows how to reason about the specific way npm installs files onto disk (it just looks in the node_modules directory to implement the namespacing - which is what CommonML does at build time).

To keep the separation clean, while also implementing proper namespacing, I would suggest forming a spec for what is required to be "easily namespaceable", and how the namespacing is to be interfaced with - something similar to what I've done in CommonML. You could imagine something like a subset of OPAM packages that are marked as such so that you can browse OPAM for "no hassle packages" that are guaranteed to Just Work.  If it catches on (and I believe it should), it could become the norm. Someone should write a scalable version of CommonML built on top of ocamlbuild or something else that does a good job of incremental builds while working with this No Hassle convention. A first version that just implements the namespacing would probably only take a day for an expert.

The specific way I've implemented namespacing is actually much simpler and more sustainable than the nodeJS module lookup convention. There are issues with how nodeJS does it (relative paths) and I didn't just want to blindly mimic their convention without questioning it. The namespacing I've proposed with CommonML applies our findings from large scale development at Facebook - an environment with massive automated refactors, thousands of daily commits, and hyper-granular sharing/modularization across multiple teams, and open source contributors. But it's absolutely not revolutionary - quite the opposite - it's the simplest possible convention imaginable.

Would anyone be interested in helping me write an ocamlbuild plugin that accomplished the same? I'd be willing to bet one already exists.

Jordan

> On Feb 27, 2015, at 8:25 PM, Louis Gesbert <louis.gesbert at ocamlpro.com> wrote:
> 
> The way your included build system integrates the build of the dependencies with the build of the tool is also very interesting, and should raise some thought in build system design too. Incidentally, that's kind of what we do (by hand) for OPAM bootstrapping, and I know several examples of that for in-house project builds. Of course, as you pointed out, this requires some discipline in package structure -- OPAM was designed to be the most tolerant possible on that front, to keep package management and project building issues separate (one thing at a time) -- but having some common project structure emerge would definitely be a good thing.
> 
>> - Jordan W, 24/02/2015 00:28 -
>> Since there is a lot of interest in OCaml from web frontend communities, I
>> thought it would be useful to imagine what the ideal development flow for
>> this audience would look like. I know there's a ton of progress being made
>> on documentation, and build systems, but I thought I would explore the
>> problem from the perspective of a frontend developer, which means starting
>> with the tooling that they are familiar with. One common tool is
>> `CommonJS/package.json`, which is a way to model and organize dependencies
>> using a single JSON file per package. The npm command line tools allow you
>> to install files from disk based purely on these package.json files.
>> 
>> I created a proof of concept called CommonML, which lets developers use
>> their familiar CommonJS workflow with OCaml:
>> https://github.com/jordwalke/CommonML
>> 
>> I also took used it as an opportunity to explore what can be done when
>> there are opinionated conventions in place. If you have a predictable
>> project structure, how can that benefit us? In this case, I created an
>> automatic docs builder (with nice styling) and also automatically generate
>> IDE autocomplete support for all your dependencies (and your project's
>> internal modules).
>> 
>> 
>> I hope there is at least something we can take away from it that helps
>> inform the design of OPAM and related tools.
>> 
>> 
>> One nice aspect is that with `CommonJS`, there needn't be an authoritative
>> package service. Your package.json file can point to arbitrary git URLs if
>> you like.  (Note: The npm command line tool is *not* the npm package
>> service - they are made by the same organization but one may be used
>> without the other). However, this prototype I've built does allow you to
>> host OCaml code on npm and depend on it.
>> 
>> By far the nicest thing about developing with `CommonJS` is that you don't
>> have to think about module namespace collisions. There is Just One Way to
>> namespace modules/packages. This prototype automatically sets up a similar
>> namespacing convention for OCaml modules. It's not flexible, and you can't
>> customize it, but it always works. It uses module aliases (thank you to Leo
>> White for helping me come up with the build conventions).
>> 
>> Another thing I like about the `CommonJS` workflow is that developing
>> packages locally is virtually the same as developing against remote
>> dependencies. (`npm link` is much like `opam pin` I'm told). When you `npm
>> install` dependencies, everything is pulled down into a local
>> sandbox(node_modules directory) instead of being installed globally by
>> default. If you want to see what versions your local package is seeing,
>> just traverse the file system! If you want to reinstall, just delete the
>> node_modules directory and then `npm install` again. I believe there is a
>> way to get it to use a global package cache so the node_modules might
>> contain symlinks to those shared packages - but that's just an
>> optimization. There isn't any notion of building in `npm`, so there
>> wouldn't be a build cache I believe.
>> 
>> In my quick prototype, every dependency must be compiled at least once for
>> the root level project that you are building. This ends up being nice in
>> cases where the build flags (such as -g) must be in effect for the
>> compilation of all my dependencies - relying on the build flags that you
>> *installed* the package with will bite you. But of course, the rebuilding
>> approach can end up being super slow. Still, the incremental build times
>> are *totally* reasonable since it does try to do some basic incremental
>> compilation. I would have used ocamlbuild which probably does a much better
>> job, but I needed to write my own totally custom operations in order to get
>> the auto-namespacing (with the help of Leo White). I wasn't sure how to do
>> that with ocamlbuild, but if I could, I imagine the incremental compilation
>> times would be way better.
>> 
>> Either way, for most of the work I do (developing libraries with many other
>> small libraries as dependencies) - I could see a development flow like this
>> being a worthwhile goal, especially if it makes OCaml much more comfortable
>> for a *huge* set of developers. `CommonJS` is likely becoming the most
>> popular development flow. It's just a hacky proof of concept, but it was
>> fun.