[opam-devel] opam and "So you want to write a package manager"

[Gabriel Scherer]

Hi all,

Sam Boyer has a very interesting piece on the requirements for package
managers at
  https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

It's a bit long (but I think it's worth the read if you are into
package managers), but I think that the central point that is of opam
relevance is the following distinction:

> OS/system package manager (SPM): this is not why we are here today
> Language package manager (LPM): an interactive tool (e.g., `go get`) that can retrieve and build specified packages of source code for a particular language. Bad ones dump the fetched source code into a global, unversioned pool (GOPATH), then cackle maniacally at your fervent hope that the cumulative state of that pool makes coherent sense.
> Project/application dependency manager (PDM): an interactive system for managing the source code dependencies of a single project in a particular language. That means specifying, retrieving, updating, arranging on disk, and removing sets of dependent source code, in such a way that collective coherency is maintained beyond the termination of any single command. Its output — which is precisely reproducible — is a self-contained source tree that acts as the input to a compiler or interpreter. You might think of it as “compiler, phase zero.”

It seems clear to me that the model of "one packaging environment per
project", rather than "one packaging environment per user-scoped
state/switch", is winning in most programming languages (Rust and Go
are mentioned in the post, but I'm observed from a distance the same
moves in the Python, Ruby and Javascript communities). And for a good
reason: it is simply a better model for developers, as it allows to
have development environments scoped at the conceptually adequate
level, which is the project level -- there is no reason especially in
languages that do not allow a given development to include two
different versions of the same package (OCaml is guilty of this as
well).

 It is in theory possible to do this with OPAM by using project-local
root directories, and in fact this is the approach that Guillaume
Claret, part of the Coq community and influenced by the Ruby packaging
practices, has long advertised:
  http://coq-blog.clarus.me/use-opam-for-coq.html

I have already mentioned this thorny question on opam-devel,
  Subject: One (or more) opam root per projet?
  Date: Mon Jan 26 20:58:29 GMT 2015
  http://lists.ocaml.org/pipermail/opam-devel/2015-January/000918.html

But I think it would be worth a re-discussion. I think that the
writing is on the wall, and that language-specific package managers
should strive to support project-local configuration. As said in the
previous discussion, having a user-scoped cache is still useful; but I
think it should be possible to use OPAM in a mode where this
resource-usage-motivated sharing is transparent to the developer,
which really has access to per-project packaging environments.

If nothing else, I think we should consider user interface ways to
make the already-possible workflow of having a project-local opam root
easier to manage. For example, we could decide that, by default, opam
walks the filesystem hierarchy towards the root and picks the first
directory named .opam as opam root. This would gracefully extend the
policy of picking ~/.opam by default.

Another topic touched in this blog post is determinism of packaging
version choices. I think that OPAM's version handling is of very high
quality compared to some other packaging systems, but this blog post
argue that having good version bounds in a package description does
not negate the use for a more precise "lock file" that specifies exact
versions for all transitive dependencies of a project, that can be
shared between developers (and even versioned in the version control
system) to improve reproducibility. This topic has been central to the
development of Haskell's new tool "stack", which haskellers are raving
about.

If I want to use this workflow with opam today, what are my options
interface-wise? Use `opam switch export` and share/version the
resulting state? In the current world of user-global package state,
this export is likely to contain packages that are completely
unrelated to the current project. Would it be possible to export only
the recursive dependencies of a given opam file, and more generally
have easier way to manipulate version state information?

I know that Louis already has his plate full for 1.3, but I hope that
some of those questions could influence the plans for 1.4.