[opam-devel] opam and "So you want to write a package manager"

[Louis Gesbert]

Indeed, a very worthwile read, thanks for linking! I pointed the author to 
Mancoosi, too, which he was glad to hear about.

Per-project dependency handling has been asked for quite a few times, and this 
is indeed something we should support. It complements and doesn't replace the 
global ~/.opam though.

Looking upwards to a `.opam` directory and use it as opam root would be 
straightforward to implement (well, I'd just have to copy-paste some code from 
ocp-indent). But I don't think it is a good idea:
1. it duplicates lots of stuff, including synchronising repositories, caches, 
and compiling OCaml
2. does it replace ~/.opam or add to it ?
3. how about `opam config env` ? Should you ensure to run it after you cd to 
your project ? Sounds dangerous, but it might be handled by the build system.
4. and what about tools such as merlin ? You surely won't install them within 
every project...

Our opam-manager¹ provides an experimental solution to 2. and 3. (and, 
partially, 1.). The idea is to wrap around installed commands (ocamlc, etc.) 
and redirect them to the correct switch, as determined by e.g. a project-local 
configuration file -- alleviating the need for `eval $(opam config env)`.

This wouldn't solve 3., but I would be more in favor of allowing project-local 
_switch trees_ while keeping the global opam root and automatically choosing 
the current switch depending on CWD. This at least allows opam to know about 
the other switches around, and may allow opam-user-setup to handle 4. through 
minor improvements (it should already work for compiler-version-insensitive 
tools). It also factorises caching and updates.

Since opam 2.0² is planned to allow selecting the repositories per-switch, 
this shouldn't be a problem. We may also be able to share compiled artifacts 
between switches at some point, which is a different issue.

> If I want to use this workflow with opam today, what are my options
> interface-wise? Use `opam switch export` and share/version the
> resulting state? In the current world of user-global package state,
> this export is likely to contain packages that are completely
> unrelated to the current project. Would it be possible to export only
> the recursive dependencies of a given opam file, and more generally
> have easier way to manipulate version state information?

For what Sam calls "lock files", indeed, there is nothing built-in, but you 
could:
- use `switch export`, but in this case you get also unrelated installed 
packages
- save the output of `opam list --installed --rec --required-by <pkg>`, and 
pass that to `opam install` (unfortunately, the --short format currently 
excludes versions). This won't, of course, handle pinned packages.

A partial export function limited to what is required by a given package would 
be very nice for this, I guess. We could imagine an opam command that picks up 
the project-local opam file, possibly a local export, and builds a local switch 
based on that. The closest we currently have is `opam pin add .` -- possibly 
with an `opam switch my-project-switch --alias-of ...`.

Note that we are somehow providing a way to do per-switch dep trees, although 
completely manually, through the way we handle multiple switches.

Another big question is that we otherwise need build systems to be opam-aware. 
Without interfering when they are run from within opam, of course. Guess I'll 
get back to working on opam-as-build-system ;)

Best,
Louis


¹ https://github.com/OCamlPro/opam-manager
² yes, it's no longer planned as 1.3, since the repository format and 
structure of ~/.opam are being changed quite a bit -- no more .comp files.


Le samedi 13 février 2016, 14:28:06 Gabriel Scherer a écrit :
> Hi all,
> 
> Sam Boyer has a very interesting piece on the requirements for package
> managers at
>  
> https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d
> 9527
> 
> It's a bit long (but I think it's worth the read if you are into
> package managers), but I think that the central point that is of opam
> 
> relevance is the following distinction:
> > OS/system package manager (SPM): this is not why we are here today
> > Language package manager (LPM): an interactive tool (e.g., `go get`) that
> > can retrieve and build specified packages of source code for a particular
> > language. Bad ones dump the fetched source code into a global,
> > unversioned pool (GOPATH), then cackle maniacally at your fervent hope
> > that the cumulative state of that pool makes coherent sense.
> > Project/application dependency manager (PDM): an interactive system for
> > managing the source code dependencies of a single project in a particular
> > language. That means specifying, retrieving, updating, arranging on disk,
> > and removing sets of dependent source code, in such a way that collective
> > coherency is maintained beyond the termination of any single command. Its
> > output — which is precisely reproducible — is a self-contained source
> > tree that acts as the input to a compiler or interpreter. You might think
> > of it as “compiler, phase zero.”
> It seems clear to me that the model of "one packaging environment per
> project", rather than "one packaging environment per user-scoped
> state/switch", is winning in most programming languages (Rust and Go
> are mentioned in the post, but I'm observed from a distance the same
> moves in the Python, Ruby and Javascript communities). And for a good
> reason: it is simply a better model for developers, as it allows to
> have development environments scoped at the conceptually adequate
> level, which is the project level -- there is no reason especially in
> languages that do not allow a given development to include two
> different versions of the same package (OCaml is guilty of this as
> well).
> 
>  It is in theory possible to do this with OPAM by using project-local
> root directories, and in fact this is the approach that Guillaume
> Claret, part of the Coq community and influenced by the Ruby packaging
> practices, has long advertised:
>   http://coq-blog.clarus.me/use-opam-for-coq.html
> 
> I have already mentioned this thorny question on opam-devel,
>   Subject: One (or more) opam root per projet?
>   Date: Mon Jan 26 20:58:29 GMT 2015
>   http://lists.ocaml.org/pipermail/opam-devel/2015-January/000918.html
> 
> But I think it would be worth a re-discussion. I think that the
> writing is on the wall, and that language-specific package managers
> should strive to support project-local configuration. As said in the
> previous discussion, having a user-scoped cache is still useful; but I
> think it should be possible to use OPAM in a mode where this
> resource-usage-motivated sharing is transparent to the developer,
> which really has access to per-project packaging environments.
> 
> If nothing else, I think we should consider user interface ways to
> make the already-possible workflow of having a project-local opam root
> easier to manage. For example, we could decide that, by default, opam
> walks the filesystem hierarchy towards the root and picks the first
> directory named .opam as opam root. This would gracefully extend the
> policy of picking ~/.opam by default.
> 
> Another topic touched in this blog post is determinism of packaging
> version choices. I think that OPAM's version handling is of very high
> quality compared to some other packaging systems, but this blog post
> argue that having good version bounds in a package description does
> not negate the use for a more precise "lock file" that specifies exact
> versions for all transitive dependencies of a project, that can be
> shared between developers (and even versioned in the version control
> system) to improve reproducibility. This topic has been central to the
> development of Haskell's new tool "stack", which haskellers are raving
> about.
> 
> If I want to use this workflow with opam today, what are my options
> interface-wise? Use `opam switch export` and share/version the
> resulting state? In the current world of user-global package state,
> this export is likely to contain packages that are completely
> unrelated to the current project. Would it be possible to export only
> the recursive dependencies of a given opam file, and more generally
> have easier way to manipulate version state information?
> 
> I know that Louis already has his plate full for 1.3, but I hope that
> some of those questions could influence the plans for 1.4.
> _______________________________________________
> opam-devel mailing list
> opam-devel at lists.ocaml.org
> http://lists.ocaml.org/listinfo/opam-devel