[opam-devel] How to manage a package manager (Was: Re: [opam] Build clarification (#1149))

[Anil Madhavapeddy]

On 1 Feb 2014, at 19:27, Fabrice Le Fessant <Fabrice.Le_fessant at inria.fr> wrote:

> Hi,
> 
>  Anil, I am worried that your workflow is not the typical workflow of
> OCaml developers. You are an expert in OPAM, having several OPAM
> repositories and so on. The typical user I imagine has little
> knowledge of OPAM, and expect upgrades of both the repository and of
> OPAM to be as simple as possible. I remember using Cabal to install
> some Haskell software: I would do it once per year, at most, and
> everytime, I was very annoyed to discover some incompatibility that
> would require an update of Cabal itself. Now, I fear to see the same
> problem with OPAM.

Hi Fabrice,

I do agree with you here of course.  I'm just noting from my experience
with running quite a few OPAM-based tutorials in the last year (starting
with the public beta release in ICFP 2012 with Thomas and Yaron, and
subsequently from helping a number of non-OCaml users get started with
Mirage and Real World OCaml, both of which require OPAM), that users
are primarily confused by two things:

- Getting started with OPAM: including utop, their ocamlinit, and all
the other bits and bobs (especially editor integration).

- Broken packages.  Part of the reason for Cabal's irritation are
incorrect package constraints. Now, if you were leave an OPAM install
for a few months and upgrade, then you are using a very untested path
in the solver.  A lot of things have to work *perfectly* to have a
successful package upgrade: the solver has to find a solution, the
uninstall targets have to be correct (or else ocamlfind complains
on the subsequent installation, and the installation must work including
depexts).  If anything goes wrong here, then the user's left to debug
the mess themselves.

I prioritise improving these above anything else to do with OPAM for
my own hacking time.  We've made very significant progress (thanks to
Roberto and the aspcud team) in OPAM 1.1.1, but there is always more to
improve.

For the second point, I must admit I'm a little disappointed by the
lack of feedback to my various posts about doing OPAM bulk builds to
improve the package repository quality.  After the Docker post [1]
(which works on any modern Linux), I also published the logfiles from
the build [2] and triaged a number of bugs [3], but not many other
people have contributed fixes.  I would very much like to get to a 
100% package installation rate, which we can then keep at 100% due to
the ongoing package testing for new pull requests.

[1] http://anil.recoil.org/2013/11/15/docker-and-opam.html
[2] https://github.com/avsm/opam-bulk-logs
[3] https://github.com/ocaml/opam-repository/issues/1304

The question of upgrading OPAM itself has never come up with anyone,
since most users simply used their OS-installed tool to upgrade from
OPAM 1.0 to 1.1.  The only major bug with this approach was because
the release of OCaml 4.01 coincided with OPAM 1.1, and a lot of Mac
users ended up upgrading their system compiler *and* OPAM simultaneously,
which led to complete meltdown of their ~/.opam (see [4])

[4] https://github.com/Homebrew/homebrew/issues/24315

This nasty bug should tell us that upgrades are just very complicated
and need a lot of testing (we tested system compiler updates, and OPAM
version updates, but never together!). I would expect any auto-update
procedure to  at least consider these issues:

- What happens during an upgrade if a newly installed OPAM binary
  inside a ~/.opam requires an upgrade of the OPAM metadata?

- If you do an opam update using the OPAM binary installed within
  the ~/.opam space, how do you avoid a 'text file busy' error
  due to being unable to replace the currently executing binary?
  We see this a lot on Mirage [5] due to the command line tool
  and the libraries being mixed up, as mentioned earlier.

- If the user invokes OPAM without first calling `opam config env`,
  they will use the *old* OPAM with a *new* repository.  This
  is currently a non-issue since there is only one OPAM binary,
  and all sub-commands spawned by it apply the correct env variables.
  The autoupdate scheme proposed will absolutely require that the
  user has correctly set up their `opam config env`, which is
  something that I've observed many users not do (including myself).
  This will thus require support and testing of downgrade.

[5] https://travis-ci.org/mirage/mirage-www/jobs/17859382

>  Even if it's not easy to achieve, because of the risk of breaking
> other OPAM features, I think "transparent update" is definitively
> something the typical user of OPAM would like and find very cool with
> OPAM, and worth adding the feature to the roadmap !

I'm obviously not against adding magical auto-updating, but I hope
that any such proposal will consider the issues i raise above.
Personally, I'm going to spend my limited time working on what
I've observed are the real (but admittedly more boring) problems
with day-to-day OPAM use that I listed above, such as upstream
packaging and improving the repository quality.

More broadly, I do not believe that we should get too dependent on
OPAM and ignore upstream binary packaging. In Xen for example,
the Debian and Fedora packages require binary OCaml packages to
link against, and telling everyone to use OPAM is not an acceptable
answer.  


-anil