[opam-devel] On self-upgrade

Louis Gesbert louis.gesbert at ocamlpro.com
Thu Jun 26 09:50:59 BST 2014

Le mercredi 25 juin 2014, 20:22:17 Anil Madhavapeddy a écrit :
> On 25 Jun 2014, at 20:15, Thomas Gazagnaire <thomas at gazagnaire.org> wrote:
> > Thanks for you message Louis, it's useful to get some insight of you design.
> > 
> >> We've had all sorts of problems in Mirage with having the tool (opam package 'mirage') invoke OPAM to satisfy package dependencies, resulting in OPAM trying to recompile Mirage itself (due to a depopt showing up), but then resulting in a failure due to the classic 'text file busy' error when trying to move the old binary out of the way.

Yes, that's the first thing I tested: renaming the file is normally enough, and `install --backup` takes care of that correctly. As a side note, it seems to be cmdliner that makes the exe "busy", don't know why, but I noticed I couldn't anymore `make install` my running program once I started using it.

> >> Is there an RT test that goes through the above sequence to simulate an upgrade and downgrade?

Not yet.

> >> I really fear that this route will cause us to ignore system/OS packaging, which is really important to keep up-to-date for a package manager.
> >> The self-upgrade route from within OPAM should never be the primary way to obtain the latest version of the package manager.

I see. Not at all my intent, and I for one prefer to always go through my system package manager when possible -- Debian is still a quite good quality insurance. I especially don't like when I am left no choice but to enter n steps of self upgrading __after__ installation (hello Steam ! Well I guess when you're an evil DRM engine it makes more sense. Even binary blobs in this case).

So I intended this more as a way to address concerns like "I want the latest OPAM version __now__, what do I do ?" and test new versions a little more widely, never thought this could replace packaging, or even become mandatory. Well then, Anil is the one doing most of the packaging work so I don't want to insist too much :) 

> > Self-upgrade is indeed kind of simple when you have only one channel to get the software you are updating. Louis, do you have any idea in mind on how this will interact with system upgrade (for instance someone doing `brew update` which will install a newer/incompatible version of your self-upgraded package) ?

Quite simply: self-upgrade only lives within an OPAMROOT, system packages live in $PATH, so they won't ever override each other. You always run from the PATH, and opam will pick the latest version of the two. There is a warning if you've got an opsolete self-upgrade installed.

> > I fear that this feature will force us to support all kind of tricky upgrade scenario (1.2 -> 1.3 is ok, but what about 1.2 -> 2.0, or 1.2 -> pin -> 1.1 ?). It's easy to limit these kind of scenario when you control the update channels. When you don't it can quickly become a nightmare.

Not sure I get you exactly, but the current behaviour is pretty strict anyway: OPAM will refuse to load an OPAMROOT when its version ($OPAMROOT/config) is newer than itself, printing a message that it needs to be upgraded. So you're not allowed to downgrade, and people getting in this situation are supposed to have an idea how they got there.

> Yeah, it's worth noting that this did indeed horribly break in the past (the system compiler upgrade from 4.00.1 to 4.01.0 and OPAM from 1.0 to 1.1 didn't work if you did a `brew update` and upgrade both simultaneously).  Upgrading in general is a very tedious, err-prone process that is usually poorly tested, with bugs only biting you a year down the line on release+1 :-)
> This might be a little radical, but it's worth considering having a completely separate directory namespace for repo version bumps, and just recompile all the packages on upgrade (it's a very rare situation, after all).
> ~/.opam/1.1/<...>
> ~/.opam/1.2/<...>

Why not -- but getting a different universe altogether in case of a downgrade could get even more confusing, so I may be in favor of a complete backup with an easy rollback command (stg like "on startup, if $OPAMROOT/config#opam-version > $self-version && exists $OPAMROOT/backups/$self-version then <print error and propose a (temporary ?) rollback to the backed-up state>")

Actually the thing that's still a little clunky with the current design is that an opam package is not the perfect fit for what we are wanting to do: self-upgrade shouldn't be recompiled in case its dependencies are, and certainly shouldn't be removed if they are -- but that's the build-dep problem. Other packages also shouldn't depend on it, but mostly, it doesn't really make sense to have it belong to a given switch only. That's an issue that has already been raised by people wanting a cross-switch binary directory to install tools though, e.g. for ocp-indent.

My current opinion on this is that the feature in itself is OK, but we should be quite careful how and to whom we advertise it. Certainly not in the Quick_install, possibly not even in the FAQ ; just checking how you feel about it.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/opam-devel/attachments/20140626/10b5f8a8/attachment.html>

More information about the opam-devel mailing list