[opam-devel] How to manage a package manager (Was: Re: [opam] Build clarification (#1149))

[Anil Madhavapeddy]

On 1 Feb 2014, at 15:12, Roberto Di Cosmo <roberto at dicosmo.org> wrote:

> On Sat, Feb 01, 2014 at 02:53:06PM +0100, Anil Madhavapeddy wrote:
>> On 1 Feb 2014, at 14:40, Roberto Di Cosmo <roberto at dicosmo.org> wrote:
>>> 
>>>> This is a very tricky interaction to get right, 
>>> 
>>> Really? I would like to know more about this; please add the factual elements
>>> to #1150
>> 
>> I think the overall problem element is obvious: the opam binary within the
>> opam installation will be picked up very suddenly during package installation
>> and upgrades.
>> 
>> Rather than me listing why every single feature of OPAM would potentially break
>> (consider the 'repo upgrade' step that OPAM 1.0 -> 1.1 does), I would again
>> suggest that you provide a strong motivation for why you want the feature in the
>> first place.
> 
> Sorry, I really do not understand what you mean. On my side, I do believe I made
> my point clear enough, and I see no point in continuing this exchange.

Let me go back to your original point, which I agree with:

> That's why one needs a clear, and simple, mechanism to ensure a smooth
> transition when incompatibilities arrive (and there will be more, and
> unexpected ones, you can bet on it). 

My point is that the user originally obtains their version of OPAM via their
OS package manager (Homebrew, apt, BSD ports, Yum, etc).  All of these managers
provide a mechanism to warn the user of any upgrade steps that need to happen
when upgrading their distribution, and so is an ideal way to update OPAM
itself in the event of incompatible future changes.

In contrast, `opam update` is part of a *development* workflow, not a system
upgrade path.  I do that when I want to recompile some part of my working
trees to a new package set, and often follow it up with an upgrade (via -u).

If you mix these things up, then it violates a lot of expectations from the
user side, and can lead to the annoying situation where you are forced to deal
with a distribution upgrade when you were expecting to just alter some local
code.  Furthermore, it is quite common to destroy and reinitialize ~/.opam
or set a new OPAMROOT to test for a clean install, and this would subsequently
downgrade OPAM (or worse, cause an error if your shell path cache becomes
invalidated).

We've already seen this problem in action with a number of tools such as
Merlin or ocp-index that really ought to be installed system-wide and not
within OPAM (since a code update could result in the editor tool being
recompiled or downgraded).  Another example is the Mirage command-line tool,
which uses libraries shared with development code.

To summarise, this feature continues to blur the divide between a system tool
(OPAM, Merlin, ocp-index) from a development tool, and should not be
encouraged. I would like to see better support for managing system tools
within OPAM, but I think prioritising upstream integration of them should
be encouraged first.  OPAM is a developer supplement to make source-based
development easier, not a complete replacement for an existing OS package
manager.

I hope that's clearer.  There are many negative reasons for upgrading OPAM
using OPAM, and I've still not seen a good reason for not focussing on
better upstream package manager (and hence external dependency) integration
before anything else.

-anil