[opam-devel] How to manage a package manager (Was: Re: [opam] Build clarification (#1149))

[Roberto Di Cosmo]

[I am moving this discussion away from the BTS and into
 the mailing list, as the issues raised here are of
 general relevance for the future of Opam, and must
 not be lost in the middle of other comments; see
 https://github.com/ocaml/opam/pull/1149 for the original
 discussion
--
Roberto
]

The original issue: what build system should we use to build opam?
Should we cook up an ad-hoc Makefile, use a patched ocamlbuild,
or use OCamlMakefile, or use ocp-build? Is it evil to embark
a build system in the package manager source code?

My quick view of these issues: 

 1) choice of the build system

     we are already plagued with multiple build systems, and one of the reason
     of opam's success is that it allows every package author the freedom to use
     the one he/she prefers/ is more comfortable with.
     Opam is no exception, and opam developers must be able to pick the build system
     they prefer, so I would say, it's their decision to make

 2) embarking a full build system into the source tree of a package

     in general I would find this a very bad idea: one should have a separate
     package for the build system, and have it installed before building anything else.

     Now, what sets opam apart from other OCaml components is the fact that it is
     the *basic* building block for all the other packages in the repository, and
     without a running opam, there is no way to install any other package; this
     is a bootstrap issue, and that's why, for example, opam embarks in src_ext
     the full source of the dose and libcudf libraries, instead of linking to
     the packages for dose and libcudf (that do exist in the repository); so
     in this particular case *only* I personally would not be shocked to see
     some extra stuff in the src_ext directory (especially considering that
     all this will need to work on multiple platforms)

  3) how do we update opam?

     Now, for another important issue, related to the special status of opam
     as *the* package manager, and how we should manage it (hence the title
     of this message).

     I am a bit unhappy with the current state of affairs, where new
     versions of opam appear with incompatible changes in the repository format,
     hence different versions of the repositories: people using old versions
     of opam get lost in a limbo-like repository with non updated packages
     until they realise they need to recompile a new version of opam
     to see the light again.

     We need to find a way to have opam become a package like the others
     (with the exception of the embarked code for bootstrapping reasons),
     and have it become part of opam-repository... the main difficulty
     lies in the fact that its binaries need to go in special places,
     and not in the 4.00/3.12 etc. dirs (correct if I'm wrong)

     Once this is done, a sensible upgrade path would be:

     Time T : I have opam version n installed, accessing repo version n

     Time T+delta: some major changes make the old opam unable to use the new
                repository format, and the repository migrates to a new place;
                a new version n+1 of opam is released, embedding logic to access
                the new repository;
                opam version n+1 is added also to repo version n

     Sometimes later: the user of opam n runs an update, sees that a new
                      version of opam is available, and upgrades it...
                      he now has the new opam, that updates the references
                      to the new repository, and keeps following the evolution
                      of the repository, with no pain
  
      I know that maybe at some point the repo format will stabilise, but this
      simple process will make sure that the lambda-users (private joke: that's
      equivalent to "jow of the street" in french, but also hints at functional
      programmers :-)) will not get lost, no matter the changes.
  
      Any chances to see this happen soon? A starting point would be to summarize
      on this list the difficulties that prevent opam from being a "normal"
      package

Cheers

--
Roberto