[opam-devel] OPAM 2.0 features

[David Allsopp]

Louis Gesbert wrote:

> Le jeudi 26 mai 2016, 10:36:48 David Allsopp a écrit :

> > Hi (Louis mainly, I expect!),

> >

> >   I've rebased work I've been very slowly doing onto the OPAM 2.0

> alpha

> > release and have been starting to get opam switch to work. There are a

> > few things which I need to be able to do for this, and I'm wondering

> > if there any opinions (or even existing features I've missed) as to

> > mechanisms...

>

> Great! All feedback is very welcome at this stage.



Biggest feedback at the moment is that compilers-as-a-package is a fabulous innovation – I had originally expected to need to add lots of horrid Windows (and OCaml, therefore)-specific stuff to the OPAM program, but in fact all I've had to do is add generic features to OPAM and then handle the rest in the package itself, which is great!



> > 1. Downloading a package from several tarballs

> >   I've not finished delving into the full details, but I think I'm

> > correct that the url file / section can only specify a single archive,

> > even if it can be made available by multiple transports? Bootstrapping

> > OCaml on Windows requires two tarballs, so I was going to extend the

> > url system to allow multiple tarballs to be specified (with control

> > over where in the overall tree they are extracted), rather than

> > hackily including the second one in files and extracting it using a

> build command.

>

> Some other compilers, e.g. metaOCaml, have the same requirement, since

> they consist of the official OCaml tarball plus a huge patch, so I have

> added a way to handle this.

>

> The `url` section or file still is a single source, and the commands are

> executed from its root (after extraction if necessary). But there is now

> also an opam field `extra-sources:` that you can use to download more

> files, unextracted, to the build directory. The format is:

>

>     extra-sources: [ [ "URL" { "target-filename" } "MD5" ] ... ]

>

> The "target-filename" part being optional. See for example

> http://opam.ocaml.org/2.0~dev/packages/ocaml/ocaml.4.01.0+BER/opam



Ta, I came across it subsequently too! I've extended it (or rather, will shortly extend it), to allow it to refer to the URL of another package, as the ocaml package needs to pick up the sources of the flexdll package installed just before it in order to bootstrap correctly.



> > 2. Specifying package variables at opam switch install

> >   I've extended the ocaml package variables to include three new ones:

> > ocaml-arch (which is exactly the architecture determined by OCaml's

> > configure script), ocaml-cc (which is "cl" for a Microsoft C compiler

> > based OCaml or "cc" for the rest of the world), and ocaml-libc (which is

> "msvc"

> > for the four native Windows OCaml ports and "libc" for the rest of the

> > world).

> >

> >   At OPAM switch time, one selects which Windows port you get by

> > specifying these three variables. So, for example, mingw64 is

> > ocaml-arch=amd64, ocaml-cc=cc, ocaml-libc=msvc; msvc32 is

> > ocaml-arch=i386, ocaml-cc=cl, ocaml-libc=msvc; cygwin64 is

> > ocaml-arch=amd64, ocaml-cc=cc, ocaml-libc=libc.

> >

> >   My question is how to specify these - the concept of "architecture",

> > "C compiler type" and "C runtime library" is generic enough that it

> > doesn't feel too bad to have opam switch --cc=cl --arch=amd64 4.03.0

> > and have the package initialise ocaml-arch, etc. based on those

> > globally available variables. However, is there any plan (or existing

> > feature) for a more generic way of specifying package input variables?

> > (discussed in https://github.com/ocaml/opam/issues/2247).

>

> a) for the _output_ variables, you can define them in your `PKG.config`

> variable (automatically installed by opam if present in your build dir).

> Note that, at the moment, variables defined by the compiler package get

> exported as global variables (so you can access e.g. `ocaml-version`

> anywhere), but there are limitations due to this and it is likely to

> disappear from the final release (see

> https://github.com/ocaml/opam/issues/2537 for details). You can always

> access them through `ocaml:VAR` inside the script parts though.

>

> b) nothing yet for _input_ variables, but #2247 is still something I need

> to work on. You can define global variables through `opam config set`,

> though, so an option in the meantime could be to use `opam switch install

> --empty`, set the variables, then install the compiler (the option to

> install and mark as compiler is missing at the moment, though, for that

> you'll have to edit

> `SWITCH/.opam-switch/switch-state`)



I've gone with the command line options for now (with an error check to ensure that they're only given when installing a switch). At the moment, they define three global variables switch-cc, switch-libc and switch-arch which are only available during the switch installation, so if they want to be kept, the compiler package has to capture them in its own .config file. ocaml.system remains a horrible hack under this mechanism (the script identifies that it's in a switch named "system" and uses that fact to probe switch-cc, switch-libc, and switch-arch...)



It's not particularly pretty, but it does seem to work in the absence of something more general for now. In particular, I've restructured the base packages and added a package called compiler whose purpose is to store variables (in particular, compiler:o, compiler:so and compiler:a giving important file extensions for object files, etc., which are necessary for writing generic package specifications). The only bug I had to fix with that so far is to thread the config files correctly through parallel_apply (before they were only re-loaded at the end of the installation), but there is a comment in the code that you already aware of problems there!



> > 3. Upgrading dependent packages for the root

> >   Any Windows OCaml installation will depend upon FlexDLL objects

> > compiled with the specific C compiler combination. Although in

> > principle these could be shared between multiple switches (the object

> > files vary according ocaml-cc, ocaml-libc and ocaml-arch; not

> > ocaml-version), it makes sense to compile these two object files

> > per-switch, rather than introduce yet another installation layer.

> >

> >   This means that in addition to the base packages, the Windows OCaml

> > package depends on a flexdll package. Is there any philosophical

> > reason for that not to be part of opam upgrade? Obviously, one doesn't

> > want to be trying to upgrade the ocaml package, but why lock its

> > dependencies at a given version? It would be quite nice if a new

> > version of flexdll is released to be able to have opam upgrade notice

> > that flexdll can be upgraded, determine that the root package must

> > therefore be recompiled and so automatically recompile the entire

> switch.

> >

> >   Is it just for philosophical reasons that the dependencies of the

> > root package are not considered for upgrade or is there a technical

> > consideration?

>

> Yes, just philosophical, there is no constraint here besides the arbitrary

> "I don't want to recompile the compiler, ever".



I've spent the last couple of days working through this, and I think it's definitely worth relaxing that restriction. I've altered opam switch so that only the root package is the compiler package – so only ocaml.version is in the compiler section of switch-state. Have a look at this transcript:


[opam:system] [git:windows] C:\DRA\opam>opam switch 4.02.0 --arch=x64

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[flexdll.0.31] http://alain.frisch.fr/flexdll/flexdll-0.31.tar.gz downloaded
[flexlink.0.31] http://alain.frisch.fr/flexdll/flexdll-0.31.tar.gz downloaded
[ocaml.4.02.0] http://caml.inria.fr/pub/distrib/ocaml-4.02/ocaml-4.02.0.tar.gz downloaded

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
✶  installed base-bigarray.base
✶  installed base-threads.base
✶  installed base-unix.base
✶  installed compiler.base
✶  installed flexdll.0.31
✶  installed ocaml.4.02.0
✶  installed flexlink.0.31
Done.

At this stage, that's a mingw64 native Windows port installed with flexlink 0.31 (several versions behind, but the current version when 4.02.0 was released). Now opam upgrade correctly offers to recompile everything (I'm going to tweak it so that it will only do it explicitly – i.e. opam upgrade flexdll):

[opam:4.02.0] [git:windows] C:\DRA\opam>opam upgrade
The following actions will be performed:
  ↑  upgrade   flexdll  0.31 to 0.34
  ↕  recompile ocaml    4.02.0                [uses flexdll]
  ↑  upgrade   flexlink 0.31 to 0.34
===== ↕  1   ↑  2 =====
Do you want to continue ? [Y/n] n

The really cool thing – principally for Alain Frisch (!) - is that it means you can pin the packages. So first I can pin the linker (the flexlink package depends on ocaml, not the other way around):

[opam:4.02.0] [git:windows] C:\DRA\opam>opam pin add C:\DRA\flexlink --kind=path --yes
[flexlink.0.31] file://C:\DRA\flexlink/ synchronized
flexlink is now pinned to file://C:\DRA\flexlink (version 0.35)

The following actions will be performed:
  ↑  upgrade flexlink 0.31 to 0.35*

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[flexlink.0.35] file://C:\DRA\flexlink/ already up-to-date

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Ø  removed   flexlink.0.31
✶  installed flexlink.0.35
Done.

but then I can also pin flexdll:

[opam:4.02.0] [git:windows] C:\DRA\opam>opam pin add C:\DRA\flexdll --kind=path --yes
[flexdll.0.31] file://C:\DRA\flexdll/ synchronized
flexdll is now pinned to file://C:\DRA\flexdll (version 0.35)

The following actions will be performed:
  ↑  upgrade   flexdll  0.31 to 0.35*
  ↕  recompile ocaml    4.02.0                [uses flexdll]
  ↕  recompile flexlink 0.35*                 [uses flexdll]
===== ↕  2   ↑  1 =====

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[ocaml.4.02.0] http://caml.inria.fr/pub/distrib/ocaml-4.02/ocaml-4.02.0.tar.gz downloaded
[flexdll.0.35] file://C:\DRA\flexdll/ already up-to-date
[flexlink.0.35] file://C:\DRA\flexlink/ already up-to-date

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Ø  removed   flexlink.0.35
[WARNING] Directory C:\Users\DRA\Documents\.opam\4.02.0\lib\ocaml is not empty, not removing
Ø  removed   ocaml.4.02.0
Ø  removed   flexdll.0.31
✶  installed flexdll.0.35
✶  installed ocaml.4.02.0
✶  installed flexlink.0.35
Done.

then add ocamlfind (tweaked slightly as at present it doesn't explicitly depend on the ocaml package – that presumably will be part of a big repository conversion when OPAM 2 is actually released):

[opam:4.02.0] [git:windows] C:\DRA\opam>opam install ocamlfind --yes
The following actions will be performed:
  ✶  install conf-pkg-config 1.0              [required by conf-ncurses]
  ✶  install conf-m4         1                [required by ocamlfind]
  ✶  install conf-ncurses    1                [required by ocamlfind]
  ✶  install ocamlfind       1.6.2
===== ✶  4 =====

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[ocamlfind.1.6.2] http://download.camlcity.org/download/findlib-1.6.2.tar.gz downloaded

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
✶  installed conf-m4.1
✶  installed conf-pkg-config.1.0
✶  installed conf-ncurses.1
[WARNING] C:\Users\DRA\Documents\.opam\4.02.0\bin\safe_camlp4 is a script; the command won't be available
✶  installed ocamlfind.1.6.2
Done.

and then I can unpin flexdll (aspcud of course identifies the upgrade from 0.31->0.34) and watch the entire switch recompile itself again:

[opam:4.02.0] [git:windows] C:\DRA\opam\src>opam pin remove flexdll --yes
flexdll is no longer pinned to file://C:\DRA\flexdll (version 0.35)
The following actions will be performed:
  ↓  downgrade flexdll   0.35 to 0.34
  ↕  recompile ocaml     4.02.0               [uses flexdll]
  ↕  recompile ocamlfind 1.6.2                [uses ocaml]
  ↕  recompile flexlink  0.35*                [uses flexdll]
===== ↕  3   ↓  1 =====

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[flexdll.0.34] http://alain.frisch.fr/flexdll/flexdll-0.34.tar.gz downloaded
[ocaml.4.02.0] http://caml.inria.fr/pub/distrib/ocaml-4.02/ocaml-4.02.0.tar.gz downloaded
[flexdll.0.35] file://C:\DRA\flexdll/ already up-to-date
[flexlink.0.35] file://C:\DRA\flexlink/ already up-to-date

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Ø  removed   flexlink.0.35
Ø  removed   ocamlfind.1.6.2
[WARNING] Directory C:\Users\DRA\Documents\.opam\4.02.0\lib\ocaml is not empty, not removing
Ø  removed   ocaml.4.02.0
Ø  removed   flexdll.0.35
✶  installed flexdll.0.34
✶  installed ocaml.4.02.0
✶  installed flexlink.0.35
[WARNING] C:\Users\DRA\Documents\.opam\4.02.0\bin\safe_camlp4 is a script; the command won't be available
✶  installed ocamlfind.1.6.2
Done.

then, finally, I can remove the flexlink pin:

[opam:4.02.0] [git:windows] C:\DRA\opam\src>opam pin remove flexlink --yes
flexlink is no longer pinned to file://C:\DRA\flexlink (version 0.35)
The following actions will be performed:
  ↓  downgrade flexlink 0.35 to 0.34

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
[flexlink.0.34] http://alain.frisch.fr/flexdll/flexdll-0.34.tar.gz downloaded

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Ø  removed   flexlink.0.35
✶  installed flexlink.0.34
Done.

That seems a potentially very useful thing to be able to do – and you don't lose the safety of the "compiler" packages – opam remove flexdll (or base-unix, etc.) will still complain that this would involve removing the compiler:

[opam:4.02.0] [git:windows] C:\DRA\opam\src>opam remove flexdll
Your request can't be satisfied:
  - Package ocaml is part of the base for this compiler and can't be changed

No solution found, exiting


David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/opam-devel/attachments/20160601/2850e78f/attachment-0001.html>