[opam-devel] On compiler package and the system compiler (fwd from: https://github.com/ocaml/opam/issues/2537)
louis.gesbert at ocamlpro.com
Mon May 2 03:57:21 BST 2016
*> Drup *commented 5 days ago
> I created an empty switch last and installed ocaml in it manually. It seems
> ocaml-version is not udpated. % opam install merlin utop tuareg
> [ERROR] utop has unmet availability conditions: ocaml-version >= "4.01"
> [ERROR] merlin has unmet availability conditions: ocaml-version >= "4.00.0"
> % opam switch show
> % opam list -i
> base-bigarray base Bigarray library distributed with the OCaml compiler
> base-threads base Threads library distributed with the OCaml compiler
> base-unix base Unix library distributed with the OCaml compiler
> ocaml 4.03.0 Official 4.03.0 release
Ah, this part is lacking documentation, and there may be some rough edges yet to polish, so let
me explain how it works at the moment (on next). And thanks again for testing ☺
*How it works*
So, compilers now being in packages, compiler-specific variables (like typically ocaml-version)
can not generally be defined at switch creation time anymore. However, packages have been
able to export opam variables since opam 1.0 by providing a <pkgname>.config file at their root
(similar to .install; but although this is an older feature it probably lacks documentation too).
These variables, however, are scoped with the package name to avoid clashes: if you have foo
installed, you can always access foo:lib to get its lib directory (<prefix>/lib/foo normally), but
also foo:bar if foo defined bar in its configuration file.
The idea, then, to keep compatibility with the previously global ocaml-* variables, is that /for
packages installed as compilers/ (/i.e./ "base packages" having the compiler flag set), the
variables they define are re-exported in the global scope of the switch. So the ocaml package
just has to define an ocaml-version variable through its .config file.
It could be much simpler to define these values in the compiler's opam file, but that would be
assuming they can be statically defined. In particular, in the case of the system compiler, the
package build is simply a polling script that will generate the configuration, and we don't know
the OCaml version in advance (so ocaml-version and ocaml:version can be different, with the
latter being system in this case). Even for compiled OCamls, we might not know in advance
whether native-dynlink is available.
In your case, ocaml was installed as a "normal" package, so its variables weren't re-exported:
look at files in <switch-prefix>/.opam-switch/config, global-config.config is where global
variables are defined, the other files are per-package. Like we have opam install --[un]set-root,
we probably need expert commands that allow adjusting the defined compiler in the current
switch (you can edit <prefix>/.opam-switch/switch-state easily, but that won't do the variable
export part, and is no excuse for not having it in the CLI anyway).
*Now let's get to the funny details...*
The reason to re-export in the global scope rather than use the variable directly (after all, we
could replace uses of ocaml-native-tools with a scoped variable ocaml:native-tools, and even
ocaml-version with ocaml:real-version or something like that) is that there is a difference
between the two:
* package variables are only available in scripts, i.e. for actions happening /after/ the solver
* global variables can be used anywhere, and, most importantly, in the available: field,
which is used to filter the packages /before/ solving, and therefore shouldn't depend on what
packages are installed.
Obviously, ocaml-version is a variable we absolutely want accessible in the available: field... so
we needed to have it in the global scope.
What this implies, however, is that it's inconsistent to change your compiler /and/ non-compiler
packages in the same set of actions: ocaml-version may be defined incorrectly. Currently, we
do /not/ handle this correctly, however it could happen on:1. upgrade of the system compiler2.
manual change of the switch's compiler (e.g. through pinning)
I have a plan to solve this situation, which requires solving once, limiting the actions to the
removals and compiler change, processing them, then re-running the solver in the new
environment to complete the expected actions. It's a bit complex to my taste, however, and
means that, by design, the user must validate (and start removing packages) before being able
to know what the expected end state will be. May be acceptable for a compiler change, though,
in 1.2.2, case 1. required a manual, full switch reinstall (that could fail due to broken
constraints, I think), and case 2. was impossible altogether.
In the current state, the reinstallation of packages following the switch change could be tried
on package unavailable for the new version and fail, and the user will need to re-run opam with
a proper request to get a correct solution and retry to install them.
*Another solution (or not)*
Now, if you expected something simpler, you can skip this. More elegant, maybe, but not
simpler. Also (spoiler), this doesn't actually solve the problem of dynamic system packages.
Generally speaking, it would be nicer to be able to write, instead of
available: ocaml-version >= "4.01"
depends: [ "ocaml" >= "4.01" ]
This allows to pass all the details to the solver, and gets rid of these annoying package/global
variables needed for available:. One issue is that currently, we have a mismatch between
OCaml versions, and the ocaml package versions, which also encode information on the variant
The idea here is to rely on the provides: feature, which allows a package to be available under
different names besides its own. This brings a whole lot of difficulties on its own, but that will
be for another time -- I have lots on notes on this too. For the time being, let's assume it just
Using this, we could have the variants defined in different packages (e.g. ocaml+BER), and
"providing" the ocaml feature with version 4.01.0.This also works for other compiler features:
we currently use two mechanisms, base- packages (e.g. base-threads) that we depend on, and
ocaml- variables (e.g. ocaml-native-dynlink) that we use in scripts (and possibly, the available:
field). Here a given compiler could "provide" native-dynlink and threads as packages, and other
packages directly depend on that. It also seems much more appropriate on a package metadata
point of view, a given compiler implementation /providing/ a set of features.
What this doesn't solve at all, however, is the system compiler issue, since these provides:
must be defined statically
We've drifted a bit, but the bulk of the issue actually boils down to this: dynamic packages, and
in particular, definition of the system compiler.
* in 1.2.2, polling and generation of a compiler description, as well as further polling to
detect changes, were hard-coded in opam
* in 2.0~alpha, there is no OCaml specific code in opam, the polling is done by the package
itself, and the compiler-package-variable-made-global feature bridges the gap to the package
There may be middle-grounds or alternate solutions:
* somehow allow the definition of dynamic packages in the repository (seems complex,
costly, and most importantly very difficult to do without compromising security -- any dynamic
package would need to do its polling before it was selected by the user in any way)
* keep the hard-coded generation, but put it in an ocaml plugin added to the ocaml-agnostic
Note that having the system package in the repository has some benefits; for example, in 1.2.2,
it is impossible to change its set of base packages, which raised some difficulties when base-
ocamlbuild started being needed.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the opam-devel