<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html><head><meta name="qrichtext" content="1" /><style type="text/css">
p, li { white-space: pre-wrap; }
</style></head><body style=" font-family:'Oxygen-Sans'; font-size:10pt; font-weight:400; font-style:normal;">
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><a name="issue-151380085"></a><a href="https://github.com/Drup"><span style=" font-weight:600; text-decoration: underline; color:#2980b9;">> D</span></a><span style=" font-weight:600; text-decoration: underline; color:#2980b9;">rup</span><span style=" font-weight:600;"> </span>commented <a href="https://github.com/ocaml/opam/issues/2537#issue-151380085"><span style=" text-decoration: underline; color:#2980b9;">5 days ago</span></a> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> I created an empty switch last and installed ocaml in it manually. It seems</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> ocaml-version is not udpated. % opam install merlin utop tuareg</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> [ERROR] utop has unmet availability conditions: ocaml-version >= "4.01"</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> [ERROR] merlin has unmet availability conditions: ocaml-version >= "4.00.0"</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> % opam switch show</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> last</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> % opam list -i</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> base-bigarray base Bigarray library distributed with the OCaml compiler</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> base-threads base Threads library distributed with the OCaml compiler</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> base-unix base Unix library distributed with the OCaml compiler</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> ocaml 4.03.0 Official 4.03.0 release</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Ah, this part is lacking documentation, and there may be some rough edges yet to polish, so let me explain how it works at the moment (on next). And thanks again for testing ☺ </p>
<p style=" margin-top:14px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-size:large; font-weight:600;">How it works</span> </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">So, compilers now being in packages, compiler-specific variables (like typically <span style=" font-family:'Courier New,courier';">ocaml-version</span>) can not generally be defined at switch creation time anymore. However, packages have been able to export opam variables since opam 1.0 by providing a <span style=" font-family:'Courier New,courier';"><pkgname>.config</span> file at their root (similar to <span style=" font-family:'Courier New,courier';">.install</span>; but although this is an older feature it probably lacks documentation too). These variables, however, are scoped with the package name to avoid clashes: if you have <span style=" font-family:'Courier New,courier';">foo</span> installed, you can always access <span style=" font-family:'Courier New,courier';">foo:lib</span> to get its lib directory (<span style=" font-family:'Courier New,courier';"><prefix>/lib/foo</span> normally), but also <span style=" font-family:'Courier New,courier';">foo:bar</span> if <span style=" font-family:'Courier New,courier';">foo</span> defined <span style=" font-family:'Courier New,courier';">bar</span> in its configuration file. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">The idea, then, to keep compatibility with the previously global <span style=" font-family:'Courier New,courier';">ocaml-*</span> variables, is that <span style=" font-style:italic;">for packages installed as compilers</span> (<span style=" font-style:italic;">i.e.</span> "base packages" having the <span style=" font-family:'Courier New,courier';">compiler</span> flag set), the variables they define are re-exported in the global scope of the switch. So the <span style=" font-family:'Courier New,courier';">ocaml</span> package just has to define an <span style=" font-family:'Courier New,courier';">ocaml-version</span> variable through its <span style=" font-family:'Courier New,courier';">.config</span> file. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">It could be much simpler to define these values in the compiler's <span style=" font-family:'Courier New,courier';">opam</span> file, but that would be assuming they can be statically defined. In particular, in the case of the system compiler, the package build is simply a polling script that will generate the configuration, and we don't know the OCaml version in advance (so <span style=" font-family:'Courier New,courier';">ocaml-version</span> and <span style=" font-family:'Courier New,courier';">ocaml:version</span> can be different, with the latter being <span style=" font-family:'Courier New,courier';">system</span> in this case). Even for compiled OCamls, we might not know in advance whether <span style=" font-family:'Courier New,courier';">native-dynlink</span> is available. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">In your case, <span style=" font-family:'Courier New,courier';">ocaml</span> was installed as a "normal" package, so its variables weren't re-exported: look at files in <span style=" font-family:'Courier New,courier';"><switch-prefix>/.opam-switch/config</span>, <span style=" font-family:'Courier New,courier';">global-config.config</span> is where global variables are defined, the other files are per-package. Like we have <span style=" font-family:'Courier New,courier';">opam install --[un]set-root</span>, we probably need expert commands that allow adjusting the defined compiler in the current switch (you can edit <span style=" font-family:'Courier New,courier';"><prefix>/.opam-switch/switch-state</span> easily, but that won't do the variable export part, and is no excuse for not having it in the CLI anyway). </p>
<p style=" margin-top:14px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-size:large; font-weight:600;">Now let's get to the funny details...</span> </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">The reason to re-export in the global scope rather than use the variable directly (after all, we could replace uses of <span style=" font-family:'Courier New,courier';">ocaml-native-tools</span> with a scoped variable <span style=" font-family:'Courier New,courier';">ocaml:native-tools</span>, and even <span style=" font-family:'Courier New,courier';">ocaml-version</span> with <span style=" font-family:'Courier New,courier';">ocaml:real-version</span> or something like that) is that there is a difference between the two: </p>
<ul style="margin-top: 0px; margin-bottom: 0px; margin-right: 0px; -qt-list-indent: 1;"><li style=" margin-top:12px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">package variables are only available in scripts, i.e. for actions happening <span style=" font-style:italic;">after</span> the solver solved</li>
<li style=" margin-top:0px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">global variables can be used anywhere, and, most importantly, in the <span style=" font-family:'Courier New,courier';">available:</span> field, which is used to filter the packages <span style=" font-style:italic;">before</span> solving, and therefore shouldn't depend on what packages are installed. </li></ul>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Obviously, <span style=" font-family:'Courier New,courier';">ocaml-version</span> is a variable we absolutely want accessible in the <span style=" font-family:'Courier New,courier';">available:</span> field... so we needed to have it in the global scope. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">What this implies, however, is that it's inconsistent to change your compiler <span style=" font-style:italic;">and</span> non-compiler packages in the same set of actions: <span style=" font-family:'Courier New,courier';">ocaml-version</span> may be defined incorrectly. Currently, we do <span style=" font-style:italic;">not</span> handle this correctly, however it could happen on:<br />1. upgrade of the system compiler<br />2. manual change of the switch's compiler (e.g. through pinning) </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">I have a plan to solve this situation, which requires solving once, limiting the actions to the removals and compiler change, processing them, then re-running the solver in the new environment to complete the expected actions. It's a bit complex to my taste, however, and means that, by design, the user must validate (and start removing packages) before being able to know what the expected end state will be. May be acceptable for a compiler change, though, in <span style=" font-family:'Courier New,courier';">1.2.2</span>, case 1. required a manual, full <span style=" font-family:'Courier New,courier';">switch reinstall</span> (that could fail due to broken constraints, I think), and case 2. was impossible altogether. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">In the current state, the reinstallation of packages following the switch change could be tried on package unavailable for the new version and fail, and the user will need to re-run opam with a proper request to get a correct solution and retry to install them. </p>
<p style=" margin-top:14px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-size:large; font-weight:600;">Another solution (or not)</span> </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Now, if you expected something simpler, you can skip this. More elegant, maybe, but not simpler. Also (spoiler), this doesn't actually solve the problem of dynamic system packages. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Generally speaking, it would be nicer to be able to write, instead of<br /><span style=" font-family:'Courier New,courier';">available: ocaml-version >= "4.01"</span><br />something in the lines of<br /><span style=" font-family:'Courier New,courier';">depends: [ "ocaml" >= "4.01" ]</span> </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">This allows to pass all the details to the solver, and gets rid of these annoying package/global variables needed for <span style=" font-family:'Courier New,courier';">available:</span>. One issue is that currently, we have a mismatch between OCaml versions, and the ocaml package versions, which also encode information on the variant (<span style=" font-family:'Courier New,courier';">4.01.0+BER</span>, <span style=" font-family:'Courier New,courier';">system</span>). </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">The idea here is to rely on the <span style=" font-family:'Courier New,courier';">provides:</span> feature, which allows a package to be available under different names besides its own. This brings a whole lot of difficulties on its own, but that will be for another time -- I have lots on notes on this too. For the time being, let's assume it just works. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Using this, we could have the variants defined in different packages (e.g. <span style=" font-family:'Courier New,courier';">ocaml+BER</span>), and "providing" the <span style=" font-family:'Courier New,courier';">ocaml</span> feature with version <span style=" font-family:'Courier New,courier';">4.01.0</span>.<br />This also works for other compiler features: we currently use two mechanisms, <span style=" font-family:'Courier New,courier';">base-</span> packages (e.g. <span style=" font-family:'Courier New,courier';">base-threads</span>) that we depend on, and <span style=" font-family:'Courier New,courier';">ocaml-</span> variables (e.g. <span style=" font-family:'Courier New,courier';">ocaml-native-dynlink</span>) that we use in scripts (and possibly, the <span style=" font-family:'Courier New,courier';">available:</span> field). Here a given compiler could "provide" <span style=" font-family:'Courier New,courier';">native-dynlink</span> and <span style=" font-family:'Courier New,courier';">threads</span> as packages, and other packages directly depend on that. It also seems much more appropriate on a package metadata point of view, a given compiler implementation <span style=" font-style:italic;">providing</span> a set of features. </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">What this doesn't solve at all, however, is the system compiler issue, since these <span style=" font-family:'Courier New,courier';">provides:</span> must be defined statically </p>
<p style=" margin-top:14px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-size:large; font-weight:600;">Dynamic packages</span> </p>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">We've drifted a bit, but the bulk of the issue actually boils down to this: dynamic packages, and in particular, definition of the system compiler. </p>
<ul style="margin-top: 0px; margin-bottom: 0px; margin-right: 0px; -qt-list-indent: 1;"><li style=" margin-top:12px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">in 1.2.2, polling and generation of a compiler description, as well as further polling to detect changes, were hard-coded in opam</li>
<li style=" margin-top:0px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">in 2.0~alpha, there is no OCaml specific code in opam, the polling is done by the package itself, and the compiler-package-variable-made-global feature bridges the gap to the package selection </li></ul>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">There may be middle-grounds or alternate solutions: </p>
<ul style="margin-top: 0px; margin-bottom: 0px; margin-right: 0px; -qt-list-indent: 1;"><li style=" margin-top:12px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">somehow allow the definition of dynamic packages in the repository (seems complex, costly, and most importantly very difficult to do without compromising security -- any dynamic package would need to do its polling before it was selected by the user in any way)</li>
<li style=" margin-top:0px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">keep the hard-coded generation, but put it in an <span style=" font-family:'Courier New,courier';">ocaml plugin</span> added to the ocaml-agnostic opam. </li></ul>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Note that having the system package in the repository has some benefits; for example, in 1.2.2, it is impossible to change its set of base packages, which raised some difficulties when <span style=" font-family:'Courier New,courier';">base-ocamlbuild</span> started being needed. </p>
<hr />
<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Ok, this got quite long, I am sure you didn't expect that in your report. I'll forward to the ML since there is worthwile design discussions to be had. </p></body></html>