[ocaml-platform] Dev Version as Package?

[Louis Gesbert]

> 1) I think that `opam update -u <my-internal-repo>` and then `opam upgrade`
> would indeed do what I need as a workaround.

`update -u` is "update then upgrade" so no need for the upgrade afterwards :)

> The incidental complexity the it creates isn't that big of a deal for
> one-offs on my box, but when managing a team of devs, and build boxes one
> wants something straightforward.
> It seems though that to make that work with the build system and to not
> have unintended updates one would then need 2 internal dev repos -- one
> that had all of the git versioned ones, and another repo that had your
> normal x.y.z versions.

Not sure I get what you are after exactly, but you could also put both version in the same repository! Making separate repositories would be useful if you want to easily configure hosts for the dev or release versions.

> 2) I like my suggested solution better because I think it adds no
> incidental complexity -- one has to describe what needs describing (ie "I
> want this dependency always updated"), and no extra lifecycle commands (ie
> no explicit "update" after one has already pushed code). And adds a clear
> functionality to the tool.
> 
> Agreed that updating a ton of packages could take a while -- but having a
> clear tool (as in what I am suggesting) to define that gives the user that
> flexibility -- it's my fault if I've decided to have 100 "alpha" packages.
> I've asked for it.
> 
> So I'm clear -- are you saying that due to how updates are currently dealt
> with within opam that the feature I'm suggesting probably would not happen?
> (I haven't taken a look at how opam works).

Granted, there is already the mentionned-pinned-package-auto-update trick, but it's quite limited in scope -- I think it's good design otherwise to keep matters separate and provide atomic commands: the update of the internal metadata, and the installation of packages, are clearly distinct operations in OPAM. Higher-level operations can easily be built on top of it, and possibly added to OPAM, but we shouldn't put too much magic in the base operations: that makes things more difficult to handle, to understand and to debug.

There is also a technical issue with what you propose: solving an installation is based on the current -- static -- metadata. So that metadata can't depend on updating other metadata, or resolving an installation would become a nightmare. That doesn't apply, though, if you only update the package source and not its metadata (like would be done with a repository package pointing to a VC repository, as opposed to a pinned package) -- it would be much easier to handle in that case, although it could become a bit inconsistent.

With the fixes I mentionned earlier, a possibility would be to add a `--update-dev` option to `install` and `upgrade`, meaning something like:
`opam install --update-dev foo`: get all installed development packages transitively related to `foo` (dependencies or dependent), update them first, then proceed with the installation, recompiling changed ones.

For `install`, this would be more or less similar to `opam install foo $(opam list --rec --required-by foo)`, once we implement the changes to do related recompilations on install.

Does that make sense ?

Best,
Louis

> 
> Thanks.
> 
> Trevor
> 
> On Sun, May 10, 2015 at 9:07 PM, Louis Gesbert <louis.gesbert at ocamlpro.com>
> wrote:
> 
> > OPAM normally never updates its metadata outside of `opam update`; this
> > was inconvenient for pins, so a trick was added: when a pinned package is
> > mentionned explicitely on an install command-line, it is first updated from
> > its upstream. Updating all the time could get very slow if you have many
> > packages pinned to their upstream, for example (like I do).
> >
> > As for reinstallations, OPAM does the minimum reinstallations to guarantee
> > consistency, i.e. it recompiles only packages that depend on a changed one
> > -- not the other way around. It will, however, mark packages that were
> > changed after an update for reinstallation. This should work in a similar
> > way for repository and pinned packages. However, that reinstallation will
> > only take place on an `opam upgrade` without argument; it would probably be
> > an improvement to process it as soon as it belongs to the dependency cone
> > of changed packages.
> >
> > Now for the dirty (and not really working) workarounds:
> > 1. `alias opam="opam update --dev; opam"`. (this would work with the fix
> > above; and the option --dev doesn't exist at the moment, we only have an
> > option --repositories. Adding it)
> > 2. Instead of `opam install foo`, fun `opam install $(opam list -s --rec
> > --required-by foo) foo`. This should trigger updates due to the trick
> > above. The reinstallations are only done on `opam upgrade` though.
> >
> > So indeed, your only real option if you want to keep your stuff up-to-date
> > is to run `opam update -u` beforehand (possibly with your dev-repo name, or
> > --dev, depending on whether you use a custom repo or pinned packages).
> >
> > Now, if we were to handle planned recompilations more often than only on
> > global `opam upgrade`, would that be enough to handle your use-case better ?
> >
> > Best,
> > Louis
> >
> > > - Trevor Smith, 10/05/2015 19:55 -
> > > Thanks Thomas.
> > >
> > > Would there be interest in making a feature to "always update this
> > > dependency"? Suggested solution:
> > >
> > > A keyword in the version could trigger "always update". This would then
> > > apply to both normal packages as well as pins. "Dev" already has meaning
> > in
> > > opam, perhaps "-alpha". This would then allow me to internally have a
> > > single repo with normal versions alongside versions that are actively
> > being
> > > developed and changing multiple times per day. Eg:
> > >
> > > my-repo/my-package.1.0-alpha/url:
> > > src: "ssh://somewhere/my-package.git"
> > >
> > > A dependent package (ie reverse dependency) "program" on: "my-package" {=
> > > "1.0-alpha" } would then always check for updates and upgrade my-package
> > > before doing anything with program. A build server could get the behavior
> > > I'm looking for by "opam install --deps-only program" and getting
> > whatever
> > > is at head of my-package (or of the given branch as the case may be).
> > >
> > > I think this solution would be a nice composition of the ideas that opam
> > > already has, and, I think, would be a strict improvement over the current
> > > situation and does not introduce any incidental complexity. This thread
> > > seems to indicate that others are interested in such a feature.
> > >
> > > Thoughts? Thanks!
> > >
> > > Trevor
> > >
> > >
> > >
> > > On Sun, May 10, 2015 at 5:50 PM, Thomas Gazagnaire <
> > thomas at gazagnaire.org>
> > > wrote:
> > >
> > > > Consider two packages: lib and program. program depends upon lib.
> > > >
> > > >
> > > > opam install lib
> > > > # Make a new commit to lib and push it
> > > > # Lib is now one commit newer
> > > > # NB I am making _no_ changes to the internal opam repo
> > > > opam install program
> > > >
> > > > lib does _not_ get recompiled.
> > > >
> > > > Is that the information you wanted?
> > > >
> > > >
> > > > yes, when pin/dev packages are modified, you need to tell opam that you
> > > > want to use the updated version (if available), so you need to run
> > `opam
> > > > update -u` before `opam install program`. Opam will check if there are
> > new
> > > > commits and recompile what needs to be recompiled.
> > > >
> > > > Thomas
> > > >
> > > >
> > > > Also: I just tried opam update program and that also did not pick up
> > the
> > > > fact that lib is git pinned.
> > > >
> > > > Thoughts? Thanks.
> > > >
> > > > Trevor
> > > >
> > > > On Sun, May 10, 2015 at 1:17 PM, Thomas Gazagnaire <
> > thomas at gazagnaire.org>
> > > > wrote:
> > > >
> > > >> I tried setting up the "option 1" -- using a repo url. This works fine
> > > >> for clean installs, however it does not work for my use case for
> > updates. I
> > > >> didn't realize until I tried it out that this won't update upon any
> > > >> dependent installation. As noted by Louis, pinning also does not
> > reinstall
> > > >> from a repo url when a dependent install happens -- it only updates
> > the
> > > >> meta-data.
> > > >>
> > > >> What I would like is a way (ideally within opam) to say "when this
> > > >> package dependend-upon it should always be checked for update and
> > upgrade".
> > > >>
> > > >> Am I correct in stating that currently there is no way to mark a
> > package
> > > >> as "update and upgrade this package whenever something that depends
> > upon it
> > > >> is installed"?
> > > >>
> > > >>
> > > >> did you run `opam update -u <package>`? If a or dev or pinned package
> > > >> changes it should normally trigger a recompilation of all the reverse
> > > >> dependencies. how did you specify the packages in your repo?
> > > >>
> > > >> Thomas
> > > >>
> > > >>
> > > >> Thanks.
> > > >>
> > > >> Trevor
> > > >>
> > > >> On Fri, May 8, 2015 at 2:24 PM, Daniel Bünzli <
> > > >> daniel.buenzli at erratique.ch> wrote:
> > > >>
> > > >>> Le vendredi, 8 mai 2015 à 20:09, Ashish Agarwal a écrit :
> > > >>> > Louis, thanks for your suggestions. I'm trying them out, but one
> > quick
> > > >>> question: how can you query with tags. I tried `opam list -e
> > foobar`, and I
> > > >>> seem to get the same output no matter what I write for foobar.
> > > >>>
> > > >>> This is not opam tags this is depexts tags (that correspond to
> > > >>> platform). You can do for example:
> > > >>>
> > > >>>   opam search -s org:erratique
> > > >>>
> > > >>> But it may not be entirely precise since opam-search matches not
> > only in
> > > >>> tags. I think opam-list should be able to filter by tags (I actually
> > > >>> thought this was possible).
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> Daniel
> > > >>>
> > > >>>
> > > >>> _______________________________________________
> > > >>> Platform mailing list
> > > >>> Platform at lists.ocaml.org
> > > >>> http://lists.ocaml.org/listinfo/platform
> > > >>>
> > > >>
> > > >> _______________________________________________
> > > >> Platform mailing list
> > > >> Platform at lists.ocaml.org
> > > >> http://lists.ocaml.org/listinfo/platform
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> >