[ocaml-platform] Dev Version as Package?

Wed May 20 02:53:11 BST 2015

Indeed, `opam update my-repo` would tell OPAM to only update the git from the repository. What you expected for updating the package should work, just run `opam update` without argument, or specifying the package -- OPAM will update any repository, and any package that is bound to version control (either through pinning, or its in-repo url file) with just `opam update`.

> By inconsistent, do you mean functionally, because the VC repo would be
> updated regularly, and the rest of the meta-data wouldn't?

Pinning a package to a given source makes OPAM fetch its metadata from that source instead of the repository, if it can find any. Therefore, if we update the package from its upstream _after_ we decided to install it using some current metadata, we could get different metadata: 
if we ignore it, it's inconsistent with the normal pinning behaviour (update package & meta and keep them in sync). If we use it, it's inconsistent with the installation scheme that we computed (e.g., a dependency may have been added), and we would have to start over, which is not really a solution.

This wouldn't happen with non-pinned development packages, though (i.e. packages with an url pointing to a VCS), since those are stuck with the metadata found in the opam repository, and will ignore an `opam` file found in the source. It would make little sense on a UI prespective to do updates in this case that we don't do in the pinned case, though...

Please anyone feel free to point out stuff that was in need of clarification that emerged from this thread, as well as useful usage scenarios: I'll gladly use them to improve the documentation (PRs to the doc pages, opam.git/doc/pages also warmly welcome, if you're so inclined).

Best,
Louis

> - Trevor Smith, 19/05/2015 19:06 -
> Sorry for the multiple posts. I see now my previous post would not work,
> because it was asking to only update the repo (opam update <repo>) and not
> the package (opam update <package).
> 
> Will post again once a get a full working solution of what I want to do.
> 
> Trevor
> 
> On Tue, May 19, 2015 at 1:42 PM, Trevor Smith <trevorsummerssmith at gmail.com>
> wrote:
> 
> > I see this works with a pin just not with a package due to the url to the
> > source repo not having this same functionality currently.
> >
> > So this is why people have the per-package pins -- which is a lot of
> > overhead.
> >
> > It seems to me the only workaround that allows package B to state it wants
> > package A to always be updated is to have either a) per-package pin-file or
> > b) per-package repo. Both of these workaround seem very not ideal as it
> > requires the dependencies for a package to live in multiple files.
> >
> > Any other thoughts on workarounds for this? Louis I hear your point about
> > wanting not a lot of magic in the core commands. However it seems to me
> > that this is a use case that isn't currently covered by the system, and,
> > again seems to be very useful.
> >
> > There is also a technical issue with what you propose: solving an
> >> installation is based on the current -- static -- metadata. So that
> >> metadata can't depend on updating other metadata, or resolving an
> >> installation would become a nightmare. That doesn't apply, though, if you
> >> only update the package source and not its metadata (like would be done
> >> with a repository package pointing to a VC repository, as opposed to a
> >> pinned package) -- it would be much easier to handle in that case, although
> >> it could become a bit inconsistent.
> >
> >
> > By inconsistent, do you mean functionally, because the VC repo would be
> > updated regularly, and the rest of the meta-data wouldn't?
> >
> > Trevor
> >
> > On Tue, May 19, 2015 at 1:16 PM, Trevor Smith <
> > trevorsummerssmith at gmail.com> wrote:
> >
> >> Louis,
> >>
> >> Thanks for your responses. Apologies for my late response. I only just
> >> got time to try this all out.
> >>
> >> I tried out the workarounds but am not seeing it pick up new commits made
> >> to a git repo.
> >>
> >> Example (not using dependencies):
> >>
> >> 1) Create package "foo" in a custom repo "my-repo". With url file set to
> >> a git repo.
> >> 2) Add repo.
> >> 3) opam install foo
> >> 4) Make new commit and push to foo's repo
> >> <following your suggested workflow>
> >> 5) opam update -u my-repo
> >> <this returns Everything as up-to-date as possible.>
> >> 6) opam install foo
> >> <package is already installed>
> >>
> >> I think I am missing something from your suggestions -- perhaps this is
> >> clearly not going to work because OPAM does not ever check the upstream git
> >> repo?). Was that workflow meant to only work with a pinned version?
> >>
> >> Thanks.
> >>
> >> Trevor
> >>
> >> On Sun, May 10, 2015 at 11:13 PM, Louis Gesbert <
> >> louis.gesbert at ocamlpro.com> wrote:
> >>
> >>> > 1) I think that `opam update -u <my-internal-repo>` and then `opam
> >>> upgrade`
> >>> > would indeed do what I need as a workaround.
> >>>
> >>> `update -u` is "update then upgrade" so no need for the upgrade
> >>> afterwards :)
> >>>
> >>> > The incidental complexity the it creates isn't that big of a deal for
> >>> > one-offs on my box, but when managing a team of devs, and build boxes
> >>> one
> >>> > wants something straightforward.
> >>> > It seems though that to make that work with the build system and to not
> >>> > have unintended updates one would then need 2 internal dev repos -- one
> >>> > that had all of the git versioned ones, and another repo that had your
> >>> > normal x.y.z versions.
> >>>
> >>> Not sure I get what you are after exactly, but you could also put both
> >>> version in the same repository! Making separate repositories would be
> >>> useful if you want to easily configure hosts for the dev or release
> >>> versions.
> >>>
> >>> > 2) I like my suggested solution better because I think it adds no
> >>> > incidental complexity -- one has to describe what needs describing (ie
> >>> "I
> >>> > want this dependency always updated"), and no extra lifecycle commands
> >>> (ie
> >>> > no explicit "update" after one has already pushed code). And adds a
> >>> clear
> >>> > functionality to the tool.
> >>> >
> >>> > Agreed that updating a ton of packages could take a while -- but
> >>> having a
> >>> > clear tool (as in what I am suggesting) to define that gives the user
> >>> that
> >>> > flexibility -- it's my fault if I've decided to have 100 "alpha"
> >>> packages.
> >>> > I've asked for it.
> >>> >
> >>> > So I'm clear -- are you saying that due to how updates are currently
> >>> dealt
> >>> > with within opam that the feature I'm suggesting probably would not
> >>> happen?
> >>> > (I haven't taken a look at how opam works).
> >>>
> >>> Granted, there is already the mentionned-pinned-package-auto-update
> >>> trick, but it's quite limited in scope -- I think it's good design
> >>> otherwise to keep matters separate and provide atomic commands: the update
> >>> of the internal metadata, and the installation of packages, are clearly
> >>> distinct operations in OPAM. Higher-level operations can easily be built on
> >>> top of it, and possibly added to OPAM, but we shouldn't put too much magic
> >>> in the base operations: that makes things more difficult to handle, to
> >>> understand and to debug.
> >>>
> >>> There is also a technical issue with what you propose: solving an
> >>> installation is based on the current -- static -- metadata. So that
> >>> metadata can't depend on updating other metadata, or resolving an
> >>> installation would become a nightmare. That doesn't apply, though, if you
> >>> only update the package source and not its metadata (like would be done
> >>> with a repository package pointing to a VC repository, as opposed to a
> >>> pinned package) -- it would be much easier to handle in that case, although
> >>> it could become a bit inconsistent.
> >>>
> >>> With the fixes I mentionned earlier, a possibility would be to add a
> >>> `--update-dev` option to `install` and `upgrade`, meaning something like:
> >>> `opam install --update-dev foo`: get all installed development packages
> >>> transitively related to `foo` (dependencies or dependent), update them
> >>> first, then proceed with the installation, recompiling changed ones.
> >>>
> >>> For `install`, this would be more or less similar to `opam install foo
> >>> $(opam list --rec --required-by foo)`, once we implement the changes to do
> >>> related recompilations on install.
> >>>
> >>> Does that make sense ?
> >>>
> >>> Best,
> >>> Louis
> >>>
> >>> >
> >>> > Thanks.
> >>> >
> >>> > Trevor
> >>> >
> >>> > On Sun, May 10, 2015 at 9:07 PM, Louis Gesbert <
> >>> louis.gesbert at ocamlpro.com>
> >>> > wrote:
> >>> >
> >>> > > OPAM normally never updates its metadata outside of `opam update`;
> >>> this
> >>> > > was inconvenient for pins, so a trick was added: when a pinned
> >>> package is
> >>> > > mentionned explicitely on an install command-line, it is first
> >>> updated from
> >>> > > its upstream. Updating all the time could get very slow if you have
> >>> many
> >>> > > packages pinned to their upstream, for example (like I do).
> >>> > >
> >>> > > As for reinstallations, OPAM does the minimum reinstallations to
> >>> guarantee
> >>> > > consistency, i.e. it recompiles only packages that depend on a
> >>> changed one
> >>> > > -- not the other way around. It will, however, mark packages that
> >>> were
> >>> > > changed after an update for reinstallation. This should work in a
> >>> similar
> >>> > > way for repository and pinned packages. However, that reinstallation
> >>> will
> >>> > > only take place on an `opam upgrade` without argument; it would
> >>> probably be
> >>> > > an improvement to process it as soon as it belongs to the dependency
> >>> cone
> >>> > > of changed packages.
> >>> > >
> >>> > > Now for the dirty (and not really working) workarounds:
> >>> > > 1. `alias opam="opam update --dev; opam"`. (this would work with the
> >>> fix
> >>> > > above; and the option --dev doesn't exist at the moment, we only
> >>> have an
> >>> > > option --repositories. Adding it)
> >>> > > 2. Instead of `opam install foo`, fun `opam install $(opam list -s
> >>> --rec
> >>> > > --required-by foo) foo`. This should trigger updates due to the trick
> >>> > > above. The reinstallations are only done on `opam upgrade` though.
> >>> > >
> >>> > > So indeed, your only real option if you want to keep your stuff
> >>> up-to-date
> >>> > > is to run `opam update -u` beforehand (possibly with your dev-repo
> >>> name, or
> >>> > > --dev, depending on whether you use a custom repo or pinned
> >>> packages).
> >>> > >
> >>> > > Now, if we were to handle planned recompilations more often than
> >>> only on
> >>> > > global `opam upgrade`, would that be enough to handle your use-case
> >>> better ?
> >>> > >
> >>> > > Best,
> >>> > > Louis
> >>> > >
> >>> > > > - Trevor Smith, 10/05/2015 19:55 -
> >>> > > > Thanks Thomas.
> >>> > > >
> >>> > > > Would there be interest in making a feature to "always update this
> >>> > > > dependency"? Suggested solution:
> >>> > > >
> >>> > > > A keyword in the version could trigger "always update". This would
> >>> then
> >>> > > > apply to both normal packages as well as pins. "Dev" already has
> >>> meaning
> >>> > > in
> >>> > > > opam, perhaps "-alpha". This would then allow me to internally
> >>> have a
> >>> > > > single repo with normal versions alongside versions that are
> >>> actively
> >>> > > being
> >>> > > > developed and changing multiple times per day. Eg:
> >>> > > >
> >>> > > > my-repo/my-package.1.0-alpha/url:
> >>> > > > src: "ssh://somewhere/my-package.git"
> >>> > > >
> >>> > > > A dependent package (ie reverse dependency) "program" on:
> >>> "my-package" {=
> >>> > > > "1.0-alpha" } would then always check for updates and upgrade
> >>> my-package
> >>> > > > before doing anything with program. A build server could get the
> >>> behavior
> >>> > > > I'm looking for by "opam install --deps-only program" and getting
> >>> > > whatever
> >>> > > > is at head of my-package (or of the given branch as the case may
> >>> be).
> >>> > > >
> >>> > > > I think this solution would be a nice composition of the ideas
> >>> that opam
> >>> > > > already has, and, I think, would be a strict improvement over the
> >>> current
> >>> > > > situation and does not introduce any incidental complexity. This
> >>> thread
> >>> > > > seems to indicate that others are interested in such a feature.
> >>> > > >
> >>> > > > Thoughts? Thanks!
> >>> > > >
> >>> > > > Trevor
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > On Sun, May 10, 2015 at 5:50 PM, Thomas Gazagnaire <
> >>> > > thomas at gazagnaire.org>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Consider two packages: lib and program. program depends upon lib.
> >>> > > > >
> >>> > > > >
> >>> > > > > opam install lib
> >>> > > > > # Make a new commit to lib and push it
> >>> > > > > # Lib is now one commit newer
> >>> > > > > # NB I am making _no_ changes to the internal opam repo
> >>> > > > > opam install program
> >>> > > > >
> >>> > > > > lib does _not_ get recompiled.
> >>> > > > >
> >>> > > > > Is that the information you wanted?
> >>> > > > >
> >>> > > > >
> >>> > > > > yes, when pin/dev packages are modified, you need to tell opam
> >>> that you
> >>> > > > > want to use the updated version (if available), so you need to
> >>> run
> >>> > > `opam
> >>> > > > > update -u` before `opam install program`. Opam will check if
> >>> there are
> >>> > > new
> >>> > > > > commits and recompile what needs to be recompiled.
> >>> > > > >
> >>> > > > > Thomas
> >>> > > > >
> >>> > > > >
> >>> > > > > Also: I just tried opam update program and that also did not
> >>> pick up
> >>> > > the
> >>> > > > > fact that lib is git pinned.
> >>> > > > >
> >>> > > > > Thoughts? Thanks.
> >>> > > > >
> >>> > > > > Trevor
> >>> > > > >
> >>> > > > > On Sun, May 10, 2015 at 1:17 PM, Thomas Gazagnaire <
> >>> > > thomas at gazagnaire.org>
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > >> I tried setting up the "option 1" -- using a repo url. This
> >>> works fine
> >>> > > > >> for clean installs, however it does not work for my use case for
> >>> > > updates. I
> >>> > > > >> didn't realize until I tried it out that this won't update upon
> >>> any
> >>> > > > >> dependent installation. As noted by Louis, pinning also does not
> >>> > > reinstall
> >>> > > > >> from a repo url when a dependent install happens -- it only
> >>> updates
> >>> > > the
> >>> > > > >> meta-data.
> >>> > > > >>
> >>> > > > >> What I would like is a way (ideally within opam) to say "when
> >>> this
> >>> > > > >> package dependend-upon it should always be checked for update
> >>> and
> >>> > > upgrade".
> >>> > > > >>
> >>> > > > >> Am I correct in stating that currently there is no way to mark a
> >>> > > package
> >>> > > > >> as "update and upgrade this package whenever something that
> >>> depends
> >>> > > upon it
> >>> > > > >> is installed"?
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> did you run `opam update -u <package>`? If a or dev or pinned
> >>> package
> >>> > > > >> changes it should normally trigger a recompilation of all the
> >>> reverse
> >>> > > > >> dependencies. how did you specify the packages in your repo?
> >>> > > > >>
> >>> > > > >> Thomas
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> Thanks.
> >>> > > > >>
> >>> > > > >> Trevor
> >>> > > > >>
> >>> > > > >> On Fri, May 8, 2015 at 2:24 PM, Daniel Bünzli <
> >>> > > > >> daniel.buenzli at erratique.ch> wrote:
> >>> > > > >>
> >>> > > > >>> Le vendredi, 8 mai 2015 à 20:09, Ashish Agarwal a écrit :
> >>> > > > >>> > Louis, thanks for your suggestions. I'm trying them out, but
> >>> one
> >>> > > quick
> >>> > > > >>> question: how can you query with tags. I tried `opam list -e
> >>> > > foobar`, and I
> >>> > > > >>> seem to get the same output no matter what I write for foobar.
> >>> > > > >>>
> >>> > > > >>> This is not opam tags this is depexts tags (that correspond to
> >>> > > > >>> platform). You can do for example:
> >>> > > > >>>
> >>> > > > >>>   opam search -s org:erratique
> >>> > > > >>>
> >>> > > > >>> But it may not be entirely precise since opam-search matches
> >>> not
> >>> > > only in
> >>> > > > >>> tags. I think opam-list should be able to filter by tags (I
> >>> actually
> >>> > > > >>> thought this was possible).
> >>> > > > >>>
> >>> > > > >>> Best,
> >>> > > > >>>
> >>> > > > >>> Daniel
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> _______________________________________________
> >>> > > > >>> Platform mailing list
> >>> > > > >>> Platform at lists.ocaml.org
> >>> > > > >>> http://lists.ocaml.org/listinfo/platform
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >> _______________________________________________
> >>> > > > >> Platform mailing list
> >>> > > > >> Platform at lists.ocaml.org
> >>> > > > >> http://lists.ocaml.org/listinfo/platform
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > >
> >>>
> >>
> >>
> >
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.ocaml.org/pipermail/platform/attachments/20150520/5f0f7ca3/attachment.sig>