[ocaml-platform] Dev Version as Package?

[Trevor Smith]

I see this works with a pin just not with a package due to the url to the
source repo not having this same functionality currently.

So this is why people have the per-package pins -- which is a lot of
overhead.

It seems to me the only workaround that allows package B to state it wants
package A to always be updated is to have either a) per-package pin-file or
b) per-package repo. Both of these workaround seem very not ideal as it
requires the dependencies for a package to live in multiple files.

Any other thoughts on workarounds for this? Louis I hear your point about
wanting not a lot of magic in the core commands. However it seems to me
that this is a use case that isn't currently covered by the system, and,
again seems to be very useful.

There is also a technical issue with what you propose: solving an
> installation is based on the current -- static -- metadata. So that
> metadata can't depend on updating other metadata, or resolving an
> installation would become a nightmare. That doesn't apply, though, if you
> only update the package source and not its metadata (like would be done
> with a repository package pointing to a VC repository, as opposed to a
> pinned package) -- it would be much easier to handle in that case, although
> it could become a bit inconsistent.


By inconsistent, do you mean functionally, because the VC repo would be
updated regularly, and the rest of the meta-data wouldn't?

Trevor

On Tue, May 19, 2015 at 1:16 PM, Trevor Smith <trevorsummerssmith at gmail.com>
wrote:

> Louis,
>
> Thanks for your responses. Apologies for my late response. I only just got
> time to try this all out.
>
> I tried out the workarounds but am not seeing it pick up new commits made
> to a git repo.
>
> Example (not using dependencies):
>
> 1) Create package "foo" in a custom repo "my-repo". With url file set to a
> git repo.
> 2) Add repo.
> 3) opam install foo
> 4) Make new commit and push to foo's repo
> <following your suggested workflow>
> 5) opam update -u my-repo
> <this returns Everything as up-to-date as possible.>
> 6) opam install foo
> <package is already installed>
>
> I think I am missing something from your suggestions -- perhaps this is
> clearly not going to work because OPAM does not ever check the upstream git
> repo?). Was that workflow meant to only work with a pinned version?
>
> Thanks.
>
> Trevor
>
> On Sun, May 10, 2015 at 11:13 PM, Louis Gesbert <
> louis.gesbert at ocamlpro.com> wrote:
>
>> > 1) I think that `opam update -u <my-internal-repo>` and then `opam
>> upgrade`
>> > would indeed do what I need as a workaround.
>>
>> `update -u` is "update then upgrade" so no need for the upgrade
>> afterwards :)
>>
>> > The incidental complexity the it creates isn't that big of a deal for
>> > one-offs on my box, but when managing a team of devs, and build boxes
>> one
>> > wants something straightforward.
>> > It seems though that to make that work with the build system and to not
>> > have unintended updates one would then need 2 internal dev repos -- one
>> > that had all of the git versioned ones, and another repo that had your
>> > normal x.y.z versions.
>>
>> Not sure I get what you are after exactly, but you could also put both
>> version in the same repository! Making separate repositories would be
>> useful if you want to easily configure hosts for the dev or release
>> versions.
>>
>> > 2) I like my suggested solution better because I think it adds no
>> > incidental complexity -- one has to describe what needs describing (ie
>> "I
>> > want this dependency always updated"), and no extra lifecycle commands
>> (ie
>> > no explicit "update" after one has already pushed code). And adds a
>> clear
>> > functionality to the tool.
>> >
>> > Agreed that updating a ton of packages could take a while -- but having
>> a
>> > clear tool (as in what I am suggesting) to define that gives the user
>> that
>> > flexibility -- it's my fault if I've decided to have 100 "alpha"
>> packages.
>> > I've asked for it.
>> >
>> > So I'm clear -- are you saying that due to how updates are currently
>> dealt
>> > with within opam that the feature I'm suggesting probably would not
>> happen?
>> > (I haven't taken a look at how opam works).
>>
>> Granted, there is already the mentionned-pinned-package-auto-update
>> trick, but it's quite limited in scope -- I think it's good design
>> otherwise to keep matters separate and provide atomic commands: the update
>> of the internal metadata, and the installation of packages, are clearly
>> distinct operations in OPAM. Higher-level operations can easily be built on
>> top of it, and possibly added to OPAM, but we shouldn't put too much magic
>> in the base operations: that makes things more difficult to handle, to
>> understand and to debug.
>>
>> There is also a technical issue with what you propose: solving an
>> installation is based on the current -- static -- metadata. So that
>> metadata can't depend on updating other metadata, or resolving an
>> installation would become a nightmare. That doesn't apply, though, if you
>> only update the package source and not its metadata (like would be done
>> with a repository package pointing to a VC repository, as opposed to a
>> pinned package) -- it would be much easier to handle in that case, although
>> it could become a bit inconsistent.
>>
>> With the fixes I mentionned earlier, a possibility would be to add a
>> `--update-dev` option to `install` and `upgrade`, meaning something like:
>> `opam install --update-dev foo`: get all installed development packages
>> transitively related to `foo` (dependencies or dependent), update them
>> first, then proceed with the installation, recompiling changed ones.
>>
>> For `install`, this would be more or less similar to `opam install foo
>> $(opam list --rec --required-by foo)`, once we implement the changes to do
>> related recompilations on install.
>>
>> Does that make sense ?
>>
>> Best,
>> Louis
>>
>> >
>> > Thanks.
>> >
>> > Trevor
>> >
>> > On Sun, May 10, 2015 at 9:07 PM, Louis Gesbert <
>> louis.gesbert at ocamlpro.com>
>> > wrote:
>> >
>> > > OPAM normally never updates its metadata outside of `opam update`;
>> this
>> > > was inconvenient for pins, so a trick was added: when a pinned
>> package is
>> > > mentionned explicitely on an install command-line, it is first
>> updated from
>> > > its upstream. Updating all the time could get very slow if you have
>> many
>> > > packages pinned to their upstream, for example (like I do).
>> > >
>> > > As for reinstallations, OPAM does the minimum reinstallations to
>> guarantee
>> > > consistency, i.e. it recompiles only packages that depend on a
>> changed one
>> > > -- not the other way around. It will, however, mark packages that were
>> > > changed after an update for reinstallation. This should work in a
>> similar
>> > > way for repository and pinned packages. However, that reinstallation
>> will
>> > > only take place on an `opam upgrade` without argument; it would
>> probably be
>> > > an improvement to process it as soon as it belongs to the dependency
>> cone
>> > > of changed packages.
>> > >
>> > > Now for the dirty (and not really working) workarounds:
>> > > 1. `alias opam="opam update --dev; opam"`. (this would work with the
>> fix
>> > > above; and the option --dev doesn't exist at the moment, we only have
>> an
>> > > option --repositories. Adding it)
>> > > 2. Instead of `opam install foo`, fun `opam install $(opam list -s
>> --rec
>> > > --required-by foo) foo`. This should trigger updates due to the trick
>> > > above. The reinstallations are only done on `opam upgrade` though.
>> > >
>> > > So indeed, your only real option if you want to keep your stuff
>> up-to-date
>> > > is to run `opam update -u` beforehand (possibly with your dev-repo
>> name, or
>> > > --dev, depending on whether you use a custom repo or pinned packages).
>> > >
>> > > Now, if we were to handle planned recompilations more often than only
>> on
>> > > global `opam upgrade`, would that be enough to handle your use-case
>> better ?
>> > >
>> > > Best,
>> > > Louis
>> > >
>> > > > - Trevor Smith, 10/05/2015 19:55 -
>> > > > Thanks Thomas.
>> > > >
>> > > > Would there be interest in making a feature to "always update this
>> > > > dependency"? Suggested solution:
>> > > >
>> > > > A keyword in the version could trigger "always update". This would
>> then
>> > > > apply to both normal packages as well as pins. "Dev" already has
>> meaning
>> > > in
>> > > > opam, perhaps "-alpha". This would then allow me to internally have
>> a
>> > > > single repo with normal versions alongside versions that are
>> actively
>> > > being
>> > > > developed and changing multiple times per day. Eg:
>> > > >
>> > > > my-repo/my-package.1.0-alpha/url:
>> > > > src: "ssh://somewhere/my-package.git"
>> > > >
>> > > > A dependent package (ie reverse dependency) "program" on:
>> "my-package" {=
>> > > > "1.0-alpha" } would then always check for updates and upgrade
>> my-package
>> > > > before doing anything with program. A build server could get the
>> behavior
>> > > > I'm looking for by "opam install --deps-only program" and getting
>> > > whatever
>> > > > is at head of my-package (or of the given branch as the case may
>> be).
>> > > >
>> > > > I think this solution would be a nice composition of the ideas that
>> opam
>> > > > already has, and, I think, would be a strict improvement over the
>> current
>> > > > situation and does not introduce any incidental complexity. This
>> thread
>> > > > seems to indicate that others are interested in such a feature.
>> > > >
>> > > > Thoughts? Thanks!
>> > > >
>> > > > Trevor
>> > > >
>> > > >
>> > > >
>> > > > On Sun, May 10, 2015 at 5:50 PM, Thomas Gazagnaire <
>> > > thomas at gazagnaire.org>
>> > > > wrote:
>> > > >
>> > > > > Consider two packages: lib and program. program depends upon lib.
>> > > > >
>> > > > >
>> > > > > opam install lib
>> > > > > # Make a new commit to lib and push it
>> > > > > # Lib is now one commit newer
>> > > > > # NB I am making _no_ changes to the internal opam repo
>> > > > > opam install program
>> > > > >
>> > > > > lib does _not_ get recompiled.
>> > > > >
>> > > > > Is that the information you wanted?
>> > > > >
>> > > > >
>> > > > > yes, when pin/dev packages are modified, you need to tell opam
>> that you
>> > > > > want to use the updated version (if available), so you need to run
>> > > `opam
>> > > > > update -u` before `opam install program`. Opam will check if
>> there are
>> > > new
>> > > > > commits and recompile what needs to be recompiled.
>> > > > >
>> > > > > Thomas
>> > > > >
>> > > > >
>> > > > > Also: I just tried opam update program and that also did not pick
>> up
>> > > the
>> > > > > fact that lib is git pinned.
>> > > > >
>> > > > > Thoughts? Thanks.
>> > > > >
>> > > > > Trevor
>> > > > >
>> > > > > On Sun, May 10, 2015 at 1:17 PM, Thomas Gazagnaire <
>> > > thomas at gazagnaire.org>
>> > > > > wrote:
>> > > > >
>> > > > >> I tried setting up the "option 1" -- using a repo url. This
>> works fine
>> > > > >> for clean installs, however it does not work for my use case for
>> > > updates. I
>> > > > >> didn't realize until I tried it out that this won't update upon
>> any
>> > > > >> dependent installation. As noted by Louis, pinning also does not
>> > > reinstall
>> > > > >> from a repo url when a dependent install happens -- it only
>> updates
>> > > the
>> > > > >> meta-data.
>> > > > >>
>> > > > >> What I would like is a way (ideally within opam) to say "when
>> this
>> > > > >> package dependend-upon it should always be checked for update and
>> > > upgrade".
>> > > > >>
>> > > > >> Am I correct in stating that currently there is no way to mark a
>> > > package
>> > > > >> as "update and upgrade this package whenever something that
>> depends
>> > > upon it
>> > > > >> is installed"?
>> > > > >>
>> > > > >>
>> > > > >> did you run `opam update -u <package>`? If a or dev or pinned
>> package
>> > > > >> changes it should normally trigger a recompilation of all the
>> reverse
>> > > > >> dependencies. how did you specify the packages in your repo?
>> > > > >>
>> > > > >> Thomas
>> > > > >>
>> > > > >>
>> > > > >> Thanks.
>> > > > >>
>> > > > >> Trevor
>> > > > >>
>> > > > >> On Fri, May 8, 2015 at 2:24 PM, Daniel Bünzli <
>> > > > >> daniel.buenzli at erratique.ch> wrote:
>> > > > >>
>> > > > >>> Le vendredi, 8 mai 2015 à 20:09, Ashish Agarwal a écrit :
>> > > > >>> > Louis, thanks for your suggestions. I'm trying them out, but
>> one
>> > > quick
>> > > > >>> question: how can you query with tags. I tried `opam list -e
>> > > foobar`, and I
>> > > > >>> seem to get the same output no matter what I write for foobar.
>> > > > >>>
>> > > > >>> This is not opam tags this is depexts tags (that correspond to
>> > > > >>> platform). You can do for example:
>> > > > >>>
>> > > > >>>   opam search -s org:erratique
>> > > > >>>
>> > > > >>> But it may not be entirely precise since opam-search matches not
>> > > only in
>> > > > >>> tags. I think opam-list should be able to filter by tags (I
>> actually
>> > > > >>> thought this was possible).
>> > > > >>>
>> > > > >>> Best,
>> > > > >>>
>> > > > >>> Daniel
>> > > > >>>
>> > > > >>>
>> > > > >>> _______________________________________________
>> > > > >>> Platform mailing list
>> > > > >>> Platform at lists.ocaml.org
>> > > > >>> http://lists.ocaml.org/listinfo/platform
>> > > > >>>
>> > > > >>
>> > > > >> _______________________________________________
>> > > > >> Platform mailing list
>> > > > >> Platform at lists.ocaml.org
>> > > > >> http://lists.ocaml.org/listinfo/platform
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/platform/attachments/20150519/c1c89226/attachment-0001.html>