[ocaml-platform] Dev Version as Package?

[Trevor Smith]

Sorry for the multiple posts. I see now my previous post would not work,
because it was asking to only update the repo (opam update <repo>) and not
the package (opam update <package).

Will post again once a get a full working solution of what I want to do.

Trevor

On Tue, May 19, 2015 at 1:42 PM, Trevor Smith <trevorsummerssmith at gmail.com>
wrote:

> I see this works with a pin just not with a package due to the url to the
> source repo not having this same functionality currently.
>
> So this is why people have the per-package pins -- which is a lot of
> overhead.
>
> It seems to me the only workaround that allows package B to state it wants
> package A to always be updated is to have either a) per-package pin-file or
> b) per-package repo. Both of these workaround seem very not ideal as it
> requires the dependencies for a package to live in multiple files.
>
> Any other thoughts on workarounds for this? Louis I hear your point about
> wanting not a lot of magic in the core commands. However it seems to me
> that this is a use case that isn't currently covered by the system, and,
> again seems to be very useful.
>
> There is also a technical issue with what you propose: solving an
>> installation is based on the current -- static -- metadata. So that
>> metadata can't depend on updating other metadata, or resolving an
>> installation would become a nightmare. That doesn't apply, though, if you
>> only update the package source and not its metadata (like would be done
>> with a repository package pointing to a VC repository, as opposed to a
>> pinned package) -- it would be much easier to handle in that case, although
>> it could become a bit inconsistent.
>
>
> By inconsistent, do you mean functionally, because the VC repo would be
> updated regularly, and the rest of the meta-data wouldn't?
>
> Trevor
>
> On Tue, May 19, 2015 at 1:16 PM, Trevor Smith <
> trevorsummerssmith at gmail.com> wrote:
>
>> Louis,
>>
>> Thanks for your responses. Apologies for my late response. I only just
>> got time to try this all out.
>>
>> I tried out the workarounds but am not seeing it pick up new commits made
>> to a git repo.
>>
>> Example (not using dependencies):
>>
>> 1) Create package "foo" in a custom repo "my-repo". With url file set to
>> a git repo.
>> 2) Add repo.
>> 3) opam install foo
>> 4) Make new commit and push to foo's repo
>> <following your suggested workflow>
>> 5) opam update -u my-repo
>> <this returns Everything as up-to-date as possible.>
>> 6) opam install foo
>> <package is already installed>
>>
>> I think I am missing something from your suggestions -- perhaps this is
>> clearly not going to work because OPAM does not ever check the upstream git
>> repo?). Was that workflow meant to only work with a pinned version?
>>
>> Thanks.
>>
>> Trevor
>>
>> On Sun, May 10, 2015 at 11:13 PM, Louis Gesbert <
>> louis.gesbert at ocamlpro.com> wrote:
>>
>>> > 1) I think that `opam update -u <my-internal-repo>` and then `opam
>>> upgrade`
>>> > would indeed do what I need as a workaround.
>>>
>>> `update -u` is "update then upgrade" so no need for the upgrade
>>> afterwards :)
>>>
>>> > The incidental complexity the it creates isn't that big of a deal for
>>> > one-offs on my box, but when managing a team of devs, and build boxes
>>> one
>>> > wants something straightforward.
>>> > It seems though that to make that work with the build system and to not
>>> > have unintended updates one would then need 2 internal dev repos -- one
>>> > that had all of the git versioned ones, and another repo that had your
>>> > normal x.y.z versions.
>>>
>>> Not sure I get what you are after exactly, but you could also put both
>>> version in the same repository! Making separate repositories would be
>>> useful if you want to easily configure hosts for the dev or release
>>> versions.
>>>
>>> > 2) I like my suggested solution better because I think it adds no
>>> > incidental complexity -- one has to describe what needs describing (ie
>>> "I
>>> > want this dependency always updated"), and no extra lifecycle commands
>>> (ie
>>> > no explicit "update" after one has already pushed code). And adds a
>>> clear
>>> > functionality to the tool.
>>> >
>>> > Agreed that updating a ton of packages could take a while -- but
>>> having a
>>> > clear tool (as in what I am suggesting) to define that gives the user
>>> that
>>> > flexibility -- it's my fault if I've decided to have 100 "alpha"
>>> packages.
>>> > I've asked for it.
>>> >
>>> > So I'm clear -- are you saying that due to how updates are currently
>>> dealt
>>> > with within opam that the feature I'm suggesting probably would not
>>> happen?
>>> > (I haven't taken a look at how opam works).
>>>
>>> Granted, there is already the mentionned-pinned-package-auto-update
>>> trick, but it's quite limited in scope -- I think it's good design
>>> otherwise to keep matters separate and provide atomic commands: the update
>>> of the internal metadata, and the installation of packages, are clearly
>>> distinct operations in OPAM. Higher-level operations can easily be built on
>>> top of it, and possibly added to OPAM, but we shouldn't put too much magic
>>> in the base operations: that makes things more difficult to handle, to
>>> understand and to debug.
>>>
>>> There is also a technical issue with what you propose: solving an
>>> installation is based on the current -- static -- metadata. So that
>>> metadata can't depend on updating other metadata, or resolving an
>>> installation would become a nightmare. That doesn't apply, though, if you
>>> only update the package source and not its metadata (like would be done
>>> with a repository package pointing to a VC repository, as opposed to a
>>> pinned package) -- it would be much easier to handle in that case, although
>>> it could become a bit inconsistent.
>>>
>>> With the fixes I mentionned earlier, a possibility would be to add a
>>> `--update-dev` option to `install` and `upgrade`, meaning something like:
>>> `opam install --update-dev foo`: get all installed development packages
>>> transitively related to `foo` (dependencies or dependent), update them
>>> first, then proceed with the installation, recompiling changed ones.
>>>
>>> For `install`, this would be more or less similar to `opam install foo
>>> $(opam list --rec --required-by foo)`, once we implement the changes to do
>>> related recompilations on install.
>>>
>>> Does that make sense ?
>>>
>>> Best,
>>> Louis
>>>
>>> >
>>> > Thanks.
>>> >
>>> > Trevor
>>> >
>>> > On Sun, May 10, 2015 at 9:07 PM, Louis Gesbert <
>>> louis.gesbert at ocamlpro.com>
>>> > wrote:
>>> >
>>> > > OPAM normally never updates its metadata outside of `opam update`;
>>> this
>>> > > was inconvenient for pins, so a trick was added: when a pinned
>>> package is
>>> > > mentionned explicitely on an install command-line, it is first
>>> updated from
>>> > > its upstream. Updating all the time could get very slow if you have
>>> many
>>> > > packages pinned to their upstream, for example (like I do).
>>> > >
>>> > > As for reinstallations, OPAM does the minimum reinstallations to
>>> guarantee
>>> > > consistency, i.e. it recompiles only packages that depend on a
>>> changed one
>>> > > -- not the other way around. It will, however, mark packages that
>>> were
>>> > > changed after an update for reinstallation. This should work in a
>>> similar
>>> > > way for repository and pinned packages. However, that reinstallation
>>> will
>>> > > only take place on an `opam upgrade` without argument; it would
>>> probably be
>>> > > an improvement to process it as soon as it belongs to the dependency
>>> cone
>>> > > of changed packages.
>>> > >
>>> > > Now for the dirty (and not really working) workarounds:
>>> > > 1. `alias opam="opam update --dev; opam"`. (this would work with the
>>> fix
>>> > > above; and the option --dev doesn't exist at the moment, we only
>>> have an
>>> > > option --repositories. Adding it)
>>> > > 2. Instead of `opam install foo`, fun `opam install $(opam list -s
>>> --rec
>>> > > --required-by foo) foo`. This should trigger updates due to the trick
>>> > > above. The reinstallations are only done on `opam upgrade` though.
>>> > >
>>> > > So indeed, your only real option if you want to keep your stuff
>>> up-to-date
>>> > > is to run `opam update -u` beforehand (possibly with your dev-repo
>>> name, or
>>> > > --dev, depending on whether you use a custom repo or pinned
>>> packages).
>>> > >
>>> > > Now, if we were to handle planned recompilations more often than
>>> only on
>>> > > global `opam upgrade`, would that be enough to handle your use-case
>>> better ?
>>> > >
>>> > > Best,
>>> > > Louis
>>> > >
>>> > > > - Trevor Smith, 10/05/2015 19:55 -
>>> > > > Thanks Thomas.
>>> > > >
>>> > > > Would there be interest in making a feature to "always update this
>>> > > > dependency"? Suggested solution:
>>> > > >
>>> > > > A keyword in the version could trigger "always update". This would
>>> then
>>> > > > apply to both normal packages as well as pins. "Dev" already has
>>> meaning
>>> > > in
>>> > > > opam, perhaps "-alpha". This would then allow me to internally
>>> have a
>>> > > > single repo with normal versions alongside versions that are
>>> actively
>>> > > being
>>> > > > developed and changing multiple times per day. Eg:
>>> > > >
>>> > > > my-repo/my-package.1.0-alpha/url:
>>> > > > src: "ssh://somewhere/my-package.git"
>>> > > >
>>> > > > A dependent package (ie reverse dependency) "program" on:
>>> "my-package" {=
>>> > > > "1.0-alpha" } would then always check for updates and upgrade
>>> my-package
>>> > > > before doing anything with program. A build server could get the
>>> behavior
>>> > > > I'm looking for by "opam install --deps-only program" and getting
>>> > > whatever
>>> > > > is at head of my-package (or of the given branch as the case may
>>> be).
>>> > > >
>>> > > > I think this solution would be a nice composition of the ideas
>>> that opam
>>> > > > already has, and, I think, would be a strict improvement over the
>>> current
>>> > > > situation and does not introduce any incidental complexity. This
>>> thread
>>> > > > seems to indicate that others are interested in such a feature.
>>> > > >
>>> > > > Thoughts? Thanks!
>>> > > >
>>> > > > Trevor
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Sun, May 10, 2015 at 5:50 PM, Thomas Gazagnaire <
>>> > > thomas at gazagnaire.org>
>>> > > > wrote:
>>> > > >
>>> > > > > Consider two packages: lib and program. program depends upon lib.
>>> > > > >
>>> > > > >
>>> > > > > opam install lib
>>> > > > > # Make a new commit to lib and push it
>>> > > > > # Lib is now one commit newer
>>> > > > > # NB I am making _no_ changes to the internal opam repo
>>> > > > > opam install program
>>> > > > >
>>> > > > > lib does _not_ get recompiled.
>>> > > > >
>>> > > > > Is that the information you wanted?
>>> > > > >
>>> > > > >
>>> > > > > yes, when pin/dev packages are modified, you need to tell opam
>>> that you
>>> > > > > want to use the updated version (if available), so you need to
>>> run
>>> > > `opam
>>> > > > > update -u` before `opam install program`. Opam will check if
>>> there are
>>> > > new
>>> > > > > commits and recompile what needs to be recompiled.
>>> > > > >
>>> > > > > Thomas
>>> > > > >
>>> > > > >
>>> > > > > Also: I just tried opam update program and that also did not
>>> pick up
>>> > > the
>>> > > > > fact that lib is git pinned.
>>> > > > >
>>> > > > > Thoughts? Thanks.
>>> > > > >
>>> > > > > Trevor
>>> > > > >
>>> > > > > On Sun, May 10, 2015 at 1:17 PM, Thomas Gazagnaire <
>>> > > thomas at gazagnaire.org>
>>> > > > > wrote:
>>> > > > >
>>> > > > >> I tried setting up the "option 1" -- using a repo url. This
>>> works fine
>>> > > > >> for clean installs, however it does not work for my use case for
>>> > > updates. I
>>> > > > >> didn't realize until I tried it out that this won't update upon
>>> any
>>> > > > >> dependent installation. As noted by Louis, pinning also does not
>>> > > reinstall
>>> > > > >> from a repo url when a dependent install happens -- it only
>>> updates
>>> > > the
>>> > > > >> meta-data.
>>> > > > >>
>>> > > > >> What I would like is a way (ideally within opam) to say "when
>>> this
>>> > > > >> package dependend-upon it should always be checked for update
>>> and
>>> > > upgrade".
>>> > > > >>
>>> > > > >> Am I correct in stating that currently there is no way to mark a
>>> > > package
>>> > > > >> as "update and upgrade this package whenever something that
>>> depends
>>> > > upon it
>>> > > > >> is installed"?
>>> > > > >>
>>> > > > >>
>>> > > > >> did you run `opam update -u <package>`? If a or dev or pinned
>>> package
>>> > > > >> changes it should normally trigger a recompilation of all the
>>> reverse
>>> > > > >> dependencies. how did you specify the packages in your repo?
>>> > > > >>
>>> > > > >> Thomas
>>> > > > >>
>>> > > > >>
>>> > > > >> Thanks.
>>> > > > >>
>>> > > > >> Trevor
>>> > > > >>
>>> > > > >> On Fri, May 8, 2015 at 2:24 PM, Daniel Bünzli <
>>> > > > >> daniel.buenzli at erratique.ch> wrote:
>>> > > > >>
>>> > > > >>> Le vendredi, 8 mai 2015 à 20:09, Ashish Agarwal a écrit :
>>> > > > >>> > Louis, thanks for your suggestions. I'm trying them out, but
>>> one
>>> > > quick
>>> > > > >>> question: how can you query with tags. I tried `opam list -e
>>> > > foobar`, and I
>>> > > > >>> seem to get the same output no matter what I write for foobar.
>>> > > > >>>
>>> > > > >>> This is not opam tags this is depexts tags (that correspond to
>>> > > > >>> platform). You can do for example:
>>> > > > >>>
>>> > > > >>>   opam search -s org:erratique
>>> > > > >>>
>>> > > > >>> But it may not be entirely precise since opam-search matches
>>> not
>>> > > only in
>>> > > > >>> tags. I think opam-list should be able to filter by tags (I
>>> actually
>>> > > > >>> thought this was possible).
>>> > > > >>>
>>> > > > >>> Best,
>>> > > > >>>
>>> > > > >>> Daniel
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> _______________________________________________
>>> > > > >>> Platform mailing list
>>> > > > >>> Platform at lists.ocaml.org
>>> > > > >>> http://lists.ocaml.org/listinfo/platform
>>> > > > >>>
>>> > > > >>
>>> > > > >> _______________________________________________
>>> > > > >> Platform mailing list
>>> > > > >> Platform at lists.ocaml.org
>>> > > > >> http://lists.ocaml.org/listinfo/platform
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > >
>>> > >
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/platform/attachments/20150519/9df6c99c/attachment-0001.html>