[ocaml-platform] Improving the opam-repository issue tracker

Fri Sep 30 14:36:53 BST 2016

On 30 Sep 2016, at 11:56, Fabrice Le Fessant <Fabrice.Le_fessant at inria.fr> wrote:
> 
> > The top 4 out of 6 contributors to opam-repository work at Docker according to the GitHub states [1], and we spend a significant amount of time maintaining the general coherence of the repository and responding to PRs, and working on CI infrastructure for OPAM (depext, ocaml-ci-scripts, and soon some more pieces).  In the top 20, I see around 10 people associated with OCaml Labs working on opam-repository.
> 
> As I said, there are already a lot of people contributing, but I think it's a very different job from fixing packages on a daily basis. The task of such an engineer would not be to work on the infrastructure, or comment/accept/reject pull-requests, but to fix all broken packages, one after the other, probably, of course, with some priorities (as Gabriel is doing on his spare time using opam-builder).

It is indeed a different job, but if you look at the contributions then you'll notice that it's not just Gabriel doing such maintenance tasks.  Most of the maintainers have done passes at various points to jump past various constraints (camlp4, ppx ast incompat, etc), and this is an ongoing part of the responsibilities of maintaining the OPAM repository.  Some of the bigger merges like Core updates from JS involve quite a lot of manual testing and constraint updating as well, as do adding depexts for new OS distributions and similar tasks, or maintaining support for less popular distributions such as OpenBSD.

So to directly address your implication that the current maintainers aren't doing repository maintenance and only doing dumb merges: it simply isn't true.

What is true is that we need more automation to assist with this process as the repository grows.  OPAM builder is a very positive step in the right direction, but is unfortunately hampered by being contributed to the OPAM community under a different and more restrictive license (for whatever reason) -- and this makes it difficult for the existing OPAM repository maintainers to contribute to.

> > Did you have a concrete suggestion in your below mail for other contributors?  In general we should make it easier for power users to contribute to the OPAM repository, as we will have an increased burden of support when code signing lands (as it requires a quorum among committers).
> 
> I think my suggestion was pretty concrete:

I meant whether you had suggestions for other companies who are relying on opam-repository.  As I pointed out earlier, a significant chunk of existing opam-repository maintenance is _already_ conducted by companies relying on opam-repository for their developments (e.g. a large number of packages come from Jane Street who invest heavily in automation, and Docker has a number of developers in the top 5 contributors).

Personally I think that this is the wrong approach -- as maintainers, we should be investing heavily in leveraging the automation infrastructure available to us and reducing the manual burden of triage, and not trying to find more people to do work that could be scripted and automated.

I'm investigating a few options on the CI front for Windows at the moment, where this will be essential given the vast number of incompatibilities, and will report back to this list with the results as soon as I get a clearer view of what is needed.

> if there are companies relying on the opam-repository for their developments, they could assign one of their engineers to such a task, and make it official, so that it will reassure users on the future of the repository, and enable us to coordinate with these engineers. I had the feeling that such a solution is easier for many companies, compared to contracting OCamlPro to do such a task (i.e. using their internal resources is easier than giving money to another company).

If by "us" you mean OCamlPro, then I've made my concerns very clear several times: releasing automation support for OPAM under a different license from the main OPAM repository makes it very difficult to work together when all the other efforts are under a more liberal LGPLv2 license.  Your alternative seems to be for commercial users to pay for OCamlPro engineers to continue to build more libraries under the alternative AGPLv3 license, which sounds like vendor lockin to me. This is a perfectly valid business model for OCamlPro, but not one which I wish to subscribe to right now :-)

regards,
Anil

> 
> On Fri, Sep 30, 2016 at 11:46 AM Anil Madhavapeddy <anil at recoil.org> wrote:
> The top 4 out of 6 contributors to opam-repository work at Docker according to the GitHub states [1], and we spend a significant amount of time maintaining the general coherence of the repository and responding to PRs, and working on CI infrastructure for OPAM (depext, ocaml-ci-scripts, and soon some more pieces).  In the top 20, I see around 10 people associated with OCaml Labs working on opam-repository.
> 
> Did you have a concrete suggestion in your below mail for other contributors?  In general we should make it easier for power users to contribute to the OPAM repository, as we will have an increased burden of support when code signing lands (as it requires a quorum among committers).
> 
> [1] https://github.com/ocaml/opam-repository/graphs/contributors
> 
> Anil
> 
> > On 30 Sep 2016, at 10:10, Fabrice Le Fessant <Fabrice.Le_fessant at inria.fr> wrote:
> >
> > Maybe it would make sense for some heavy industrial users of OCaml to devote one engineer part-time (one or two days a week, for example) to the maintenance of the opam-repository ? There are already people from different organizations contributing, but mostly on their spare time, so they cannot really spend a lot of time understanding, for every broken package, how to fix the problem. Having engineers, officially dedicated to that task by their organizations, would help a lot. Do you think it would be possible ? I think that the Haskell community as such an engineer working full-time at maintaining the stability of one of their repositories (Hackage ? Stackage ?).
> >
> > On Thu, Sep 29, 2016 at 4:59 PM Anil Madhavapeddy <anil at recoil.org> wrote:
> > There have been a couple of informal get-togethers in Cambridge, and we're happy to host more of course.
> >
> > However, in this instance what the opam-repository needs is fairly simple curation and labelling. Thomas has been working on a GitHub PR library that should be open sourced soon that will help us tag issues more automatically, so we can do a sweep through opam-repository when this is done.
> >
> > In general, the Debian philosophy elsewhere in the thread is correct -- we do not have the resources to be a user support channel, but should keep issues open that are real breakage that can be actioned in the repository.  Thoughts on where feature requests should go are welcome...
> >
> > regards,
> > Anil
> >
> > > On 27 Sep 2016, at 15:55, Gabriel Scherer <gabriel.scherer at gmail.com> wrote:
> > >
> > > Perhaps what is needed is a somewhat tedious day with maintainers in the same (virtual) place, so that (brief) discussions can take place immediately, to control the backlog?
> > >
> > > Maybe for another time, but have opam-repository maintainers and contributors considered having an actual get-together event? Given the current distribution, Cambridge or London could be good starting points. (I'm personally stuck on the wrong side of the Atlantic before January, but in general terms I would consider attending such an event. There would also be interesting discussions to be had regarding opam 2.0 migration and Conex.)
> > >
> > > On Tue, Sep 27, 2016 at 4:48 AM, Thomas Gazagnaire <thomas at gazagnaire.org> wrote:
> > > >> Nowadays I consider it a lost cause when I file an issue on the opam-
> > > >> repository.
> > > >>
> > > >> I think this is an issue.
> > > >>
> > > >> I perfectly understand that from the point of view of repo maintainers the
> > > >> amount of issues (136 now) doesn't entice them to go through the backlog
> > > >> to try to fix or close them. However I believe that if we try to limit the
> > > >> backlog or tag them more appropriately there may be a better chance that
> > > >> issues do not simply get ignored.
> > > >>
> > > >> Going through the least recently updated issues:
> > > >>
> > > >> https://github.com/ocaml/opam-
> > > >> repository/issues?q=is%3Aopen+is%3Aissue+sort%3Aupdated-asc
> > > >>
> > > >> here are a few things that come to mind:
> > > >>
> > > >> 1. Kill that `request for package` tag. Being a developer-oriented package
> > > >> system I don't think the opam repository is the place to ask for
> > > >> packaging, people should ask upstream (I don't say this didn't make sense
> > > >> when opam was a baby).
> > > >> 2. Kill too open ended questions with the `question` tag.
> > > >> 3. Go through the `bug` tag. It seems a lot of old things can be closed.
> > > >
> > > > Agreed - I was briefly involved with Git-for-Windows. I disliked hugely the way the principal maintainer runs that project, but one thing which was very impressive was his rapid triage of issues. For standard FAQ questions, "we" (i.e. a maintainer) should comment with the appropriate FAQ link (number 1 would be advice either to contact upstream or a pointer to the packaging instructions; number 2 would either link to the manual or a general FAQ to open an issue on the appropriate docs repository; etc.) and immediately *close* the issue. It doesn't prevent the poster from commenting a little further, but it removes a "pointless" issue from the list as quickly as possible. Also, if an issue was woefully lacking in required information, the issue was closed, rather than requesting further information and leaving it open. The OP can always re-open the issue having supplied further details (or start a fresh one).
> > > >
> > > > If your issue survives that process, his next stage was tag it and determine who was going to fix it - if it a maintainer volunteers, it's assigned; otherwise if you don't agree to fix it, it's closed at once (happens with feature requests more than bugs, obviously).
> > > >
> > > > Finally, about once a month, he'd go through old issues and ping them for status - and close anything which seemed not to be making progress.
> > > >
> > > > It seems to me that for opam-repository a ruthless model would work well! Or, as we can see, you can't see the wood for them trees...
> > > >
> > > >> 4. There seem to be a lot of old install glitches that I'm sure are no
> > > >> longer relevant.
> > > >> 5. There are a few open issues where people say that the problem is
> > > >> solved, they should be closed...
> > > >>
> > > >> I think we should walk up from the oldest issues and whenever things are
> > > >> are unclear tag them with `scheduled for closure` and comment that without
> > > >> any further feedback in 7 days, the issue will be closed. Also in general
> > > >> it would be nice to introduce tags to distinguish between repo
> > > >> organisation issues like [1] (may be long lived) and end-user repo install
> > > >> failures like [2] (should be short lived).
> > > >
> > > > Perhaps what is needed is a somewhat tedious day with maintainers in the same (virtual) place, so that (brief) discussions can take place immediately, to control the backlog?
> > >
> > > I agree, I rarely look at the issue tracker and its current state makes me quite sad (these two are maybe related). Any help to triage these issues would be greatly appreciated. I will make a quick first scan to close the obvious ones.
> > >
> > > Thomas
> > >
> > > _______________________________________________
> > > Platform mailing list
> > > Platform at lists.ocaml.org
> > > http://lists.ocaml.org/listinfo/platform
> > >
> > > _______________________________________________
> > > Platform mailing list
> > > Platform at lists.ocaml.org
> > > http://lists.ocaml.org/listinfo/platform
> >
> > _______________________________________________
> > Platform mailing list
> > Platform at lists.ocaml.org
> > http://lists.ocaml.org/listinfo/platform
> 