[ocaml-platform] Is it taking too long for OCaml software to become 4.03-compatible? Would release process changes help?

[Gabriel Scherer]

TL;DR:
1. We need to have a "test:" field in OPAM metadata and get packages to use
it.
2. We should do a weekly build of OPAM using the ocaml *trunk* version, to
help find and fix (with the cooperation of compiler developers upstream)
issues before the actual release.
3. If such a weekly build is effective, the policy of "stay in beta as long
as allegedly-installable OPAM packages fail to build on the new release"
would be much easier to accept. We could target *all* opam-repository
packages instead of any subset, and it could take less than a month -- say
two weeks.
4. It would be nice to have an automated tool to add a given version bound
on a set of opam files. (And that is easy to implement.)


Simon Cruanes wrote:

> As usual, the issue is: what set of packages is monitored, and who
> chooses?


In this case there is a canonical answer: all of them (on the
public/central OPAM repository).

Note that OWS only tests installability from metadata, which is a useful
metric, but doesn't help when the metadata is wrong and build actually
fails. Damien has a script to compile all of OPAM that could give
buildability result.

Which gets us to the next point: we really should use a "test:" field in
opam metadata and use that. In the atdgen case for example, the software
would compile fine, but it would segfault at usage -- already in the
testsuite. I think that introducing such a "test:" field should be among
our priorities to improve the quality of the OPAM repository.

I agree with the workflow you propose, as long as the beta-testing period
is long enough -- I suggested "one month" and I think it's a good proposal.
Because we could do this with the one-week-long beta period we had in 4.03,
and it would amount to just adding an upper bound on all software, and that
would better for users but only marginally so -- they don't get a crippled
opam-repository, they get a crippled release with no installable software.

In fact I think we should go further and try to make all of OPAM work with
trunk OCaml versions. We could have a weekly build of all-opam from trunk
and collectively fix the issues. This would help ocaml-trunk development
(being able to compile software to test trunk changes is one pain point
right now), and this would help have working software by the release time
-- maybe we wouldn't even need such a long beta period if we did this
effectively.

I don't add upper bounds myself on my own packages (too tedious to do it on
> 15 old releases), so it should probably be automated.
>

I agree; this should be very easy to do using existing opam scripting
capabilities. Any volunteers?

Thomas Gazagnaire wrote:

> The main issue with 4.03 is that killing camlp4 took a bit longer than
> expected. [...] The jump from 4.01 to 4.02 was much easier in comparison. I
> think that’s instability is a one-off.
>

I don't buy it. I think the rate of OCaml change is accelerating rather
than slowing down. Think of what will happen if we merge some form of
multicore-ocaml in the future; all FFI code will break loose (well maybe
not all, but like Obj.magic in 4.03, enough will).

Martin Jambon:

> Bugs due to incompatibilities with newer versions of OCaml have been
> reported and sometimes fixed by others, and while this is great, I'd rather
> (1) not have these bugs in the first place and (2) fix the bugs before they
> start affecting users. I must say I really lack the motivation for such
> maintenance, compared to developing a new feature that I need.
>

I agree. I think that staring testing packages using trunk rather than
waiting for the release could help with that. First, if the breakage is
under the eyes of the upstream OCaml devs, it may be easier for them to
understand the cause and find the fix -- in my experience helping people
un-break software after a new OCaml version, having a deep knowledge of
said software was rarely necessary, and understanding the compiler was more
useful. Plus it may also encourage people to revert or tame changes that
break user software, reducing the number of changes required of package
maintainers.


On Wed, Jun 29, 2016 at 2:09 PM, Martin Jambon <martin at mjambon.com> wrote:

> First, thank you Gabriel, Damien, and others who've been working hard on
> improving package quality with each new release of OCaml.
>
> I have the chance to be most of the time a passive user of OCaml, i.e. we
> develop a commercial application, and we typically upgrade OCaml only when
> it becomes necessary. Also it's normally not a problem for us to wait for 6
> months to a year to upgrade. We have really no complaints there. I feel
> however that it's a missed opportunity to contribute back to the community.
> We generally don't mind bugs in features that we start using, but we're
> really afraid of regression. So it would be good for us to confidently
> upgrade OCaml as soon as we know it's unlikely to break our application,
> but not delay the upgrade out of fear.
>
> I'm also involved in maintaining a small number of public tools because I
> started them. Bugs due to incompatibilities with newer versions of OCaml
> have been reported and sometimes fixed by others, and while this is great,
> I'd rather (1) not have these bugs in the first place and (2) fix the bugs
> before they start affecting users. I must say I really lack the motivation
> for such maintenance, compared to developing a new feature that I need.
>
> What could be useful is a few user-visible stats on what's installable and
> what's usable in opam, for a given OCaml version. It's the same idea as
> labeling an OCaml/opam distribution as beta, but finer-grained. Each
> package known to have version-specific problems would be labeled as such. A
> package could be in one of a few states such as: OK, Usable but has some
> version-specific bugs, Uninstallable. For the opam user wondering whether
> they should upgrade OCaml, it would be useful to get the percentage of
> packages in each state for a given OCaml version, as well as the list of
> packages in each state and links to the relevant bug reports. Hopefully
> this would help users be more adventurous and make informed decisions on
> whether to upgrade.
>
>
> On 06/29/2016 09:40 AM, Gabriel Scherer wrote:
>
>> Dear platform,
>>
>> TL;DR: I propose that we change the OCaml release process to remain in
>> "beta" stage while there remain OPAM packages that are broken by the new
>> releases and their version bounds have not been updated.
>>
>> I'm a bit frustrated with the time it takes for the OCaml ecosystem to
>> catch up with the 4.03 release -- which has been released two months ago
>> now, on April 25.
>>
>> A large chunk of my personal hacking time following the release was
>> spent on updating Batteries, with a 4.03-compatible release happening on
>> May 12, so two weeks after the release -- a time I would still consider
>> too long for our downstream users waiting to try the 4.03 goodness but
>> held back by a Batteries dependency. I recently helped make atdgen
>> compatible with 4.03 (it's a dependency of IOCaml for example), and just
>> today found out that Coq releases (or its CoqIDE part) did not work on
>> 4.03 (with no upper-bound in opam, it would just miscompile).
>>
>> I find it frustrating because that means that there is a large time
>> window where we advertise (externally, to beginners, etc.) that the best
>> OCaml versions ever is 4.03.0, while running opam from this switch to
>> install any major software whatsoever feels like playing a rigged
>> russian roulette. I feel bad whenever I'm informed that my software (or
>> anyone else's, in fact) does not work by end-users that would rather
>> have spent their day doing something else than trying broken software.
>>
>> Do you agree that this is a problem? What could we do to fix that?
>>
>> When I started writing this email I intended to send it to Damien
>> Doligez, as the release manager of the compiler distribution, with the
>> following suggestion: maybe we could have a *much longer* beta-testing
>> period for OCaml versions that would be used not (only) for finding bugs
>> in the new version, but also to let important pieces of the ecosystem
>> update to the new version. Say, at least a month. We could have a fixed
>> set of software that we monitor, or in fact all of what's in OPAM, and
>> delay the release until they get updated/fixed or their OPAM version
>> bounds get fixed. (Aha, a Platform indeed!)
>>
>> Does that sound like a reasonable idea to you? Should we monitor all
>> OPAM, or only a subset?
>>
>> This suggestion is rooted in the idea that when we announce the release
>> to our end-users, stuff should just work -- or refuse to install itself.
>> Maybe this is an old idea of the SVN times and we're now in the era of
>> continuously-released, always-broken stuff?
>>
>> Finally, there has been all this work on the "OPAM weather services",
>> continuous build of opam packages etc., plus the work that Damien is
>> doing around release time. It doesn't seem to have worked very well for
>> 4.03. Who is monitoring this stuff? Do we need to fix them to make them
>> work better, or to fix our process to use them better? To be honest, I'm
>> never looking at these services -- I just don't think of using them. Can
>> I get an email whenever they detect that my stuff is broken? Say I'm a
>> new package manager, are there guidelines somewhere on what the best
>> practices are and what I should do?
>>
>>
>>
>>
>> _______________________________________________
>> Platform mailing list
>> Platform at lists.ocaml.org
>> http://lists.ocaml.org/listinfo/platform
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/platform/attachments/20160629/40de50a6/attachment.html>