[opam-devel] opam2{rpm,deb}

[Anil Madhavapeddy]

On 20 Mar 2015, at 23:22, Török Edwin <edwin+ml-ocaml at etorok.net> wrote:
> 
> Hi,
> 
> [On Anil's suggestion I'm moving the discussion to opam-devel.
> TL;DR: I am about to start writing opam2{rpm,deb} for my package and its dependencies. Does anyone else need such a tool or want to help?]

Apologies for the delay replying -- I continue to see significant interest in a tool like this for generating standalone installable packages easily for end-user tools.

There seems to be a basic tension in upstream OS packaging these days: to add all the libraries for every language as a package, or to just "vendor" third party libraries and build a standalone binary with fewer dependencies.  In OCaml's case the latter works great, just as with Go, due to the static linking.

> I am about to package my own OCaml application for Fedora and Debian, which indirectly depends on 12 opam packages
> (3 of which already review requests on Fedora's bugzilla as they are shared dependencies with opam itself).
> I am looking for some automation in preparing spec and Debian templates, for checking their quality, and tracking the packages through their lifecycle,
> and to avoid making the same common mistakes for each package.

Out of interest, do you also need Ubuntu PPA (or the Fedora equivalent) support?  The issue with upstreaming libraries is the difficulty of managing updates, and so I'd also like to be able to build a complete PPA archive with the latest OPAM libraries.  This shouldn't be too difficult with a mechanical OPAM transformation, but I suspect there will be lots of versioning issues.

> Since such a tool (opam2rpm/opam2deb) doesn't exist now, I'll describe what my idea/requirements are.
> Please let me know what you think and if you have new / different requirements. Perhaps together we can design/create an opam2rpm tool that will be useful for more than just a oneshot conversion.
> 
> I am quite new to packaging, have just recently went through packaging another (non-OCaml) application for Fedora so I've
> documented a package's (and package maintainer's) states here: https://fedoraproject.org/wiki/Fedora_Package_Lifecycle_notes.
> This is still rather incomplete and rough, take it with a grain of salt and consult the official docs when in doubt.
> 
> What we have now:
> * opam metadata: depexts, conf-* opam packages, build rules, constraints, etc.
> * opam-lib
> * opam-repository, where some packages implicitly assume a "modern" Linux environment (has native compiler, dynlink, certain tools and C headers, etc.).
> For distribution packages all these dependencies will have to be made explicit
> * oasis2deb, generates decent starting point for a Debian package, but fails #3
> 
> What are my requirements?
> #1 (local) opam repository as input, .spec/Debian as output
> #2 lint the output (both opam and possibly manually edited spec/Debian), there are already external tools for this
> #3 test that the build requirements are complete, i.e. it builds in a minimal environment. there are external tools mock/pbuilder for a clean/official solution,
> or docker for an insecure and somewhat faster solution
> #4 allow to manually modify the generated .spec and debian/ files and still keep track of changes in opam metadata
> #5 allow incremental refinement of the spec/debian rules, don't require everything to be 100% compliant/correct syntax
> #6 allow to override opam metadata (could just git clone the opam repo and use that)
> #7 opam2rpm and opam2deb should be similar to use (i.e. maybe it should be opam2pkg rpm and opam2pkg deb), and I don't necesarely want to limit this to rpm and deb,
> someone is welcome to create an opam2openbsd or opam2brew, etc.
> #8 developing the automation tool shouldn't take (significanlty) longer than manually writing the packaging for those 9 packages :)
> #9 the goal would also be to make it easier for upstream reviewers to review all these packages, at some point we should consider involving them in the discussion once we have some draft tool
> to make sure we generate good quality packages :)

Complete agreement so far from me.  What are your thoughts, Louis and Thomas?

> Most (but not all) of the packages I need to write spec files for are based on OASIS, so we could try to load the missing metadata from oasis, or detect that oasis is used
> and simplify some steps, however not all packages use it so I think that starting at the opam metadata is still the right way.

Yeah, the `findlib` file in OPAM was a first step to close the gap between OASIS and OPAM, and we can continue to add automated scripts to keep this information in sync.  It would be good to only depend on one package database for this translation, and OPAM is the outermost layer and so most appropriate.

> What kind of automation is needed for opam2rpm and opam2deb for me:
> * mapping of opam to Fedora/Debian naming conventions (some opam packages have ocaml- already in them, so take care not to duplicate that)
> * generate .spec and debian/ file templates from opam metadata + some overrides. If this was just a one-way/one-time conversion I'd go with
> something simple (perhaps like ocaml-mustache), however see below
> * check license name compliance (Fedora and Debian have their standard name lists), and make sure the package actually has a license *file*
> * don't need full roundtrip capability between opam and spec/debian, the tool should serve three purposes only:
>  * generate initial templates
>  * show a smart diff
>  * lint the opam metadata / spec file
> * shouldn't require to update the conversion tool just to support an obscure corner-case that can be fixed more easily by hand in the spec file
> * doesn't have to be perfect or support the full features of spec/debian/opam files. lets start with the minimum requirements and add fields/features as we need them
> * its not a substitute for human review, the final output always has to be tested and reviewed by a human

Yep, I think the last thing is key, since the OPAM version numbers also don't map perfectly onto the upstream packaging ones.  For Debian or Ubuntu, we'd need to bump the extra versions for any small change to the package, whereas OPAM doesn't track these small changes.

> 
> Implementation thoughts:
> * ocaml module describing spec file an debian rules with a toplevel type t and nested records, with type 'a field
> * some form of conversion to/from a string -> string map
> * a lexer that loads that map from the spec/debian files, and a formatter that generates it (ocamllex + format, or just ocaml-re + printf?)
> * please if we can lets avoid camlp4 for this, I'd suggest ppx_deriving+cconv or just manually writing the parse/generate calls for the record fields
> * validation should be optional
> * fields should be easy to declare (name, of_string, to_string, validate, smart_diff), initially all would be raw strings, more parsing/diffing can be added as needed
> * some way to fill the 't' type from opam metadata
> * certain files may not need all this advanced conversion, and might be easier to generate from a text template with substitutions (ocaml-mustache, or mpp?), and just show a textual diff
> (docdiff at character/word level is quite nice)
> 
> Automation for tracking a package through its lifecycle (separate tool):
> Since packaging an OCaml application can involve packaging/tracking a large number of dependant libraries some tool to track the status of each package (lifecycle)
> would be useful. This is not specific to OCaml, I'm surprised that upstream distros don't have this already, but anyway using Fedora terminology I want a single CLI tool that can:
> * query yum repo for stable packages
> * query pkgdb for existence of package in other releases
> * query bodhi for pending/testing packages
> * query koji for builds submitted/completed
> * query for existence of scm (git) repo
> * query bugzilla for package review and find out its state (submitted, under review, approved, waiting for scm repo, scm repo created)

Jon Ludlam has been working with Koji recently -- any thoughts on the above?

> For Debian a similar tool that:
> * query bugs.debian.org for RFP (request for packaging), ITP (intent to package) and its status
> * query NEW queue for uploaded source package
> * query package tracker for buildd status
> * this can also avoid duplicating work, i.e. finding out that someone else is already packaging my package or its dependencies
> * might also be useful for other packages
> * can be done manually by visiting a bunch of webpages in turn and searching on each, but there ought to be some CLI tools to help or at least some (web)API
> * doesn't have to be necesarely written in Ocaml, if we can reuse existing tools and have this just as a shell/python script the better
> 
> I wanted to go through the policy docs and write some details here on what is missing from opam metadata, but this mail is long enough and it is late enough already,
> that I thought to start the discussion with what I have so far, and discuss the details as we go.
> 
> Once we agree on the major design points of this tool I plan on starting with opam2rpm, but if someone else wants to own it I'd be more than happy with that as well.


I do agree with this all, and that it's the next important stage to make it easier to distribute OCaml applications upstream.  It's quite a bit of work, so perhaps we should transfer this design to the OPAM wiki to make tracking it easier.  Once there is some stability, we can start ticking off various bits of low hanging fruit like the lifecycle query tools, and figure out who can work on what component.

I'm personally keen on seeing this so that we can have binary distributions of Core/Async more easily, as well as distribute the MirageOS command-line more sensibly on Ubuntu and Fedora.  

cheers,
Anil