[opam-devel] opam2{rpm,deb}
Török Edwin
edwin+ml-ocaml at etorok.net
Sat Mar 21 11:41:12 GMT 2015
On 03/21/2015 11:39 AM, Sylvain Le Gall wrote:
>
>
> Le sam. 21 mars 2015 à 10:35, Sylvain Le Gall <sylvain+ocaml at le-gall.net <mailto:sylvain%2Bocaml at le-gall.net>> a écrit :
>
>
>
> Le sam. 21 mars 2015 à 00:22, Török Edwin <edwin+ml-ocaml at etorok.net <mailto:edwin%2Bml-ocaml at etorok.net>> a écrit :
>
> Hi,
>
> [On Anil's suggestion I'm moving the discussion to opam-devel.
> TL;DR: I am about to start writing opam2{rpm,deb} for my package and its dependencies. Does anyone else need such a tool or want to help?]
>
> I am about to package my own OCaml application for Fedora and Debian, which indirectly depends on 12 opam packages
> (3 of which already review requests on Fedora's bugzilla as they are shared dependencies with opam itself).
> I am looking for some automation in preparing spec and Debian templates, for checking their quality, and tracking the packages through their lifecycle,
> and to avoid making the same common mistakes for each package.
>
> Since such a tool (opam2rpm/opam2deb) doesn't exist now, I'll describe what my idea/requirements are.
> Please let me know what you think and if you have new / different requirements. Perhaps together we can design/create an opam2rpm tool that will be useful for more than just a oneshot conversion.
>
> I am quite new to packaging, have just recently went through packaging another (non-OCaml) application for Fedora so I've
> documented a package's (and package maintainer's) states here: https://fedoraproject.org/__wiki__/Fedora_Package_Lifecycle___note__s <https://fedoraproject.org/wiki/Fedora_Package_Lifecycle_notes>.
> This is still rather incomplete and rough, take it with a grain of salt and consult the official docs when in doubt.
>
> What we have now:
> * opam metadata: depexts, conf-* opam packages, build rules, constraints, etc.
> * opam-lib
> * opam-repository, where some packages implicitly assume a "modern" Linux environment (has native compiler, dynlink, certain tools and C headers, etc.).
> For distribution packages all these dependencies will have to be made explicit
> * oasis2deb, generates decent starting point for a Debian package, but fails #3
>
>
> Not sure to understand how oasis2deb fails #3 (test that build requirement is complete).
>
> It actually declares build dependencies for the source package, but it doesn't embed the source of the dependencies (which is by the way forbidden by Debian policy) but depends on what should be the package to do it. I.e. it integrates into the target build system.
>
>
> Sentence got mixed in my head:
> It actually declares build dependencies for the source package, but it doesn't embed the source of the dependencies. This is, by the way, forbidden by Debian policy. So it depends on what the expected build-depends package should be (it tries to derive it from the findlib package name). The overall generated package integrates into the target build system.
>
>
Sorry I should've been more clear that this is not a fault of oasis2debian, but it simply is outside the scope of OASIS to fully specify distribution specific *non-OCaml* dependencies.
For example build-time dependencies on C libraries (zlib, sqlite, etc.), or runtime dependencies on non-OCaml tools.
opam has such metadata (depexts or conf-*) -- although perhaps not fully specified for each package yet -- and in fact it might be possible for opam2deb to just call oasis2debian with --library-dev-extra-depends and other appropriate flags
when the package is actually using oasis.
This is what I meant when I said "detect that oasis is used and simplify some steps"
>
> Example on my dlnasync generated package:
>
> Source: dlnasync
> Section: ocaml
> Priority: optional
> Maintainer: Debian OCaml Maintainers <debian-ocaml-maint at lists.debian.org <mailto:debian-ocaml-maint at lists.debian.org>>
> Uploaders:
> Sylvain Le Gall <sylvain at le-gall.net <mailto:sylvain at le-gall.net>>
> Build-Depends:
> ocaml-nox (>= 3.11.1-3~),
> ocaml-findlib,
> ocaml-base-nox,
> libsekred-ocaml-dev (>= 0.2.0),
> libounit-ocaml-dev (>= 2.0.0),
> libocamlnet-ocaml-dev (>= 3.5.1),
> libfileutils-ocaml-dev (>= 0.4.2),
> libbatteries-ocaml-dev,
> libav-tools,
I think you give a good example here, you had to use --library-dev-extra-depends to add libav-tools here, but where do I store that so I will have it next time
when upstream releases a new version and the deb package has to be updated?
opam metadata seems like a good place to store that information that benefits other tools as well (opam-installext, or the yet non-existent opam2rpm/oasis2rpm).
This is also my point #4, although oasis2debian can be rerun and standard diff tools used to compare what changed.
> dh-ocaml (>= 0.9~),
> debhelper (>= 7.0.50~)
> Standards-Version: 3.9.1
> Homepage: http://forge.ocamlcore.org/projects/dlnasync
> Vcs-Git: git://git.debian.org/git/pkg-ocaml-maint/packages/dlnasync.git <http://git.debian.org/git/pkg-ocaml-maint/packages/dlnasync.git>
> Vcs-Browser:
> http://git.debian.org/?p=pkg-ocaml-maint/packages/dlnasync.git
>
> You can see that all build dependencies are here.
>
> I made this decision on purpose. If you want to push this kind of tools to build packages to be distributed by the normal RPM/Deb scheme of your distribution, they need to somehow comply with policy.
>
> (I also created tools that DON'T do that but I don't publish them because they are basic hacks)
Yes, complying with policy is my intention, which is exactly why I want a tool that allows me to easily package all the dozen or so of my dependencies as separate packages, leveraging the already existing
opam/OASIS metadata (and manually improving them where they are not complete).
BTW could you add oasis2debian to opam, or tell me where I can download debian-formats? I forgot where I got it from last time.
>
>
>
> What are my requirements?
> #1 (local) opam repository as input, .spec/Debian as output
> #2 lint the output (both opam and possibly manually edited spec/Debian), there are already external tools for this
> #3 test that the build requirements are complete, i.e. it builds in a minimal environment. there are external tools mock/pbuilder for a clean/official solution,
> or docker for an insecure and somewhat faster solution
> #4 allow to manually modify the generated .spec and debian/ files and still keep track of changes in opam metadata
> #5 allow incremental refinement of the spec/debian rules, don't require everything to be 100% compliant/correct syntax
> #6 allow to override opam metadata (could just git clone the opam repo and use that)
> #7 opam2rpm and opam2deb should be similar to use (i.e. maybe it should be opam2pkg rpm and opam2pkg deb), and I don't necesarely want to limit this to rpm and deb,
> someone is welcome to create an opam2openbsd or opam2brew, etc.
> #8 developing the automation tool shouldn't take (significanlty) longer than manually writing the packaging for those 9 packages :)
> #9 the goal would also be to make it easier for upstream reviewers to review all these packages, at some point we should consider involving them in the discussion once we have some draft tool
> to make sure we generate good quality packages :)
>
> Most (but not all) of the packages I need to write spec files for are based on OASIS, so we could try to load the missing metadata from oasis, or detect that oasis is used
> and simplify some steps, however not all packages use it so I think that starting at the opam metadata is still the right way.
>
> What kind of automation is needed for opam2rpm and opam2deb for me:
> * mapping of opam to Fedora/Debian naming conventions (some opam packages have ocaml- already in them, so take care not to duplicate that)
> * generate .spec and debian/ file templates from opam metadata + some overrides. If this was just a one-way/one-time conversion I'd go with
> something simple (perhaps like ocaml-mustache), however see below
> * check license name compliance (Fedora and Debian have their standard name lists), and make sure the package actually has a license *file*
> * don't need full roundtrip capability between opam and spec/debian, the tool should serve three purposes only:
> * generate initial templates
> * show a smart diff
> * lint the opam metadata / spec file
> * shouldn't require to update the conversion tool just to support an obscure corner-case that can be fixed more easily by hand in the spec file
> * doesn't have to be perfect or support the full features of spec/debian/opam files. lets start with the minimum requirements and add fields/features as we need them
> * its not a substitute for human review, the final output always has to be tested and reviewed by a human
>
> Implementation thoughts:
> * ocaml module describing spec file an debian rules with a toplevel type t and nested records, with type 'a field
> * some form of conversion to/from a string -> string map
> * a lexer that loads that map from the spec/debian files, and a formatter that generates it (ocamllex + format, or just ocaml-re + printf?)
> * please if we can lets avoid camlp4 for this, I'd suggest ppx_deriving+cconv or just manually writing the parse/generate calls for the record fields
> * validation should be optional
> * fields should be easy to declare (name, of_string, to_string, validate, smart_diff), initially all would be raw strings, more parsing/diffing can be added as needed
> * some way to fill the 't' type from opam metadata
> * certain files may not need all this advanced conversion, and might be easier to generate from a text template with substitutions (ocaml-mustache, or mpp?), and just show a textual diff
> (docdiff at character/word level is quite nice)
>
> Automation for tracking a package through its lifecycle (separate tool):
> Since packaging an OCaml application can involve packaging/tracking a large number of dependant libraries some tool to track the status of each package (lifecycle)
> would be useful. This is not specific to OCaml, I'm surprised that upstream distros don't have this already, but anyway using Fedora terminology I want a single CLI tool that can:
> * query yum repo for stable packages
> * query pkgdb for existence of package in other releases
> * query bodhi for pending/testing packages
> * query koji for builds submitted/completed
> * query for existence of scm (git) repo
> * query bugzilla for package review and find out its state (submitted, under review, approved, waiting for scm repo, scm repo created)
> For Debian a similar tool that:
> * query bugs.debian.org <http://bugs.debian.org> for RFP (request for packaging), ITP (intent to package) and its status
> * query NEW queue for uploaded source package
> * query package tracker for buildd status
> * this can also avoid duplicating work, i.e. finding out that someone else is already packaging my package or its dependencies
> * might also be useful for other packages
> * can be done manually by visiting a bunch of webpages in turn and searching on each, but there ought to be some CLI tools to help or at least some (web)API
> * doesn't have to be necesarely written in Ocaml, if we can reuse existing tools and have this just as a shell/python script the better
>
> I wanted to go through the policy docs and write some details here on what is missing from opam metadata, but this mail is long enough and it is late enough already,
> that I thought to start the discussion with what I have so far, and discuss the details as we go.
>
> Once we agree on the major design points of this tool I plan on starting with opam2rpm, but if someone else wants to own it I'd be more than happy with that as well.
>
> Anil also suggested to look at:
> On 03/19/2015 10:15 AM, Anil Madhavapeddy wrote:
> > This annotated changeset about a recent update from Jon might be useful as well:
> >
> > https://github.com/jonludlam/__o__caml4021-buildroot/commit/__9ef9__554c283ea1cda8f64742b73bb6__3807__b791c7#commitcomment-__10071274 <https://github.com/jonludlam/ocaml4021-buildroot/commit/9ef9554c283ea1cda8f64742b73bb63807b791c7#commitcomment-10071274>
> >
>
> Best regards,
> --Edwin
> ___________________________________________________
> opam-devel mailing list
> opam-devel at lists.ocaml.org <mailto:opam-devel at lists.ocaml.org>
> http://lists.ocaml.org/__listinf__o/opam-devel <http://lists.ocaml.org/listinfo/opam-devel>
>
More information about the opam-devel
mailing list