[opam-devel] opam2{rpm,deb}

[Sylvain Le Gall]

Le sam. 21 mars 2015 à 10:35, Sylvain Le Gall <sylvain+ocaml at le-gall.net> a
écrit :

>
>
> Le sam. 21 mars 2015 à 00:22, Török Edwin <edwin+ml-ocaml at etorok.net> a
> écrit :
>
> Hi,
>>
>> [On Anil's suggestion I'm moving the discussion to opam-devel.
>> TL;DR: I am about to start writing opam2{rpm,deb} for my package and its
>> dependencies. Does anyone else need such a tool or want to help?]
>>
>> I am about to package my own OCaml application for Fedora and Debian,
>> which indirectly depends on 12 opam packages
>> (3 of which already review requests on Fedora's bugzilla as they are
>> shared dependencies with opam itself).
>> I am looking for some automation in preparing spec and Debian templates,
>> for checking their quality, and tracking the packages through their
>> lifecycle,
>> and to avoid making the same common mistakes for each package.
>>
>> Since such a tool (opam2rpm/opam2deb) doesn't exist now, I'll describe
>> what my idea/requirements are.
>> Please let me know what you think and if you have new / different
>> requirements. Perhaps together we can design/create an opam2rpm tool that
>> will be useful for more than just a oneshot conversion.
>>
>> I am quite new to packaging, have just recently went through packaging
>> another (non-OCaml) application for Fedora so I've
>> documented a package's (and package maintainer's) states here:
>> https://fedoraproject.org/wiki/Fedora_Package_Lifecycle_notes.
>> This is still rather incomplete and rough, take it with a grain of salt
>> and consult the official docs when in doubt.
>>
>> What we have now:
>> * opam metadata: depexts, conf-* opam packages, build rules, constraints,
>> etc.
>> * opam-lib
>> * opam-repository, where some packages implicitly assume a "modern" Linux
>> environment (has native compiler, dynlink, certain tools and C headers,
>> etc.).
>> For distribution packages all these dependencies will have to be made
>> explicit
>> * oasis2deb, generates decent starting point for a Debian package, but
>> fails #3
>>
>
> Not sure to understand how oasis2deb fails #3 (test that build requirement
> is complete).
>
> It actually declares build dependencies for the source package, but it
> doesn't embed the source of the dependencies (which is by the way forbidden
> by Debian policy) but depends on what should be the package to do it. I.e.
> it integrates into the target build system.
>
>
Sentence got mixed in my head:
 It actually declares build dependencies for the source package, but it
doesn't embed the source of the dependencies. This is, by the way,
forbidden by Debian policy. So it depends on what the expected
build-depends package should be (it tries to derive it from the findlib
package name). The overall generated package integrates into the target
build system.



> Example on my dlnasync generated package:
>
> Source: dlnasync
> Section: ocaml
> Priority: optional
> Maintainer: Debian OCaml Maintainers <debian-ocaml-maint at lists.debian.org>
> Uploaders:
>   Sylvain Le Gall <sylvain at le-gall.net>
> Build-Depends:
>   ocaml-nox (>= 3.11.1-3~),
>   ocaml-findlib,
>   ocaml-base-nox,
>   libsekred-ocaml-dev (>= 0.2.0),
>   libounit-ocaml-dev (>= 2.0.0),
>   libocamlnet-ocaml-dev (>= 3.5.1),
>   libfileutils-ocaml-dev (>= 0.4.2),
>   libbatteries-ocaml-dev,
>   libav-tools,
>   dh-ocaml (>= 0.9~),
>   debhelper (>= 7.0.50~)
> Standards-Version: 3.9.1
> Homepage: http://forge.ocamlcore.org/projects/dlnasync
> Vcs-Git: git://git.debian.org/git/pkg-ocaml-maint/packages/dlnasync.git
> Vcs-Browser:
>   http://git.debian.org/?p=pkg-ocaml-maint/packages/dlnasync.git
>
> You can see that all build dependencies are here.
>
> I made this decision on purpose. If you want to push this kind of tools to
> build packages to be distributed by the normal RPM/Deb scheme of your
> distribution, they need to somehow comply with policy.
>
> (I also created tools that DON'T do that but I don't publish them because
> they are basic hacks)
>
>
>>
>> What are my requirements?
>> #1 (local) opam repository as input, .spec/Debian as output
>> #2 lint the output (both opam and possibly manually edited spec/Debian),
>> there are already external tools for this
>> #3 test that the build requirements are complete, i.e. it builds in a
>> minimal environment. there are external tools mock/pbuilder for a
>> clean/official solution,
>> or docker for an insecure and somewhat faster solution
>> #4 allow to manually modify the generated .spec and debian/ files and
>> still keep track of changes in opam metadata
>> #5 allow incremental refinement of the spec/debian rules, don't require
>> everything to be 100% compliant/correct syntax
>> #6 allow to override opam metadata (could just git clone the opam repo
>> and use that)
>> #7 opam2rpm and opam2deb should be similar to use (i.e. maybe it should
>> be opam2pkg rpm and opam2pkg deb), and I don't necesarely want to limit
>> this to rpm and deb,
>> someone is welcome to create an opam2openbsd or opam2brew, etc.
>> #8 developing the automation tool shouldn't take (significanlty) longer
>> than manually writing the packaging for those 9 packages :)
>> #9 the goal would also be to make it easier for upstream reviewers to
>> review all these packages, at some point we should consider involving them
>> in the discussion once we have some draft tool
>> to make sure we generate good quality packages :)
>>
>> Most (but not all) of the packages I need to write spec files for are
>> based on OASIS, so we could try to load the missing metadata from oasis, or
>> detect that oasis is used
>> and simplify some steps, however not all packages use it so I think that
>> starting at the opam metadata is still the right way.
>>
>> What kind of automation is needed for opam2rpm and opam2deb for me:
>>  * mapping of opam to Fedora/Debian naming conventions (some opam
>> packages have ocaml- already in them, so take care not to duplicate that)
>>  * generate .spec and debian/ file templates from opam metadata + some
>> overrides. If this was just a one-way/one-time conversion I'd go with
>> something simple (perhaps like ocaml-mustache), however see below
>>  * check license name compliance (Fedora and Debian have their standard
>> name lists), and make sure the package actually has a license *file*
>>  * don't need full roundtrip capability between opam and spec/debian, the
>> tool should serve three purposes only:
>>   * generate initial templates
>>   * show a smart diff
>>   * lint the opam metadata / spec file
>>  * shouldn't require to update the conversion tool just to support an
>> obscure corner-case that can be fixed more easily by hand in the spec file
>>  * doesn't have to be perfect or support the full features of
>> spec/debian/opam files. lets start with the minimum requirements and add
>> fields/features as we need them
>>  * its not a substitute for human review, the final output always has to
>> be tested and reviewed by a human
>>
>> Implementation thoughts:
>>  * ocaml module describing spec file an debian rules with a toplevel type
>> t and nested records, with type 'a field
>>  * some form of conversion to/from a string -> string map
>>  * a lexer that loads that map from the spec/debian files, and a
>> formatter that generates it (ocamllex + format, or just ocaml-re + printf?)
>>  * please if we can lets avoid camlp4 for this, I'd suggest
>> ppx_deriving+cconv or just manually writing the parse/generate calls for
>> the record fields
>>  * validation should be optional
>>  * fields should be easy to declare (name, of_string, to_string,
>> validate, smart_diff), initially all would be raw strings, more
>> parsing/diffing can be added as needed
>>  * some way to fill the 't' type from opam metadata
>>  * certain files may not need all this advanced conversion, and might be
>> easier to generate from a text template with substitutions (ocaml-mustache,
>> or mpp?), and just show a textual diff
>> (docdiff at character/word level is quite nice)
>>
>> Automation for tracking a package through its lifecycle (separate tool):
>> Since packaging an OCaml application can involve packaging/tracking a
>> large number of dependant libraries some tool to track the status of each
>> package (lifecycle)
>> would be useful. This is not specific to OCaml, I'm surprised that
>> upstream distros don't have this already, but anyway using Fedora
>> terminology I want a single CLI tool that can:
>>  * query yum repo for stable packages
>>  * query pkgdb for existence of package in other releases
>>  * query bodhi for pending/testing packages
>>  * query koji for builds submitted/completed
>>  * query for existence of scm (git) repo
>>  * query bugzilla for package review and find out its state (submitted,
>> under review, approved, waiting for scm repo, scm repo created)
>> For Debian a similar tool that:
>>  * query bugs.debian.org for RFP (request for packaging), ITP (intent to
>> package) and its status
>>  * query NEW queue for uploaded source package
>>  * query package tracker for buildd status
>>  * this can also avoid duplicating work, i.e. finding out that someone
>> else is already packaging my package or its dependencies
>>  * might also be useful for other packages
>>  * can be done manually by visiting a bunch of webpages in turn and
>> searching on each, but there ought to be some CLI tools to help or at least
>> some (web)API
>>  * doesn't have to be necesarely written in Ocaml, if we can reuse
>> existing tools and have this just as a shell/python script the better
>>
>> I wanted to go through the policy docs and write some details here on
>> what is missing from opam metadata, but this mail is long enough and it is
>> late enough already,
>> that I thought to start the discussion with what I have so far, and
>> discuss the details as we go.
>>
>> Once we agree on the major design points of this tool I plan on starting
>> with opam2rpm, but if someone else wants to own it I'd be more than happy
>> with that as well.
>>
>> Anil also suggested to  look at:
>> On 03/19/2015 10:15 AM, Anil Madhavapeddy wrote:
>> > This annotated changeset about a recent update from Jon might be useful
>> as well:
>> >
>> > https://github.com/jonludlam/ocaml4021-buildroot/commit/9ef9
>> 554c283ea1cda8f64742b73bb63807b791c7#commitcomment-10071274
>> >
>>
>> Best regards,
>> --Edwin
>> _______________________________________________
>> opam-devel mailing list
>> opam-devel at lists.ocaml.org
>> http://lists.ocaml.org/listinfo/opam-devel
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/opam-devel/attachments/20150321/e27e08a2/attachment.html>