[opam-devel] OPAM 1.3 roadmap

[Anil Madhavapeddy]

On 21 Feb 2015, at 04:01, Louis Gesbert <louis.gesbert at ocamlpro.com> wrote:
> 
> With 1.2.1 almost out of the door, time has come to review the roadmap discussed back in December and choose where we'll be going for 1.3. Original mail attached for reference.
> 
> 
> The topic that is burning hot at the moment is, specially after the Debian Haskell build host has been compromised, security: we have no signing at all at the moment, and we need to improve on this before it becomes a problem. TUF [1] has devised a sane amount of rules for repository management and signing that should make it easier to get it right in OPAM. Hannes has mentionned writing an OCaml implementation for TUF, which could get very helpful.
> 
> 
> Also of importance is Windows support. It should at least be straighforward and documented to get a basic Cygwin setup working. That goes with adding automated tests (appveyor can now be added in Github alongside Travis). Related is cleaning up external command usage (even if not really justified by a Windows port only, as David Allsopp pointed out) -- replacing curl calls by cohttp, use ocaml-fileutils...
> 
> 
<...>
> 
> * Tracking of files installed by packages. While unrelated to repo signing, this might have some security implications, so we might want to bring it in.
<...>
> - **Improve security**: just checking md5s is quite light ; package scripts can
>   write anywhere 

I completely agree that this is a very high priority.  Some thoughts in no particular order

- I'd like the automated builds to do the `build` phase without network access, so that we can verify that there is no external network access going on in the normal course of events.

- Registering the files that are installed is going to be quite a large undertaking within the repository, but also makes eventual binary distributions much easier. This is traditionally done by having a 'fakeroot' into which packages are installed, but this requires changing every package.  One interesting option is that we could turn ~/.opam/<switch> into a .git root, and simply version control it using ocaml-git.  This way anything can write into the checkout, and we simply do a `git add` after each installation to figure what changed in the staging area.  It also makes it significantly easier to revert back to an older snapshot.  If filesystem space is a concern, this feature can be deactivated and no git tracking done.

- The `source linux` tags have to go.  Not sure how though...

- Moving to integrate cohttp and ocaml-git do help quite a bit with portability, but also let us fix the tools when they break.  It's particularly frustrating that Curl on MacOS X is currently broken with no obvious way to fix it beyond replacing the system curl.


> These are the other main features, that'll probably take more time if we are to focus eg. on security:
> 
> * a plugin mechanism with plugins for example for OCaml (for better agnosticity), external dependency handling [2], documentation generation...
> 
> * a 'provides:' field in OPAM files [3]. This is a loose requirement if we want to switch the repository to have OCaml itself in a package (which is already possible, but the system compiler may yet be an issue).
> 
> * More flexible switch handling (multi-switch packages, per-switch remotes, layered switches for cross-compilation...)
> 
> * With file tracking, generating a binary repo (with some limitations) could be quite straight-forward.

This is a very comprehensive list, thanks!  The only other branch I don't have much view on is the compilers-as-a-package branch.  While that's a feature that could help out other communities like Coq, I'm not sure how it fits in.  It's going to be very difficult to do more than one major repository migration at a time, and security+Windows is already a lot to take on...

-anil