[opam-devel] OPAM 1.3 roadmap
Roberto Di Cosmo
roberto at dicosmo.org
Fri Feb 27 08:20:31 GMT 2015
Ok, so this really seems the moment to start the pluging infrastructure.
A first draft proposal for the depext plugin API is in
doc/design/depexts-plugins
we may add one for the sandpoxing API, and then see
what can be shared.
On Fri, Feb 27, 2015 at 10:27:27AM +0900, Louis Gesbert wrote:
> A plugin mechanism was actually within my possible plans for 1.3, with a range of features in view including depexts, better agnosticity (an OCaml plugin), etc. But that would probably be too much for the next milestone.
>
> On the other hand, the entry point for calling external processes is quite clean now, and already has metadata like working dir specified in the call. So adding sandbox permissions to these parameters, and delegating process startup to an OS-dependent new module handling sandboxes could probably be done without adding too much noise to the code.
>
> Cheers,
> Louis
>
> > - Roberto Di Cosmo, 25/02/2015 11:58 -
> > I fully share this point of view: having opam in control of the sandbox
> > provides much finer grain. So, from a functional standpoint, I would not
> > hesitate.
> >
> > What makes me hesitate is that we would need to change the opam code for
> > this, and take into account all possible different mechanisms according to
> > os/platform. This is pretty well doable, if we set up a proper plugin
> > mechanism, and we could share code with a plugin-based depext support, but
> > it requires more effort :-)
> >
> >
> > 2015-02-25 11:54 GMT+01:00 Anil Madhavapeddy <anil at recoil.org>:
> >
> > > I'm also not sure how viable activating it for day-to-day testing except
> > > by advanced users is. However, we can at least get the infrastructure in
> > > place for CI, which is necessary before we need to decide that.
> > >
> > > I forgot one point in my previous mail: why this needs to be in OPAM
> > > instead
> > > of just an external `sandbox` binary. We don't actually want to sandbox
> > > OPAM itself (which we trust!), but rather the packages it invokes. In my
> > > OpenBSD sandbox, I am constraining OPAM and all its future subshells.
> > > Instead, I would prefer to have OPAM invoke an external sandbox only when
> > > it is doing a build, with sufficient arguments to let it know which dirs
> > > are ok to write into.
> > >
> > > This would let a package only write into the specific subdirectory relevant
> > > to the current switch. This actually bites me quite often with pinning,
> > > since a stale OASIS setup.data can cause an installation to write into the
> > > wrong switch if a full clean is not done ahead of time. It also lets us
> > > sandbox network access of builds independently of OPAM, which is needed to
> > > "unplug" an OPAM package build.
> > >
> > > cheers,
> > > Anil
> > >
> > > > On 25 Feb 2015, at 10:48, Roberto Di Cosmo <roberto at dicosmo.org> wrote:
> > > >
> > > > Many thanks Anil, that's precious information:
> > > > this is clearly another very OS-specific feature!
> > > >
> > > > The question is whether we want to add these policies and tools
> > > >
> > > > - outside opam (pro: no changes to opam, cons: opam has no control)
> > > > - or make opam aware of them (pro: opam gets control, cons: various
> > > changes to opam)
> > > >
> > > > For CI testing, the "outside" approach seems to be quite sufficient.
> > > >
> > > > For day to day use, though, I am not sure...
> > > >
> > > > On Tue, Feb 24, 2015 at 04:22:31PM +0000, Anil Madhavapeddy wrote:
> > > >> Hi Roberto, Simon,
> > > >>
> > > >> Sandboxing mechanisms have come along quite a bit in the last few years.
> > > >> It's important to separate the two threat models that we want, since
> > > their
> > > >> use can be quite intrusive if made mandatory. I see two modes of
> > > operation:
> > > >>
> > > >> - A mandatory sandbox in CI testing, where we use it to check that the
> > > >> package isn't violating obvious policies such as network downloads
> > > >> from within the package, or writing outside of ~/.opam or /tmp.
> > > >>
> > > >> - Optional sandbox for day-to-day use by end users. This will catch
> > > >> even malicious behaviours, but imposes a rather heavy support burden.
> > > >>
> > > >> In terms of how to sandbox applications, I know of:
> > > >>
> > > >> - OpenBSD: I use the built-in systrace system call permissions subsystem
> > > >> already with OPAM. I've uploaded my local policy to GitHub at:
> > > >> https://gist.github.com/avsm/8293aa52c6cee772a9cb
> > > >> This policy is used by "systrace opam install foo" and pulls up an
> > > >> interactive dialog if an application attempts to write outside of
> > > >> either ~/.opam or /tmp.
> > > >> See: http://www.openbsd.org/cgi-bin/man.cgi?query=systrace
> > > >> or: http://www.citi.umich.edu/u/provos/systrace/
> > > >> paper: http://www.citi.umich.edu/u/provos/papers/systrace.pdf
> > > >>
> > > >> - FreeBSD: the latest versions have the Capsicum capability system
> > > >> integrated, and the Casper system daemon that grants common operations
> > > >> is also being worked on upstream (e.g. for name resolution)
> > > >> See: https://www.cl.cam.ac.uk/research/security/capsicum/
> > > >>
> > > >> - Linux: As always with Linux, there are a myriad of possible solutions.
> > > >> I'd discourage the use of LD_PRELOAD based solutions since they don't
> > > >> work in several situations reliably (most obviously with static
> > > binaries).
> > > >> The fakeroot-ng project uses ptrace instead, which is more reliable.
> > > >>
> > > >> However, the primary thing we want is to only let the package only
> > > >> write into ~/.opam, and so the mount namespace feature (see
> > > CLONE_NEWNS)
> > > >> may be sufficient for our needs -- a lightweight filesystem container,
> > > >> in effect. David Sheets has done some investigation into this for
> > > >> another project we're working on.
> > > >>
> > > >> - Windows: various, Sandboxie is one solution, but the underlying
> > > >> mechanism is the Windows Integrity Mechanism that was introduced in
> > > >> Vista: https://msdn.microsoft.com/en-us/library/bb625964.aspx
> > > >> This lets applications drop privileges, and is used by the Chrome
> > > >> sandbox (in general, following what Chrome does for sandboxing is a
> > > >> good idea, since their needs are a superset of ours).
> > > >>
> > > >> - MacOS X: The App Sandbox requires code signing, but does almost
> > > >> exactly what we need: it gives each app/user a unique directory
> > > >> into which they can write files. All we should need to do is to
> > > >> set OPAMROOT to that directory, and everything should just work.
> > > >> In practise, this requires some investigation into how the App
> > > >> signing infrastructure works, since I've only seen it done for
> > > >> bundles and not for CLI tools.
> > > >>
> > > >> This is a quick off-the-top-of-my-head survey, but I think it's viable
> > > >> and useful for us to build an `opam sandbox` in the same style as
> > > >> `opam depext` that attempts to invoke a relevant sandboxing mechanism
> > > >> depending on the OS. In the longer term, this will really serve us
> > > >> well as the package database grows.
> > > >>
> > > >> I'm less sure about the viability of recording installed files
> > > >> strictly -- I view thatas an advisory rather than enforcement-based
> > > >> mechanism. The reason I like the "make ~/.opam a git store" is that
> > > >> its possible for applications to write directly into the store as they
> > > >> do right now, but still let us track changes precisely. In fact, if
> > > >> we forbid subshells from writing into `~/.opam/.git`, this would be
> > > >> a production grade solution that also offers instant-rollback in case
> > > >> of compilation errors (no more waiting for a full recompilation of
> > > >> the original dependencies!).
> > > >>
> > > >> cheers,
> > > >> Anil
> > > >>
> > > >>> On 21 Feb 2015, at 09:16, Roberto Di Cosmo <roberto at dicosmo.org>
> > > wrote:
> > > >>>
> > > >>> Anil, Simon, can you provide more details on the sandboxing mechanisms
> > > you know of?
> > > >>>
> > > >>> We looked into all this for Mancoosi years ago; the most complete tool
> > > >>> out there was installwatch (now checkinstall) that hijacks filesystem
> > > modifying
> > > >>> commands using the standard LD_PRELOAD trick and a wrapper for system
> > > calls.
> > > >>> Checkinstall does not alter user priviledges, though, so one sometimes
> > > needed
> > > >>> a combination of fakeroot (that only alter user priviledges) with it.
> > > >>>
> > > >>> The best approach I know of was described in a Master thesis from ...
> > > Cambridge
> > > >>> :-) It was under the supervision of Peter Sewell, and used the ptrace
> > > mechanism
> > > >>> instead of the LD_PRELOAD trick, because LD_PRELOAD is blind to
> > > statically
> > > >>> compiled binaries that have system calls hardcoded, while ptrace gets
> > > them all.
> > > >>>
> > > >>> The dissertation is still available today here
> > > http://robot101.net/files/diss.ps.gz
> > > >>> and contains a very nice discussion of the issues related to
> > > monitoring and
> > > >>> rolling back file system changes performed by a command in the Linux
> > > system.
> > > >>> The source code is also available here
> > > http://robot101.net/files/trace.tar.gz
> > > >>> and one can get in touch with Robert Mcqueen that will be delighted to
> > > see his
> > > >>> work being used.
> > > >>>
> > > >>> Since all this is almost 10 years old, I suppose many exciting new
> > > ideas, tools
> > > >>> and approaches surfaced in the meantime, and I'd really like to know
> > > more.
> > > >>>
> > > >>> Cheers
> > > >>>
> > > >>> --
> > > >>> Roberto
> > > >>>
> > > >>> On Sat, Feb 21, 2015 at 09:37:07AM +0100, Simon Cruanes wrote:
> > > >>>> Sandboxing the build would also be a big security improvement. I
> > > think cabal
> > > >>>> now does it, and signing packages doesn't protect against malicious
> > > or buggy
> > > >>>> packages (see: bumblebee's uninstall target). That also goes hand in
> > > hand with
> > > >>>> file tracking. I don't know how difficult it is, though.
> > > >>>>
> > > >>>> Cheers!
> > > >>>>
> > > >>>> Le 21 février 2015 05:01:56 UTC+01:00, Louis Gesbert
> > > >>>> <louis.gesbert at ocamlpro.com> a écrit :
> > > >>>>
> > > >>>> With 1.2.1 almost out of the door, time has come to review the
> > > roadmap discussed back in December and choose where we'll be going for 1.3.
> > > Original mail attached for reference.
> > > >>>>
> > > >>>>
> > > >>>> The topic that is burning hot at the moment is, specially after the
> > > Debian Haskell build host has been compromised, security: we have no
> > > signing at all at the moment, and we need to improve on this before it
> > > becomes a problem. TUF [1] has devised a sane amount of rules for
> > > repository management and signing that should make it easier to get it
> > > right in OPAM. Hannes has mentionned writing an OCaml implementation for
> > > TUF, which could get very helpful.
> > > >>>>
> > > >>>>
> > > >>>> Also of importance is Windows support. It should at least be
> > > straighforward and documented to get a basic Cygwin setup working. That
> > > goes with adding automated tests (appveyor can now be added in Github
> > > alongside Travis). Related is cleaning up external command usage (even if
> > > not really justified by a Windows
> > > >>>> port only, as David Allsopp pointed out) -- replacing curl calls by
> > > cohttp, use ocaml-fileutils...
> > > >>>>
> > > >>>>
> > > >>>> These are the other main features, that'll probably take more time
> > > if we are to focus eg. on security:
> > > >>>>
> > > >>>> * a plugin mechanism with plugins for example for OCaml (for better
> > > agnosticity), external dependency handling [2], documentation generation...
> > > >>>>
> > > >>>> * a 'provides:' field in OPAM files [3]. This is a loose
> > > requirement if we want to switch the repository to have OCaml itself in a
> > > package (which is already possible, but the system compiler may yet be an
> > > issue).
> > > >>>>
> > > >>>> * More flexible switch handling (multi-switch packages, per-switch
> > > remotes, layered switches for cross-compilation...)
> > > >>>>
> > > >>>> * Tracking of files installed by packages. While unrelated to repo
> > > signing, this might have some security implications, so we might want to
> > > bring it in.
> > > >>>>
> > > >>>> * With file tracking, generating a binary repo (with some
> > > limitations) could be quite
> > > >>>> straight-forward.
> > > >>>>
> > > >>>>
> > > >>>> Which of these do you think is most important ? Have I forgotten
> > > anything ?
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Louis
> > > >>>>
> > > >>>>
> > > >>>> [1] http://theupdateframework.com/
> > > >>>> [2]
> > > https://github.com/ocaml/opam/blob/master/doc/design/depexts-plugins
> > > >>>> [3]
> > > https://github.com/ocaml/opam/blob/master/doc/design/provides.md
> > > >>>>
> > > >>>> message suivi
> > > >>>>
> > > >>>> De : Louis Gesbert
> > > >>>> À : opam-devel at lists.ocaml.org
> > > >>>> Envoyé : Wed Dec 17 11:07:40 UTC+01:00 2014
> > > >>>> Objet : [opam-devel] OPAM Roadmap -- what next ?
> > > >>>>
> > > >>>> Hi all,
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> with some lag after the 1.2 release, I'd like to open some space
> > > for a
> > > >>>> collective discussion of the priorities for where the energies
> > > should go
> > > >>>> next. We have mainly 3 directions for improvements: first,
> > > portability,
> > > >>>> with the main goal of a Windows version. Second, agnosticity, with
> > > the goal
> > > >>>> to make OPAM more generic, and try and open it to users outside of
> > > the
> > > >>>> OCaml community (wouldn't OPAM for Haskell be fun ?). Third, there
> > > are
> > > >>>> always lots of features and improvements that have been asked for,
> > > and
> > > >>>> would improve the experience of current users.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> So here is a summary of what I've gathered. Feel free to add your
> > > own.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ## Portability
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Rewrite parallel command engine.** / done.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Native system manipulation (cp, rm, curl...).**
> > > >>>>
> > > >>>> These are mostly done through calls to the shell or external
> > > programs. It's
> > > >>>>
> > > >>>> not very pretty but quite pragmatic actually... until we want to
> > > run on
> > > >>>>
> > > >>>> Windows without Cygwin. Anyway, this wouldn't be the end of
> > > portability
> > > >>>>
> > > >>>> issues.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Windows support.**
> > > >>>>
> > > >>>> for OPAM itself to begin with. This should be pretty close.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Packages on Windows.**
> > > >>>>
> > > >>>> Locate common issues and attempt to find generic fixes.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - Allow **direct use of more solvers** or solver servers.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Support cross-compilation**
> > > >>>>
> > > >>>> This is quite an open issue at the moment.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ## Agnosticity
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Isolate OCaml-specific stuff.**
> > > >>>>
> > > >>>> E.g. specific opam-file variables. See ocaml-specific.md
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Separate as plugins**
> > > >>>>
> > > >>>> To open the gate to OPAM without these plugins, or with other ones
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Compilers as packages.**
> > > >>>>
> > > >>>> This should mostly work now, but needs some tests and
> > > strengthening. The
> > > >>>> main
> > > >>>>
> > > >>>> thing still to do is to handle the system compiler changes properly
> > > ; that
> > > >>>>
> > > >>>> part should be made more generic anyway (see discussion on hooks)
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ## Features
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - A **provides** field. Generally useful, but particulary so with
> > > >>>>
> > > >>>> compilers-as-packages
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - Releasing the **"features" field** for easier package
> > > configuration
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Track installed files**
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Improve security**: just checking md5s is quite light ; package
> > > scripts
> > > >>>> can
> > > >>>>
> > > >>>> write anywhere
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **OS-specific handling of dependencies** (eg dependencies on
> > > packages
> > > >>>> from the
> > > >>>>
> > > >>>> OS) with plugins (#1519)
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - Specify and implement **hooks**
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - Find a nicer way to **share dev repos** / undoable "pinning
> > > sources"
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Per-switch remotes**
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - **Multi-switch packages**
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - Support for (automatic generation of) **binary packages**
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> - Nicer **ocamlfind interaction**
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Cheers,
> > > >>>>
> > > >>>> Louis Gesbert
> > > >>>>
> > > >>>>
> > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> > > >>>>
> > > >>>> opam-devel mailing list
> > > >>>> opam-devel at lists.ocaml.org
> > > >>>> http://lists.ocaml.org/listinfo/opam-devel
> > > >>>>
> > > >>>>
> > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> > > >>>>
> > > >>>> opam-devel mailing list
> > > >>>> opam-devel at lists.ocaml.org
> > > >>>> http://lists.ocaml.org/listinfo/opam-devel
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Simon
> > > >>>
> > > >>>> _______________________________________________
> > > >>>> opam-devel mailing list
> > > >>>> opam-devel at lists.ocaml.org
> > > >>>> http://lists.ocaml.org/listinfo/opam-devel
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Roberto Di Cosmo
> > > >>>
> > > >>> ------------------------------------------------------------------
> > > >>> Professeur En delegation a l'INRIA
> > > >>> PPS E-mail: roberto at dicosmo.org
> > > >>> Universite Paris Diderot WWW : http://www.dicosmo.org
> > > >>> Case 7014 Tel : ++33-(0)1-57 27 92 20
> > > >>> 5, Rue Thomas Mann
> > > >>> F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo
> > > >>> FRANCE. Twitter: http://twitter.com/rdicosmo
> > > >>> ------------------------------------------------------------------
> > > >>> Attachments:
> > > >>> MIME accepted, Word deprecated
> > > >>> http://www.gnu.org/philosophy/no-word-attachments.html
> > > >>> ------------------------------------------------------------------
> > > >>> Office location:
> > > >>>
> > > >>> Bureau 3020 (3rd floor)
> > > >>> Batiment Sophie Germain
> > > >>> Avenue de France
> > > >>> Metro Bibliotheque Francois Mitterrand, ligne 14/RER C
> > > >>> -----------------------------------------------------------------
> > > >>> GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3
> > > >>>
> > > >>
> > > >
> > > > --
> > > > Roberto Di Cosmo
> > > >
> > > > ------------------------------------------------------------------
> > > > Professeur En delegation a l'INRIA
> > > > PPS E-mail: roberto at dicosmo.org
> > > > Universite Paris Diderot WWW : http://www.dicosmo.org
> > > > Case 7014 Tel : ++33-(0)1-57 27 92 20
> > > > 5, Rue Thomas Mann
> > > > F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo
> > > > FRANCE. Twitter: http://twitter.com/rdicosmo
> > > > ------------------------------------------------------------------
> > > > Attachments:
> > > > MIME accepted, Word deprecated
> > > > http://www.gnu.org/philosophy/no-word-attachments.html
> > > > ------------------------------------------------------------------
> > > > Office location:
> > > >
> > > > Bureau 3020 (3rd floor)
> > > > Batiment Sophie Germain
> > > > Avenue de France
> > > > Metro Bibliotheque Francois Mitterrand, ligne 14/RER C
> > > > -----------------------------------------------------------------
> > > > GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3
> > > >
> > >
> > >
> >
> >
> >
--
Roberto Di Cosmo
------------------------------------------------------------------
Professeur En delegation a l'INRIA
PPS E-mail: roberto at dicosmo.org
Universite Paris Diderot WWW : http://www.dicosmo.org
Case 7014 Tel : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo
FRANCE. Twitter: http://twitter.com/rdicosmo
------------------------------------------------------------------
Attachments:
MIME accepted, Word deprecated
http://www.gnu.org/philosophy/no-word-attachments.html
------------------------------------------------------------------
Office location:
Bureau 3020 (3rd floor)
Batiment Sophie Germain
Avenue de France
Metro Bibliotheque Francois Mitterrand, ligne 14/RER C
-----------------------------------------------------------------
GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3
More information about the opam-devel
mailing list