[opam-devel] OPAM 1.3 roadmap
Roberto Di Cosmo
roberto at dicosmo.org
Wed Feb 25 10:58:09 GMT 2015
I fully share this point of view: having opam in control of the sandbox
provides much finer grain. So, from a functional standpoint, I would not
hesitate.
What makes me hesitate is that we would need to change the opam code for
this, and take into account all possible different mechanisms according to
os/platform. This is pretty well doable, if we set up a proper plugin
mechanism, and we could share code with a plugin-based depext support, but
it requires more effort :-)
2015-02-25 11:54 GMT+01:00 Anil Madhavapeddy <anil at recoil.org>:
> I'm also not sure how viable activating it for day-to-day testing except
> by advanced users is. However, we can at least get the infrastructure in
> place for CI, which is necessary before we need to decide that.
>
> I forgot one point in my previous mail: why this needs to be in OPAM
> instead
> of just an external `sandbox` binary. We don't actually want to sandbox
> OPAM itself (which we trust!), but rather the packages it invokes. In my
> OpenBSD sandbox, I am constraining OPAM and all its future subshells.
> Instead, I would prefer to have OPAM invoke an external sandbox only when
> it is doing a build, with sufficient arguments to let it know which dirs
> are ok to write into.
>
> This would let a package only write into the specific subdirectory relevant
> to the current switch. This actually bites me quite often with pinning,
> since a stale OASIS setup.data can cause an installation to write into the
> wrong switch if a full clean is not done ahead of time. It also lets us
> sandbox network access of builds independently of OPAM, which is needed to
> "unplug" an OPAM package build.
>
> cheers,
> Anil
>
> > On 25 Feb 2015, at 10:48, Roberto Di Cosmo <roberto at dicosmo.org> wrote:
> >
> > Many thanks Anil, that's precious information:
> > this is clearly another very OS-specific feature!
> >
> > The question is whether we want to add these policies and tools
> >
> > - outside opam (pro: no changes to opam, cons: opam has no control)
> > - or make opam aware of them (pro: opam gets control, cons: various
> changes to opam)
> >
> > For CI testing, the "outside" approach seems to be quite sufficient.
> >
> > For day to day use, though, I am not sure...
> >
> > On Tue, Feb 24, 2015 at 04:22:31PM +0000, Anil Madhavapeddy wrote:
> >> Hi Roberto, Simon,
> >>
> >> Sandboxing mechanisms have come along quite a bit in the last few years.
> >> It's important to separate the two threat models that we want, since
> their
> >> use can be quite intrusive if made mandatory. I see two modes of
> operation:
> >>
> >> - A mandatory sandbox in CI testing, where we use it to check that the
> >> package isn't violating obvious policies such as network downloads
> >> from within the package, or writing outside of ~/.opam or /tmp.
> >>
> >> - Optional sandbox for day-to-day use by end users. This will catch
> >> even malicious behaviours, but imposes a rather heavy support burden.
> >>
> >> In terms of how to sandbox applications, I know of:
> >>
> >> - OpenBSD: I use the built-in systrace system call permissions subsystem
> >> already with OPAM. I've uploaded my local policy to GitHub at:
> >> https://gist.github.com/avsm/8293aa52c6cee772a9cb
> >> This policy is used by "systrace opam install foo" and pulls up an
> >> interactive dialog if an application attempts to write outside of
> >> either ~/.opam or /tmp.
> >> See: http://www.openbsd.org/cgi-bin/man.cgi?query=systrace
> >> or: http://www.citi.umich.edu/u/provos/systrace/
> >> paper: http://www.citi.umich.edu/u/provos/papers/systrace.pdf
> >>
> >> - FreeBSD: the latest versions have the Capsicum capability system
> >> integrated, and the Casper system daemon that grants common operations
> >> is also being worked on upstream (e.g. for name resolution)
> >> See: https://www.cl.cam.ac.uk/research/security/capsicum/
> >>
> >> - Linux: As always with Linux, there are a myriad of possible solutions.
> >> I'd discourage the use of LD_PRELOAD based solutions since they don't
> >> work in several situations reliably (most obviously with static
> binaries).
> >> The fakeroot-ng project uses ptrace instead, which is more reliable.
> >>
> >> However, the primary thing we want is to only let the package only
> >> write into ~/.opam, and so the mount namespace feature (see
> CLONE_NEWNS)
> >> may be sufficient for our needs -- a lightweight filesystem container,
> >> in effect. David Sheets has done some investigation into this for
> >> another project we're working on.
> >>
> >> - Windows: various, Sandboxie is one solution, but the underlying
> >> mechanism is the Windows Integrity Mechanism that was introduced in
> >> Vista: https://msdn.microsoft.com/en-us/library/bb625964.aspx
> >> This lets applications drop privileges, and is used by the Chrome
> >> sandbox (in general, following what Chrome does for sandboxing is a
> >> good idea, since their needs are a superset of ours).
> >>
> >> - MacOS X: The App Sandbox requires code signing, but does almost
> >> exactly what we need: it gives each app/user a unique directory
> >> into which they can write files. All we should need to do is to
> >> set OPAMROOT to that directory, and everything should just work.
> >> In practise, this requires some investigation into how the App
> >> signing infrastructure works, since I've only seen it done for
> >> bundles and not for CLI tools.
> >>
> >> This is a quick off-the-top-of-my-head survey, but I think it's viable
> >> and useful for us to build an `opam sandbox` in the same style as
> >> `opam depext` that attempts to invoke a relevant sandboxing mechanism
> >> depending on the OS. In the longer term, this will really serve us
> >> well as the package database grows.
> >>
> >> I'm less sure about the viability of recording installed files
> >> strictly -- I view thatas an advisory rather than enforcement-based
> >> mechanism. The reason I like the "make ~/.opam a git store" is that
> >> its possible for applications to write directly into the store as they
> >> do right now, but still let us track changes precisely. In fact, if
> >> we forbid subshells from writing into `~/.opam/.git`, this would be
> >> a production grade solution that also offers instant-rollback in case
> >> of compilation errors (no more waiting for a full recompilation of
> >> the original dependencies!).
> >>
> >> cheers,
> >> Anil
> >>
> >>> On 21 Feb 2015, at 09:16, Roberto Di Cosmo <roberto at dicosmo.org>
> wrote:
> >>>
> >>> Anil, Simon, can you provide more details on the sandboxing mechanisms
> you know of?
> >>>
> >>> We looked into all this for Mancoosi years ago; the most complete tool
> >>> out there was installwatch (now checkinstall) that hijacks filesystem
> modifying
> >>> commands using the standard LD_PRELOAD trick and a wrapper for system
> calls.
> >>> Checkinstall does not alter user priviledges, though, so one sometimes
> needed
> >>> a combination of fakeroot (that only alter user priviledges) with it.
> >>>
> >>> The best approach I know of was described in a Master thesis from ...
> Cambridge
> >>> :-) It was under the supervision of Peter Sewell, and used the ptrace
> mechanism
> >>> instead of the LD_PRELOAD trick, because LD_PRELOAD is blind to
> statically
> >>> compiled binaries that have system calls hardcoded, while ptrace gets
> them all.
> >>>
> >>> The dissertation is still available today here
> http://robot101.net/files/diss.ps.gz
> >>> and contains a very nice discussion of the issues related to
> monitoring and
> >>> rolling back file system changes performed by a command in the Linux
> system.
> >>> The source code is also available here
> http://robot101.net/files/trace.tar.gz
> >>> and one can get in touch with Robert Mcqueen that will be delighted to
> see his
> >>> work being used.
> >>>
> >>> Since all this is almost 10 years old, I suppose many exciting new
> ideas, tools
> >>> and approaches surfaced in the meantime, and I'd really like to know
> more.
> >>>
> >>> Cheers
> >>>
> >>> --
> >>> Roberto
> >>>
> >>> On Sat, Feb 21, 2015 at 09:37:07AM +0100, Simon Cruanes wrote:
> >>>> Sandboxing the build would also be a big security improvement. I
> think cabal
> >>>> now does it, and signing packages doesn't protect against malicious
> or buggy
> >>>> packages (see: bumblebee's uninstall target). That also goes hand in
> hand with
> >>>> file tracking. I don't know how difficult it is, though.
> >>>>
> >>>> Cheers!
> >>>>
> >>>> Le 21 février 2015 05:01:56 UTC+01:00, Louis Gesbert
> >>>> <louis.gesbert at ocamlpro.com> a écrit :
> >>>>
> >>>> With 1.2.1 almost out of the door, time has come to review the
> roadmap discussed back in December and choose where we'll be going for 1.3.
> Original mail attached for reference.
> >>>>
> >>>>
> >>>> The topic that is burning hot at the moment is, specially after the
> Debian Haskell build host has been compromised, security: we have no
> signing at all at the moment, and we need to improve on this before it
> becomes a problem. TUF [1] has devised a sane amount of rules for
> repository management and signing that should make it easier to get it
> right in OPAM. Hannes has mentionned writing an OCaml implementation for
> TUF, which could get very helpful.
> >>>>
> >>>>
> >>>> Also of importance is Windows support. It should at least be
> straighforward and documented to get a basic Cygwin setup working. That
> goes with adding automated tests (appveyor can now be added in Github
> alongside Travis). Related is cleaning up external command usage (even if
> not really justified by a Windows
> >>>> port only, as David Allsopp pointed out) -- replacing curl calls by
> cohttp, use ocaml-fileutils...
> >>>>
> >>>>
> >>>> These are the other main features, that'll probably take more time
> if we are to focus eg. on security:
> >>>>
> >>>> * a plugin mechanism with plugins for example for OCaml (for better
> agnosticity), external dependency handling [2], documentation generation...
> >>>>
> >>>> * a 'provides:' field in OPAM files [3]. This is a loose
> requirement if we want to switch the repository to have OCaml itself in a
> package (which is already possible, but the system compiler may yet be an
> issue).
> >>>>
> >>>> * More flexible switch handling (multi-switch packages, per-switch
> remotes, layered switches for cross-compilation...)
> >>>>
> >>>> * Tracking of files installed by packages. While unrelated to repo
> signing, this might have some security implications, so we might want to
> bring it in.
> >>>>
> >>>> * With file tracking, generating a binary repo (with some
> limitations) could be quite
> >>>> straight-forward.
> >>>>
> >>>>
> >>>> Which of these do you think is most important ? Have I forgotten
> anything ?
> >>>>
> >>>> Cheers,
> >>>> Louis
> >>>>
> >>>>
> >>>> [1] http://theupdateframework.com/
> >>>> [2]
> https://github.com/ocaml/opam/blob/master/doc/design/depexts-plugins
> >>>> [3]
> https://github.com/ocaml/opam/blob/master/doc/design/provides.md
> >>>>
> >>>> message suivi
> >>>>
> >>>> De : Louis Gesbert
> >>>> À : opam-devel at lists.ocaml.org
> >>>> Envoyé : Wed Dec 17 11:07:40 UTC+01:00 2014
> >>>> Objet : [opam-devel] OPAM Roadmap -- what next ?
> >>>>
> >>>> Hi all,
> >>>>
> >>>>
> >>>>
> >>>> with some lag after the 1.2 release, I'd like to open some space
> for a
> >>>> collective discussion of the priorities for where the energies
> should go
> >>>> next. We have mainly 3 directions for improvements: first,
> portability,
> >>>> with the main goal of a Windows version. Second, agnosticity, with
> the goal
> >>>> to make OPAM more generic, and try and open it to users outside of
> the
> >>>> OCaml community (wouldn't OPAM for Haskell be fun ?). Third, there
> are
> >>>> always lots of features and improvements that have been asked for,
> and
> >>>> would improve the experience of current users.
> >>>>
> >>>>
> >>>>
> >>>> So here is a summary of what I've gathered. Feel free to add your
> own.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> ## Portability
> >>>>
> >>>>
> >>>>
> >>>> - **Rewrite parallel command engine.** / done.
> >>>>
> >>>>
> >>>>
> >>>> - **Native system manipulation (cp, rm, curl...).**
> >>>>
> >>>> These are mostly done through calls to the shell or external
> programs. It's
> >>>>
> >>>> not very pretty but quite pragmatic actually... until we want to
> run on
> >>>>
> >>>> Windows without Cygwin. Anyway, this wouldn't be the end of
> portability
> >>>>
> >>>> issues.
> >>>>
> >>>>
> >>>>
> >>>> - **Windows support.**
> >>>>
> >>>> for OPAM itself to begin with. This should be pretty close.
> >>>>
> >>>>
> >>>>
> >>>> - **Packages on Windows.**
> >>>>
> >>>> Locate common issues and attempt to find generic fixes.
> >>>>
> >>>>
> >>>>
> >>>> - Allow **direct use of more solvers** or solver servers.
> >>>>
> >>>>
> >>>>
> >>>> - **Support cross-compilation**
> >>>>
> >>>> This is quite an open issue at the moment.
> >>>>
> >>>>
> >>>>
> >>>> ## Agnosticity
> >>>>
> >>>>
> >>>>
> >>>> - **Isolate OCaml-specific stuff.**
> >>>>
> >>>> E.g. specific opam-file variables. See ocaml-specific.md
> >>>>
> >>>>
> >>>>
> >>>> - **Separate as plugins**
> >>>>
> >>>> To open the gate to OPAM without these plugins, or with other ones
> >>>>
> >>>>
> >>>>
> >>>> - **Compilers as packages.**
> >>>>
> >>>> This should mostly work now, but needs some tests and
> strengthening. The
> >>>> main
> >>>>
> >>>> thing still to do is to handle the system compiler changes properly
> ; that
> >>>>
> >>>> part should be made more generic anyway (see discussion on hooks)
> >>>>
> >>>>
> >>>>
> >>>> ## Features
> >>>>
> >>>>
> >>>>
> >>>> - A **provides** field. Generally useful, but particulary so with
> >>>>
> >>>> compilers-as-packages
> >>>>
> >>>>
> >>>>
> >>>> - Releasing the **"features" field** for easier package
> configuration
> >>>>
> >>>>
> >>>>
> >>>> - **Track installed files**
> >>>>
> >>>>
> >>>>
> >>>> - **Improve security**: just checking md5s is quite light ; package
> scripts
> >>>> can
> >>>>
> >>>> write anywhere
> >>>>
> >>>>
> >>>>
> >>>> - **OS-specific handling of dependencies** (eg dependencies on
> packages
> >>>> from the
> >>>>
> >>>> OS) with plugins (#1519)
> >>>>
> >>>>
> >>>>
> >>>> - Specify and implement **hooks**
> >>>>
> >>>>
> >>>>
> >>>> - Find a nicer way to **share dev repos** / undoable "pinning
> sources"
> >>>>
> >>>>
> >>>>
> >>>> - **Per-switch remotes**
> >>>>
> >>>>
> >>>>
> >>>> - **Multi-switch packages**
> >>>>
> >>>>
> >>>>
> >>>> - Support for (automatic generation of) **binary packages**
> >>>>
> >>>>
> >>>>
> >>>> - Nicer **ocamlfind interaction**
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Louis Gesbert
> >>>>
> >>>>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> >>>>
> >>>> opam-devel mailing list
> >>>> opam-devel at lists.ocaml.org
> >>>> http://lists.ocaml.org/listinfo/opam-devel
> >>>>
> >>>>
> ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
> >>>>
> >>>> opam-devel mailing list
> >>>> opam-devel at lists.ocaml.org
> >>>> http://lists.ocaml.org/listinfo/opam-devel
> >>>>
> >>>>
> >>>> --
> >>>> Simon
> >>>
> >>>> _______________________________________________
> >>>> opam-devel mailing list
> >>>> opam-devel at lists.ocaml.org
> >>>> http://lists.ocaml.org/listinfo/opam-devel
> >>>
> >>>
> >>> --
> >>> Roberto Di Cosmo
> >>>
> >>> ------------------------------------------------------------------
> >>> Professeur En delegation a l'INRIA
> >>> PPS E-mail: roberto at dicosmo.org
> >>> Universite Paris Diderot WWW : http://www.dicosmo.org
> >>> Case 7014 Tel : ++33-(0)1-57 27 92 20
> >>> 5, Rue Thomas Mann
> >>> F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo
> >>> FRANCE. Twitter: http://twitter.com/rdicosmo
> >>> ------------------------------------------------------------------
> >>> Attachments:
> >>> MIME accepted, Word deprecated
> >>> http://www.gnu.org/philosophy/no-word-attachments.html
> >>> ------------------------------------------------------------------
> >>> Office location:
> >>>
> >>> Bureau 3020 (3rd floor)
> >>> Batiment Sophie Germain
> >>> Avenue de France
> >>> Metro Bibliotheque Francois Mitterrand, ligne 14/RER C
> >>> -----------------------------------------------------------------
> >>> GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3
> >>>
> >>
> >
> > --
> > Roberto Di Cosmo
> >
> > ------------------------------------------------------------------
> > Professeur En delegation a l'INRIA
> > PPS E-mail: roberto at dicosmo.org
> > Universite Paris Diderot WWW : http://www.dicosmo.org
> > Case 7014 Tel : ++33-(0)1-57 27 92 20
> > 5, Rue Thomas Mann
> > F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo
> > FRANCE. Twitter: http://twitter.com/rdicosmo
> > ------------------------------------------------------------------
> > Attachments:
> > MIME accepted, Word deprecated
> > http://www.gnu.org/philosophy/no-word-attachments.html
> > ------------------------------------------------------------------
> > Office location:
> >
> > Bureau 3020 (3rd floor)
> > Batiment Sophie Germain
> > Avenue de France
> > Metro Bibliotheque Francois Mitterrand, ligne 14/RER C
> > -----------------------------------------------------------------
> > GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3
> >
>
>
--
Roberto Di Cosmo
------------------------------------------------------------------
Professeur En delegation a l'INRIA
PPS E-mail: roberto at dicosmo.org
Universite Paris Diderot WWW : http://www.dicosmo.org
Case 7014 Tel : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo
FRANCE. Twitter: http://twitter.com/rdicosmo
------------------------------------------------------------------
Attachments:
MIME accepted, Word deprecated
http://www.gnu.org/philosophy/no-word-attachments.html
------------------------------------------------------------------
Office location:
Bureau 320 (3rd floor)
Batiment Sophie Germain
Avenue de France
Metro Bibliotheque Francois Mitterrand, ligne 14/RER C
-----------------------------------------------------------------
GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/opam-devel/attachments/20150225/39c58818/attachment-0001.html>
More information about the opam-devel
mailing list