[opam-devel] docker hub update?

[Anil Madhavapeddy]

On 8 Apr 2016, at 16:51, Török Edwin <edwin+ml-ocaml at etorok.net> wrote:
> 
> Hi,
> 
> How often are the ocaml/opam docker containers rebuilt and what determines the build order?

Hi Edwin,

The Docker packages up on ocaml/opam are still somewhat experimental, so please do bear with me as I rearrange things around to make it work more efficiently.  On the other hand, this is a great time to bring up your usecases for them as well so I can make breaking changes in the Hub namespace before announcing them properly.

There are a few goals for the Docker Hub containers:

- make it easy to publish an OCaml-based application (such as your Skylable LibreS3, or Mirage, Irmin, Flowtype, Hack, Infer, etc etc)
- continuous integration with a variety of distros and compiler version variants
- development environments for OCaml

Each of these three are slightly at odds with each other, since:

- for publishing, we want a tiny base package and a statically linked OCaml executable (ideally with musl libc) for small, self-contained containers
- for CI, we want the entire matrix of possible distro/arch/compiler versions
- for development environments, we want the richest possible container with pre-configured Emacs/Vi/Merlin/debuggers.

The current container setup is split into:

- ocaml/ocaml
- ocaml/opam
- ocaml/opam-archive
- ocaml/dev

- The OCaml containers provide a version of OCaml current enough to compile OPAM, but do not include OPAM.
- The OPAM containers provide a comprehensive matrix of all the possible variations, and have a preinitialised OPAM repository embedded in the container.
- The dev ones install a few other things like Merlin and Jenga.

This is all autogenerated from the OCaml Dockerfile DSL via scripts at https://github.com/avsm/ocaml-docker-infra and pushed to GitHub (which in turn triggers a Docker Hub build). The major issue with it currently is that since the OPAM repository checkout is embedded within the container, and so all the containers need to be rebuilt every time to get the latest package database.


To fix this and address every usecase, I'd like to break up the container layers to be more fine-grained:

- ocaml/ocaml - remains as it is right now
- ocaml/opam-base - provides an installed OPAM binary, but _not_ initialised, and has the capability to volume mount an OPAM repository from which it can initialise the local base without any data being held in the container.
- ocaml/opam - embeds an OPAM repository checkout and works as-is right now 
- ocaml/opam-archive: also volume mounts in an external archive/ directory so we do not have to embed the volume in a container

The issue with an out-of-container OPAM archive is that it becomes slightly less easy to use out of the box, since you have to have a Git checkout on the host. Hence my thinking is to leave ocaml/opam easy to use (with a repository checked out) and use ocaml/opam-base for automated CI tools that want finer control over the exact revision of the repository to use.


> E.g. 'ocaml/opam:latest' and 'ocaml/opam:debian' was last updated 9 days ago according to docker hub, but then some images
> were updated a few hours ago. 
> That looks confusing, would it be possible to prioritize building 'ocaml/opam:latest'?

It's difficult to prioritise unfortunately -- Docker (my employer) is sponsoring the builds so that we have up to 100 in parallel, and they are pushed when they were built.

I was on vacation last week and it was an errant cron job that did the push that you saw, but the eventual plan is to have a cron job that rebuilds every day.


> As a workaround I can git pull and opam update if I want to install a new package:
> 
> $ sudo docker run -it ocaml/opam opam install qtest.2.2
> [ERROR] No package matches qtest.2.2.
> 
> $ sudo docker run -it ocaml/opam bash -c 'cd /home/opam/opam-repository; git pull && opam update && opam install qtest.2.2'
> [...]
> OK, it installs it


regards,
Anil