[opam-devel] OPAM on Windows (1.3 Roadmap)

[David Allsopp]

For OCaml 4.02, instead of spending the time I would normally use on a
compiler upgrade bringing my private build scripts up-to-date, I've been
doing some work porting OPAM to Windows. 

I'm chipping away at this - mainly at 'weekends'! - and will publish to
dra27/opam on GitHub hopefully within the next fortnight (if someone else is
also actively working on a Windows port and would like to see the stuff
sooner, then ping me a message - there're just too many hacks on the TODO
list to bother with pushing commits at the moment). So far I've got opam
init working (including installing the base packages) though I'm still
polishing some of the scripts for opam config env. Next up is being able to
switch to another compiler instance and install some packages, but I'm
feeling that the bulk of the hard work is already done now.

All that said, the main purpose of this message is to offer some comment on
the portability section of the 1.3 Roadmap
(https://github.com/ocaml/opam/wiki/1.3-Roadmap) which I hope will be
helpful.

* Rewrite the parallel command engine
This is a 'de facto' requirement for porting to Windows, because it's used
to install the base packages and removing it was easy. My own code only
avoids Unix.fork (and Unix.wait) for Windows, but it would be beneficial for
both Unix and Windows to use the same code path, obviously. I simply
converted everything to Marshal - the child closure is sent on stdin to
additional opam processes (using a special --fork command line switch). It
seems to be working, though some subtleties with global variables might rear
their heads when I start installing packages and trying to push the --jobs
switch later.

* Native system manipulation
While I think invocations of wget/curl could beneficially be replaced with
ocamlnet and tar with ocaml-tar, I think replacing all other calls (rm, cp,
etc.) this is a bad idea or, at least it should certainly not be done on the
false altar of making Windows-use better/easier. OPAM is a source-based
package manager - OCaml itself requires a bash-like environment to compile
(for *all* Windows ports, including the Microsoft-based ones) and let's not
forget that a possible bash-free alternative for building OCaml was
subsequently removed from the source tree. And that's before looking at
packages - it's naïve to imagine, given the majority of package authors
being Unix-based in one flavour or another, that there will ever a point
where anything approaching a majority of package build systems are totally
shell-agnostic. It seems to me to be much better to head towards a system
which can expect various standard Unix commands to be available (via Cygwin,
MSYS, GnuWin32, etc.) but aim not to require the actual shell to be Unix
(i.e. work within Command Prompt, PowerShell, etc.). In this model, the aim
is to make Windows sufficiently Unix-like so that supporting Windows becomes
a matter of small configure-style tweaks already required to support various
flavours of Unix anyway. In my experience, packages which end up with two
build systems (the main one, and something hacked for Windows) end up with a
very stale, broken Windows build system, because the maintainer quite
reasonably ends up releasing later versions without testing whether the
Windows build system still works.

* Test & fix on Windows
"Cygwin, then, maybe, native." I'm afraid implies a crucial misunderstanding
of how OCaml-on-Windows works. I haven't checked, but OPAM should already
compile for *Cygwin* because Cygwin is simply Unix. The only thing which may
be broken is the odd configure check (i.e. correctly parsing the output of
uname). But understand that that means compiling OPAM in *Cygwin*'s OCaml
(which is OCaml built using ./configure in the normal way, not by using the
manual files in config/). Having OPAM working for Cygwin is a good idea,
indeed it'd be a good to have a Cygwin package of OPAM if someone were
willing to maintain it. But it's a trivial target - virtually nothing should
require altering, because that's the point of Cygwin. So it's either
"native" (i.e. the MinGW and MSVC ports) or not really worth talking about!

The four Windows ports are cross-compiled using a Cygwin bash shell as the
*environment* (note that a lot of this can be done using an MSYS bash shell
as the environment too). But the compilers are all cross-compilers (although
because the architecture for both compilation and target is the same, most
configure scripts do not realise that they are cross compiling!) - which is
why the build system can use things like "/bin/sh" (which Cygwin C runtime
translates it) but the resulting OCaml environment cannot (because the
Microsoft Visual C Runtime, unsurprisingly, does not). 

Fortunately, it's easy to interact with Cygwin commands (rm, cp, etc.)
because they're all coded to recognise Windows paths if they're specified
with forward slashes (C:/Windows\system32 is a perfectly valid Windows path
and, indeed, for the Command Interpreter, even C:/"W"indows\""system32 is a
valid path!).

My experience (based on my own installation of around 45 or so OCaml and
OCaml-related packages) is that building for Windows usually just involves
overriding some lazy assumptions in build systems (for example, the most
common one being the assumption "we're building on Windows" => "object files
end in .obj") so nothing which can't be quickly committed upstream, or
applied in opam-repository patches. I don't expect there will be "generic"
fixes, beyond trying to cp "foo.exe" if "foo" cannot be found in a .install
file (which I'm coding up anyway in my fork). What may be useful is to have
certain non-OCaml packages packaged up in OPAM for Windows-only - for
example, openssl, pcre and so on to deal with libraries which aren't
normally packaged for Windows.

Related to this, are there general plans at the moment for how OPAM may deal
with cross-compiling? This becomes relevant with four Windows ports - one of
the big advantages with opam (and the whole model of opam config env) is
that it will make it much easier to run all four Windows ports on the same
machine, but it seems very tedious (and difficult to maintain) to need to
specify every single compiler switch 5 times (e.g. 4.02.0, 4.02.0+msvc,
4.02.0+mingw, 4.02.0+msvc64, 4.02.0+mingw64). I was thinking of adding a
Windows-only (for now) --target option to opam switch for an enumeration of
auto/mingw/msvc/mingw64/msvc64. So opam switch 4.02.0+foo --target=mingw64
would build the 64-bit MinGW port of OCaml in ~/.opam/4.02.0+foo+mingw64. It
would obviously also be useful for opam files to be able to reference the
value of --target.


David