[opam-devel] OPAM on Windows (1.3 Roadmap)

[Louis Gesbert]

Thanks for these comments !

* Rewrite the parallel command engine

This is well underway, I've got a working prototype with limited backends at the moment.
I'm not relying on fork anymore, but we'll still need some windows way of implementing `Unix.waitpid 0`; and to add OS switches to the build system.

* Native system manipulation

I agree on re-coding cp, mv etc.; here is also a possible idea, it may be worth checking if it's practical: make sure we may work if limited to busybox, and include some Windows version of busybox in the Windows distribution.

* Test & fix on Windows
> "Cygwin, then, maybe, native." I'm afraid implies a crucial misunderstanding
> of how OCaml-on-Windows works.

I know that Cygwin is hardly a step towards native windows compatibility and about the cross-compilation tangle. But given the difference in cost, I think it's worth ensuring Cygwin is well supported before attacking the bulk of the problem: you quickly trip over tiny issues even with Cygwin at the moment.

> Related to this, are there general plans at the moment for how OPAM may deal
> with cross-compiling? This becomes relevant with four Windows ports

There is quite some amount of thought about it, but no plan yet.

@Anil, on the "os" variable, we have "cygwin" and "win32", following Sys.os_type().


Cheers,
Louis


Le dimanche 2 novembre 2014, 20:48:37 David Allsopp a écrit :
> For OCaml 4.02, instead of spending the time I would normally use on a
> compiler upgrade bringing my private build scripts up-to-date, I've been
> doing some work porting OPAM to Windows. 
> 
> I'm chipping away at this - mainly at 'weekends'! - and will publish to
> dra27/opam on GitHub hopefully within the next fortnight (if someone else is
> also actively working on a Windows port and would like to see the stuff
> sooner, then ping me a message - there're just too many hacks on the TODO
> list to bother with pushing commits at the moment). So far I've got opam
> init working (including installing the base packages) though I'm still
> polishing some of the scripts for opam config env. Next up is being able to
> switch to another compiler instance and install some packages, but I'm
> feeling that the bulk of the hard work is already done now.
> 
> All that said, the main purpose of this message is to offer some comment on
> the portability section of the 1.3 Roadmap
> (https://github.com/ocaml/opam/wiki/1.3-Roadmap) which I hope will be
> helpful.
> 
> * Rewrite the parallel command engine
> This is a 'de facto' requirement for porting to Windows, because it's used
> to install the base packages and removing it was easy. My own code only
> avoids Unix.fork (and Unix.wait) for Windows, but it would be beneficial for
> both Unix and Windows to use the same code path, obviously. I simply
> converted everything to Marshal - the child closure is sent on stdin to
> additional opam processes (using a special --fork command line switch). It
> seems to be working, though some subtleties with global variables might rear
> their heads when I start installing packages and trying to push the --jobs
> switch later.
> 
> * Native system manipulation
> While I think invocations of wget/curl could beneficially be replaced with
> ocamlnet and tar with ocaml-tar, I think replacing all other calls (rm, cp,
> etc.) this is a bad idea or, at least it should certainly not be done on the
> false altar of making Windows-use better/easier. OPAM is a source-based
> package manager - OCaml itself requires a bash-like environment to compile
> (for *all* Windows ports, including the Microsoft-based ones) and let's not
> forget that a possible bash-free alternative for building OCaml was
> subsequently removed from the source tree. And that's before looking at
> packages - it's naïve to imagine, given the majority of package authors
> being Unix-based in one flavour or another, that there will ever a point
> where anything approaching a majority of package build systems are totally
> shell-agnostic. It seems to me to be much better to head towards a system
> which can expect various standard Unix commands to be available (via Cygwin,
> MSYS, GnuWin32, etc.) but aim not to require the actual shell to be Unix
> (i.e. work within Command Prompt, PowerShell, etc.). In this model, the aim
> is to make Windows sufficiently Unix-like so that supporting Windows becomes
> a matter of small configure-style tweaks already required to support various
> flavours of Unix anyway. In my experience, packages which end up with two
> build systems (the main one, and something hacked for Windows) end up with a
> very stale, broken Windows build system, because the maintainer quite
> reasonably ends up releasing later versions without testing whether the
> Windows build system still works.
> 
> * Test & fix on Windows
> "Cygwin, then, maybe, native." I'm afraid implies a crucial misunderstanding
> of how OCaml-on-Windows works. I haven't checked, but OPAM should already
> compile for *Cygwin* because Cygwin is simply Unix. The only thing which may
> be broken is the odd configure check (i.e. correctly parsing the output of
> uname). But understand that that means compiling OPAM in *Cygwin*'s OCaml
> (which is OCaml built using ./configure in the normal way, not by using the
> manual files in config/). Having OPAM working for Cygwin is a good idea,
> indeed it'd be a good to have a Cygwin package of OPAM if someone were
> willing to maintain it. But it's a trivial target - virtually nothing should
> require altering, because that's the point of Cygwin. So it's either
> "native" (i.e. the MinGW and MSVC ports) or not really worth talking about!
> 
> The four Windows ports are cross-compiled using a Cygwin bash shell as the
> *environment* (note that a lot of this can be done using an MSYS bash shell
> as the environment too). But the compilers are all cross-compilers (although
> because the architecture for both compilation and target is the same, most
> configure scripts do not realise that they are cross compiling!) - which is
> why the build system can use things like "/bin/sh" (which Cygwin C runtime
> translates it) but the resulting OCaml environment cannot (because the
> Microsoft Visual C Runtime, unsurprisingly, does not). 
> 
> Fortunately, it's easy to interact with Cygwin commands (rm, cp, etc.)
> because they're all coded to recognise Windows paths if they're specified
> with forward slashes (C:/Windows\system32 is a perfectly valid Windows path
> and, indeed, for the Command Interpreter, even C:/"W"indows\""system32 is a
> valid path!).
> 
> My experience (based on my own installation of around 45 or so OCaml and
> OCaml-related packages) is that building for Windows usually just involves
> overriding some lazy assumptions in build systems (for example, the most
> common one being the assumption "we're building on Windows" => "object files
> end in .obj") so nothing which can't be quickly committed upstream, or
> applied in opam-repository patches. I don't expect there will be "generic"
> fixes, beyond trying to cp "foo.exe" if "foo" cannot be found in a .install
> file (which I'm coding up anyway in my fork). What may be useful is to have
> certain non-OCaml packages packaged up in OPAM for Windows-only - for
> example, openssl, pcre and so on to deal with libraries which aren't
> normally packaged for Windows.
> 
> Related to this, are there general plans at the moment for how OPAM may deal
> with cross-compiling? This becomes relevant with four Windows ports - one of
> the big advantages with opam (and the whole model of opam config env) is
> that it will make it much easier to run all four Windows ports on the same
> machine, but it seems very tedious (and difficult to maintain) to need to
> specify every single compiler switch 5 times (e.g. 4.02.0, 4.02.0+msvc,
> 4.02.0+mingw, 4.02.0+msvc64, 4.02.0+mingw64). I was thinking of adding a
> Windows-only (for now) --target option to opam switch for an enumeration of
> auto/mingw/msvc/mingw64/msvc64. So opam switch 4.02.0+foo --target=mingw64
> would build the 64-bit MinGW port of OCaml in ~/.opam/4.02.0+foo+mingw64. It
> would obviously also be useful for opam files to be able to reference the
> value of --target.
> 
> 
> David 
> 
> _______________________________________________
> opam-devel mailing list
> opam-devel at lists.ocaml.org
> http://lists.ocaml.org/listinfo/opam-devel