[opam-devel] OPAM on Windows (1.3 Roadmap)

[Sylvain Le Gall]

 On Tue, Nov 4, 2014, 08:42 Louis Gesbert <louis.gesbert at ocamlpro.com>
wrote:

Thanks for these comments !

* Rewrite the parallel command engine

This is well underway, I've got a working prototype with limited backends
at the moment.
I'm not relying on fork anymore, but we'll still need some windows way of
implementing `Unix.waitpid 0`; and to add OS switches to the build system.

* Native system manipulation

I agree on re-coding cp, mv etc.; here is also a possible idea, it may be
worth checking if it's practical: make sure we may work if limited to
busybox, and include some Windows version of busybox in the Windows
distribution.


I am about to release a new version of ocaml-fileutils that have better
implementation of cp, mv than the previous version. You may consider using.
POSIX implementation of those command require some work.

* Test & fix on Windows
> "Cygwin, then, maybe, native." I'm afraid implies a crucial
misunderstanding
> of how OCaml-on-Windows works.

I know that Cygwin is hardly a step towards native windows compatibility
and about the cross-compilation tangle. But given the difference in cost, I
think it's worth ensuring Cygwin is well supported before attacking the
bulk of the problem: you quickly trip over tiny issues even with Cygwin at
the moment.

> Related to this, are there general plans at the moment for how OPAM may
deal
> with cross-compiling? This becomes relevant with four Windows ports

There is quite some amount of thought about it, but no plan yet.

@Anil, on the "os" variable, we have "cygwin" and "win32", following
Sys.os_type().

Cheers,
Louis

Le dimanche 2 novembre 2014, 20:48:37 David Allsopp a écrit :
> For OCaml 4.02, instead of spending the time I would normally use on a
> compiler upgrade bringing my private build scripts up-to-date, I've been
> doing some work porting OPAM to Windows.
>
> I'm chipping away at this - mainly at 'weekends'! - and will publish to
> dra27/opam on GitHub hopefully within the next fortnight (if someone else
is
> also actively working on a Windows port and would like to see the stuff
> sooner, then ping me a message - there're just too many hacks on the TODO
> list to bother with pushing commits at the moment). So far I've got opam
> init working (including installing the base packages) though I'm still
> polishing some of the scripts for opam config env. Next up is being able
to
> switch to another compiler instance and install some packages, but I'm
> feeling that the bulk of the hard work is already done now.
>
> All that said, the main purpose of this message is to offer some comment
on
> the portability section of the 1.3 Roadmap
> (https://github.com/ocaml/opam/wiki/1.3-Roadmap) which I hope will be
> helpful.
>
> * Rewrite the parallel command engine
> This is a 'de facto' requirement for porting to Windows, because it's used
> to install the base packages and removing it was easy. My own code only
> avoids Unix.fork (and Unix.wait) for Windows, but it would be beneficial
for
> both Unix and Windows to use the same code path, obviously. I simply
> converted everything to Marshal - the child closure is sent on stdin to
> additional opam processes (using a special --fork command line switch). It
> seems to be working, though some subtleties with global variables might
rear
> their heads when I start installing packages and trying to push the --jobs
> switch later.
>
> * Native system manipulation
> While I think invocations of wget/curl could beneficially be replaced with
> ocamlnet and tar with ocaml-tar, I think replacing all other calls (rm,
cp,
> etc.) this is a bad idea or, at least it should certainly not be done on
the
> false altar of making Windows-use better/easier. OPAM is a source-based
> package manager - OCaml itself requires a bash-like environment to compile
> (for *all* Windows ports, including the Microsoft-based ones) and let's
not
> forget that a possible bash-free alternative for building OCaml was
> subsequently removed from the source tree. And that's before looking at
> packages - it's naïve to imagine, given the majority of package authors
> being Unix-based in one flavour or another, that there will ever a point
> where anything approaching a majority of package build systems are totally
> shell-agnostic. It seems to me to be much better to head towards a system
> which can expect various standard Unix commands to be available (via
Cygwin,
> MSYS, GnuWin32, etc.) but aim not to require the actual shell to be Unix
> (i.e. work within Command Prompt, PowerShell, etc.). In this model, the
aim
> is to make Windows sufficiently Unix-like so that supporting Windows
becomes
> a matter of small configure-style tweaks already required to support
various
> flavours of Unix anyway. In my experience, packages which end up with two
> build systems (the main one, and something hacked for Windows) end up
with a
> very stale, broken Windows build system, because the maintainer quite
> reasonably ends up releasing later versions without testing whether the
> Windows build system still works.
>
> * Test & fix on Windows
> "Cygwin, then, maybe, native." I'm afraid implies a crucial
misunderstanding
> of how OCaml-on-Windows works. I haven't checked, but OPAM should already
> compile for *Cygwin* because Cygwin is simply Unix. The only thing which
may
> be broken is the odd configure check (i.e. correctly parsing the output of
> uname). But understand that that means compiling OPAM in *Cygwin*'s OCaml
> (which is OCaml built using ./configure in the normal way, not by using
the
> manual files in config/). Having OPAM working for Cygwin is a good idea,
> indeed it'd be a good to have a Cygwin package of OPAM if someone were
> willing to maintain it. But it's a trivial target - virtually nothing
should
> require altering, because that's the point of Cygwin. So it's either
> "native" (i.e. the MinGW and MSVC ports) or not really worth talking
about!
>
> The four Windows ports are cross-compiled using a Cygwin bash shell as the
> *environment* (note that a lot of this can be done using an MSYS bash
shell
> as the environment too). But the compilers are all cross-compilers
(although
> because the architecture for both compilation and target is the same, most
> configure scripts do not realise that they are cross compiling!) - which
is
> why the build system can use things like "/bin/sh" (which Cygwin C runtime
> translates it) but the resulting OCaml environment cannot (because the
> Microsoft Visual C Runtime, unsurprisingly, does not).
>
> Fortunately, it's easy to interact with Cygwin commands (rm, cp, etc.)
> because they're all coded to recognise Windows paths if they're specified
> with forward slashes (C:/Windows\system32 is a perfectly valid Windows
path
> and, indeed, for the Command Interpreter, even C:/"W"indows\""system32 is
a
> valid path!).
>
> My experience (based on my own installation of around 45 or so OCaml and
> OCaml-related packages) is that building for Windows usually just involves
> overriding some lazy assumptions in build systems (for example, the most
> common one being the assumption "we're building on Windows" => "object
files
> end in .obj") so nothing which can't be quickly committed upstream, or
> applied in opam-repository patches. I don't expect there will be "generic"
> fixes, beyond trying to cp "foo.exe" if "foo" cannot be found in a
.install
> file (which I'm coding up anyway in my fork). What may be useful is to
have
> certain non-OCaml packages packaged up in OPAM for Windows-only - for
> example, openssl, pcre and so on to deal with libraries which aren't
> normally packaged for Windows.
>
> Related to this, are there general plans at the moment for how OPAM may
deal
> with cross-compiling? This becomes relevant with four Windows ports - one
of
> the big advantages with opam (and the whole model of opam config env) is
> that it will make it much easier to run all four Windows ports on the same
> machine, but it seems very tedious (and difficult to maintain) to need to
> specify every single compiler switch 5 times (e.g. 4.02.0, 4.02.0+msvc,
> 4.02.0+mingw, 4.02.0+msvc64, 4.02.0+mingw64). I was thinking of adding a
> Windows-only (for now) --target option to opam switch for an enumeration
of
> auto/mingw/msvc/mingw64/msvc64. So opam switch 4.02.0+foo --target=mingw64
> would build the 64-bit MinGW port of OCaml in ~/.opam/4.02.0+foo+mingw64.
It
> would obviously also be useful for opam files to be able to reference the
> value of --target.
>
>
> David
>
> _______________________________________________
> opam-devel mailing list
> opam-devel at lists.ocaml.org
> http://lists.ocaml.org/listinfo/opam-devel

_______________________________________________
opam-devel mailing list
opam-devel at lists.ocaml.org
http://lists.ocaml.org/listinfo/opam-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/opam-devel/attachments/20141105/5620b925/attachment.html>