[wg-parallel] About Lwt and Async

[Yaron Minsky]

I'm not sure that the right goal is "to make them evolve so they are
close enough".  If we're going to end up with just one implementation,
it's hard for me to see how it's not going to be Async, because Jane
Street is not going to be able to move away from Async as a practical
matter.

If people sign on to that understanding (a big if), then we should
focus less on evolving the two towards each other, and more on:

- Making Async more hospitable to LWT users
- Making it easier to interoperate between the two in the meantime

In terms of hospitality, there are a few clear things to do that Jane
Street can work on directly.

  - Breaking out Core and Async to have _kernel_ versions (e.g.,
    Core_kernel and Async_kernel) that don't depend on things like
    threads, C-bindings or objects.

  - Rearrange functionality so that you have the maximum available
    functionality when you're only using the kernel.

  - add an "immediate deferred" version of the Deferred monad that has
    the semantics that Lwt users are used to that the RHS of a bind
    whose LHS is filled in fires immediately.

There are also some pieces we're less well equipped to do, in
particular, porting to Windows and Javascript.  The windows part in
particular is perhaps an area where we can get help from OCamlPro.

Here are some other things that might help:

- Have a skin to the Async library that is easier to use without
  Core.  I suspect that opening Async.Std without opening Core.Std is
  already not a complete disaster, but there's probably a less
  invasive wrapper for Core that could help here, and maybe an
  alternate layout for Async.  Anil, do you have a view as to what's
  needed here?  Have you tried Async without opening Core.Std?

- Have a skin for Async that makes transitioning over from Lwt easier.
  I'm not sure whether this makes sense, but it certainly seems
  reasonable.  Maybe we can get some experience porting some existing
  Lwt code over and figure out what the pain points are.

In terms of interop, making an LWT equivalent of Async-RPC certainly
seems appealing.  I don't know if there are any other reasonable
interop tricks that one could want.  You could imagine having an
in-process Lwt/Async interop layer, but I suspect that madness lies
that way....

y

On Mon, Apr 29, 2013 at 6:18 AM, Jeremie Dimino <jdimino at janestreet.com> wrote:
> Hi everybody and welcome to the wg-parallel mailing list,
>
> The aim of this group is to discuss the future of monadic cooperative
> concurrent programing in OCaml.  There are currently two well-known
> libraries for that purpose, lwt and async. Lwt is historically the
> first one, async was developed at Jane Street and has been
> open-sourced recently.
>
> These libraries are quite 'invasive' since they require to write the
> code in a cooperative way.  It is not possible to use both of them in
> the same program and even if a third-party library can sometime be
> made to work with both, it is in general complex and not very
> convenient.
>
> So while they are both important having two such 'core' and
> incompatible libraries is not a very good thing for the OCaml
> community.  Switching brutally from one to the other is not realistic
> since it would require a huge amount of time and efforts.  The idea
> would be to make them evolve so they are close enough and, if
> possible, eventually have only one.
>
> One strategy developped by big projects using both lwt and async such
> as Mirage/Xen is to interact at the RPC level.  This can be done on
> top of lwt and async without modifying them and there are people
> working in this direction.
>
> They are currently two big semantic divergences in the kernel of both
> libraries: execution order and error handling. Plus they are a number
> of other more or less important differences.  I try to describe them
> in the rest of this email.
>
> Semantic of bind
> ----------------
>
> The general idea in lwt is to run the most it can immediatly and
> yielding is always explicit, while in async the precedence is given to
> knowing exactly when the code is going to be executed.
>
> So for [return x >>= f]:
>
> - lwt evaluates [f x] immediatly
> - async returns a non-determined deferred and will evaluate [f x]
> during the next cycle
>
> This semantic of the bind is very important for lwt users, but async
> is already planning to add a second monad with lwt's behavior, so it
> shouldn't be an issue.
>
> Lwt and async also differs regarding Lwt.wait/Ivar.fill but there
> seems to be an agreement that the async's behavior is better and it is
> trivial to change in lwt so that shouldn't be an issue.
>
> Error handling
> --------------
>
> The ['a Lwt.t] type 'inline' the error monad [Value of 'a | Error of
> exn] while in async all errors are explicit.  So the async equivalent
> of ['a Lwt.t] would be [('a, exn) Result.t Deferred.t].
>
> As a result, while most functions in lwt return a [something Lwt.t],
> in async they return a variant or a [something Or_error.t Deferred.t]
> when the result may be an error status.
>
> This may be tedious to handle.
>
> Local storage
> -------------
>
> Lwt allows to define global variables with different values in
> different threads.  This was introduced so logs can display 'thread
> identifiers' but is now also used for other things.  Async internally
> has such a mechanism, it could be adapted and exported.
>
> Scheduler
> ---------
>
> The kernel of lwt can be used without a scheduler, which is important
> for javascript for instance.  The scheduler is part of the lwt.unix
> package and can be run multiple times in the lifetime of a program.
>
> This is not true for async but it is something that can be considered.
>
> Portability
> -----------
>
> The kernel of lwt depends only on the standard library and has no C.
> As a result it is very portable and can be used without problem in
> javascript or any other exotic environment.
>
> The kernel of async (Async_core) currently depends on Core which
> depends on Unix and contains C bindings.  It is planned to split Core
> to get a small kernel for Core and Async.
>
> Lwt is also more portable, for example it runs on Windows.  Windows
> support could be added to Async.
>
> There are questions about whether this should be done like in
> unix/lwt.unix, i.e. emulate unix's behavior on Windows, or binds the
> Win32 API directly and write the dispatching code in OCaml.
>
> General experience
> ------------------
>
> Async is part of the core suite and use core's convention.  Code using
> it starts with:
>
>     open Core.Std
>     open Async.Std
>
> which brings all the core and async world into the environment.
>
> The core style is generally considered better and async has a huge
> number of utilities well integrated together, but switching code to it
> may require a lot of work.
>
> On the other hand lwt is more independent so it is easier to integrate
> it into an existing project.
>
>
>
> In conclusion there is a lot to think about but no blocker.
>
> Jeremie
> _______________________________________________
> wg-parallel mailing list
> wg-parallel at lists.ocaml.org
> http://lists.ocaml.org/listinfo/wg-parallel