Making Async play better with others

[Anil Madhavapeddy]

A pluggable engine will give you the lowest common denominator and/or a poorly tested set of alternatives (most users will just one the default one). There are observable differences between the backends (e.g. select returns immediately if you pass in a block device fd, whereas epoll can often wedge forever and never return).

The reason I like libuv is because its most aligned with the Core "full service" feature set, whereas the rest are mostly just network I/O.  Node.js is very similar in terms of its runtime architecture to Core+Async (C non-blocking runtime, language VM activations).  Letting that community deal with platform portability would save a lot of time.
 
I've written servers in both libevent and libev, and they are both pretty good for network AIO. The scalability differences between them won't matter for 99% of users.

Anil

On 1 Nov 2011, at 10:32, Yaron Minsky wrote:

> Poking around a bit more, it looks like libuv uses libev (rather than forking/reimplementing), which makes libuv sound more interesting...
> 
> y
> 
> On Tue, Nov 1, 2011 at 6:30 AM, Yaron Minsky <yminsky at gmail.com> wrote:
> I agree that you want to use one of those libraries if you're programming against epoll and kqueue.  I'm less sure that you should use them in preference to select (which, from both some experiments I think Markus Mottl ran, and other things I've heard is faster for moderate numbers of file descriptors) is less clear.
> 
> But even once you agree you need one of these libraries, it's till not clear to me how to pick amongst libev, libuv and libevent.  libevent seems like the most mature, libev is claimed (by the author anyway) to be more efficient, and libuv seems to treat Windows as a first-class citizen.  This is to some degree why I like the idea of a pluggable engine, which would prevent us from needing to commit to a single answer.
> 
> y
> 
> 
> On Tue, Nov 1, 2011 at 6:24 AM, Dmitry Astapov <dastapov at gmail.com> wrote:
> I read quite a lot about libev, which seems to be an ancestor of libuv
> while I hunted for bugs in the code which had direct bindings for
> epoll and kqueue. And in my experience, libev handles so many bugs and
> quirky situations in underlying polling mechanisms that one must be
> really, really condfident that he is rejecting libev (or any other
> wrapper) for a good reason (that is, that he would do the same job
> better).
> 
> Just search for EVBACKEND_EPOLL and EVBACKEND_KQUEUE here
> http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod and then look at
> the relevant parts of ev_epoll.c or ev_kqueue.c and think if you want
> to have a walk through this particular minefield :)
> 
> On Oct 31, 3:06 pm, Yaron Minsky <ymin... at janestreet.com> wrote:
> > On 31/10/11 14:29, Anil Madhavapeddy wrote:
> >
> > > You could combine most of these by basing Async on top of libuv,
> > > which is a superset of libev (the event library that Lwt uses to
> > > abstract network I/O).
> >
> > >https://github.com/joyent/libuv
> >
> > > Crucially, libuv has good support for the Windows AIO model, and is
> > > also what node.js uses to drive its event loop (so has widespread
> > > usage).
> >
> > > At first glance at Core, it looks like many of the Linux-specific
> > > bits like high-res timers and thread scheduling are also covered by
> > > it.
> >
> > How much do you know about the performance and quality of libuv?  I'm
> > always nervous about shimming yet another layer between us and the
> > bare metal.
> >
> > y
> >
> >
> >
> > > Anil
> >
> > > On 31 Oct 2011, at 14:25, Yaron Minsky wrote:
> >
> > > > I'm wondering if anyone has opinions on what we should do to make
> > > > Async more widely applicable.  Here are some issues that jump to mind.
> >
> > > > - Porting to windows.  People have proposed using libuv to do that.
> > > > - Porting to non-Linux UNIX's: Right now, we depend on Core's
> > > >  Linux_ext module.  I think the dependency is not deep, so
> > > >  conditional compilation might be a way to fix that.
> > > > - Support for other polling mechanisms: like epoll and friends.  Right
> > > >  now, it's all select.  A pluggable engine mechanism like the one Lwt
> > > >  has could make sense here.
> > > > - Interoperability with Lwt:  It would be nice to make Async and Lwt
> > > >  libraries play together.  Jeremie Dimino suggested that the best
> > > >  thing here might be to build a library for Lwt on top of Async.
> > > >  That seems plausible enough, but an engine in the other direction
> > > >  might be good too.
> >
> > > > I'm curious if people have opinions as to how to best handle these.
> > > > I'm in particular wondering about conditional compilation as a
> > > > solution.  It seems messy, but I'm not sure what would be a better
> > > > approach.  One thing you could imagine is using first-class modules,
> > > > where you stored a system-dependent module that provided the
> > > > OS-specific functionality.  But there's some performance compromise
> > > > there (probably too small to matter compared to a system-call,
> > > > though), and I'm not sure it comes out cleaner.
> >
> > > > y
> >
> > > > --
> > > > Yaron Minsky
> >
> > --
> > Yaron Minsky
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/core/attachments/20111102/58990559/attachment.html>