[ocaml-ctypes] Best practices for wrapping ocaml-ctypes?

[Andi McClure]

Hello—

Thanks Jeremy for this response. After some time I've been able to come
back and try these things out. I do have some new questions :) I might
write this out a little more verbosely than I need to just to be certain
I'm speaking precisely.

I have a test program with a c function

    int increment3(int x) { return x + 3; }

On the OCaml side, this is accessed as

    let increment3 = foreign "increment3" (int @-> returning int) in
    increment3 3

And that works and I'm happy with it. Actually, after playing with the
foreign / "@->" interface, I'm finding it seems to be about as good as I
need, and I'm trying to figure out if I can use that instead of having to
work with the low-level interfaces (which as discussed before are not
necessarily directly exposed at this time anyway).

Anyway, I'm trying to set up my OCaml-implemented interpreter, so that this
line of code in Emily is equivalent to the "let increment3" above:

    (* Just pretend .int is shorthand for "int" *)
    increment3 = package.emily.ffi.c.function [ name = .increment3; args =
[ .int, ]; return = .int ]

The syntax could use some work :P But never mind that. I've so far got it
where when you invoke ffi.c.function, the function specification you
provided gets piped into OCaml and turned into this OCaml data structure:

    type foreignSpec = { name : string; args : string list; returning:
string; }

This seems like a straightforward representation of the information @->
encodes; it seems something parsing a header would need to represent the
data the same way.

My naive impulse is to imagine that this is straightforward, and I need to
just build up a typ variable by repeatedly chaining @->, something like:

    let typeConvert = function "void" -> void | "int" -> int | _ ->
failwith "??"
    let functionFrom spec =
        foreign spec.name ( List.fold_right
            ( @-> )
            ( List.map typeConvert wrap.args )
            ( returning ( typeConvert wrap.returning ) ) )

If I start trying to implement this, OCaml immediately objects that even
just typeConvert is impossible to compile:

    Error: This expression has type int Ctypes.typ = int Ctypes_static.typ
           but an expression was expected of type
             unit Ctypes.typ = unit Ctypes_static.typ
           Type int is not compatible with type unit

This seems to make immediate sense: I don't fully understand what the type
of the thing @-> returns is (Merlin says it's '_a -> '_b -> '_c ...) but it
seems like whatever @-> may be doing, in all cases @-> or foreign actually
gets used, the type of that @->'s return must be something known at the
call site at compile time-- because when OCaml code invokes "foreign" OCaml
will need to statically compile a statically typed result. So I can't
possibly store one of the typs in a variable which could contain sometimes
"int" and sometimes "void". I'm assuming this works something like C++
template arguments, which are something I have a little more familiarity
with than OCaml type variables.

It seems like, if I really understood what @-> is doing, I might be able to
maybe make this work anyway. In Emily I currently have only one "type", a
big variant which is modeled in OCaml like:

type value = Null | True | FloatValue of float | StringValue of string |
AtomValue of string
    | BuiltinFunctionValue of (value -> value)

I eventually intend for the ffi.c.function implementation to return a
BuiltinFunctionValue that takes an argument; either unpacks it from a
"value" into the correct tagged type or throws an error; then packs the
return back into an appropriately tagged "value". The "value" pack and
unpack functions will know their types statically, so maybe in principle
they can work in participation with whatever type-chaining trick @-> does
and still produce well-typed code. But it's very unclear to me how to get
started with that, or whether it will ultimately work.

Do you have any thoughts on this? Is this approach (engaging foreign / @->
directly) one which can eventually work, or should I back off and attempt
one of the lower-level approaches mentioned in your previous email? Your
previous email proposed building out a Ctypes_untyped with a slightly
different interface, but it was not clear to me what implementation of that
would look like.

Thanks!

 - Andi

On Mon, Jun 29, 2015 at 4:16 PM, Jeremy Yallop <yallop at gmail.com> wrote:

> Hi Andi,
>
> Please excuse the slow response.
>
> On 27 June 2015 at 14:25, Andi McClure <andi.m.mcclure at gmail.com> wrote:
> > Hello, I have a project which is a programming language (
> > http://emilylang.org/ ) and the current interpreter is implemented in
> ocaml.
> > I want to add a C FFI, and I am looking at ocaml-ctypes, but I am having
> > trouble figuring out how best to use the ocaml-ctypes library given my
> > project's specific needs as a language.
>
> Well, using ocaml-ctypes to build an FFI for another language is a bit
> different from the usual use case of exposing a particular C library
> to OCaml, so it'll be interesting to see how things turn out.
>
> > Basically: ocaml-ctypes seems to take libFFI and recast its operations in
> > terms of ocaml idioms and primitives. This looks great if I am writing an
> > ocaml program. However, my goal is to present an interface to libffi (or
> > something like it) in terms of *Emily* idioms and primitives (Emily
> being my
> > language) so that I can write an Emily program. This means directly
> wrapping
> > ocaml-ctypes probably won't work out well (the ocaml idioms ocaml-ctypes
> > uses can't all be directly represented in my language, and they might
> come
> > across as weird to the language users who are probably not ocaml users).
> >
> > My initial thought was that what I probably wanted to do was try to
> present
> > an interface between the interpreter and interpreted code that closely
> > tracks the programming interface of libffi. Then I could write an
> > in-language library on top of this with a friendlier/more idiomatic
> > interface. (The reason I believe I want the interpreter-interpreted
> > interface to resemble libffi is that I might someday switch from ocaml to
> > another language, and libffi will probably be available in other contexts
> > but ocaml-ctypes will not). Looking over the ctypes code it looked like a
> > "raw" libffi-flavored interface might be possible using
> > ctypes-foreign-base/ctypes_ffi.mli and the function_of_pointer interface,
> > but it looks like this mli is present in the code but not exposed in the
> > opam package.
>
> Right: that module is intended to be internal-only.  However, some of
> the functionality is expressible via the public interface.  For
> example, here are alternative implementations of function_of_pointer
> and pointer_of_function using only functions that are currently
> publicly exposed:
>
>   let function_of_pointer ?name ~abi ~check_errno ~release_runtime_lock fn
> p =
>     Ctypes.coerce (ptr void)
>       (Foreign.funptr ?name ~abi ~check_errno
> ~runtime_lock:release_runtime_lock fn)
>       p
>
>   let pointer_of_function ~abi ~acquire_runtime_lock fn f =
>     Ctypes.coerce (Foreign.funptr ~abi ~runtime_lock:acquire_runtime_lock
> fn)
>       (ptr void)
>       f
>
> > What would you recommend in this case?
>
> I can think of three approaches that are not obviously wrong, and it's
> likely that there are others.
>
> First, you might do what you've been considering already -- i.e. using
> the low-level parts of ctypes as a basis for a more Emily-flavoured
> libffi binding.  The two essential modules are probably
> Ctypes_memory_stubs (src/ctypes/ctypes_memory_stubs.ml), which
> provides an untyped API for accessing C-managed memory, and
> Ctypes_ffi_stubs (src/ctypes-foreign-base/ctypes_ffi_stubs.ml), which
> provides a fairly direct OCaml binding onto libffi.  Neither of these
> modules is exposed directly, but it should be reasonably
> straightforward to make them available by tweaking the Makefile to add
> them to the appropriate '.public' list.
>
> Second, you might find some way to massage the ctypes API into a form
> that's more suitable for use in Emily.  For example, I can imagine the
> type parameters causing problems if you want to use the type
> representations (for int, float, etc.) in a more uniform way.  It
> probably wouldn't be too much work to build an 'untyped' interface on
> top of ctypes along the following lines
>
>   module Ctypes_untyped :
>   sig
>     type typ
>     val int : typ
>     val float : typ
>     val (@->) :  typ -> typ -> typ
>     (* ... *)
>
> This is just a guess, of course, and perhaps there are other OCaml
> idioms that aren't so easy to factor out.
>
> Finally, you might use ctypes to build a binding to libffi -- i.e.
> just treat libffi like any other C library and describe its interface
> using ctypes, ignoring the fact that ctypes uses libffi internally.
> This approach isn't as absurd as it might sound at first, since recent
> versions of ctypes make it possible to bind to C libraries by
> generating C and OCaml code rather than by routing calls through
> libffi, so you wouldn't necessarily end up using libffi twice.
>
> Feel free to ask if you'd like more detail on any of the above!
>
> Kind regards,
>
> Jeremy.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocaml.org/pipermail/ctypes/attachments/20150809/9d2853dd/attachment.html>