[ocaml-platform] on the need and design of OCaml namespaces

[Yaron Minsky]

On Fri, Mar 1, 2013 at 4:56 AM, Didier Remy <Didier.Remy at inria.fr> wrote:
>>> The main question whether namespaces are hierarchical or flat is still
>>> not
>>> answered, Flat namespaces are not much of an extension to OCaml and do
>>> not
>>> raise many questions about their semantics.  This is the benefit of
>>> Alain's
>>> proposal---but also its weakness to be limited in expressiveness.
>>
>>
>> I don't really understand the goal of full hierarchical models.  I
>> would have thought that flat namespaces that include auto-open modules
>> would be quite sufficient.
>
>
> I thought you wanted a hierarchical model!
> And you keep using hierarchical notation in your examples.
>
>         open Core
>         open Std
> or
>         open Core#Std
>
> Isn't this hierarchical?
> Do you mean hierarchical with just two levels?
> If this what you mean by not being fully hierarchical?
>
> But I don't think two levels or unbounded makes a significant difference,
> technically.

I see how I've given that impression.  I've sometimes used the
hierarchical # notation that others have mooted, but it does not seem
necessary to me.  Core definitely needs multiple overlapping
namespaces, but I don't know that any hierarchy is required.  I think
if they were called Core_Std and Core_Stable, I think I would be quite
happy.

(That said, hierarchy in such things is often natural, and if it
didn't present technical difficulties to have the namespaces be
hierarchical in some sense, I wouldn't object.  But I don't yet know
what I'd use it for in practice.)

>> One difference of opinion that I do have is that my intuition about
>> shadowing and dependencies on orders of opens is the opposite of
>> yours.  In particular, shadowing of names seems quite ordinary in
>> OCaml, it's an expected part of the language, and I would prefer for
>> namespaces to share that property.  Indeed, places where shadowing of
>> names is not possible (modules, in particular) have been a point of
>> frustration for us.
>>
>> My inclination is that if you have two namespaces that define the same
>> module, and you open them in sequence, the second should simply shadow
>> the first.
>>
>>     open namespace Core#Std   (* defines a UNIX module *)
>>
>>     ... some code that uses Core's UNIX ....
>>
>>     open namespace Async#Std  (* defines a different UNIX module *)
>>
>>     ... some code that uses Async's UNIX ....
>>
>> I would be quite distressed to find that the namespace system
>> prevented the second open on the ground that it shadowed something in
>> the first.  This is precisely the behavior I would want and expect!
>
>
> I am not sure of what you exactly want.
>
> In the simplest hierarchical model, namespaces are nodes of a graph whose
> directed edges are names to access sub-namespaces (internal nodes)
> or module objects (leaves).
>
> In this model namespaces are not ordered, i.e. edges leaving from one node
> are not ordered.
>
> This is, I think, an easier model, because you can populate a namespace by
> listing available module objects in a directory---which is not ordered.
>
> In this model:
>
> 1) you can open a node, with either a strict semantics (detecting overrides)
>    or an overriding semantics (given priority to the names coming from the
>    open).
>
> 2) However, you cannot open several nodes at the same time without
>    specifying an ordering between those nodes.

What does it mean to open several nodes "at the same time"?  I would
have thought that opening of nodes would always be ordered, since the
open statements would need to be written in some order.

> 3) Hence, you cannot do auto-open if you have several auto-open edges
>    leaving from the same node unless auto-open edges are ordered.

I agree that the module opens assocaited with a namespace must be
ordered.  I'm not sure why that's problematic, though.  It just seems
like the right design decision.

> If wish to do (3) you would need to order edges (at least auto-open ones) in
> the namespace. You may then wish all edges to be ordered.  However, I think
> this is not a good model.  An intermediate option is to just order auto-open
> nodes when you declare them.  This is better than ordering all nodes---we
> may assume that auto-open nodes are few.  But this mixed model is more
> complicated for the user.  (See at the end of this message while the fully
> ordered model is not good in my opinion.)

I would have assumed that one would order all nodes, and I don't see
the downsides of this, actually.  Why would one prefer an unordered
semantics?  All of the rest of OCaml's namespace manipulations
(variable bindings, module opens, module bindings) are order
dependent.  Why make namespaces different?

>
>>>          ROOT . --Core--> C --Pervasives*--> M
>>>                            \
>>>                              --Std*--> S
>>
>>
>> I apologize, but I just don't understand the example.  Is Pervasives a
>> module or a namespace?  I would assume it would be a module, since
>> I thought only modules are auto-open.  And what is the * for after Std
>> and Pervasives?
>
>
> Here Pervasives is the name of the edge that goes from namespace C to the
> module object M.  Core is the name of the edge that goes from the toplevel
> name space to the sub-namespace C.  You have to distinct the name and the
> object in the model.  You could have several paths leading to the same
> object.
>
> The * is the auto-open flag, as used in Gabriel's description.
> Sorry for leaving this implicit.

Got it.  That's quite helpful, though I'm still a little lost about
the problem you're describing.  It sounds to me like the issue dervies
from the fact that you're proposing opening up a parent namespace as
an operation that opens all children as well.  Is that right?  I
hadn't even begun to imagine wanting that feature, so maybe that isn't
a real issue at all?  Or perhaps I'm just confused....

>>> Perhaps, an alternative to auto-open would be to allow one to write open
>>> directives in .ns files as well. Then, a file myenv.ns could both build a
>>> namespace and pre-open some of the nodes, including some of its leaves.
>>> The
>>> user could then just put something as concise as
>>>
>>>          use myenv.ns
>>>
>>> at the top of his *.ml files. And if he opens myenv.ns he will
>>> have a very clear idea of what names should be visible.
>>>
>>> Just an idea, which I don't like so much either.  But is shows
>>> that there are still important details to be thought of and
>>> choices to be made.
>>
>>
>> How would the declaration "use myenv.ns" differ from "open namespace
>> Myenv"?
>
>
> Here Myenv is not a namespace but a file containing namespaces commands.
> Some of the commands would build the environment, and my proposal is that
> namespace file could also contain open commands.  (opening a name space can
> be seen as the construction of a namespace, but opening a module cannot.)
>
>
>> Are you proposing two different primitives, or would one only
>> have the "use" declaration?
>
>
> This is syntax. The point is that auto-open complicates things and if the
> only reason to have it is that you do not wish to repeat or have every user
> write the sequence
>
>         open Core (* that contains Pervasives, Std, etc. *)
>         open Pervasives
>         open Std
>
> then you could disallow auto-opens and instead allow a namespace file
> myenv.ns to also contain commands that open modules. They you would only
> invoke this file (with whatever syntax to be chosen) at the beginning of
> your .ml source files.

I had thought that in Alain's world, namespaces are almost nothing
more than sequences of module aliases, to which I was proposing adding
module opens.  We do need more than this structure in order to get
error messages and documentation right, so that model is somewhat too
impoverished, no doubt.

>> I think my mental model of how namespaces should work is quite similar
>> to what you described above.  In particular, I would think that
>> opening a namespace would be the equivalent of adding a collection of
>> module aliases (possibly shadowing existing module names), and opening
>> some collection of modules (possible shadowing yet more module names
>> and other values).  All this should happen in whatever order is
>> described in the namespace definition file.
>
>
> The last sentence means that namespaces objects are ordered in your mind.
> Not in mine. In my (simplest) model, namespace objects are not ordered,
> but open (module) directives are ordered.
>
>> And in all of this, I view the dependency on order to be a feature,
>> not a bug.   OCaml is through and through full of this kind of
>> dependency, and I can see no justification for breaking that property
>> specifically with namespaces.  We like the behavior for values and
>> types, after all.  Why should namespaces be different?
>
>
> I think you are talking about dependency on on the order when you open
> things.  This need does not imply a dependency on the order or edges in the
> namespace structure.  If you open things (namespaces or modules) one by one,
> you need not order edges in the namespace structure.
>
> ----------------
>
> I think namespaces should be unordered to scale up.
>
> You would like to be able to represent all available libraries in the OCaml
> world as a namespace, with libraries coming from many different sources and
> there is no natural order on these, i.e. no order that could be given a
> meaningful semantics.  Would you put World#JaneStreet before World#Lexify or
> the other way round?
>
> If name spaces are unordered, it is also easy to populate them by reading
> and merging different sources in a strict way: name spaces are meant to
> avoid conflicting names to start with, so if a merge introduces conflicts,
> this is really the right place to fix it by renaming one side or
> reorganizing the hierarchy rather than do some silent overriding.

I wonder if this all comes down to different thoughts about hierarchy.
I guess I wouldn't think the namespace World#JaneStreet would be
ordered compared to World#Lexify, but the module aliases and opens
contained in a namespace would, I thought, have been ordered.  And
surely, when you open multiple namespaces in your environment, you can
be including the same name multiple times, at which point I believe
the proper semantics is shadowing.

Again, my feeling about this is that we should to the degree possible
stick to the way OCaml behaves now, and now, non-commutative
declarations with shadowing is the norm.  Deviating from this would I
think only confuse developers.

> When getting closer to the leaves, the situation is a bit different, because
> nodes may all come from the same source/library.  Here, it may be desirable
> (yet to be confirmed) to have auto-opens and thus introduce some ordering.
> Ordering towards the leaves is less of a problem in practice and those name
> spaces may be built by explicitly listing all components---rather than
> scanning and merging different sources. So it seems ok in practice.  Still,
> it implies optional ordering of edges and slightly complicates the mental
> model.
>
> Hope this helps,

I think what I'm mostly confused about is what is the nature of the
hierarchical namespace system you're thinking about. What are the
operations available?  Do you intend to have namespace opens also
recursively open all children?  Is there an operation for merging
namespaces?  Aliasing upper namespaces?  Right now, I lack both a
precise mental model of what you mean by hierarchical namespaces, and
an understanding of the language features that motivate this
hierarchy.


>     Didier