[ocaml-platform] on the need and design of OCaml namespaces

[Gabriel Scherer]

> I think what I'm mostly confused about is what is the nature of the
> hierarchical namespace system you're thinking about. What are the
> operations available?  Do you intend to have namespace opens also
> recursively open all children?  Is there an operation for merging
> namespaces?  Aliasing upper namespaces?  Right now, I lack both a
> precise mental model of what you mean by hierarchical namespaces, and
> an understanding of the language features that motivate this
> hierarchy.

I recommend that you have a look at the document I posted as the
starting point for the discussion:
  http://gallium.inria.fr/~scherer/namespaces/spec.pdf

It is not monolithic in the sense that it must be understood as
describing the problem space and possible design choices.

It does not mention the "flat access" feature (the auto-open for
Pervasives, etc.) which comes from an older, more advanced (and less
pertinent for now) document. (A previous mail by Didier explained the
design issues around auto-open well. Regarding the compromises to
make, I would tend to slightly diverge by accepting a fully ordered
model and giving up on the robustness of strict merging on leaves. I
disagree, however, with the liking you have for name shadowing at
large scales and think this should be restricted as much as possible,
but it's a rather orthogonal aspect that's best left for another
discussion.).

Another distinct aspect of the proposal is its insistence on
introducing a difference between internal compilation unit names (as
used by the linker) and external compilation unit names (as seen by
the programmer), currently both derived from the filename. That's a
gray part of the design discussion that has seen little contributions,
and Alain's suggestion to just do nothing at all there is probably a
reasonable basis for experimentation. In this light, the "Compilation
unit information" part of the document may be ignored.

Regarding "features that motivate a hierarchy": 'open'. Alain's
proposal is a hierarchy with only two levels, the names of the .ns
files (first level) and the compilation unit names and their aliases
(second level). In your examples both are apparent in source code and
have distinct semantics properties (leaves are modules, non-leaves
("namespace names") are not). As Didier said, I believe there is
little technical difference between two-levels hierarchies and a full
hierarchical model (in particular believing that it makes design
problems simpler is delusive), but I don't have a particular opinion
on what should end up in an actual implementation, other than a desire
for a solution that *can* gracefully scale towards a full hierarchy if
we start needing it in the future.

On Fri, Mar 1, 2013 at 12:53 PM, Yaron Minsky <yminsky at janestreet.com> wrote:
> On Fri, Mar 1, 2013 at 4:56 AM, Didier Remy <Didier.Remy at inria.fr> wrote:
>>>> The main question whether namespaces are hierarchical or flat is still
>>>> not
>>>> answered, Flat namespaces are not much of an extension to OCaml and do
>>>> not
>>>> raise many questions about their semantics.  This is the benefit of
>>>> Alain's
>>>> proposal---but also its weakness to be limited in expressiveness.
>>>
>>>
>>> I don't really understand the goal of full hierarchical models.  I
>>> would have thought that flat namespaces that include auto-open modules
>>> would be quite sufficient.
>>
>>
>> I thought you wanted a hierarchical model!
>> And you keep using hierarchical notation in your examples.
>>
>>         open Core
>>         open Std
>> or
>>         open Core#Std
>>
>> Isn't this hierarchical?
>> Do you mean hierarchical with just two levels?
>> If this what you mean by not being fully hierarchical?
>>
>> But I don't think two levels or unbounded makes a significant difference,
>> technically.
>
> I see how I've given that impression.  I've sometimes used the
> hierarchical # notation that others have mooted, but it does not seem
> necessary to me.  Core definitely needs multiple overlapping
> namespaces, but I don't know that any hierarchy is required.  I think
> if they were called Core_Std and Core_Stable, I think I would be quite
> happy.
>
> (That said, hierarchy in such things is often natural, and if it
> didn't present technical difficulties to have the namespaces be
> hierarchical in some sense, I wouldn't object.  But I don't yet know
> what I'd use it for in practice.)
>
>>> One difference of opinion that I do have is that my intuition about
>>> shadowing and dependencies on orders of opens is the opposite of
>>> yours.  In particular, shadowing of names seems quite ordinary in
>>> OCaml, it's an expected part of the language, and I would prefer for
>>> namespaces to share that property.  Indeed, places where shadowing of
>>> names is not possible (modules, in particular) have been a point of
>>> frustration for us.
>>>
>>> My inclination is that if you have two namespaces that define the same
>>> module, and you open them in sequence, the second should simply shadow
>>> the first.
>>>
>>>     open namespace Core#Std   (* defines a UNIX module *)
>>>
>>>     ... some code that uses Core's UNIX ....
>>>
>>>     open namespace Async#Std  (* defines a different UNIX module *)
>>>
>>>     ... some code that uses Async's UNIX ....
>>>
>>> I would be quite distressed to find that the namespace system
>>> prevented the second open on the ground that it shadowed something in
>>> the first.  This is precisely the behavior I would want and expect!
>>
>>
>> I am not sure of what you exactly want.
>>
>> In the simplest hierarchical model, namespaces are nodes of a graph whose
>> directed edges are names to access sub-namespaces (internal nodes)
>> or module objects (leaves).
>>
>> In this model namespaces are not ordered, i.e. edges leaving from one node
>> are not ordered.
>>
>> This is, I think, an easier model, because you can populate a namespace by
>> listing available module objects in a directory---which is not ordered.
>>
>> In this model:
>>
>> 1) you can open a node, with either a strict semantics (detecting overrides)
>>    or an overriding semantics (given priority to the names coming from the
>>    open).
>>
>> 2) However, you cannot open several nodes at the same time without
>>    specifying an ordering between those nodes.
>
> What does it mean to open several nodes "at the same time"?  I would
> have thought that opening of nodes would always be ordered, since the
> open statements would need to be written in some order.
>
>> 3) Hence, you cannot do auto-open if you have several auto-open edges
>>    leaving from the same node unless auto-open edges are ordered.
>
> I agree that the module opens assocaited with a namespace must be
> ordered.  I'm not sure why that's problematic, though.  It just seems
> like the right design decision.
>
>> If wish to do (3) you would need to order edges (at least auto-open ones) in
>> the namespace. You may then wish all edges to be ordered.  However, I think
>> this is not a good model.  An intermediate option is to just order auto-open
>> nodes when you declare them.  This is better than ordering all nodes---we
>> may assume that auto-open nodes are few.  But this mixed model is more
>> complicated for the user.  (See at the end of this message while the fully
>> ordered model is not good in my opinion.)
>
> I would have assumed that one would order all nodes, and I don't see
> the downsides of this, actually.  Why would one prefer an unordered
> semantics?  All of the rest of OCaml's namespace manipulations
> (variable bindings, module opens, module bindings) are order
> dependent.  Why make namespaces different?
>
>>
>>>>          ROOT . --Core--> C --Pervasives*--> M
>>>>                            \
>>>>                              --Std*--> S
>>>
>>>
>>> I apologize, but I just don't understand the example.  Is Pervasives a
>>> module or a namespace?  I would assume it would be a module, since
>>> I thought only modules are auto-open.  And what is the * for after Std
>>> and Pervasives?
>>
>>
>> Here Pervasives is the name of the edge that goes from namespace C to the
>> module object M.  Core is the name of the edge that goes from the toplevel
>> name space to the sub-namespace C.  You have to distinct the name and the
>> object in the model.  You could have several paths leading to the same
>> object.
>>
>> The * is the auto-open flag, as used in Gabriel's description.
>> Sorry for leaving this implicit.
>
> Got it.  That's quite helpful, though I'm still a little lost about
> the problem you're describing.  It sounds to me like the issue dervies
> from the fact that you're proposing opening up a parent namespace as
> an operation that opens all children as well.  Is that right?  I
> hadn't even begun to imagine wanting that feature, so maybe that isn't
> a real issue at all?  Or perhaps I'm just confused....
>
>>>> Perhaps, an alternative to auto-open would be to allow one to write open
>>>> directives in .ns files as well. Then, a file myenv.ns could both build a
>>>> namespace and pre-open some of the nodes, including some of its leaves.
>>>> The
>>>> user could then just put something as concise as
>>>>
>>>>          use myenv.ns
>>>>
>>>> at the top of his *.ml files. And if he opens myenv.ns he will
>>>> have a very clear idea of what names should be visible.
>>>>
>>>> Just an idea, which I don't like so much either.  But is shows
>>>> that there are still important details to be thought of and
>>>> choices to be made.
>>>
>>>
>>> How would the declaration "use myenv.ns" differ from "open namespace
>>> Myenv"?
>>
>>
>> Here Myenv is not a namespace but a file containing namespaces commands.
>> Some of the commands would build the environment, and my proposal is that
>> namespace file could also contain open commands.  (opening a name space can
>> be seen as the construction of a namespace, but opening a module cannot.)
>>
>>
>>> Are you proposing two different primitives, or would one only
>>> have the "use" declaration?
>>
>>
>> This is syntax. The point is that auto-open complicates things and if the
>> only reason to have it is that you do not wish to repeat or have every user
>> write the sequence
>>
>>         open Core (* that contains Pervasives, Std, etc. *)
>>         open Pervasives
>>         open Std
>>
>> then you could disallow auto-opens and instead allow a namespace file
>> myenv.ns to also contain commands that open modules. They you would only
>> invoke this file (with whatever syntax to be chosen) at the beginning of
>> your .ml source files.
>
> I had thought that in Alain's world, namespaces are almost nothing
> more than sequences of module aliases, to which I was proposing adding
> module opens.  We do need more than this structure in order to get
> error messages and documentation right, so that model is somewhat too
> impoverished, no doubt.
>
>>> I think my mental model of how namespaces should work is quite similar
>>> to what you described above.  In particular, I would think that
>>> opening a namespace would be the equivalent of adding a collection of
>>> module aliases (possibly shadowing existing module names), and opening
>>> some collection of modules (possible shadowing yet more module names
>>> and other values).  All this should happen in whatever order is
>>> described in the namespace definition file.
>>
>>
>> The last sentence means that namespaces objects are ordered in your mind.
>> Not in mine. In my (simplest) model, namespace objects are not ordered,
>> but open (module) directives are ordered.
>>
>>> And in all of this, I view the dependency on order to be a feature,
>>> not a bug.   OCaml is through and through full of this kind of
>>> dependency, and I can see no justification for breaking that property
>>> specifically with namespaces.  We like the behavior for values and
>>> types, after all.  Why should namespaces be different?
>>
>>
>> I think you are talking about dependency on on the order when you open
>> things.  This need does not imply a dependency on the order or edges in the
>> namespace structure.  If you open things (namespaces or modules) one by one,
>> you need not order edges in the namespace structure.
>>
>> ----------------
>>
>> I think namespaces should be unordered to scale up.
>>
>> You would like to be able to represent all available libraries in the OCaml
>> world as a namespace, with libraries coming from many different sources and
>> there is no natural order on these, i.e. no order that could be given a
>> meaningful semantics.  Would you put World#JaneStreet before World#Lexify or
>> the other way round?
>>
>> If name spaces are unordered, it is also easy to populate them by reading
>> and merging different sources in a strict way: name spaces are meant to
>> avoid conflicting names to start with, so if a merge introduces conflicts,
>> this is really the right place to fix it by renaming one side or
>> reorganizing the hierarchy rather than do some silent overriding.
>
> I wonder if this all comes down to different thoughts about hierarchy.
> I guess I wouldn't think the namespace World#JaneStreet would be
> ordered compared to World#Lexify, but the module aliases and opens
> contained in a namespace would, I thought, have been ordered.  And
> surely, when you open multiple namespaces in your environment, you can
> be including the same name multiple times, at which point I believe
> the proper semantics is shadowing.
>
> Again, my feeling about this is that we should to the degree possible
> stick to the way OCaml behaves now, and now, non-commutative
> declarations with shadowing is the norm.  Deviating from this would I
> think only confuse developers.
>
>> When getting closer to the leaves, the situation is a bit different, because
>> nodes may all come from the same source/library.  Here, it may be desirable
>> (yet to be confirmed) to have auto-opens and thus introduce some ordering.
>> Ordering towards the leaves is less of a problem in practice and those name
>> spaces may be built by explicitly listing all components---rather than
>> scanning and merging different sources. So it seems ok in practice.  Still,
>> it implies optional ordering of edges and slightly complicates the mental
>> model.
>>
>> Hope this helps,
>
> I think what I'm mostly confused about is what is the nature of the
> hierarchical namespace system you're thinking about. What are the
> operations available?  Do you intend to have namespace opens also
> recursively open all children?  Is there an operation for merging
> namespaces?  Aliasing upper namespaces?  Right now, I lack both a
> precise mental model of what you mean by hierarchical namespaces, and
> an understanding of the language features that motivate this
> hierarchy.
>
>
>>     Didier
> _______________________________________________
> Platform mailing list
> Platform at lists.ocaml.org
> http://lists.ocaml.org/listinfo/platform