[ocaml-platform] on the need and design of OCaml namespaces

[Yaron Minsky]

I should mention that the term "UNSANITARY" might sound a bit
negative, but that is not intended.  My actual opinion is that Alain's
proposal sounds to me quite practical, and is basically my favorite
proposal thus far (though I consider my own opinion to be of limited
value, because of my lack of understanding of many implementation
issues).  I just wanted for us to call out the downsides explicitly,
so as to better understand what the opposition to Alain's plan might
be.

On Tue, Feb 26, 2013 at 10:12 AM, Gabriel Scherer
<gabriel.scherer at gmail.com> wrote:
> Yaron Minsky <yminsky at janestreet.com> wrote:
>> One thing I'll say is that it is important to be able to add values,
>> and not just modules, to the namespace.  Open Core.Std also adds
>> top-level values, as does the traditional standard library (i.e.,
>> Pervasives), and I don't want to lose that.
>
> This is handled as a "bonus feature" discussed in
> http://gallium.inria.fr/~scherer/namespaces/pack_et_functor_pack.html
> , under the name "flat access". The idea is that adding values to
> namespaces directly doesn't work very well (in OCaml, values cannot
> live outside a compilation unit), but you can have some modules that
> will be explicitly opened if you open the namespace (or included if
> you use it as a module). However, we're not yet sure that this works
> as well as the other aspects of the design, and would recommend
> starting without it to get a feel of the system.

That sounds dandy.

> On Tue, Feb 26, 2013 at 3:38 PM, Yaron Minsky <yminsky at janestreet.com> wrote:
>> Leo, do you have a summary of what you don't like about Alain's
>> namespace proposal?  I'm not presently able to identify any obvious
>> weaknesses in it.  The downsides I see are:
>>
>> - UNSANITARY: Having both "open namespace Core.Std" and Core_List as
>>   names seems a little unsanity.  Indeed, to provide a decent user
>>   experience, you probably want to hide the Core_List name almost
>>   everywhere.  You don't want it showing up in error messages,
>>   documentation, source files, etc.  When you need to do a bunch of
>>   work to hide something, maybe it's better not to include it at all.
>
> I want to point out that Alain's idea of using long, hopefully-unique
> names for the source files, that you deem "unsanitary" here, is also a
> workaround for the problem of internal name conflict. In Alain's
> proposal, internal names conflicts are (hopfeully) avoided with no
> implementation change, just by judicious name choices and letting the
> implementation do its work. This imposes this "unsanitary" aspect you
> mention, but it is a complexity tradeoff: the other alternative would
> be to design a more clever implementation to choose internal names
> (than the module's source filename), which could be hidden from the
> user (no exposed Core_List) but require an implementation change. I
> think that could warrant an exploration (because it could support use
> cases such as linking together two versions of the same library, which
> isn't possible with a purely filename-based solution), but in the
> context Alain's proposal this "unsanitary" aspect has more advantages
> than downsides.

I don't disagree with any of this.

>> - NO HIDING: I'm not sure that the other namespace proposals do
>>   support this, but I'd like to be able to hide some modules so that
>>   they are not reachable outside of the namespace.  We can do this
>>   with the current Core.Std, but I don't see how to do it in Alain's
>>   proposal.
>
> I'm surprised by this "hiding" idea. What does it mean, and what would
> be an use case for that?
>
>> Are there other issues I'm missing?
>
> My gut feeling is that a hierarchical model would add little
> complexity to Alain's proposal and give a saner semantics to "open
> namespace". With a flat model there can be no real notion of "open
> namespace", only handcrafted renamings en masse (Core_list -> List,
> Core_array -> Array...). (On the other hand "open namespace", if they
> are done at the source level, are a source language change, while none
> are necessary so far, so they will require more work to handle in any
> case.)
>
> There is the problem of internal name conflicts, but that is one of
> the more "advanced" questions that may be left out of a first
> experiment.
>
> When deciding to leave out some questions for later, it is however
> important to wonder whether we're sure that we will actually be able
> to extend the design to support them. Some early design choices may
> make future extensions harder (without breaking compatibility). One
> solution is to make a coherent design that covers advanced features,
> and decide to implement only a subset. One other is to try to guess
> what will be easy to add afterwards, and be lucky. Finally, one can
> experiment with a first design and move to another, but without
> committing to backward compatibility (meaning no language release
> between the two design iterations).

I appreciate your point about picking a modest design.  I would argue,
however, that it's not worth actually implementing a namespace
mechanism that doesn't handle the case of a library like Core.  I
think libraries like Core are really one of the biggest drivers for
namespaces, and it would seem a mistake to build and deploy something
that doesn't solve that problem.

> My personal guess is that
> - "flat access" will be easy an easy extension.
> - "hierarchical rather than flat mappings" will be relatively easy in
> theory but make a painful transition period for providers that have
> modules organized flatly and desire to change; but it depends on a
> question of "seeing non-leaf namespaces as modules" that Alain's flat
> namespaces don't need to consider.
> - "extending the mapping description language" must be planned for
> beforehand to avoid growing pains.
> - "finer implementation of internal names to avoid linking conflicts"
> is yet of unknown difficulty, either today and after any first
> proposal not tackling this aspect is acted upon.

This all sounds reasonable.


>>
>> y
>>
>> On Tue, Feb 26, 2013 at 9:30 AM, Yaron Minsky <yminsky at janestreet.com> wrote:
>>> On Tue, Feb 26, 2013 at 8:03 AM, Alain Frisch <alain.frisch at lexifi.com> wrote:
>>>> On 02/25/2013 10:53 PM, Yaron Minsky wrote:
>>>>>
>>>>> I understand your point Alain, but while what you're saying is
>>>>> technically reasonable, I think it doesn't hold together.  When
>>>>> programming in the large, it is useful to be able to manipulate the
>>>>> namespace and group parts of the world together.
>>>>
>>>> Can you give concrete examples of which manipulations are desired (and why)?
>>>
>>> I think the conversation has gotten confused.  You said "who needs
>>> pack?  Just use hierarchically named modules".  I'm saying: that's
>>> crazy, you need to be able to do simple manipulations of namespaces
>>> (like the ones implied by "open Core.Std" or "open Core.Stable".)
>>>
>>> Now you're pointing out that your namespace proposal covers the
>>> manipulations I'm describing.  That may well be right, and I wasn't
>>> contesting that point.
>>>
>>> All I'm saying is that simply relying on long module names without any
>>> kind of explicit namespace control does not scale.  I stand by that,
>>> without necessarily objecting to your namespace proposal.
>>>
>>>>> Many libraries, not
>>>>> just Core or Async, want to be able to remap the world by adding a
>>>>> collection of related names to the namespace.  The ability to do the
>>>>> moral equivalent of:
>>>>>
>>>>>     open Core.Std
>>>>>
>>>>> is powerful and important.  Your proposal of having people add
>>>>> prefixes to module names does not fit the bill, and the resulting
>>>>> system does not, in my opinion, scale.
>>>>
>>>>
>>>> I believe my proposal covers this use case.  If you depend a lot on a
>>>> specify library which exports many modules, you might indeed want to avoid
>>>> using long names everywhere to access these modules.  That's why I propose
>>>> to give simple ways to alias long modules names to short ones, in a way
>>>> which can be factorized (with external mapping files).
>>>>
>>>> In my proposal, the equivalent of "open Core.Std" would simply be to tell
>>>> the compiler (through a command-line option or with a directive in the code)
>>>> to use a mapping file.  The same can be done within the library itself.
>>>>
>>>> Here is a minimalistic version of my proposal, restricted to specifying
>>>> those mapping files on the compiler and tools command-lines.  To make it
>>>> clear that this is only about mapping module references to compiled units,
>>>> let's piggy-back the -I option.  If its argument is a file with a .ns suffix
>>>> and not a directory, the compiler interprets it as a mapping file (a
>>>> sequence of lines of the form "Module_name =
>>>> relative_path_to_compiled_unit", e.g. "List = core_list"), and when this -I
>>>> option is considered during the resolution of a module reference (such as
>>>> "List") and the module is defined in the file, the compiler simply resolves
>>>> the module to the corresponding unit.
>>>>
>>>> This approach could also be used to restrict which units are visible by the
>>>> compiler (and avoid repeated lookup on the file system) without moving files
>>>> around on the file system.
>>>>
>>>> ocamldep (without -modules) would apply the same logic; ocamldep -modules
>>>> could either implement the same logic, or leave it to the build system
>>>> interpreting its output (allowing more dynamic scenarios where the mapping
>>>> file themselves are generated).  I expect ocamldoc to work mostly
>>>> out-of-the-box, even though one could think about using the mapping files
>>>> (in reverse direction) to provide shorter names in the generated
>>>> documentation (or not).
>>>>
>>>> Users of a library are never forced to use the mapping files, and they can
>>>> always refer to a module with its full name (provided the corresponding -I
>>>> <path> is used).
>>>>
>>>> The OCaml stdlib would be adapted to use longer names (stdlib_list,
>>>> stdlib_array, etc) and shipped with a stdlib.ns file opened by default
>>>> (unless -nostdlib is used).  To be clear: this mapping file will be used by
>>>> the stdlib itself, so references within itself don't need to use long names.
>>>>
>>>> I'm interested to see concrete examples of manipulation or scenarios not be
>>>> covered by this proposal.
>>>
>>> I am unaware of any.  I think this does mostly support the
>>> Core.Std/Core.Stable tricks we use.
>>>
>>> One thing I'll say is that it is important to be able to add values,
>>> and not just modules, to the namespace.  Open Core.Std also adds
>>> top-level values, as does the traditional standard library (i.e.,
>>> Pervasives), and I don't want to lose that.
>>>
>>> y
>> _______________________________________________
>> Platform mailing list
>> Platform at lists.ocaml.org
>> http://lists.ocaml.org/listinfo/platform