[Merlin-discuss] Merlin's configuration

[Roshan James]

A running merlin instance can compute, for each .ml file that one
visits:

  1. the type environment in scope at the top of the file
  2. the preprocessors used on the file
  3. the location of the .cmt files for modules in the environment

Merlin accomplishes this via .merlin files, which give this
information at the granularity of a directory (all files in a
directory have the same toplevel environment).

We propose:

  A. There should be a way to state this information per file.
  B. The merlin command should use the same syntax as an ocamlopt
     call for expressing the information.
  C. The information from the command-line in B also be used to
     determine other aspects of type checking, e.g. warnings.
  D. An auxiliary file (.merlin) be used for expressing only things
     that the ocamlopt call doesn't express, e.g. (3).
  E. A running merlin process should accept a command that updates
     this information.
  F. A running merlin process should accept a command that tells it
     when a .cmt or .cmi changes, and should update its internal
     cache.

The thinking behind using an ocamlopt call as the mechanism of
expression is that it is already present in all build systems,
and is thus very likely to reflect what the build system is
actually doing. Here is an example:

ocamlopt.opt -I +camlp4 -thread -w @a-4-7-29 -strict-sequence
-bin-annot -inline 20 -nodynlink -g
-pp 'camlp4o ../../lib/pa_type_conv/pa_type_conv.cmo
../../lib/pa_sexp_conv/pa_sexp_conv.cmo
../../lib/pa_bin_prot/pa_bin_prot.cmo
../../lib/pa_fields_conv/pa_fields_conv.cmo
../../lib/pa_variants_conv/pa_variants_conv.cmo
../../lib/pa_compare/pa_compare.cmo ../../lib/pa_pipebang/pa_pipebang.cmo
../../lib/pa_here/pa_here.cmo
../../lib/pa_custom_printf/pa_custom_printf.cmo
../../lib/pa_test/pa_test.cmo ../../lib/pa_enumerate/pa_enumerate.cmo
../../lib/pa_ounit/pa_ounit.cmo -pa-ounit-lib bins'
-I ../../lib/pa_ounit_lib -I ../../lib/bin_prot -I ../../lib/fieldslib
-I ../../lib/sexplib -I ../../lib/variantslib -I ../../lib/core_kernel
-I ../../lib/core -I ../../lib/textutils -I ../../lib/async_core
-I ../../lib/async_extra -I ../../lib/async_unix -I ../../lib/pcre
-I ../../lib/core_extended -I ../../lib/async -I ../../lib/async_extended
-I ../../lib/varg -I ../../lib/sh -I ../../lib/jane_extended -c z.ml

In more detail:

(1) This command specifies the exact namespace visible to each
.ml file. From each include path one can collect the relevant
.cmi files to construct the namespace visible to the file being
compiled.

(2) Any additional compiler flags relevant to compiling the .ml
files should be available here. For example, it would be valuable
for us if merlin flagged the right warnings are errors.

(3) Every build system already knows how to generate such
compilation commands. Merlin should be able to parse this
relatively easily given its roots in the ocaml toplevel.
Further, merlin can ignore switches it doesn't understand for now
and silently add more support in the future without needing
everyone to change their build scripts.

(4) The missing information in the compilation command includes
where to find the .cmt files of each library and the mapping of
preprocessors to merlin's EXT specifications. These can be
communicated in a .merlin file. For example:

EXT binprot  ../../lib/pa_bin_prot/pa_bin_prot.cmo
EXT sexp     ../../lib/pa_bin_prot/pa_bin_prot.cmo
EXT here ../../lib/pa_here/pa_here.cmo
CMT core_kernel ../../base/core/kernel/lib

i.e. if a call to camlopt mentions core_kernel, then merlin can
use this mapping to find the [.cmt] files.  This mapping can be
specified either in each local directory or once at the root of
the repo.

(5) How should the compilation command be communicated to merlin?
There are at least three approaches that one can think of, of
which we think the third has several advantages:

(i) Write things into files: As you proposed, we can write out
such compilation commands into a file. If any .ml specific
customization is required, it will be captured in this format. If
all the files of a dir are compiled the same way, one need
specify only one compilation command with some glob patterns.

(ii) Have merlin ask the build system: As you proposed, we can
specify a command that merlin calls, that returns the info in (i)
to merlin.

(iii) Have the build system notify merlin: Merlin can provide a
command that the build system calls to communicate the ocamlopt
compilation command to the current running instance of
merlin. This has the advantage that merlin gets updates whenever
the build system needs to recompile files. Further, merlin can be
notified of successful build completion of each .ml file.

(6) Finally, one property worth thinking about is how to make
merlin provide "best effort" completion: Consider the situation
where one is working on both an exe and a lib that it depends
on. During development there maybe some .ml files in the lib that
compile and some that don't. The ones that don't yet compile
prevent the lib.cmi file from being built/updated. In such cases,
it would be nice if merlin can still provide the latest
completion information for the .ml files of the lib that are
compiling.

Roshan



On Tue, Nov 19, 2013 at 11:26 AM, Sylvain Le Gall <sylvain at le-gall.net>wrote:

> In OASIS we have a function that automatically generates _tags from
> _oasis, generating a somehow similar tool-readable output in a plugin
> will be super easy for OASIS.
>
> We can also fine tune a lot for Merlin. We can implement new fields to
> embed most of the configuration needed. If we get it right -- and if
> you want -- I think it may be possible to replace .merlin by a few
> fields in _oasis and reuse information from this file.
>
> 2013/11/19 Gabriel Scherer <gabriel.scherer at gmail.com>:
> > Re. the second point, I think trying to reimplement the build system's
> > job in Merlin is a dead end. You must defer to the existing build
> > system(s) to provide you this information.
> >
> > ocamlbuild has a -show-tags option that does a fair bit of what you
> > want (but maybe not all: it reads _tags but does not provide as much
> > information as computed dynamically by the actual compilation rules;
> > and it returns an unspecified ocamlbuild pretty-printing format that
> > was planned for humans, not tools, to read). More can be implemented.
> >
> > .merlin has always been a bit of a hack, duplicating information that
> > was already known by the build logic (plus possibly some
> > merlin-specific configuration that has its place). We should be
> > working with tool authors to *reduce* it, not extend it.
> >
> > On Tue, Nov 19, 2013 at 12:26 PM, Sylvain Le Gall <sylvain at le-gall.net>
> wrote:
> >> Hello,
> >>
> >> I will be more than happy to help you on the second point to integrate
> >> an external process that can ask directly oasis to generate a
> >> configuration. I can help you on this project and even help you write
> >> the extension.
> >>
> >> Regards
> >> Sylvain
> >>
> >>
> >>
> >> 2013/11/19 Thomas Refis <thomas.refis at gmail.com>:
> >>> Hi!
> >>>
> >>> There have been some changes recently on merlin's configuration, and
> >>> we're also considering some more heavy changes on the syntax for the
> >>> .merlin file.  We'd like to have people's input on this.
> >>>
> >>>    * * *
> >>>
> >>> First we've added two new "directives": CMT and CMI, these are
> >>> refinements of the "B" directive, which was used to add directories to
> >>> the build path. The need for this is primarily felt at JaneStreet.
> >>>
> >>> At janestreet, there is one big repository including, among others,
> >>> core and async.  Every source directory is also a build directory but
> >>> the cmi for packed modules are afterwards copied to a particular lib/
> >>> directory at the root of this repo.
> >>> You then have two options to set-up your build path in merlin: add
> >>> only lib/, or add every source directory.
> >>> The first option is unsatisfactory since you won't have access to cmt
> >>> files (that implies no fancy source browsing), but the second is also
> >>> problematic.  Indeed there is a lot of module name conflicts, and
> >>> there's no predictable way to tell which one will be choosen. That
> >>> results in the frequent presence of the error message "Module Foo and
> >>> Bar make inconsistent assumptions over interface Baz".
> >>>
> >>> With the new directives, one can now add lib/ to the "CMI" dirs (i.e.
> >>> the buildpath) and every other directory to the "CMT" dirs.
> >>>
> >>> On simpler setups, one can of course still simply use "B".
> >>>
> >>> NB: This is not on the opam package yet, only on the git repository.
> >>> We should however make a new release soon (in the next two weeks I'd
> >>> say).
> >>>
> >>>    * * *
> >>>
> >>> We are considering moving to a more expressive syntax for the .merlin
> >>> files, the goal is to be more precise about which files/directories
> >>> require which packages/compiler flags and which do not.
> >>> There is two different paths we can follow here:
> >>>
> >>>   The first one is to adopt the syntax ocamlbuild uses in its _tags
> >>> file. This should be precise enough to express everything one would
> >>> ever need. Also it has the advantage of being a syntax everyone
> >>> already knows, and one can also imagine that at some point in the
> >>> future, merlin would be able to understand (at least partially) _tags
> >>> files whenever there is no .merlin.
> >>>
> >>>   The second one, a bit more ambitious, consist of finding a way to
> >>> delegate the work to external tools. That is: merlin would call an
> >>> (arbitrary) external process, giving it the path of the file currently
> >>> under edition. And that process would then output merlin's
> >>> configuration.
> >>> That way, people could first have the configuration file in their
> >>> language of choice, or call a tool to generate a configuration from an
> >>> _tags, _oasis, ?, but would also be able to have merlin integrated in
> >>> their build process. For example people using a build system running
> >>> in "daemon" mode (jenga, OMake) could then feed it the path of the
> >>> file being edited, and would then receive a list of the dependencies
> >>> of that file, as well as compiler flags required.
> >>>
> >>> We would like to know if you think either of this approach is useful,
> >>> or interesting... or not.
> >>>
> >>>    * * *
> >>>
> >>> That's about it.
> >>>
> >>> Cheers,
> >>>
> >>> Frederic and Thomas.
> >>> _______________________________________________
> >>> Merlin-discuss mailing list
> >>> Merlin-discuss at lists.forge.ocamlcore.org
> >>> https://lists.ocamlcore.org/cgi-bin/listinfo/merlin-discuss
> >> _______________________________________________
> >> Merlin-discuss mailing list
> >> Merlin-discuss at lists.forge.ocamlcore.org
> >> https://lists.ocamlcore.org/cgi-bin/listinfo/merlin-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ocamlcore.org/pipermail/merlin-discuss/attachments/20131119/f702e79a/attachment-0001.html>