[wg-camlp4] Reconciling Merlin and ppx

[Gabriel Scherer]

> If we retransform the whole AST during each invocation, we also need to retype the whole AST after.

You could probably write a function that takes two AST, and compare
them left-to-right, to cut them into a common prefix and a new chunk.
You would lose incremental syntax-extension processing but still
regain incremental type-checking afterwards (assuming the rewriter
modifications are local enough: adding stuff at the toplevel would
break that property). This would also help a lot to implement merlin
frontend for editors with broken buffer-change support (you can send
the whole buffer to the parser and still save on the type-checking
time).

> Ok. So if some work had to be done to ensure proper compatibility between
> merlin and ppx, it would be for supporting stateful rewriters.

I think we should consider making statefulness explicit in the
rewriters API, that is asking stateful rewriter to use a marshallable
state type and express their transform in the state monad (state * AST
-> state * result), instead of silently initializing internal state.
This would be compatible with an imperative style in the code of the
rewriter themselves (you just need to reify the state at the end of
the transformation).

On Mon, Nov 4, 2013 at 3:56 PM, Frédéric Bour <defree at gmail.com> wrote:
>
> On 04/11/2013 15:06, Alain Frisch wrote:
>>
>> On 11/04/2013 02:46 PM, Frédéric Bour wrote:
>>>
>>> Merlin is used in an interactive context (were we expect a response in
>>> less than ~ 60ms, on average).
>>>
>>> If we retransform the whole AST during each invocation, we also need to
>>> retype the whole AST after.
>>
>>
>> I assume there is already some logic to detect from which point the code
>> has changed, and to resume type-checking from the start of the phrase, using
>> some kind of snapshot.  Couldn't this logic be applied after the
>> preprocessing?
>
>
> The process is roughly the following:
> - Each chunk has an ending position.
> - The editor tells the coordinates of the first character that changed.
> - Merlin drops everything ending after that position then notify the editor
> of the ending position of the last chunk on merlin side.
> - The editor is expected to tell the rest of the buffer (up to the cursor)
> starting from this last chunk.
>
> So the "detection logic" has more to do with the editor than with the AST.
>
>> Still, calling the PPX transformers as external processes might already be
>> too slow (there is always the option to link them in).
>
> Constantly invoking external processes should be fast enough in merlin's
> case, as long as the work to do is not proportional to the size of the file.
>
>
>>
>> > Then an "open Core.Std" alone is enough to
>>>
>>> blow up the time budget. (And this existimation is done on a desktop
>>> computer, some users have EEE).
>>
>>
>> For this specific example, this patch might help:
>> http://caml.inria.fr/mantis/view.php?id=5916
>> (I'm still waiting for a reproducible case before accepting the patch...)
>
>
> Merlin's actually use a patch to make substitution lazy, guaranteeing "open"
> time proportional to the number of identifier introduced, not to the number
> of path substituted.
> So "open Core.Std" is instantaneous (except on cold start, where the vast
> majority of the time is spent unmarshalling data on C-side).
>
> I can't tell if this is the same approach as in the patch you linked, but it
> works well for Merlin.
>
>
>>
>>> However, I wonder how this state threading would be handled in the REPL ?
>>
>>
>> Currently, it's not.  See:
>>
>> http://caml.inria.fr/mantis/view.php?id=5904#c8797
>>
>>
>> Even though some rewriters might need to maintain state (or even have
>> access to the whole unit), many of them (and in particular those
>> corresponding to the extensions currently supported by Merlin) won't
>> probably need it.
>
>
> Ok. So if some work had to be done to ensure proper compatibility between
> merlin and ppx, it would be for supporting stateful rewriters.
>
>
>>
>>
>> Alain
>
>
> _______________________________________________
> wg-camlp4 mailing list
> wg-camlp4 at lists.ocaml.org
> http://lists.ocaml.org/listinfo/wg-camlp4