[wg-camlp4] Reconciling Merlin and ppx

[Frédéric Bour]

On 04/11/2013 17:17, Alain Frisch wrote:
> On 11/04/2013 04:15 PM, Gabriel Scherer wrote:
>> I think we should consider making statefulness explicit in the
>> rewriters API, that is asking stateful rewriter to use a marshallable
>> state type and express their transform in the state monad (state * AST
>> -> state * result), instead of silently initializing internal state.
>
> We probably need to turn "state" into "state option" for the first 
> invocation of the rewriter. This is a nice idea, but it would require 
> to change the current format of AST files (shared with Camlp4), and 
> introduce some unsafe behavior (the state will be untyped in the 
> external representation).

Correct me if I am wrong, but the current code already relies on 
marshalling and this is where the unsafety lies − the external state can 
just be seen as an abstract type ?

> Moreover, it would not cover all possible cases. The problem is so 
> much that a rewriter can have an internal state, but rather than it 
> can be completely non compositional. For instance, sedlex collects all 
> lexer specifications and generate shared transition tables/functions 
> at the beginning of the unit. Admittedly, this is just an optimization 
> (and it works fine to process phrase by phrase), but other examples of 
> ppx rewriters could really do non-local processing which cannot be 
> simply captured by a state threaded sequentially.

Right. The point is that some extensions (like eliom) don't fit in the 
framework that merlin or the toplevel could expect for working 
incrementally.

I think that we should focus on providing a simple API for the most 
common cases (stateless transformers, then state monad, then 
backtracking stack, etc.). As long as the plugin informs its host of the 
way it is intended to be used, it is possible to provide graceful 
degradation.

Non compositional transformers are likely to be the exception rather 
than the rule (optimizations aside).
Also, I may stress that in the specific case of merlin, we are not 
interested by the implementation produced by a transformer, only by the 
interface, and this is how extensions are reimplemented right now. So as 
a last resort, plugins could be provide specific support for use inside 
an editor.