Dear wg-camp4 users,<div><br></div><div> So far, the discussion is really interesting and quite helpful, but I want to talk about the meta-programming <b>from the point of the view of implementation side,</b> features are easy to propose, but maybe only the compiler/library writers know how hard to implement, I do appreciate that you can sit dow and read the long email, I am also starting to blog(<a href="http://hongboz.wordpress.com/">http://hongboz.wordpress.com/</a>) about how to do syntactic meta programming (SMP) in a right way.</div>
<div> I rewrite the whole camlP4(named Fan) from scratch, building the quotation kit and throw away the crappy grammar parser, so plz believe me <b>that I do understand the whole technology stack of camlP4</b>, if we could reach some consensus, I would be happy to handle over the maintaining of Fan, Fan does not loose any feature compared with camlP4, in fact it has more interesting featrues.</div>
<div><br></div><div> Let's begin with some easy, not too technical parts which has a significant effect on user experience though:</div><div> 1. Performance</div><div> Performance does matter, it's a shame that the most time spent in compiling the ocaml compiler is dedicated to camlP4, but it is an engineering problem, currently compiling Fan only takes less than 20s, and it can be improved further</div>
<div> 2. Building issues</div><div> The design of having side effects by dynamic loading is generically a bad idea, in Fan<b> the dynamic loading only register some functionality the Fan support,</b> it <b>does not have any other side effec</b>t, each file stands alone says which (ppx , or filters, or syntax) it want to use with a good default option. so the building is always something like '-pp fan pluging1 plugin2 plugin3', <b>the order of pulgings does not matter</b>, also, l<b>oading all the plugins you have does not have any side effect, even better, you can do the static linking all the plugins you collected, the building process is simplified. </b></div>
<div><b> </b> 3. Grammar Extension (<b>Language namespace</b>)</div><div><b> </b>I concur that grammar extension arbitrarily is a bad idea, and I agree with Gabrier that so far only the quotation(Here quotation means delimited DSL, quosi-quotation means Lisp style macros) is modular, composable, and I also agree with Gabrier -ppx<b> should not be used to do syntax overriding (this should not be called syntax extension actually), </b>that's a terrible idea to do syntax overriding, since the user never understand what's going on underly without reading the Makefile. So here some my suggestion is that some really conevenient syntax extesion, i.e, (let try.. in) should be merged to the built in parser. quotations does not bring too much heavy syntax (imho). In Fan, we proposed the concept of a hierarchical language name space, since once quotation is heavily used, it's really easy to introduce conflict, <b>the language namespace querying is exactly like java package namespace,</b> you can import, close import to save some typing.</div>
<div> Here is a taste</div><div> -----------------------------------------------------------------------------------------------</div><div> {:.Fan.Lang.Meta.expr| a + b |} ------> </div><div> <span style="background-color:rgb(255,255,255);color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px">`App (`App ((`Id (`Lid "+")), (`Id (`Lid "a")))), (`Id (`Lid "b")))</span></div>
<div> {:.Fan.Lang.Meta.N.expr| a + b |} -----></div><div> <span style="background-color:rgb(255,255,255);color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px">`App</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
(_loc,</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)"> (`App</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)">
(_loc, (`Id (_loc, (`Lid (_loc, "+")))),</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px;background-color:rgb(255,255,255)"> (`Id (_loc, (`Lid (_loc, "a")))))),</div>
<div><span style="background-color:rgb(255,255,255);color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.800000190734863px"> (`Id (_loc, (`Lid (_loc, "b")))))</span> </div><div><div> -----------------------------------------------------------------------------------------------</div>
<div> the .Fan.Lang.Meta.expr the first '.' means it's from the absolute namespace, the <b>N.expr shares exactly the same syntax without location</b>, though</div></div><div><br></div><div> 4. Portable to diffierrent compiler extensions(like LexiFi's fork of ocaml)</div>
<div> I am pretty sure it's pretty easy to do in Fan, only Ast2pt (dumping the intemediate Ast into Parsetree) part need to be changed to diffierent compilers.<br clear="all"><div><br></div><div>----------------------------------------------------------------------------------------------------------------</div>
<div>Now let's talk about some internal parts of SMP.</div><div>Quasi-Quotation is the essential part of SMP, I am surprised so far that the discussion <b>silently ignores the quasi-quotation,</b> Leo's answer of writing three parsers is neither satisfying nor practical(imho). </div>
<div><br></div><div>Camlp4 is mainly composed of two parts, one is the extensible parser and <b>the other significant part is Ast Lifting</b>. Since we all agree that extensible parser increases the complexity too much, let's simply ignore that part.</div>
<div><br></div><div>The Ast Lifting are tightly coupled <b>with the design of the Abstract Syntax Tree.</b> People complain about that Camlp4 Ast is hard to learn and using quasi-quotation to do the pattern match is a bad idea.</div>
<div><br></div><div>Let me explain the topic a bit:</div><div> Camlp4Ast is hard to learn, I agree, it has some alien names that nobody understand what it means, quosi-quotation <b>is definitely a great idea</b> to boom the meta-programming, but my experience here is <b>for very very small Ast fragment, using the Abstract Syntax Tree directly,</b> otherwise Quasi-quotation is a life saver to do the meta programming.</div>
<div> Luckily the quotation kit has nothing to do with the parser part, it's simply several functions(I did some simplify a bit) which turns a normal runtime </div><div>value into an Ast node generically, <b>such kind functions are neither easy to write nor easy to read</b>, <b>the idea case is that it should be generated once for all, and all the data types in normal ocaml</b> <b>should be derived automatically</b>(some ADT with functions can not be derived). <b>I bet it's mostly likely a nightmare if we maintain 3 parsers for the ocaml grammar while two other parsers dumping to a meta-level</b></div>
<div> </div><div> So, how to make Ast Lifting easier, </div><div> The first guideline is <b>"Don't mixing with records", </b></div><div><b> </b>Once you encoding AST with records, you have to encode the records in the meta level which increases the complexity without bringing any new features, <b>it's simply not worthwhile.</b></div>
<div><b><br></b></div><div><b> </b> The second guideline is "Don't do <b>any </b>syntax desugaring" , syntax desguaring makes the semantics of syntax meta programming a bit weird. Syntax desguaring happens everywhere in Parsetree, think about the list literals, it uses the syntax desuaring, if you don't use any syntax desugaring, for example, you want to match the bigarray access, you simply needed to match `Bigarray(..)' instead of </div>
<div><div><br></div><div>Pexp_apply</div><div> ({pexp_desc=Pexp_ident</div><div> {txt= Ldot (Ldot (Lident "Bigarray", array), ("get"|"set" as gs)) ;_};_},</div><div>
label_exprs)</div></div><div>----------------------------</div><div> The third guideline is to<b> </b>make it <b>as uniform as possible</b></div><div><b> </b>This not only helps the user, but <b>it helps the meta-programming over types to derive some utility types. </b>Take a look at my Ast encoding in Fan <a href="https://github.com/bobzhang/Fan/blob/master/src/Ast.ml">https://github.com/bobzhang/Fan/blob/master/src/Ast.ml</a> (it needs to be polished, plz don't panic when you see variants I use here)</div>
<div><b> </b>The initial Ast has locations and ant support, but<b> here we derive 3 other Asts thanks to my very regular design</b>.<b> AstN is the Ast without locations</b>, the locations are important, but it is simply not too much helpful when you only do the code generation, but it complicates the expanded code a lot), <b>AstA is the Ast without antiquotations(simply remove the ant branch), </b>it is a subtype of Ast(thanks to the choice we use variants here), <b>AstNA is the Ast without neither locations nor antiquotations</b>), it is a subtype of AstN. <b>In practice, I found the Ast without locations is particular helpful when you only do the code generation, it simplifies this part significantly.<i><u> The beautif</u></i></b><u><b><i>u</i></b><i style="font-weight:bold">l part is that all the four Ast share the same grammar with the same quosiquotatoin mechanism, as I showed .Fan.Lang.N.expr and .Fan.Lang.expr</i></u></div>
<div> I don't know how many parsers you have to maintain to reach such a goal or it's never going to happen.</div><div> Using variants to encode the intermediate ast has a lots of other benefits, but I don't want to cover it in such a short mail.</div>
<div><br></div><div> So,<b> my proposal is that the community design an Intermediate Ast together, and write a built-in parser to such Intermediate Ast then dump to Parsetree, but I am for that Parsetree still needs to be cleaned a bit but not too much change . </b>I do appreciate you can take something away from Fan, I think the Parsetree is<b> not the ideal part</b> to do SMP, HTH</div>
<div><div><br></div>-- <br>-- Regards, Hongbo
</div></div>