moonbit-code-transform

# MoonBit Code Transformations Guide for building source-to-source code generation tools in MoonBit using `moonbitlang/parser` to parse source into AST and `moonbitlang/formatter` to emit generated code. Uses `golem_sdk_tools` as the reference implementation. ## Pipeline Overview Every code transformation follows the same pattern: ``` Source (.mbt files) → Parse with @parser.parse_string() → Extract information from @syntax.Impl list → Construct new @syntax.Impl nodes (AST) → Format with @formatter.impls_to_string() → Write output file ``` ## Required Dependencies In `moon.mod.json`: ```json { "deps": { "moonbitlang/parser": "0.1.16", "moonbitlang/formatter": "..." } } ``` In the library package `moon.pkg` (where AST construction happens): ``` import { "moonbitlang/core/list", "moonbitlang/parser", "moonbitlang/parser/syntax", "moonbitlang/parser/basic", } import { "moonbitlang/formatter", } for "test" ``` In the CLI/main package `moon.pkg` (where formatting and file I/O happen): ``` import { "moonbitlang/formatter", "moonbitlang/x/fs", "your/module/lib" @lib, } ``` **Key separation**: The library package constructs `@list.List[@syntax.Impl]`. The CLI package calls `@formatter.impls_to_string(impls)` and writes the result. Tests in the library import `moonbitlang/formatter` via `for "test"` to verify output without depending on it at runtime. ## Parsing Source Files Use `@parser.parse_string()` to parse MoonBit source into a list of top-level AST items: ```mbt nocheck let (impls, _reports) = @parser.parse_string(content) // impls : @list.List[@syntax.Impl] // _reports : Array[...] — diagnostics, usually ignored for codegen ``` ### Walking the AST to Extract Information Pattern-match on `@syntax.Impl` variants to find declarations: ```mbt nocheck impls.each(fn(impl_) { match impl_ { // Function declarations TopFuncDef(fun_decl~, ..) => { // fun_decl : @syntax.FunDecl let name = fun_decl.name.name // String let vis = fun_decl.vis // @syntax.Visibility let params = fun_decl.decl_params // Option[@list.List[@syntax.Parameter]] let ret = fun_decl.return_type // Option[@syntax.Type] let doc = fun_decl.doc // @syntax.DocString let type_name = fun_decl.type_name // Option[@syntax.TypeName] (for methods) ... } // Type definitions (struct, enum) TopTypeDef(type_decl) => { let name = type_decl.tycon // String let attrs = type_decl.attrs // @list.List[@syntax.Attribute] let doc = type_decl.doc // @syntax.DocString ... } _ => () } }) ``` ### Checking Attributes (Annotations) Attributes like `#derive.agent` are accessed via `type_decl.attrs`: ```mbt nocheck let mut is_agent = false type_decl.attrs.each(fn(attr) { if attr.raw == "#derive.agent" { is_agent = true } }) ``` ### Extracting Function Parameters Parameters come as `@list.List[@syntax.Parameter]`: ```mbt nocheck match fun_decl.decl_params { Some(ps) => ps.each(fn(p) { match p { Positional(binder~, ty=Some(ty)) | Labelled(binder~, ty=Some(ty)) => { let name = binder.name // String // ty : @syntax.Type — process recursively } _ => () } }) None => () } ``` ### Processing Types Recursively `@syntax.Type` is an enum with variants like `Name`, `Option`, `Tuple`, etc.: ```mbt nocheck fn process_type(ty : @syntax.Type) -> MyTypeRepr { match ty { Option(ty~, ..) => MyOptional(process_type(ty)) Name(constr_id~, tys~, ..) => match constr_id.id { Ident(name~) => if name == "Array" { match tys { More(inner, ..) => MyList(process_type(inner)) Empty => ... // error } } else if tys.is_empty() { MySimple(name) } else { ... } Dot(pkg~, id~) => MyQualified(pkg, id) } _ => ... // unsupported } } ``` ## Constructing AST Nodes All AST nodes require a `loc` field. For generated code, use dummy locations: ```mbt nocheck let dummy_pos : @basic.Position = { fname: "", lnum: 0, bol: 0, cnum: 0 } let dummy_loc : @basic.Location = { start: dummy_pos, end: dummy_pos } ``` ### The `@list.List` Pattern The AST uses `@list.List[T]` (immutable linked lists), not `Array[T]`. Build with `Array` first, then convert: ```mbt nocheck fn[T] to_list(arr : Array[T]) -> @list.List[T] { @list.List::from_array(arr) } ``` ### DSL Helper Functions Create a set of small helper functions to make AST construction readable. Organize by category. The reference implementation is in `golem_sdk_tools/lib/ast_helpers.mbt`. #### Primitives ```mbt nocheck fn make_binder(name : String) -> @syntax.Binder { @syntax.Binder::{ name, loc: dummy_loc } } fn make_label(name : String) -> @syntax.Label { @syntax.Label::{ name, loc: dummy_loc } } ``` #### Type Constructors ```mbt nocheck // Simple type: String, Int, MyStruct, etc. fn make_type(name : String) -> @syntax.Type { @syntax.Type::Name( constr_id=@syntax.ConstrId::{ id: @syntax.LongIdent::Ident(name~), loc: dummy_loc, }, tys=to_list([]), loc=dummy_loc, ) } // Generic type: Array[T], Result[T, E], etc. fn make_parameterized_type( name : String, type_args : Array[@syntax.Type], ) -> @syntax.Type { @syntax.Type::Name( constr_id=@syntax.ConstrId::{ id: @syntax.LongIdent::Ident(name~), loc: dummy_loc, }, tys=to_list(type_args), loc=dummy_loc, ) } // Option type: T? fn make_option_type(inner : @syntax.Type) -> @syntax.Type { @syntax.Type::Option(ty=inner, loc=dummy_loc, question_loc=dummy_loc) } ``` #### Literal Expressions ```mbt nocheck fn make_string_expr(s : String) -> @syntax.Expr { @syntax.Expr::Constant(c=@syntax.Constant::String(s), loc=dummy_loc) } fn make_int_expr(n : Int) -> @syntax.Expr { @syntax.Expr::Constant(c=@syntax.Constant::Int(n.to_string()), loc=dummy_loc) } fn make_bool_expr(b : Bool) -> @syntax.Expr { @syntax.Expr::Constant(c=@syntax.Constant::Bool(b), loc=dummy_loc) } ``` Other numeric types use their respective `@syntax.Constant` variants: `UInt("0")`, `Int64("0")`, `UInt64("0")`, `Float("0.0")`, `Double("0.0")`, `Byte("\\x00")`, `Char('a')`. #### Identifier Expressions ```mbt nocheck // Simple identifier: foo fn make_ident_expr(name : String) -> @syntax.Expr { @syntax.Expr::Ident( id=@syntax.Var::{ name: @syntax.LongIdent::Ident(name~), loc: dummy_loc }, loc=dummy_loc, ) } // Qualified identifier: @pkg.foo fn make_qualified_expr(pkg : String, id : String) -> @syntax.Expr { @syntax.Expr::Ident( id=@syntax.Var::{ name: @syntax.LongIdent::Dot(pkg~, id~), loc: dummy_loc }, loc=dummy_loc, ) } // Method reference: TypeName::method_name fn make_method_ref(type_name : String, method_name : String) -> @syntax.Expr { @syntax.Expr::Method( type_name=@syntax.TypeName::{ name: @syntax.LongIdent::Ident(name=type_name), is_object: false, loc: dummy_loc, }, method_name=make_label(method_name), loc=dummy_loc, ) } ``` #### Constructor Expressions ```mbt nocheck // Enum variant without args: None, Ok, Err, etc. fn make_constr_no_args(name : String) -> @syntax.Expr { @syntax.Expr::Constr( constr=@syntax.Constructor::{ name: @syntax.ConstrName::{ name, loc: dummy_loc }, extra_info: @syntax.ConstructorExtraInfo::NoExtraInfo, loc: dummy_loc, }, loc=dummy_loc, ) } // Qualified enum variant: @pkg.Type::Variant fn make_qualified_constr( pkg : String, type_name : String, variant : String, ) -> @syntax.Expr { @syntax.Expr::Constr( constr=@syntax.Constructor::{ name: @syntax.ConstrName::{ name: variant, loc: dummy_loc }, extra_info: @syntax.ConstructorExtraInfo::TypeName(@syntax.TypeName::{ name: @syntax.LongIdent::Dot(pkg~, id=type_name), is_object: false, loc: dummy_loc, }), loc: dummy_loc, }, loc=dummy_loc, ) } ``` #### Function Application ```mbt nocheck fn make_positional_arg(expr : @syntax.Expr) -> @syntax.Argument { @syntax.Argument::{ value: expr, kind: @syntax.ArgumentKind::Positional } } // f(arg1, arg2, ...) fn make_apply(func : @syntax.Expr, args : Array[@syntax.Expr]) -> @syntax.Expr { @syntax.Expr::Apply( func~, args=to_list(args.map(make_positional_arg)), attr=@syntax.ApplyAttr::NoAttr, loc=dummy_loc, ) } // self.method(arg1, arg2, ...) fn make_dot_apply( self_ : @syntax.Expr, method_name : String, args : Array[@syntax.Expr], ) -> @syntax.Expr { @syntax.Expr::DotApply( self=self_, method_name=make_label(method_name), args=to_list(args.map(make_positional_arg)), return_self=false, attr=@syntax.ApplyAttr::NoAttr, loc=dummy_loc, ) } // lhs op rhs (e.g., a + b, x == y) fn make_infix( op : String, lhs : @syntax.Expr, rhs : @syntax.Expr, ) -> @syntax.Expr { @syntax.Expr::Infix( op=@syntax.Var::{ name: @syntax.LongIdent::Ident(name=op), loc: dummy_loc }, lhs~, rhs~, loc=dummy_loc, ) }