ffigen-migrate-yaml-to-dart

# Migrating FFIgen YAML Configuration to Modern Dart Code ## Contents - [Introduction & Motivation](#introduction--motivation) - [Step-by-Step Migration Workflow](#step-by-step-migration-workflow) - [Comprehensive YAML to Dart API Mapping](#comprehensive-yaml-to-dart-api-mapping) - [1. General & Clang Settings](#1-general--clang-settings) - [2. Output Configuration](#2-output-configuration) - [3. Binding Styles & Library Settings](#3-binding-styles--library-settings) - [4. Declaration Filters & Renaming](#4-declaration-filters--renaming) - [5. Category-Specific Settings](#5-category-specific-settings) - [6. Symbol Files & Type Mappings](#6-symbol-files--type-mappings) - [7. Objective-C Configuration](#7-objective-c-configuration) - [8. Obsolete & Ignored YAML Fields](#8-obsolete--ignored-yaml-fields) - [9. Modern Dart-Only Features](#9-modern-dart-only-features) - [Before & After Migration Examples](#before--after-migration-examples) - [Example 1: Dynamic Library C Bindings](#example-1-dynamic-library-c-bindings) - [Example 2: Modern Static Native External Bindings](#example-2-modern-static-native-external-bindings) - [Example 3: Objective-C Framework Bindings](#example-3-objective-c-framework-bindings) - [Verification Checklist](#verification-checklist) - [Troubleshooting & Common Pitfalls](#troubleshooting--common-pitfalls) --- ## Introduction & Motivation Historically, `package:ffigen` relied on a static YAML configuration specified either in `ffigen.yaml` or under the `ffigen:` key in `pubspec.yaml`, executed via `dart run ffigen`. Starting with `ffigen` 16+, programmatic code configuration via `FfiGenerator` in a Dart script (typically `tool/ffigen.dart`) is the standard and recommended approach: 1. **Compile-Time Safety & Autocomplete**: Benefit from static typing, IDE code completion, and immediate compile-time errors instead of cryptic runtime YAML parsing failures. 2. **Full Dart Power**: Use arbitrary Dart logic (loops, sets, custom regular expressions, closures, transformations) inside AST `Visitor`s rather than dealing with fragile YAML regex mappings. 3. **Future Proofing**: YAML configuration support is being deprecated and phased out in favor of the programmatic Dart API. 4. **Advanced Features**: Direct access to modern features such as `@RecordUse` tree-shaking maps (`recordUseMapping`), experimental C++ support (`Cpp`), and composable symbol file loaders (`importFromSymbolFiles`). --- ## Step-by-Step Migration Workflow Follow this systematic 9-step workflow to migrate any package from YAML configuration to a modern Dart generator script without introducing breaking changes or unintended diffs: ```mermaid flowchart TD S1["1. Upgrade package:ffigen & pub get"] --> S2["2. Run legacy YAML config (baseline)"] S2 --> S3["3. Backup generated file (<output>.temp_backup.dart)"] S3 --> S4["4. Create tool/ffigen.dart"] S4 --> S5["5. Translate YAML keys to FfiGenerator API"] S5 --> S6["6. Run dart run tool/ffigen.dart"] S6 --> S7{"7. Diff against backup"} S7 -- "Differences found" --> S5 S7 -- "Identical public API" --> S8["8. Delete backup & legacy YAML config"] S8 --> S9["9. Verify: dart format, analyze & test"] ``` ### Step 1: Upgrade `package:ffigen` and Update Dependencies Ensure the package has the latest `ffigen` dependency under `dev_dependencies` in `pubspec.yaml`: ```bash dart pub add dev:ffigen ``` Run `dart pub get` to ensure all lockfiles and dependencies are up to date. ### Step 2: Establish a Clean Baseline Before making any changes, run the existing legacy YAML generator to ensure that the current bindings are cleanly generated and reproducible: ```bash # If config is in pubspec.yaml or ffigen.yaml: dart run ffigen # Or if a custom config file was used: dart run ffigen --config ffigen.yaml ``` Verify that `git status` reflects a clean working tree (or commit existing changes first). If there are significant changes to the bindings output due to upgrading ffigen, inform the user. ### Step 3: Copy Existing Generated Bindings to a Temporary Backup Make a temporary copy of every generated file so that the newly generated bindings can be diffed line-by-line. For example: ```bash cp lib/src/generated_bindings.dart lib/src/generated_bindings.temp_backup.dart ``` *(If Objective-C `.m` files or symbol files are also generated, back them up as well).* ### Step 4: Create the Dart Configuration Script Create the generator script. The typical location is `tool/ffigen.dart`: ```dart import 'dart:io'; import 'package:ffigen/ffigen.dart'; Future<void> main() async { final packageRoot = Platform.script.resolve('../'); final generator = FfiGenerator( // Configuration translated in Step 5... ); await generator.generate(); } ``` ### Step 5: Translate YAML Keys into Modern Dart API Inspect the YAML configuration and systematically translate each section into `FfiGenerator` parameters using the [Comprehensive YAML to Dart API Mapping](#comprehensive-yaml-to-dart-api-mapping) below. The most important change is that filtering and renaming is now performed using `Visitor`s, and the default behavior is to *exclude* all top level bindings. ### Step 6: Execute the New Generator Script Run the newly created generator script from the package root: ```bash dart run tool/ffigen.dart ``` ### Step 7: Diff Newly Generated Bindings Against the Backup Compare the new output with the temporary backup. Do this for each binding file if there are multiple files. For example: ```bash git diff --no-index lib/src/generated_bindings.temp_backup.dart lib/src/generated_bindings.dart ``` **Verification rules**: - **Allowed diffs**: Trivial differences such as formatting, renaming of internal-only methods or variables, reordering of the bindings, or the names of positional parameters. - **Forbidden diffs**: Any differences in public APIs, class names, method signatures, struct fields, enum constants, native types, leaf annotations, packing annotations, etc. It's critical that there are no breaking changes, but we also don't want to add new classes or methods unnecessarily. - If unintended diffs exist, adjust the script and re-run until the diff is clean. ### Step 8: Clean Up Legacy Files and References 1. Delete the temporary backup file: ```bash rm lib/src/generated_bindings.temp_backup.dart ``` 2. Remove the legacy YAML configuration: - If using `ffigen.yaml`, delete the file (`rm ffigen.yaml`). - If the configuration is inside `pubspec.yaml`, remove the `ffigen:` section entirely. 3. Update repository scripts and documentation: - Update any `README.md` instructions referencing `dart run ffigen` to `dart run tool/ffigen.dart`. - Update any CI workflows (e.g., `.github/workflows/*.yml`), Makefile targets, or build scripts to invoke `dart run tool/ffigen.dart`. ### Step 9: Final Verification Loop (Format, Analyze, Test) Execute the complete Dart verification suite in the target package root: 1. **Format code**: ```bash dart format . ``` Ensures both `tool/ffigen.dart` and generated files adhere to canonical Dart formatting. 2. **Static analysis**: ```bash dart analyze ``` Ensures zero errors, warnings, or lints across the entire package. If generated bindings trigger lints, append the appropriate `// ignore_for_file:` directives to the `preamble` of `Output` in `tool/ffigen.dart` (never disable lints globally in `analysis_options.yaml`). 3. **Run tests**: ```bash dart test ``` Ensures all unit, integration, and FFI tests execute and pass cleanly. --- ## Comprehensive YAML to Dart API Mapping ### 1. General & Clang Settings | Legacy YAML Key | Type | Modern Dart API Equivalent | Notes | | :--- | :--- | :--- | :--- | | `headers.entry-points` | `List<String>` | `Input(entryPoints: [packageRoot.resolve('header.h')])` | Takes `List<Uri>`. Globs can be resolved via `Directory.listSync()` or explicit file lists. | | `headers.include-directives` | `List<String>` | `Input(include: (Uri header) => bool)` | Filter closure to include/exclude transitively imported headers. | | `compiler-opts` | `String` or `List<String>` | `Input(compilerOptions: ['-I/path', ...])` | List of command-line compiler options passed directly to libclang. | | `compiler-opts-automatic.macos.include-c-standard-library` | `bool` | `defaultCompilerOpts(Logger.root, macIncludeStdLib: true)` | Automatically includes macOS standard library headers when compiling with Clang on macOS. | | `ignore-source-errors` | `bool` | `Input(ignoreSourceErrors: true)` | Silences compiler warnings/errors occurring inside native source headers. | | `llvm-path` | `List<String>` | `FfiGenerator(..., libclangDylib: Uri.file('/path/to/libclang.dylib'))` | Custom libclang path. By default, FFIgen automatically locates libclang on Linux, macOS, and Windows. | | `language` | `'c'` or `'objc'` | `FfiGenerator(objectiveC: const ObjectiveC())` | Set `objectiveC` to enable Objective-C parsing. Default (`null`) is C. | ### 2. Output Configuration | Legacy YAML Key | Type | Modern Dart API Equivalent | Notes | | :--- | :--- | :--- | :--- | | `output` (string) | `String` | `Output(dart: DartOutput(path: packageRoot.resolve('bindings.dart')))` | Primary Dart bindings output path. | | `output.bindings` | `String` | `Output(dart: DartOutput(path: packageRoot.resolve('bindings.dart')))` | Output Dart file when `output` is a map. | | `output.objc-bindings` | `String` | `Output(..., objectiveCFile: packageRoot.resolve('bindings.m'))` | Objective-C implementation glue file (defaults to `${dart.path}.m`). | | `output.symbol-file.output` & `output.symbol-file.import-path` | `Map` | `Output(symbolFile: SymbolFile(Uri.parse('package:pkg/bindings.dart'), packageRoot.resolve('symbols.yaml')))` | Exports symbols for reuse by downstream packages. | | `preamble` | `String` | `Output(preamble: '...')` | Header string injected at the top of the generated file (license, lints suppression). | ### 3. Binding Styles & Library Settings | Legacy YAML Key | Type | Modern Dart API Equivalent | Notes | | :--- | :--- | :--- | :--- | | `ffi-native` | `Map` or empty | `Output(style: const NativeExternalBindings(assetId: '...'))` | Generates `@Native` external functions for `dart:ffi`. | | `ffi-native.asset-id` | `String` | `NativeExternalBindings(assetId: 'package:my_pkg/asset')` | Asset identifier for native assets loading. | | `name` | `String` | `DynamicLibraryBindings(wrapperName: 'MyLib')` | Class name when using dynamic library binding style. Default is `NativeLibrary`. | | `description` | `String` | `DynamicLibraryBindings(wrapperDocComment: 'Bindings to MyLib.')` | Documentation comment placed on the generated wrapper class. | | `comments` | `bool` or `Map` | `Output(commentType: CommentType(...))` | Configures doc comments: `CommentType.none()`, `CommentType.def()`, or `CommentType(CommentStyle.any, CommentLength.full)`. | ### 4. Declaration Filters & Renaming In YAML, filters and renames were configured with string matching and regex maps under declaration keys. In the Dart API, all filtering and renaming occurs inside the `visitors: [Visitor(...)]` list. All top level nodes are excluded by default. | Legacy YAML Feature | Modern Dart `Visitor` Implementation | | :--- | :--- | | `exclude-all-by-default: true` | Do not set `node.isIncluded = true` generally; only set `node.isIncluded = true` on desired nodes. | | `functions.include: ['funcA', 'prefix_.*']` | `Visitor(func: (node) { if (node.name == 'funcA' \|\| node.name.startsWith('prefix_')) node.isIncluded = true; })` | | `functions.exclude: ['dispose']` | `Visitor(func: (node) { if (node.name == 'dispose') node.isIncluded = false; })` | | `structs.rename: {'_(.*)': '$1'}` | `Visitor(struct: (node) { if (node.name.startsWith('_')) node.name = node.name.substring(1); })` | | `structs.member-rename: {'.*': {'_(.*)': '$1'}}` | `Visitor(field: (node) { if (node.name.startsWith('_')) node.name = node.name.substring(1); })` | | `enums.member-rename: {'MyEnum': {'kValue': 'value'}}` | `Visitor(enumConstant: (node) { if (node.parent.originalName == 'MyEnum' && node.name == 'kValue') node.name = 'value'; })` | | `functions.member-rename` (parameter rename) | `Visitor(param: (node) { if (node.parent.originalName == 'myFunc' && node.name == 'in') node.name = 'input'; })` | ### 5. Category-Specific Settings | Legacy YAML Key | Type | Modern Dart API Equivalent | | :--- | :--- | :--- | | `functions.leaf` | Include/Exclude | `Visitor(func: (node) { if (node.name == 'fastFunc') node.isLeaf = true; })` | | `functions.symbol-address` | Include/Exclude | `Visitor(func: (node) { if (node.name == 'sum') node.exposeSymbolAddress = true; })` |