jnigen-migrate-yaml-to-dart

# Migrating JNIgen YAML Configuration to Modern Dart Code ## Contents - [Introduction & Motivation](#introduction--motivation) - [Step-by-Step Migration Workflow](#step-by-step-migration-workflow) - [Comprehensive YAML to Dart API Mapping](#comprehensive-yaml-to-dart-api-mapping) - [1. Input & Java API Discovery](#1-input--java-api-discovery) - [2. Output Configuration](#2-output-configuration) - [3. Maven Dependencies](#3-maven-dependencies) - [4. Android SDK & Gradle Integration](#4-android-sdk--gradle-integration) - [5. Symbol Imports & Class Hiding](#5-symbol-imports--class-hiding) - [6. Nullability Annotations](#6-nullability-annotations) - [7. Logging & Execution](#7-logging--execution) - [8. Obsolete, Removed & Deprecated Fields](#8-obsolete-removed--deprecated-fields) - [9. Modern Dart-Only Capabilities](#9-modern-dart-only-capabilities) - [Before & After Migration Examples](#before--after-migration-examples) - [Example 1: Android SDK & Gradle Flutter Plugin](#example-1-android-sdk--gradle-flutter-plugin) - [Example 2: Maven Java Library (Multi-File Package Structure)](#example-2-maven-java-library-multi-file-package-structure) - [Example 3: Cross-Package Symbol Imports & Nullability](#example-3-cross-package-symbol-imports--nullability) - [Example 4: Legacy Dart + C Bindings Migration](#example-4-legacy-dart--c-bindings-migration) - [Example 5: AST Filtering & Renaming with Modern Visitors](#example-5-ast-filtering--renaming-with-modern-visitors) - [Verification Checklist](#verification-checklist) - [Troubleshooting & Common Pitfalls](#troubleshooting--common-pitfalls) --- ## Introduction & Motivation Historically, `package:jnigen` used static YAML configuration specified either in a standalone `jnigen.yaml` file or under the `jnigen:` key in `pubspec.yaml`, executed via `dart run jnigen`. Starting with `jnigen` 1.0+, programmatic configuration in a dedicated Dart script (typically `tool/jnigen.dart`) using `JniGenerator` is the standard and recommended approach: 1. **Compile-Time Safety & Autocomplete**: Benefit from static typing, IDE code completion, and immediate compile-time errors instead of cryptic runtime YAML parsing failures. 2. **Full Dart Power**: Use arbitrary Dart logic (loops, sets, regular expressions, environment variables, closures) and AST `Visitor` passes for renaming and fine-grained symbol filtering. 3. **Future Proofing**: YAML configuration support is being phased out in favor of programmatic Dart configuration scripts across the Dart native interop ecosystem (`jnigen`, `ffigen`). 4. **Direct Path Resolution**: Reliable path resolution via `Platform.script.resolve('../')` avoids directory-dependent path breakage when running code generators from subdirectories or CI pipelines. --- ## Step-by-Step Migration Workflow Follow this systematic 9-step workflow to migrate any package from YAML configuration to a modern Dart generator script without introducing breaking changes or unintended diffs: ```mermaid flowchart TD S1["1. Upgrade package:jnigen & pub get"] --> S2["2. Run legacy YAML config (baseline)"] S2 --> S3["3. Backup generated files (<output>.temp_backup)"] S3 --> S4["4. Create tool/jnigen.dart"] S4 --> S5["5. Translate YAML keys to JniGenerator API"] S5 --> S6["6. Run dart run tool/jnigen.dart"] S6 --> S7{"7. Diff against backup"} S7 -- "Differences found" --> S5 S7 -- "Identical public API" --> S8["8. Delete backup & legacy YAML config"] S8 --> S9["9. Verify: dart format, analyze & test"] ``` ### Step 1: Upgrade `package:jnigen` and Update Dependencies Ensure the package has the latest `jnigen` dependency under `dev_dependencies` in `pubspec.yaml`: ```bash dart pub add dev:jnigen ``` Run `dart pub get` (or `flutter pub get` for Flutter projects) to update lockfiles and dependencies. ### Step 2: Establish a Clean Baseline Before making any changes, run the existing legacy YAML generator to ensure that the current bindings are cleanly generated and reproducible: ```bash # If config is in jnigen.yaml or pubspec.yaml: dart run jnigen # Or if a custom config file was used: dart run jnigen --config jnigen.yaml ``` Verify that `git status` reflects a clean working tree (or commit existing changes first). If there are significant changes to the bindings output due to upgrading jnigen, inform the user. ### Step 3: Copy Existing Generated Bindings to a Temporary Backup Make a temporary copy of every generated file or directory so that the newly generated bindings can be diffed line-by-line. For example: - **Single-file layout**: ```bash cp lib/src/generated_bindings.dart lib/src/generated_bindings.temp_backup.dart ``` - **Package-structure layout (multi-file directory)**: ```bash cp -r lib/src/third_party lib/src/third_party_temp_backup ``` - **Generated symbol file (`symbols.yaml`) (if applicable)**: ```bash cp symbols.yaml symbols.temp_backup.yaml ``` ### Step 4: Create the Dart Configuration Script Create the generator entrypoint. The typical location is `tool/jnigen.dart`: ```dart import 'dart:io'; import 'package:jnigen/jnigen.dart'; void main() async { final packageRoot = Platform.script.resolve('../'); final generator = JniGenerator( // Configuration translated in Step 5... ); await generator.generate(); } ``` ### Step 5: Translate YAML Keys into Modern Dart API Inspect the legacy YAML configuration and systematically translate each section into `JniGenerator` parameters using the [Comprehensive YAML to Dart API Mapping](#comprehensive-yaml-to-dart-api-mapping) below: The most important change is that filtering and renaming is now performed using `Visitor`s. ### Step 6: Execute the New Generator Script Run the newly created generator script from the package root: ```bash dart run tool/jnigen.dart ``` ### Step 7: Diff Newly Generated Bindings Against the Backup Compare the new output with the temporary backup: - **For single-file output**: ```bash git diff --no-index lib/src/generated_bindings.temp_backup.dart lib/src/generated_bindings.dart ``` - **For directory / package structure output**: ```bash git diff --no-index lib/src/third_party_temp_backup lib/src/third_party ``` **Verification rules**: - **Allowed diffs**: Trivial differences such as formatting, renaming of internal-only methods or variables, reordering of the bindings, or the names of positional parameters. - **Forbidden diffs**: Any differences in public APIs, class names, method signatures, field types, etc. It's critical that there are no breaking changes, but we also don't want to add new classes or methods unnecessarily. - If unintended diffs exist, adjust the script and re-run until the diff is clean. ### Step 8: Clean Up Legacy Files and References 1. Delete the temporary backup files: ```bash rm -rf lib/src/*.temp_backup* lib/src/*_temp_backup ``` 2. Remove the legacy YAML configuration: - If using `jnigen.yaml`, delete the file (`rm jnigen.yaml`). - If the configuration is inside `pubspec.yaml`, remove the `jnigen:` section entirely. 3. Update repository scripts and documentation: - Update any `README.md` instructions referencing `dart run jnigen` to `dart run tool/jnigen.dart`. - Update any CI workflows (e.g., `.github/workflows/*.yml`), Makefile targets, or build scripts to invoke `dart run tool/jnigen.dart`. ### Step 9: Final Verification Loop (Format, Analyze, Test) Execute the complete Dart verification suite in the target package root: 1. **Format code**: ```bash dart format . ``` Ensures both `tool/jnigen.dart` and generated files adhere to canonical Dart formatting. 2. **Static analysis**: ```bash dart analyze ``` Ensures zero errors, warnings, or lints across the entire package. If generated bindings trigger lints, append the appropriate `// ignore_for_file:` directives to the `preamble` of `Output` in `tool/jnigen.dart` (never disable lints globally in `analysis_options.yaml`). 3. **Run tests**: ```bash dart test ``` Ensures all unit, integration, and JNI runtime tests execute and pass cleanly. --- ## Comprehensive YAML to Dart API Mapping ### 1. Input & Java API Discovery All input sources, classes, classpaths, and Java summarizer settings map into `Input(...)`: | Legacy YAML Key | Type | Modern Dart API Equivalent | Notes | | :--- | :--- | :--- | :--- | | `classes` * | `List<String>` | `Input(classes: ['com.example.MyClass', 'com.example.pkg'])` | Target classes or packages. Specifying a parent class pulls in nested classes. Do not use `$` notation for nested classes. | | `source_path` | `List<String>` | `Input(sourcePath: [packageRoot.resolve('android/src/main/java')])` | Directories to search for Java source files. Takes `List<Uri>`. | | `class_path` | `List<String>` | `Input(classPath: [packageRoot.resolve('libs/library.jar')])` | Paths to JAR files or directories for compiled Java classes. Takes `List<Uri>`. | | `summarizer.backend` | `'auto'`, `'doclet'`, or `'asm'` | `Input(backend: SummarizerBackend.asm)` | Backend engine for summarizer. `auto` in YAML is represented by omitting `backend` (defaults to `null`). | | `summarizer.extra_args` | `List<String>` | `Input(extraArgs: ['-Xlint:none'])` | Extra CLI arguments passed directly to the summarizer tool. | | `summarizer.working_dir` | `String` | `Input(workingDirectory: packageRoot.resolve('build/'))` | Working directory where summarizer executes. Defaults to `Uri.directory('.')`. | | `summarizer.command` | `String` | `Input(summarizerCommand: 'java -jar /path/to/ApiSummarizer.jar')` | Command used to invoke a prebuilt summarizer JAR instead of compiling it with Gradle. | ### 2. Output Configuration Dart code emission, file structure, symbol export, and header comments map into `Output(...)` and `DartOutput(...)`: | Legacy YAML Key | Type | Modern Dart API Equivalent | Notes | | :--- | :--- | :--- | :--- | | `output.dart.path` * | `String` | `Output(dart: DartOutput(path: packageRoot.resolve('lib/bindings.dart')))` | Primary Dart bindings output path (`Uri`). For single-file mode, must end in `.dart`. For package structure, must end in `/` or use `Uri.directory`. | | `output.dart.structure` | `'single_file'` or `'package_structure'` | `DartOutput(..., structure: OutputStructure.singleFile)` | Layout mode: `OutputStructure.singleFile` or `OutputStructure.packageStructure` (default). | | `output.symbols` | `String` | `Output(..., symbols: SymbolsOutput(packageRoot.resolve('symbols.yaml')))` | Path to export generated symbol definitions (`Uri`). Must end in `.yaml`. | | `preamble` | `String` | `Output(..., preamble: '// Header comments\n')` | Header string injected at the top of every generated Dart file (licenses, lints suppression). | | `generate_stubs` | `bool` | `Output(..., generateStubs: true)` | Generates minimal Dart stubs for referenced but unincluded Java classes. Defaults to `true`. | | `format` | `bool` | `Output(..., format: true)` | Automatically runs `dart format` on generated files. Defaults to `true`. | ### 3. Maven Dependencies Automatic downloading of Maven source jars and compiled jars maps into `MavenDownloads(...)` under `Input(mavenDownloads: ...)`: | Legacy YAML Key | Type | Modern Dart API Equivalent | Notes | | :--- | :--- | :--- | :--- | | `maven_downloads.source_deps` | `List<String>` | `MavenDownloads(sourceDeps: ['org.apache.pdfbox:pdfbox:2.0.26'])` | Maven dependencies (`groupId:artifactId:version`) to download and unpack sources for. Sources are automatically added to `sourcePath`. | | `maven_downloads.source_dir` | `String` | `MavenDownloads(sourceDir: packageRoot.resolve('mvn_java/'))` | Directory where Maven sources are extracted (`Uri`). Defaults to `mvn_java/`. | | `maven_downloads.jar_only_deps` | `List<String>` | `MavenDownloads(jarOnlyDeps: ['org.slf4j:slf4j-api:1.7.36'])` | Maven dependencies to download JARs for only (without sources). JARs are automatically added to `classPath`. | | `maven_downloads.jar_dir` | `String` | `MavenDownloads(jarDir: packageRoot.resolve('mvn_jar/'))` | Directory where Maven JARs are stored (`Uri`). Defaults to `mvn_jar/`. | ### 4. Android SDK & Gradle Integration Android platform stub resolution and Gradle classpath discovery map into `AndroidSdk(...)` under `Input(androidSdk: ...)`: | Legacy YAML Key | Type | Modern Dart API Equivalent | Notes | | :--- | :--- | :--- | :--- | | `android_sdk_config.versions` | `List<int>` | `AndroidSdk(versions: [34, 33])` | Android SDK API versions to search in decreasing preference order. | | `android_sdk_config.sdk_root` | `String` | `AndroidSdk(sdkRoot: Uri.directory('/path/to/android-sdk'))` | Custom Android SDK installation directory (`Uri`). If omitted, uses the `ANDROID_SDK_ROOT` environment variable. | | `android_sdk_config.add_gradle_deps` | `bool` | `AndroidSdk(addGradleDeps: true)` | Runs a Gradle stub to determine the actual compile classpath of the Android subproject. Requires `flutter pub get` beforehand. Defaults to `false`. | | `android_sdk_config.add_gradle_sources` | `bool` | `AndroidSdk(addGradleSources: true)` | Runs a Gradle stub to obtain source dependencies of the Android project. Defaults to `false`. |