# .NET Source Build Assets
This repository contains source, tools, and infrastructure for auxilary packages required to [build .NET from source](https://github.com/dotnet/source-build).
This repository is often referred to as `SBRP`.
This repo supports the following package types:
1. **External** - A package that resides outside of the [dotnet](https://github.com/dotnet) organization required to build .NET from source.
Examples include Newtonsoft.Json and Application Insights for .NET.
Git submodules are utilized to reference the external source.
1. **Reference** - A package that contains API signatures, no implementation.
It enables developers to build, targeting a specific library version without having the full implementation assembly for that version.
1. **Targeting** - A bundle of reference packages that target a specific .NET TFM.
1. **Text only** - Packages that contain text-only assets.
## Supported Platforms
.NET source build currently only supports Linux but generating reference and text-only packages
is supported on both Windows and Unix based operating systems.
## Building
``` bash
./build.sh -sb
```
## Adding/Upgrading Packages
New packages are needed from time to time as
[existing dependency versions are upgraded](https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/update-dependencies.md)
and [new dependencies are added](https://github.com/dotnet/source-build/blob/main/Documentation/sourcebuild-in-repos/new-dependencies.md)
to .NET. The following sections describe how to add/upgrade the various types of packages.
> **Note:** The instructions below apply to the `main` branch. Servicing branches (e.g. `release/9.0`)
> may have different processes for generating and adding packages. When working on a servicing branch,
> always refer to the README on that branch for the correct instructions.
* [External](#external)
* [Reference](#reference)
* [Targeting](#targeting)
* [Text only](#text-only)
### External
#### Adding a New External Component
1. Add the repo as a submodule to `./src/externalPackages/src`
```bash
git submodule add <remote_url> ./src/externalPackages/src/<destination_dir>
git commit -m "<commit_message>"
```
1. Define a [project](src/externalPackages/projects) for the new component.
The project is responsible for building the submodule with the appropriate configuration for source build.
See the [existing projects](src/externalPackages/projects) for examples.
Include a `SourceRevisionId` property set to the submodule's commit hash.
This ensures the correct commit hash is embedded in the built binaries rather than the VMR's.
If the component computes `FileVersion` from non-deterministic values (e.g., `DateTime.Now`),
include a `FileVersionRevision` property set to the revision from the Microsoft-shipped package
and pass it via `/p:FileVersion` in the build command args. Per-component overrides like this
live in a sibling `src/externalPackages/projects/<component>.props` file that the `.proj`
imports via `<Import Project="$(MSBuildThisFileDirectory)$(MSBuildProjectName).props" />`.
In that same `<component>.props`, also declare a `FileVersionValidationPackage` **item** naming a
NuGet package produced by the component so the metadata-update script and tests can validate the
version. The default property name bindings (defined in `src/externalPackages/projects/validation.props`
via `<ItemDefinitionGroup>`, imported by each per-component `.props`) point at conventional
property names — for the common case of one item per project using those names, no per-item
metadata is needed:
```xml
<ItemGroup>
<FileVersionValidationPackage Include="Microsoft.MyPackage" />
</ItemGroup>
```
The defaults bind `FileVersionRevisionProperty=FileVersionRevision`,
`AssemblyVersionOverrideProperty=AssemblyVersionOverride`, and
`InformationalVersionOverrideProperty=InformationalVersionOverride`. Each aspect is validated
only if the named property is defined in the `<component>.props` — items don't need to opt out
of aspects they don't override. If an aspect's property name is overridden per-item, the named
property **must** exist in the `<component>.props` or the tool/test will error.
When a single submodule produces multiple packages with different release versions or revisions
(each from its own release tag), declare one item per package, each with its own per-package
property names so the items don't compete to write the same property:
```xml
<ItemGroup>
<FileVersionValidationPackage Include="MyOrg.PackageA">
<FileVersionRevisionProperty>MyOrgPackageAFileVersionRevision</FileVersionRevisionProperty>
<ReleaseVersionProperty>MyOrgPackageAReleaseVersion</ReleaseVersionProperty>
</FileVersionValidationPackage>
<FileVersionValidationPackage Include="MyOrg.PackageB">
<FileVersionRevisionProperty>MyOrgPackageBFileVersionRevision</FileVersionRevisionProperty>
<ReleaseVersionProperty>MyOrgPackageBReleaseVersion</ReleaseVersionProperty>
</FileVersionValidationPackage>
</ItemGroup>
```
By default the release version is auto-derived from the `.proj` filename
(e.g. `my-component.proj` → `<MyComponentReleaseVersion>` in `eng/Versions.props`).
Setting `ReleaseVersionProperty` overrides that lookup.
After defining the project, run the metadata update script to automatically populate
`SourceRevisionId` and the package-derived properties from the submodule and published package:
```bash
# Linux/macOS
./eng/update-external-metadata.sh <component-name>
# Windows
./eng/update-external-metadata.ps1 <component-name>
```
1. [Build](#building) locally and resolve any build errors.
Source changes must be applied via [patches](src/externalPackages/patches).
See [below](#patches) for more info on patches.
1. Validate the version of the NuGet packages and binaries produced by the build.
See the contents of `./artifacts/packages/<build_configuration>/Shipping`.
1. If the original binaries have strong name signatures, validate the source built ones have them as well.
1. Open a PR.
1. Trigger a full source build within the VMR from your PR by adding a `/azp run source-build-assets-unified-build` comment.
#### Updating an External Component to a Newer Version
1. Update the `./src/externalPackages/src/<external_repo_dir>` to the desired sha
``` bash
cd src/externalPackages/src/<external_repo_dir>
git fetch
git checkout <updated_sha>
cd ..
git add .
git commit -m "<commit_message>"
```
1. [Build](#building) locally
1. Update any [patches](src/externalPackages/patches) as needed.
1. Review the [repo's project](src/externalPackages/projects) to ensure it is appropriate for the new version.
There are a number of projects that utilize MSBuild properties to specify the version.
These need to be manually updated with each upgrade.
1. Run the metadata update script to refresh `SourceRevisionId` and package-derived version metadata (for example `FileVersionRevision`, `AssemblyVersionOverride`, and `InformationalVersionOverride`):
```bash
# Linux/macOS
./eng/update-external-metadata.sh <component-name>
# Windows
./eng/update-external-metadata.ps1 <component-name>
```
1. Resolve build errors.
Source changes must be applied via [patches](src/externalPackages/patches).
See [below](#patches) for more info on patches.
1. Validate the version of the NuGet packages and binaries produced by the build.
See the contents of `./artifacts/packages/<build_configuration>/Shipping`.
1. Open a PR.
1. Trigger a full source build within the VMR from your PR by adding a `/azp run source-build-assets-unified-build` comment.
This will validate the new version will build without adding prebuilts.
It will also ensure the new version does not contain prohibited checked-in binaries.
1. After the PR is merged to update a component, coordination is often needed in the darc dependency flows.
The source-build-assets source may need to flow in at the same time as the cooresponding changes in product repos which take a dependency on the new component version.
Sometimes it can be easier to add the new upgraded version along side the older version and then delete the old version after all product repos have been upgraded to the new version.
#### Building Multiple Versions of an External Component
When consumers require different versions, keep a separate submodule and `.proj` for each source revision.
The `.proj` filename must match its submodule directory so source copying, patches, logs, and build outputs
remain isolated. Each version must have its own `SourceRevisionId`, release version property, and sibling
`.props` validation metadata. Set `ReleaseVersionProperty` on the `FileVersionValidationPackage` item when
the release version property cannot be derived from the component name.
OpenTelemetry builds both 1.16.0 (`opentelemetry-dotnet`) and 1.18.0 (`opentelemetry-dotnet-1.18`).
They share `opentelemetry-dotnet.targets`, but use separate upstream sources and package metadata.
The contrib project continues to depend on `opentelemetry-dotnet` (1.16.0).
Add new submodules in this repository first, then forward-flow them into the VMR.
VMR backflow can update existing tracked submodules, but does not create new submodules.
#### Patches
1. When creating/updating patches, it is desirable to backport the changes whenever feasible as this reduces
the maintenance burden when [updating a component to a newer version](#updating-an-external-component-to-a-newer-version).
1. **Minimize patch surface.** Patches are diffs from upstream and may conflict when updating to a newer version.
Keep the number of changed files and lines as small as possible.
Prefer pushing behavioral changes (e.g. disabling analyzers, NuGet audit, code-style enforcement, version overrides)
into the [project file](src/externalPackages/projects) via `/p:` command-line arguments rather than patching the source.
Many upstream projects already define MSBuild properties for these behaviors—pass the right values from the `.proj` instead of editing the upstream files.
1. **Use MSBuild conditions instead of removing XML.**
When a change *must* be made in the upstream source, prefer adding an MSBuild `Condition` to logically disable or override behavior rather than deleting XML elements.
For example, add `Condition="'$(DotNetBuildSourceOnly)' != 'true'"` to an analyzer `PackageReference` instead of removing the element entirely.
Conditional changes are smaller diffs, less likely to conflict with upstream updates, and can potentially be contributed upstream so the project builds in isolation for other interested parties.
1. **Keep Central Package Management (CPM) enabled.**
If an upstream project uses CPM (`ManagePackageVersionsCentrally`), do not disable it—doing so requires adding explicit `Version` attributes to every `PackageReference` across many files, inflating the patch.
Instead, update the version pins in `Directory.Packages.props` to align with the versions available in this SBRP repo.
Use a `Condition="'$(DotNetBuildSourceOnly)' == 'true'"` block to override versions only during source-build,
keeping the upstream defaults intact for normal development.
1. Steps to create new patches:
1. Make changes in the submodule.
1. Commit changes in the submodule.
1. From the root directory of the submodule, run [extract-patches.sh](src/externalPackages/patches/extract-patches.sh)/[extract-patches.ps1](src/externalPackages/patches/extract-patches.ps1).
The script will prepare a patch based on the base sha of the submodule and the latest committed changes.
The patch will be added to `patches/<component>/*.patch`
1. To apply a patch, or multiple patches, use `git am` while inside the submodule directory.
For example, to apply *all* `humanizer` patches:
```sh
# cd src/externalPackages/src/humanizer
# git am "../../patches/humanizer/*"
```
### Reference
The [generate script](https://github.com/dotnet/source-build-assets/blob/main/generate.sh)
supports generating reference packages.
Run `generate.sh --help` for usage details.
When generating a reference package(s), the tooling will detect and generate all dependent packages.
> **Note:** Reference packages should be for released stable versions. Adding preview/release candidate
packages are for exceptional cases only and require approval from
[dotnet/source-build](https://github.com/orgs/dotnet/teams/source-build).
> **Note:** Reference packages should only be added to this repo if they are required during the product
source build (e.g. a [VMR](https://github.com/dotnet/dotnet) build). Reference packages that are only
required for building a repo level source build should not be added to this repo. In this case, it is
appropriate to add these types of package as allowed prebuilt via the `eng/SourceBuildPrebuiltBaseline.xml`
file. See the [Eliminating pre-builts documentation](https://github.com/dotnet/source-build/blob/main/Documentation/eliminating-pre-builts.md)
for detailed guidance.
> **Important:** For a .NET `N` product release, reference packages in this repository must target only
older .NET versions. Do not add `N.x` reference packages or current-TFM targeting packages to
`release/N.0` or the corresponding development branch. A generated current-major reference package
generally targets `netN.0` and depends on `Microsoft.NETCore.App.Ref` `N.0`. Source-build-assets is
part of the bootstrap layer: it builds before runtime, while runtime depends on source-build-assets.
Depending on the current product targeting pack would invert that dependency, create a build cycle,
and undermine N-1 bootstrapping. Resolve current-version dependencies through the current
source-build dependency graph or, when necessary to break a legitimate bootstrap cycle, use
previously-source-built (PSB) packages instead of current-major reference-package API stubs.
``` bash
./generate.sh --package system.buffers,4.5.1
```
The tooling does not handle all situations and sometimes the generated code will need manual tweeks to get
it to compile. If this occurs when generating a newer version of an existing package, it can be helpful to
regenerate the older version to see what customizations to the generated code were made.
> **Suggestion:** Open an [issue](https://github.com/dotnet/source-build-assets/issues/new) that describes the packages to add and assign it to Copilot to do the work (example [issue](https://github.com/dotnet/source-build-assets/issues/1324) and [resulting PR](https://github.com/dotnet/source-build-assets/pull/1325)).
#### Workflow
1. Generate reference package and its depencencies running the `./generate.sh --package <package>,<version>` script.
1. Inspect any changes to packages that already existed in the repository. There are two reasons why previously
generated packages show changes when being regenerated.
1. The package contains intentional code modifications on top of the generated code.
This may be code fixups because the generate tooling does not support a scenario.
When this occurs, there should be code comments explaining why the code modification was made.
If this is the case, the changes to the existing package should be reverted.
1. The generate tooling has changed since the last time this package was generated.
The new changes should be considered better/correct and should be committed.
1. Run build with the `./build.sh -sb` command. This includes API compatibility validation
that compares the generated package against the official baseline from NuGet.
1. If the build produces **API compatibility errors** (e.g., CP0001, CP0002, CP0008, CP0021):
- Determine whether the difference is a real API gap (fix the generated code) or a
generator limitation (the generator cannot perfectly reproduce certain metadata).
- For generator limitations, ensure there is a tracking issue in
[dotnet/sdk](https://github.com/dotnet/sdk/issues) with the `Area-GenAPI` label.
- Add a `CompatibilitySuppressions.xml` file in the package version directory.
You can auto-generate it by building with `/p:GenerateCompatibilitySuppressionFile=true`.
- Commit the suppression file as part of the package.
1. If the compilation produces numerous compilation issue - run the `./build.sh --projects <path to .csproj file>`
command for each generated reference package separately.
It may be necessary to manually tweak the generated artifacts to address compilation issues.
When this occurs, please ensure there is an [tracking issue](#filing-issues) to address the underlying problem with the generator.
When making changes to the generated artifacts, it is recommended to utilize the following pre-defined constructs if possible.
1. Customizations.props - Automatically imported by the generated project.
Use it for additive changes such as NoWarns or additional source files.
1. Customizations.cs - Automatically included by the generated project.
Use it to add new types or members to partial classes.
You can search the code base to see example usages.
The benefit of using these files is that they will be preserved when the packages are regenerated.
You can also search for [known GenAPI issues](https://github.com/dotnet/sdk/issues?q=is%3Aissue+label%3AArea-GenAPI).
1. Add comments calling out any modifications to the generated code that were necessary.
> **Note:** When porting new packages between branches, you must regenerate the packages when crossing the 10.0/9.0 boundary.
This is because in 10.0 the generated projects switched from using PackageReference to ProjectReference.
Porting new packages across 10.0/9.0 boundary will introduce prebuilts.
See the workflow documented in the servicing branch readmes for additional requirements when adding new packages pre 10.0.
#### Regenerating all Reference Packages
As bugs are fixed or enhancements are made to the generate tooling, it may be desirable or necessary to
regenerate the existing packages. The following command will regenerate all of the reference packages.
``` bash
./generate.sh -a
```
### Targeting
The [generate script](https://github.com/dotnet/source-build-reference-packages/blob/main/generate.sh) supports generating targeting packages.
Run `generate.sh --help` for usage details.
``` bash
./generate.sh --type target --package Microsoft.NetCore.App.Ref,10.0.0
```
> **Note:** Generating targeting packages uses ILDasm to disassemble reference assemblies.
### Text Only
The [generate script](https://github.com/dotnet/source-build-assets/blob/main/generate.sh)
supports generating text-only packages.
Run `generate.sh --help` for usage details.
``` bash
./generate.sh --type text --package microsoft.build.traversal,3.1.6
```
> **Suggestion:** Open an [issue](https://github.com/dotnet/source-build-assets/issues/new) that describes the packages to add and assign it to Copilot to do the work (example [issue](https://github.com/dotnet/source-build-assets/issues/1324) and [resulting PR](https://github.com/dotnet/source-build-assets/pull/1325)).
## Vulnerable Packages
CVEs may exist for reference packages included in this repo. Because the packages do not contain any
implementation, they do not pose a security risk. CG is configured in this repo to ignore the reference
packages. If product repos migrate off these vulnerable packages, they can be [removed](#cleanup).
## Filing Issues
This repo should contain [issues](https://github.com/dotnet/source-build-assets/issues) that are tied to reference, text-only, external, and target packages used by source build.
Other source build related issues should be opened in [dotnet/source-build](https://github.com/dotnet/source-build/issues/new/choose)
## Cleanup
Periodically, packages that are unreferenced by the product source build should be deleted. The number of
unreferenced packages build up over time as the product repositories upgrade their dependencies to newer
versions. Ideally this cleanup would be performed around RC1 timeframe as the product locks down in preparation
for the GA release. To find which packages are unreferenced, you can run a VMR build with the `ReportSbaUsage`
option to generate an SBA package usage report. The resulting report will be written to
`artifacts/log/<configuration>/sbaPackageUsage.json`.
``` bash
./build.sh -sb /p:ReportSbaUsage=true
```
The VMR CI runs with the `ReportSbaUsage` option set therefore you can grab the usage report from any build's
artifacts.
The [cleanup-unreferenced-packages](.github/skills/cleanup-unreferenced-packages/skill.md) AI skill
provides an intelligent cleanup workflow that:
- Automatically detects packages added to SBA that haven't flowed into the VMR yet
- Reads a [config file](.github/skills/cleanup-unreferenced-packages/cleanup-packages-config.json) of known false positives
- Prompts the user to identify packages whose uptake in the VMR is still in progress before deletion
To invoke the skill, ask an AI agent to "clean up unreferenced packages" in this repository.
## License
This repo is licensed with [MIT](LICENSE.txt).