run-integration-tests

name

run-integration-tests

description

Run the New Relic .NET agent integration tests locally. Use whenever you need to run, debug, or reproduce an integration test (host-run IntegrationTests, UnboundedIntegrationTests, or ContainerIntegrationTests), are verifying an agent/wrapper change end to end, or find yourself reaching for `dotnet test` against a test project in this repo. You run these from the CLI yourself -- secrets and infra are already in place. Covers the build-first prerequisite, picking the layer, the run command per layer, and the env gotchas behind confusing failures.

Running integration tests locally

Start a real app with the agent attached, exercise it, wait for a harvest, assert on the parsed agent log. Flow: build agent (if changed) -> build test solution (if changed) -> run via the layer's runner -> read results.

1. Build (solutions only, never individual projects)

a. Agent -- only if agent/wrapper code changed. Tests read built home dirs src/Agent/newrelichome_*, not source; stale home = #1 false failure. Build FullAgent.sln (see build-dotnet-agent). Already current for this branch? Skip.

b. Test solution -- once, only if test/app/agent code changed since the last test-solution build. One build serves all later runs; never rebuild between tests. Building the .sln builds the test project, shared MultiFunctionApplicationHelpers, and every launched app (the test .csproj doesn't reference the apps, but the solution includes all Applications/* + SharedApplications/ConsoleMultiFunctionApplication{FW,Core}) -- so never build an app by hand. App sourcing at test time: web apps copied from a Deploy/ folder; FW console from bin/<Config>/<tfm>; Core console dotnet published from source (always current).

Use full VS MSBuild.exe (not dotnet build -- legacy non-SDK web projects have WebPublish targets the SDK MSBuild can't build), Debug (fixture looks for apps under Debug). These flags mirror CI (all_solutions.yml) and are the only proven-good set: -p:DeployOnBuild=true -p:PublishProfile=LocalDeploy drives the web publish, and no -t:Build (the web csproj's <Target Name="Deploy" AfterTargets="Build"> re-invokes WebPublish; -t:Build -> error MSB4006: circular dependency ... "Deploy"). Slow -- background/tee. Swap the .sln per layer.

MSBUILD=$("/c/Program Files (x86)/Microsoft Visual Studio/Installer/vswhere.exe" \
  -latest -prerelease -products '*' -requires Microsoft.Component.MSBuild \
  -find 'MSBuild/**/Bin/MSBuild.exe')
"$MSBUILD" tests/Agent/IntegrationTests/IntegrationTests.sln \
  -restore -m -p:Configuration=Debug -p:DeployOnBuild=true -p:PublishProfile=LocalDeploy

2. Pick the layer

Layer	Solution / test project	Notes
Host-run (default)	IntegrationTests.sln / IntegrationTests.csproj	Windows + home dirs; real NR staging collectors. FW web-app tests (RemoteWebApplication/HostedWebCore) need an elevated (Administrator) terminal; console (ConsoleDynamicMethodFixture) tests don't.
Unbounded	UnboundedIntegrationTests.sln / .csproj	Real DBs/brokers (MySQL, Postgres, SQL Server, Mongo, Redis, RabbitMQ, Kafka, Elasticsearch, Couchbase...).
Container	ContainerIntegrationTests.sln / .csproj	Docker Desktop; Linux-agent coverage. Runner differs -- section 3.
Performance	PerformanceTests.sln	run-perf-test.py, not dotnet test. Out of scope.

Host-run answers most things. Secrets auto (user secrets on Shared.csproj, already set). Unbounded infra is always up (restart: unless_stopped) -- no docker compose up; check UnboundedServices only on a connection failure.

3. Run -- runner depends on the layer

xUnit v3 projects (OutputType Exe, Microsoft.Testing.Platform).

Host-run / Unbounded: run the built exe <testproj>/bin/Debug/net10.0/NewRelic.Agent.<layer>.exe (as CI/VS do). Not dotnet test -- the VSTest bridge is ~3x slower (measured 7 ASB classes: ~5.7 min exe vs ~17 min dotnet test). Filters (native xUnit v3; cross-type AND; * at start/end): -namespace "<FQN>", -class "*Suffix"/"FQCN", -method "FQN.Method". Dry-run -list classes (/json to parse) before a slow run. One invocation spans everything in one assembly. Default -parallel collections, but effective concurrency ~2 -- fixtures serialize per-app publish/tool-install behind a lock (DotnetTool/RemoteApplication); don't force wider.

Run in a sub-agent (Sonnet; Haiku for one test) returning only: TEST EXECUTION SUMMARY (Total/Failed/Skipped), failing names + asserts, NR-ERROR/NR-FATAL. Never raw read/tail/wide-grep the agent log -- counts, capped width, /tmp distill.

cd tests/Agent/IntegrationTests/UnboundedIntegrationTests/bin/Debug/net10.0
EXE=./NewRelic.Agent.UnboundedIntegrationTests.exe
"$EXE" -namespace "NewRelic.Agent.UnboundedIntegrationTests.AzureServiceBus" -class "*CoreLatest" \
       -trx "C:/IntegrationTestWorkingDirectory/TestResults/asb.trx" > /tmp/asb.log 2>&1

Exe prints === TEST EXECUTION SUMMARY === (Total/Errors/Failed/Skipped/Time); -trx writes a TRX; the agent-log temp path is printed in output.

Container: use dotnet test (runner is noise -- docker build + harvest waits dominate; ~2m16s dotnet test vs ~2m46s exe, same test). Multi-targets net10.0/net11.0 so pass --framework; filter by [Trait] like linux_container_tests.yml. Needs Docker Desktop + Linux home src/Agent/newrelichome_x64_coreclr_linux (libNewRelicProfiler.so); x64 needs no emulation, arm64 needs QEMU.

dotnet test tests/Agent/IntegrationTests/ContainerIntegrationTests/ContainerIntegrationTests.csproj \
  --framework net10.0 --no-build --filter "Architecture=amd64&Distro=Ubuntu"
#   ... --filter "FullyQualifiedName~UbuntuX64ContainerTest"   # one class

4. Logging env -- don't touch

NEW_RELIC_LOG_DIRECTORY/NEW_RELIC_LOG_LEVEL are inherited from the user's env -- don't set/override. Change agent behavior via NewRelicConfigModifier/WebConfigModifier/fixture.SetEnvironmentVariable, never ad-hoc XML.

5. On failure, check in order

1. Elevated terminal? FW web-app tests (RemoteWebApplication/HostedWebCore) need Administrator; else HostedWebCore exits code 2.
2. Builds current? FullAgent.sln fresh (if agent changed) + test solution built Debug with no app erroring out. (A stale bin/Release/<tfm>/<rid>/publish from NR_DOTNET_TEST_PREBUILT_APPS=1 can mask source changes -- delete it.)
3. AzureFunction namespace: Core Tools must be >= 4.13.0. Symptom: Timed out waiting for Azure function host to become ready, and the captured func stdout carries Error configuring services in an external startup class ... Could not load file or assembly 'Microsoft.Extensions.Options, Version=10.0.0.0'. Cause: the isolated AzureFunctionApplication extension bundle resolves Microsoft.Extensions.* 10.0.x (Worker.Extensions.Storage.Queues -> WebJobs.Extensions.Storage.Queues 5.3.8 -> Microsoft.Extensions.Hosting >= 10.0.3) and does not ship those assemblies; the host supplies them. Core Tools retargeted to .NET 10 in 4.13.0 (2026-08-10) -- older hosts are net8.0 and ship 8.0.0.0. Verify with func.runtimeconfig.json ("tfm": "net10.0"); winget lags the GitHub release. Select a specific build with azure_func_exe_path=<...>\func.exe (CI sets it to the choco path). In-proc classes are unaffected: AzureFunctionInProcApplication references the WebJobs extensions directly and builds no bundle.
4. Fixture env vars set? 5. Ports free (localhost)? 6. Agent log: grep NR-ERROR/NR-FATAL, distilled. 7. App stdout/stderr.

Container: Docker up with resources, image built, container stayed up, host<->container reachable.