azure-functions

# Azure Functions Dev/Test Workflow This skill helps tracer engineers test changes to the `Datadog.AzureFunctions` package: build a local dev version, deploy it to a test Azure Function App, trigger the function, and analyze traces/logs to verify instrumentation behavior. ## Prerequisites This skill requires the following tools (assume they are installed and only troubleshoot if errors occur): - **PowerShell**: `pwsh` (PowerShell 7+) preferred, or `powershell.exe` (PowerShell 5.1+ on Windows) - Always prefer `pwsh` over `powershell.exe` when available - Scripts use PowerShell-specific cmdlets like `Expand-Archive` - **Azure CLI**: `az` (must be authenticated) - **Azure Functions Core Tools**: `func` - **.NET SDK**: Matching target framework of sample app **Only if a tool fails, provide installation links**: - **PowerShell**: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell - **Azure CLI**: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli - **Azure Functions Core Tools**: https://learn.microsoft.com/en-us/azure/azure-functions/functions-run-local - **.NET SDK**: https://dotnet.microsoft.com/download ## Commands When invoked with an argument, perform the corresponding workflow: - `/azure-functions build-nuget` - Build the Datadog.AzureFunctions NuGet package - `/azure-functions deploy [app-name]` - Deploy to Azure Function App - `/azure-functions configure [app-name]` - Configure environment variables for Datadog instrumentation - `/azure-functions test [app-name]` - Trigger and verify function execution - `/azure-functions logs [app-name]` - Download and analyze logs If no argument is provided, guide the user through the full workflow interactively. ## Context **Current repository**: This skill assumes you are working from the root of the `dd-trace-dotnet` repository. **Prerequisites**: Users provide their own Azure Function App name (`-AppName`), resource group (`-ResourceGroup`), and sample app path (`-SampleAppPath`). The sample app must: 1. Reference the `Datadog.AzureFunctions` NuGet package 2. Have a `nuget.config` file (in the app directory or a parent directory) that defines a local NuGet feed pointing to a directory on disk, for example: ```xml <?xml version="1.0" encoding="utf-8"?> <configuration> <packageSources> <clear /> <add key="local" value="nuget/local-source" /> <add key="nuget" value="https://api.nuget.org/v3/index.json" /> </packageSources> </configuration> ``` The `-CopyTo` parameter of `Build-AzureFunctionsNuget.ps1` must point to the same directory as the local feed (e.g. if the feed `value` is `nuget/local-source` relative to the `nuget.config` location, then `-CopyTo` should be the absolute path to that directory). ## Workflow Steps ### 1. Build NuGet Package **CRITICAL**: Before building, verify that `tracer/src/Datadog.AzureFunctions/Datadog.AzureFunctions.csproj` uses **PackageReference** (not ProjectReference) for `Datadog.Trace` and `Datadog.Trace.Annotations`. This ensures the locally-built package references the latest releases from nuget.org instead of building them from source. **Check the .csproj** — it should contain: ```xml <PackageReference Include="Datadog.Trace" Version="*"/> <PackageReference Include="Datadog.Trace.Annotations" Version="*" /> ``` If instead it contains **ProjectReference** lines like these, replace them with the PackageReference lines above: ```xml <!-- These are the production references — replace for local testing: --> <ProjectReference Include="$(MSBuildThisFileDirectory)..\Datadog.Trace.Manual\Datadog.Trace.Manual.csproj" /> <ProjectReference Include="$(MSBuildThisFileDirectory)..\Datadog.Trace.Annotations\Datadog.Trace.Annotations.csproj" /> ``` **IMPORTANT**: The PackageReference change is for local testing only. Do NOT commit it. If it's already using PackageReference, no change is needed. Build the `Datadog.AzureFunctions` NuGet package with your changes: ```powershell ./tracer/tools/Build-AzureFunctionsNuget.ps1 -CopyTo <output-dir> ``` **What this does**: 1. (If `-BuildId` specified and files not already downloaded) Downloads bundle from Azure DevOps build once 2. Generates a unique prerelease version from a timestamp (e.g. `3.38.0-dev20260209143022`) 3. Cleans previous builds 4. Builds `Datadog.Trace` (net6.0) 5. Publishes to bundle folder 6. Packages `Datadog.AzureFunctions.nupkg` with the generated version (referencing latest nuget.org releases) 7. Copies to the directory specified by `-CopyTo` **Versioning**: Each build gets a unique version, so NuGet caching is never an issue. The sample app should use a floating version like `3.38.0-dev.*` in its package reference (or `Directory.Packages.props`) to always resolve the latest local dev build. **Options**: - `-CopyTo <output-dir>` - Copy the built package to the specified directory (typically your local NuGet feed) - `-Version '3.38.0-dev.custom'` - Use a specific version instead of auto-generating - `-BuildId 12345` - One-time download of bundle files from Azure DevOps build (only needed once per dd-trace-dotnet release, then reused for subsequent local builds) **Examples**: ```powershell # Typical local build (after bundle files already downloaded) ./tracer/tools/Build-AzureFunctionsNuget.ps1 -CopyTo <output-dir> # First build after new dd-trace-dotnet release (download bundle files once) ./tracer/tools/Build-AzureFunctionsNuget.ps1 -BuildId 12345 -CopyTo <output-dir> ``` ### 2. Deploy Function **IMPORTANT**: Before deploying, verify prerequisites: **Verify nuget.config exists**: ```powershell $nugetConfig = ./.claude/skills/azure-functions/Find-NuGetConfig.ps1 -StartPath "<path-to-sample-app>" if (-not $nugetConfig) { Write-Error "nuget.config not found in sample app directory or parent directories" exit 1 } Write-Host "Found nuget.config at: $nugetConfig" ``` **Verify environment variables are configured** (skip if already done on a previous deploy): ```powershell $envCheck = ./.claude/skills/azure-functions/Test-EnvVars.ps1 -AppName "<app-name>" -ResourceGroup "<resource-group>" if (-not $envCheck.AllRequiredPresent) { Write-Warning "Required environment variables are missing. Run '/azure-functions configure' first, or proceed if you plan to configure after deploying." } ``` Use the `Deploy-AzureFunction.ps1` script to automate deployment, wait, and trigger: ```powershell ./tracer/tools/Deploy-AzureFunction.ps1 ` -AppName "<app-name>" ` -ResourceGroup "<resource-group>" ` -SampleAppPath "<path-to-sample-app>" ``` **What this does**: 1. Runs `dotnet restore` in the sample app directory 2. Publishes to Azure with `func azure functionapp publish` 3. Waits 60 seconds (default) for worker process to restart 4. Triggers the HTTP endpoint and captures execution timestamp 5. Outputs a result object for pipeline usage **Options**: - `-SkipBuild` - Skip `dotnet restore` - `-SkipWait` - Skip 2-minute wait (not recommended) - `-WaitSeconds 60` - Custom wait duration - `-SkipTrigger` - Skip HTTP trigger - `-TriggerUrl "https://..."` - Custom trigger URL **Note**: `-AppName` and `-ResourceGroup` are required parameters. **Pipeline usage** (save output for log analysis): ```powershell $deploy = ./tracer/tools/Deploy-AzureFunction.ps1 ` -AppName "<app-name>" ` -ResourceGroup "<resource-group>" ` -SampleAppPath "<path-to-sample-app>" ``` ### 3. Configure Environment Variables Configure Datadog instrumentation environment variables for an Azure Function App using `Set-EnvVars.ps1`: ```powershell ./.claude/skills/azure-functions/Set-EnvVars.ps1 ` -AppName "<app-name>" ` -ResourceGroup "<resource-group>" ` -ApiKey "<api-key>" ` -Tier recommended ` -Env "dev-lucas" ``` **What this does**: 1. Detects the OS/platform (Linux or Windows) 2. Builds the correct set of variables for the chosen tier 3. Sets platform-specific profiler paths automatically 4. Applies all settings via Azure CLI 5. Restarts the function app (unless `-SkipRestart`) **Tiers**: - **required** - Minimum variables for instrumentation (CORECLR_*, DD_API_KEY, DD_DOTNET_TRACER_HOME, DOTNET_STARTUP_HOOKS) - **recommended** - required + feature disables (AppSec, CI Visibility, RCM, Agent Feature Polling, Process) - **debug** - recommended + debug logging (DD_TRACE_DEBUG, DD_LOG_LEVEL, DD_TRACE_LOG_SINKS, direct log submission) **Options**: - `-Tier required|recommended|debug` - Configuration tier (default: required) - `-Env "dev-lucas"` - Set DD_ENV - `-Service "my-service"` - Set DD_SERVICE - `-Version "1.0.0"` - Set DD_VERSION - `-SamplingRules '<json>'` - Set DD_TRACE_SAMPLING_RULES - `-ExtraSettings @{"KEY"="value"}` - Set additional variables - `-SkipRestart` - Don't restart the app after applying - `-WhatIf` - Preview changes without applying **Preview before applying**: ```powershell ./.claude/skills/azure-functions/Set-EnvVars.ps1 ` -AppName "<app-name>" ` -ResourceGroup "<resource-group>" ` -ApiKey "<api-key>" ` -Tier debug ` -WhatIf ``` **Complete reference**: See [environment-variables.md](environment-variables.md) for all available variables. ### 4. Test Function Trigger an already-deployed function and capture the execution timestamp. Useful for re-testing after a deploy, or testing an app that was deployed earlier. **Discover available triggers**: ```bash # List HTTP-triggered functions and their URLs func azure functionapp list-functions <app-name> --show-keys ``` Or via Azure CLI: ```bash az functionapp function list --name <app-name> --resource-group <resource-group> --query "[].{name:name, href:invokeUrlTemplate}" -o table ``` **Trigger and capture timestamp**: ```powershell $timestamp = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd HH:mm:ss") $response = Invoke-WebRequest -Uri "<trigger-url>" -UseBasicParsing Write-Host "HTTP Status: $($response.StatusCode)" Write-Host "Execution timestamp (UTC): $timestamp" ``` Save the timestamp — you'll need it for log filtering in the next step. **Note**: The Deploy script (step 2) already triggers and captures a timestamp. Use this step when you want to re-test without redeploying. ### 5. Download and Analyze Logs Use the `Get-AzureFunctionLogs.ps1` script to download, extract, and analyze logs: ```powershell ./tracer/tools/Get-AzureFunctionLogs.ps1 ` -AppName "<app-name>" ` -ResourceGroup "<resource-group>" ` -OutputPath $env:TEMP ` -ExecutionTimestamp "<YYYY-MM-DD HH:MM:SS>" ` -All ``` **IMPORTANT**: Always specify `-OutputPath $env:TEMP` to save logs to the system temp folder instead of cluttering the repository directory. **What this does**: 1. Downloads logs from Azure to a timestamped zip file 2. Extracts the archive 3. Identifies host and worker log files 4. Analyzes tracer version, span count, and trace parenting **Analysis options**: - `-ShowVersion` - Display Datadog tracer version from worker logs - `-ShowSpans` - Count spans at execution timestamp (split by host/worker) - `-CheckParenting` - Validate trace parenting (detect root span duplication) - `-All` - Enable all analysis (recommended) **Pipeline usage** (with Deploy script): ```powershell $deploy = ./tracer/tools/Deploy-AzureFunction.ps1 ` -AppName "<app-name>" ` -ResourceGroup "<resource-group>" ` -SampleAppPath "<path-to-sample-app>" ./tracer/tools/Get-AzureFunctionLogs.ps1 ` -AppName $deploy.AppName ` -ResourceGroup "<resource-group>" ` -OutputPath $env:TEMP ` -ExecutionTimestamp $deploy.ExecutionTimestamp ` -All ``` **Log file patterns**: - **Host process**: `dotnet-tracer-managed-Microsoft.Azure.WebJobs.Script.WebHost-{pid}.log` - **Worker process**: `dotnet-tracer-managed-dotnet-{pid}.log` **CRITICAL**: The script automatically filters logs by execution timestamp. Raw log files are append-only and contain entries from multiple deployments/restarts. **Parenting analysis**: - **Span fields**: `s_id` (span ID), `p_id` (parent ID), `t_id` (trace ID) - **Healthy trace**: Worker spans have same `t_id` as host, `p_id` matching host span IDs - **Broken trace**: Worker spans have different `t_id` or `p_id: null` (orphaned root) ## Verification Checklist After deployment and testing: - [ ] Function responds successfully (HTTP 200) - [ ] Worker loaded correct tracer version (check "Assembly metadata") - [ ] Host and worker spans share same trace ID - [ ] Span parent-child relationships are correct (check `p_id` → `s_id` links) - [ ] Process tags are correct (`aas.function.process:host` or `worker`) - [ ] No error logs at execution timestamp - [ ] AsyncLocal context flows correctly (if debugging context issues) ## Common Troubleshooting **General guidance**: For environment variable configuration issues, see [environment-variables.md](environment-variables.md) for complete reference on required, recommended, and debugging variables. ### Function Not Responding **First, check if the app is running**: ```powershell $envCheck = ./.claude/skills/azure-functions/Test-EnvVars.ps1 -AppName "<app-name>" -ResourceGroup "<resource-group>" if ($envCheck.State -ne "Running") {