accordant-test-generation

name	accordant-test-generation
description	How Accordant generates and runs tests - use this skill when configuring test generation, understanding state graphs, or running tests

Test Generation in Accordant

Accordant generates tests by exploring the state graph defined by your spec, then running the generated sequences against your real system.

The State Graph

Starting from an initial state, Accordant applies each input to compute resulting states, then recurses. This builds a graph where:

• Nodes = States the system can be in
• Edges = Operations that transition between states

Generating Tests

Basic Usage

var spec = CreateSpec();
var initialState = new BankState();

// Define inputs to explore
var inputs = new InputSet
{
    spec.GetOperation<string, ApiResult<decimal>>("CreateAccount").With("alice", "Create alice"),
    spec.GetOperation<(string, decimal), ApiResult<decimal>>("Deposit").With(("alice", 100m), "Deposit 100"),
    spec.GetOperation<(string, decimal), ApiResult<decimal>>("Withdraw").With(("alice", 30m), "Withdraw 30"),
    spec.GetOperation<(string, decimal), ApiResult<decimal>>("Withdraw").With(("alice", 200m), "Withdraw 200 (overdraft)"),
    spec.GetOperation<string, ApiResult<decimal>>("DeleteAccount").With("alice", "Delete alice"),
};

// Generate test cases
var testCases = spec.GenerateTests(initialState, inputs, new TestGenerationOptions
{
    MaxDepth = 5
});

TestGenerationOptions

Option	Description	Default
MaxDepth	Maximum sequence length	5
StateConstraint	Prune states during exploration	None
MaxConcurrencyLevel	For concurrent test generation	3
SequentialTestCaseAlgorithm	How paths are extracted	StateCoverage

Constraining the State Space

Prevent state explosion with StateConstraint:

var options = new TestGenerationOptions
{
    MaxDepth = 5,
    StateConstraint = state =>
    {
        var s = (BankState)state;
        // Limit: max 2 accounts, balances under 500
        return s.Accounts.Count <= 2 && 
               s.Accounts.Values.All(b => b <= 500);
    }
};

Test Case Algorithms

StateCoverage (Default)

Visits each unique state at least once:

var options = new TestGenerationOptions
{
    SequentialTestCaseAlgorithm = SequentialTestCaseAlgorithms.StateCoverage
};

TransitionCoverage

Exercises every edge (transition) in the graph — can generate many tests:

var options = new TestGenerationOptions
{
    SequentialTestCaseAlgorithm = SequentialTestCaseAlgorithms.CreateTransitionCoverage(maxSequenceLength: 4)
};

RandomWalk

Samples paths probabilistically — good for large state spaces:

var options = new TestGenerationOptions
{
    SequentialTestCaseAlgorithm = SequentialTestCaseAlgorithms.CreateRandomWalk(
        numberOfWalks: 200,
        maxWalkLength: 8,
        seed: 42)  // For reproducibility
};

Running Tests

Basic Execution

var context = spec.CreateTestingContext();

// Register your test client
var client = new BankApiClient(httpClient);
context.Register(client);

var results = await spec.RunTests(context, initialState, testCases, new TestExecutionOptions
{
    BeforeEachAsync = async _ =>
    {
        // Reset system before each test
        await client.DeleteAccount("alice");
        await client.DeleteAccount("bob");
    }
});

// Check results
var failures = results.Where(r => !r.Success).ToList();
Assert.IsEmpty(failures, $"Failed: {failures.FirstOrDefault()?.LastFailureMessage}");

TestExecutionOptions

Option	Description
BeforeEachAsync	Reset system before each test case
AfterEachAsync	Cleanup after each test case
OnStepExecuted	Log each operation's request/response for debugging

Resetting State

Each test expects the system to start in the initial state. Reset with BeforeEachAsync:

var knownAccounts = new[] { "alice", "bob" };

var results = await spec.RunTests(context, initialState, testCases, new TestExecutionOptions
{
    BeforeEachAsync = async _ =>
    {
        foreach (var accountId in knownAccounts)
        {
            await client.DeleteAccount(accountId);  // Ignore 404
        }
    }
});

Visualizing the State Graph

Generate a GraphViz visualization:

var dot = spec.VisualizeStateSpace(initialState, inputs, new TestGenerationOptions { MaxDepth = 4 });
File.WriteAllText("state-graph.dot", dot);

// Convert to PNG: dot -Tpng state-graph.dot -o state-graph.png

Custom Node Labels

var dot = spec.VisualizeStateSpace(
    initialState,
    inputs,
    new TestGenerationOptions { MaxDepth = 3 },
    new VisualizationOptions
    {
        NodeLabelLambda = node =>
        {
            var state = (BankState)node.State;
            return $"Accounts: {state.Accounts.Count}\\nTotal: {state.Accounts.Values.Sum()}";
        }
    });

Manual Conformance Testing

Validate individual operations without auto-generation:

var stateProfile = new StateProfile(new BankState());

// Execute and validate
var response = await client.CreateAccount("alice");
(bool isValid, string message, stateProfile) = 
    spec.Allows(spec.GetOperation("CreateAccount"), "alice", response, stateProfile);

Assert.IsTrue(isValid, message);

// Continue with more operations...
var withdrawResponse = await client.Withdraw("alice", 50);
(isValid, message, stateProfile) = 
    spec.Allows(spec.GetOperation("Withdraw"), ("alice", 50m), withdrawResponse, stateProfile);

Assert.IsTrue(isValid, message);

Complete Example

[Test]
public async Task AutoGeneratedTests()
{
    var spec = CreateSpec();
    
    // Bind to real API
    spec.ExecuteWith<BankApiClient>()
        .Bind<string, ApiResult<decimal>>("CreateAccount", (c, id) => c.CreateAccount(id).Result)
        .Bind<(string, decimal), ApiResult<decimal>>("Deposit", (c, r) => c.Deposit(r.Item1, r.Item2).Result)
        .Bind<(string, decimal), ApiResult<decimal>>("Withdraw", (c, r) => c.Withdraw(r.Item1, r.Item2).Result)
        .Bind<string, ApiResult<decimal>>("DeleteAccount", (c, id) => c.DeleteAccount(id).Result);

    var inputs = new InputSet
    {
        spec.GetOperation<string, ApiResult<decimal>>("CreateAccount").With("alice", "Create alice"),
        spec.GetOperation<(string, decimal), ApiResult<decimal>>("Deposit").With(("alice", 100m), "Deposit 100"),
        spec.GetOperation<(string, decimal), ApiResult<decimal>>("Withdraw").With(("alice", 30m), "Withdraw 30"),
    };

    var testCases = spec.GenerateTests(new BankState(), inputs, new TestGenerationOptions { MaxDepth =  });

     context = spec.CreateTestingContext();
    context.Register( BankApiClient(CreateHttpClient()));

     results =  spec.RunTests(context,  BankState(), testCases,  TestExecutionOptions
    {
        BeforeEachAsync =  _ =>  ResetDatabase()
    });

    Assert.That(results.All(r => r.Success), );
}

Common Issues

Too Many Test Cases

• Reduce MaxDepth
• Add StateConstraint to prune uninteresting states
• Use fewer inputs or more targeted inputs

Tests Failing Unpredictably

• Ensure BeforeEachAsync properly resets state
• Check for timing issues (async operations may not complete)
• Verify inputs match what the spec expects

Next Steps

• Concurrency Testing: Test race conditions with GenerateConcurrentTests and RunTests
• Async Operations: Handle background work with polling