shiny-datasync

name	shiny-datasync
description	Generate code using Shiny.DataSync, an AOT-compliant offline-first data synchronization engine for .NET MAUI and desktop apps with SQLite local storage and HTTP API sync
auto_invoke	true
triggers	["data sync","datasync","offline sync","offline first","sync service","sync entity","sync pull","sync push","push queue","pull data","push data","sync job","sync background","sync interceptor","ISyncService","ISyncInterceptor","SyncDirection","SyncEvent","SyncMetadata","SyncQueueItem","SyncEntityRegistration","DataSyncConfiguration","AddDataSync","AddEntity","PullUri","PushUri","PullDateVariable","PushBufferSize","VersionSelector","SoftDeletePredicate","ExpiryPredicate","TombstoneUri","[Truncated]"]

Shiny.DataSync — Offline-First Data Synchronization for .NET

You are helping a developer integrate the Shiny.DataSync NuGet package into their .NET MAUI or desktop application. This library provides offline-first data synchronization between a local SQLite store and HTTP APIs.

Quick Facts

• Package: Shiny.DataSync
• Also required: Shiny.DocumentDb.Sqlite, Shiny.DocumentDb.Extensions.DependencyInjection for local storage; Shiny.Hosting.Maui for MAUI apps
• Target: net10.0+
• AOT compliant: Yes — requires System.Text.Json source-generated JsonSerializerContext
• Background sync: Automatically registers a Shiny background job via Shiny.Jobs
• Conflict resolution: Server wins — no merge logic
• Sync order: Sync() runs Clean → Push → Pull

Installation

dotnet add package Shiny.DataSync

Core Concepts

• Entities are plain classes — no base classes, no attributes, no interfaces. Just POCOs with an ID property.
• AOT compliance is required — all entity types must be registered in a System.Text.Json source-generated JsonSerializerContext. The type, its array type, and its List type must all be included.
• Sync direction per entity — SyncDirection.Both (default), PullOnly, or PushOnly.
• Server wins — no merge conflict resolution. Local changes are pushed as-is, pulled data overwrites local.
• SyncJob auto-registered — AddDataSync() automatically registers a Shiny background job. No manual AddJob() needed.

Setup Pattern

Every integration follows these steps:

1. Define entity classes

public class TodoItem
{
    public string Id { get; set; } = Guid.NewGuid().ToString();
    public string Title { get; set; } = "";
    public bool IsComplete { get; set; }
    public bool IsDeleted { get; set; }
    public long Version { get; set; }
    public DateTimeOffset? UpdatedAt { get; set; }
}

2. Create a JsonSerializerContext

[JsonSerializable(typeof(TodoItem))]
[JsonSerializable(typeof(TodoItem[]))]
[JsonSerializable(typeof(List<TodoItem>))]
public partial class AppJsonContext : JsonSerializerContext;

Important: Always include typeof(T), typeof(T[]), and typeof(List<T>) for every entity type. Missing any of these will cause runtime failures.

3. Register DocumentDB (entity storage)

builder.Services.AddDocumentStore(opts =>
{
    opts.DatabaseProvider = new SqliteDatabaseProvider(
        $"Data Source={Path.Combine(FileSystem.AppDataDirectory, "app.db")}"
    );
    opts.MapTypeToTable<TodoItem>();
    // Add MapTypeToTable for every entity type
});

4. Register DataSync

builder.Services.AddDataSync(ds =>
{
    ds.MetadataDatabasePath = Path.Combine(FileSystem.AppDataDirectory, "sync_meta.db");
    ds.MaxPushAttempts = 5; // 0 = disabled (never auto-remove failed items)

    ds.HttpJsonSerializerOptions = new JsonSerializerOptions(JsonSerializerDefaults.Web)
    {
        TypeInfoResolver = AppJsonContext.Default
    };

    ds.AddEntity<TodoItem>(x => x.Id, e =>
    {
        e.Direction = SyncDirection.Both;
        e.PullUri = "/api/todos";
        e.PullDateVariable = "since";
        e.PullMinimumTime = TimeSpan.FromMinutes(5);
        e.PushUri = "/api/todos";
        e.PushBufferSize = 50;
        e.VersionSelector = x => x.Version;
    });
});

5. Configure the HttpClient

The library uses IHttpClientFactory with the client name "DataSync":

builder.Services.AddHttpClient("DataSync", client =>
{
    client.BaseAddress = new Uri("https://your-api.com");
});

6. (Optional) Register ISyncInterceptor for auth

public class MySyncInterceptor : ISyncInterceptor
{
    public Task BeforePull(Type documentType, DateTimeOffset lastRun, HttpRequestMessage request)
    {
        request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", "token");
        return Task.CompletedTask;
    }

    public Task BeforePush(Type documentType, object[] items, HttpRequestMessage request)
    {
        request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", "token");
        return Task.CompletedTask;
    }
}

builder.Services.AddSingleton<ISyncInterceptor, MySyncInterceptor>();

7. For MAUI apps, call UseShiny()

builder.UseMauiApp<App>().UseShiny();

Using ISyncService

Inject ISyncService to perform CRUD and sync operations:

// Insert a new entity (marks for push)
await syncService.Insert(new TodoItem { Title = "Buy milk" });

// Update an entity (marks for push)
todo.IsComplete = true;
await syncService.Update(todo);

// Mark for deletion (marks for push)
await syncService.Remove<TodoItem>(todo.Id);

// Run full sync cycle: Clean -> Push -> Pull
await syncService.Sync();

// Or run phases independently
await syncService.Clean();
await syncService.Push();
await syncService.Pull();

// Type-specific operations
await syncService.Push<TodoItem>();
await syncService.Pull<TodoItem>();

// Get pending push queue
var queue = await syncService.GetQueue();
var todoQueue = await syncService.GetQueue<TodoItem>();

// Clear the push queue (discard pending changes)
await syncService.Clear();
await syncService.Clear<TodoItem>();

// Get sync metadata for a specific entity
var meta = await syncService.GetMetadata<TodoItem>(todoId);

// Subscribe to sync events
syncService.WhenSync().Subscribe(evt =>
{
    // evt.Type: PushStarted, PushCompleted, PushFailed, PullStarted, PullCompleted, PullFailed
    // evt.EntityType: the Type being synced
    // evt.Count: number of entities involved
    // evt.Error: exception if failed
});

Entity Registration Options

All properties on SyncEntityRegistration<T>:

ds.AddEntity<MyEntity>(x => x.Id, e =>
{
    // --- Direction ---
    e.Direction = SyncDirection.Both; // Both | PullOnly | PushOnly

    // --- Pull ---
    e.PullUri = "/api/items";              // required for Pull/Both
    e.PullHttpMethod = HttpMethod.Get;     // default: GET
    e.PullDateVariable = "since";          // appends ?since=<timestamp> to PullUri
    e.PullMinimumTime = TimeSpan.FromMinutes(5); // throttle pulls

    // --- Push ---
    e.PushUri = "/api/items";              // required for Push/Both
    e.PushHttpMethod = HttpMethod.Post;    // default: POST
    e.PushBufferSize = 50;                 // max entities per batch (default: 50)

    // --- Delete (optional separate endpoint) ---
    e.DeleteUri = "/api/items/delete";     // if unset, deletes go to PushUri
    e.DeleteHttpMethod = HttpMethod.Post;  // default: POST

    // --- Version tracking ---
    e.VersionSelector = x => x.Version;   // extract server version from entity

    // --- Removal Strategies (all optional, any combination) ---

    // 1. Soft-delete: remove locally if server flags entity as deleted
    e.SoftDeletePredicate = x => x.IsDeleted;

    // 2. Tombstone: fetch deleted IDs from a separate endpoint
    e.TombstoneUri = "/api/items/tombstones";
    e.TombstoneHttpMethod = HttpMethod.Get;       // default: GET
    e.TombstoneDateVariable = "since";            // appends ?since=<timestamp>

    // 3. Reconciliation: fetch all valid IDs, remove anything not in the set
    e.ReconciliationUri = "/api/items/ids";
    e.ReconciliationHttpMethod = HttpMethod.Get;   // default: GET
    e.ReconciliationMinimumTime = TimeSpan.FromMinutes(30); // throttle (expensive)

    // 4. Expiry: remove locally if server-driven state change detected
    e.ExpiryPredicate = x => x.AssignedTo == null;
});

DataSyncConfiguration Options

builder.Services.AddDataSync(ds =>
{
    ds.MetadataDatabasePath = "sync_meta.db";  // SQLite file for internal metadata
    ds.MaxPushAttempts = 5;                    // Clean drops items after N failures (0 = disabled)
    ds.RegisterJob = true;                     // auto-register SyncJob (default: true)
    ds.HttpJsonSerializerOptions = ...;        // must include source-gen context

    ds.AddEntity<T>(...);
});

Heterogeneous Push Queue Pattern

For a general-purpose "sync up" queue that can push multiple unrelated types to one endpoint:

public class SyncUpEntry
{
    public string Id { get; set; } = Guid.NewGuid().ToString();
    public string EntryType { get; set; } = "";   // "Feedback", "CrashReport", etc.
    public JsonElement Payload { get; set; }        // serialized data
    public DateTimeOffset CreatedAt { get; set; } = DateTimeOffset.UtcNow;
}

Register as push-only, create entries by serializing into JsonElement:

ds.AddEntity<SyncUpEntry>(x => x.Id, e =>
{
    e.Direction = SyncDirection.PushOnly;
    e.PushUri = "/api/syncup";
    e.PushBufferSize = 100;
});

// Usage:
var entry = new SyncUpEntry
{
    EntryType = "Feedback",
    Payload = JsonSerializer.SerializeToElement(obj, AppJsonContext.Default.Feedback)
};
await syncService.Insert(entry);

Data Removal Strategies

Four strategies for removing data that the client should no longer have. All are optional and composable — use any combination on the same entity.

Strategy	When to use	Server provides
Soft-Delete	Server marks entities as deleted but still returns them in pull	A boolean flag on the entity (e.g. IsDeleted)
Tombstone	Server tracks deleted IDs separately	string[] of deleted IDs at a dedicated endpoint
Reconciliation	Need to catch all cases (unassigned work, bulk deletes)	string[] of all valid IDs at a dedicated endpoint
Expiry	Server changes entity state to mean "not for this client anymore"	A state change on the entity (e.g. AssignedTo becomes null)

Reconciliation skips entities with pending local changes to avoid removing in-progress work.

Server API Contracts

The library expects these HTTP endpoint shapes:

Endpoint	Method	Request Body	Response Body
Pull	GET	-	T[] (entity array)
Push	POST	T[] (entity array)	2xx status
Delete	POST	T[] (entities to delete)	2xx status
Tombstone	GET	-	string[] (deleted entity IDs)
Reconciliation	GET	-	string[] (all valid entity IDs)

• Pull and Tombstone endpoints support an optional ?since=<ISO8601> query parameter when PullDateVariable / TombstoneDateVariable is configured.
• Push and Delete endpoints receive the full entity objects as a JSON array, not just IDs.

Common Mistakes

1. Missing array type in JsonSerializerContext — Must include [JsonSerializable(typeof(T[]))] and [JsonSerializable(typeof(List<T>))] for every entity type, not just typeof(T).
2. Forgetting MapTypeToTable<T>() — Every entity must be registered in both DocumentDB and DataSync.
3. Wrong HttpClient name — The library looks for a client named "DataSync" via IHttpClientFactory.
4. Not calling UseShiny() — Required in MAUI apps for Shiny Jobs background scheduling to work.
5. Manual AddJob() for SyncJob — Not needed. AddDataSync() registers it automatically. Set RegisterJob = false only if you need custom job parameters.

Best Practices

1. Always set MetadataDatabasePath to a path under FileSystem.AppDataDirectory in MAUI apps.
2. Use PullDateVariable on pull endpoints for incremental sync — avoids pulling the entire dataset each time.
3. Set MaxPushAttempts > 0 to prevent poison queue items from blocking sync indefinitely.
4. Use PullMinimumTime to throttle frequent pull operations.
5. Subscribe to WhenSync() for UI feedback during sync operations.
6. Use ISyncInterceptor for authentication headers rather than configuring the HttpClient directly — the interceptor is called per-request and can refresh tokens.
7. Register entities as PullOnly for reference data that the client never modifies.
8. Register entities as PushOnly for telemetry, feedback, or audit log entries.
9. Use the heterogeneous SyncUpEntry pattern for mixed-type push-only queues.
10. Combine removal strategies: soft-delete for real-time detection + reconciliation as a safety net.

See reference/api-reference.md for the complete API surface.