sensor-ingest

name

sensor-ingest

description

Configure and troubleshoot sensor data ingest into weather-station-dashboard via MQTT or HTTP POST. Use when asked to set up MQTT broker connection, debug why readings are not appearing, configure a new sensor sending data over HTTP, check MQTT topic configuration, test sensor connectivity, or understand the ingest pipeline. Triggers include "MQTT not receiving", "sensor not sending", "configure MQTT", "HTTP POST reading", "ingest setup", "topic configuration", "reading not showing up", or any task involving getting sensor data into the system.

sensor-ingest

Configure, test, and troubleshoot MQTT and HTTP POST ingest pipelines for weather-station-dashboard.

When to Use

• Setting up a new sensor to send data via MQTT
• Configuring HTTP POST ingest from a Raspberry Pi or ESP32
• Debugging why a station is showing offline despite sending data
• Understanding which MQTT topics are accepted and which are rejected
• Testing the ingest pipeline end-to-end before deploying hardware

Ingest Methods

HTTP POST (always available)

Send readings to POST /api/readings. No broker required.

curl -X POST http://localhost:3005/api/readings \
  -H "Content-Type: application/json" \
  -d '{
    "station_id": "abc-123-uuid",
    "temperature_c": 22.4,
    "humidity_pct": 68.0,
    "pressure_hpa": 1013.2,
    "rainfall_mm": 0.0,
    "wind_speed_ms": 3.1,
    "wind_dir_deg": 225.0,
    "uv_index": 4,
    "soil_temp_c": 18.2,
    "soil_moisture_pct": 52.0
  }'

All sensor fields are optional except station_id. Send only the sensors your hardware supports.

MQTT Ingest (optional, set MQTT_ENABLED=1)

Start the server with MQTT_ENABLED=1 and set MQTT_BROKER_URL.

MQTT_ENABLED=1 \
MQTT_BROKER_URL=mqtt://192.168.1.100:1883 \
MQTT_TOPIC_PREFIX=farm/weather \
pnpm start

The server subscribes to farm/weather/# and matches incoming messages to stations by their configured mqtt_topic field.

MQTT Topic Routing

Each station has an optional mqtt_topic field (set in Settings or via API). This is the full topic path the station publishes to.

Example station configuration:

# Update a station's MQTT topic
curl -X PUT http://localhost:3005/api/stations/{station_id} \
  -H "Content-Type: application/json" \
  -d '{ "mqtt_topic": "farm/weather/north-field" }'

When a message arrives on farm/weather/north-field, it is matched to the station with that mqtt_topic value and ingested. Messages on topics not matching any station are rejected and logged.

MQTT Payload Format

Publish JSON to the station's topic. All numeric fields are optional - include only what your sensor reports:

{
  "temperature_c": 22.4,
  "humidity_pct": 68.0,
  "pressure_hpa": 1013.2,
  "rainfall_mm": 0.0,
  "wind_speed_ms": 3.1,
  "wind_dir_deg": 225.0,
  "uv_index": 4,
  "soil_temp_c": 18.2,
  "soil_moisture_pct": 52.0
}

Do not include station_id in the MQTT payload - it is inferred from the topic.

Testing MQTT Ingest

Use Mosquitto clients or any MQTT client:

# Publish a test reading
mosquitto_pub \
  -h 192.168.1.100 \
  -t "farm/weather/north-field" \
  -m '{"temperature_c":22.4,"humidity_pct":68,"pressure_hpa":1013.2}'

# Subscribe to verify the broker is receiving
mosquitto_sub -h 192.168.1.100 -t "farm/weather/#" -v

Then check if the reading appeared:

curl "http://localhost:3005/api/readings?station_id={id}&limit=5"

Ingest Pipeline Steps

When a reading arrives (HTTP or MQTT):

1. Zod validation - reject if payload is malformed
2. Station lookup - reject if station_id not found or inactive
3. MQTT topic check - reject if topic does not match any station's mqtt_topic
4. INSERT into readings table with generated UUID and current timestamp
5. UPDATE station.last_seen to current time
6. Evaluate all enabled alert thresholds for this station
7. If threshold crossed: INSERT alert_event, broadcast WSMessage { type: 'alert' }
8. Broadcast WSMessage { type: 'reading' } to subscribed WebSocket clients
9. Return 201 Created (HTTP) or log success (MQTT)

Troubleshooting

Readings not appearing via HTTP POST

Check the response body for validation errors. Common issues:

• station_id is missing or not a valid UUID
• Station with that ID does not exist - create it first via POST /api/stations
• Station has active: 0 - re-enable via PUT /api/stations/{id}

# Verify station exists
curl http://localhost:3005/api/stations/{station_id}

MQTT messages not being ingested

1. Verify MQTT_ENABLED=1 in server environment
2. Check server logs for "MQTT connected" message on startup
3. Confirm the station's mqtt_topic matches the topic you are publishing to exactly
4. Check server logs for "REJECTED" entries - topic mismatch is the most common cause
5. Verify the JSON payload is valid: echo '{"temperature_c":22}' | python3 -m json.tool

Station stays offline after sending readings

The station's last_seen field is updated on each successful ingest. If it is not updating:

• The reading may be failing validation silently - check server logs
• The station_id in HTTP POST or the mqtt_topic mapping may be wrong
• Confirm with GET /api/readings?station_id={id}&limit=1 whether any readings exist

MQTT reconnect behavior

The server uses the mqtt package with automatic reconnect (exponential backoff, max 30s). On reconnect it resubscribes to MQTT_TOPIC_PREFIX/#. No manual intervention is needed. Check MQTT_BROKER_URL is reachable from the server container/host.

Data retention pruning

Raw readings older than DATA_RETENTION_DAYS (default 365) are deleted nightly at 00:05 by the same cron job that computes daily summaries. Daily summaries are not pruned. To change retention:

curl -X PUT http://localhost:3005/api/settings \
  -H "Content-Type: application/json" \
  -d '{ "data_retention_days": 180 }'