discover-models

name	discover-models
description	This skill should be used when the user asks to discover models, list models, find what models are available, check model pricing, or see which models are actually used in their Langfuse project. It lists both the registered model definitions (with pricing) and the models observed in actual trace data via the Langfuse REST API with a direct-database fallback.

Discover Models

List registered Langfuse model definitions (including pricing) and cross-reference them with models observed in trace data. Return a unified view showing available models, pricing, and which models are actively generating observations.

---

Prerequisites

Before executing any requests, ensure the following credentials are available (ask the user if missing):

Variable	Example	Purpose
HOST	http://localhost:3000	Langfuse base URL
PUBLIC_KEY	pk-lf-...	API Basic Auth username
SECRET_KEY	sk-lf-...	API Basic Auth password
PROJECT_ID	clxxxxxxxxxxxxxxxxxxxxxxxxx	Multi-tenancy project ID
DB_CONN	postgresql://user:pass@host:5432/db	Direct DB fallback (optional)

Validate credentials before proceeding:

curl -s -o /dev/null -w "%{http_code}" \
  -u "$PUBLIC_KEY:$SECRET_KEY" \
  "$HOST/api/public/models?limit=1"

A 200 confirms valid authentication. A 401 or 403 means the keys are incorrect -- report the error and ask the user to verify the public and secret keys.

---

Step 1 -- Fetch Registered Model Definitions via REST API

Call the Langfuse public models endpoint to retrieve all model definitions with their pricing configuration. Paginate until all models are collected.

PAGE=1
while true; do
  RESPONSE=$(curl -s -u "$PUBLIC_KEY:$SECRET_KEY" \
    "$HOST/api/public/models?limit=100&page=$PAGE")

  COUNT=$(echo "$RESPONSE" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('data',[])))")

  if [ "$COUNT" -eq 0 ]; then
    break
  fi

  echo "$RESPONSE" >> /tmp/langfuse_models_raw.json
  PAGE=$((PAGE + 1))
done

For each model definition, the relevant fields are:

• modelName -- the canonical model name (e.g., "gpt-4o", "model-name-20250514")
• unit -- pricing unit, typically "TOKENS" or "CHARACTERS"
• inputPrice -- cost per unit for input/prompt tokens (decimal, e.g., 0.000003)
• outputPrice -- cost per unit for output/completion tokens (decimal, e.g., 0.000015)
• totalPrice -- optional flat cost per request (some models use this instead of per-token pricing)
• matchPattern -- regex pattern used to match observation model strings to this definition

Parse the collected model definitions:

python3 -c "
import json

models = []
with open('/tmp/langfuse_models_raw.json') as f:
    for line in f:
        try:
            page = json.loads(line)
            models.extend(page.get('data', []))
        except json.JSONDecodeError:
            continue

print('=== REGISTERED MODELS ===')
for m in sorted(models, key=lambda x: x.get('modelName', '')):
    name = m.get('modelName', '(unknown)')
    unit = m.get('unit', 'N/A')
    inp = m.get('inputPrice')
    out = m.get('outputPrice')
    inp_str = f'{inp}' if inp is not None else 'N/A'
    out_str = f'{out}' if out is not None else 'N/A'
    print(f'| {name} | {unit} | {inp_str} | {out_str} |')
"

---

Step 2 -- Discover Models Actually Used in Observations

Registered model definitions are what Langfuse knows for pricing. Observations may reference model strings that do not match any registered definition, resulting in missing cost calculations. Discover which models are used in observations.

Via REST API

Fetch a sample of observations and extract unique model values:

PAGE=1
while true; do
  RESPONSE=$(curl -s -u "$PUBLIC_KEY:$SECRET_KEY" \
    "$HOST/api/public/observations?limit=100&page=$PAGE")

  COUNT=$(echo "$RESPONSE" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('data',[])))")

  if [ "$COUNT" -eq 0 ]; then
    break
  fi

  echo "$RESPONSE" >> /tmp/langfuse_observations_raw.json
  PAGE=$((PAGE + 1))

  # Limit to 20 pages for sampling
  if [ "$PAGE" -gt 20 ]; then
    break
  fi
done

Extract unique model names from the observations:

python3 -c "
import json
from collections import Counter

observations = []
with open('/tmp/langfuse_observations_raw.json') as f:
    for line in f:
        try:
            page = json.loads(line)
            observations.extend(page.get('data', []))
        except json.JSONDecodeError:
            continue

model_counts = Counter()
for obs in observations:
    model = obs.get('model')
    if model:
        model_counts[model] += 1

print('=== MODELS IN OBSERVATIONS ===')
for model, count in sorted(model_counts.items(), key=lambda x: -x[1]):
    print(f'| {model} | {count} |')
"

For exact counts on large projects, use the database fallback in Step 3.

---

Step 3 -- Database Fallback

When the REST API is unavailable, returns errors, or exact counts are needed, query PostgreSQL directly.

Option A: psycopg2 (Preferred)

Run a Python script with parameterized queries:

import psycopg2

conn = psycopg2.connect("CONNECTION_STRING_HERE")
cur = conn.cursor()

# Registered model definitions
cur.execute("""
    SELECT model_name, unit, input_price, output_price
    FROM models
    WHERE project_id = %s
    ORDER BY model_name
""", (PROJECT_ID,))
print("=== REGISTERED MODELS ===")
for row in cur.fetchall():
    name, unit, inp, out = row
    inp_str = str(inp) if inp is not None else "N/A"
    out_str = str(out) if out is not None else "N/A"
    print(f"| {name} | {unit or 'N/A'} | {inp_str} | {out_str} |")

# Models actually used in observations
cur.execute("""
    SELECT model, COUNT(*) as cnt
    FROM observations
    WHERE project_id = %s AND model IS NOT NULL
    GROUP BY model
    ORDER BY cnt DESC
""", (PROJECT_ID,))
print("\n=== MODELS IN OBSERVATIONS ===")
for row in cur.fetchall():
    print(f"| {row[0]} | {row[1]} |")

cur.close()
conn.close()

If psycopg2 is not installed, install it first:

uv add psycopg2-binary

Option B: docker exec psql (Fallback)

When the database is only reachable from within a Docker container:

# Registered models
docker exec CONTAINER_NAME psql -U USER -d DBNAME -t -A -F '|' -c "
  SELECT model_name, unit, input_price, output_price
  FROM models
  WHERE project_id = 'PROJECT_ID'
  ORDER BY model_name;
"

# Models used in observations
docker exec CONTAINER_NAME psql -U USER -d DBNAME -t -A -F '|' -c "
  SELECT model, COUNT(*) as cnt
  FROM observations
  WHERE project_id = 'PROJECT_ID' AND model IS NOT NULL
  GROUP BY model
  ORDER BY cnt DESC;
"

Parse the pipe-delimited output and format it into the standard tables.

Note: The latency field is computed, not stored. In Postgres queries, compute it as EXTRACT(EPOCH FROM (end_time - start_time)) (returns seconds as a float). The REST API returns latency as a pre-computed field.

---

Step 4 -- Cross-Reference and Merge

After collecting registered definitions and observed model names, merge them into one table. For each model, indicate whether it has registered pricing and appears in observations.

python3 -c "
import json
from collections import Counter

# Load registered models
registered = {}
try:
    with open('/tmp/langfuse_models_raw.json') as f:
        for line in f:
            try:
                page = json.loads(line)
                for m in page.get('data', []):
                    name = m.get('modelName', '')
                    registered[name] = {
                        'inputPrice': m.get('inputPrice'),
                        'outputPrice': m.get('outputPrice'),
                    }
            except json.JSONDecodeError:
                continue
except FileNotFoundError:
    pass

# Load observed models
observed = Counter()
try:
    with open('/tmp/langfuse_observations_raw.json') as f:
        for line in f:
            try:
                page = json.loads(line)
                for obs in page.get('data', []):
                    model = obs.get('model')
                    if model:
                        observed[model] += 1
            except json.JSONDecodeError:
                continue
except FileNotFoundError:
    pass

# Merge into unified view
all_models = set(registered.keys()) | set(observed.keys())

print('| Model Name | Input Price | Output Price | Used in Observations |')
print('|------------|-------------|--------------|----------------------|')
for name in sorted(all_models):
    reg = registered.get(name)
    inp = str(reg['inputPrice']) if reg and reg['inputPrice'] is not None else 'N/A'
    out = str(reg['outputPrice']) if reg and reg['outputPrice'] is not None else 'N/A'
    obs_count = observed.get(name, 0)
    used = f'Yes ({obs_count})' if obs_count > 0 else 'No'
    print(f'| {name} | {inp} | {out} | {used} |')
"

---

Step 5 -- Format and Present Output

Present the merged result as one Markdown table:

| Model Name                  | Input Price | Output Price | Used in Observations |
|-----------------------------|-------------|--------------|----------------------|
| model-name-20250514    | 0.000003    | 0.000015     | Yes (1245)           |
| gpt-4o                      | 0.000005    | 0.000015     | Yes (3400)           |
| gpt-4o-mini                 | 0.00000015  | 0.0000006    | Yes (890)            |
| text-embedding-3-small      | 0.00000002  | N/A          | Yes (5600)           |
| custom-fine-tuned-v1        | N/A         | N/A          | Yes (12)             |

After the table, provide a brief summary:

• Total number of registered model definitions.
• Total number of distinct models observed in trace data.
• Models appearing in observations without a registered pricing definition -- flag them because Langfuse cost calculations will be missing, and recommend registering them via the Langfuse UI or API.
• Registered models with zero observations -- note they may be outdated or not yet deployed.
• If pricing is configured, note the relative cost difference between the most and least expensive models.

---

Error Handling

• Empty model definitions: If no model definitions are found in the registry, this is normal for projects that rely on Langfuse's built-in model matching. Note this and focus on the observation-based model discovery.
• Empty observations: If no observations exist, report that the project has no observation data yet and suggest verifying that the application is instrumented and generating LLM calls through the Langfuse SDK.
• API timeout or 5xx errors: Log the error, switch to the DB fallback path, and note to the user that the API was unreachable.
• DB connection refused: Report the connection error and ask the user to verify the connection string, database host reachability, and whether the Langfuse database container is running.
• Permission denied on DB: Report that the database user lacks SELECT permission on the models or observations table and suggest verifying the credentials.
• Models endpoint returns 404: Some older Langfuse versions may not expose the /api/public/models endpoint. Fall back to the database query for registered models.

---

Cleanup

Remove any temporary files created during execution:

rm -f /tmp/langfuse_models_raw.json /tmp/langfuse_observations_raw.json