langsmith-create-dataset

name	langsmith_create_dataset
description	Guidance for creating a LangSmith dataset from data and uploading examples in bulk. Always write a custom script in the working directory.

Overview

This skill provides guidance for writing custom dataset-upload scripts to LangSmith. Always create a custom script in the working directory tailored to the specific data format and user needs.

When to use

• When run_evals.py does not have a specified dataset ID
• When you need to upload user data to LangSmith for evaluation

Process

1. Read the user's data to understand its structure (columns, format)
2. Ask the user for the dataset name they want
3. Write a custom script in the working directory (NOT in this skill folder) that:
   • Loads the data from the specified path
   • Creates a LangSmith dataset with the user's chosen name
   • Uploads examples in batches
   • Prints the dataset ID at the end
4. Run the script and capture the dataset ID
5. Use the dataset ID in run_evals.py

Key concepts for custom scripts

• Authentication: Use LANGSMITH_API_KEY (preferred) or LANGCHAIN_API_KEY environment variable
• Dataset creation: client.create_dataset(name, description=...)
• Example format: Examples are dicts with "inputs" and "outputs" keys. Both values can be nested structures (e.g., inputs: {"text": ...}, outputs: {"label": ...})
• Batching: Upload examples in batches to avoid timeouts (e.g., 200 at a time)

Example code pattern

from langsmith import Client
import pandas as pd

client = Client()
df = pd.read_csv("/path/to/data.csv")

dataset = client.create_dataset("Dataset Name", description="...")
print(f"Created dataset: {dataset.id}")

examples = []
for _, row in df.iterrows():
    examples.append({
        "inputs": {"text": row["text_column"]},
        "outputs": {"label": row["label_column"]},
    })

# Upload in batches
batch_size = 200
for i in range(0, len(examples), batch_size):
    batch = examples[i:i+batch_size]
    client.create_examples(dataset_id=dataset.id, examples=batch)
    print(f"Uploaded {len(batch)} examples")

print(f"Done. Dataset ID: {dataset.id}")

Important

• Do NOT use a pre-made script from this skill folder
• Always write a custom script in the working directory tailored to the user's data
• Always ask the user for the dataset name before creating it