palantir-core-workflow-a

name	palantir-core-workflow-a
description	Build Palantir Foundry data pipelines using Python transforms. Use when creating ETL pipelines, writing @transform decorators, or building dataset-to-dataset processing in Foundry. Trigger with phrases like "palantir pipeline", "foundry transform", "palantir ETL", "palantir data pipeline", "foundry python transform".
allowed-tools	Read, Write, Edit, Bash(pip:*), Grep
version	1.5.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","palantir","foundry","transforms","pipelines","spark"]
compatibility	Designed for Claude Code

Palantir Core Workflow A — Data Pipelines with Transforms

Overview

Build Foundry data pipelines using the transforms-python library. Covers the @transform and @transform_df decorators, input/output dataset wiring, incremental transforms, and @configure for Spark tuning. This is the primary workflow for all data processing in Foundry.

Prerequisites

• Completed palantir-install-auth setup
• A Foundry Code Repository (Python Transforms type)
• Understanding of PySpark DataFrames (Foundry runs Spark under the hood)

Instructions

Step 1: Project Structure

my-transforms-repo/
├── src/
│   └── myproject/
│       ├── __init__.py
│       ├── pipeline.py          # Main transforms
│       ├── utils.py             # Shared logic
│       └── datasets.py          # Dataset path constants
├── build.gradle                 # Foundry build config
├── conda_recipe/meta.yaml       # Dependency declarations
└── settings.gradle

Step 2: Basic Transform with @transform_df

# src/myproject/pipeline.py
from transforms.api import transform_df, Input, Output

@transform_df(
    Output("/Company/datasets/cleaned_orders"),
    orders=Input("/Company/datasets/raw_orders"),
)
def clean_orders(orders):
    """Clean raw orders: drop nulls, normalize dates, filter test data."""
    from pyspark.sql import functions as F

    return (
        orders
        .filter(F.col("order_id").isNotNull())
        .filter(~F.col("email").like("[e-mail]"))
        .withColumn("order_date", F.to_date("order_date_str", "yyyy-MM-dd"))
        .withColumn(, (F.col() * ).cast())
        .drop(, )
    )