# Smart Redaction Tool
An intelligent document redaction application that automatically detects and redacts Personally Identifiable Information (PII) from documents using a two-pass detection system combining regex pattern matching and AI-powered semantic analysis.
## Features
- **Two-Pass PII Detection**
- **Pass 1: Regex Pattern Matching** - Fast, high-precision detection of standard formats (SSNs, credit cards, phone numbers, emails, etc.)
- **Pass 2: AI/LLM Analysis** - Aggressive semantic detection using Case.dev's LLM to find non-standard formats, contextual PII, names, and addresses
- **Supported PII Types**
- Social Security Numbers (SSN)
- Bank Account Numbers
- Credit Card Numbers
- Names
- Addresses
- Phone Numbers
- Email Addresses
- Dates of Birth
- **Document Processing**
- PDF text extraction using pdf.js
- Text file support
- Image OCR support (via Case.dev OCR API)
- **User-Friendly Workflow**
1. **Upload** - Drag and drop or select documents
2. **Configure** - Choose from prefigured redaction presets or custom selection
3. **Review** - Preview detected entities, toggle individual redactions, edit masked values
4. **Export** - Download redacted PDFs and audit logs
## Architecture
```
src/
โโโ app/
โ โโโ page.tsx # Main application UI
โ โโโ api/
โ โโโ detect-pii/route.ts # Two-pass PII detection endpoint
โ โโโ export-pdf/route.ts # PDF generation with redactions
โ โโโ detect/route.ts # Database-backed detection (full workflow)
โ โโโ export/route.ts # Database-backed export
โ โโโ jobs/route.ts # Job management
โ โโโ upload/route.ts # File upload handling
โโโ components/
โ โโโ redaction/
โ โ โโโ PatternSelector.tsx # Redaction type selection UI
โ โ โโโ EntityList.tsx # Detected entities list with controls
โ โ โโโ DocumentPreview.tsx # Document preview with highlights
โ โโโ upload/
โ โ โโโ DropZone.tsx # File upload component
โ โโโ ui/ # Shadcn UI components
โโโ lib/
โ โโโ redaction/
โ โ โโโ detector.ts # Two-pass detection logic
โ โ โโโ patterns.ts # Regex patterns and presets
โ โโโ case-dev/
โ โ โโโ client.ts # Case.dev API client
โ โโโ db.ts # Database connection
โ โโโ utils.ts # Utility functions
โโโ prisma/
โโโ schema.prisma # Database schema
```
## Tech Stack
- **Frontend**: Next.js 14, React, TypeScript, Tailwind CSS
- **UI Components**: Shadcn UI
- **PDF Processing**: pdf.js (extraction), pdf-lib (generation)
- **AI/LLM**: Case.dev API (GPT-4o)
- **Database**: PostgreSQL with Prisma ORM (optional, for full workflow)
## Setup
### Prerequisites
- Node.js 18+
- npm or yarn
- Case.dev API key (for AI-powered detection)
### Installation
1. **Clone the repository**
```bash
git clone <repository-url>
cd redaction-tool
```
2. **Install dependencies**
```bash
npm install
```
3. **Configure environment variables**
Copy the example environment file:
```bash
cp .env.example .env
```
Edit `.env` and add your Case.dev API key:
```env
CASEDEV_API_KEY=your_api_key_here
```
4. **Run the development server**
```bash
npm run dev
```
5. **Open the application**
Navigate to [http://localhost:3000](http://localhost:3000)
### Database Setup (Optional)
For the full workflow with job persistence:
1. **Set up PostgreSQL** and add the connection string to `.env`:
```env
DATABASE_URL="postgresql://user:password@localhost:5432/redaction_tool"
```
2. **Run migrations**:
```bash
npx prisma migrate dev
```
## Usage
### Quick Start
1. **Upload Documents**: Drag and drop PDF or text files into the upload zone
2. **Select Redaction Types**: Choose a preset (e.g., "SSNs and Financial Account Numbers") or customize
3. **Scan Documents**: Click "Scan Documents" to run detection
4. **Review Results**: Toggle individual entities on/off, edit masked values if needed
5. **Export**: Generate and download the redacted PDF
### Redaction Presets
| Preset | Description | Types Included |
|--------|-------------|----------------|
| SSNs and Financial | Financial document redaction | SSN, Account Numbers, Credit Cards |
| All Personal Information | Comprehensive PII redaction | All types |
| Contact Information Only | Communication data | Phone, Email |
| Financial Only | Banking data | Account Numbers, Credit Cards |
### API Endpoints
#### POST `/api/detect-pii`
Detect PII in text using two-pass detection.
**Request:**
```json
{
"text": "Document text content...",
"types": ["SSN", "CREDIT_CARD", "NAME"]
}
```
**Response:**
```json
{
"success": true,
"count": 5,
"matches": [
{
"id": "entity-123",
"type": "SSN",
"value": "123-45-6789",
"maskedValue": "***-**-6789",
"confidence": 0.95,
"detectionMethod": "regex"
}
]
}
```
#### POST `/api/export-pdf`
Generate a redacted PDF from text and entities.
**Request:**
```json
{
"text": "Document text content...",
"entities": [...],
"filename": "redacted-document.pdf"
}
```
**Response:** PDF file download
## Detection Methodology
### Pass 1: Regex Pattern Matching
High-precision patterns for standard formats:
- SSN: `XXX-XX-XXXX` format with validation
- Credit Cards: Luhn-valid card number patterns
- Phone: US phone number formats
- Email: Standard email format
- Dates: Common date formats
### Pass 2: AI/LLM Analysis
Aggressive semantic detection for:
- Non-standard formats (e.g., "SSN: one two three...")
- Contextual references (e.g., "my social security number is...")
- Names and addresses (cannot be reliably detected with regex)
- OCR errors and typos
- Obfuscated or partial data
The AI is configured to be **moderately aggressive** - it's better to flag potential PII (which users can unselect) than to miss actual sensitive data.
## Configuration
### Environment Variables
| Variable | Description | Required |
|----------|-------------|----------|
| `CASEDEV_API_KEY` | Case.dev API key for LLM access | Yes |
| `DATABASE_URL` | PostgreSQL connection string | No (for full workflow) |
### Customizing Detection
Edit `src/lib/redaction/patterns.ts` to:
- Add new regex patterns
- Modify confidence thresholds
- Create new redaction presets
Edit `src/lib/redaction/detector.ts` to:
- Adjust LLM prompts
- Modify detection aggressiveness
- Add new detection passes
## License
Copyright 2025 Case.dev
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
## Support
For issues or questions, please open a GitHub issue or contact the development team.