ai-check

name	ai-check
description	Detect AI/LLM-generated text patterns in research writing. Use when: (1) Reviewing manuscript drafts before submission, (2) Pre-commit validation of documentation, (3) Quality assurance checks on research artifacts, (4) Ensuring natural academic writing style, (5) Tracking writing authenticity over time. Analyzes grammar perfection, sentence uniformity, paragraph structure, word frequency (AI-typical words like 'delve', 'leverage', 'robust'), punctuation patterns, and transition word overuse.
allowed-tools	Read, Grep, Bash
version	1.0.0

AI-Generated Text Detection Skill

Purpose

Detect patterns typical of LLM-generated text to ensure natural, human-authored academic writing. This skill helps maintain authenticity in research publications, dissertations, and documentation.

When to Use This Skill

Primary Use Cases

1. Pre-Commit Validation - Automatically check manuscripts and documentation before git commits
2. Manuscript Review - Validate academic writing before submission to journals or committees
3. Quality Assurance - Part of systematic QA workflow for research artifacts
4. On-Demand Analysis - Manual review of any text file for AI patterns
5. Writing Evolution Tracking - Monitor writing style changes over time

Specific Scenarios

• Before submitting dissertation chapters to advisor
• Prior to journal article submission
• When reviewing team member contributions
• After making significant edits to documentation
• When suspicious patterns are noticed in writing
• During peer review or committee review preparation

Detection Methodology

1. Grammar Pattern Analysis

What We Check:

• Excessive Perfection: Zero typos, missing commas, or minor errors throughout
• Comma Placement: Perfect comma usage in all complex sentences
• Formal Register: Consistent formal tone with no informal elements
• Grammar Consistency: No variations in grammatical choices

Red Flags:

• Absolutely no grammatical errors in 10+ pages
• Every semicolon and colon used perfectly
• No sentence fragments or run-ons even in appropriate contexts
• Overly formal language even in methods sections

Human Writing Typically Has:

• Occasional minor typos or comma splices
• Some inconsistency in formality
• Natural variations in grammar choices
• Context-appropriate informality

2. Sentence Structure Uniformity

What We Check:

• Sentence Length Distribution: Variation in sentence lengths
• Structural Patterns: Repetitive sentence structures
• Complexity Variation: Mix of simple, compound, complex sentences
• Opening Patterns: How sentences begin

Red Flags:

• Most sentences 15-25 words (AI sweet spot)
• Repetitive subject-verb-object patterns
• Every paragraph starts with topic sentence
• Excessive use of transition words at sentence starts
• Predictable sentence complexity patterns

Human Writing Typically Has:

• Wide variation (5-40+ word sentences)
• Unpredictable sentence structures
• Occasional fragments for emphasis
• Natural flow without forced transitions

3. Paragraph Structure Analysis

What We Check:

• Paragraph Length: Uniformity vs. natural variation
• Structural Pattern: Topic sentence + support + conclusion pattern
• Information Flow: Natural vs. algorithmic organization
• Paragraph Transitions: Connection between paragraphs

Red Flags:

• All paragraphs 4-6 sentences long
• Every paragraph follows same structure
• Mechanical transitions between paragraphs
• Perfectly balanced paragraph lengths
• No single-sentence paragraphs

Human Writing Typically Has:

• Paragraph length variation (1-10+ sentences)
• Structural diversity based on content
• Natural transitions
• Strategic use of short/long paragraphs for emphasis

4. Word Frequency Analysis (AI-Typical Words)

High-Risk AI Words (overused by LLMs):

Verbs:

• "delve" (rarely used by humans)
• "leverage" (business jargon)
• "utilize" (instead of "use")
• "facilitate" (overly formal)
• "demonstrate" (overused)
• "implement" (in non-technical contexts)
• "enhance" (marketing language)

Adjectives:

• "robust" (technical overuse)
• "comprehensive" (vague intensifier)
• "innovative" (buzzword)
• "cutting-edge" (cliché)
• "significant" (statistical overuse)
• "substantial" (formal overuse)
• "considerable" (formal overuse)
• "crucial" (intensity overuse)

Transition Words (overused):

• "furthermore" (very formal)
• "moreover" (archaic feeling)
• "additionally" (redundant)
• "consequently" (overused)
• "subsequently" (temporal overuse)
• "nevertheless" (formal overuse)
• "nonetheless" (synonym overuse)

Phrases:

• "it is important to note that"
• "it should be emphasized that"
• "a comprehensive analysis of"
• "in the context of"
• "with respect to"
• "in terms of"
• "in order to" (instead of "to")

Detection Criteria:

• Count frequency per 1000 words
• Compare to human academic writing baselines
• Flag if 3+ high-risk words per 1000 words
• Weight by word rarity (delve = high weight)

5. Punctuation Patterns

What We Check:

• Semicolon Usage: Frequency and correctness
• Colon Usage: Perfect usage patterns
• Em-Dash Usage: Consistent stylistic choices
• Comma Patterns: Perfection vs. natural variation
• Ellipsis/Exclamation: Absence in informal contexts

Red Flags:

• Excessive semicolon use (2+ per paragraph)
• Perfect colon usage throughout
• Consistent em-dash formatting (—)
• No missing commas anywhere
• Zero informal punctuation

Human Writing Typically Has:

• Inconsistent punctuation choices
• Occasional missing/extra commas
• Variable dash formatting (- vs -- vs —)
• Some informal punctuation where appropriate

Confidence Scoring System

Scoring Formula

Overall Confidence = Weighted average of:

• Grammar perfection: 20%
• Sentence uniformity: 25%
• Paragraph structure: 20%
• AI-typical words: 25%
• Punctuation patterns: 10%

Each metric scored 0-100, then combined with weights.

Confidence Levels

Low Confidence (0-30%): Likely Human Writing

Characteristics:

• Natural sentence length variation (5-40+ words)
• Occasional grammatical imperfections
• Authentic voice and natural flow
• Domain-specific terminology used naturally
• Structural variety in paragraphs
• Minimal AI-typical words (0-2 per 1000 words)

Action: ✅ Writing appears authentic, no changes needed

Medium Confidence (30-70%): Possible AI Assistance

Characteristics:

• Some uniformity in sentence structure
• Mix of AI-typical and natural patterns
• May be human-edited AI output
• Overly formal in places
• Some transition word overuse
• 3-5 AI-typical words per 1000 words

Action: ⚠️ Review flagged sections, apply suggestions selectively

Examples of Mixed Writing:

• AI-generated first draft with heavy human editing
• Human writing that mimics academic formality excessively
• Non-native English speakers using formal templates
• Multiple authors with different styles

High Confidence (70-100%): Likely AI-Generated

Characteristics:

• Excessive uniformity across all metrics
• Multiple AI-typical word clusters
• Perfect grammar and punctuation throughout
• Artificial transition patterns
• Mechanical paragraph structure
• 6+ AI-typical words per 1000 words

Action: 🚫 Significant revision needed, rewrite in authentic voice

Output Format

When running AI-check analysis, generate a comprehensive report:

1. Executive Summary

Overall Confidence Score: 65%
Status: MEDIUM - Possible AI assistance detected
Files Analyzed: 1
Total Words: 3,456
Recommendation: Review flagged sections

2. Metric Breakdown

Grammar Perfection:     85% (High - suspiciously few errors)
Sentence Uniformity:    72% (High - repetitive structures)
Paragraph Structure:    68% (Medium - some variation)
AI-Typical Words:       58% (Medium - 4.2 per 1000 words)
Punctuation Patterns:   45% (Low - natural variation)

3. Flagged Sections

Lines 45-67 (Confidence: 82%)
  Pattern: Excessive transition words + uniform sentences
  AI Words: "moreover", "furthermore", "leverage", "robust"
  
Lines 112-134 (Confidence: 76%)
  Pattern: Perfect grammar + mechanical structure
  AI Words: "delve", "comprehensive", "facilitate"

4. Specific Issues Detected

High-Risk AI Words Found (per 1000 words):
  • "delve" (2 occurrences) - RARELY used by humans
  • "leverage" (3 occurrences) - Business jargon overuse
  • "robust" (4 occurrences) - Technical overuse
  • "furthermore" (6 occurrences) - Formal transition overuse

Sentence Uniformity Issues:
  • 67% of sentences are 15-25 words (AI sweet spot)
  • 82% of paragraphs start with transition words
  • Low variation in sentence complexity

Paragraph Structure Issues:
  • All paragraphs 4-6 sentences long
  • Mechanical topic-sentence pattern throughout

5. Word Frequency Report

Top AI-Typical Words:
1. "furthermore" - 6x (baseline: 0.5x per 1000 words)
2. "robust" - 4x (baseline: 0.8x per 1000 words)
3. "leverage" - 3x (baseline: 0.3x per 1000 words)
4. "comprehensive" - 3x (baseline: 1.2x per 1000 words)
5. "delve" - 2x (baseline: 0.1x per 1000 words)

Comparison to Human Academic Writing:
  Your text: 4.2 AI-typical words per 1000
  Human baseline: 1.5 AI-typical words per 1000
  Ratio: 2.8x higher than human baseline

Improvement Suggestions

For High Confidence (70-100%) Detections

Sentence Structure:

• ❌ "Furthermore, the results demonstrate a comprehensive analysis of the robust dataset."
• ✅ "The results show our analysis covered the full dataset."

Why Better: Simpler words, no transition word, more direct

Word Choice:

• ❌ "This study delves into the utilization of innovative methodologies."
• ✅ "We examine how researchers use new methods."

Why Better: Active voice, common words, clearer meaning

Paragraph Variation:

• ❌ All paragraphs 5 sentences, topic sentence + 3 support + conclusion
• ✅ Mix paragraph lengths: 2, 7, 4, 3, 6 sentences based on content needs

Why Better: Natural flow based on content, not formula

Specific Suggestion Categories

1. Vary Sentence Lengths

Current: 15-25 word sentences consistently
Suggestion: Mix short (5-10), medium (15-20), long (25-35) sentences
Example:
  - Short: "The effect was significant."
  - Medium: "We observed a 23% increase across all conditions."
  - Long: "This finding aligns with previous work showing that..."

2. Replace AI-Typical Words

Replace → With
- "delve into" → "examine", "explore", "investigate"
- "leverage" → "use", "apply", "employ"
- "utilize" → "use"
- "robust" → "strong", "reliable", "thorough"
- "facilitate" → "enable", "help", "allow"
- "furthermore" → "also", "next", [or remove]
- "moreover" → "additionally", "also", [or use dash]
- "comprehensive" → "complete", "thorough", "full"

3. Add Natural Imperfections (Where Appropriate)

- Use contractions in appropriate contexts ("it's", "we'll")
- Include domain-specific jargon naturally
- Allow informal phrasing in methods/procedures
- Use occasional sentence fragments for emphasis
- Add personal observations or interpretations
- Include field-specific colloquialisms

4. Break Paragraph Uniformity

Current: All paragraphs follow topic-support-support-support-conclusion
Suggestion: Vary based on content
  - Use single-sentence paragraphs for emphasis
  - Combine related ideas into longer paragraphs
  - Don't force every paragraph to have 5 sentences
  - Let content determine structure, not formula

5. Remove Mechanical Transitions

❌ "Furthermore, the results show... Moreover, the analysis reveals..."
✅ "The results show... The analysis also reveals..." [simpler transitions]
✅ "The results show... Looking closer, the analysis..." [natural bridges]

Integration Points

1. Pre-Commit Hook Integration

Automatic checking before git commits

# Configured in .claude/settings.json
"gitPreCommit": {
  "command": "python3 hooks/pre-commit-ai-check.py",
  "enabled": true
}

Behavior:

• Runs on staged .md, .tex, .rst files
• Warns if confidence 30-70%
• Blocks commit if confidence >70%
• User can override with git commit --no-verify

Exit Codes:

• 0: Pass (confidence <30%)
• 1: Warning (confidence 30-70%, commit allowed)
• 2: Block (confidence >70%, commit blocked)

2. Quality Assurance Integration

Part of comprehensive QA workflow

Integrated into code/quality_assurance/qa_manager.py:

• Runs during manuscript phase QA validation
• Checks all deliverable documents
• Generates detailed QA report section
• Fails QA if confidence >40%

Configuration (.ai-check-config.yaml):

qa_integration:
  enabled: true
  max_confidence_threshold: 0.40
  check_manuscripts: true
  check_documentation: true
  generate_detailed_reports: true

3. Manuscript Writer Agent Integration

Real-time feedback during writing

Agent checks writing incrementally:

• After drafting each section
• Before moving to next phase
• Applies suggestions automatically
• Re-checks until confidence <30%

Agent Workflow:

1. Draft section
2. Run ai-check skill
3. Review detection results
4. Apply improvement suggestions
5. Re-check until authentic
6. Proceed to next section

4. Standalone Skill Usage

Manual invocation by user or agents

User Invocation:

Please run ai-check on docs/manuscript/discussion.tex and provide detailed feedback.

Agent Invocation:

I'll use the ai-check skill to verify this text before proceeding.

CLI Tool:

python tools/ai_check.py path/to/file.md
python tools/ai_check.py --directory docs/
python tools/ai_check.py --format html --output report.html

Tracking System

Historical Tracking

Log all AI-check runs to database for evolution tracking:

Database Schema (PostgreSQL via research-database MCP):

CREATE TABLE ai_check_history (
  id SERIAL PRIMARY KEY,
  file_path TEXT NOT NULL,
  git_commit TEXT,
  timestamp TIMESTAMP DEFAULT NOW(),
  overall_confidence FLOAT,
  grammar_score FLOAT,
  sentence_score FLOAT,
  paragraph_score FLOAT,
  word_score FLOAT,
  punctuation_score FLOAT,
  ai_words_found JSONB,
  flagged_sections JSONB
);

Trend Analysis

Track writing evolution:

File: docs/manuscript/discussion.tex

Version History:
2025-01-15: 78% confidence (HIGH - likely AI)
2025-01-18: 52% confidence (MEDIUM - revision 1)
2025-01-20: 34% confidence (LOW-MEDIUM - revision 2)
2025-01-22: 18% confidence (LOW - authentic writing)

Trend: ✅ Improving toward authentic writing

Use Cases:

• Monitor dissertation chapters over time
• Track improvements after applying suggestions
• Demonstrate writing authenticity to committee
• Identify sections needing more work

Configuration

Configuration File: .ai-check-config.yaml

# AI-Check Skill Configuration

# Pre-Commit Hook Settings
pre_commit:
  enabled: true
  check_files: [".md", ".tex", ".rst", ".txt"]
  check_docstrings: true  # Check Python docstrings
  block_threshold: 0.70   # Block commit if >= 70%
  warn_threshold: 0.30    # Warn if >= 30%
  exclude_patterns:
    - "*/examples/*"
    - "*/tests/*"
    - "*/node_modules/*"
    - "*/.venv/*"

# Quality Assurance Integration
qa_integration:
  enabled: true
  max_confidence_threshold: 0.40  # Fail QA if >= 40%
  check_manuscripts: true
  check_documentation: true
  generate_detailed_reports: true
  track_history: true

# Detection Parameters
detection:
  # Weight each metric (must sum to 1.0)
  weights:
    grammar_perfection: 0.20
    sentence_uniformity: 0.25
    paragraph_structure: 
     
     
  
  
  
     [, , ]
     [, , ]
     [, , ]
  
  
   
   
  


     
   
   
   
   



   
   
   

Per-Project Overrides

Create .ai-check.local.yaml for project-specific settings:

# Project-specific overrides
pre_commit:
  block_threshold: 0.60  # More lenient for early drafts
  
detection:
  ai_words:
    high_risk: ["delve"]  # Only flag worst offenders

Examples

Example 1: High Confidence Detection

Input Text:

Furthermore, this comprehensive study delves into the robust 
methodologies utilized to facilitate the implementation of innovative 
approaches. Moreover, the analysis demonstrates significant findings 
that leverage state-of-the-art techniques. Subsequently, the results 
indicate substantial improvements across all metrics. Nevertheless, 
additional research is crucial to fully comprehend the implications.

AI-Check Report:

Overall Confidence: 89% (HIGH - Likely AI-generated)

Issues Detected:
- 8 AI-typical words in 60 words (13.3 per 1000 words!)
- Every sentence starts with transition word
- Uniform sentence length (15-18 words each)
- Perfect grammar, zero natural imperfections
- Mechanical paragraph structure

AI Words Found:
- furthermore, comprehensive, delves, robust
- utilized, facilitate, innovative, leverage
- demonstrates, significant, subsequently, substantial
- nevertheless, crucial, comprehend

Recommendation: Complete rewrite recommended

Suggested Revision:

We examined the methods used in this approach. The analysis shows 
clear improvements across metrics. However, more research is needed 
to understand the full implications.

(23 words, 12% confidence - much more natural)

Example 2: Medium Confidence Detection

Input Text:

The experimental design followed standard protocols established in 
previous work (Smith et al., 2023). We collected data from 150 
participants over six months. Statistical analysis used mixed-effects 
models to account for repeated measures. The results showed a 
significant main effect of condition (p < 0.001).

AI-Check Report:

Overall Confidence: 35% (MEDIUM - Possible minor AI assistance)

Issues Detected:
- Slightly uniform sentence length (11-15 words)
- One AI-typical word: "significant" (statistical context acceptable)
- Otherwise natural academic writing

Recommendation: Minor revisions optional, writing appears largely authentic

Example 3: Low Confidence (Human Writing)

Input Text:

OK so here's what we found. The effect was huge - way bigger than 
expected. Participants in the experimental group scored 23% higher 
on average. This wasn't just statistically significant; it was 
practically meaningful.

We're still not sure why. Maybe it's the timing? Could be the 
instructions were clearer. Need to run follow-ups.

AI-Check Report:

Overall Confidence: 8% (LOW - Clearly human writing)

Human Writing Indicators:
- Natural sentence variation (4-19 words)
- Informal elements ("OK so", "way bigger")
- Incomplete thoughts and questions
- Natural uncertainty expressions
- Zero AI-typical words
- Authentic voice throughout

Recommendation: Writing is authentic, no changes needed

Best Practices

For PhD Students

1. Run Before Advisor Meetings

   • Check chapters before sending to advisor
   • Ensure authenticity before committee review
   • Track improvements over time

2. Use During Drafting

   • Check each section after writing
   • Apply suggestions immediately
   • Develop natural writing habits

3. Pre-Submission Validation

   • Run on complete manuscripts before journal submission
   • Check supplementary materials
   • Verify all documentation

For Research Teams

1. Establish Team Standards

   • Set agreed-upon confidence thresholds
   • Define when to block vs. warn
   • Create team-specific word lists

2. Code Review Integration

   • Check documentation in pull requests
   • Validate README files and guides
   • Ensure authentic technical writing

3. Track Team Writing

   • Monitor trends across team members
   • Identify systematic issues
   • Share improvement strategies

For Journal Submission

1. Pre-Submission Checklist

   • Overall confidence <30%
   • No flagged high-risk sections
   • AI-typical words <2 per 1000 words
   • Natural sentence variation present
   • Authentic academic voice throughout

2. Demonstrating Authenticity

   • Include AI-check reports in submission materials
   • Show writing evolution over time
   • Document revision process

Limitations

What This Skill Cannot Do

1. Not 100% Accurate

   • LLMs constantly improving
   • Patterns evolve over time
   • False positives possible (very formal human writing)
   • False negatives possible (heavily edited AI text)

2. Cannot Detect All AI Usage

   • Well-edited AI text may pass
   • Human writing in AI style may be flagged
   • Paraphrasing tools may evade detection
   • Future models may have different patterns

3. Domain Limitations

   • Trained primarily on academic writing
   • May not work well for creative writing
   • Technical jargon may affect scores
   • Non-English text not supported

Use Alongside Human Judgment

This skill is a tool, not a replacement for human judgment:

• Use confidence scores as guidance, not absolute truth
• Consider context and field-specific norms
• Combine with plagiarism detection tools
• Maintain academic integrity standards
• Update word lists as AI patterns evolve

Support

Troubleshooting

Problem: False positive on authentic writing Solution: Check if writing is overly formal. Consider field-specific norms. Adjust thresholds in config.

Problem: AI text passing with low confidence Solution: Update AI-typical word lists. Check for heavily edited text. Report patterns for skill updates.

Problem: Pre-commit hook too slow Solution: Reduce checked file types. Enable caching. Check only modified sections.

Problem: Disagreement with manual review Solution: Generate detailed report. Review flagged sections specifically. Consider multiple metrics not just overall score.

Getting Help

• Documentation: See docs/skills/ai-check-reference.md
• Issues: https://github.com/astoreyai/ai_scientist/issues
• Updates: Check for skill updates regularly as AI patterns evolve

---

Last Updated: 2025-11-09 Version: 1.0.0 License: MIT