<div align="center">
<img src="https://github.com/user-attachments/assets/2137f529-e5de-4d8e-9ae2-3d67a797d0c9" width="300"/>
# **Substrate**
### **Infrastructure for Human Knowledge & Progress**
[](https://github.com/danielmiessler/Substrate/commits/main)
[](https://opensource.org/licenses/MIT)
[](https://github.com/danielmiessler/Substrate/stargazers)
[](https://www.typescriptlang.org/)
[](https://bun.sh)
<p align="center">
<a href="#-what-is-substrate">What Is It</a> โข
<a href="#-data--evidence">Data</a> โข
<a href="#-quick-start">Quick Start</a> โข
<a href="#-documentation">Docs</a> โข
<a href="#-contribute">Contribute</a> โข
<a href="#-roadmap">Roadmap</a>
</p>
</div>
---
## ๐ฏ What Is Substrate?
<div align="center">
<a href="https://www.youtube.com/watch?v=ky7ejowc_qY">
<img src="https://img.youtube.com/vi/ky7ejowc_qY/0.jpg" alt="Substrate Introduction" width="600">
</a>
**[โ Watch Introduction Video](https://www.youtube.com/watch?v=ky7ejowc_qY)** โข **[โ Read Blog Post](https://danielmiessler.com/blog/how-my-projects-fit-together)**
</div>
---
### **A Shared Foundation for Human Progress**
Think of **substrate** as the base layerโthe common ground where we can all work together to understand problems and drive solutions forward. Instead of fragmented discussions and reinventing the wheel, Substrate gives us **one place** to:
- **Document problems** โ What's actually broken, with evidence
- **Track solutions** โ What works, what doesn't, with results
- **Connect progress** โ Link people, organizations, and projects actually moving things forward
- **Build on each other** โ Arguments and ideas that build on shared evidence
- **Measure outcomes** โ Did it work? What actually changed?
**The Purpose: Accelerate Human Progress**
We can't solve problems we don't understand. We can't build on solutions we can't find. Substrate provides:
- ๐ฏ **Shared understanding** โ One place to understand what's wrong and what works
- ๐ **Faster progress** โ Build on existing knowledge instead of starting over
- ๐ **Evidence-based action** โ Ground decisions in authoritative data, not opinions
- ๐ **Connected knowledge** โ See how problems, solutions, people, and data interconnect
- ๐ **Collective intelligence** โ Human insight + AI analysis working together toward progress
**An open-source framework connecting 17+ knowledge components:**
```mermaid
graph TB
subgraph "๐ The Substrate"
A[๐งฉ Problems]
B[๐ก Solutions]
C[๐ Data Sources]
D[๐ฃ๏ธ Arguments]
E[๐ Claims]
F[๐ฅ People]
G[๐ข Organizations]
H[๐ Projects]
I[๐ Plans]
J[๐ฏ Values]
K[๐ญ Ideas]
end
L[๐ค Human Contributors] --> A
L --> B
L --> D
M[๐ค AI Analysis] --> A
M --> B
M --> D
A -.connects to.-> B
B -.connects to.-> C
D -.connects to.-> C
E -.connects to.-> C
F -.connects to.-> G
G -.connects to.-> H
H -.connects to.-> B
A --> N[๐ Shared Understanding]
B --> N
C --> N
D --> N
```
<table>
<tr>
<td width="50%">
### ๐๏ธ **Structured Components**
- **Problems** - Documented challenges with evidence
- **Solutions** - Proven approaches with results
- **Arguments** - Reasoning chains with quality scores
- **Claims** - Assertions linked to evidence
- **Plans** - Actionable strategies with metrics
- **Ideas** - Frameworks and concepts
- **People & Organizations** - Who's working on what
- **Projects** - Active initiatives with outcomes
</td>
<td width="50%">
### ๐ **Authoritative Data**
- **13 Data Sources** across 100+ years
- **Library science methodology** (8 dimensions)
- **Government agencies** + verified databases
- **TypeScript automation** with Bun runtime
- **Full documentation** for every dataset
- **Free APIs** with excellent access
- **Human wellbeing indicators** beyond GDP
- **Real-time** to annual update frequencies
</td>
</tr>
</table>
---
## ๐ What's New
> [!TIP]
> **October 2025 - Massive Data Expansion!**
>
> We just added **8 authoritative data sources** measuring human wellbeing beyond traditional economic metrics. Now track financial stress, health crises, social connection, worker agency, and environmental health.
<details>
<summary><strong>๐
Latest Updates (Click to expand)</strong></summary>
### ๐ฅ October 27, 2025 - Human Wellbeing Infrastructure
**5 NEW US Wellbeing Data Sources:**
| Added | Source | What It Measures |
|-------|--------|------------------|
| ๐ | **DS-00004** FRED Economic Wellbeing | Debt stress, unemployment, inequality |
| ๐ | **DS-00005** CDC WONDER Mortality | Overdoses, suicides, deaths of despair |
| ๐ | **DS-00006** Census ACS Social | Living alone, commute times, isolation |
| ๐ | **DS-00007** BLS JOLTS Labor | Worker "permission to quit" index |
| ๐ | **DS-00008** EPA Air Quality | PM2.5, ozone, environmental health |
**Why This Matters:**
> "If we measure only GDP and unemployment, we will miss the slow-motion collapse of human thriving happening in plain sight."
These sources capture:
- ๐ฐ **Leading indicators** - Financial stress predicts recessions
- ๐ญ **Behavioral truth** - Actions reveal what surveys miss
- ๐๏ธ **Structural determinants** - Air quality you can't "self-care" out of
- ๐จ **Crisis detection** - County-level suffering visibility
- ๐ **Worker agency** - Quit rate = economic freedom measure
### October 25, 2025 - Core Data Updates
- โ
Pulitzer Prize, GDP, inflation data refreshed
- โ
Knowledge Worker Salaries validation completed
### October 16-18, 2025 - Infrastructure
- ๐๏ธ Library science methodology (8-dimension evaluation)
- โก TypeScript automation with auto-discovery
- ๐ Central logging and health monitoring
- ๐ Complete documentation suite
---
<details>
<summary><strong>๐ Previous Updates (Click to expand)</strong></summary>
### September 2024 - Community Growth
- โ
Claims, Arguments, and Values frameworks
- โ
6+ community contributors
- โ
10+ merged pull requests
- โ
TELOS integration
### August 2024 - Framework Expansion
- โ
Extended component types (Plans, Ideas, Models)
- โ
Improved documentation structure
- โ
Enhanced contribution guidelines
### July 2024 - Foundation
- โ
Single-repo structure with 17+ object types
- โ
Core framework and documentation
- โ
Public launch with initial datasets
- โ
Community contribution framework
</details>
---
**[โ View Full Changelog](./UPDATES.md)**
</details>
---
## ๐ Data & Evidence
> [!NOTE]
> All data sources include **complete library science cataloging** with 8-dimension evaluation: Authority, Currency, Objectivity, Accuracy, Methodology, Coverage, Reliability, and Provenance.
> [!IMPORTANT]
> **We know data can be controversial.** That's exactly why we:
> - ๐ **Collect from multiple sources** - Cross-reference data from different authoritative providers
> - ๐ **Provide complete transparency** - Every source fully documented with provenance, methodology, and limitations
> - ๐ **Full logging** - All data pulls logged with timestamps, source versions, and processing steps
> - ๐ **Open source everything** - TypeScript update scripts show exactly how data is fetched and transformed
>
> You can verify, audit, and challenge our data. That's the point.
### **Core Datasets** (`Data/`)
| Dataset | Time Span | Data Points | Source | Status |
|---------|-----------|-------------|--------|--------|
| ๐บ๐ธ **US GDP** | 1929-2025 | 96 years + 314 quarters | FRED/BEA | โ
Active |
| ๐ต **US Inflation** | 1947-2025 | 945 months | FRED/BLS | โ
Active |
| ๐ฆ **Bay Area COVID Wastewater** | 2022-2025 | 161 weeks | CDPH | โ
Active |
| ๐ **Pulitzer Prize Winners** | 1918-2024 | 249 winners | Wikidata | โ
Active |
| ๐ผ **Knowledge Worker Salaries** | Global | Multi-region | Research | โ
Active |
### **Wellbeing Data Sources** (`Data-Sources/`) ๐
<details>
<summary><strong>๐ Global Health & Development (Click to expand)</strong></summary>
| ID | Name | Coverage | Indicators | Update |
|----|------|----------|------------|--------|
| **DS-00001** | [WHO Global Health Observatory](./Data-Sources/DS-00001โWHO_Global_Health_Observatory/) | 194 countries | 2,000+ health indicators | Quarterly |
| **DS-00002** | [UN SDG Indicators](./Data-Sources/DS-00002โUN_SDG_Indicators/) | 193 countries | 231 SDG targets | Biannual |
| **DS-00003** | [World Bank Open Data](./Data-Sources/DS-00003โWorld_Bank_Open_Data/) | Global | Development metrics | Varies |
</details>
<details open>
<summary><strong>๐บ๐ธ US Human Wellbeing (October 2025)</strong></summary>
<table>
<tr>
<th width="15%">Source</th>
<th width="35%">Key Indicators</th>
<th width="30%">Why It Matters</th>
<th width="20%">Update Freq</th>
</tr>
<tr>
<td><strong>DS-00004</strong><br/><a href="./Data-Sources/DS-00004โFRED_Economic_Wellbeing/">FRED Economic</a></td>
<td>
โข Debt service ratio<br/>
โข Credit card delinquency<br/>
โข Financial stress index<br/>
โข Underemployment (U-6)<br/>
โข Consumer sentiment<br/>
โข GINI inequality
</td>
<td>๐ฐ <strong>Economic Security Foundation</strong><br/>Debt >12% = stress, >14% = crisis. Leading indicator of recession.</td>
<td>Weekly to Annual</td>
</tr>
<tr>
<td><strong>DS-00005</strong><br/><a href="./Data-Sources/DS-00005โCDC_WONDER_Mortality/">CDC WONDER</a></td>
<td>
โข Drug overdose deaths<br/>
โข Opioid-specific deaths<br/>
โข Suicide rates<br/>
โข All-cause mortality
</td>
<td>๐จ <strong>Crisis Detection</strong><br/>Deaths of despair reveal community breakdown surveys miss. County-level visibility.</td>
<td>Annual (1-2yr lag)</td>
</tr>
<tr>
<td><strong>DS-00006</strong><br/><a href="./Data-Sources/DS-00006โCensus_ACS_Social_Wellbeing/">Census ACS</a></td>
<td>
โข Living alone (isolation)<br/>
โข Commute times (time poverty)<br/>
โข No internet (digital divide)<br/>
โข Household demographics
</td>
<td>๐ค <strong>Social Connection</strong><br/>Structural isolation and time poverty constrain flourishing. Census tract granularity.</td>
<td>Annual</td>
</tr>
<tr>
<td><strong>DS-00007</strong><br/><a href="./Data-Sources/DS-00007โBLS_JOLTS_Labor_Market/">BLS JOLTS</a></td>
<td>
โข <strong>Quit rate</strong> (worker agency)<br/>
โข Job openings<br/>
โข Hire rate<br/>
โข Layoff/discharge rate
</td>
<td>๐ <strong>"Permission to Quit"</strong><br/>People only quit when they have options. Measures economic freedom and dignity.</td>
<td>Monthly</td>
</tr>
<tr>
<td><strong>DS-00008</strong><br/><a href="./Data-Sources/DS-00008โEPA_Air_Quality_System/">EPA Air Quality</a></td>
<td>
โข PM2.5 (particulate matter)<br/>
โข Ozone (O3)<br/>
โข SO2, CO, NO2<br/>
โข AQI exceedances
</td>
<td>๐ซ๏ธ <strong>Environmental Health</strong><br/>You can't "self-care" out of breathing toxic air. Structural wellbeing constraint.</td>
<td>Real-time</td>
</tr>
</table>
</details>
### **Composite Wellbeing Indices**
> [!TIP]
> Combine multiple data sources to create powerful wellbeing metrics:
- ๐ธ **Financial Stress Composite** - Debt + delinquency + evictions + stress index
- ๐จ **Crisis Alert Composite** - Overdoses + suicides + long-term unemployment
- ๐ค **Community Health Composite** - Living alone + commute + digital divide (inverted)
- ๐ **Worker Agency Index** - Quit rate + job openings / unemployment
- ๐ซ๏ธ **Environmental Health Index** - PM2.5 + ozone (inverted)
**[โ Wellbeing Implementation Guide](./Data-Sources/WELLBEING_DATA_SOURCES.md)** | **[โ Data Directory](./Data/README.md)**
---
## ๐ค Human & AI Collaboration
> **Substrate provides the pieces. You and your AI create the connections.**
<table>
<tr>
<td width="50%">
### ๐ค **Humans Contribute**
- Document problems and solutions
- Add authoritative data sources
- Create arguments and claims
- Link entities explicitly
- Validate AI suggestions
- Rate quality of evidence
</td>
<td width="50%">
### ๐ค **AI Analyzes**
- Scan thousands of components
- Suggest relationships automatically
- Detect patterns across datasets
- Score argument quality
- Find contradictions
- Generate knowledge graphs
</td>
</tr>
</table>
**Everything is designed for dual consumption:**
- โ
**Human-readable** - Markdown and CSV anyone can open
- โ
**Machine-parseable** - Consistent formats AI can query
- โ
**Fully documented** - Complete methodology for every dataset
- โ
**Linked with IDs** - Unambiguous entity references
### Use Cases
<details>
<summary><strong>๐ OSINT & Investigation</strong></summary>
- Cross-reference public records, corporate filings, government data
- Link people โ organizations โ transactions โ policies
- Surface connections invisible in isolated databases
- Build evidence chains from claims to verifiable records
- Pattern detection for fraud, corruption, illicit networks
</details>
<details>
<summary><strong>๐ Research & Analysis</strong></summary>
- Track claims against authoritative data sources
- Evaluate argument quality based on evidence backing
- Compare solutions across different implementations
- Measure progress toward stated goals with real metrics
- Cross-correlate economic, health, social, environmental data
</details>
<details>
<summary><strong>๐ Public Accountability</strong></summary>
- Verify political claims against documented evidence
- Track campaign promises โ policy outcomes โ measured results
- Link donations โ voting records โ policy positions
- Monitor government spending against stated objectives
- Environmental justice analysis (who breathes toxic air?)
</details>
---
## ๐ Quick Start
### **View the Data** (No Installation Required)
All datasets are available as **CSV and Markdown files** you can browse directly:
```bash
# Clone the repository
git clone https://github.com/danielmiessler/Substrate.git
cd Substrate
# Explore core datasets
ls Data/*/
# Explore wellbeing data sources
ls Data-Sources/*/
```
### **Run the Automation** (Optional)
> [!WARNING]
> Requires [Bun](https://bun.sh) runtime. Install: `curl -fsSL https://bun.sh/install | bash`
```bash
# Install dependencies
bun install
# Update a specific dataset
cd Data/US-GDP
bun run update.ts
# Update a wellbeing source (requires API key)
export FRED_API_KEY="your_key_here"
cd Data-Sources/DS-00004โFRED_Economic_Wellbeing
bun run update.ts
# Update all datasets
bun run scripts/update-all.ts
```
### **Get API Keys** (Free)
| Data Source | Get Key | Rate Limit |
|-------------|---------|------------|
| FRED Economic | [fred.stlouisfed.org/docs/api](https://fred.stlouisfed.org/docs/api/api_key.html) | 120 req/min |
| Census ACS | [api.census.gov/data/key_signup](https://api.census.gov/data/key_signup.html) | 500 req/day |
| EPA Air Quality | Email: aqs.support@epa.gov | 10 req/min |
| BLS JOLTS | [bls.gov/developers/home](https://www.bls.gov/developers/home.htm) | 500 req/day |
| CDC WONDER | No key required | Fair use |
**[โ Complete Getting Started Guide](./GETTING_STARTED.md)** | **[โ Quick Reference](./QUICK_REFERENCE.md)**
---
## ๐ก Contribute
> [!IMPORTANT]
> **Anyone can submit components.** No gatekeeping on ideasโjust structured formats.
### **What You Can Add**
<table>
<tr>
<td width="33%">
**๐ Problems**
- Water quality issues
- Healthcare access gaps
- Climate change impacts
**๐ก Solutions**
- Filtration systems
- Telemedicine networks
- Infrastructure adaptation
**๐ Plans**
- Political platforms
- Policy proposals
- Action roadmaps
</td>
<td width="33%">
**๐ฃ๏ธ Arguments**
- "This works because X, Y, Z"
- "This failed due to A, B"
- (We don't judgeโcommunity rates)
**๐ Data**
- Public records
- Research datasets
- Compiled statistics
**๐ญ Ideas**
- Theoretical frameworks
- Novel measurement approaches
- Conceptual models
</td>
<td width="33%">
**๐ฅ People & Orgs**
- Researchers on problems
- Organizations implementing
- Projects with outcomes
**๐ Metrics**
- Success criteria
- Measurement frameworks
- Progress indicators
**๐ฏ Values**
- Guiding principles
- Evaluation criteria
</td>
</tr>
</table>
### **How to Submit**
1. **Fork the repository** on GitHub
2. **Add your component** in the appropriate directory (`Problems/`, `Solutions/`, etc.)
3. **Follow the format** in that directory's README
4. **Submit a Pull Request**
> [!NOTE]
> We're building a **web interface** to make this easier for non-technical contributors!
**[โ Contribution Guidelines](./GETTING_STARTED.md#contributing)**
---
## ๐ Documentation
<table>
<tr>
<td width="50%">
### **Getting Started**
- ๐ [**Getting Started Guide**](./GETTING_STARTED.md) - Complete setup walkthrough
- โก [**Quick Reference**](./QUICK_REFERENCE.md) - Command cheatsheet
- ๐ [**Data Directory**](./Data/README.md) - Data philosophy & standards
- ๐ฏ [**Wellbeing Data Guide**](./Data-Sources/WELLBEING_DATA_SOURCES.md) - Implementation details
</td>
<td width="50%">
### **Technical**
- ๐๏ธ [**Project Summary**](./PROJECT_SUMMARY.md) - Architecture overview
- ๐ [**Library Science Framework**](./Data/README-LIBRARY-SCIENCE.md) - 8-dimension methodology
- ๐ [**Migration Guide**](./Data/MIGRATION-GUIDE.md) - Structure changes
- ๐ [**UPDATES.md**](./UPDATES.md) - Complete changelog
</td>
</tr>
</table>
---
## ๐บ๏ธ Roadmap
### โ
**Completed**
<details>
<summary><strong>Phase 1: Foundation (July 2024)</strong></summary>
- โ
Single-repo structure with 17+ object types
- โ
Core framework and documentation
- โ
Public launch with initial datasets
- โ
Community contribution framework
</details>
<details>
<summary><strong>Phase 2: Community (Aug-Sep 2024)</strong></summary>
- โ
Claims, Arguments, and Values frameworks
- โ
6+ community contributors
- โ
10+ merged pull requests
- โ
TELOS integration
</details>
<details open>
<summary><strong>Phase 3: Data Infrastructure (Oct 2025) ๐ฅ</strong></summary>
- โ
**13 authoritative data sources** (5 core + 8 wellbeing)
- โ
**Library science methodology** (8-dimension evaluation)
- โ
**TypeScript automation system** with Bun runtime
- โ
**6,000+ lines of documentation** across all sources
- โ
**Comprehensive wellbeing indicators** (economic, health, social, labor, environmental)
- โ
**Free API access** with rate limiting and retry logic
</details>
### ๐ง **Planned**
**Phase 4: Enhanced Access** (Q4 2025 - Q1 2026)
- ๐จ Web-based contribution interface (no coding required)
- ๐ Interactive data visualizations
- ๐ RESTful API for programmatic access
- ๐ฑ Mobile-friendly exploration
**Phase 5: Dataset Expansion** (2026)
- ๐ Additional international sources (UNICEF, OECD, IHME)
- โก Real-time data feeds integration
- ๐ณ๏ธ Community-driven dataset requests
- ๐ค Partnerships with research institutions
**Phase 6: Intelligence Layer** (2026+)
- ๐ค Automated relationship discovery
- ๐ Confidence scoring for AI-suggested links
- ๐ฏ Pattern detection algorithms
- ๐ Email/Slack notifications for data updates
- ๐ Machine-readable catalog (DCAT/CKAN compliance)
---
## ๐ Integration with TELOS
> **Substrate** provides evidence. **[TELOS](https://github.com/danielmiessler/Telos)** provides intention.
<table>
<tr>
<th width="50%">TELOS (Goals & Strategy)</th>
<th width="50%">Substrate (Evidence & Solutions)</th>
</tr>
<tr>
<td>
- **Goals** - What you want to achieve
- **Strategies** - How you'll pursue it
- **Challenges** - Obstacles you face
- **Metrics** - Progress tracking
</td>
<td>
- **Problems** - What stands in the way
- **Solutions** - Proven approaches
- **Data** - Measured evidence
- **Plans** - Implementation roadmaps
</td>
</tr>
</table>
**Together:** Complete system for intention + evidence-based action.
---
## ๐ Credits
### **Created By**
**[Daniel Miessler](https://danielmiessler.com)** โข July 2024
[](https://twitter.com/danielmiessler)
[](https://danielmiessler.com/subscribe)
### **Special Thanks**
**Inspiration & Contributions:**
- **Jonathan Dunn** [@xssdoctor](https://github.com/xssdoctor) - Similar vision and collaboration
- **Joel Parish** - Structure wisdom and guidance
- **Joseph Thacker** - Continuous flow of innovative ideas
**Community Contributors:**
[@ThatNateGuy](https://github.com/ThatNateGuy) โข [@JaymanW](https://github.com/JaymanW) โข [@karai114](https://github.com/karai114) โข [@DesertEaglePWN](https://github.com/DesertEaglePWN) โข [@ktfth](https://github.com/ktfth)
---
## ๐ License
MIT License - see the [LICENSE](LICENSE) file for details.
---
<div align="center">
### **๐ Repository Stats**
**Data:** 13 sources โข 1,700+ data points โข 107-year span (1918-2025)
**Code:** TypeScript automation โข 6,000+ lines documentation โข GitHub Actions
**Community:** 6+ contributors โข 10+ PRs merged โข 17 object types
---
**[โฌ Back to Top](#substrate)**
<sub>Built with โค๏ธ for human understanding and progress</sub>
<sub>Powered by [TypeScript](https://www.typescriptlang.org/) โข [Bun](https://bun.sh) โข Open Data</sub>
---
## Star History
[](https://star-history.com/#danielmiessler/Substrate&Date)
</div>