doris-debug-data-lake
Use for Doris external catalog issues: Hive/Iceberg/Paimon/Hudi query failures, metadata refresh, filesystem S3/HDFS connectivity, external MV rewrite misses.
소스 정보
- 저장소
- apache/doris-skills
- 최근 소스 활동
- 2026년 7월 24일 23:41
- 감지된 SKILL.md 언어
- 영어
- 스타
- 46
- 포크
- 15
설치 방법
기본적으로 소스를 먼저 확인하는 Prompt가 선택됩니다. 직접 명령으로 전환하거나 로컬 사본을 다운로드할 수도 있습니다.
소스 파일 검토
설치 여부를 결정하기 전에 SKILL.md와 SkillsMP에 표시된 보조 파일을 읽어 보세요.
파일 탐색기
2 개 파일SKILL.md 표시 중
SKILL.md
소스 지침 · 읽기 전용 미리보기- name
- doris-debug-data-lake
- description
- Use for Doris external catalog issues: Hive/Iceberg/Paimon/Hudi query failures, metadata refresh, filesystem S3/HDFS connectivity, external MV rewrite misses.
- version
- 0.2.0
- category
- data-lake
# Data Lake (Multi-Catalog)
## Causes
| ID | Cause | Evidence | Source anchor |
|----|-------|----------|---------------|
| A | Metadata stale | Query returns old data / missing partitions after upstream write | `HMSClient.java`, `CachedHMSClient.java` |
| B | S3/HDFS connectivity | `Access Denied`, `Timeout`, `NoSuchBucket` | `S3FileSystem.java`, `HdfsResource.java` |
| C | External MV rewrite miss | EXPLAIN shows no MATERIALIZED_REWRITE | `MTMVService.java` rewrite rules |
| D | Schema mismatch | Parquet/ORC schema ≠ Hive Metastore schema | `ParquetReader.cpp`, schema merge |
| E | Credential / IAM expiry | STS / IAM token rotated; 403 on S3 | `S3FileSystem.java` credential refresh |
| F | Too many open files | FD exhaustion listing large table directories | BE `ulimit -n`, `FileSystem.listFiles()` |
## 10 min triage
```sql
-- Verify catalog is alive
SHOW CATALOGS;
-- Refresh metadata (hive / iceberg / paimon)
REFRESH CATALOG hive_catalog;
REFRESH DATABASE hive_catalog.db_name;
REFRESH TABLE hive_catalog.db_name.tbl_name;
-- Check if external MV rewrite is enabled
SET materialized_view_rewrite_enable_contain_external_table = true;
-- Analyze external table for CBO stats
ANALYZE TABLE hive_catalog.db_name.tbl_name;
```
```bash
# BE logs for S3/HDFS errors
./scripts/doris-debug log-grep be/log --query-id "$QID"
grep -r "403\|Access Denied\|Timeout\|NoSuchBucket\|Token.*expired" be/log/
# Check BE file descriptor limit
cat /proc/$BE_PID/limits | grep "open files"
```
## Cause A — Metadata staleness
Hive Metastore metadata is cached; `REFRESH CATALOG` invalidates the cache. For large catalogs, prefer:
```sql
-- Refresh only the changed partition
REFRESH TABLE catalog.db.tbl PARTITION (dt='2026-07-15');
```
If refresh is slow, check HMSClient cache settings (`fe.conf`):
```properties
hive_metastore_client_timeout_second = 10
```
## Cause B — S3/HDFS connectivity
```properties
# be.conf — S3 credential chain
aws_access_key_id = ...
aws_secret_access_key = ...
aws_region = us-east-1
```
```bash
# Test S3 reachability from BE host
curl -I "https://<bucket>.s3.<region>.amazonaws.com"
```
```properties
# fe.conf for HDFS catalog
hadoop_conf_dir = /path/to/hadoop/conf # must contain core-site.xml + hdfs-site.xml
```
## Cause E — Credential rotation
For STS/AssumeRole-based S3 access, the cached credential may expire:
```sql
-- Force credential refresh (Doris 2.1+ / 3.0)
REFRESH CATALOG hive_catalog;
```
For long-running queries against external tables, ensure credential TTL > query timeout.
## Source
- `fe/.../datasource/hive/HMSExternalCatalog.java`
- `fe/.../datasource/hive/HMSClient.java`
- `be/src/io/fs/s3/S3FileSystem.cpp` — S3 connector
- `be/src/io/fs/hdfs/HdfsFileSystem.cpp` — HDFS connector
- `be/src/vec/exec/format/parquet/vparquet_reader.cpp` — parquet reader
- `fe/.../mtmv/MTMVService.java` — external MV rewrite
GitHub에서 보기