| name | semantic-search-cwicr |
| description | Semantic search in the DDC CWICR construction cost database using vector embeddings (BGE-M3, 1024-dim, per-language Qdrant collections). Find similar work items and resources for cost estimation across 8 national bases and 30 markets in 26 languages. |
| homepage | https://datadrivenconstruction.io |
| metadata | {"openclaw":{"emoji":"🗄️","os":["darwin","linux","win32"],"homepage":"https://datadrivenconstruction.io","requires":{"bins":["python3"],"env":["QDRANT_URL"]},"primaryEnv":"QDRANT_URL"}} |
Semantic Search in DDC CWICR Database
Business Case
Problem Statement
Construction cost estimation requires finding relevant work items from large databases. Traditional keyword search fails when:
- Users describe work in natural language
- Terminology varies across regions and languages
- Similar work items have different naming conventions
Solution
DDC CWICR provides pre-computed embeddings (BAAI/bge-m3, 1024 dimensions) enabling multilingual semantic search across 8 national bases (78,228 positions) plus the 30-market global base in 26 languages, with 48 PPP-repriced market catalogs per national base.
Business Value
- 90% faster work item lookup compared to manual search
- Multi-language: Arabic, Bulgarian, Chinese, Croatian, Czech, Danish, Dutch, English, Finnish, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Mongolian, Norwegian, Polish, Portuguese, Romanian, Russian, Spanish, Swedish, Thai, Turkish, Vietnamese
- Higher accuracy by finding semantically similar items, not just keyword matches
Data landscape (2026)
| National base | Region id | Positions |
|---|
| Turkey (Birim Fiyat) | TR_NATIONAL | 22,704 |
| China (Beijing Dinge + Bole) | ZH_CHINA | 11,312 |
| Brazil (SINAPI) | BR_NATIONAL | 9,723 |
| Spain (BCCA Andalucía) | ES_ANDALUCIA | 6,453 |
| Italy (Prezzario Toscana) | IT_TOSCANA | 5,836 |
| Vietnam (Dinh Muc) | VN_NATIONAL | 4,299 |
| Indonesia (AHSP) | ID_NATIONAL | 2,784 |
| Greece (GGDE) | GR_NATIONAL | 2,647 |
Each base ships the 95-column CWICR master schema (rate_code, rate_original_name, rate_final_name, rate_unit, total_cost_per_position, classification hierarchy collection/department/section/subsection/category, resource_* component lines with is_material/is_machine/is_labor flags) plus 26 language editions and 48 markets/*.csv catalogs.
Latest data release: v0.4.0 (see releases).
Technical Implementation
Prerequisites
pip install qdrant-client pandas sentence-transformers
Collections (2026)
The vector store uses BAAI/bge-m3 (1024-dim dense + sparse + colbert in one forward pass, MIT license, 100+ languages). Production collections are named cwicr_{LANG}_v3 (e.g. cwicr_tr_v3, cwicr_zh_v3). An ONNX-int8 variant (gpahal/bge-m3-onnx-int8, ~700 MB) is used on VPS-sized hosts.
Python Implementation
import pandas as pd
from qdrant_client import QdrantClient
from sentence_transformers import SentenceTransformer
class CWICRSemanticSearch:
def __init__(self, host="localhost", port=6333, lang="en"):
self.client = QdrantClient(host=host, port=port)
self.collection = f"cwicr_{lang}_v3"
self.model = SentenceTransformer("BAAI/bge-m3")
def search_work_items(self, query, limit=10):
vec = self.model.encode(query).tolist()
hits = self.client.search(
collection_name=self.collection,
query_vector=vec,
limit=limit,
)
return pd.DataFrame([{**h.payload, "score": h.score} for h in hits])
def search_by_category(self, query, category, limit=10):
vec = self.model.encode(query).tolist()
hits = self.client.search(
collection_name=self.collection,
query_vector=vec,
query_filter={"must": [{"key": "category", "match": {"value": category}}]},
limit=limit,
)
pd.DataFrame([{**h.payload, : h.score} h hits])
Inside OpenConstructionERP
The platform's costs module already exposes semantic matching:
POST /api/v1/costs/suggest-for-element — rank cost items for a BIM element body./qdrant-search — multilingual candidate retrieval for a query.- The SQL fallback (
GET /api/v1/costs/?q=...) works without Qdrant.
Database Schema (95-column master)
Key fields the payload carries:
| Field | Type | Description |
|---|
rate_code | string | Unique work item code (e.g. 15.115.1008) |
rate_original_name | string | Source-language description |
rate_final_name | string | Display/translated description |
rate_unit | string | m², m³, m, kg, Ad, Sa… |
total_cost_per_position | float | Total unit price |
total_resource_cost_per_position | float | Resource sum (before markup) |
collection_name / department_name / section_name / subsection_name | string | Classification hierarchy |
category_type | string | Normalized category (e.g. CONSTRUCTION WORK) |
resource_name / resource_quantity / resource_price_per_unit_current / resource_cost | mixed | Component lines |
is_material / is_machine / is_labor | bool | Component nature flags |
Usage Examples
Basic Search
search = CWICRSemanticSearch(lang="tr")
results = search.search_work_items("tuğla duvar örülmesi")
print(results[["rate_code", "rate_original_name", "total_cost_per_position", "score"]])
Cost Estimation
foundation = search.search_work_items("reinforced concrete foundation", limit=20)
quantities = {"15.115.1008": 150.0}
total = sum(quantities[c] * row["total_cost_per_position"]
for _, row in foundation.iterrows() if row["rate_code"] in quantities)
print(f"Estimated: {total:,.2f} TRY")
Best Practices
- Use specific queries - "reinforced concrete slab 200mm" beats "concrete"
- Filter by category - Narrow results to relevant work types
- Check similarity scores - Low scores need manual verification
- Combine with QTO - Use BIM quantities for automated estimation
- Mind the coefficient bases - Vietnam and Indonesia have no prices (rate 0); price them via a market resource sheet
- Trust the source column -
rate_original_name holds the source wording; translations live in rate_final_name
Resources
