ukbsci-preprocess

name

ukbsci-preprocess

description

Variable cleaning and feature construction for UK Biobank Research Analysis Platform (RAP) cohorts using UKBAnalytica. Negative-code sanitation (ukb_clean_missing), unified preprocessing with predefined variable sets (preprocess_baseline, prepare_analysis_dataset), composite variable builders (calculate_blood_pressure, calculate_air_pollution, calculate_diet_score), and curated variable-set lookups (get_variable_set, get_variable_sets, get_variable_info). Use this skill when the user asks to clean UKB negative-codes (-1, -3, -7, …), derive blood pressure / air pollution / diet score, compose an analysis-ready table, or look up predefined UKB variable sets. Triggers: UKB preprocessing, variable cleaning, derive BP, air pollution exposure, diet score, negative code, UKB 变量预处理, 缺失码处理, 复合变量, /ukbsci-preprocess. Hard rule: local agents must not read or inspect real UKB RAP participant-level data; generate scripts for RAP execution and interpret aggregate outputs only.

ukbsci-preprocess — UKB variable cleaning & composite-feature builders

0. RAP guardrails

Strict local-agent boundary: this skill is for script generation, workflow planning, package guidance, and interpretation of aggregate outputs. The agent must not read, inspect, summarize, or process real UK Biobank RAP participant-level data, including de-identified row-level tables, raw RAP fields, exact dates, per-row predictions, row-level SHAP matrices, screenshots, tracebacks, or logs containing row-level values. Generate scripts for the user to run inside RAP; only aggregate results or rendered figures may be shared back with the agent. See ../references/agent-privacy-boundary.md.

The cleaned cohort is participant-level. Stay inside /mnt/project/.... Summary statistics, missingness counts, and pre-vs-post sanity tables are aggregate and may be shared with the local agent.

---

1. When to load

• Convert UKB negative sentinel codes (-1, -2, -3, -7, -9, -10, -11, -13, -15) to NA.
• Build composite variables: averaged blood pressure, averaged air pollution, diet score.
• Apply a predefined variable set (e.g. clinical_core) to a freshly extracted RAP table.
• Wrap up preprocessing + optional imputation in one call (prepare_analysis_dataset).

2. When NOT to load

• Pulling fields off RAP → ukbsci-rap-extract.
• Multiple imputation (more than the optional flag here) → ukbsci-imputation.
• Defining disease cases / survival outcome → ukbsci-cohort.

---

3. Prerequisites

library(UKBAnalytica); library(data.table)
stopifnot(data.table::is.data.table(dt))

---

4. Pipeline

Phase 1 — Browse predefined variables

get_variable_info(category = "all")         # 9 categories supported
get_variable_info(category = "biomarkers")
get_variable_sets(category = "lifestyle")    # all sets, with field IDs
get_variable_set("clinical_core", output = "field_id")  # int vector

The catalogues cover: demographics, anthropometrics, lifestyle, socioeconomic, blood_pressure, medications, biomarkers, pollution, diet.

Phase 2 — Sanitize negative codes

dt <- ukb_clean_missing(
  dt,
  cols          = NULL,              # NULL = scan all numeric cols
  invalid_codes = c(-1, -3, -7, -9, -11, -13, -15)
)

This is the first transformation. Run it before computing composite variables — otherwise the means / scores are silently corrupted by sentinels.

Phase 3 — Build composite variables

# Mean SBP / DBP across two readings; falls back to manual if automated missing
dt <- calculate_blood_pressure(dt, type = "sbp", prefer = "auto")
dt <- calculate_blood_pressure(dt, type = "dbp", prefer = "auto")

# Average air pollution across time points
dt <- calculate_air_pollution(dt, pollutants = c("NO2","PM10","PM2.5","NOx"))

# Healthy diet score 0–7
dt <- calculate_diet_score(
  dt,
  components  = c("fruit","vegetable","fish","meat","cereal","milk"),
  na_handling = "strict"           # "strict" = NA in any component → NA score
)

Phase 4 — Unified call (preprocess_baseline)

For routine workflows:

dt <- preprocess_baseline(
  dt,
  variables       = c("age","sex","bmi","sbp","dbp","ldl_cholesterol",
                       "hba1c","crp","smoking_status","alcohol_intake"),
  custom_mapping  = NULL,
  missing_action  = "keep",        # "drop" filters rows with any NA
  invalid_codes   = c(-1, -3)
)

preprocess_baseline() understands the same variable names as get_variable_info(); passing a name not in the catalogue raises an error.

Phase 5 — Optional imputation

imp <- run_imputation(
  dt,
  variables = c("age","sex","bmi","sbp","ldl_cholesterol","hba1c"),
  m = 5,
  method = "pmm",
  seed = 1234
)

For full multiple-imputation model fitting and pooling, use ukbsci-imputation.

---

5. Common pitfalls

1. Order matters. Always ukb_clean_missing() before any composite variable; otherwise sentinel codes contaminate the means.
2. calculate_diet_score() rounds to one decimal place in some versions. If you need higher precision for downstream Cox, scale the raw components yourself.
3. prefer = "manual" in calculate_blood_pressure() swaps the precedence — handy if your cohort has more manual readings than automated.
4. na_handling = "partial" in calculate_diet_score() averages available components and scales — useful but introduces a different missingness mechanism. Document it explicitly.
5. missing_action = "drop" in preprocess_baseline() removes any row with any NA in the listed variables. Confirm this is what the user wants, especially for high-dimensional feature sets.
6. Custom mappings. custom_mapping = list(my_alias = "p123_i0") lets you rename without touching the catalogue. Useful for non-standard fields (e.g. derived imaging metrics).
7. Predefined variables can drift. When UKB releases new fields, the catalogue is updated. Run get_variable_info() interactively before freezing scripts.

---

6. Key functions

Function	Purpose
ukb_clean_missing(df, cols = NULL, invalid_codes = c(-1,-3,-7,-9,-11,-13,-15))	Replace sentinel codes with NA
calculate_blood_pressure(df, type = c("sbp","dbp"), prefer = c("auto","manual"))	Mean BP across readings
calculate_air_pollution(df, pollutants = c("NO2","PM10","PM2.5","NOx"))	Time-averaged exposures
calculate_diet_score(df, components = c("fruit","vegetable","fish","meat","cereal","milk"), na_handling = c("strict","partial"))	0–7 healthy diet score
preprocess_baseline(df, variables, custom_mapping = NULL, missing_action = c("keep","drop"), invalid_codes = c(-1,-3))	Unified pipeline using predefined variable map
get_variable_info(category = "all")	Catalogue rows
get_variable_set(set, output = c("data.frame","field_id","ukb_col"))	One set
get_variable_sets(set = NULL, category = NULL)	All sets with filter

See references/variable-catalog.md for the 9 categories and example variables.

---

7. Related skills

Skill	When
ukbsci-rap-extract	Get the raw fields first.
ukbsci-cohort	Build outcomes from the cleaned table.
ukbsci-baseline	Table 1 of the cleaned cohort.
ukbsci-imputation	Full MI pooling (vs. the one-shot option here).

---

8. References

• references/functions.md
• references/variable-catalog.md — the 9 categories and what's in each
• references/examples.md