Design cross-national survey experiments covering power, equivalence, and localization. Use for multi-country sampling, measurement comparability, and instrument adaptation.
Design cross-national survey experiments covering power, equivalence, and localization. Use for multi-country sampling, measurement comparability, and instrument adaptation.
Cross-National Comparative Designer
Related skills. This skill composes with hypothesis-building (per-country predictions and estimands), survey-design (question wording, acquiescence, sensitivity), conjoint-design (origin-country stimuli, power), methods-reporting (per-country CONSORT, APSA/JARS/DA-RT), and pre-registration-writing (study-level PAP with per-country tiers).
Reference files.references/example-4-country-immigration.md -- worked four-country immigration-conjoint cascade covering case-selection logic, shared attributes with origin-country calibration, the full TRAPD translation protocol, per-country SESOI/predictions, and a pooled-vs-per-country analysis table.
Instructions
1. Case Selection and Theoretical Justification
Per-Country Predictions Before Case Selection (Why -> If-Then): Before finalizing the country set, articulate, for each candidate case, the specific pattern of results that would support the theoretical claim versus disconfirm it. Cross-national leverage comes from the contrast between predicted country-level patterns, not from running the same experiment in multiple places and describing what happens. If a case cannot be tied to a distinct, falsifiable prediction, it is decorative rather than diagnostic. See hypothesis-building for the Why -> If-Then funnel and estimand specification.
Variation by Design: Cases should be selected to vary on theoretically relevant dimensions (e.g., regime type, institutional structure, immigration history, partisan polarization), not for convenience or data availability. Each case should serve a specific theoretical function identified in the step above.
Case Selection Table: Produce a summary table listing each country, its relevant contextual features (e.g., immigration salience, institutional architecture, regime type), the per-country prediction, and the specific motivation for its inclusion. This table belongs in the PAP and should be referenced in the narrative.
Avoid the "Most Similar" Trap: Cross-national designs are often most informative when cases are deliberately diverse. Including cases that vary on regime type (e.g., democracy vs. hybrid regime) or institutional structure (e.g., parliamentary vs. presidential) enables stronger tests of whether a theoretical mechanism generalizes beyond a single political context.
Typical Case Count (3--5): For survey experiments with ~2,500 respondents per country, 3--5 cases is a common range in published work that balances theoretical leverage with practical feasibility (budget, translation, vendor logistics). Fewer than 3 limits comparison; more than 5 strains resources without proportional analytical gain. Treat this as a planning heuristic rather than a rule and tie the final count to the per-country predictions above.
Information Environment Variation: The information environment differs across countries -- media systems, political knowledge levels, and the accessibility of policy information vary. A treatment that is "informative" in one context may be redundant or incomprehensible in another. Document the expected information environment per country and consider how it affects treatment interpretation (Druckman 2022).
2. Instrument Localization
Conceptual Equivalence over Literal Translation: The goal is not identical wording across countries but equivalent conceptual meaning. A "skilled worker visa" means H-1B in the US, E-7 in South Korea, S Pass in Singapore. Use country-specific referents that carry the same conceptual weight. Word choice constitutes a framing treatment and must be calibrated per country, not just translated -- terms like "unauthorized," "undocumented," and "illegal" carry different connotations across linguistic and political contexts (Stantcheva 2023, citing Merolla et al. 2013). The Cross-Cultural Survey Guidelines (CCSG 2016) are the canonical source for cross-cultural comparability, including translation and adaptation procedures.
Qualitative Treatment Framing: For information experiments fielded across countries, consider using qualitative descriptions rather than exact statistics. Qualitative information "can help make the treatment more homogeneous when doing an experiment in several countries or settings" because exact numbers carry different meanings in different contexts (Stantcheva 2023).
Anti-Acquiescence Measurement: Acquiescence bias varies by culture. Use item-specific scales (not agree-disagree or true-false formats), balanced scales with equal numbers of positive and negative options, and randomized question direction to ensure measurement equivalence across countries (Stantcheva 2023).
Coarse Measurement as Principled Commitment: When implementing the same conceptual measure across diverse contexts, accept that some precision is lost in translation. Treat this as a principled commitment to comparability rather than a limitation -- coarse measurement that is consistent across contexts is preferable to precise measurement that means different things in different places (Sniderman 2018).
Institutional Referents: Localize all political actors (president, cabinet, ministry), legislative bodies (Congress, Bundestag, National Assembly, Parliament), international organizations (UN, EU, OECD, ASEAN), and policy instruments (visa types, permit categories). Document all localizations in a country-specific table.
Cultural Calibration of Stimuli: For attributes like "country of origin" that signal cultural proximity or distance, the specific countries must be calibrated per respondent country. "Culturally distant" means different things in the US (e.g., Somalia) than in South Korea (e.g., Yemen) or Singapore (e.g., Afghanistan). Document the calibration rationale.
Ecological Validity Check: Every experimental stimulus should be plausible in the respondent's national context. A vignette about EU directives makes sense for German respondents but not Singaporean ones. Test each localized version against the question: "Would a newspaper in this country plausibly report this?" Note that being "outside the lab" does not automatically confer ecological validity -- it requires deliberate calibration of stimuli to real-world conditions in each country (Mutz 2011). Where feasible, benchmark stimuli against real-world behavioral outcomes in at least one case; Hainmueller, Hangartner, & Yamamoto (2015) validate immigration-conjoint preferences against actual Swiss naturalization referendum votes, demonstrating that vignette and conjoint designs can recover behaviorally meaningful effects when localized carefully.
3. Composition and Origin Country Selection
Origin Countries as Group Cues: When immigrant origin countries are used as experimental stimuli (common in immigration conjoint studies; see Hainmueller, Hopkins, & Yamamoto 2014 for the foundational AMCE framework and its original application to immigration attitudes), select origins that: (a) are recognizable to respondents, (b) vary on the intended dimension (e.g., cultural proximity), and (c) do not carry overwhelming confounding associations (e.g., ongoing war, recent political crisis) unless those associations are part of the theoretical design. See conjoint-design for the attribute-architecture rules that govern origin-country levels.
Nesting Origins within Domains: If the experiment crosses policy domain (e.g., labor vs. asylum) with immigrant composition, it may be necessary to use different origin countries for different domains within the same respondent country. This preserves ecological validity (asylum seekers plausibly come from different countries than labor migrants) but means the composition attribute cannot be cleanly separated from domain. Document this nesting and acknowledge it in the analysis.
Avoiding Origin Overlap: Within a single respondent country, avoid reusing the same origin country across different experimental roles (e.g., using Nigeria as both the "distant" labor origin and the "distant" asylum origin). This prevents respondents from noticing repeated countries and developing response patterns.
4. Cross-National Power and Error Management
Per-Country Power Analysis: Conduct power analyses separately for each country, not just for the pooled sample. Country-specific samples (typically 1,500--2,500) may be underpowered for detecting small AMCEs or interaction effects even when the pooled analysis is well-powered.
Compromise Power for Small-N Countries: When country-specific sample sizes are fixed by budget constraints, use a compromise power analysis that minimizes the combined Type I + Type II error rate. An alpha > 0.05 may be justified for country-level analyses where the per-country N is small but fixed (Lakens 2025).
Context-Specific SESOI: The smallest meaningful effect size may differ by country. A 3-percentage-point AMCE for a policy attribute may be substantively important in a country where the policy is highly salient but trivial where it is not. Document the SESOI rationale per country when it varies.
Minimum Effect Tests for Pooled Data: When pooling data across countries, large pooled N makes it easy to find "significant" trivial correlations. Use minimum effect tests (testing whether the effect exceeds a meaningful threshold, not just whether it differs from zero) to ensure that pooled results are practically significant (Lakens 2025).
Sequential Factorials Across Countries: In multi-experiment, multi-country designs, consider a "sequential factorial" approach where the factorial structure builds across experiments while the country set remains constant. This enables cumulative theoretical leverage: Experiment 1 establishes baseline effects across countries; Experiment 2 "splices in" additional factors to probe mechanisms (Sniderman 2018).
5. Cross-National Analytical Strategy
Per-Country Estimation First: Always estimate the primary model separately for each country before pooling. This respects the possibility that the data-generating process differs across contexts and provides country-specific effect sizes that are interpretable on their own. Name the per-country estimand explicitly (country-specific AMCE, marginal mean, or subgroup AMCE on the same respondent population) rather than leaving the target quantity implicit (Lundberg, Johnson, & Stewart 2021).
Pooled Models with Country Interactions: For formal tests of cross-national heterogeneity, estimate a pooled model that includes attribute $\times$ country interaction terms. Wald tests on the interaction coefficients provide a statistical test of whether effect sizes differ across countries. Note that respondent fixed effects absorb country main effects when country is a respondent-level constant.
Scope Conditions, Not Country Hypotheses: When the theory predicts that effect magnitudes will vary by institutional context but does not make precise predictions about which country will show the largest effect, frame country-specific expectations as "contextual expectations and scope conditions" rather than confirmatory hypotheses. This avoids the proliferation of underpowered country-level tests while preserving theoretical nuance.
Response Style Biases: Moderacy, extreme response, and response order biases vary systematically across cultures and socioeconomic groups. Accounting for these biases "may be particularly important in cross-country studies" because response tendencies differ across cultural contexts (Stantcheva 2023). Assess response style distributions per country and consider whether they distort cross-national comparisons.
Sensitivity Bias by Context: Social desirability bias may vary systematically across countries with different political regimes and norms of expression. Blair et al. (2020) provide domain-specific estimates of sensitivity bias -- use these to assess whether direct-question measures are trustworthy per country or whether indirect measurement (e.g., list experiments) is needed in some contexts but not others.
Visualization: Present cross-national results as side-by-side AMCE panels (one per country) or forest plots. These enable visual comparison of effect magnitudes and patterns, which is often more informative than formal statistical tests of heterogeneity.
6. Practical Considerations
Survey Vendors: Cross-national online surveys typically require country-specific vendors or an international panel provider. Verify that quota-matching (age, gender, education, region) is available in all target countries. Note whether samples are probability-based or nonprobability panels and the implications for PATE estimation in each country (Mutz 2011). For developing or middle-income country cases, acknowledge that online samples are "online representative" rather than nationally representative (Stantcheva 2023) and discuss implications for generalizability.
Survey Mode Consistency: Maintain consistent survey mode across countries (all online, all CATI, etc.) to avoid confounding treatment effects with mode effects. If mode must vary by country, document this and assess its impact.
Cross-National Piloting: Pilot the localized instrument in each country, not just one. Pilots should test whether localized stimuli function equivalently -- whether respondents access, attend to, and interpret the information as intended across contexts (Druckman 2022). Pretesting is the "P" of the TRAPD protocol and should surface translation-review issues that earlier stages missed before the instrument is locked (CCSG 2016).
Translation Protocol: Follow the TRAPD team-translation model from the Cross-Cultural Survey Guidelines (CCSG 2016) -- Translation (two or more independent translators produce parallel drafts), Review (translators and reviewers discuss drafts together), Adjudication (a designated adjudicator resolves disputes and decides when the version is ready), Pretesting (cognitive interviews or pilots in each target language before fielding), and Documentation (logged decisions at each step for quality assurance). Simple back-translation alone is not sufficient; experimental stimuli in particular benefit from a reviewer who understands the experimental design, not just the language. See references/example-4-country-immigration.md for a worked TRAPD walkthrough.
Ethical Approvals: Some countries or institutions require separate ethics review. Plan for varying timelines and requirements across sites.
Data Sharing Across Privacy Regimes: Cross-national studies involve data from multiple countries with potentially different privacy regimes (e.g., GDPR in Europe). Plan for data sharing, archiving, and de-identification given varying legal and ethical constraints across national contexts; Christensen, Freese, & Miguel (2019) discuss general open-science and data-sharing principles that apply, though region-specific legal guidance (e.g., GDPR, cross-border transfer rules) should be consulted separately.
Pre-Registration: Pre-register at the study level (not per-country). The pre-analysis plan should include both the pooled analytical strategy and the per-country estimation plan, with per-country predictions locked before fielding. See for PAP structure and for per-country CONSORT/attrition reporting.
Quality Checks
Per-Country Predictions: Does each country have a distinct, pre-registered prediction whose pattern would support or disconfirm the theoretical claim?
Case Justification: Is each country's inclusion theoretically motivated and documented in a case selection table?
Localization Documented: Are all country-specific referents (actors, institutions, visa types, origin countries) listed in a localization table?
Conceptual Equivalence: Do localized stimuli carry equivalent conceptual meaning across countries, even if wording differs?
Ecological Validity: Is every stimulus plausible in each national context?
Origin Calibration: Are origin countries calibrated for cultural proximity/distance per respondent country, with rationale documented?
No Origin Overlap: Within each respondent country, is each origin country used in only one experimental role?
Per-Country Power: Has a per-country power analysis been conducted, with MDE reported for each country?
Per-Country Models: Does the analysis plan include per-country estimation before pooling?
Scope Conditions: Are cross-national expectations framed as scope conditions rather than confirmatory hypotheses (unless the theory makes precise directional predictions)?
Sensitivity Audit: Has the risk of social desirability bias been assessed per country, with indirect measurement considered where needed?
Mode Consistency: Is the survey mode consistent across countries, or are mode differences documented?
Cross-National Piloting: Has the localized instrument been piloted in each country?
Translation: Is a TRAPD team-translation protocol specified? (Back-translation alone is not sufficient -- see Translation Protocol above.)
pre-registration-writing
methods-reporting
Firehouse Readiness: Internet-based panels make cross-national experiments feasible for rapid-response "firehouse studies" after international events. Having pre-established vendor relationships in multiple countries reduces deployment time (Mutz 2011).
Response Style Biases: Have moderacy, extreme response, and acquiescence biases been assessed across countries?
Online Representativeness: For developing/middle-income country samples, is the "online representative" limitation acknowledged?
Treatment Homogeneity: Are information treatments designed to maintain equivalence across countries (e.g., qualitative rather than quantitative framing where appropriate)?
Word Choice Calibration: Has terminology been tested for differential framing effects across languages?
