literature-filtering

name	literature-filtering
description	Filter literature by publication year, journal, and predefined screening rules to produce inclusion/exclusion lists; use when conducting preliminary screening or systematic review screening to narrow the literature scope.
license	MIT
author	AIPOCH

Source: https://github.com/aipoch/medical-research-skills

When to Use

• You need to quickly narrow a large bibliography by publication year range (e.g., 2015–2024).
• You must restrict results to a target journal set (e.g., a whitelist/blacklist of journals).
• You are running preliminary screening before full-text review and need traceable inclusion/exclusion decisions.
• You are conducting systematic review screening and must record consistent reasons for exclusion.
• You need standardized outputs (lists + logs) for collaboration, auditing, or downstream analysis.

Key Features

• Rule-based filtering by year, journal, and literature type/criteria.
• Journal name normalization to match abbreviations and full names consistently.
• Structured recording of exclusion reasons for transparency and reproducibility.
• Support for borderline/controversial item review to improve consistency.
• Standardized outputs: inclusion list, exclusion list, and screening statistics/summary.

Dependencies

• None (documentation-driven workflow).
• Optional template file:
  • assets/screening_log_template.csv

Example Usage

The following example is a complete, runnable Python script that:

1. normalizes journal names, 2) filters by year and journal whitelist, 3) applies simple inclusion/exclusion rules, and 4) outputs inclusion/exclusion CSV files plus a screening log.

#!/usr/bin/env python3
import csv
import re
from dataclasses import dataclass
from typing import Dict, List, Tuple

# ----------------------------
# Configuration (edit as needed)
# ----------------------------
YEAR_MIN = 2018
YEAR_MAX = 2024

# Journal whitelist after normalization
JOURNAL_WHITELIST = {
    "journal of finance",
    "journal of financial economics",
    "review of financial studies",
}

# Abbreviation/full-name mapping (extend as needed)
JOURNAL_ALIASES = {
    "j. finan.": "journal of finance",
    "j finan": "journal of finance",
    "jfe": "journal of financial economics",
    "rev. financ. stud.": "review of financial studies",
    "rfs": "review of financial studies",
}

# Simple keyword-based screening rules (example)
INCLUDE_KEYWORDS = {"asset pricing", "corporate finance", "risk premium"}
EXCLUDE_KEYWORDS = {"editorial", "book review", "erratum"}

# ----------------------------
# Data model
# ----------------------------
@dataclass
class Record:
    id: str
    title: str
    year: 
    journal: 
    abstract: 




 () -> :
    
      name:
         
    raw = name.strip().lower()
    raw = re.sub(, , raw)  
    raw = re.sub(, , raw).strip()

    
     raw  aliases:
         aliases[raw]

    
    nodot = raw.replace(, )
     nodot  aliases:
         aliases[nodot]

    
    canonical = re.sub(, , raw)
    canonical = re.sub(, , canonical).strip()
     canonical

 () -> :
    t = (text  ).lower()
     (k  t  k  keywords)

 () -> [, ]:
    
     r.year < YEAR_MIN  r.year > YEAR_MAX:
         , 

    norm_journal = normalize_journal(r.journal, JOURNAL_ALIASES)
     norm_journal   JOURNAL_WHITELIST:
         , 

    text = 
     contains_any(text, EXCLUDE_KEYWORDS):
         , 

      contains_any(text, INCLUDE_KEYWORDS):
         , 

     , 




 () -> [Record]:
    
    out = []
     (path, , newline=, encoding=)  f:
        reader = csv.DictReader(f)
         row  reader:
            out.append(
                Record(
                    =row.get(, ).strip(),
                    title=row.get(, ).strip(),
                    year=(row.get(, )),
                    journal=row.get(, ).strip(),
                    abstract=row.get(, ).strip(),
                )
            )
     out

 () -> :
     (path, , newline=, encoding=)  f:
        w = csv.DictWriter(f, fieldnames=fieldnames)
        w.writeheader()
        w.writerows(rows)

 ():
    input_path = 
    records = read_input_csv(input_path)

    included, excluded, log = [], [], []
     r  records:
        norm_journal = normalize_journal(r.journal, JOURNAL_ALIASES)
        ok, reason = screen_record(r)

        log.append({
            : r.,
            : r.title,
            : (r.year),
            : r.journal,
            : norm_journal,
            :   ok  ,
            : reason,
        })

        base = {
            : r.,
            : r.title,
            : (r.year),
            : norm_journal,
        }
        (included  ok  excluded).append(base)

    write_csv(, included, [, , , ])
    write_csv(, excluded, [, , , ])
    write_csv(
        ,
        log,
        [, , , , , , ],
    )

    
    stats = {
        : (records),
        : (included),
        : (excluded),
    }
    (, stats)
    ()

 __name__ == :
    main()

Minimal input file example (input_literature.csv):

id,title,year,journal,abstract
1,Asset Pricing with Risk Premiums,2020,J. Finan.,We study asset pricing and the risk premium...
2,An Editorial Note,2021,Journal of Finance,This editorial summarizes...
3,Corporate Finance Evidence,2017,JFE,Empirical corporate finance results...

Implementation Details

1. Rule Setting

• Year rules: define an inclusive range [YEAR_MIN, YEAR_MAX].
• Journal rules:
  • Use a whitelist (or blacklist) of canonical journal names.
  • Apply normalization before matching to avoid false mismatches.
• Screening criteria:
  • Define explicit inclusion/exclusion criteria (e.g., topic, study type, population, method).
  • Ensure each exclusion has a single primary reason (or a controlled multi-reason scheme).

2. Journal Name Normalization

Recommended normalization steps (in order):

1. Convert to lowercase.
2. Remove/standardize punctuation and collapse whitespace.
3. Apply abbreviation/full-name mapping (e.g., J. Finan. → Journal of Finance).
4. Output a canonical form used for matching and reporting.

Key parameters:

• JOURNAL_ALIASES: dictionary for abbreviation/full-name mapping.
• Normalization policy choices:
  • Case sensitivity (typically disabled by lowercasing).
  • Punctuation handling (strip most punctuation; optionally preserve dots for alias keys).
  • Whitespace collapsing.

3. Execution of Screening

• Apply filters in a stable order to keep decisions consistent and auditable:
  1. Year range
  2. Journal match (after normalization)
  3. Inclusion/exclusion criteria
• Record a decision and reason for every record in a screening log.

4. Review and Consistency

• Flag borderline items (e.g., unclear abstracts, ambiguous journal names) for manual review.
• Keep a shared, versioned rule set (year range, journal list, alias map, criteria) to ensure consistent application across reviewers.

5. Output Organization

Produce at minimum:

• included.csv: records that pass all rules.
• excluded.csv: records that fail at least one rule.
• screening_log.csv: full trace with normalized journal and exclusion reason.
• Optional: screening statistics and a reason summary (counts by reason).

Reference formats and checkpoints can be aligned with references/guide.md if available.