retire-add-ast-query

name

retire-add-ast-query

description

Add an AST query for a JavaScript library to the retire.js vulnerability database. Given a library name, downloads multiple versions, analyzes the code structure where the version string is assigned, writes an ASTronomical XPath-like query anchored with library-specific identifiers, tests the query locally, adds it to jsrepository-ast.js, and runs validate/test-detection. Use this skill whenever adding an AST-based version detection query to retire.js.

retire-add-ast-query

This skill adds an ASTronomical query for a JavaScript library to the retire.js AST detection system. It downloads real files from CDNs, reads the code structure around version assignments, builds a query anchored to library-specific identifiers, tests it locally with astronomical, and validates with test-detection.js.

The repo is at /Users/erlend/code/github/RetireJS/retire.js. All file edits and validation commands run there.

Overview

AST queries detect the library version by traversing the JavaScript AST rather than matching raw text. Queries must be anchored to library-specific identifiers on sibling or parent nodes — otherwise they'll fire on unrelated files that happen to have a version assignment. The query returns the version string directly (no §§version§§ placeholder needed — the Literal node's value is the result).

All queries live in repository/jsrepository-ast.js (a CommonJS module). After editing it, run ./convertToVersioned to propagate to the versioned JSON files used at runtime.

Step 1 — CDN Lookup

Fetch library metadata from cdnjs:

https://api.cdnjs.com/libraries/{NAME}?fields=name,versions,filename,description

or jsdelivr if not on cdnjs:

https://data.jsdelivr.com/v1/packages/npm/{NAME}

Note the filename field (minified name) and derive the non-minified name (strip .min). Record the full list of versions.

Step 2 — Select Versions

Pick 4–5 versions spread across history: 1 early (not the very first), 2–3 middle, and the latest.

Step 3 — Download Files

mkdir -p /tmp/retire-ast-{NAME}
# For each VERSION:
curl -sL "https://cdnjs.cloudflare.com/ajax/libs/{NAME}/{VERSION}/{MIN_FILENAME}" \
  -o /tmp/retire-ast-{NAME}/{VERSION}.min.js
curl -sL "https://cdnjs.cloudflare.com/ajax/libs/{NAME}/{VERSION}/{UNMIN_FILENAME}" \
  -o /tmp/retire-ast-{NAME}/{VERSION}.js

Verify each downloaded successfully (non-zero, not an HTML 404 page):

ls -la /tmp/retire-ast-{NAME}/
head -3 /tmp/retire-ast-{NAME}/{VERSION}.js

If cdnjs returns 404, fall back to jsDelivr:

curl -sL "https://cdn.jsdelivr.net/npm/{NAME}@{VERSION}/dist/{FILENAME}" \
  -o /tmp/retire-ast-{NAME}/{VERSION}.js

Step 4 — Find Version Strings

Grep each downloaded file for the exact version number:

grep -n "1\.2\.3" /tmp/retire-ast-{NAME}/{VERSION}.js
grep -n "1\.2\.3" /tmp/retire-ast-{NAME}/{VERSION}.min.js

For single-line minified files where grep context flags fail, use Python:

python3 -c "
import re
with open('/tmp/retire-ast-{NAME}/{VERSION}.min.js', 'r', errors='replace') as f:
    content = f.read()
for m in re.findall(r'.{0,100}1\.2\.3.{0,100}', content)[:5]:
    print(m)
"

Step 5 — Understand the Code Pattern

Read 10–20 lines around each version occurrence in non-minified files:

grep -n "1\.2\.3" /tmp/retire-ast-{NAME}/{VERSION}.js
# Then:
sed -n '${LINE_MINUS_10},${LINE_PLUS_10}p' /tmp/retire-ast-{NAME}/{VERSION}.js

Identify two things:

1. What node holds the version: e.g. x.VERSION = "1.2.3" (AssignmentExpression → right → Literal), {version: "1.2.3"} (ObjectExpression → Property → value → Literal), var version = "1.2.3" (VariableDeclarator → init → Literal)
2. Library-specific anchors nearby: an identifier that uniquely identifies this library. The anchor must appear in the same block/expression as the version assignment — or inside a method body of the same object.

Anchor quality, best to worst:

• Gold: an identifier that contains the library name — e.g. tinyMCEPreInit, isMoment, HandlebarsEnvironment, __SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED. These cannot match any other library by definition.
• Good: a highly domain-specific sibling property or method that wouldn't appear together with a version string in any other library — e.g. migrateVersion, focusedEditor combined with majorVersion.
• Weak: generic property names like releaseDate, activeEditor, major, init, build. These could plausibly appear in other libraries and should only be used in combination with other anchors, never alone.

Look inside method bodies too. If the version is in an object literal, scan the body of methods on that same object for library-name identifiers — e.g. setup() { ... window.tinyMCEPreInit ... } inside the same ObjectExpression. A //MemberExpression or //Identifier traversal into a method body provides a much stronger anchor than any generic sibling property name.

Ask yourself: would this query fire on a file from an unrelated library that happens to assign a version string? If you're not confident the answer is no, strengthen the anchor.

Step 6 — Write a Candidate Query

Use ASTronomical syntax to navigate from the root to the version Literal, filtering with library-specific anchors.

ASTronomical syntax quick reference

Syntax	Meaning
//NodeType	Descendant at any depth
/NodeType	Direct child
/:attr	Named attribute of current node
/$:name	Resolve identifier binding (follow variable reference)
/$$:name	Binding or attribute fallback (when value may be variable or literal)
[cond]	Filter; use &&, ||, ==
../	Go to parent (use sparingly — causes extra traversal)
/fn:concat(sel1, ".", sel2)	Concatenate results (for major.minor.patch)

Common patterns

Named property assignment in a block with a sibling anchor:

//BlockStatement[
  /ExpressionStatement/AssignmentExpression[/:left/:property/:name == "ANCHOR_PROP"]
]/ExpressionStatement/AssignmentExpression[
  /:left/:property/:name == "version"
]/$$:right/:value

Named object property with a co-occurring sibling property:

//ObjectExpression[
  /Property/:key/:name == "ANCHOR_PROP"
]/Property[/:key/:name == "version"]/:value/:value

Assignment where the object is named directly:

//AssignmentExpression[
  /:left/:object/:name == "LibName" &&
  /:left/:property/:name == "VERSION"
]/:right/:value

Two properties sharing the same object binding (stronger anchor):

//SequenceExpression[
  /AssignmentExpression[/:left/:property/:name == "ANCHOR_PROP"]/:left/$:object ==
  /AssignmentExpression[/:left/:property/:name == "version"]/:left/$:object
]/AssignmentExpression[/:left/:property/:name == "version"]/$$:right/:value

Object with library-name identifier inside a method body (gold anchor):

//ObjectExpression[
  /Property[/:key/:name == "majorVersion"] &&
  /Property[/:key/:name == "setup"]//MemberExpression[/:property/:name == "libraryNamePreInit"]
]/fn:concat(
  /Property[/:key/:name == "majorVersion"]/:value/:value,
  ".",
  /Property[/:key/:name == "minorVersion"]/:value/:value
)

Start simple, add anchors until the query is specific enough. Multiple queries in the array are OR'd — add a second entry for a different version era if the code structure changed across major versions.

Step 7 — Test the Query Locally

Create a test script:

// /tmp/test-ast-query.js
const { multiQuery } = require('/Users/erlend/code/github/RetireJS/retire.js/node/node_modules/astronomical');
const fs = require('fs');
const code = fs.readFileSync(process.argv[2], 'utf8');
const query = `YOUR QUERY HERE`;
const results = multiQuery(code, { version: query });
console.log(JSON.stringify(results.version, null, 2));

Run against each downloaded version:

node /tmp/test-ast-query.js /tmp/retire-ast-{NAME}/{VERSION}.js
node /tmp/test-ast-query.js /tmp/retire-ast-{NAME}/{VERSION}.min.js

Expected output: a JSON array containing exactly the version string, e.g. ["1.2.3"]. If the output is [], the query didn't match — revise it. If it returns multiple values, the anchor is too loose.

Iterate across all tested versions until each returns the correct version string. If an early version has a structurally different pattern, write a second query string.

Step 8 — Add to jsrepository-ast.js

Edit repository/jsrepository-ast.js. Add the library key to exports.queries (in alphabetical order is nice but not required):

"library-name": [
  /*
  Code pattern this query matches, e.g.:
    LibName.isMoment = function; LibName.version = "1.2.3"
  */
  `//SequenceExpression[
    /AssignmentExpression[/:left/:property/:name == "isMoment"]/:left/$:object ==
    /AssignmentExpression[/:left/:property/:name == "version"]/:left/$:object
  ]/AssignmentExpression[/:left/:property/:name == "version"]/$$:right/:value`,
],

If the library already has an entry (added by someone else), append to its array rather than replacing it.

Step 9 — Update testcases.json if needed

If the library is already in repository/testcases.json, check if any of the tested versions have a different code structure that the query can't reach. Add allowAstMiss for those versions:

"library-name": {
  "https://cdnjs.cloudflare.com/.../§§version§§/...§§subversion§§.js": {
    "versions": ["1.0.0", "2.0.0", "3.0.0"],
    "subversions": ["", ".min"],
    "allowAstMiss": ["1.0.0"]
  }
}

If the library is not yet in testcases.json, add a minimal entry covering the 3–5 tested versions. Without a testcases.json entry, test-detection.js won't exercise the AST query.

Step 10 — Validate and Test

cd /Users/erlend/code/github/RetireJS/retire.js/repository
./convertToVersioned && ./validate
node test-detection.js {LIBRARY_NAME}
node test-detection.js          # full suite — catch any false positives introduced

Fixing failures:

• validate error: check jsrepository-ast.js syntax — it's a JS module, not JSON; look for unclosed template literals or missing commas
• Did not detect ... using ast: query doesn't match that version — refine the query (back to Step 7) or add allowAstMiss for that version
• Detect multiple components using ast: query is too broad — strengthen the anchor (back to Step 6)

Step 11 — Report Results

Show:

• The final query string(s) added to jsrepository-ast.js
• Which versions were successfully detected via AST
• The timing output from test-detection.js (C: content ms, B: AST ms per version)