hn-summarize

# HN Summarize `hckrnews.com` is a JavaScript-rendered front end - curling it returns an empty shell, so **do not scrape it**. Instead use the official Hacker News APIs (Firebase + Algolia), which give the same stories with points, comment counts, and full comment trees. These APIs return plain JSON, so plain `curl` works fine. ## 1. Current top stories (the "top 10") `topstories.json` returns 500 story IDs in front-page rank order. Take the first N and look up each item. ```bash curl -sL 'https://hacker-news.firebaseio.com/v0/topstories.json' -o /tmp/top.json python3 -c " import json,urllib.request ids=json.load(open('/tmp/top.json'))[:10] for i,sid in enumerate(ids,1): d=json.load(urllib.request.urlopen(f'https://hacker-news.firebaseio.com/v0/item/{sid}.json')) print(f\"{i}. {d.get('title')} | {d.get('score')} pts | {d.get('descendants',0)} comments | id {sid}\") print(f\" {d.get('url','(text post)')}\") " ``` ## 2. Find a specific story by topic (Algolia search) ```bash curl -sL 'https://hn.algolia.com/api/v1/search?query=YOUR+QUERY&tags=story' -o /tmp/s.json python3 -c " import json for h in json.load(open('/tmp/s.json'))['hits'][:8]: print(h['objectID'], '|', h.get('points'), 'pts |', h.get('num_comments'), 'comments |', h['title']) print(' ', h.get('url')) " ``` - Add `&numericFilters=created_at_i>UNIXTS` to restrict to recent stories (avoids matching an old duplicate of the same headline). - `search` ranks by relevance; `search_by_date` ranks by recency. - Pick the `objectID` with the highest points/comments - that's the live front-page discussion. ## 3. Fetch a story + its comment tree ```bash curl -sL 'https://hn.algolia.com/api/v1/items/OBJECT_ID' -o /tmp/hn.json ``` The response is a nested tree: top-level `children` are root comments, each with their own `children`. Flatten and print root comments in thread order (HN's default ranking ≈ this order): ```bash python3 -c " import json,re d=json.load(open('/tmp/hn.json')) def clean(t): t=re.sub('<[^>]+>',' ',t) for a,b in [('&#x27;',chr(39)),('&gt;','>'),('&lt;','<'),('&amp;','&'),('&quot;','\"')]: t=t.replace(a,b) return re.sub(' +',' ',t).strip() for c in d.get('children',[])[:15]: if c.get('text'): print(f\"{c.get('author')}: {clean(c['text'])[:550]}\") print('---') " ``` Note: Algolia's per-comment `points` field is now always `null`, so sort by thread order (already roughly HN's ranking) rather than by points. For deeper threads, recurse into `children` and track depth. ## 4. Fetch the linked article Fetch the story's article with `curl -sL <url>`, then strip tags with `sed 's/<[^>]*>//g'` to extract readable text, or grep for the key sentences. If the page is JS-heavy or paywalled, try a Wayback Machine snapshot: ```bash curl -sL 'http://archive.org/wayback/available?url=ARTICLE_URL' -o /tmp/wb.json python3 -c "import json;print(json.load(open('/tmp/wb.json'))['archived_snapshots'].get('closest',{}).get('url'))" ``` Then fetch the snapshot URL the same way. If the host blocks outbound `curl` requests, fetch through a container or proxy you have available. ## Summary format For each story give: title, points, comment count, source, a few sentences on what the article says, then **comment themes** - group the discussion into 3-6 recurring threads (agreement, rebuttals, tangents) rather than listing comments one by one. Note when the top thread is a critical/contrarian take, since that's common on HN.