periodic-web-scraper-with-redis-storage

Develop a Python script to scrape the latest news items from a website, store detailed content (text, URL, date, media links) in Redis, and schedule periodic updates with deduplication.

Jump to install

Repository: ECNU-ICALK/AutoSkill
Last source activity: March 13, 2026 at 06:15
Detected SKILL.md language: English
Stars: 553
Forks: 51

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

A direct command skips the review prompt. Inspect the source before running it.

npx skills add https://github.com/ECNU-ICALK/AutoSkill --skill periodic-web-scraper-with-redis-storage

The command stays on one line. Scroll horizontally to inspect it before copying.

Prefer a local copy? Download the files currently available to SkillsMP.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

SKILL.md

Source instructions · Read-only preview

Back to search Back to installation

autoskill

Manage personal local Agent Skill files as an installable skill manager. Proactively and periodically detect reusable user-specific, team-specific, or broadly reusable skill material during or after meaningful sessions; run non-blocking extraction checks; offer candidate skill titles or accept a user-supplied topic when extraction direction is ambiguous; preserve the appropriate output language; search local and external skill ecosystems for similar skills; score candidates by evidence, recurrence, personal value, and portability; fully draft proposed skills or diffs before asking for approval; then, after explicit user approval, discard, improve, merge, or create `SKILL.md` folders.

2026-05-10

cbt

布置结构化家庭作业，引导求助者记录现实互动事件、自动思维及情绪反应，并设计简易验证行动（如主动询问、观察反证），用于检验投射性认知偏差。适用于已识别出具体非理性信念（如‘别人肯定不喜欢我’）且情绪稳定者。

2026-05-06

cbt

结构化8次CBT咨询流程，按评估性会谈→咨询性会谈→巩固性会谈三阶段推进，整合悬搁接地技术与认知行为策略，专用于强迫症伴轻度抑郁状态、具自省力与作业执行力的成年来访者。

2026-05-06

View all from this repository

Operational Rules & Constraints

Scraping Logic:

Target the last 10 news items from the source.
Extract specific fields for each item: news text, news URL, news date, and links to photos and videos.

Database Storage:

Use Redis as the database.
Store the extracted data in Redis.

Scheduling:

The program must run periodically every n hours.
The value of n must be obtained from the user via input.

Deduplication:

Implement logic to check if a news item already exists in the database.
Do not save duplicate messages.

Output Format:

Provide the Python code.
Explain the steps in order.

Anti-Patterns

Do not hardcode the website URL or specific news category (e.g., sports) unless provided in the specific request; treat them as variables or placeholders.

Do not omit the deduplication logic.

Triggers

write a python scraper to store news in redis

periodic web scraping script with redis

scrape website every n hours and save to database

python program to scrape and deduplicate news

redis news scraper with scheduling