# Universal Content Intake
[中文说明](README.zh-CN.md)
Universal Content Intake is a download tool for macOS and Windows. **Give it one link
and it downloads what is behind it**: a video, a document, images, an article
or a whole web page. The result is filed in your Downloads folder.
On top of that it offers an optional companion feature, **automatic
monitoring**. Name a set of YouTube creators and the system finds their new
videos that are taking off, downloads them, translates them and burns in
Chinese subtitles, then adds the video details and a publish description. You
decide whether to use it the first time you run the tool.
## Quick start
macOS (Terminal):
```bash
git clone https://github.com/YIAN6557/universal-content-intake.git
cd universal-content-intake
bin/uci https://example.com/report.pdf
```
Windows (Command Prompt or PowerShell; install Python 3.11+ and Git first,
for example with `winget install Python.Python.3.13 Git.Git`):
```bat
git clone https://github.com/YIAN6557/universal-content-intake.git
cd universal-content-intake
bin\uci https://example.com/report.pdf
```
Everything below works the same on both systems. The examples write
`bin/uci`; on Windows type `bin\uci` (in PowerShell `bin/uci` works too).
The first run asks two questions:
1. **Turn on automatic monitoring?** Yes starts a guided setup. No keeps the
system a pure downloader, and you can switch monitoring on later.
2. **When you download a video, translate it and burn in Simplified Chinese
subtitles?** The answers are on, off, or ask every time. This is
independent of the first question. Videos picked by automatic monitoring
always get Chinese subtitles.
It then lists the preparation your choices need, such as checking that ffmpeg
and yt-dlp are installed. Change your answers any time with
`bin/uci settings`.
## Downloading: one link is enough
```bash
bin/uci https://example.com/some-page # just a link: the type is detected
bin/uci "download the PDF at https://example.com/report" # say what you want and it does exactly that
bin/uci "下载这个视频 https://www.youtube.com/watch?v=…"
```
- **Say what you want and it downloads exactly that.** "The PDF at this
link", "this video" or "save this page" pick the matching downloader
without probing. `--type video|pdf|document|image|images|article|webpage`
does the same.
- **Give only a link and it decides.** It checks, in order:
- known platforms: video sites such as YouTube and Bilibili, Google Drive,
image sites;
- the file extension;
- what the server returns. HTML pages are then classified as an article or
a page to keep whole.
- **Videos.** Any of the thousands of sites yt-dlp supports, at 1080p by
default (`--quality 720p|1440p|4k|highest`). Low-resolution sources are
never upscaled.
- Subtitles follow your setting: on adds Chinese subtitles, off downloads
the original only, and ask asks you on each download.
- `--zh` or `--no-zh` overrides the setting for one download.
- **Documents.** Direct links to PDF, Word, Excel, PowerPoint, archives and
more; Google Drive shares; configured rclone remotes.
- **Images.** A single picture, or a whole album or gallery.
- **Web pages.** Articles are extracted as Markdown. Whole pages are saved as
offline HTML, Markdown and PDF.
Downloads land in your Downloads folder under their original title. A short
description file sits next to them with the source, author and original link.
Multi-file results, such as galleries, get their own folder. `--to <folder>`
saves them elsewhere.
### Keeping the download engines current
The downloads rely on open-source engines from GitHub. Setup installs two
scheduled background jobs, whether or not monitoring is on:
- **yt-dlp updates itself weekly.** Each new version is tried on a real
lookup and rolled back if it fails.
- **The other engines are checked every two weeks:** gallery-dl, gdown,
trafilatura, SingleFile, rclone, aria2, deno, and whisper.cpp for
subtitles. Each is compared with its latest GitHub release. When one is
newer you get a desktop notification. **Nothing is installed automatically.**
Each update is one command; a whisper.cpp upgrade is checked on a test
recording first, and the old version stays if the check fails.
On macOS these are LaunchAgents; on Windows they are tasks in the "Universal
Content Intake" folder of Task Scheduler.
### Chinese subtitles: who translates
Speech recognition (whisper.cpp) always runs on your computer. Translation:
- **macOS:** Apple Translation, on the device and free, by default. You can
switch to an online model instead.
- **Windows:** an online model, since Windows has no built-in translation.
Two are offered: **DeepSeek** (the default, very cheap) and **Qwen**
(Alibaba Cloud Bailian, with a free quota for new users).
When you turn subtitles on, the tool asks whether to connect a model, explains
step by step how to get an API key, tries the key on one sentence and keeps it
in the Keychain or Windows Credential Manager. Any time later:
```bash
bin/uci setup translation use deepseek # or qwen; shows how to get a key
bin/uci setup translation key # paste the key (not shown on screen); it is tested before it is saved
bin/uci setup translation test # translate one sentence with the saved key
bin/uci setup translation apple # macOS: back to Apple Translation
```
```bash
bin/uci engines # check now: each engine's version and how to update it
bin/uci status # the result of the last check
```
## Companion feature: automatic monitoring (optional)
```
Creators ─► periodic discovery ─► early view tracking ─► rules + optional AI editorial check ─► daily pick
│
Delivery folder ◄─ burn-in ◄─ translation ◄─ subtitles / on-device speech recognition ◄─ local worker
```
- **In the cloud (Google Apps Script + one Google Sheet).**
- During your discovery window it checks the creators every 10 minutes and
records views at 30, 60 and 120 minutes after publishing.
- It compares each video with that creator's own history to find the ones
taking off.
- It applies your length and title rules. With Gemini connected, it also
checks each video against your editorial direction.
- Once a day it picks one or two videos for your computer.
- **On your computer (a background worker).** It downloads the picked videos,
translates them, burns in Chinese subtitles and delivers each one with a
publish sheet: title, copy, hashtags and source.
Turning it on and off:
```bash
bin/uci settings --monitoring on # on; if it has never been set up, run bin/uci setup as prompted
bin/uci settings --monitoring off # off: stops the local worker and pauses the cloud; creators and settings are kept
bin/uci status # how automatic monitoring is running, and whether the engines are current
```
Setup needs a Google account, and a wizard walks you through it in about 30–60
minutes. **No YouTube API key is needed.** The cloud part uses your own Google
authorization, with a free quota of 10,000 units a day, of which a typical
setup uses one to two thousand. A free Gemini API key is recommended, which
takes about two minutes to create. It filters videos by your editorial
direction and writes the publish copy.
## Requirements
| Feature | Needs |
|---|---|
| Downloading (everyone) | macOS 15 or newer (Apple silicon recommended) or Windows 10 (1809+) / 11; Python 3.11+, ffmpeg, yt-dlp, deno; gallery-dl for images, SingleFile and Chrome for whole pages, rclone for cloud drives |
| Subtitles on macOS (when set to on or ask) | Xcode Command Line Tools; the wizard downloads the ~490 MB Whisper speech model and builds whisper.cpp and the translation helper; Apple Translation language pack (English → Simplified Chinese), or a DeepSeek / Qwen API key |
| Subtitles on Windows (when set to on or ask) | the wizard downloads the ~490 MB Whisper speech model and whisper.cpp's ready-made Windows build; a DeepSeek or Qwen API key |
| Automatic monitoring (when on) | All of the above, plus a Google account and Node.js (for `clasp`); a free Gemini API key is recommended |
`bin/uci setup doctor` lists anything missing with the command to install it
(Homebrew on macOS, winget on Windows).
Tools for features you have switched off are marked as not needed.
## Settings and setup
```bash
bin/uci settings # show or change the two choices
bin/uci setup # guided setup: only the steps your choices need, and what to do next
bin/uci setup verify # acceptance check: a trial download
```
Every wizard step says whether an agent can do it or you must do it yourself,
for example signing in to Google, approving access or pasting a secret.
[SETUP.md](SETUP.md) walks through every step. If you use a coding agent such
as Claude Code, point it at [SKILL.md](SKILL.md). It asks you the two
questions first, then guides you through setup.
The older `bin/uci-get`, `bin/uci-setup` and `bin/uci-status` still work as
aliases of `bin/uci get`, `bin/uci setup` and `bin/uci status`.
## Where settings live
| Where | What |
|---|---|
| `~/.config/universal-content-intake/config.yaml` | the two choices (`features.monitoring`, `features.video_subtitles`); downloads, delivery and work folders; video quality (1080p by default); Chrome sign-in fallback (off by default); cloud endpoint |
| Google Sheet → `Config` (monitoring; edit with `bin/uci setup config set key=value`) | timezone, discovery window, final sweep, selection time, daily count, latest start time for the second pick, view thresholds, maximum length, title rules, Gemini check and your editorial direction |
| Google Sheet → `Creators` (monitoring; edit with `bin/uci setup creators …`) | the creator whitelist |
| macOS Keychain / Windows Credential Manager | the shared secret between your computer and the cloud; the DeepSeek / Qwen key for subtitles; optional Gemini / Anthropic keys for publish copy (`bin/uci setup apikey gemini`) |
| Apps Script → Script Properties | the same shared secret; optional `UCI_GEMINI_API_KEY` |
Your creator list and editorial direction stay in your own Google Sheet and
local config; none of it is stored in this repository.
## Privacy and security
- Downloads are anonymous. Chrome's sign-in state is used for sign-in-gated
videos only when you enable `video.allow_browser_cookies`.
- Speech recognition runs on your computer, and so does Apple Translation on
macOS. With an online model (always on Windows) the subtitle text is sent to
DeepSeek or Qwen for translation. The optional Gemini check sends video
titles, descriptions and statistics to Google. The optional publish writer
also sends translated subtitle excerpts.
- With automatic monitoring, every request between your computer and the cloud
is signed with HMAC-SHA256. The secret lives only in your Keychain or
Credential Manager and in Script Properties, and the wizard never displays it.
## Responsible use
Download, process and publish content only where you have the rights or
permission to do so, and follow each platform's terms of service and the laws
that apply to you. The software is provided as is, and its authors accept no
liability for how it is used.
## Development
```bash
python3 -m unittest discover -s tests # Python
node --test tests/*.js # Apps Script (run in a Node VM)
```
GitHub Actions runs the Python tests on macOS and Windows, plus real downloads,
a full subtitle run and the background tasks on Windows
(`.github/workflows/tests.yml`). What differs between the two systems lives in
`src/core/compat.py`.
The per-stage command-line tools and module ownership are described in
[docs/REFERENCE.md](docs/REFERENCE.md).
## License
Universal Content Intake is dual-licensed:
- **Personal, non-commercial use** is free under the
[Universal Content Intake Personal Use License 1.0](LICENSE).
- **Commercial use** requires a written commercial license and the applicable
fee, obtained in advance; see [Commercial Licensing](COMMERCIAL_LICENSING.md)
or email `ohhhhhmya@gmail.com`.
Use by companies, organizations, studios, teams, employees, contractors or for
client projects counts as commercial use. So does use for paid services,
commercial content, marketing, advertising, brand communication, or
integration into products and workflows. Internal use, trials, not yet being
profitable, or not charging a client separately do not make a use
non-commercial.
Third-party components are not relicensed by this repository; see
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
Maintainer: [@YIAN6557](https://github.com/YIAN6557)