rclone-wiki-backup

name	rclone-wiki-backup
description	Sync a local wiki (Obsidian vault) to Google Drive using rclone with .gitignore rules, including critical rclone-vs-git syntax differences discovered through trial and error.

rclone Wiki Backup to Google Drive

Key Learnings (Trial-Error Results)

0. CRITICAL: -* at end of filter file excludes EVERYTHING

A filter file ending with -* will exclude ALL directories not explicitly included with + rules. This is a common mistake that results in only explicitly-included files syncing.

Symptom: Dry-run shows directories like entities:, concepts:, raw: as "Excluded" when they should sync.

Fix: Remove -* from the end of filter file. rclone defaults to including everything; only add exclusions for what you DON'T want.

1. skills/**/node_modules pattern fails in rclone

Git's .gitignore uses skills/**/node_modules to exclude node_modules at any depth.
rclone requires trailing / to treat this as a directory exclusion. Without /, rclone still enters the directory and scans its contents.

Fix in .gitignore:

# Wrong for rclone (works in git only):
skills/**/node_modules

# Correct for rclone (trailing / = directory):
skills/**/node_modules/

2. !.github/copilot-instructions.md negation rule doesn't work in rclone

Git supports negation patterns (!) in .gitignore. rclone's --exclude-from does NOT support negation includes — exclusion rules are final.

Also: mixing --include and --exclude-from causes indeterminate ordering ERROR:

ERROR: Using --filter is recommended instead of both --include and --exclude as the order they are parsed in is indeterminate

Fix: Use a filter file (--filter-from) with explicit ordering — include BEFORE exclude for the same path:

+ .github/copilot-instructions.md   # include FIRST
- .github/                          # exclude AFTER
- *                                 # exclude everything else

3. IPv6 causes "i/o timeout" on Google OAuth

When the Mac has IPv6 connectivity but no actual route to Google, rclone authorize drive fails with:

Error: failed to get token: Post "https://oauth2.googleapis.com/token": dial tcp [2607:f8b0:400e:c01b::5f]:443: i/o timeout

Fix: Use v2rayN proxy at port 10808:

export HTTPS_PROXY=http://127.0.0.1:10808
export HTTP_PROXY=http://127.0.0.1:10808
rclone authorize drive

4. ~ doesn't expand in zsh alias single quotes

Alias commands with ~ in the path won't expand correctly inside single quotes. Use $HOME instead:

# Wrong:
alias rclone-wiki-sync='rclone sync ... --filter-from=~/.rclone/filter-wiki.txt'

# Correct:
alias rclone-wiki-sync='rclone sync ... --filter-from=$HOME/.rclone/filter-wiki.txt'

5. Proxy auto-set in .zshrc (conditional) — only for other tools that need it, NOT for rclone sync

if nc -z 127.0.0.1 10808 2>/dev/null; then
  export HTTPS_PROXY=http://127.0.0.1:10808
  export HTTP_PROXY=http://127.0.0.1:10808
fi

Important: This proxy auto-set is for curl/browser/brew etc. For rclone sync, DO NOT rely on this — it degrades performance. Either unset before sync or run in a clean subshell.

Filter File Format (~/.rclone/filter-wiki.txt)

Critical: include rules must come BEFORE exclude rules for the same path. rclone processes filter files top-to-bottom.

WARNING: Do NOT end filter file with -* — this excludes everything not explicitly included with + rules. rclone defaults to include; only add exclusions for what you DON'T want.

- /.tmp.driveupload
- /.tmp.drivedownload
- skills/**/node_modules/
- database.base
- .smart-env/
- .venv/
- __pycache__/
- .DS_Store
- .idea/
- .claude/settings.local.json
- .obsidian/workspace.json
- .obsidian/workspace (1).json
- .obsidian/workspace-mobile.json
- .obsidian/graph.json
- .obsidian/backlink.json
- .obsidian/plugins/
- .obsidian/core-plugins.json
- .obsidian/core-plugins (1).json
- .obsidian/types.json
- .obsidian/icons/
- copilot/
- .omc/
- .git/                              # exclude git repo metadata
- rclone-sync-*.log                  # exclude rclone log files (prevents hash mismatch on sync)
- cron-status.log                    # exclude cron status log
- heartbeat/                         # exclude heartbeat dir (see section 10b — TOCTOU md5 race with cron-heartbeat.py)

+ .github/copilot-instructions.md    # include BEFORE the matching exclude
- .github/                           # exclude after include

Proxy: Environment-Dependent

There are TWO scenarios:

Scenario A: Direct connection works (default)

Critical finding from 2026-05-10: Using proxy (10808) for actual sync causes:

• ~100x slowdown: 11.7s dry-run → 7+ minutes for 26% completion with proxy
• Incomplete file listing: proxy reports 1,683 files vs 2,679 via direct connection
• False ETA: proxy shows 3h+ ETA for 9 MB of data

Rule for standard environments: Proxy only for rclone authorize drive. Always try direct connection first for sync:

# This WORKS fine — no proxy needed for actual sync:
/Users/jinguo/bin/rclone sync . gdrive-wiki:wiki ...

Scenario B: Direct connection times out (requires proxy)

Found in 2026-05: Some environments (e.g., cron jobs, certain network configs) cannot reach Google Drive API directly — they must use proxy. In this case, proxy is necessary and works correctly (~8 min for this wiki).

When to use proxy:

• Cron job environment can't reach Google Drive API directly
• Direct connection hangs/fails with timeout
• Error: dial tcp: i/o timeout

export HTTPS_PROXY=http://127.0.0.1:10808
export HTTP_PROXY=http://127.0.0.1:10808
/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before

Always verify connectivity first:

nc -z 127.0.0.1 10808 && echo "Proxy OK"

Full Sync Commands

# Dry-run (always first — direct connection, no proxy)
rclone-wiki-sync --dry-run

# Actual sync (direct connection)
# NOTE: This wiki (~500 files) takes 30+ minutes. Do NOT assume hung if no output.
/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before

Background execution (recommended for cron):

# WRONG for Hermes - will error: "Foreground command uses '&' backgrounding"
# Use terminal(background=true) instead

# Correct approach for Hermes cron:
terminal(background=true, command="""
export https_proxy=http://127.0.0.1:10808
export http_proxy=http://127.0.0.1:10808
/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before \
  > ~/wiki/rclone-sync-tmp.log 2>&1
""")

Current cron prompt pattern (in use since 2026-06-08) — SIMPLER, NO inner &: After Hermes tightened the background-process check, the simplest working pattern is terminal(background=true) wrapping exec rclone ... directly. No &, no disown, no nohup inside the command body. Hermes tracks the process via the returned session_id.

terminal(background=true, command="""
export https_proxy=http://127.0.0.1:10808
export http_proxy=http://127.0.0.1:10808
echo "[$(date '+%Y-%m-%d %H:%M')] rclone-wiki-backup: starting" >> ~/wiki/cron-status.log
exec /Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \\
  --filter-from=$HOME/.rclone/filter-wiki.txt \\
  --delete-before
""", notify_on_complete=true, timeout=600)

Verified 2026-06-08: this pattern completed cleanly in 174s with exit_code: 0. The notify_on_complete=true flag is the right complement — it triggers exactly one notification on exit, so the cron run reports real completion status (not just "started") without needing an explicit wait/poll loop in the prompt.

⚠️ Pitfall: omitting the redirect inside terminal(background=true) leaves rclone-sync-tmp.log stale The canonical example above wraps exec rclone ... directly, with NO > ~/wiki/rclone-sync-tmp.log 2>&1 redirect inside the background command. That means stdout/stderr go ONLY to Hermes's process buffer (visible via process(action='poll')), and the rclone-sync-tmp.log file on disk stays 0 bytes / stale from the previous run. Verified 2026-06-11: heartbeat touched, sync completed, exit=0 — but ~/wiki/rclone-sync-tmp.log mtime was from a previous run because the new run never wrote to it.

If you need on-disk log continuity (for grep / diff / tail from another terminal), include the redirect inside the background command body:

terminal(background=true, command="""
export https_proxy=http://127.0.0.1:10808
export http_proxy=http://127.0.0.1:10808
exec /Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \\
  --filter-from=$HOME/.rclone/filter-wiki.txt \\
  --delete-before \\
  > ~/wiki/rclone-sync-tmp.log 2>&1
""", notify_on_complete=true, timeout=600)

Two consequences of the redirect:

• process(action='poll') shows the live tail of the log file (not rclone's own stdout), so you'll see the same content as tail -f ~/wiki/rclone-sync-tmp.log.
• The log file is on the rclone-exclude list (- rclone-sync-*.log is filtered out), so the next sync will not transfer it to Drive — safe to keep writing.

If you don't need the on-disk log (relying on Hermes process buffer + cron-status.log entries instead), the simpler exec rclone ... form is fine. Pick one and stay consistent.

⚠️ CRITICAL anti-patterns that will error in 2026-06+ Hermes:

• ❌ nohup rclone ... & inside a foreground terminal() call → ERROR: "Foreground command uses shell-level background wrappers (nohup/disown/setsid). Use terminal(background=true)..."
• ❌ rclone ... & inside a foreground terminal() call → ERROR: "Foreground command uses '&' backgrounding. Use terminal(background=true) for long-lived processes"
• ❌ disown inside a foreground terminal() call → same error as nohup

All three are caught at the shell-wrapper layer before rclone even starts. If you see the error, switch to terminal(background=true) — do not try to work around it with & inside the body.

Legacy shell approach (for manual terminal, not Hermes):

# Run in background — rclone produces NO output until complete for large syncs
# BUT: if log stays 0 bytes for >5 minutes with no progress, it may be hung — see section 11
/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before \
  > ~/wiki/rclone-sync-tmp.log 2>&1 &
echo "rclone PID: $!" >> ~/wiki/rclone-sync-tmp.log

• Log file stays 0 bytes for the entire duration — this is normal for large syncs
• Process runs 30-60+ minutes for this wiki size — patience, not intervention
• Check completion via: pgrep -f "rclone sync.*gdrive-wiki" || echo "done"

Verify network activity (diagnostic if stuck):

# Check if process is still running
pgrep -f "rclone sync.*gdrive-wiki" && echo "still running" || echo "completed"

# Verify network connections through proxy (if stuck)
lsof -p $(pgrep -f "rclone sync.*gdrive-wiki" | head -1) | grep TCP
# Should show established connections to proxy (localhost:10808)

Completion detection via process tool (Hermes cron): When rclone is launched with terminal(background=true, notify_on_complete=true), the session_id from the launch response is the canonical handle. Two ways to check:

1. Block on completion (preferred when nothing else to do in the cron run):

   process(action='wait', session_id='proc_xxx', timeout=120)
   

   ⚠️ Pitfall: timeout parameter is silently clamped to the 60s platform limit even if you pass 300/600. Plan polling around this — call process(action='poll', timeout=180) separately if you need to wait longer than 60s, since wait will return {"status": "timeout"} after 60s regardless of the value you passed.

   Verified 2026-06-10: A 23-minute sync required SIX consecutive process(action='wait', timeout=600) calls, each returning {"status": "timeout", "timeout_note": "Requested wait of 600s was clamped to configured limit of 60s"}, before the seventh wait finally returned {"status": "exited", "exit_code": 0}. Each timeout response is a no-op for the actual process — rclone keeps running unaffected in the background. Two practical patterns:

   1. Loop the wait (simple, works for any duration):

      while True:
          r = process(action='wait', session_id='proc_xxx', timeout=600)
          if r.get('status') == 'exited':
              break
          # status == 'timeout' after 60s — loop and wait again
      

   2. Poll until exited (preferred — no busy-loop, returns the same data):

      while True:
          r = process(action='poll', session_id='proc_xxx')
          if r.get('status') == 'exited':
              break
          time.sleep(30)  # back off between polls
      

   Both are correct; pick by taste. The "right" timeout value to pass is moot — the clamp is a hard cap. Do NOT keep calling wait with bigger and bigger timeout values hoping the clamp lifts — it won't.

2. Poll after the fact (preferred when other steps run between start and end):

   process(action='poll', session_id='proc_xxx', timeout=180)
   # returns {"status": "exited", "exit_code": 0, "uptime_seconds": 81} when done
   # returns {"status": "running", "uptime_seconds": 12} while still going
   

When the rclone sync is short (~80s for incremental), process(action='wait', timeout=120) returns status: "exited" cleanly without ever hitting the clamp. Cross-check pgrep -f "rclone sync.*gdrive-wiki" from a separate terminal() call for a belt-and-suspenders "is it really done?" check.

⚠️ Pitfall: session_id PID is the bash wrapper, NOT the rclone binary

Verified 2026-06-12: when terminal(background=true) returns a session_id and a PID (e.g. 24370), that PID is the /bin/bash -lic ... wrapper, not the rclone binary. The actual /Users/jinguo/bin/rclone process is a child (e.g. PID 24373, ppid=24370). This means:

• ps -p <session_pid> shows the bash wrapper, NOT rclone. Its command line is the literal multi-line bash invocation (with \012 separators) and its CPU% stays at 0.0 even when sync is in full swing. This is normal — the wrapper is just exec-ing and waiting.
• To see real rclone activity, walk children of the wrapper PID:

  ps -axo pid,ppid,etime,%cpu,%mem,command | awk -v p=<session_pid> '$2 == p'
  # or equivalently:
  pstree -p <session_pid> 2>/dev/null
  

  Expect ONE child row like /Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki .... Its etime is the authoritative elapsed time for the sync, and %cpu shows real work.
• pgrep -f "rclone sync.*gdrive-wiki" works too (the pattern matches the child command line), but the ps -axo ppid=... approach is the only one that ties the child back to its session_id, which matters when you have multiple backgrounded syncs and need to disambiguate.
• After the process exits, both the wrapper and the rclone child are gone — pgrep -f "rclone sync.*gdrive-wiki" returns nothing, and process(action='poll', session_id=...) returns status: "exited". Both signals should agree; if one says "exited" and the other says "still running", wait 5s and re-check (the rclone child sometimes reaps slightly after the wrapper).

⚠️ Stale cron prompts in jobs.json may still use the broken & pattern

The current canonical pattern is terminal(background=true, command="exec rclone ...") or the terminal(background=true, command="rclone ... > log 2>&1") redirect form (both are valid; pick one). A cron prompt that contains nohup ... & or rclone ... & inside a foreground terminal() call will fail at launch with:

Foreground command uses shell-level background wrappers (nohup/disown/setsid). Use terminal(background=true)...

If a cron run starts failing with that error after a prompt edit, the prompt body still has the old & form — fix by removing the &, nohup, and disown from the cron prompt and relying on terminal(background=true) for lifecycle. The shell-level backgrounding is doubly wrong inside Hermes: (a) Hermes rejects it, and (b) the rclone child would be reparented to PID 1 when the foreground bash exits, losing any output not already redirected.

⚠️ Pitfall: uptime_seconds field in process(action='poll') responses is STALE Verified 2026-06-08: the uptime_seconds value reported by process(action='poll') can lag the real elapsed time by minutes — observed uptime_seconds stuck at 149s across three polls spanning ~6s wall-clock each, while ps -o etime -p <PID> showed 02:33 → 02:49 (correctly advancing). The poll eventually does return status: "exited" and a final exit_code once the process finishes, so correctness is preserved — but don't use uptime_seconds from poll to drive decisions like "should I kill it?". For a real elapsed-time signal during a long sync, use:

ps -o pid,etime,stat,command -p <PID>   # etime is the authoritative elapsed wall clock

Treat uptime_seconds in poll responses as a best-effort hint, not ground truth.

Proxy only when the first attempt fails with network errors:

export HTTPS_PROXY=http://127.0.0.1:10808
export HTTP_PROXY=http://127.0.0.1:10808
/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before \
  -P

Alias (add to .zshrc)

# rclone wiki backup — dry-run first, then drop --dry-run for actual sync
# Proxy only if direct connection fails; direct is ~100x faster
alias rclone-wiki-sync='/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki --filter-from=$HOME/.rclone/filter-wiki.txt --delete-before -P'

11. Rclone hangs silently with no output — add -v (or -vv) flag to debug and proceed

When running rclone in certain environments (proxy required, cron), the process may start but produce zero output for the entire duration — log file stays 0 bytes. This is DIFFERENT from normal behavior (where log is also 0 bytes but sync completes).

KEY INSIGHT: Without -v flag, rclone produces NO output at all in background mode — even when sync is working perfectly. This is normal behavior, not a hang. Use -v to see actual progress.

11c. Incremental syncs are fast — ~20 seconds for small changes

After the initial full sync, subsequent syncs only transfer changed files. Typical incremental sync:

• Duration: 20-30 seconds
• Files checked: ~3,700 (full file listing still runs)
• Transferred: 2-10 files, 1-50 KiB typical

This is dramatically faster than the 60-120 minute full sync. Use -P flag to monitor:

rclone sync ... -P

11b. Rclone slow startup — 60+ seconds before first progress is NORMAL

Symptom: rclone process starts (PID visible), but log shows only NOTICE messages for 60+ seconds before any "Transferred" progress appears. Process appears to hang.

Reality: This is NORMAL behavior. rclone takes significant time to:

1. Initialize OAuth token refresh through proxy
2. List all files on both source and destination (this wiki: ~14,600 files)
3. Compute checksums for change detection

How to distinguish from actual hang:

• With -v flag: you'll see "Listed X files" messages after ~60 seconds
• Check process state: if CPU time increases (ps -o time), it's working
• Use rclone lsd to verify connectivity works at all

Example from 2026-05-23 (609 files, 6.3 MB sync):

21:13:59 NOTICE: ...symlink warnings...     # Initial startup
21:14:44 INFO: ...first Copied message...   # ~45 seconds later
21:24:03 INFO: Transferred: 138.963 KiB...  # ~10 min later (first progress output)

Solution: Be patient. Do NOT kill the process before 2 minutes with -v flag showing no progress. The sync is working even before output appears.

For progress visibility in cron/background jobs, use both flags:

/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before \
  -v \
  --stats=10s \
  > ~/wiki/rclone-sync-tmp.log 2>&1

This outputs progress every 10 seconds:

Transferred:  	  25.979 KiB / 1.066 MiB, 2%, 1.623 KiB/s, ETA 10m56s
Checks:              3091 / 3091, 100%, Listed 12832
Transferred:           0 / 107, 0%
Elapsed time:        50.0s

⚠️ PITFALL: --stats-one-shot flag does NOT exist When adding stats output to rclone commands, do NOT use --stats-one-shot — it will cause rclone to fail with Error: unknown flag: --stats-one-shot.

Use --stats=10s or --stats=1m instead (the value is the reporting interval):

/Users/jinguo/bin/rclone sync ... --stats=1m

Symptoms:

• Process shows as running (pgrep -f "rclone sync.*gdrive-wiki" returns PID)
• Log file stays 0 bytes for 5+ minutes
• No progress, no errors, no output whatsoever

Debugging first attempt: Kill the hung process:

pkill -f "rclone sync.*gdrive-wiki"

Recovery: Add -vv (double-verbose) flag and retry:

export https_proxy=http://127.0.0.1:10808
export http_proxy=http://127.0.0.1:10808

/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before \
  -vv \
  > ~/wiki/rclone-sync-tmp.log 2>&1

The -v flag enables verbose output which can help rclone proceed past whatever was causing the hang, and provides visibility into what's happening.

Cron Job (auto backup)

⚠️ CRITICAL: Cron prompt must use terminal(background=true), NOT shell &

The cron prompt should invoke the terminal tool with background=true parameter. Do NOT use shell & — it will error:

Foreground command uses '&' backgrounding. Use terminal(background=true) for long-lived processes

Correct cron prompt structure:

python3 ~/wiki/scripts/cron-heartbeat.py touch rclone-wiki-backup

# 设置代理（必须）
export https_proxy=http://127.0.0.1:10808
export http_proxy=http://127.0.0.1:10808

# 后台同步 — use terminal(background=true) in Hermes, NOT shell &
terminal(background=true, command="""
export https_proxy=http://127.0.0.1:10808
export http_proxy=http://127.0.0.1:10808
/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=/Users/jinguo/wiki/.rclone/filter-wiki.txt \
  --delete-before \
  -L \
  > ~/wiki/rclone-sync-tmp.log 2>&1""")

Note on --filter-from path: Use the absolute path ~/wiki/.rclone/filter-wiki.txt in the cron prompt (not the $HOME/.rclone/filter-wiki.txt the skill's main config-location line suggests). The deployed filter file as of 2026-06-15 is at the wiki path, and $HOME/.rclone/filter-wiki.txt does not exist on disk. See "Config Locations" above for the full layout.

Legacy shell approach (NOT recommended for Hermes cron):

# WRONG for Hermes cron prompts — will fail with backgrounding error
/Users/jinguo/bin/rclone sync ... > ~/wiki/rclone-sync-tmp.log 2>&1 &

Cron job rclone-wiki-backup runs every 20m automatically.

# Manual trigger
hermes cron run job-id=0372e9a009a3

Cron prompt uses source ~/.wiki-cron.env and runs the sync command without proxy. Executes as background process — cron delivery reflects start-time status, not completion. Log to ~/wiki/cron-status.log after process exits.

Cron Heartbeat & Status Logging

Touch heartbeat at start (required for stale task detection):

python3 ~/wiki/scripts/cron-heartbeat.py touch rclone-wiki-backup

Log start/completion to cron-status.log:

# At start:
echo "[$(date '+%Y-%m-%d %H:%M')] rclone-wiki-backup: started in background (PID $!)" >> ~/wiki/cron-status.log

# At completion:
echo "[$(date '+%Y-%m-%d %H:%M')] rclone-wiki-backup: completed successfully" >> ~/wiki/cron-status.log

8. Expired refresh token causes rclone to hang indefinitely

When the GDrive OAuth token's access_token has expired AND the refresh_token is also invalid (revoked, expired, or blocked), rclone does NOT produce a clean error — it hangs indefinitely trying to refresh. Both the custom binary (/Users/jinguo/bin/rclone) and the system binary (/opt/homebrew/bin/rclone) exhibit this behavior.

Symptoms:

• rclone process starts but produces zero output
• Log file stays 0 bytes indefinitely
• pgrep -f "rclone sync.*gdrive-wiki" shows the process is still running
• ps aux | grep rclone shows the process in S (sleep) state
• Any diagnostic command (rclone lsd, rclone lsf, rclone about) also hangs

Diagnosis: Check the token expiry in ~/.config/rclone/rclone.conf:

grep -A5 '\[gdrive-wiki\]' ~/.config/rclone/rclone.conf

If the expiry field is in the past AND the refresh token appears truncated or the token block looks incomplete, the refresh token is dead.

Recovery: Requires manual re-authentication on the Mac:

rclone authorize "gdrive-wiki"
# or
rclone config reconnect gdrive-wiki:

Prevention: Consider using a service account file (service_account_file in config) instead of OAuth for cron jobs, since service account tokens auto-refresh and don't expire.

9a. Source file modified during sync (transient)

When a file is being actively edited during sync, rclone may fail with:

ERROR: entities/2026-05-14-code-intelligence.md: Failed to copy: Patch "...googleapis.com/upload/drive/...": googleapi: Copy failed: can't copy - source file is being updated (size changed from 10905 to 10947)

Symptoms:

• Single file ERROR in log
• Error message includes "source file is being updated" and "size changed"
• rclone continues processing other files

Resolution: This is a transient error. The next scheduled cron run will sync the file successfully. No intervention needed unless it persists across multiple runs.

Prevention: Schedule wiki edits to avoid edit-write conflicts during backup windows, or use file locking if feasible.

9b. Google Drive API quota exceeded (403 rate limit)

When rclone produces no output and eventually exits, or when debug mode shows:

googleapi: Error 403: Quota exceeded for quota metric 'Queries' and limit 'Queries per minute' 
of service 'drive.googleapis.com' for consumer 'project_number:202264815644'

This is a temporary rate limit - Google Drive API allows ~840 queries/minute/project. Large syncs can hit this.

Symptoms:

• rclone continues running but outputs 403 ERROR messages to log
• Multiple ERROR entries appear (one per file that hit the limit)
• rclone automatically retries after brief delays
• Eventually completes or fails after all retries exhausted

⚠️ CRITICAL: Quota exceeded during --delete-before phase causes stuck retry loop When the quota error occurs during the initial "Waiting for deletions to finish" phase (required by --delete-before), rclone may enter a retry loop that hangs indefinitely:

• Process shows as running with established TCP connections to proxy
• Log file contains ERROR messages but no new progress
• No files are transferred even after 10+ minutes
• Verbose mode (-vv) shows repeated "Attempt N/3 failed" messages

Detection: If rclone is running but log shows no progress (only repeated ERROR entries) for >10 minutes, it's stuck in retry loop.

Recovery: Kill the process and wait before retrying:

# Kill stuck process
pkill -f "rclone sync.*gdrive-wiki"

# Wait for quota to reset (30-60 seconds)
sleep 60

# Retry - quota should clear
/Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before \
  > ~/wiki/rclone-sync-tmp.log 2>&1

Prevention:

• Use --transfers=1 to reduce concurrent API calls
• Schedule large syncs during off-peak hours
• Split large syncs across multiple cron runs

10. Rclone log file causes MD5 hash mismatch on sync (CRITICAL for cron)

When running rclone with output redirection (> logfile 2>&1), the log file itself gets synced to Google Drive, causing:

ERROR : rclone-sync-tmp.log: corrupted on transfer: md5 hashes differ
src(Local file system at /Users/jinguo/wiki) "xxx" vs dst(Google drive root 'wiki') "yyy"

Root cause: The log file is open/being written during sync, or contains different content between local and remote (buffering, different timestamps).

Fix: ALWAYS exclude log files in filter:

- rclone-sync-*.log
- cron-status.log

Even better: Delete the temp log file before sync:

rm -f ~/wiki/rclone-sync-tmp.log
/Users/jinguo/bin/rclone sync ... > ~/wiki/rclone-sync-tmp.log 2>&1

10b. Heartbeat file md5 mismatch — TOCTOU race with the cron-heartbeat.py touch (verified 2026-06-16)

A different md5 mismatch can hit heartbeat/cron.last-run itself, with a different root cause than section 10 (which is the log file self-reference). Symptom observed in a real 09:04 cron run:

2026/06/16 09:05:56 ERROR : heartbeat/cron.last-run: corrupted on transfer: md5 hashes differ
src(Local file system at /Users/jinguo/wiki) "924d59635713ee9c55a2660fa93658dc" vs
dst(Google drive root 'wiki') "6bb9e6e62d182350563037166b165d16"
2026/06/16 09:06:55 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: md5 hashes differ
...
2026/06/16 09:12:42 ERROR : Attempt 2/3 succeeded

Root cause — TOCTOU race between two cron jobs:

1. rclone-wiki-backup cron prompt begins with python3 ~/wiki/scripts/cron-heartbeat.py touch rclone-wiki-backup. This writes a new timestamp to heartbeat/cron.last-run.
2. rclone hashes the wiki source listing, captures md5 of the new cron.last-run.
3. Another cron job (e.g. wechat-inbox-pipeline, rss-feed-scan, anything that runs cron-heartbeat.py touch) fires in the same 20-min window and overwrites heartbeat/cron.last-run with a fresh timestamp.
4. rclone uploads the old content; Google Drive-side md5 of the current Drive copy (or vice versa) doesn't match the bytes being PUT. rclone marks the transfer "corrupted".
5. rclone retries (Attempt 2/3) and usually succeeds — the heartbeat is just a one-line timestamp file, so by retry time no other job is touching it.

Distinguishing feature: The file in the error message is heartbeat/cron.last-run (not a log file). If you see this filename in an ERROR : corrupted on transfer line, it is the TOCTOU race, not section 10's log-file self-reference.

Why rclone's internal retry works:

• The file is tiny (~30 bytes), uploads in milliseconds.
• After Attempt 1 fails, the next attempt's source-side md5 is captured afresh. By then the other cron job has long since finished writing, so md5(src) == md5(uploaded).
• Total impact: ~6 min delay in this run (one retry cycle that included the full wiki re-listing). Exit code was 0.

Fixes — pick by environment:

1. Add heartbeat/ to the filter (simplest, but loses Drive backup of heartbeat state — acceptable since heartbeat is a local artifact, not user data):

   - heartbeat/
   

   Trade-off: Drive won't have a copy of heartbeat/cron.last-run, but that file is regenerated by every cron run anyway. The rclone sync will stop flagging it.

2. Schedule the heartbeat touch as a separate cron job that runs at a non-overlapping offset (e.g. every 20m but :00, :20, :40, while rclone-wiki-backup runs :05, :25, :45):

   • Eliminates the race entirely — only one job touches cron.last-run at a time.
   • Verified pattern; needs the cron schedule (not the rclone command) to be edited.

3. Excluding the file in-flight from rclone's hash check — rclone does not support per-file "skip md5 verify" without skipping the file entirely. The retry-and-succeed path (which already works) is the simplest "fix" if you're comfortable with the ~6 min penalty.

Recommended: Add - heartbeat/ to the filter. Heartbeat files are operational ephemera (regenerated every cron tick), not content that needs Drive backup. This eliminates the error line entirely.

Do NOT confuse with section 9a (transient "source file is being updated" errors on actively-edited wiki pages). That error includes the literal phrase "source file is being updated" and a size diff. The heartbeat md5 mismatch has neither — it's purely a hash-comparison failure on a file the user did NOT just edit, with the message "md5 hashes differ".

7. Google Drive unreachable — diagnostic and recovery flow

When rclone sync hangs with no output (log file stays 0 bytes for >5 minutes):

Diagnosis: Google Drive API is unreachable (OAuth token expired, network down, or proxy issue).

Recovery steps:

1. Kill ALL rclone processes first — two concurrent instances cause conflicts:

   pkill -f "rclone sync.*gdrive-wiki"
   

2. Test connectivity before attempting sync:

   source ~/.wiki-cron.env
   /Users/jinguo/bin/rclone lsf gdrive-wiki:wiki 2>&1 | head -5
   

   If this times out after 60s, Google Drive is unreachable.
3. Log the failure to cron-status.log:

   echo "[$(date '+%Y-%m-%d %H:%M')] rclone-wiki-backup: FAILED - Google Drive API unreachable (timeout)" >> ~/wiki/cron-status.log
   

4. Do NOT retry in the same cron run — the next scheduled run will retry automatically.

Common causes: OAuth token expired (check ~/.config/rclone/rclone.conf), IPv6 routing issue, or proxy not configured.

6. OAuth token missing or invalid (cron failure modes)

There are TWO failure modes with different symptoms:

Mode A: Token completely missing

CRITICAL: Failed to create file system for "gdrive-wiki:wiki":
drive: failed when making oauth client: failed to create oauth client:
empty token found - please run "rclone config reconnect gdrive-wiki:"

• Diagnosis: config shows only type and scope, no token field at all
• Fix: rclone config reconnect gdrive-wiki:

Mode B: Token exists but expired/invalid (hangs indefinitely)

• Symptoms: rclone hangs (no output, log stays 0 bytes), any command (lsd, lsf, about) times out after 60s+
• Diagnosis: Check token expiry:

  grep -A3 'token = {' ~/.config/rclone/rclone.conf | grep expiry
  

  If expiry time is in the past (e.g., expiry: "2026-05-17T12:00:20", current time is after 12:00), the token is expired. If rclone hangs when trying to refresh, the refresh_token is also invalid.
• Fix: Manual browser-based OAuth on Mac:

  rclone config reconnect gdrive-wiki:
  

  Note: reconnect will prompt "Already have a token - refresh? y) Yes" — you must answer interactively, so this cannot run in cron.

Mode C: Token valid but cron environment cannot reach Google API (CRITICAL)

• Symptoms: Token is valid (not expired), but direct API call times out
• Diagnosis: Test connectivity:

  rclone lsd gdrive-wiki:wiki --timeout=30s
  # Times out even with valid token
  

• Root cause: Cron environment has different network path - no IPv6 route to Google, or direct connection blocked
• Fix: Use proxy in cron prompt:

  export https_proxy=http://127.0.0.1:10808
  export http_proxy=http://127.0.0.1:10808
  /Users/jinguo/bin/rclone sync ...
  

• Note: Even after re-authenticating with browser, cron may still need proxy because the OAuth token refresh goes through a different network path than browser OAuth

Prevention: Use service account file (service_account_file in config) instead of OAuth — tokens auto-refresh and don't expire.

Config Locations

• rclone config: ~/.config/rclone/rclone.conf
• Filter rules: ~/.rclone/filter-wiki.txt (skill canonical) — but verified 2026-06-15 the deployed file lives at ~/wiki/.rclone/filter-wiki.txt, not ~/.rclone/filter-wiki.txt. If $HOME/.rclone/filter-wiki.txt is missing, check ~/wiki/.rclone/filter-wiki.txt instead. The current ~/wiki/.rclone/filter-wiki.txt contains only + * (include everything), which means the deployed filter is effectively a no-op — the wiki syncs in full, no exclusions. This is FINE for a small wiki but the longer exclusion list in the "Filter File Format" section below is the skill's recommended shape for noise reduction.
• Remote name: gdrive-wiki
• Drive destination: gdrive-wiki:wiki
• Cron heartbeat script: ~/wiki/scripts/cron-heartbeat.py (called as python3 ~/wiki/scripts/cron-heartbeat.py touch rclone-wiki-backup)

Sync Metrics

See references/sync-metrics.md for runtime data points and performance characteristics.

--stats* flag quick reference

See references/stats-flags.md for the exact flag combination to use in cron (--stats=30s --stats-one-line -v), what each variant does, and why --stats-one-shot does NOT exist as a flag.

Recurring cron noise: playwright-profile/node_modules symlinks

Every cron run logs the same two NOTICE lines about symlinks that can't be followed without -L:

NOTICE: scripts/playwright-profile/node_modules/.bin/playwright: Can't follow symlink without -L/--copy-links
NOTICE: scripts/playwright-profile/node_modules/.bin/playwright-core: Can't follow symlink without -L/--copy-links

Why: scripts/playwright-profile/ lives under scripts/ (not skills/), so the existing skills/**/node_modules/ rule in the filter doesn't reach it. The .bin/playwright* files are symlinks, which rclone refuses to follow by default.

Impact: Cosmetic only — the symlinks aren't transferred, but rclone still scans the directory above them. The sync proceeds and exits 0. The NOTICE count is the only signal in the log when nothing else changes.

Fix options (any of these silences the noise):

1. Add to ~/.rclone/filter-wiki.txt:

   - scripts/playwright-profile/node_modules/
   - scripts/playwright-profile/node_modules/**
   

2. Add --copy-links to the rclone invocation to follow symlinks (slower for this path).
3. Delete the symlinks locally if the profile isn't actively used.
4. Add -L (follow symlinks, transfer target) to the rclone invocation. Verified 2026-06-15 on rclone-wiki-backup cron: this silences BOTH the scripts/playwright-profile/node_modules/.bin/playwright* symlink NOTICEs and the ~/.git/fsmonitor--daemon.ipc non-file NOTICE in one shot. -L is the same flag rclone uses internally for --copy-links mode — it dereferences the symlink on the source side and uploads the target. For wiki/ this is safe: every symlink in playwright-profile/node_modules/.bin/ points to a real playwright/playwright-core JS file inside the same package, and .git/fsmonitor--daemon.ipc is a transient socket that rclone's -L will skip with a "Can't transfer non file/directory" notice (also a NOTICE, not an error). Cost: marginal — the wiki has only ~3 symlinks and they are tiny. This is the simplest one-flag fix when the symlinks are safe to follow.

Symptom → fix mapping: If a cron log shows ONLY 2 NOTICE lines and exit=0, this is the expected "already in sync" outcome — do not chase it. The presence of the NOTICEs is a signal of "incremental no-op sync completed cleanly", not an error.

Re-verified 2026-06-17 in the rclone-wiki-backup 00:00 run: a 2-NOTICE log (the two playwright-profile/node_modules/.bin/playwright* symlink warnings) appeared twice in the file — once at 00:01:00 from the failed first attempt, once at 00:05:38 from the recovered second attempt. Both runs produced 530 bytes total, both exited 0, and the heartbeat TOCTOU pitfall (section 10b) did NOT fire — confirming the - heartbeat/ filter rule added on 2026-06-15 is doing its job.

⚠️ PITFALL: rclone binary path on Apple Silicon — /usr/local/bin/rclone does NOT exist (verified 2026-06-17)

This Mac is Apple Silicon (Darwin arm64), so homebrew installs to /opt/homebrew/bin/, NOT the Intel-era /usr/local/bin/. The custom build lives at /Users/jinguo/bin/rclone. On this machine:

$ ls -la /Users/jinguo/bin/rclone /opt/homebrew/bin/rclone /usr/local/bin/rclone
-rwxr-xr-x  /Users/jinguo/bin/rclone        ← canonical, the one cron must use
lrwxr-xr-x  /opt/homebrew/bin/rclone        ← homebrew shim, also works
             /usr/local/bin/rclone          ← DOES NOT EXIST — do NOT use

Symptom of using the wrong path (observed 2026-06-17 in the rclone-wiki-backup cron run at 00:00):

bash: /usr/local/bin/rclone: No such file or directory
exit_code: 127          # rclone never started, log file stays 0 bytes
uptime_seconds: 4       # bash exits in 4s

What this looks like in the cron delivery: a 4-second "started" report with empty log + exit 127 from the wrapper. The session_id returned by terminal(background=true) is the bash wrapper, so process(action='poll') will eventually return {"status": "exited", "exit_code": 127}.

Recovery recipe (proven 2026-06-17, total wall time ~6 min):

1. Verify the path is wrong before re-running:

   ls -la /Users/jinguo/bin/rclone /opt/homebrew/bin/rclone 2>&1
   

2. Rotate the stale log so a fresh run gets a clean 0-byte file:

   rm -f ~/wiki/rclone-sync-tmp.log
   

3. Re-launch with the correct path via terminal(background=true, notify_on_complete=true):

   terminal(background=true, command="""
   export https_proxy=http://127.0.0.1:10808
   export http_proxy=http://127.0.0.1:10808
   /Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \\
     --filter-from=$HOME/.rclone/filter-wiki.txt \\
     --delete-before \\
     > ~/wiki/rclone-sync-tmp.log 2>&1
   """, notify_on_complete=True, timeout=600)
   

4. Log both events to cron-status.log so the trail is clear:

   echo "[$(date '+%Y-%m-%d %H:%M')] rclone-wiki-backup: started in background (PID $correct_rclone_pid)" >> ~/wiki/cron-status.log
   # ...after process() returns exited, exit_code 0:
   echo "[$(date '+%Y-%m-%d %H:%M')] rclone-wiki-backup: completed (exit 0)" >> ~/wiki/cron-status.log
   

Always use /Users/jinguo/bin/rclone in cron prompts — never /usr/local/bin/. If a cron prompt you read contains /usr/local/bin/, treat it as a stale recipe and patch it. The canonical examples in the "Full Sync Commands" and "Cron Job" sections of this skill all use the correct path; do not be misled by older /usr/local/bin/rclone snippets in third-party blog posts.

Cross-check rule for any new rclone recipe: before running, command -v rclone or which rclone will print the homebrew path. That is fine for ad-hoc manual use, but for cron prompts always pass the full /Users/jinguo/bin/rclone so the binary location is independent of $PATH at cron-exec time.

⚠️ PITFALL: A 2-line log is AMBIGUOUS — never trust it alone (verified 2026-06-13)

The "2 NOTICE lines = clean no-op" mapping above is correct ONLY when combined with two other signals. A bare 2-line log file can mean three different things, and you cannot tell them apart from the log contents alone:

Log state	pgrep -f "rclone sync.*gdrive-wiki"	~/wiki/rclone-sync-tmp.log mtime	Meaning
2 lines	no PID	recent	✅ Clean no-op — incremental sync found no changes, exited 0
2 lines	PID still running	recent	⚠️ Sync in progress — stats haven't fired yet; rclone is mid-transfer, wait
2 lines	no PID	stale (from previous run)	❌ rclone died early — or > log 2>&1 redirect was missing and new run never wrote here (see "omitting the redirect" pitfall above)

Always cross-check three things before declaring success:

pgrep -f "rclone sync.*gdrive-wiki" > /dev/null && echo "STILL_RUNNING" || echo "DONE"
stat -f "%Sm %z bytes" ~/wiki/rclone-sync-tmp.log
tail -3 ~/wiki/rclone-sync-tmp.log

Why this matters: The 2026-06-13 cron run produced an exactly-2-line log on the first attempt that looked identical to a successful no-op. The wrapper exited on line 2 of the command (after echo "rclone PID: $!"), the inner nohup ... &; disown rclone was still running asynchronously, and the log had not yet received its first stats line. Only the pgrep check disambiguated "process is still running, just not done yet" from "process completed cleanly".

Fix for the cron prompt: Always include --stats=30s --stats-one-line so the log always contains a Transferred: ... line within 30s of any transfer activity, even a zero-byte one. This eliminates the "ambiguous 2-line log" failure mode entirely:

# Canonical cron invocation — stats line guarantees log is never just 2 NOTICEs
exec /Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki \
  --filter-from=$HOME/.rclone/filter-wiki.txt \
  --delete-before \
  --stats=30s --stats-one-line -v \
  > ~/wiki/rclone-sync-tmp.log 2>&1

The --stats-one-line flag (no value) keeps output compact — one summary line per interval, not the full multi-line block. The default value is 1m; 30s is friendlier for cron runs that themselves run every 20m.

⚠️ Do NOT confuse --stats-one-line with --stats-one-shot — the latter does not exist (confirmed error: unknown flag: --stats-one-shot). --stats-one-line formats stats as a single line per interval; it does NOT mean "print stats once and exit".

Note on nohup ... &; disown inside terminal(background=true): This pattern is technically allowed (Hermes does not reject the & when the outer call is already background=true), but it is REDUNDANT — the outer terminal(background=true) already detaches the process. The &; disown adds nothing useful and creates the "wrapper exits on line 2, child keeps running" footgun above where a pgrep check is required to know the true state. Prefer the canonical exec rclone ... form (no inner &, no nohup, no disown) for cron prompts.

⚠️ PITFALL: 6-hour silent hang — the cron prompt recipe let a stuck rclone live undetected (verified 2026-06-16)

The previous cron-prompt recipe (nohup rclone ... & inside a foreground terminal call) returned immediately after writing the start line. The cron delivery reflected "started" status, the inner rclone was reparented to PID 1 with no heartbeat or completion detection, and a rclone that hung on its first kqueue call sat for 6+ hours and 3 cron ticks before being noticed. Symptoms that distinguish a true silent hang from a slow sync — and a full recovery recipe — are in references/diagnosing-silent-hangs.md. The four-axis liveness check (pgrep + etime + time/%cpu + sample frames) is the right first move when a log file hasn't grown in >5 min; the proxy-socket kqueue case (_pthread_cond_wait on a TCP socket to 127.0.0.1:10808 with flat CPU time) is the failure mode to recognize — SIGTERM the hung rclone, restart via terminal(background=true, notify_on_complete=true) with --stats=30s --stats-one-line -v so the new run's log can't go silent.

Stale PID in cron-status.log from the &; disown pattern

When a cron prompt uses the nohup rclone ... &; echo "rclone PID: $!" >> ...log form, the PID captured by $! is the inner rclone child PID at the moment of &, but the line gets written to cron-status.log by the outer wrapper which exits immediately. Two consequences:

1. The PID is correct at the moment of capture — $! is the rclone child PID, not the wrapper.
2. But by the time you cat cron-status.log, that PID is long gone — the rclone child lived for ~3 minutes, exited, and was reaped. The PID number in the log is now meaningless.

Fix: Either (a) drop the &; disown and use the canonical exec rclone ... form so Hermes tracks the rclone process via session_id (and process(action='poll') reports the real exit_code on completion), or (b) write the completion status to cron-status.log from a follow-up terminal() call after pgrep confirms the rclone is done. The 2026-06-13 run adopted pattern (b):

# In the cron prompt, after the background sync has been launched:
sleep 60
pgrep -f "rclone sync.*gdrive-wiki" > /dev/null && echo "STILL_RUNNING" || echo "DONE"
tail -3 ~/wiki/rclone-sync-tmp.log
echo "[$(date '+%Y-%m-%d %H:%M')] rclone-wiki-backup: completed successfully (~26 KiB, 2 file updates)" >> ~/wiki/cron-status.log

Pattern (a) is cleaner but requires giving up the "fire-and-forget, check next cron tick" model that the current cron prompt relies on. Pattern (b) preserves the fire-and-forget model and adds completion detection within the same run.

Diagnosing silent hangs

For the full diagnostic recipe (four-axis liveness check, the specific 2026-06-16 proxy-kqueue failure mode, the recovery steps that worked, and root-cause hypotheses), see references/diagnosing-silent-hangs.md.

Notes

• Runtime: This wiki (~3200 files, ~70+MiB) takes 60-120+ minutes to sync. rclone produces no stdout/stderr during this time. Do NOT kill a running rclone that shows no output — it is working.
• --delete-before: deletes files in Drive not present in source after sync
• Dry-run always before actual sync
• Filter file order is deterministic; --include+--exclude-from mix is not
• Debug filter issues with -vv: If dry-run shows unexpected exclusions, run with -vv flag to see detailed logs:

  /Users/jinguo/bin/rclone sync /Users/jinguo/wiki gdrive-wiki:wiki --filter-from=$HOME/.rclone/filter-wiki.txt --dry-run -vv 2>&1 | grep -E '(Excluded|Need to transfer)'
  

  This reveals exactly which directories/files are excluded and why.
• Do NOT rely on .zshrc proxy auto-set for sync — it degrades performance ~100x. Use the explicit alias or direct command without proxy env vars.
• Background process: Use pgrep -f "rclone sync.*gdrive-wiki" to check if sync is still running. Log file stays 0 bytes until completion — this is normal rclone behavior for large syncs.