f2fs-qemu-agent-pipeline

name

f2fs-qemu-agent-pipeline

description

orchestrate safe and reproducible f2fs or qemu work inside a learn_os style kernel workspace. use when chatgpt needs to source .vars.sh, start qemu without blocking, validate kernel builds, run guest commands, run guest test scripts, verify f2fs/mounts, and collect logs with evidence. default to ssh when available, but if ssh is unavailable/blocked or the user explicitly requests qemu guest agent execution, use `scripts/qga_exec.py` as the primary guest command runner.

f2fs / qemu agent pipeline

Follow this skill when working inside a learn_os or similar F2FS development workspace where QEMU boot, guest SSH, kernel builds, and filesystem validation must be performed carefully and reproducibly.

Core behavior

• Treat these instructions as default operating behavior for F2FS, QEMU, guest-SSH, kernel build, and filesystem regression tasks in this workspace.
• Be conservative with destructive actions. Do not reset git state, delete unrelated files, or overwrite user-authored scripts unless explicitly asked.
• Prefer reproducible scripted operations over ad hoc one-off terminal commands.
• Before any build, boot, mount, SSH, or test command, locate the workspace root and source .vars.sh if it exists.
• Reuse variables and helper functions exported by .vars.sh instead of re-declaring paths manually.
• If required variables or helpers are missing after sourcing, stop and report exactly what is missing.

Use this bootstrap pattern unless the workspace already provides a stronger equivalent:

set -a
[ -f ./.vars.sh ] && . ./.vars.sh
set +a

Workflow decision tree

1. Need to boot or re-boot the guest? → follow QEMU start policy.
2. Need to run a guest-side command? → follow Guest interaction policy.
3. Need to validate kernel-side code changes? → follow Build policy.
4. Need to investigate F2FS behavior, mount effects, or on-disk changes? → follow F2FS-focused workflow.
5. Need to report results? → follow Logging and evidence and Safety checks before claiming completion.

Directory structure introduction

• $TEST refer to myscripts/shared_with_qemu/testhas all test script that should run in vm instance.
• $SCRIPT refer to myscripts has QEMU launchers:
  • single-VM: qemu_start_ori.sh (primary launcher)
• guest_console.log is real time log and changes fast. Prefer targeted rg patterns and/or tail -n when analyzing.

QEMU start policy

When starting the VM, use one command that includes built-in verification:

source .vars.sh && bash .agents/tools/vm_start_bg.sh

This script starts QEMU in background and blocks until readiness is confirmed:

• waits for qemu-system-aarch64 process (90s timeout)
• waits for /tmp/qga.sock and /tmp/qemu-qmp.sock
• runs QGA handshake (echo qga_ok && uname -a)
• prints status=ready and the real QEMU PID, or status=failed with reason

Do not run separate post-start verification commands. The script IS the verification. If the script exits non-zero, read the status=... and reason=... output and act accordingly.

If the workspace-local vm_start_bg.sh does not actually print the readiness contract above, or if it exits 0 but no real qemu-system-aarch64 process remains, treat the wrapper result as a false-positive startup. In that case, fall back to a foreground PTY launch with myscripts/qemu_start_ori.sh and keep guest work on QGA/SSH after the handshake. This is a branch, not a success path: record the false-positive evidence and do not claim the VM is ready until the real process, sockets, and QGA handshake pass.

If the launcher log shows a host-forwarding error such as Could not set up host forwarding rule, treat that as a strong signal that another QEMU instance may already be holding the forwarded port. In that situation, inspect the existing QEMU process list first and prefer reusing the already-running VM instead of claiming the new launch succeeded.

Note: the guest can be controlled either via SSH (when available) or via QEMU Guest Agent (QGA) using .agents/tools/qga_exec.py (when SSH is unavailable/blocked or the user requests QGA).

QGA startup failure signals (read script output)

The script prints status=failed with a reason= field. Act on the reason directly:

• reason=no_qemu_process → qemu never appeared. Check launch_log= path.
• reason=no_qga_socket → qemu started but sockets never appeared. QEMU may have crashed; check console_log= for panic.
• reason=qga_handshake_failed → sockets exist but QGA is unresponsive. Guest kernel may be hung; check console_log=.

Do not re-attempt startup without first reading the launch_log and console_log.

When the user explicitly forbids SSH, keep guest command execution on QGA only after startup. It is acceptable to keep QEMU itself attached to a PTY for host-side liveness while all guest-side setup and test commands go through .agents/tools/qga_exec.py.

Detailed troubleshooting checklist: references/qga-startup-troubleshooting.md

QEMU stop policy

When stopping the VM, use a fixed and inspectable pattern instead of ad hoc process killing. The goal is to stop the intended QEMU instance, preserve useful crash evidence when needed, and explain why shutdown is happening.

Use this decision rule:

• Normal completion / user wants to exit after finishing guest work: stop the running QEMU instance after confirming the relevant task is done.
• Kernel crash, panic, or suspected deadlock during testing: do not kill QEMU immediately. Wait briefly first so console output, panic text, or lockup evidence has time to flush into guest_console.log, then stop it.
• Need to re-launch but ports are still occupied by an old VM: identify the existing QEMU instance first, decide whether it should be reused, and only stop it if the current workflow really requires a fresh boot.

Required verification and stop pattern:

1. Inspect the real process list with ps aux | grep qemu.
2. Identify the target qemu-system-aarch64 process, not just the wrapper shell.
3. If the stop reason is crash/deadlock related, wait a short period first to collect evidence.
4. Use a fixed stop command pattern against the verified QEMU PID.
5. Re-check ps aux | grep qemu afterward to confirm the QEMU process is gone.
6. Report the stop reason, target PID, and whether shutdown completed.

Default stop command pattern:

ps aux | grep qemu
kill <qemu-system-aarch64-pid>
ps aux | grep qemu

Crash/deadlock-oriented variant:

sleep 5
ps aux | grep qemu
kill <qemu-system-aarch64-pid>
ps aux | grep qemu

Important constraints:

• Prefer stopping the qemu-system-aarch64 PID directly. Do not assume killing the launcher shell is sufficient.
• Do not claim shutdown succeeded unless the post-stop process check confirms that the target QEMU instance disappeared.
• If multiple QEMU instances are present, explicitly state which PID is being stopped and why.
• If the user still needs the guest running, do not stop it just because the skill can.

Guest interaction policy

This workspace supports two control planes for running commands inside the VM:

• SSH (default when available): best for interactive-ish workflows and bulk file movement.
• QEMU Guest Agent (QGA) via .agents/tools/qga_exec.py (preferred when SSH is unavailable/blocked or when the user explicitly requests it).

Treat .agents/tools/qga_exec.py as a high-level tool: use it instead of ad-hoc socat/raw QGA JSON whenever the task is “run a command/script inside the guest”.

Decision rule (SSH vs QGA)

Use QGA (.agents/tools/qga_exec.py) if any of these are true:

• The user says SSH is unavailable, broken, blocked, or intentionally not used.
• The user explicitly requests qga_exec.py or “QGA execute”.
• The guest has no network / no forwarded port, but QGA is up.

Otherwise, use SSH via .agents/tools/vm_ssh.sh.

QGA execution policy (when selected)

1. Verify host-side QGA socket exists and is reachable: /tmp/qga.sock
2. Run guest commands via:

python3 .agents/tools/qga_exec.py --sock /tmp/qga.sock '<guest command>'

3. For long-running tests or noisy output, do not stream unlimited stdout back through QGA. Prefer:
   • ... | tail -n 200 for quick diagnosis, or
   • redirect logs to a guest-local path (e.g. /tmp/test.log) and then tail / sed -n it.
4. Always capture the exit code and report it.
5. If the QGA probe succeeds but background launch is flaky in this workspace, keep the VM process alive through a foreground PTY host session and still use QGA as the only guest command channel.

QGA evidence pull policy

When a guest-side failure leaves important evidence under guest-local paths and SSH/scp is unavailable:

1. Try small text evidence first through qga_exec.py with host-side redirection:

python3 .agents/tools/qga_exec.py --sock /tmp/qga.sock 'cat /tmp/test.log' > /tmp/test.log

2. If the file is large or binary, use QGA file-read instead of streaming through guest-exec:

python3 .agents/tools/qga_pull_file.py --sock /tmp/qga.sock /guest/path/file.img /host/path/file.img

3. Verify the pulled artifact with size and hash on both sides before cleaning the guest:

sha256sum /host/path/file.img
python3 .agents/tools/qga_exec.py --sock /tmp/qga.sock 'sha256sum /guest/path/file.img; stat -c "%s %n" /guest/path/file.img'

Use this path when 9p/shared-directory writes from guest fail with Operation not permitted; host-created files in the shared directory do not prove the guest can create large artifacts there.

SSH execution policy (when selected)

• For non-interactive guest operations, prefer SSH command injection instead of opening an interactive terminal.
• Prefer a wrapper such as .agents/tools/vm_ssh.sh.
• Before the first SSH action of a fresh boot, verify the guest is reachable and ready.
• Prefer key-based SSH when it is configured.
• If key-based SSH is not configured, explain the concrete blocker and the smallest next step.
• Prefer idempotent remote commands.
• Do not claim a guest-side step ran unless the SSH command actually succeeded.

Recommended order:

1. Confirm VM is up.
2. Select control plane (QGA vs SSH) using the decision rule above.
3. Verify reachability (QGA socket or SSH port).
4. Run a small probe command.
5. Execute the intended guest command through the chosen wrapper.
6. Capture exit status and any relevant log path.

Script entrypoints and references

When these scripts are available under this skill's scripts/ directory, prefer them over ad-hoc command reconstruction. In this workspace, the absolute path is under /home/nzzhao/.agents/skills/f2fs-qemu-agent-pipeline/scripts/.

• scripts/vm_start_bg.sh: reference: references/script-vm_start_bg.md
• scripts/qga_exec.py: reference: references/script-qga_exec.md
• scripts/qga_pull_file.py: reference: references/script-qga_pull_file.md
• scripts/vm_stop.sh: reference: references/script-vm_stop.md
• scripts/vm_ssh.sh: reference: references/script-vm_ssh.md
• /home/nzzhao/.agents/skills/f2fs-qemu-agent-pipeline/scripts/vm_enable_dyn_debug.sh: reference: references/script-vm_enable_dyn_debug.md
• /home/nzzhao/.agents/skills/f2fs-qemu-agent-pipeline/scripts/vm_hunt_trunc_write_delete.sh: reference: references/script-vm_hunt_trunc_write_delete.md
• /home/nzzhao/.agents/skills/f2fs-qemu-agent-pipeline/scripts/vm_hunt_rwtest_deadlock_watch.sh: reference: references/script-vm_hunt_rwtest_deadlock_watch.md

Rule: if script behavior and SKILL prose differ, update both SKILL and references together in the same change set.

Deadlock-hunting note:

• Do not treat D state as deadlock by itself.
• For this workspace's inline-encryption hunt, prefer a two-phase confirmation:
  1. sustained rw_test.py D state window
  2. repeated sysrq w/t/l snapshots separated by a gap
  3. classify as deadlock-suspect only when key stack signatures remain stable and workload log shows no forward progress.

Build policy

When asked to validate, test, or sanity-check a kernel-side change: Loop: 1. Identify changed .c files relevant to the current work using git diff or git status. 2. For each changed .c file, run the single-file or object-level build helper from .vars.sh which is a function called kobj 3. Only after object-level checks succeed, run bash $SCRIPT/make_upstream.sh the full kernel image build command. 4. bash $SCRIPT/make_upstream.sh already capture build logs in a file you can inspect.s 5. Summarize failures with the exact failing file, target, or phase. 6. Do not claim success unless the relevant command exited successfully. 7. Fix small syntax errors that the log reports. Especially be careful when deal with macro bug report. They can recursive flatten a lot of logs, but many cases the errors of use macro are just small errors.Don't be confused by the vase error massege.Thinking and reasoning the root case. Until: no more errors are reported.

When reporting build results, always include:

• the command or script that ran,
• whether it succeeded,
• where the log file is,
• the next recommended step if it failed.

F2FS-focused workflow

When the task is about F2FS behavior, mount behavior, on-disk effects, regression checks, or guest-visible filesystem state:

• Prefer booting the prepared QEMU guest instead of reasoning purely from source.
• Prefer scripted reproduction steps over informal shell experimentation.
• Keep host-side orchestration on the host and guest-side file operations inside the guest.
• When shared directories or 9p mounts are required, verify the mounts explicitly before proceeding.
• When an encrypted directory path is not confirmed, do not invent one; report that it is unknown.
• If the workflow depends on guest-visible artifacts, verify they actually exist in the guest.
• Reference (code-path overview): references/f2fs-verity-buffered-writeback-flow.md
• Reference (interface mapping): references/f2fs-sysfs-procfs-interface.md
• Reference (knob usage + adding): references/f2fs-sysfs-knobs-usage-and-adding.md
• Reference (fscrypt inlinecrypt/fallback bounce triage): references/fscrypt-inlinecrypt-bounce-diagnosis.md
• Reference (xfstests F2FS/QEMU triage): references/xfstests-f2fs-qemu-triage.md

Tracefs-first debugging pattern

When debugging guest-side page-cache, mmap, readahead, or writeback behavior:

• Prefer running guest commands through scripts/vm_ssh.sh with one wrapped remote command block instead of many ad hoc SSH calls.
• Prefer clearing and enabling only the needed tracepoints in tracefs, then running the reproduction, then filtering the captured trace before reporting.
• If the bug is inode-specific, always collect the guest file inode with stat, convert the decimal inode to hex, and filter trace output by the hex inode because trace events such as mm_filemap_fault and mm_filemap_add_to_page_cache print inode values in hex.
• Do not rely on tail of the whole trace buffer when the task is about a specific file; inode-filtered extraction is the default.
• When a workflow is repeatedly useful, prefer creating or extending reusable wrappers under the skill's scripts/ directory rather than duplicating long command sequences in chat.
• When validating mmap/readahead behavior, capture both MM/filemap tracepoints and filesystem tracepoints when available, because one side alone can hide folio lifecycle transitions.
• Use $TEST wisely and based on user needs to validate and continuesly see the logs.

Logging and evidence

Every meaningful operation should leave evidence when practical:

• the exact command line or wrapper used,
• the exit status,
• a stable log file path when available.

Present results concisely, but never hide the exact failing command.

Use this reporting shape by default:

command/script: <what ran>
status: success | failed | blocked
log: <path or "none">
next step: <smallest useful follow-up>

Safety checks before claiming completion

Before saying the task is done, verify the relevant facts actually happened:

• the build finished successfully,
• the VM is really running if you said it started,
• SSH is really reachable if you said guest access is ready,
• required mount points are present if the task depends on them,
• expected output files or logs were created.

Refusal to fake progress

• Do not say a VM is up unless you verified it.
• Do not say a kernel built unless the build exited successfully.
• Do not say a guest-side operation ran unless SSH execution actually succeeded.
• If a required script, helper, path, or mount is missing, say so directly and propose the exact missing piece.

Preferred response style

• Be operational.
• Be explicit about paths, scripts, and next commands.
• Keep summaries short, but include enough detail for manual reproduction.
• When blocked, give the smallest actionable unblock step.

Guidedance of adding logs to kernel

• Must load skill "kernel-log-instrumentor"

Example triggers this skill should cover

• “帮我在 learn_os 里把 qemu 起起来，但不要卡住对话。”
• “用ssh操作虚拟机,发送命令和调试”
• “ssh 不可用/不想用 ssh，用 .agents/tools/qga_exec.py 在虚拟机里执行命令或跑测试脚本。”
• “先检查这次改动涉及的 f2fs .c 文件能不能单独编，再跑整镜像编译。”
• “进 guest 看一下共享目录有没有挂上，顺便跑个非交互命令。”
• “这个 F2FS 改动不要只看代码，帮我进 qemu 做一个可复现验证。”
• “给我一个明确结论：到底是没启动、没连上 ssh，还是编译失败，日志在哪。”