| name | emergency-rollback |
| description | Emergency rollback procedures when changes break the ISUPIPE application or benchmark fails |
Emergency Rollback — ISUPIPE
When a change breaks the application and the benchmark fails or score drops significantly.
Quick Diagnosis
exec host="vm1" command="systemctl status isupipe-go --no-pager"
exec host="vm1" command="journalctl -u isupipe-go --no-pager -n 30 --since '5 minutes ago'"
exec host="vm1" command="sudo nginx -t && systemctl status nginx --no-pager"
exec host="vm1" command="systemctl status mysql --no-pager"
exec host="vm1" command="mysql -u isucon -pisucon -e 'SELECT 1'"
Rollback: Go Application
If Go code changes broke the build or runtime:
exec host="vm1" command="cd /home/isucon/isucon13/webapp/go && git checkout -- ."
exec host="vm1" command="cd /home/isucon/isucon13/webapp/go && go build -o isupipe ."
exec host="vm1" command="sudo systemctl restart isupipe-go"
Rollback: Database
If schema changes or data corruption occurred:
exec host="vm1" command="cd /home/isucon/isucon13/webapp/sql && sudo -u isucon bash init.sh"
Warning: This resets all data AND removes any added indexes. Re-apply indexes after init.
Rollback: nginx Configuration
exec host="vm1" command="sudo nginx -t"
exec host="vm1" command="cd /home/isucon/isucon13 && git checkout -- ../env/*/etc/nginx/"
exec host="vm1" command="sudo systemctl reload nginx"
Rollback: MySQL Configuration
exec host="vm1" command="sudo rm -f /etc/mysql/mysql.conf.d/isucon.cnf"
exec host="vm1" command="sudo systemctl restart mysql"
Rollback: Multi-Server Distribution
If multi-server setup is broken, return to single-server:
exec host="vm2" command="sudo systemctl start mysql"
exec host="vm3" command="sudo systemctl start mysql"
exec host="vm2" command="sudo sed -i 's|ISUCON13_MYSQL_DIALCONFIG_ADDRESS=.*|ISUCON13_MYSQL_DIALCONFIG_ADDRESS=127.0.0.1|' /home/isucon/env.sh && sudo systemctl restart isupipe-go"
exec host="vm3" command="sudo sed -i 's|ISUCON13_MYSQL_DIALCONFIG_ADDRESS=.*|ISUCON13_MYSQL_DIALCONFIG_ADDRESS=127.0.0.1|' /home/isucon/env.sh && sudo systemctl restart isupipe-go"
exec host="vm1" command="sudo systemctl start isupipe-go"
Full Reset (Nuclear Option)
Re-initialize everything on all VMs:
for vm in vm1 vm2 vm3; do
exec host="$vm" command="cd /home/isucon/isucon13/webapp/go && git checkout -- . && go build -o isupipe . && sudo systemctl restart isupipe-go"
exec host="$vm" command="cd /home/isucon/isucon13/webapp/sql && sudo -u isucon bash init.sh"
exec host="$vm" command="sudo systemctl restart nginx mysql pdns"
done
Post-Rollback Verification
benchmark_start options="--pretest-only"
benchmark_status
If pretest passes, run a full benchmark to confirm score is back to expected level.