Programming-Contests/tests/README.md

Testing a build

These tools start a throwaway copy of build/acpc2026.qcow2 in QEMU and drive it from the build machine, over SSH and through the virtual keyboard, so that the image can be checked as a team and an administrator would use it. The built image itself is not modified. They need qemu-storage-daemon, debugfs, sshpass and Python with Pillow on the build machine, and the administrator password in ADMIN_PASS.

export ADMIN_PASS='PASSWORD'
tests/check_vm.sh start --wrong-clock --usb
tests/feature_check.sh
tests/technotes_check.sh
tests/ide_check.sh
tests/check_vm.sh stop

feature_check.sh runs, as root in the virtual machine, one check for each feature of the playbooks. technotes_check.sh runs, from the team account, the statements of the technical notes that can be checked on the machine. Both write their results to test_report/ and end with the number of failed checks, which should be 0. When a playbook or the notes change, add the matching check.

Tool	Use
check_vm.sh start [--wrong-clock] [--usb]	boots the copy (tests/work/) with SSH open for the tests and waits for the team desktop. --wrong-clock starts with the hardware clock 3 hours behind, --usb adds a USB controller
check_vm.sh stop	stops the virtual machine and deletes the copy
vssh.sh team|icpc 'command'	runs a command over SSH
gui.sh 'program arguments'	starts a program on the team desktop
key.sh 'text{{ret}}'	types on the keyboard of the virtual machine. Special keys use the QEMU names, as in {{esc}}, {{ctrl-s}} and {{f5}}, and key.sh @file types a file
shot.sh NAME	saves a screenshot as test_report/NAME.png
add_test_tools.sh	installs xdotool in the copy only, for click.sh
click.sh X Y [right|double]	clicks on the team desktop at a screen position, read on a screenshot (1280 by 800 pixels)
usb.sh plug|unplug	plugs a small USB stick into a copy started with --usb
make_report.py	builds test_report/index.html from the screenshots, test_report/captions.txt (one NAME | caption line per screenshot), test_report/intro.txt and the two checklists
ide_check.sh	verifies the installed IDE commands, C23/C++23, static linking, Java VM options, PyPy, paths with spaces, compiler probes and failed-build handling
make_ide_report.py	builds the separate, self-contained test_report/ide-config/index.html from reviewed screenshots and results.json

A full test goes through the following steps, with a screenshot of each.

1. Boot with the wrong clock. The team desktop appears with the right time.
2. Open Firefox from the Applications menu, then the contest, standings, problems, login and status pages, and a few blocked sites.
3. Write a short program in each language in a different editor, then compile and run it in a terminal with the compile and run commands. submit prints the address of the problems on Kattis.
4. Run /test_firewall.sh in the team terminal.
5. Try to get around the restrictions from the team terminal (sudo, reboot, Wi-Fi, reading the administration scripts, a USB stick, a forced address, DNS).
6. Build and run a program in each language of each IDE, with the input typed in the IDE, and check which compiler or interpreter the IDE used. Cover all 19 combinations: Geany and VS Code and Eclipse (C, C++, Java, Python), Code::Blocks (C, C++), CLion (C, C++, Python), IntelliJ IDEA (Java), and PyCharm (Python). Use the actions in the technical notes; the ordinary Java run buttons in VS Code and Eclipse use ECJ, so test their supplied ACPC actions to verify javac. The fixtures in ide_samples/ assert the language standards and runtime versions, exercise C++23 std::println, and check Java's seven VM options. Enter 3000000000 4; each program must print SUM = 3000000004 and exit successfully.
7. In the team terminal, run su - icpc, then contest-restrictions status, off usb (red banner), plug a USB stick, report and on usb. Check that the restrictions are active and the banner has disappeared.
8. Open Contest Help (offline) from the Contest menu. Disconnect the VM network with python3 -I tests/monitor.py tests/work/monitor.sock 'set_link net0 off'. Reload the page, search for EOF, follow an article link and open the local Technical Notes PDF. Use key.sh and shot.sh while SSH is unavailable. Restore the connection with python3 -I tests/monitor.py tests/work/monitor.sock 'set_link net0 on' before continuing other checks.

Do not run /reset_team_home.sh as part of this procedure or any automated test. It permanently erases team files and is reserved for an explicitly authorized manual staff action. The feature checks inspect its installation without executing it.

Keyboard input goes to the window that has the focus, so close dialogs and other windows before typing. First-launch windows depend on what was used before (PyCharm offers to import the settings of VS Code once VS Code has run), so the IDEs are worth testing on a fresh copy as well. The JetBrains IDEs take up to a minute to open in the virtual machine.

For the IDE report, record each reviewed result in test_report/ide-config/results.json: a cases list with id, ide, language, action, status, evidence, and screenshots (each with file and caption), plus verdict, build, notes, and sources (each with title and url). Keep the original report and screenshots. Run the final graphical checks on a fresh Ansible build without manually repairing compiler settings. Test fixtures may be copied into the VM; compile and run them through the actual IDE interface and inspect every selected screenshot.