VirtualBox disk corruption
For a while now I've been using vagrant (with VirtualBox) VMs to test an application I'm working on. Unfortunately quite often these VMs would only last one or perhaps two tests before apparently having corrupted their disk.
The symptoms would tend to be "Input/output error", eg things like:
-bash: /usr/bin/less: Input/output error
...although it could be almost anything that failed to run. Sometimes the machine wouldn't boot at all, others it would but "vagrant ssh" couldn't connect or would connect but couldn't provide a shell.
Thankfully for my testing I could just throw away the VM and let it rebuild (vagrant is great!) to get things working again.
Today I finally investigated the actually problem. It seems it stemmed from the application I'm testing being very disk intensive. After a random amount of time of running the application, the guest kernel would spit out some of these:
[ 63.034642] mptscsih: ioc0: attempting task abort! (sc=ffff8800dbb99800) [ 63.034650] sd 2:0:0:0: [sda] tag#1 CDB: Read(10) 28 00 00 5c 67 50 00 00 08 00 [ 63.146718] mptscsih: ioc0: task abort: FAILED (rv=2003) (sc=ffff8800dbb99800)
For reference, I'm running VirtualBox from Ubuntu 16.10 with Ubuntu 16.04 as the guest.
Based on those reports, it's an issue with using multiple processors and SCSI disks in VirtualBox. Which managed to lead me to an actual bug report.
At the moment it seems the latest release (5.1.12) does include the fix "Storage: fixed a problem with the LsiLogic SCSI controller where requests could be lost with SMP guests" - see the changelog.
The 5.1.12 version of VirtualBox has already been packaged for Zesty (Ubuntu 17.04) and it seems to install fairly cleanly onto 16.10. Below is the process I followed.
Fetch a couple of dependencies:
...and install them:
sudo dpkg -i libgsoap10_2.8.35-4_amd64.deb libvpx4_1.6.0-3_amd64.deb
Then grab the main VirtualBox packages:
...and install those too:
sudo dpkg -i virtualbox-dkms_5.1.12-dfsg-2_all.deb virtualbox-qt_5.1.12-dfsg-2_amd64.deb virtualbox_5.1.12-dfsg-2_amd64.deb
...and hopefully all will be well.
So far, it seems to have fixed the disk corruption. I have now had a guest kernel panic relating to memory allocation but at least when I restarted the VM I was still able to login and carry on from where it stopped.