We’ve been tracking down issues with Windows Server 2016 on a multitude of servers this week where the servers will reboot and come back with Unmountable Boot Volume which is a pretty nasty experience for oncall. So far we’ve mainly seen it on Domain Controllers but also on a Hyper-V server. The solution is typically to do a last known good boot on the machine and then try to work out what has changed on the server and needs redoing. So far we’ve had issues with duplicate servers in Webroot and Automate along with a couple of server functions not working correctly.
Initially we thought it was a problem with Windows Updates, but it seems that the culprit is Storagecraft’s SPX version 6.7.4
The solution is either to downgrade to version 6.5 or get a patch for 6.7.4 that fixes this issue.
Download location for SPX 6.5.2:
For 6.7.4, You will need to get the patched stcvsm.sys from Storagecraft and then apply these instructions.
Patch is a very manual process. New version of the stcvsm.sys driver is 2.2.73.0.36
1. Install SPX 6.7.2:
2. Do NOT reboot
3. Rename %windir%\system32\drivers\stcvsm.sys to %windir%\system32\drivers\stcvsm-rtm.sys
4. Copy the 2.2.73 driver to %windir%\system32\drivers. Be sure to select the correct ‘bitness’.
5. Reboot
It’s been very frustrating to have gone through this issue without any notification of this pretty serious bug from #Storagecraft
Edit: Today I discovered that Storagecraft now have a more detailed knowledge base article about resolving Inaccessible Boot Device after upgrade to 6.7.x. Judging from the comments I’ve had here, I’m not the only one who has had this issue and it still keeps happening for some users.
Comments
Andy – thanks for sharing this information. The StorageCraft technical support team is delighted to assist with this issue. A support request may be submitted via support.storagecraft.com.
Regards,
Connie Whiteside
Sr Director Customer Success
[email protected]
Author
Thanks for the follow up Connie. My team is in the process of rolling out the patch or downgrading to the previous version which is working for us.
How were you able to resolve the blue screen issue?
Disregard previous comment. How did you determine that it was StorageCraft SPX causing the issue?
Author
Hi Jorge,
Thanks for commenting – we fixed it by rolling back to the previous version – it was quicker than applying the patch and I’ve been swamped with other things this week which means I haven’t had a chance to try the manual patch. As you also need to contact Storagecraft for the patch itself, it’s quicker just to roll back (as long as you don’t have an auto upgrade function on your servers.
As to how we found it – a lot of very painful rollbacks and a process of elimination. Iniitially we thought it was a windows update as it occurred after machines rebooted due to the patches, but after building a new server, applying the various windows patches we were not able to reproduce the issue. As one of the machines we had to rebuild was also going to be in production we installed spx on it and it failed to complete the boot cycle afterwards. We rolled back the image and were able to reproduce the issue, contacted Storagecraft and were told it was a known issue and were provided the patch file to test or to revert back to the previous version.
I am not providing the actual patch file here as it’s not my property to distribute and Storagecraft need to be aware of how many customers this affected which can only be done by customers contacting support.
Experiencing this now, random issue with a relatively new 2016 server deployment. Storagecraft customer.
Very helpful
Thank you
I ended up copying the same file version 2.2.71.0 product version 2.2.71.0.29 from known working server 16 over to another server 16 running same SPX version but had the blue screen. Server booted at that point
Ran into this problem last week. It’s still an issue.
Thanks Andy… may have helped one of our customers here at Dell.
I thought I had it fixed as well but it did not so I went to version 6.5.2
Strange though I have version 6.7.4 on Server 16 VM with it being DC, Webroot AV, and Automate RMM tools installed. But that server is fine.
I’m having this issue now on Server 2016 physical servers, but I have 2016 VM’s with Storagecraft SPX 6.7.4/Webroot/Automate RMM tools and they seem to be just fine (so far
Author
We’re actually still getting this on some machines – the latest occurred this week 🙁
I had the same problem today.
Server was also a DC, webroot installed, Shadow Protect SPX installed, and Connectwise Automate.
Unfortunately, booting to last known good did not fix this issue for me.
Because this was a secondary DC with the Essentials Experience role installed, I opted to build a new server from scratch.
Getting this on terminal server 6.7.4 server 16.
However I am tending to only see this during which the host shuts down the VMs, then upon host reboot, the VM does not start.
Strange thing is other VM (cloned, then sysprepped etc) is a DC with same version of spx, LT, and SC. That one does not have a problem.
I do wonder if SecureBoot has anything to do with it? I came across one article where a user had issues with bluescreens and SecureBoot enabled. Anyone else check that?
Hi there,
I got the same problem too. Windows 2016 server, Webroot , RMM and installed Storagecraft 4.7.x..
It restarted and gave me a bluescreen, then I read your post and restarted again with Last known good boot. Then the server started successfully, I checked to see if the Storeagecraft service is still there, but it disappeared. also, I tried to uninstall 4.7.x and install 4.5.x but it failed to uninstall and failed to install.
anyone got any idea? i really need 4.5.x to backup my server
Thanks
After I restarted the server by using “Last known good boot”
I tried to uninstall the 6.7.4 but it will not uninstall.
and then i tried to install the 6.5.x , it will not install.
please help. i really need to back my server up asap, thanks
Author
I would recommend you log a ticket and call into Storagecraft support to get it installed on your machine properly, especially with the new version of Storagecraft.
thank you for this. saved my ass this morning, brotha.
This seems to only be an issue with servers that have webroot installed, I had a 2016 server running SPX 6.7.2 for over 12 months with no issue, installed webroot this week, and first reboot since then came up with the same error. Replacing the storagecraft sys file as per the KB article does resolve the issue.