We’ve been tracking down issues with Windows Server 2016 on a multitude of servers this week where the servers will reboot and come back with Unmountable Boot Volume which is a pretty nasty experience for oncall. So far we’ve mainly seen it on Domain Controllers but also on a Hyper-V server. The solution is typically to do a last known good boot on the machine and then try to work out what has changed on the server and needs redoing. So far we’ve had issues with duplicate servers in Webroot and Automate along with a couple of server functions not working correctly.
Initially we thought it was a problem with Windows Updates, but it seems that the culprit is Storagecraft’s SPX version 6.7.4
The solution is either to downgrade to version 6.5 or get a patch for 6.7.4 that fixes this issue.
Download location for SPX 6.5.2:
For 6.7.4, You will need to get the patched stcvsm.sys from Storagecraft and then apply these instructions.
Patch is a very manual process. New version of the stcvsm.sys driver is 188.8.131.52.36
1. Install SPX 6.7.2:
2. Do NOT reboot
3. Rename %windir%\system32\drivers\stcvsm.sys to %windir%\system32\drivers\stcvsm-rtm.sys
4. Copy the 2.2.73 driver to %windir%\system32\drivers. Be sure to select the correct ‘bitness’.
It’s been very frustrating to have gone through this issue without any notification of this pretty serious bug from #Storagecraft
Edit: Today I discovered that Storagecraft now have a more detailed knowledge base article about resolving Inaccessible Boot Device after upgrade to 6.7.x. Judging from the comments I’ve had here, I’m not the only one who has had this issue and it still keeps happening for some users.
For the short answer go here.
For those of you that follow me on Facebook, you may have seen my frustration with Microsoft support. I had to log a support call with their support as every time my computer rebooted, offline files were disabled. Unfortunately the technician attempting to support me obviously had no idea of what offline files were or even how to create a new user account in Windows7 (to troubleshoot if it was my account or not that was causing the problem). At no point in time did either of the 3 techs look at the event log, despite me trying to tell them there was issues reported in the log.
To cut a very long and frustrating story short, I initially followed the diagnostic steps at Jonathan’s blog and the follow up post but the setting was still disabled after a reboot. I did find that enabling the offline files service from disabled (or manual) and then starting the service, the offline files were temporarily enabled.
As part of this troubleshooting step, the issue was obviously something resetting the services on startup so off to the eventlogs. The Offline Event Logs section had nothing in it out of the ordinary but my Application log was full of Bootshieldsvc errors. One “offline files bootshieldsvc” search later
shows this is a known problem with Lenovo’s rapid boot software. The Rapidshield patch from Lenovo was downloaded, extracted, installed and the pc rebooted and sure enough offline files was STILL not enabled. Urgh.
clicking start, and entering RapidBoot confirms the version is 1.23 and the gui has an option to disable the feature. This requires a reboot so I used the opportunity to re-enable offline files again and rebooted – STILL disabled. After another reboot, just to be sure Offline files was still disabled. Checking RapidShield Gui, the software claims it is not running, but checking the application log after a reboot still shows the various errors – for example Event 256 for BootShieldsvc – “An error has occured (—query FLAG_AUTO_SVC_CHANGED key success failed with 1,The Code is:0x24.).
Apart from the fact that there is a “success failed” – it’s really not encouraging that software is still running and making changes despite it being disabled. Next stop was an uninstall (and enabling offline files again) followed by yet another reboot. It’s a good thing this machine is fast at rebooting.
So after uninstalling – the problem still occurs – arghhhhhhhh.
Checking the status of the Offline Files service I noticed it was still set to manual. Comparing against another machine where offline files works, the service should be set to automatic. I changed the service to automatic and started. Offline files was now showing it was enabled. Another reboot and the service was back to manual again and offline files disabled – more arghhhhhhhhhhh
Next stop is to try and hack out the Fast Boot Service of Lenovo. UNfortunately I get access denied when trying to stop the service. msconfig stops the service but it still somehow manages to try and make changes as evidenced by the events in the eventlog after a reboot.
sc delete “BootShieldSvc” from an elevated command line also removes the service but it comes back after a reboot. After a reboot into safe mode I was able to disable the Fast Boot Service of Lenovo, set the Offline Files to be automatic and now offline files works.
In Summary –
Uninstall the Lenovo Rapidshield software (or upgrade).
If that fails – boot into safe mode, disable Fast Boot Service of Lenovo, Set Offline Files to automatic.
Powershell 3 was released this week and is now available to download for Windows 7 (sp1) platforms. I tried to install it on my home machine this weekend and got
“The update is not applicable to your computer”. It turns out that this is actually because .net framework 4 (or higher) has not been installed.
The full .net 4 framework package is available at http://www.microsoft.com/en-us/download/details.aspx?id=17718 or you could install the newer 4.5 framework at http://www.microsoft.com/en-us/download/details.aspx?id=30653. Make sure that you close the false powershell installation before attempting the .net installation or the .net will try to install for about 5 minutes on your computer and then complain that another install is already in use and does not give you the option to retry. You can only abort and then run the whole installation again. All in all a pretty bad user experience for trying to install the software.
Thanks to the Troubleshooting guide for the beta version of Powershell 3 that tipped me off for the pre-reqs which are not mentioned on the original download page for Powershell 3 and unfortunately there is no place on that web page to provide feedback.
Firefox updated itself to v15 this morning and now all my tabs are wrong.
If I open multiple tabs then the url that I type in on the right most tab actually gets loaded in the tab to the left of it. As you can see from the screenshot shown here, my current url is showing LogMeIn and the “Add New Post” screen is actually to the right.
Interestingly, the far left tab does show my gmail content but the second from left only shows the google page – the bing results are not found anywhere.
Also anything I type in the current window (ie this blog post) only appears when my mouse goes up to the bookmark toolbar or above.
Update It doesn’t do it when running in safe mode so has to be an extension somewhere. Thats going to take some time to debug 🙁
Update 2 This issue is actually caused by New Tab King. Disabling this addon gets firefox working again
fighting working with SharePoint for about a week and trying to get the Search Service started on my SharePoint Server. The only thing that seemed consistent in all the troubleshooting was that the SharePoint error messages were only slightly more helpful than “An error has occurred”. I ended logging a PSS support call with Microsoft and didn’t get very far for a while. My SharePoint farm consisted of the SharePoint Server and a separate SQL server to host the data and attempting to start the service I would get ‘Could not access the Search administration database. A generic error occurred while trying to access the database to obtain the schema version info.’
There are several other posts out there on updating the version of SharePoint to the latest Service Pack, Installing the latest cumulative update(2598321) and ensuring that the protocols were enabled on the SQL instance. All things I corrected, applied and did not fix the issue. (Note that installing the latest cumulative update DOES require a reboot and may stop SharePoint working until you do reboot – so make sure you install this out of hours.)
Upgrading the database is done with ‘psconfig -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures’ After running this command I noticed that the checking the status of the server with (get-spserver servername).NeedsUpdate would work fine on the SQL server, but running against the SQL server from the Sharepoint Server, it would tell me the database needed updating.
On one of my servers, Add/Remove Programs said that the hotfix was not required yet the Admin console on the website said it was. This issue was fixed with a “psconfig -cmd installcheck -noinstallcheck” (Thanks to http://tinyurl.com/7pkrbem)
After starting the service on the sql server instance, we wanted to get the SharePoint Server working as originally intended. After a long time of troubleshooting, our next step was to uninstall the SQL Native Client and reinstall it. As I went to uninstall the Native Client, Add/Remove programs told me the package was not installed. A repair or modify would not work either. Opening Regedit and searching for Native Client under HKey_Classes_Root\installer and deleting this key meant I was then able to reinstall the NativeClient.
We then tried starting the service and this time it worked. The strange thing is that some communications between the SharePoint server and the SQL server were obviously working fine – the database on SQL was created with no problems and SharePoint could see the data – it’s just weird that the initialising/upgrading of the database required the SQL native client but did not give any useful information that pointed to this fact.
Update – Never mind – see bottom of article.
Discovered an interesting flaw in the requirement for two factor authentication with gmail today. I like to use the application on my cell phone to ensure that only I have access to my account – and if somehow a keylogger was in place, my password to gmail is not any use as the 2nd factor authentication would also require access to my cell phone.
However, today I logged into google reader first (which doesn’t support 2nd factor authentication) and used my username and password only. I then clicked the gmail tab at the top of the reader – and hey presto I’m into gmail.
Bottom line – don’t think that just because you have enabled 2nd factor authentication you are safe from keyloggers on a pc or network sniffing/man in the middle attacks. I’ve not reported this to Google yet but it will be interesting to see what they say.
Update After signing out AND restarting firefox I was prompted for the 2nd factor password. Interestingly I wasn’t prompted until I restarted the browser – so as usual – always restart browsers once you’ve finished with them.
After several hours of work today, Powerpoint suddenly gave the error message “PowerPoint was unable to display some of the text, images, or objects on slides in the file, filename because they have become corrupted. Affected slides have been replaced by blank slides in the presentation and it not possible to recover the lost information. To ensure that the file can be opened in previous versions of PowerPoint, use the Save As command (File menu) and save the file with either the same or a new name.”
Now it is all very well giving a really verbose error message, but to totally blank out slides and wipe out missing data is a very peculiar way of fixing the issue. It looks like a hotfix was released in May 2011 but in our case, I saved the file to a usb drive, copied it across to my machine that had office 2010 installed and then opened the file in Powerpoint 2010. I was able to open the file but this time I got another warning about some data being corrupted but the slides that were empty in 2003 were displayed ok. I then resaved the file back to a new filename on the usb drive, opened the new file back in 2003 and we were really relieved to have a working powerpoint file to continue working on.
Not only is the data back, it also means another 4 hours of work does not need to be repeated and instead more time can be spent surfing waves – a great result all around.
I’ve had an issue with a new SBS2008 server, running Exchange service pack3 rollup 2 where the information store service does not start after a reboot, especially annoying after the server is rebooting with a scheduled maintenance task. Apparently this issue was fixed in service pack 1, roll up 5 but I’m still getting it 2 service packs and 2 rollups later.
Microsoft have a “fast publish” knowledge base article 940845 and the first solution is to start the services manually – really helpful! Thankfully there are other solutions that involve changing the dependencies of the services to ensure Exchange does not try to start before AD has finished.
One word of warning – using the Microsoft KB to determine the latest service pack or rollup for Exchange 2007 returns Service Pack 3, rollup 1 from http://support.microsoft.com/kb/937052. However Rollup 2 has been available since Dec 14th 2010. I’ve put a note on the original kb article but the better way to determine the latest rollup is probably to search for Exchange 2007 service pack 3 rollup
Update Knowledge base 940845 now has a fixit file you can download that will change the dependencies for you along with instructions on how to fix it manually. The article no longer has references to this issue being fixed in previous rollups – probably because this was obviously not the case.
We had a weird issue this morning where the Archive options were missing in Outlook 2007. This is apparently a known issue with the kb2412171 December 14, 2010 outlook update which allegedly improves stability. It is hard to see how removing functionality, breaking send and receive and reducing performance when you switch folders comes under the category of improving stability and increasing performance.
Thankfully the solution is simple, just remove 2412171 from add/remove programs and everything should go back to normal.
Further details on the patch are also available which includes the known issues when installing 2412171.
Needless to say, we have unapproved this patch on our WSUS servers.
I was getting the “‘gtLV’ is null or not an object” message when I replied to an email using our Microsoft Online Hosted Exchange email account. Ironically enough, the problem would always occur when I replied to a new email from a Microsoft support engineer. The email would go through but I would get the ” ‘gtLV’ is null or not an object” error message popup on the screen. If I replied to the email again the problem would not occur. A very similar message can be seen in the Microsoft Exchange Server forums where I also posted the provided solution.
After many emails to the very patient support tech at Microsoft (as I would reply and then send an email to let him know if the reply worked or not) we escalated the ticket and I got back the following resolution.
1. type regedit on command prompt or run
2. go to: HKCU\Software\Microsoft\Internet Explorer\Main
3. create TabProcGrowth (string or dword) and set the value to 0
This solution worked for me. From what I can see at the ie8blog this has the side effect of reducing the protectedmode protection and I think the browser tabs use the same process rather than running in seperate processes. This is a slight downside, but I doubt many users will care – they’re more than happy to have OWA working.