ConfigMgr Backup failure - Error 145

We had backup failure on two of our Primaries, the root cause of which seemed to stem from file locks being present on the files in the backup folder while the backup is running.

I scheduled a job to run just before the backup to collect any file handles and dump to a log file (handles.exe) but none showed up. No process had file locks in the backup location, or at least at that time there were none.

Bit more digging and we identified that Software Inventory is running minutes before backup starts. The Inventory Agent is running a SINV scan on the backup location at the same time as the ConfigMgr backup service is started.

This causes the following to appear in the SMSBCK log:

Deleting Z:\SMSBCK\XXXBackup\SiteServer\SMSServer\inboxes\replmgr.box, FAILED, Win32 Error = 145
Failed to delete the contents of the backup folder. Error Code = 0x0
Error: Deleting the existing files in the backup location failed....
<STATUS MESSAGE IS PREPARED AND POSTED TO THE SITE>
SMS site backup failed. Please see previous errors. 

The path in the first line varies between failures, which makes sense as SINV is sweeping through that backup folder and all its files and subfolders.

There are four ways to work this out

1. Move the backup window forward\backward to move around the SINV schedule
2. Switch the SINV schedule from Simple to a full schedule and position it before\after backup takes place
3. Use a Local Policy to alter the SINV schedule on a machine-by-machine basis
4. Alter the schedule directly on a machine-by-machine basis via WMI

We opted for 4 as we didn't want to change the SINV schedule (site wide setting) for the clients assigned to the primary with the backup problem, and, I like using WMI. The easiest option for us is to alter the schedule directly using WBEMTEST on the two primaries that are experiencing the backup failure.

Before you begin, you can switch on Verbose Logging for confirmation that the change took place,  or use WBEMTEST to confirm the change. Entirely up to you, at the end of this write up I link you to the steps for enabling Verbose Logging as well as showing you what will appear in the clients Scheduler log after the change has been made. Both WBEMTEST and the Scheduler log will give you the same confirmation of the change taking effect.

So, here goes, instructions on how to accomplish this task easily.

Usual disclaimer: Don't do this if you don't understand what you are doing. Don't do this in production, familiarise yourself on a test system first. Be careful with WBEMTEST, few clicks and you can accidently delete classes and instances in the WMI repository. Deletions are not too serious for the client, as a Repair from the ConfigMgr Control Panel applet gets the data back in there, and the clients activity will repopulate the repository eventually. Anything not exposed in the GUI or documented by Microsoft is usually unsupported, you enter unsupported land by applying this. Raise a call with Microsoft CSS to get guidance on this if it is an issue.

1. Open WBEMTEST
2. Click Connect
3. Enter root\ccm\scheduler
4. Select Enum Classes with the Recursive option
5. Identify and double click the class called CCM_Scheduler_History
6. Select Instances
7. Identify and double click the instance called {00000000-0000-0000-0000-000000000002}
8. Select Show MOF
9. Put the contents of the displayed MOF in to notepad
10. Add #pragma namespace("\\\\.\\root\\ccm\\scheduler") to the top of the new MOF on a new line to define it as a usable MOF
11. Should look something like this:

#pragma namespace("\\\\.\\root\\ccm\\scheduler")

instance of CCM_Scheduler_History
{
 FirstEvalTime = "20100215220000.000000+000";
 LastTriggerTime = "20100510095149.000000+000";
 ScheduleID = "{00000000-0000-0000-0000-000000000002}";
 UserSID = "Machine";
};

The FirstEvalTime attribute needs to be changed. It's format is YYYYMMDDHHMMSS

Change HHMMSS to the time you want the SINV to run. The reason this works is because this field is used to determine the next run time and is set at installation time so SINV will run on the frequency set in the simple schedule at the time the first SINV ran. This is how Simple schedule works, time of install and frequency of the Simple schedule. Simple. Changing this seed forces the schedule backwards or forwards. Groovy!

Modify the file and save as C:\Change_Schedule.MOF

To import this new MOF follow these steps:

1. Open a CMD prompt
2. Use the following command to stop the SMS Agent Host service
   SC STOP CCMEXEC
   or
   NET STOP CCMEXEC
3. MOFCOMP C:\Change_Schedule.MOF
4. You should see "Success!" displayed if the MOF went in OK. If it didn't, it should have, you've make a typo somewhere, re-read above.
5. Now start the SMS Agent Host service
   SC START CCMEXEC
   or
   NET START CCMEXEC

That's pretty much it. Not too complicated, and using WBEMTEST you can again connect to the repository and actually check the schedule has changed by following the first 8 steps in the procedure above and confirming the time is correct,. Or you can enable verbose logging before you begin the exercise and open the Clients (this is a site server, you'll need to go to <DRIVE>:\SMS_CCM or whereever you've installed the client) log folder and check out the Scheduler.log for something like this showing up:

CSMSTrigger::Init("0001200000100038"):
ScheduleLoadedTime="02/15/2010 22:00:00"
LastFireTime="05/10/2010 09:51:00"
CurrentTime="05/10/2010 13:09:59"
Adjusting next occurrence for DST; subtracting 60 minutes.
CSMSTrigger::Init("0001200000100038") HasMissedOccurrence=FALSE Adjusting next occurrence for DST; subtracting 60 minutes.
SMSTrigger '0001200000100038' will fire at 05/10/2010 10:00:00 PM.
Successfully activated schedule 'Machine/{00000000-0000-0000-0000-000000000002}'.
CScheduleManager::AddSchedule: Successfully added schedule Machine/{00000000-0000-0000-0000-000000000002}

Here's a handy URL to get Verbose Logging enabled on your ConfigMgr client

http://blogs.msdn.com/gabeb/archive/2008/08/06/enabling-debug-and-verbose-logging-in-configmgr-2007.aspx