Remove IIS servers from SCOM monitoring
In our environment we have alot of Windows servers running IIS (Internet Information Services). Many of these servers do not need to have IIS monitoring enabled in SCOM because they are development or test servers that have their IIS services stopped, restarted, or specific sites stopped at random by developers and engineers. So, the problem I am faced with is removing these servers from IIS monitoring in SCOM (we’re using the Microsoft-provided IIS management packs) while still getting things like OS-related alerts.
To do this, create a group in SCOM under the Authoring tab. Use explicit membership and add each Windows computer object for the servers running IIS that we don’t want to monitor. Then, we need to set overrides on the object discoveries. Click on Object Discoveries in the Authoring section, and search for the following three object discovery rules:
- Windows Internet Information Services Server Role Discovery Rule (for Windows 2003/IIS 6 servers)
- IIS 7 Server Role Discovery
- IIS 8 Server Role Discovery
Right-click each of these rules, click “Overrides – Override the Object Discovery – For a group…” and select the group created in the previous step. Check the override for the “Enabled” parameter and change the value to “False.” The next time the object discovery runs, it will remove IIS monitoring for the servers in the group. (Note: By default, the object discoveries for IIS on Windows 2003 and 2012 run every hour, but for 2008/2008 R2, they run every 4 hours.)
BUT WAIT…there’s more! After the discovery runs and the IIS objects are disabled, you’ll notice that they still show up in the views, and old alerts will remain in the console. There is a powershell cmdlet you need to run in order to clean these up. Open the Operations Manager Shell and run the following:
- SCOM 2007: remove-disabledmonitoringobject
- SCOM 2012: remove-scomdisabledclassinstance
You’ll receive a warning that the operation will permanently delete class instances and relationships from the database and it can take a significant amount of time to complete. I haven’t experienced any long delays in my environment of over 700 agents, it typically takes less than one minute to complete.
If you need to add IIS monitoring back for one of these servers, simply remove it from the group you used to disable monitoring, and the IIS objects will be added back to SCOM on the next discovery cycle.
WMI Errors in Application Log After Every Reboot
On some of our servers running Windows Server 2008 R2, we’ve noticed a specific event regarding WMI showing up in the Application event log after every reboot. When we reboot, we get the following:
Log Name: Application
Source: Microsoft-Windows-WMI
Date: <Date>
Event ID: 10
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: <ServerName>
Description:
Event filter with query “SELECT * FROM __InstanceModificationEvent WITHIN 60 WHERE TargetInstance ISA “Win32_Processor” AND TargetInstance.LoadPercentage > 99″ could not be reactivated in namespace “//./root/CIMV2” because of error 0x80041003. Events cannot be delivered through this filter until the problem is corrected.
Event Xml:
According to Microsoft, this issue is normal and can be safely ignored. There is a fix available as well, which will remove the error from the event logs if needed. See the following article for more information:
http://support.microsoft.com/default.aspx?scid=kb;en-US;2545227
Also, if you are running Vista or Windows Server 2008 with SP1, you may receive the same error, which can be fixed by running a script. Here’s the Microsoft KB article regarding that fix:
Strange SCOM CScript.exe Errors On Domain Controllers
I recently encountered a domain controller running Windows Server 2008 R2 in our Active Directory domain that was recording a strange event in the Application event log every five minutes. The event information was:
Log Name: Application
Source: Application Error
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Description:
Faulting application name: cscript.exe, version: 5.8.7600.16385, time stamp: 0x4a5bca2a
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc00000fd
Fault offset: 0x0000000000053560
Faulting process id: 0x25ac
Faulting application start time: 0x01cdffe08a2d9a20
Faulting application path: C:\Windows\system32\cscript.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: c8dd9c2a-6bd3-11e2-89a7-2c768a515630
After doing a little research online I discovered that there was a problem with one of the AD monitoring scripts in System Center Operations Manager 2007 R2. In my case, it was the AD_General_Response.vbs script. It appeared to be related to information this script reads and writes to/from the registry. The registry path in question can be found at:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Modules\{D0C83F7B-F7B6-91F9-47DE-B63140166447}\S-1-5-18\Script\AD Management Pack\AD General Response
(Note that the unique identifiers may vary on your system.) In this path there are two keys, ErrorCount and ErrorDescription, that contained data. ErrorCount contained a number, and ErrorDescriptions contained a long string of data that was too big to see. It is this long ErrorDescriptions key that I believe contained faulty data and needed to be cleared to prevent these errors from being logged. After backing up the registry (always back up your registry before making registry changes!) I deleted the values in both of these keys. I noticed after doing this that within 5 minutes, the ErrorCount key was repopulated with the value “0”.
Note that the ErrorDescriptions key, when I opened it, didn’t contain any text in the box, but if I held down the delete key for several seconds and then hit OK, it cleared the data from the value. Again, something was definitely unusual about this key.
SCOM Domain Controllers Alert Script Based Test Failed to Complet
If you are using the Microsoft Active Directory management pack in System Center Operations Manager 2007 R2, you may encounter some strange alerts on a new domain controller after adding it to your domain. Specifically, you may receive “Script Based Test Failed to Complete” alerts. There will probably be two alerts that show up, the description for these alerts will state:
“AD Lost And Found Object Count: The script ‘AD Lost And Found Object Count’ failed to create object ‘McActiveDir.ActiveDirectory’. This is an unexpected error. The error returned was ‘ActiveX component can’t create object’ (0x1AD)”
“AD Database and Log: The script ‘AD Database and Log’ failed to create object ‘McActiveDir.ActiveDirectory’. The error returned was: ‘ActiveX component can’t create object’ (0x1AD)”
These scripts run every five minutes, and you will see corresponding errors in the OperationsManager Event log on the domain controller that is raising the alerts.
More than likely, the problem domain controller had a manual installation of the Operations Manager agent, rather than a push from the management server. When you do this, it prevents the Active Directory Management Helper Object for SCOM to be installed, which is automatic for agents that were pushed from the management server. This helper object is necessary for these scripts to run properly.
You can manually install the helper object by copying the “OomADs.msi” file from the management server’s “C:\Program Files\System Center Operations Manager 2007\HelperObjects” folder to the domain controller and running the MSI installer. My experience has been that no reboot is required and the installation should only take a few seconds or so to complete.
After installing the helper object, you can manually close the associated alerts in the Operations Manager console.
Citrix Alternate Addresses
I ran into a problem recently with a Citrix XenApp 6.5 farm we have setup for external users in an overseas office. These users access the Citrix farm via external IP addresses which are setup on our firewall. The internal IP address of the Citrix XenApp server is different than the external IP address that the clients overseas use to access the published applications. This caused the connection to the published applications to fail because the Citrix Web Interface server was sending the clients to the internal IP address rather than the external address.
In order to make this work, you have to first assign an alternate (external) IP address for the servers running XenApp using the ALTADDR command. The alternate address is returned to clients that request it and is used to access a server that is behind a firewall. Before this will work, though, you also must setup the Citrix Web Interface to use the alternate address.
So, as an example, let’s say that you have a Citrix XenApp server hosting published applications and is assigned the internal IP address of 192.168.1.10. Your firewall is configured to allow access to this internal IP address via the external public IP address of 100.1.1.10. The first step to make this work is to use the ALTADDR command line utility on the Citrix XenApp server to specify the alternate IP address of 100.1.1.10. The syntax would be:
altaddr /set 100.1.1.10
The next step is to configure the Citrix Web Interface to use this address. Now, let’s say that this XenApp server is accessed by both clients on your internal network and clients at an overseas location. Assume that the overseas office uses the subnet 10.2.2.0/24 for their PCs. We can setup the Web Interface server to use the alternate address for the overseas office, and all other clients will use the internal (default) address.
On the Citrix Web Interface server, launch the Citrix Web Interface Management console. You should have two options on the left side, XenApp Web Sites, and XenApp Services Sites. Select the site that you want to setup for alternate addresses, right-click the site name and click “Secure Access”. Click the “Add…” button, and using the IP subnet from our example, you will enter 10.2.2.0 for the IP address and 255.255.255.0 for the Mask. In the “Access Method” drop-down box, select “Alternate” and click OK. Click Finish.
Now when a user on the 10.2.2.x subnet attempts to access a published app, the web interface server will send the ICA session to 100.1.1.10 instead of 192.168.1.10.
Exchange Management Shell failures
Ran into a problem on an Exchange server today when launching the management shell and management console. The error was similar to the following:
The following error occured when searching for On-Premises Exchange server: [servername.company.com] Connecting to remote server failed with the following message: The WSMan client cannot process the request. Proxy is not suported under http transport. Change the transport to https and specify valid proxy information and try again. For more information, see the about troubleshooting help topic.
After searching the web, I found the solution. I had to browse to the following registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\Connections
The WinHttpSettings key contains some sort of proxy information for the system. I deleted this key and relaunched the management shell and the problem was fixed.
I found this solution at the following Microsoft Technet forum:
Citrix SQL Gotchas
Every Citrix XenApp farm requires a data store, which is basically a SQL database. The database can exist on either a fully-licensed Microsoft SQL server implementation, or a freely available (and included) SQL Express server. When running Citrix XenApp setup, you can opt to setup the included SQL Express edition. However, you may wish to do a manual installation of SQL Express for various reasons. If you choose to do a manual installation, there are a couple of things you will need to know.
First of all, when setting up a new SQL Express installation, always choose a named instance, and use the name “CITRIX_METAFRAME” whenever possible. This is the default choice for an out-of-the-box Citrix SQL Express installation, and makes things easier. When you create the new data store during XenApp installation, it should automatically create the MF20 database for you.
When you are attempting to join a new Citrix XenApp server, you may encounter a “Network-Failure” error when testing the connection (during the join process). If you encounter this issue, it is most likely due to the configuration of SQL Express on the data store server. By default, the TCP/IP and Named Pipes protocols are disabled on a new SQL Express 2008 R2 installation. To change this, on the SQL Express server hosting the data store, launch the SQL Server Configuration Manager (found under the start menu, under Microsoft SQL Server 2008 R2\Configuration Tools), and expand the SQL Server Network Configuration node. Select “Protocols for NAMED_INSTANCE” (should be CITRIX_METAFRAME), right-click on TCP/IP and Named Pipes and select “Enable”. It should look like this:
When you are finished you should restart the SQL services, then attempt to join the new XenApp server to the farm again.
Citrix XenApp IMAService Login Failed
On a new Citrix XenApp 6.5 farm that I built recently, I noticed that I was no longer able to connect to the farm using the AppCenter console. When doing a discovery, I received errors that it could not connect to the data store. I checked the server hosting the Citrix farm’s data store, and everything appeared to be ok. So, I checked the event viewer on one of the servers running Citrix XenApp, and discovered a recurring error in the System event viewer:
Log Name: System
Source: IMAService
Event ID: 3989
Description: Citrix XenApp failed to connect to the Data Store. ODBC error while connecting to the database: 28000 -> [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed. The login is from an untrusted domain and cannot be used with Windows authentication.
I opened the DSN file located at C:\Program Files (x86)\Citrix\Independent Management Architecture named “MF20.DSN” with Notepad and noticed that the UID value was set to my domain user account, which I had used to install the Citrix farm’s components. Since I have to change the password on this username frequently (and had recently changed it), I decided I should change this to a domain service account with a password that never expired. To do this, I had to use the dsmaint command line utility. The syntax is as follows:
dsmaint config /user:<username> /pwd:<password> /dsn:”C:\Program Files (x86)\Citrix\Independent Management Architecture\mf20.dsn”
You should get a notification that it “Successfully connected to the data store” and that “Configuration successfully changed.” Restart the IMA service and everything should work again.
Understanding how AutoComplete works in Outlook 2010
Microsoft Outlook will automatically suggest a recipient in the To, CC, and BCC fields of a new email message based upon previous recipients that you have sent mail to in the past. In versions prior to Outlook 2010, these were stored in an NK2 file that was stored in your user profile on the PC where you were running Outlook. Starting with Outlook 2010, this auto-complete cache is stored in your Inbox on the Exchange server (if you are running in a Microsoft Exchange environment) so that the auto-complete cache will contain the same information when using OWA, Windows Mobile, or Outlook 2010 on another PC.
I have been asked “How does the Auto-Complete work regarding what information it searches against?” For example, if you type certain information, such as the display name, or the primary email address, it suggests the correct recipient. However, if you type other information, such as last name, it does not. I did some research on how this works, and here is what I found.
Let me start by defining some terms. Auto-Complete is the feature that accesses a cache file that contains the recipients you’ve sent to or resolved already, it doesn’t search live against AD or the GAL. It refers to the pop-down menu when you type on the “To” or “CC” line in a new message, as shown in the following screenshot:
Auto-Resolve refers to the feature that resolves the recipient’s name when you hit either “CTRL+K” or “tab” when typing information on the “To” or “CC” line.
Auto-Complete Behavior
By default, the Auto-Complete cache in Outlook is empty. It starts recording addresses in the cache as soon as you start resolving them (using auto-resolve via CTRL+K, or by inserting addresses from the Global Address List or Contacts). These resolved addresses are only stored in the auto-complete cache during the current Outlook session and are purged as soon as you exit the application, unless you actually send a message to these recipients. Once you send a message to them, they are kept in the auto-complete cache semi-permanently (more on this later).
Consider a user named Jane Doe. Her username is jdoe, her primary email address is jane_doe@domain.com, her Exchange Alias is janedoe, and her display name is “Jane K. Doe”. When searching for a recipient based on text you type in the “To”, “CC”, or “BCC” fields, auto-complete looks at the following fields:
- Display Name (i.e. Jane K. Doe)
- Primary SMTP E-mail address (i.e. Jane_Doe@domain.com)
- Exchange Alias (i.e. janedoe)
- legacyExchangeDN attribute in Active Directory (more on this later)
Auto-complete does NOT search against any of the following criteria:
- First Name (if it is different from the display name, for example if the display name contained a nickname instead of the legal first name, which is listed in the recipients’ “first name” field in Active Directory)
- Last Name (unless one of the above four fields starts with the last name)
- Username (unless one of the above four fields starts with the username, or is set to match the username)
Auto-Resolve Behavior
Auto-Resolve can query multiple fields beyond the limitations of the fields that Auto-Complete works with. There is a good chart of the fields that Auto-Resolve uses when Outlook is in offline mode vs. online mode here.
How long is a recipient’s information kept in the Auto-Complete cache?
In Outlook 2003 and 2007, the cache file (NK2 file) could store up to 1000 recipients in it. Once it hit this limit, it would start to purge old recipients in the order that they were added (keeping the most recently used). I have been unable to find out if the Outlook 2010 cache has a similar limit. If anyone has any information regarding this, I would encourage you to leave it in the comments.
More information
For information about clearing the Auto-complete cache:
http://support.microsoft.com/kb/287623
http://aspoc.net/archives/2010/05/04/how-to-clear-the-autocomplete-list-in-outlook-2010/
For information on the legacyExchangeDN Active Directory attribute:
http://www.msexchange.org/tutorials/Understanding-LegacyExchangeDN.html
Configure SCOM to alert when new shares are created
My company, like many other companies, are under strict regulations from governments at various levels to control who has access to sensitive data. We not only have to worry about who can access data on our network from outside the corporate firewall, but also users who are internal. One of the most difficult struggles we have had is controlling who has access to shared folders on our internal servers.
Since we delegate administrative privileges to many of our servers to other groups in IT (such as DBAs or Application Developers), it is a frequent occurrence that one of these individuals will create a new file share but not lock it down properly. As you may know, shares created on a server running Windows Server 2003 are, by default, open to “Everyone” with read-only access. So, how can you get some control on this situation? You can beat systems administrators over the head with policies until your arms are tired, or you can configure SCOM to send an alert any time a share is created on any of your monitored servers.
SCOM can be configured to monitor for WMI events in addition to events in the Windows event logs. Fortunately for us, a WMI event is triggered every time a shared folder is created in Windows. Unfortunately for us, the WMI event looks the same no matter what is shared, so SCOM will not differentiate between a new shared printer and a new shared folder. However, if you are like me, you won’t mind getting notified when printer shares are created, as long as you are also notified when a folder is shared.
I want to make it clear that I didn’t come up with this on my own, I simply pulled information from other sources on the Internet and used what I needed to accomplish my goals. The following two blog posts were very helpful for me:
Now, here’s what I did:
- We’re going to be creating a rule. Launch the Operations Manager Console UI, go to Authoring, select Rules.
- Right-click Rules, select “Create New Rule.”
- On the Created Rule Wizard, expand Collection Rules >Event Based > WMI Event and click Next (don’t forget to select a custom management pack other than the “Default Management Pack”.
- Enter a Rule Name (I used “New network file share created”) and if you would like, you can also enter a description.
- On Rule Category, select Alert.
- For Rule Target, click Select, click on View All Targets, and select Windows Computer
- For WMI Namespace, type root\cimv2 and for Query, enter the following:SELECT * FROM __InstanceCreationEvent WITHIN 120 WHERE TargetInstance ISA ‘Win32_Share’
- Set the Poll Interval to 120 seconds.
When finishing out the rule, you’ll want to configure this rule to have a response of an Alert. Here was the tricky part for me. I needed to get useful information out of the WMI Event to put into the alert so I could make sense of what the alert was telling me. Specifically, I wanted to know the name of the share and the local path to the folder on the server. Configure your alert to look like the following screen shot:
After configuring the alert, I setup a subscription in SCOM for the alert to email me whenever a new share is created. Now I am able to check new shares when they are created to make sure they are locked down with the appropriate permissions.
The Server Monkey's Blog 