Category Archives: OSD / SCCM / MDT

Securing Web Services

One of the main criticisms of using web services is that they’re inherently insecure. By default anyone can access them and if they have functions to actually change anything then one must proceed with caution.

My current client was somewhat skeptical about their introduction and the only way I could bring them round to the wonderful gifts that they offer was to promise that we’d investigate a secure way to present them. There was some trial and error but I think we came up with a pretty good solution which I shall share here.

Why use web services at all?

There are an awful lot of reasons so I’ll keep it to why I like to use them. They’re a cheap and cheerful way to provide functionality by proxy.  They can be used to off-load many tasks which, if you don’t have the budget for something like the wonderful System Center Orchestrator product, can make a great shoe-in. OK a little more explanation…

From a deployment perspective they can provide a mechanism to access Active Directory, MDT and SCCM without a client OS necessarily being part of AD. For example, they could be called from a WinPE session to update a database or query/update an OU. They’re relatively easy to write if you possess basic programming skills but if not, then I fully recommend you check out Maik Koster’s toolset here.  Installation instructions are provided but do fall a little short on security, so let me crack on.

Securing the Web Services

First of all, if you run a PKI infrastructure, let me recommend you you change the URL to run under HTTPS. I am not going into detail in this respect here as there are plenty of how-to’s on this topic elsewhere on the web. Suffice to say it’s a no-brainer if you’re truly concerned about security, particularly if there are any services which need to pass confidential information such as passwords.

Next, ensure you have read Maik’s security blog for his web services. They’re basic but a good start. Now to secure it properly:

Securing the website via pass-through authentication

Follow these steps to lock down page to an AD group. I won’t go into detail on  clicky-clicky, I assume if you have come this far you’ll know this stuff from within IIS and from the screen shots provided. If enough people tell me otherwise, I’ll review this though.

  1. Before changing anything, this is the expected configuration:

2. Install url authorisation feature and windows authentication features from server manager or Powershell.

3. After installation, change authentication model as follows (apologies, image is a little blurry, I’ll try to update in due course).

4. Update Authorisation rules. Note that All Users verbs have been changed to POST. This prevents the web page appearing at all without a login prompt (ie the initial GET action is prevented from running) for all users other than those in that are members of the specified AD group.

 

5. Providers should remain at their defaults:

6. Update local Intranet sites. If the site isn’t trusted you may need to add this to local intranet sites to prevent a login box appearing.

 

You should now have full pass-through authentication for your web service, dependent on membership of the AD group of your choice.

SCCM Client “Currently Intranet” doesn’t change to “Currently Internet”

Environment: SCCM 1602, full HTTPS communication throughout.

I ran into this problem recently at a client where we’d installed SCCM 1602 with full HTTPS communication throughout. One of the requirements was to deploy software and software updates to clients on the internet as well as the intranet. All went pretty much according to plan until I put a laptop on the internet to test deployment of said software.  The issue I faced was that whatever I did, I couldn’t make the laptop drop to Currently Internet from Currently Intranet. Looking at the LocationServices.log confirmed my suspicions: it was trying to contact an MP on the internal network.

Attempting to refresh TRK from AD LocationServices 22/06/2016 16:27:50 3024 (0x0BD0)
Unexpected row count (0) retrieved from AD. LocationServices 22/06/2016 16:27:50 3024 (0x0BD0)
Failed to get TRK from AD LocationServices 22/06/2016 16:27:50 3024 (0x0BD0)
Failed to send request to /SMS_MP/.sms_aut?MPKEYINFORMATIONEX at host ICSKSCCMMP02.local.com, error 0x2ee2 LocationServices 22/06/2016 16:28:11 3024 (0x0BD0)
[CCMHTTP] ERROR: URL=https://ICSKSCCMMP02.local.com/SMS_MP/.sms_aut?MPKEYINFORMATIONEX, Port=443, Options=63, Code=12002, Text=ERROR_WINHTTP_TIMEOUT LocationServices 22/06/2016 16:28:11 3024 (0x0BD0)
Raising event:
instance of CCM_CcmHttp_Status
{
ClientID = “GUID:0abd2d73-79a1-4b55-91de-1bc56d93050c”;
DateTime = “20160622152811.098000+000”;
HostName = “ICSKSCCMMP02.local.com”;
HRESULT = “0x80072ee2”;
ProcessID = 2948;
StatusCode = 600;
ThreadID = 3024;
};
LocationServices 22/06/2016 16:28:11 3024 (0x0BD0)
Successfully queued event on HTTP/HTTPS failure for server ‘ICSKSCCMMP02.local.com’. LocationServices 22/06/2016 16:28:11 3024 (0x0BD0)
MP ICSKSCCMMP02 capability is not available LocationServices 22/06/2016 16:28:11 3024 (0x0BD0)
Executing Task LSRefreshDefaultMPTask LocationServices 22/06/2016 16:28:12 2136 (0x0858)
Current AD site of machine is North-West LocationServices 22/06/2016 16:28:12 6028 (0x178C)
Failed to send request to /SMS_MP/.sms_aut?MPKEYINFORMATIONEX at host ICSKSCCMMP02, error 0x2ee2 LocationServices 22/06/2016 16:28:32 3024 (0x0BD0)

 During my investigations I looked into what criteria ConfigMgr used to discover if it was on the internet and found the answer here:

When the client detects a change in network, this kicks off service location to find its intranet management point (the default management point in its assigned site or proxy management point if it’s within the boundaries of a secondary site that belongs to its assigned site).  If service location fails, the client deduces that it must be on the Internet and so tries to communicate with its assigned Internet-based management point.  The assigned Internet-based management point always directs the client to the Internet-based site systems in the site, and never to intranet-based site systems or to Internet-based site systems in another site.

So I looked at my default management point. This was also set (via an alias) as the internet management point (owing to the IP policy here they don’t really ‘do’ DMZs but that’s another story). So what was happening? Well essentially when the local network was disconnected and the computer was switched over to an internet connection, it wasn’t able to differentiate the default MP from the internet MP and hence thought it was still on the local network.

The Solution

I changed the default Management Point from the internet facing MP to a local MP that wasn’t accessible via the internet. This allowed the client to figure out that it was no longer on the local network and change over to the internet. Once this happened, it was then able to pick the correct MP and the correct DP to talk to and order was once again restored. I guess this isn’t a common scenario but something to look out for if you’re experiencing the problem described.

SQL Server AlwaysOn Availability Groups, SCCM and MultiSubnetFailover

EDIT: The issue below has finally been addressed in version 1906:

https://docs.microsoft.com/en-gb/sccm/core/plan-design/changes/whats-new-in-version-1906#improvements-to-support-for-sql-always-on

——————————————————————————————————-

At the time of writing there appear to be very little on the internet about this particular configuration.

In short, we needed to use the MultiSubnetFailover option string with our SQL AOAG making up the CM2016 back end. We have two SQL DBs, one in London and one in Slough. As one might typically expect, these exist on separate subnets.  After hunting around the internet to try to ensure the supportability of this configuration it quickly became clear that a call to Premier Support was required to confirm. In particular, we were interested to know whether the ODBC System DSN driver should remain the same on the site server. By default this uses an old version which doesn’t have the MultiSubnetFailover checkbox available. We figured this might need changing to the newer 11.x native client when AOAG were used over separate subnets. There was an awful lot of to-and-fro between ourselves and Microsoft and it was clear this was an area where there was little clear knowledge. We did finally prize an answer out of them however and the following is an edited transcript of the outcome:

  1. We wish to know if SCCM 1511 and 1602 should be able to talk to such a multi-site SQL cluster OK?

ANS: Multi-site Cluster is supported but it`s a complex solution to implement. It is heavily dependent on the Hardware Supportability and Networking. Storage should also be carefully considered:https://blogs.technet.microsoft.com/meamcs/2013/11/09/microsoft-windows-multi-site-failover-cluster-best-practices/

Configurations for the SQL Server Site Databasehttps://technet.microsoft.com/en-us/library/gg682077.aspx#BKMK_SupConfigSQLDBconfig

  1. Should we make SCCM use the “SQL server native client V 11.0” ODBC connection “system DSN” connection which has a check box that seems to have “multisubnetfailover” option or the default SCCM installed “SQL driver” for ODBC connection?

ANS: We should not be making any changes to the default configuration made as part of the ConfigMgr installation. ConfigMgr does not support SQL Server Multi-subnet clustering.

  1. If we upgrade SCCM from 1511 to 1602  (or to a later version in future) will the upgrade make the site server use the “SQL driver ODBC” connection again, needing a manual change to SQL native client again?

ANS: We should not be altering the default configuration at ODBC

  1. [PS Engineer] wrote the following “…SCCM Components make use of dll’ (some components) andSQL Server native Client both.  Native Client will get updated as per SCCM need, and we can leave sqlsvr32.dll’ as its default version.”

Our response: But slqsvr32.dll, does it have the multisubnetfailover option? We cannot see this. Where and how do we configure ALL the components to use multisubnetfailover on primary site server and other component servers as appropriate?

ANS: SQL Server Multi Subnet cluster is not supported.

Always On Availability Groups in SCCM 1511

There has been a lot of talk about the use of SQL AlwaysOn Availability Groups (AOAGs) from SCCM 2012 onward but it wasn’t available in any iteration of this version and promises were made by Microsoft to include this is as a feature in 1511. Indeed it seems to have been planned and implemented in the Technical Previews but evidently didn’t make it into the final official release of 1511. Given that officially* you need to go via 1511 to install 1602 this is very annoying for anyone wishing to use this feature as a number of extra steps are required. Below I try to clear up much of the contradictory information available at present (May 2016) on the internet.

A number of websites run by respected MVP’s continue to suggest availability in several 1511 versions:

https://4sysops.com/archives/system-center-configuration-manager-2016-tp3/ which suggests it is available in TP3.
And this from another fairly well respected site:
https://www.systemcenterdudes.com/sccm-1511-new-features/

In a nutshell, it seems AOAGs are supported only on 1511 Tech Preview 4 and above but to confuse matters, the following from Microsoft explains this definition:
https://technet.microsoft.com/en-us/library/mt595861 (snippet below)

Technical Preview 1511 – This version is also known as the Configuration Manager Technical Preview 4. It represents a baseline for Technical Previews that begin with the release of the current branch for System Center Configuration Manager, version 1511…

This is where it gets interesting because Technical Preview 4 covers a number of separate releases which flank the ‘official’ 1511 release on Dec 8th 2015:
https://buildnumbers.wordpress.com/sccm/
Well that that clears it up then.

After filtering through a good deal of contradictory information, it appears that SQL AOAGs made it into the 1512 TP4 version and that 1602 is the first official CB release to have full support.
So to be clear, when installing 1602 and above from scratch, install 1511 first by pointing to the single DB instance then change to AOAGs later via the procedure below.

https://blogs.technet.microsoft.com/arnabm/2016/05/14/step-by-step-configuration-manager-site-database-hosted-on-sql-server-alwayson-availability-group/

If you don’t, and you are using the Dec 8th (official) release of 1511 then when adding the AV Listener you’ll see the following error message:

AVG_1511

And nobody likes to see that 🙂

 

* I hesitate to say this but it is perfectly possible to install 1602 direct from certain media sources** for 1602. This is not supported and therefore not recommended, I am informed one of the side effects of doing this is that you’ll not be able to update in future. There may be other, undocumented side effects too.

** Media I obtained came from an unconfirmed source although it is possible it may have been from CD.Latest (I didn’t originally source it so can’t confirm for definite). Basically don’t do it if you want a supported environment!

Unable to open SCCM admin console following 1511 upgrade

ENVIRONMENT:
SCCM 2012 R2 SP1 with SCCM database on the same server.

So in a lab I upgraded my SCCM 2012 R2 SP1 instance to 1511. This is a Hyper-V environment running on my Surface Pro and I have SCCM, SQL and the SCCM admin console  running on the same VM.
The upgrade itself appeared to go smoothly but when I opened the admin console I just got the grey screen telling me I was unable to connect and that I should check I had permissions and that the SMS provider was installed and available, etc. Obviously it was, it was on the same machine and I had been working with it 20 minutes beforehand.

My immediate reaction was something had gone wrong with the DB upgrade. I hadn’t run the /TestDBUpgrade switch as this was very much a test environment and I didn’t have another server I could quickly and easily test it with. I did however take a snapshot and reverted back. A few tweaks here and there and a re-run of the upgrade produced the same result. Then I decided to test whether this was actually a problem with the installation or simply a problem with the console.
I decided to try and build a machine off this install (PXE boot) and sure enough it built perfectly.
Next, I installed the 1511 console on another VM (my lab DC in this case) and sure enough again, it connected perfectly.
Great.
Next step, check the console logs, in particular the SMSAdminUI.log file under C:\Program Files (x86)\Microsoft Configuration Manager\AdminConsole\AdminUILog. Under here I noticed that there were a number of performance counter errors eg as below:

[1, PID:844][04/08/2016 11:11:59] :The performance counter ‘# images’ was not found
[4, PID:844][04/08/2016 11:12:01] :The performance counter ‘# result objects in memory’ was not found
[4, PID:844][04/08/2016 11:12:01] :The performance counter ‘# exceptions’ was not found

LogCapture

SOLUTION
Seems the upgrade broke the performance counter settings in some way so this needed fixing:
1. (Important) Open a command prompt and CD\Windows\SysWow64
2. LODCTR /R (ensure you are in the dir path above or you are likely to get error code 2).
3. Reinstall the console from the 1511 upgrade setup.
4. Et Voila, (hopefully) all should start working again.

The task sequence execution engine failed executing the action (Gather) in the group (State Restore) with the error code 2147942402…The operating system reported error 2147942402: The system cannot find the file specified.

Thought I’d do a quick post on this as there is precious little to be found in forums about this task sequence ‘bug’. I say bug, really it’s probably just lazy programming (I haven’t checked the specifics but the evidence leads me to believe this is the case).

Anyway…the scenario is this:

ENVIRONMENT:
SCCM R2 SP1
MDT 2013 U1

MDT-infused Client Task Sequence goes all the way through until it hits the Gather step under the State Restore phase. At this point you are hit with:
Gather

In this case the Set Variable For Drive Letter step under the Install section had set the OSDPreserveDriveLetter task sequence variable to TRUE which caused my OS disk to be set to X: for some reason! I suspect the same may happen if the OS had been installed on another drive other than C: but I haven’t tested this so can’t say for sure. Equally it appears to happen only with Gather Only Local Data. In any case the fix is to ensure you are installing to C:

Software Updates My Way :-) (plus a useful tool to help out)

There are a 101 ways of deploying software updates out there and you’ll undoubtedly get a different answer on the best way to deploy updates from every ConfigMgr admin you ask. Indeed it obviously depends on the environment and company politics how best the solution is implemented. This is my tuppence worth.

As an admin, you have enough to do in setting up the required updates in the first place. In many environments you’ll then have to raise the necessary change requests and all the headaches that entails. Then there is the deployment itself:
“Why did it install that?”
“Why didn’t it install that?”
My solution to this is to move the onus to the application or server owner to set up when their computers are patched.

1, STRATEGY:

Select you maintenance window(s). Ensure the window is of sufficient length to take into account all the updates that need deploying. If a ‘hard’ maintenance window is in force, bear in mind you might not finish installing and rebooting before it closes if it is set too short. For my example I will select three per day for different deployment types. This doesn’t mean there will necessarily be computers populated in all three collections, but the option is there. Agree these with all interested parties first. For example:

1800-2000 (Special applications)
2300-0100 (Workstations)
0300-0500 (Servers)

2. DURATION:

By duration I’m talking about how long to complete the deployment to the enterprise. For me, there are two scenarios here. I will typically roll out patches to the servers and/or workstations in my test domain before rolling out to live. The duration for the full rollout is 14 days for each environment (test/non prod and production). This (usually) leaves a few days in between patch Tuesday to download the new updates and deliver the update list for the coming month to interested parties to approve, before rollout on the following Sunday (‘Patch Sunday’). In many situations you may not have the advantages of a test environment to hand. In these situations you may potentially argue that a longer rollout period was necessary. However there are few situations I have ever seen where a 14 day cycle isn’t sufficient if proper testing is put in place.

3. DEFER OWNERSHIP:

As alluded to earlier, IMO the onus should be on the SCCM admin to set up the job but after that it should be up to the application owner or server team to administer when their servers are patched. Ideally the patch window for the server should be set at the server build stage and should be specified in the original request.
My approach to this is to ensure all the software update collections have a query rule assigned which defines a particular AD group that can be updated by the interested party. Responsibility then falls squarely on the owner for any problems related to when the machine is updated.

4. AD GROUPS, QUERIES AND COLLECTIONS
This is how I set up my collections in SCCM along with the queries and groups.
EXAMPLE AD GROUP:
Name: PR1-SCCM PRD-07-2300-0100 (Site PR1, Prod environment, day 7, 2300-0100)
EXAMPLE COLLECTION:
Software Updates – WK1 – Saturday – 2300-0100 or Software Updates – WK2 – Wednesday- 2300-0100
EXAMPLE QUERY RULE:
select SMS_R_SYSTEM.ResourceID,SMS_R_SYSTEM.ResourceType,SMS_R_SYSTEM.Name,SMS_R_SYSTEM.SMSUniqueIdentifier,SMS_R_SYSTEM.ResourceDomainORWorkgroup,SMS_R_SYSTEM.Client from SMS_R_System where SMS_R_System.SystemGroupName = ‘DOMAIN\\PR1-SCCM PRD-07-2300-0100’ and SMS_R_System.Name not in (select distinct SMS_R_System.Name from SMS_R_System where SMS_R_System.SystemGroupName = ‘DOMAIN\\MW Exception Group’)
Note:
DOMAIN\\MW Exception Group represents a group that a computer can be dropped into if for some reason it needs removing from its usual schedule temporarily.

5. NO REBOOT GROUP
In addition to the three daily maintenance window collections, another daily collection I have set up is one with no reboot. Indeed this is of particular interest to certain application owners who only have a narrow window where servers can be down. They take the responsibility on themselves to manually reboot these machines at a time convenient to them.

6. ADMINISTRATION
So…if should you decide the above all sounds like a great idea (that’s a big if, but go with me here) we’re talking about over 60 collections to deploy updates to. That’s going to take all week to set up isn’t it, particularly if there are a lot of update to deploy? Well yes, it is. If done manually. Powershell would traditionally be your friend here and when I first started this, that is exactly what I was using. What I found in practice though was even with this I wanted to see what was going to happen first and again you could write scripts to do this too. What I decided on though was to write a tool that would do everything for me. When I say everything, I am including creating collections, hard maintenance windows (if required), AD groups, queries and of course the selection, deployment schedule and deployment of the updates themselves. This tool is adaptable for different environments and should make the whole process very simple. Whilst I take no responsibility for how you use it in your environment, I have used it in live and prod environments at clients I have consulted and all sorts of different test environments. I include the download and source code at the end of this blog.

7. TO WRAP UP

This blog has detailed my specific approach but the high-level is really what it’s about. As I mentioned in my opening paragraph, one size does not fit all but I do believe some of the pressure (and work) can and should take a shift sideways. Some computers you’ll most likely always have to deal with, eg workstations/laptops. The systems that cause the most headaches though are the servers and I firmly believe ownership can and should be shifted here. It won’t work everywhere but it’s all about changing culture.

Toolio_SoftwareUpdateTool_Instructions

ToolioSoftwareUpdateTool_Source_v2.9.1 (C# source code)

Toolio_Installer_v2.9.1 (exe)

Let me know any issues you have and hopefully I’ll be able to help you out. Please make sure you test before putting into production and I take no responsibility for use!

 

SMS_MP_CONTROL_MANAGER – Error communicating with component

Had this recently in an SCCM 2007 environment. After a bit of poking around I decided to reinstall the management point but this failed to fix the problem. Looking in the MPSetup.log is the error:

Fatal MSI Error – mp.msi could not be installed

After ensuring all the WebDAV setting were correct I started to scratch my head a bit before seeing one or two forum entries which suggested fairly major surgery but crucially included the step to run CCMClean. Well, this did make me think and I checked the SCCM client on the primary site. This was a lab environment and there was an overlapping boundary which caused a CM2012 client to be installed on this box.

Resolution:

I simply removed this client, removed and reinstalled the MP and everything was fine.

Patching workgroup computers and template VMs in CM2012

I recently needed to update the templates of our VMWare servers to reduce the amount of patching required after each deployment. IMO it’s good practice to do this every six months or so and reduces the time it takes for the server to be in a state that is ready for business.

Our template is essentially a non-domain-connected workgroup with no licence key as you’d expect (this all gets added during the orchestration process). As such this presents additional challenges as regards deployment from SCCM which would typically be to a domain-joined machine. The process below can be used for any deployment  to a workgroup-based machine of any type from SCCM, it just so happens I need to do this for software updates to a server.

1.) Ensure you have a local administrative account set up on the workgroup machine, eg %computername%\SBond in my example below.
2.) RDP onto the machine using account above. Copy the CM2012 client install folder from a site server (eg Primary or MP) and paste it via an RDP session on the desktop.
TIP: Copy over CMTrace.exe too, you’ll more than likely use it for something.
3.) Add the local account to ConfigMgr:
UserAccount

4.) Bring up an administrative cmd prompt and CD to C:\Users\<localuser>\Desktop\Client. Run the following command (update as appropriate):

ccmsetup.exe SMSSITECODE=WPR SMSMP=<APPROPRIATE MANAGEMENT POINT>.domain.com DNSSUFFIX=Domain.com FSP=<FSP.domain.com>  CCMLOGMAXSIZE=10240 CCMENABLELOGGING=true SMSCACHESIZE=10240

5.) Recommended: Depending on your setup, you may wish to add the following registry entry for preferred MPs. Select at least one MP on the same subnet as your machine:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CCM\AllowedMPs
REG_Multi_SZ
MP.domain.com (eg)
This should stop much of the noise you’ll get trying to contact MPs you’re not interested in. This might be the case if you have cross forest relationships, etc. This step can safely be ignored if your configuration is reasonably simple.
6.)  Leave for 10 mins or so and then find machine in ConfigMgr. Right-click and select Approve.
7.) Open the CM Client applet and if you don’t already have all the tabs you’d expect usually ~7) then click Machine Policy Refresh under the Actions tab and Find Site under the Site Tab. Give it a few more minutes and the tabs should all appear.
8.) Finally, Add the computer to the appropriate patch collection and wait for patches to download. Ensure there are no further patches to run after reboot.
9.) Remember to remove the registry entries from earlier before handing back the machine!
 
NOTE
1.) In my case, because the template machine hasn’t been deployed, it doesn’t have a licence. If you select to run the software update scan cycle you’ll see a popup like that below. You can safely ignore this, it will still install the updates. You will also see this popup from time to time just before they start installing. Just ignore it.
error

 “You may be a victim of software counterfeiting.”

2.) I noticed all the machines had the same name. Because they’re in a workgroup, this is acceptable to CM and had no noticeable detrimental impact on deployment.

Software Updates Failure – 0x800F0902

0x800F0902

Symptom

You try to deploy software updates to a system and no updates will install. The system picks up the updates, tries to install but ultimately fails after hanging for ages. Eventually you are presented with the error above.

Resolution

BITS is most likely not running. It is unlikely it isn’t installed as you would expect it to be installed if the ConfigMgr agent is installed but it may not be running. Start the service and try again.