Feed Info - Kirk Evans Blog

While patching one of my SharePoint 2013 farms with the April 2013 Cumulative Update, I kept receiving an error:

“The installation of this package failed”.

I had extracted the contents from the executable ZIP file and then just copied the ubersrv2013-kb2726992-fullfile-x64-glb.exe file to my VM.  After a bit of cussing and searching, I found a discussion forums thread, “Trouble installing April update to SharePoint 2013” where the poster saw the same symptom.  Following the advice on the thread, I went to Windows Explorer and put “%TMP% in the address bar, which then navigated to my temp file location:

C:\Users\administrator.CONTOSO\AppData\Local\Temp\2

In there I see a log file, “opatchinstall(1).log”.  Opened it up to find the following error:

OPatchInstall: Getting the data from the file 'C:\temp\UBERSRV_1.CAB'
OPatchInstall: Thrown CDetectionExceptionWin32(2)

The discussion forum thread mentions a missing CAB file.  Went back to where I originally extracted the contents, found the file UBERSRV_1.CAB, and then copied to my VM in the same folder.  Ran the CU installer again, and it worked.

The fix, then, is to make sure you grab both files if you are copying to a new location.

Another tip… see Russ Maxwell’s blog post on Why SharePoint 2013 Cumulative Update takes 5 hours to install? This seems to have saved me a huge amount of time as the installation itself only took a few minutes (while the troubleshooting and cussing took a little longer). 

I am presenting tonight to the Arizona PowerShell User Group on “SharePoint and Windows PowerShell”.  Here is the agenda for the talk:

• Adding SharePoint snap-in to ISE
• SharePoint Nouns and Verbs
• SharePoint –isms
• Site structure creation
• Shell administrators
• Service applications
• On-premises remoting
• SharePoint Online

For everyone’s enjoyment, the deck I am presenting is attached. 

One of the things that frustrates customers when implementing claims authentication in SharePoint is how the people picker works for SAML claims users.  If you try to add a SAML claims user to a group in SharePoint, anything you type is considered valid.  For instance, I don’t have a user named “THIS IS NOT VALID”, but when I type that into the people picker, it works just fine.  In fact, it shows me two results!

Huh?

The title of this post says “fixing people picker”, but it is actually working just as it was designed.  The way this works is that you enter a claim value and by doing so you are asserting that any user with that claim value has access to the site.  There is no function to get a list of users using SAML claims, so SharePoint just accepts whatever we type and assumes we are typing the correct value.  There are two values that it is looking for, either the email address (which is the identifier claim in this configuration) or a role claim. 

What we want, instead, is to see something like this:

With a little code, we can add the capability to the people picker so that we can select from a list of valid results.  This code is called a Claim Provider.  The code for this solution is attached at the end of the post.

Introducing Claim Providers

Claim providers can be difficult at first to understand what they do, but once you understand what they do you will see how incredibly powerful they can be.  A claim provider serves two purposes:

• When someone logs into SharePoint, you might want to give additional claims to a user that they didn’t have before.  For instance, I wrote a previous blog entry on How to Allow Only Users Who Have a Community Badge to Your SharePoint 2013 Site.  That solution used a claim provider to add additional claims to the user when they log in such as the Achievements they have unlocked, and this is called entity augmentation.
• When you search for a user or group in SharePoint, such as when adding a user to a group, a claim provider can provide the search results.  Similarly, when you type something into the textbox to add a user or group, a claim provider is used to validate the value.  This is called name resolution. 

One of the limitations of using SAML claims is that there is no standard way to provide a list of users, which is needed when searching for users based on part of their name.  To provide this behavior, we can use a custom claim provider, which takes care of steps 4 and 5 in the following diagram.

1. The user logs into SharePoint by going to ADFS or another SAML authentication provider. 
2. The SAML authentication provider validates the requested user against some authentication store or directory such as Active Directory and gets the attributes for the user and even perhaps their group memberships.
3. A token is returned that contains claims about the user, which are used to gain access to resources within SharePoint.
4. We may want to add additional claims to the user’s token that are not passed back from the authentication service such as the Achievements they have unlocked.  For this, we can add additional claims using entity augmentation to add additional claims to the user’s token.
5. When we are searching for a user, we cannot go directly to ADFS because there is no search function.  Instead, we use a custom claim provider to query directly to some authentication store or directory such as Active Directory to retrieve information about a list of users in order to provide name resolution.

This post will focus solely on number 5 in this list, providing name resolution.  Start by creating a new empty SharePoint project created as a Farm Solution.

Setting Up the Data Model

The first thing I want to do is to define the data model that will be used with my claim provider.  I start with the basic data being modeled and create a data entity class called LDAPUser.

using System;

namespace Microsoft.PFE.ClaimsProviders
{
    public class LDAPUser
    {
        public string DisplayName { get; set; }
        public string GivenName { get; set; }
        public string SurName { get; set; }
        public string Mail { get; set; }
        public string sAMAccountName { get; set; }
    }
}

I am going to query via LDAP, so I create a class with methods that let me search on partial values or exact values using LDAP.  I wrote in a previous blog post about Querying Active Directory that shows how to use System.DirectoryServices to query using LDAP. 

using Microsoft.SharePoint;
using System;
using System.Collections.Generic;
using System.DirectoryServices;

namespace Microsoft.PFE.ClaimsProviders
{

    public class LDAPHelper
    {
        public static List<LDAPUser> Search(string pattern)
        {
            List<LDAPUser> ret = new List<LDAPUser>();

            //Run with elevated privileges to get the context of the service account 
            SPSecurity.RunWithElevatedPrivileges(delegate()
            {
                //TODO: Where to store the LDAP string?
                using (DirectoryEntry entry = new DirectoryEntry("LDAP://OU=SAMLEMPLOYEES,DC=CONTOSO,DC=LAB"))
                {
                    using (DirectorySearcher ds = new DirectorySearcher(entry))
                    {
                        ds.PropertiesToLoad.Add("displayName");
                        ds.PropertiesToLoad.Add("sAMAccountName");
                        ds.PropertiesToLoad.Add("givenName");
                        ds.PropertiesToLoad.Add("sn");
                        ds.PropertiesToLoad.Add("mail");

                        ds.Filter = "(|((displayName=" + pattern + "*)(sAMAccountName=" + pattern + "*)" + 
                            "(givenName=" + pattern + "*)(sn=" + pattern + "*)(mail=" + pattern + "*)))";

                        SearchResultCollection results = ds.FindAll();

                        foreach (SearchResult result in results)
                        {
                            ret.Add(new LDAPUser
                            {
                                DisplayName = result.Properties["displayName"][0].ToString(),
                                sAMAccountName = result.Properties["sAMAccountName"][0].ToString(),
                                GivenName = result.Properties["givenName"][0].ToString(),
                                SurName = result.Properties["sn"][0].ToString(),
                                Mail = result.Properties["mail"][0].ToString()
                            });
                        }
                    }
                }
            });
            return ret;

        }

        public static LDAPUser FindExact(string pattern)
        {
            LDAPUser ret = null;

            //Run with elevated privileges to get the context of the service account
            SPSecurity.RunWithElevatedPrivileges(delegate()
            {
                //TODO: Where to store the LDAP string?
                using (DirectoryEntry entry = new DirectoryEntry("LDAP://OU=SAMLEMPLOYEES,DC=CONTOSO,DC=LAB"))
                {

                    using (DirectorySearcher ds = new DirectorySearcher(entry))
                    {
                        ds.PropertiesToLoad.Add("displayName");
                        ds.PropertiesToLoad.Add("sAMAccountName");
                        ds.PropertiesToLoad.Add("givenName");
                        ds.PropertiesToLoad.Add("sn");
                        ds.PropertiesToLoad.Add("mail");

                        ds.Filter = "(|((displayName=" + pattern + ")(sAMAccountName=" + pattern + ")" + 
                                "(givenName=" + pattern + ")(sn=" + pattern + ")(mail=" + pattern + ")))";

                        SearchResult result = ds.FindOne();
                        if (null != result)
                        {
                            ret = new LDAPUser
                            {
                                DisplayName = result.Properties["displayName"][0].ToString(),
                                sAMAccountName = result.Properties["sAMAccountName"][0].ToString(),
                                GivenName = result.Properties["givenName"][0].ToString(),
                                SurName = result.Properties["sn"][0].ToString(),
                                Mail = result.Properties["mail"][0].ToString()
                            };
                        }
                    }
                }
            });
            return ret;

        }

    }
}

Now that the data entity model is created, we can start on the claim provider implementation.

Creating the Claim Provider

The first step is to add a class that derives from SPClaimProvider.  We add a few properties to the class including 4 properties that tell SharePoint what our provider is capable of doing.

using Microsoft.SharePoint.Administration.Claims;
using Microsoft.SharePoint.WebControls;
using System;
using System.Collections.Generic;


namespace Microsoft.PFE.ClaimsProviders
{
    public class LDAPClaimProvider : SPClaimProvider
    {
        #region ctor
        public LDAPClaimProvider(string displayName) : base(displayName) 
        { 
        }
        #endregion

        
        #region Properties
        internal static string ProviderInternalName
        {
            get { return "LDAPClaimProvider"; }
        }

        public override string Name
        {
            get { return ProviderInternalName; }
        }

        internal static string ProviderDisplayName
        {
            get { return "LDAP Claim Provider"; }
        }

        private static string LDAPClaimType
        {
            //The type of claim that we will return. Our provider only returns the
            //email address, which is the user identifier claim.
            get { return "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"; }
        }
        private static string LDAPClaimValueType
        {
            //The type of value that we will return. Our provider only returns email address
            //as a string.
            get { return Microsoft.IdentityModel.Claims.ClaimValueTypes.String; }
        }

        internal static string SPTrustedIdentityTokenIssuerName
        {
            //This is the same value returned from:
            //Get-SPTrustedIdentityTokenIssuer | select Name
            get { return "ADFS SAML Provider"; }  
        }


        public override bool SupportsEntityInformation
        {
            //Not doing claims augmentation
            get { return false; }
        }

        public override bool SupportsHierarchy
        {
            //Not modeling search results as a hierarchy.
            get { return false; }
        }

        public override bool SupportsResolve
        {
            //Yes, we will resolve search results 
            get { return true; }
        }

        public override bool SupportsSearch
        {
            //Yes, we will enable searching for users
            get { return true; }
        }
        #endregion

The next two methods tell SharePoint what type of data we are going to return.  We will return the user’s email address as a string.

        protected override void FillClaimTypes(List<string> claimTypes)
        {
            if (claimTypes == null)
                  throw new ArgumentNullException("claimTypes");
   
              // Add our claim type.
              claimTypes.Add(LDAPClaimType);
        }

        protected override void FillClaimValueTypes(List<string> claimValueTypes)
        {
             if (claimValueTypes == null)
                 throw new ArgumentNullException("claimValueTypes");
 
            // Add our claim value type.
            claimValueTypes.Add(LDAPClaimValueType);
        }

The next method tells SharePoint what type of entity types we are returning.  We will return claims that uniquely identify a user.

            protected override void FillEntityTypes(List<string> entityTypes)
        {
            if (null == entityTypes)
             {
                 throw new ArgumentNullException("entityTypes");
             }
             entityTypes.Add(SPClaimEntityTypes.User); 
        }

Once we have gotten the basics out of the way, we are left with 3 methods to implement. The first is called FillSearch, which populates the search results.  This method becomes ridiculously easy to implement because we separated out data entities in a separate class.  The FillSearch method is called when we type a partial name and hit the search button to display a list of results.  In this case we call our LDAPHelper.Search method which allows us to do a partial match on multiple values.

        protected override void FillSearch(Uri context, string[] entityTypes, string searchPattern, 
                string hierarchyNodeID, int maxCount, 
            SharePoint.WebControls.SPProviderHierarchyTree searchTree)
        {
            if (!EntityTypesContain(entityTypes, SPClaimEntityTypes.FormsRole))
            {
                return;
            }

            List<LDAPUser> users = LDAPHelper.Search(searchPattern);
            foreach (var user in users)
            {
                PickerEntity entity = GetPickerEntity(user);
                searchTree.AddEntity(entity);
            }
        }

We use a custom method called GetPickerEntity, we’ll see the details of that method in a minute, but let’s take a look at the last two methods that provide validation for the item we select.  Here we use our LDAPHelper.FindExact method to find a single match.

        protected override void FillResolve(Uri context, string[] entityTypes, 
                SPClaim resolveInput, List<SharePoint.WebControls.PickerEntity> resolved)
        {
            FillResolve(context, entityTypes, resolveInput.Value, resolved);
        }

        protected override void FillResolve(Uri context, string[] entityTypes, 
                string resolveInput, List<SharePoint.WebControls.PickerEntity> resolved)
        {
            LDAPUser user = LDAPHelper.FindExact(resolveInput);
            if (null != user)
            {
                PickerEntity entity = GetPickerEntity(user);
                resolved.Add(entity);                
            }
        }

In both the FillSearch and FillResolve methods, we call a custom method “GetPickerEntity”.  This method performs the task of adding the results to the people picker, which is the control that you interact with when adding users or groups to SharePoint.

        private PickerEntity GetPickerEntity(LDAPUser user)
        {
            PickerEntity entity = CreatePickerEntity();
            entity.Claim = new SPClaim(LDAPClaimType, user.Mail, LDAPClaimValueType, 
                    SPOriginalIssuers.Format(SPOriginalIssuerType.TrustedProvider, SPTrustedIdentityTokenIssuerName));
            entity.Description = user.DisplayName;
            entity.DisplayText = user.DisplayName;
            entity.EntityData[PeopleEditorEntityDataKeys.DisplayName] = user.DisplayName;
            entity.EntityData[PeopleEditorEntityDataKeys.Email] = user.Mail;
            entity.EntityData[PeopleEditorEntityDataKeys.AccountName] = user.sAMAccountName;
            entity.EntityType = SPClaimEntityTypes.User;
            entity.IsResolved = true;
            return entity;
        }

We are issuing identity claims in this scenario, so we use the SPClaim constructor to create the claim as discussed in Replacing the out of box Name Resolution in SharePoint 2010 - Part 2.  To make sure we use the correct claims encoded value, we add the SPTrustedIdentityTokenIssuerName to the claim.  We then add a few properties such as the display name, email, and account name so that the people picker will display them, and return the resolved entity.

There are a few other methods that we implement from the abstract class that I point out the fact they are not implemented within a region.

        #region Not Implemented
        protected override void FillClaimsForEntity(Uri context, SPClaim entity, List<SPClaim> claims)
        {
            throw new NotImplementedException();
        }

        protected override void FillHierarchy(Uri context, string[] entityTypes, string hierarchyNodeID, 
                int numberOfLevels, SharePoint.WebControls.SPProviderHierarchyTree hierarchy)
        {
            throw new NotImplementedException();
        }

        protected override void FillSchema(SharePoint.WebControls.SPProviderSchema schema)
        {
            throw new NotImplementedException();
        }
        #endregion

Deploying the Claim Provider

In order to deploy the claim provider, we need to register the claim provider using a feature receiver.  Add a new farm-scoped feature to the class and add a feature receiver to the feature. 

Next, change the type that it derives from to SPClaimProviderFeatureReceiver and fill in the data for the properties.

using System;
using System.Runtime.InteropServices;
using System.Security.Permissions;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Administration.Claims;

namespace Microsoft.PFE.ClaimsProviders.Features.Farm
{   
    [Guid("fa2cc96b-68f5-4428-9812-9ade4033e41a")]
    public class FarmEventReceiver : SPClaimProviderFeatureReceiver
    {
        public override string ClaimProviderAssembly
        {
            get { return typeof(LDAPClaimProvider).Assembly.FullName; }
        }

        public override string ClaimProviderDescription
        {
            get { return "LDAP claim provider sample provider written by Kirk Evans"; }
        }

        public override string ClaimProviderDisplayName
        {
            get { return LDAPClaimProvider.ProviderDisplayName; }
        }

        public override string ClaimProviderType
        {
            get { return typeof(LDAPClaimProvider).FullName; }
        }

    }
}

Next, right-click on the project node in Solution Explorer and choose Publish to create a WSP. 

Once you have the WSP, use some PowerShell to register the solution and install it.  The feature should automatically register the claim provider.

Add-SPSolution -LiteralPath C:\temp\Microsoft.PFE.ClaimsProviders.wsp
Install-SPSolution microsoft.pfe.claimsproviders.wsp  -GACDeployment -Force

This might take a minute to deploy, but once it is done you should be able to confirm that the claim provider now appears in the list of claim providers when you use Get-SPClaimProvider.

The last step is to register your claim provider as the default for the SPTrustedIdentityTokenIssuer. 

$ap = Get-SPTrustedIdentityTokenIssuer "ADFS SAML Provider"
$ap.ClaimProviderName = "LDAPClaimProvider"
$ap.Update()

This step is necessary because it will allow us to override the claims issued by this provider, otherwise we would issue duplicate claims and get some weird errors in SharePoint about users not being unique.

What’s The Payoff?

Thanks for sticking around this long   The payoff here is that the people picker will now validate any text we enter in the people picker control, and we can select from a set of results that display information about users.  When I search for “Eva”, my LDAPHelper.Search method is called that looks for users that have a partial match on email, first name, last name, or account name.  This particular search matches on last name to find an entry for Kirk Evans.

Notice that there are two results, one for my LDAP Claim Provider, the other entry is for Active Directory.  I have both types of authentication enabled for this zone.  If you want to see how to enable a claim provider for a particular zone, see Steve Peschka’s blog on Configuring a Custom Claims Provider to be Used only on Select Zones in SharePoint 2010.

Even better, though, is that now when we type “THIS IS NOT VALID”, we see that the results do not match anything.

This happens because our claim provider validates the list of entries in FillResolve to look for an exact match.

Looking For a Pre-Built Solution?

The purpose of this post was to introduce you to how claims work and show you how you can build a custom solution for yourself.  However, as I was writing this code and showing it to a friend, he mentioned that there is a CodePlex solution that does something quite similar to this, and they had made it much more configurable.  I took a look, and holy smokes the guys who wrote LDAP/AD Claims Provider for SharePoint 2013 did a great job, including writing application pages for it.  The purpose of my blog has always been to introduce you to concepts, but if you want a solution that is mostly done for you then go download and evaluate this solution.

For More Information

Writing a Custom Claims Provider for SharePoint 2010 - Part 1: Claims Augmentation and Registering Your Provider

Writing a Custom Claims Provider for SharePoint 2010 - Part 2: Adding Support for Hierarchy Nodes

Writing a Custom Claims Provider for SharePoint 2010 - Part 3: Searching Claims

Writing a Custom Claims Provider for SharePoint 2010 - Part 4: Supporting Resolve Name

Replacing the out of box Name Resolution in SharePoint 2010 - Part 2

LDAP/AD Claims Provider for SharePoint 2013

I have been working with claims authentication quite a bit lately, and something that can be frustrating when using claims authentication for Forms Based Authentication (FBA) or SAML claims is that when you log in you see the claims identifier instead of the user’s name.  As an example, I configured an FBA provider to use LDAP to authenticate users and when I log in, I see the following:

Similarly, I configured a trusted provider using ADFS and when I log in as a user using SAML claims, the user’s name is shown as the following:

This is because the user profile for the user has not been populated.  In my environment, I have 3 providers configured for the same zone.

To understand how to configure the trusted identity provider, see Steve Peschka’s post Configuring SharePoint 2010 and ADFS v2 End to End.  To understand how to configure FBA, see my posts SQL Server Provider for FBA in SharePoint 2010, configuring FBA with SqlMembershipProvider in SharePoint 2010 using PowerShell, and Configuring LDAP for FBA in SharePoint 2010 or SharePoint 2013 with PowerShell. 

This post is going to show how to configure user profile synchronization when you have multiple authentication providers.  In my scenario, all of the users are being imported from the same Active Directory instance.  This provides a challenge for using apps.  Working with apps in SharePoint 2013 requires user profiles to be populated (see Steve Peschka’s article, “OAuth and the Rehydrated User in SharePoint 2013 – How’d They do That and What do I Need to Know”).  It becomes increasingly important in SharePoint 2013 to properly configure the user profiles with UPN, Email, or SIP attributes when working with apps.  It doesn’t matter how you populate these, you could use PowerShell or a custom program, but SharePoint provides the ability to populate these attributes using User Profile Synchronization.  The challenge is that apps will rehydrate the user typically based on email, so the email has to be unique across all of the user profiles. This makes it important to make sure that each user in your directory has exactly one profile in SharePoint.

Steve Peschka did a great job showing how to accomplish this in his blog Mapping User Profiles for SAML Users with an AD Import in SharePoint 2013.  The part I want to highlight is how to configure multiple sync connections.

Multiple Synchronization Connections

In my scenario, I have 3 different connections to the same Active Directory.

The challenge here is making sure that your users do not overlap and that they are unique per connection.  If you have a user that is imported from AD and authenticates both as a Windows user and as a SAML user, the authentication for apps will likely fail because you cannot uniquely identify the user based on email.  This is why it is important to make sure that each user account has a unique profile.

Configuring Multiple Connections for User Profile Synchronization

I am not going into the details of how to start the user profile synchronization service instance.  For details, see Spence Harbar’s seminal post Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization.  The steps are identical for SharePoint 2013.  The part we are going to focus on is configuring the connections for our scenario that includes multiple connection sources.

Configuring a Connection for Active Directory

This is the best-documented and easiest to configure because you only need to configure the connection and you’re done.  Spence covers the details in his blog post Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization.  Even though we are authenticating with Windows claims, the only thing you need to do is configure user profile synchronization to Active Directory.

The important thing to watch out for is that you import only the users who will log in via Windows.  I have an OU called “Employees” in Active Directory that contains all of my users who authenticate via Windows to avoid overlap.

When configuring the user profile synchronization connection, I choose only this OU.

Click OK and you’re done.

Configuring a Connection for FBA

My FBA implementation is using LDAP, which points to Active Directory to authenticate the users.  Because I am using Active Directory as the LDAP provider for FBA, I can import the users with a connection to Active Directory with one special change. The key is to set the Authentication Provider Type to Forms Based Authentication, and choose the FBA provider that you’ve already configured (this value is picked up from the web.config in Central Administration).

To make sure that the account does not have multiple profiles, I constrain the synchronization to a specific OU that contains only those users who authenticate via LDAP.

Once you have configured the connection, the next step is to map the claim identifier.  Go to the User Profile Service Application and click on User Profile Properties.  Edit the property “Claim User Identifier” and add a mapping for the attribute that will be used to identify the user via claims.

Setting the authentication provider type, selecting the FBA provider, and mapping the claim identifier is important because this is used to map the claims encoded account name.  Once the user is imported, you can see the claims encoded account name to identify the user.

If you see two profiles for the same user, the problem might be because you didn’t map the provider type on the connection, or didn’t add the mapping for the claims identifier.

Configuring a Connection for a Trusted Identity Provider

I am using ADFS to authenticate my users.  When I use user profile sync, I am not connecting to ADFS, I am connecting to the Active Directory where the users are authenticated to.  I can sync the users to that Active Directory to populate their attributes.  As mentioned several times earlier, I need to make sure that the user profiles are unique.  To ensure this, I have a single OU for the users who authenticate via ADFS.

I then set up a user profile sync connection to Active Directory.  Just like when we configured FBA, we need to specify the Authentication Provider Type (this time we choose “Trusted Claims Provider Authentication”), and the Authentication Provider Instance (which is the name of the trusted identity provider you’ve already configured).

To make sure that these users do not have multiple profiles, I import them from an OU where only those users authenticate via ADFS.

The next step is to go to the user profile properties and add a mapping for the Claim User Identifier.  Note that when you have multiple connections, you will need to map each individually.  I could have the FBA users mapped using sAMAccountName instead of mail, I just chose to map them using the mail attribute since all my users have an email address.  Edit the property Claim User Identifier and add a mapping.

The claim identifier that you map for the ADFS connection needs to be the same attribute that you specified in the IdentifierClaim parameter when registering the SPTrustedIdentityTokenIssuer.

$ap = New-SPTrustedIdentityTokenIssuer -Name "SAML Provider" 
    -Description "SharePoint secured by SAML" 
    -realm $realm 
    -ImportTrustCertificate $cert 
    -ClaimsMappings $map,$map2 
    -SignInUrl "https://congen1.contoso.local/adfs/ls" 
    -IdentifierClaim "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"

See Steve Peschka’s blog Configuring SharePoint 2010 and ADFS v2 End to End for more information on configuring ADFS.

One other side note… when you are adding users to your site that are SAML users, take care to use the same identifierclaim when adding them as a user to a site. I type the full email, and then make sure I select the EmailAddress claim.  If you want to change this behavior, see Steve Peschka’s blog for creating custom claims providers.  The user will show as the IdentifierClaim until the user profile is updated with their name.

What if My User Doesn’t Come From Active Directory?

FBA users are a little more difficult because you may be using SQL for FBA, in which case there’s really no option for user profile synchronization out of the box, you need to update it through some other means.  Additionally, you might be authenticating to a trusted provider where you cannot sync all the users from the source into user profiles.  Luckily it’s pretty easy to update via using the object model, which can be accessed via PowerShell or C#.

$mySiteUrl = "http://my.contoso.lab"

$gc = Start-SPAssignment 

$site = ($gc | Get-SPSite $mySiteUrl)
$context = ($gc | Get-SPServiceContext -Site $site)
$upm = new-object Microsoft.Office.Server.UserProfiles.UserProfileManager($context)
$profile = $upm.GetUserProfile("i:0#.f|ldapmember|barneyrubble@contoso.lab");
$profile["WorkEmail"].Value = "barneyrubble@live.com";
$profile.Commit()

Stop-SPAssignment $gc

Other properties that you might want to map are SIP (mapped to the SharePoint property “SPS-SipAddress”) or the UPN (mapped to the SharePoint property “SPS-UserPrincipalName”).  The important point to note here is that it doesn’t matter if you use user profile sync or if you populate the attributes through some other process, the part that matters is that the properties are populated for the user.  You could also provide the user’s name while you’re at it to provide a friendly name at the top right of the screen. 

Showing The User Name

This blog started off showing the problem of claims users not showing their name in the top right of the page when they log in.  The only thing you have to do is make sure that their name is populated in the user profile, either through sync or through your own process.  To prove this, the same user that used to show the email now shows their name instead.

The name is displayed as part of their user profile, so you can take advantage of the fact that apps require a populated user profile and simply populate the name as part of your sync process.

For More Information

Configuring SharePoint 2010 and ADFS v2 End to End

Configuring LDAP for FBA in SharePoint 2010 or SharePoint 2013 with PowerShell

configuring FBA with SqlMembershipProvider in SharePoint 2010 using PowerShell

Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization

Mapping User Profiles for SAML Users with an AD Import in SharePoint 2013

While speaking in Kuala Lumpur for the Ignite developer training, Chris O’Connor (aka @grumpiewookie) jokingly said that the whole apps for Office thing needed something catchy, something nostalgic.  We need some obscure JavaScript library to do something interesting, something entertaining yet educational.

Something like Clippy.

I searched for all of 30 seconds, and sure enough someone built a JavaScript library that lets you put Clippy in a web page!  I love their tagline, “Our research shows that people love two things: failed Microsoft technologies and obscure Javascript libraries. Naturally, we decided to combine the two.”

During the lunch hour, I worked on a funny app (with the help of Keenan Newton) and moments later, demo’d an app that featured Clippy.  This is not just an image, Clippy is actually animated, makes sounds, and prints whatever message you feed him to the screen!

There are a number of actions you can provide the agents, such as moveTo, animate, speak, congratulate, and a bunch of other tricks that you can use to amuse your fellow geeks.

Creating a Clippy app is ridiculously easy.  There are a few steps, but only a handful of lines of code… the rest is just deployment goo.  We’ll even go one step further by deploying our app for Office as part of a SharePoint document library so that when someone creates a new document we can be sure that Clippy will be there.

Create a Document in Excel

This demonstration requires Excel 2013 to support the new app model.  Open Excel 2013 and create a document.  This document will be the template used for our app (we could use a blank Excel document, but that doesn’t look as nice in screen shots now does it?)  In Excel 2013 go to the File tab, click New, and choose the Expense Report template.  Save the document somewhere on your hard drive.

Creating the Project

There are four steps that we need to perform to create the Clippy app.  First, we create a SharePoint-hosted app.  Next, we add an App for Office project item to the app.  We then add a document library that uses the document as its template, and finally we add Clippy to the App for Office.

Creating the SharePoint Hosted App

This is the really easy part.  In Visual Studio 2012, create a new App for SharePoint 2013 named “MyExpenseReportApp” and click OK.  Give the project a name, URL for debugging, and changed the hosting model to “SharePoint-hosted”.

Done with that step.

Add an App for Office

Right-click the project node in Visual Studio 2012 and choose Add / New Item.  Choose the “App for Office” project item type and name it “ExpenseReportApp”. 

On the next page, uncheck Word and PowerPoint, leaving the option for creating a Task pane app in Excel.

On the next page, choose the option to insert the app into an existing document, and browse to your expense report document that you just created.

Click Finish.

Add a Document Library to the SharePoint App

We now need a document library that will use the App for Office as its template for new documents.  Right-click the project node in Visual Studio 2012 and choose “Add / New Item”.  Choose the “List” project item and name it “ExpenseReports”.  Choose the option to create a customizable list template and a list instance using the Document Library content type.

Choose Next.

On the next screen in the wizard, choose the option to use the following document as the template for this library.  Click the browse button, and you should be in the folder that contains your app.  There is a child folder, “OfficeDocuments”, that contains the expense report Excel document that was bound to your app for Office. 

Once you choose the file, click Finish in Visual Studio.  You now have a document template that contains an app as the template for a document library.  When someone creates a new document for that library, they will see your app.

The document library will be created in the SharePoint app web, the only way that users will be able to access it is if you provide them the URL or add a UI element like a ribbon button.  Another easy way to get to the document library is simply to update the app manifest to point to the document library.

When the app starts, it will be on the all items view for your document library.

Add Clippy

Now for a little developer ingenuity.  Visit https://www.smore.com/clippy-js and download Clippy from GitHub.  There are several folders in the ZIP file that you download, the only ones you need are the 3 files in the Build folder.

Extract those 3 files and add them to Visual Studio 2012 in the Home folder for your App for Office.

Once you have the files, the rest is easy.  Open the Home.html file.  Add the following CSS reference in the HEAD section.

<link rel="stylesheet" type="text/css" href="clippy.css" media="all">

You can see where I included it:

Finally, go to the bottom of the Home.html page and add the implementation details. 

    <script src="clippy.min.js"></script>
    
    <script type="text/javascript">
    clippy.load('Clippy', function (agent) {
        agent.moveTo(0, 200);
        agent.show();
    });
</script>

Finally, hit F5 to debug your project.  If you followed along, the app should open to your “ExpenseReports” document library in your app web.  Go to the Files tab and choose New Document / ExpenseReportsContentType.

That will open Excel using the Expense Report template.  The document contains a reference to our App for Office, so when the app opens we will see Clippy.

Can’t you just imagine their surprise when they open the new expense report app that you’ve been laboring over, and they are delighted that it comes with a built-in help system!

For More Information

https://www.smore.com/clippy-js – source and documentation for Clippy.js

http://dev.office.com – Learn to build apps for Office 2013

I received the following question on the SPYam group on Yammer, and decided to share the information with the broader audience via my blog.

For SharePoint security Microsoft always recommended to use Active directory group inside SharePoint group. But this guidance now changed Microsoft now recommend:

"We do not recommend SharePoint groups to assign permissions to sites. When a SharePoint group is used to assign permissions, a full crawl of the index occurs. Instead, we recommend Active Directory Domain Services (AD DS) groups."

[from http://technet.microsoft.com/en-us/library/ff758656(v=office.14).aspx]

Can someone from Microsoft confirm this please?!

 

Happy to try

Configuring the Environment

To demonstrate the effect of adding a user to a SharePoint group, adding an Active Directory group, and adding users to an Active Directory group, I want to isolate any changes as much as possible from the rest of my environment.  I configured my environment with a new web application CrawlDemo.Contoso.local, and a root site collection using the Team Site template.  The site collection administrator is a single user, “contoso\administrator”. 

In order to isolate the effect of search crawls to just this web application, I configured a new content source that contains my new web application’s URL.

This lets me isolate any crawls to just that one web application.  The rest of the settings for the content source:

• Crawl Settings: Crawl everything under the hostname for each start address
• Crawl Schedules: Enable Incremental Crawls
• Content Source Priority: Normal

I do not have continuous crawls configured for this environment, and none of my content sources are currently scheduled, so I can be sure that crawl information will be isolated to this single web application. 

Before continuing, I make sure that the crawl account has Full Read permission to the web application by checking the User Policy for the web application in Central Administration.

Note the text in red:

Adding or updating Web application policy with new users or groups will trigger a SharePoint Search crawl over all content covered by that policy. This can reduce search crawl freshness and increase crawl load. Consider using security groups at the policy level and add/remove users from security groups to avoid this.

We’ll come back to that later in this post as it is directly relevant to what we are discussing.

Once everything is configured, I initiate a full crawl of the content source.  Reviewing the Crawl Health report in Central Administration for Crawl Rate, I see that all 37 items were crawled.

If we focus our attention on the Summary table to the right of the screen, we can see the total items crawled was 37 items.

This means that the full number of items crawled was 37 items.  When we run an incremental crawl, we should only crawl the changed items and not every item.  The rest of this post focuses on the impact that security changes have on crawling.

A Note on Performance

I am in no way indicating that my demonstrations in this post be used for any type of performance benchmarking.  I am running all of these tests on a single laptop on a single drive using virtualized images that are not optimized for performance.  Further, I have other processes running on the machine, so the numbers that I show do not represent a production environment.  That said, we can make some general assumptions based on the data represented.

Understanding the EventCache

In each content database, there is a table named “EventCache” which contains a change log for everything in that content database.  Once the crawl is complete, I query it using the following SQL statement:

select EventType, ObjectType, EventTime
from EventCache with (nolock)
where EventTime > '2013-05-06 16:37:36.750'

Note that it is unsupported to query SharePoint’s databases directly and should not be done in production without the direct involvement of a Microsoft support engineer.  To minimize any potential effects of the query, I use the SQL NOLOCK hint to read uncommitted records and avoid introducing a table lock. 

Initially this query returns zero results because the time stamp used in the query is the last time stamp from when I created the site collection.  Let’s add a few documents to the Documents library in our new site.

We query the table again, and this time we see results in the EventCache table in the content database where the documents were added.

That makes complete sense, 4 documents added, 4 rows in the database.  We initiate an incremental crawl (not a full crawl) and wait for it to complete.  Once complete, click the Crawl Health Reports link in the search service application in Central Administration. 

The important part to call out is the Summary table on the right of the screen that shows 12 items were crawled during this incremental crawl, none of which were security items.

When the search crawler tries to determine what changes were applied, it reads the information in the EventCache table to calculate what needs to be crawled. The thing to learn from this section is that there is a table, EventCache, which the crawler uses to determine changes from each of the content databases. 

I run another incremental crawl, and the number of items decreases to 6 crawled items, and another decreases it to 5 items crawled.  No new rows were in the EventCache table, so SharePoint is simply crawling the top level pages to determine if any content changes appear.

Effect of Adding a User to a SharePoint Group

Let’s add a single domain user to the “Crawl Demo Members” group.

We now run our SQL query and see that a row has been added to the EventCache table. 

The new value, EventType 2097152, indicates a change where a member was added to a group (actually, 3 new rows were added, but I am only showing the relevant new row).  Recall that the search crawler reads the information in the EventCache table to calculate what needs to be crawled.  We run another incremental crawl and then inspect the health reports to see what happened.  Notice the red bar that now shows, the legend for the chart indicates this is a “security only” crawl.

Because we added a user to the group, everything in the site needs to be crawled in order to re-calculate the ACLs for the site.  Focusing on the Summary table at the right, we see that 36 security items were crawled simply because we added a user to a group.

We next turn to the crawl log to inspect a summary of our crawls so far.

Notice the value in the “Type” column has a letter “F” (indicating full crawl) or “I” (indicating incremental crawl).  The time for the incremental crawl went from 2:30 to 2:10 to 1:40 before we added a user to the group.  When we added the user to the group, it jumped up to 2:50 again because the ACLs for the items had to be recalculated.  The entire index had to be crawled to calculate ACLs.

In a large SharePoint environment with many documents, adding a user to a SharePoint group can have a significant impact on crawl performance.

Effect of Adding an Active Directory Group to a SharePoint Group

We just saw what happens when adding Active Directory Domain Services users to a SharePoint group.  Now let’s see the impact of adding an Active Directory group to a SharePoint group.  The effect will be less educational than before because the results will look nearly identical to what we just saw.  I create a new AD group, “MyCrawlGroup1” and add 12 users to the group in Active Directory.

Next, I add “MyCrawlGroup1” to the “Crawl Demo Members” SharePoint group.

We then go to the EventCache table to see what the changes were.  I get 3 new rows, the most important one is highlighted.

As we saw before, this change indicates that a member was added to a group.  We initiate an incremental crawl again, and just like before we see that we have the same number of security-only updates in the index.  Notice the red bar again.

Focusing on the Summary table, we see that we again have 36 security items, the same number as when we added a single user to a SharePoint group. 

If we check the crawl times, we can see that the time increased again.  This time, the time increased because SharePoint had to calculate the ACLs for the members of the domain group.  While the difference between the time spent calculating the members of the domain group versus adding a user directly is not statistically relevant (the difference of 10 seconds can be attributed to a number of factors), the fact that adding a domain group caused a security crawl is significant.

The point to understand in this section is that any time you add users or AD groups to SharePoint groups, your next incremental crawl causes the entire index to be crawled in order to calculate the ACLs. 

Effect of Adding an Active Directory Group to a Site

This should be predictable by now.  We will add an Active Directory group directly to a site instead of adding to a SharePoint group.  This time, I create a group with 12 different users in it called “MyCrawlGroup2”. 

Next, we will add the group directly to the site, granting this domain group Edit permissions just like the “Crawl Demo Members” group has.  When I go into “Site settings / Site permissions” and click the “Grant Permissions” button, an interesting thing happens, highlighted below.

SharePoint defaults to adding the new user or AD group to an existing SharePoint group.  Since we already proved what happens when you add a domain group to a SharePoint group, let’s change the drop-down to “Edit” instead of “Crawl Demo Members [Edit]”, meaning the group now is added to the site with the same level of permissions as the SharePoint group.

We query the EventCache table and highlight the interesting row.

EventType 524288 indicates “change assignment add”, and object type 4 means “Web”, so we are changing the permission on the web itself.  As expected, we run the incremental crawl, and you can guess what happens… a security crawl again.  Notice the red bar.

Inspecting the crawl log, we see that the time increased again.  Hopefully not statistically relevant, but the time has increased a bit in our crawl logs.

This should come as no surprise, adding a security object directly to the site permissions will cause the entire index to be crawled to calculate the ACLs.

Effect of Adding Users to Active Directory Groups

By now, you’re hopefully wondering, where’s the payoff for trudging through this post and all the pictures and data so far?  Well, you’re here!  Let’s see what happens when I add 12 more users to the security group “MyCrawlGroup1”, testing if adding users to a domain group that is contained within a SharePoint group causes security crawls. 

First, we query the EventCache table to see if any changes were recorded since the last record was written.  Nope!

With no changes in the EventCache table, our new users will be able to access the site because we added them to the Active Directory group, but no changes were recorded to SharePoint to trigger a security crawl.  We check the report, and see that the “Security only” red bar no longer shows in the graph.

Further, looking at the crawl log, our crawl time significantly decreased from 3:31 to just 1:50.

The lesson to learn is that adding users to an Active Directory group that already has permission in a SharePoint group does not trigger a full crawl of the index to calculate changes in permissions.

Now let’s try adding users to our “MyCrawlGroup2”, testing to see if adding users to an Active Directory group that already has permissions to the site causes a full crawl of the index.  I add 12 users to the AD group.

Of course, there are no new records in SQL.

And our incremental crawl runs again.  The time stays consistent with zero security updates.

The lesson here is that adding users to an Active Directory group that has already been granted permission to a SharePoint site directly does not initiate a full crawl of the index to calculate security changes.

Bonus Material: Adding a User Policy to the Web Application

By now, you’ve seen that adding a user or AD group to a SharePoint group or site causes a full index to calculate the permissions.  At the beginning of this post, I called out the User Policy button in Central Administration, noting the text in red:

Adding or updating Web application policy with new users or groups will trigger a SharePoint Search crawl over all content covered by that policy.

Let’s see what happens when we try to add a user.  In a previous post, I covered PowerShell that configures the object cache accounts.  Configuring the object cache accounts requires that you add Full Read permission to the Portal Super Reader and Full Control to the Portal Super Reader account.  I add the two accounts as part of the configuration steps necessary.

Next, I check SQL.

EventType 8192 means change or modify, and object type 2048 means Security Policy.  As you can reasonably expect by now, this causes a full index to calculate permissions as described in the red text on that page.  Notice the red bar is back.

The lesson to learn is that it is important that you configure the object cache accounts so they have permission to the cache (see my blog posting Setting Object Cache Accounts in SharePoint 2010, the same steps apply for SharePoint 2013) while understanding the impact that adding the permission can have on crawl performance.  Adding a user policy to the web application causes a full index to calculate permissions of all objects covered by that policy.  You still need to configure the object cache accounts, but make sure that you plan for the potential impact on crawl performance when you apply this change.

Summary

Hopefully this post shed some light on how security works with SharePoint and the effect that adding users or Active Directory groups can have on crawl performance. Adding users to Active Directory groups that have already been granted permissions to SharePoint does not affect crawl performance.  Adding users to Active Directory groups is the recommendation for managing security permissions in SharePoint.

This post will show how you can secure a web site based on if a user has been given a badge in a SharePoint 2013 community site.

Badges and Reputation in SharePoint 2013 Community Sites

SharePoint 2013 Community Sites provide a new set of features to implement game theory, or gamification.  One feature is called “reputation”, where participation earns you points.  As you earn points, you earn achievements that are reflected as your reputation.  The settings for reputation allow you to define your own points system, defining the points earned for creating a post, replying to a post, having a post marked with 4 or 5 stars or “liked”, and having a post marked as “Best Reply”.  You further define the number of points per level, and whether the achievements are represented as an image or as text.

The second feature to point out is called a Gifted Badge.  The site owner will define a set of badges that allow you to define the name of each badge.

This allows a community site owner to define a set of badges and assign them to community members.  For instance, you might want to create a badge that calls out if someone has achieved a particular certification level.  Just highlight a community site member and the “Give Badge” button will be enabled in the ribbon in the Moderation tab.

Once users have been assigned badges, they appear on the front page of the Community site in the “Top contributors” web part.

When I first saw the new Community Sites in SharePoint 2013, I was excited.  Of course, there are a ton of things that I would like to see improved, but this is a decent start.  To me, the ability to assign badges seemed really cool and I saw a bunch of opportunities such as assigning badges when employees are on-boarded and they complete training courses.  Out of the box, all you can do with them is either earn them or assign them.  What would be really cool is if you could somehow use them to protect content such that only people who have a specific badge are allowed to use a site.  For instance, I could create a badge called “Microsoft Certified Professional” and only users who have that badge are allowed into the “MCPs Only” web site.  That would allow me to do this:

That is exactly what we are going to build.

Wanna Come In?  Show Me Your Badge!

Badges are simply items in a list, so how would you use that in order to allow or deny someone access to a site?  The solution is a custom claims provider.   A claims provider is used to augment a user’s claims and to provide name resolution.  Augmenting claims simply means adding more claims into the user’s token.  For instance, I can add a claim that says what the user’s favorite basketball team is, what their hair color is, whatever information I want to add to their user token.  Here is a screen shot where I added a new claim type “http://Microsoft.PFE.Claims/Badges” with a value “Microsoft Certified Master”.

When the user logs in, our custom claim provider will augment the user’s claim set by adding an additional claim indicating if the user has been assigned a badge.  If the user has not been assigned a badge, then no claim is added.

This scenario could be adapted for many other scenarios.  Perhaps you have a document library that contains your company’s intellectual property, and only people who have authored intellectual property have access to a special site.  Maybe you require people to complete training courses before allowing them access to content.  There are many ways that you can adapt this scenario.

The Badge Custom Claims Provider

There are a number of claim providers already registered out of the box, and you can register more than one claim provider to augment claims.  Using the Get-SPClaimProvider cmdlet in PowerShell, we can see the claim providers that have been registered.

Start by creating a new SharePoint 2013 farm solution using the “SharePoint 2013 – Empty Project” template.  Next, add a class that derives from Microsoft.SharePoint.Administration.Claims.SPClaimProvider.  Here is a class diagram of what we will implement.  

Before we go into the claims provider, let’s get our data access layer stuff out of the way.

Reading Badges From a Community Site

The first step is to create a few classes that store the data we want to work with.  The first class is Badge that just holds the ID and Title for the badge.

namespace Microsoft.PFE.ClaimsProviders
{
    public class Badge
    {
        public int ID { get; set; }
        public string Title { get; set; }
    }
}

The next class is BadgeUser, which holds information about the user and the badges that they have been assigned.

using System.Collections.Generic;

namespace Microsoft.PFE.ClaimsProviders
{
    public class BadgeUser
    {
        public string Name { get; set; }
        public string LoginName { get; set; }
        public List<Badge> Badges { get; set; }
    }
}

Now that we’ve defined some classes to hold the data, let’s create a helper class that can fill them with data.  The first method allows us to search for any badges that start with a set of characters.

 1: using Microsoft.SharePoint;

 2: using System.Collections.Generic;

 3:  

 4: namespace Microsoft.PFE.ClaimsProviders

 5: {

 6:     public class BadgeHelper

 7:     {   

 8:         /// <summary>

 9:         /// Get the list of badges that start with the partial name.

 10:         /// </summary>

 11:         /// <param name="partialBadgeName">The characters the badge

 12:         /// name should start with.

 13:         /// </param>

 14:         /// <returns>A list of badges that match the criteria.</returns> 

 15:         public static List<Badge> GetBadgesByPartialName(string partialBadgeName)

 16:         {

 17:  

 18:             List<Badge> badges = new List<Badge>();

 19:  

 20:             //Use RWEP here because the user may not have 

 21:             //access to the site.

 22:             SPSecurity.RunWithElevatedPrivileges(delegate()

 23:             {

 24:                 using (SPSite site = new SPSite("http://intranet.contoso.local"))

 25:                 {

 26:                     SPWeb web = site.OpenWeb("Community");

 27:                     

 28:                     SPQuery query = new SPQuery();

 29:                     query.Query = "<Where>" +

 30:                                        "<BeginsWith>" +

 31:                                            "<FieldRef Name='Title' />" +

 32:                                            "<Value Type='TEXT' >" + 

 33:                                                partialBadgeName + 

 34:                                            "</Value>" +

 35:                                        "</BeginsWith>" + 

 36:                                    "</Where>";

 37:  

 38:                     query.ViewFields = "<FieldRef Name='ID'/>" +

 39:                                         "<FieldRef Name='Title'/>";

 40:                                         

 41:                     SPList badgeList = web.Lists["Badges"];

 42:                     SPListItemCollection items = badgeList.GetItems(query);

 43:  

 44:                     

 45:                     if (null != items)

 46:                     {

 47:                         foreach (SPListItem item in items)

 48:                         {

 49:                             badges.Add(new Badge 

 50:                             { 

 51:                                 ID = item.ID, 

 52:                                 Title = item.Title 

 53:                             });

 54:                         }

 55:                     }

 56:                     

 57:                 }

 58:             });

 59:             return badges;

 60:         }

The call on line 22 is necessary because the current user may not have access to the community site that we are obtaining data from.  You can see this is a pretty simple method that just gets data from a SharePoint list. The next method looks nearly identical in that it gets a list of data based on an exact match rather than a partial match.

        /// <summary>
        /// A list of badges that match the exact name.
        /// </summary>
        /// <param name="name">The name of the badge to match.</param>
        /// <returns>A list of badges that have the exact name.</returns>
        public static List<Badge> GetBadgesByExactName(string name)
        {            
            List<Badge> badges = new List<Badge>();

            //Use RWEP here because the user may not have 
            //access to the site.
            SPSecurity.RunWithElevatedPrivileges(delegate()
            {
                using (SPSite site = new SPSite("http://intranet.contoso.local"))
                {
                    SPWeb web = site.OpenWeb("Community");

                    SPQuery query = new SPQuery();
                    query.Query = "<Where>" +
                                       "<Eq>" + 
                                            "<FieldRef Name='Title' />" +
                                            "<Value Type='TEXT' >" + 
                                                name + 
                                            "</Value>" +
                                       "</Eq>" +
                                   "</Where>";
                    query.ViewFields = "<FieldRef Name='ID'/>" +
                                        "<FieldRef Name='Title'/>";

                    SPList badgeList = web.Lists["Badges"];
                    SPListItemCollection items = badgeList.GetItems(query);


                    if (null != items)
                    {
                        foreach (SPListItem item in items)
                        {
                            badges.Add(new Badge
                            {
                                ID = item.ID,
                                Title = item.Title
                            });
                        }
                    }

                }
            });
            return badges;
        }

The last method bears a little more explanation.  Now we want to get the badges for a particular user.  The user may not have access to the site, so we have to use RunWithElevatedPrivileges.  In fact, the user may not even be in the site users list, so we have to use EnsureUser in order to obtain a valid reference to the SPUser object for that user. 

 1:         /// <summary>

 2:         /// Gets the badges for a particular user

 3:         /// </summary>

 4:         /// <param name="loginName">The login name in encoded claims

 5:         /// format. For instance, "i:0#.w|contoso\administrator".</param>

 6:         /// <returns>The user information and badges for that user.</returns>

 7:         public static BadgeUser GetBadgesForUser(string loginName)

 8:         {

 9:             BadgeUser badgeUser = null;

 10:  

 11:             //Use RWEP here because the user may not have 

 12:             //access to the site.

 13:             SPSecurity.RunWithElevatedPrivileges(delegate()

 14:             {

 15:                 using (SPSite site = new SPSite("http://intranet.contoso.local"))

 16:                 {

 17:                     SPWeb web = site.OpenWeb("Community");

 18:  

 19:                     //The encoded claim will be in the format

 20:                     //0#.f|ldapmember|kirkevans. Add the i: prefix

 21:                     //to indicate this is an identity claim.

 22:                     if (!loginName.StartsWith("i:"))

 23:                     {

 24:                         loginName = "i:" + loginName;

 25:                     }

 26:  

 27:                     //EnsureUser will only work if the same login provider

 28:                     //is available in this site. For example, if you are

 29:                     //calling this from a user authenticated in another web 

 30:                     //application, and this site does not have FBA configured,

 31:                     //the call to EnsureUser will fail.

 32:                     SPUser user = user = web.EnsureUser(loginName);

 33:  

 34:                     SPQuery query = new SPQuery();

 35:                     query.Query = "<Where>" +

 36:                                     "<And>" +

 37:                                         "<Eq>" +

 38:                                            "<FieldRef Name='Member' LookupId='TRUE' />" +

 39:                                            "<Value Type='USER' >" + user.ID + "</Value>" +

 40:                                         "</Eq>" +

 41:                                         "<IsNotNull>" +

 42:                                            "<FieldRef Name='GiftedBadgeLookup'/>" +

 43:                                         "</IsNotNull>" +

 44:                                     "</And>" +

 45:                                   "</Where>";

 46:  

 47:                     query.ViewFields = "<FieldRef Name='Title'/>" +

 48:                                         "<FieldRef Name='Member'/>" +

 49:                                         "<FieldRef Name='GiftedBadgeLookup'/>";

 50:                     SPList memberList = web.Lists["Community Members"];

 51:                     SPListItemCollection items = memberList.GetItems(query);

 52:  

 53:                     if (items.Count > 0)

 54:                     {

 55:                         badgeUser = new BadgeUser

 56:                         {

 57:                             LoginName = user.LoginName,

 58:                             Name = items[0].Title

 59:                         };

 60:                         List<Badge> badges = new List<Badge>();

 61:  

 62:                         foreach (SPListItem item in items)

 63:                         {

 64:                             SPFieldLookupValue badgeName =

 65:                                 new SPFieldLookupValue(item["GiftedBadgeLookup"].ToString());

 66:                             //The current data structure only allows

 67:                             //one badge, but we may want to add badges

 68:                             //from another system in the future.

 69:                             badges.Add(new Badge

 70:                             {

 71:                                 ID = badgeName.LookupId,

 72:                                 Title = badgeName.LookupValue

 73:                             });

 74:                         }

 75:                         badgeUser.Badges = badges;

 76:                     }

 77:                 }

 78:             });

 79:             return badgeUser;

 80:         }

 81:     }

 82: }

The call to EnsureUser requires that the provider used to authenticate the user is available to this web application.  If we are in one web application that uses FBA claims and then we make a call to another web site (in this case http://intranet.contoso.local), then both web applications must provide the same authentication capabilities.  If one web application uses FBA, the site you are calling EnsureUser for must also be configured to use the same FBA provider.

Implementing the Badge Claim Provider

Now that we have the data access stuff out of the way, let’s focus on the really cool part of the solution.  We’ll start by implementing the properties as shown in the previous class diagram.

 1: using System;

 2: using System.Collections.Generic;

 3: using Microsoft.SharePoint.Administration;

 4: using Microsoft.SharePoint.Administration.Claims;

 5: using Microsoft.SharePoint.WebControls;

 6:  

 7: namespace Microsoft.PFE.ClaimsProviders

 8: {

 9:     public class BadgeClaimProvider : SPClaimProvider

 10:     {

 11:         #region Constructor

 12:         public BadgeClaimProvider(string displayName)

 13:             : base(displayName)

 14:         {

 15:         }

 16:         #endregion

 17:  

 18:         #region Properties

 19:         internal static string ProviderInternalName

 20:         {

 21:             get { return "BadgeClaimsProvider"; }

 22:         }

 23:  

 24:         public override string Name

 25:         {

 26:             get { return ProviderInternalName; }

 27:         }

 28:  

 29:         internal static string ProviderDisplayName

 30:         {

 31:             get { return "Badges for User"; }

 32:         }

 33:  

 34:         private static string BadgeClaimType

 35:         {

 36:             get { return "http://Microsoft.PFE.Claims/Badges"; }

 37:         }

 38:  

 39:         private static string BadgeClaimValueType

 40:         {

 41:             get { return Microsoft.IdentityModel.Claims.ClaimValueTypes.String; }

 42:         }

 43:  

 44:         public override bool SupportsEntityInformation

 45:         {

 46:             get { return true; }

 47:         }

 48:  

 49:         public override bool SupportsHierarchy

 50:         {

 51:             get { return false; }

 52:         }

 53:  

 54:         public override bool SupportsResolve

 55:         {

 56:             get { return true; }

 57:         }

 58:  

 59:         public override bool SupportsSearch

 60:         {

 61:             get { return true; }

 62:         }

 63:  

 64:         #endregion

The ProviderDisplayName property on line 29 can be seen in the DisplayName property when we use the Get-SPClaimProvider cmdlet.  The BadgeClaimType (line 34) defines the URI for the claim, and the BadgeClaimValueType (line 39) defines the type of value the claim will contain. 

The four properties starting with “Support” indicate to SharePoint what capabilities this provider has. 

• EntityInformation – provides claims augmentation for a specific user.
• Hierarchy – used in the People Picker to create a hierarchical representation of claims.
• Resolve – Find an exact match for the claim value. 
• Search – Search for a value based on a partial word or search pattern.

Our provider will support entity information, search, and resolve. 

Claims Augmentation - Providing Entity Information

We need to add a claim to the user’s token that shows a badge that they might have earned.  To do that, we first tell SharePoint that we are going to fill a claim with a specific type (“http://Microsoft.PFE.Claims/Badges”) and what type its value will be (Microsoft.IdentityModel.Claims.ClaimValueTypes.String).  We then use the FillClaimsForEntity method to augment the claims for the current user.  This method accepts a parameter, “entity”, the value of which is the user’s identity claim (for instance, “0#.f|ldapmember|kirkevans”).  We then pass that value to our helper method GetBadgesForUser.

 1:         protected override void FillClaimTypes(List<string> claimTypes)

 2:         {

 3:             if (claimTypes == null)

 4:                 throw new ArgumentNullException("claimTypes");

 5:  

 6:             // Add our claim type.

 7:             claimTypes.Add(BadgeClaimType);

 8:         }

 9:  

 10:         protected override void FillClaimValueTypes(List<string> claimValueTypes)

 11:         {

 12:             if (claimValueTypes == null)

 13:                 throw new ArgumentNullException("claimValueTypes");

 14:  

 15:             // Add our claim value type.

 16:             claimValueTypes.Add(BadgeClaimValueType);

 17:         }

 18:  

 19:  

 20:         protected override void FillClaimsForEntity(Uri context, SPClaim entity, List<SPClaim> claims)

 21:         {            

 22:             if (entity == null)

 23:                 throw new ArgumentNullException("entity");

 24:  

 25:             if (claims == null)

 26:                 throw new ArgumentNullException("claims");

 27:  

 28:                         

 29:             var badgeUser = BadgeHelper.GetBadgesForUser(entity.Value);

 30:  

 31:             

 32:             if (null != badgeUser)

 33:             {

 34:                 if (null != badgeUser.Badges)

 35:                 {

 36:                     foreach (var badge in badgeUser.Badges)

 37:                     {

 38:                         claims.Add(CreateClaim(BadgeClaimType, badge.Title, BadgeClaimValueType));

 39:                     }   

 40:                 }

 41:             }                     

 42:         }

 43:         

 44:         

 45:  

 46:         protected override void FillEntityTypes(List<string> entityTypes)

 47:         {

 48:             if (null == entityTypes)

 49:             {

 50:                 throw new ArgumentNullException("entityTypes");

 51:             }

 52:             entityTypes.Add(SPClaimEntityTypes.FormsRole);            

 53:         }

Enabling the People Picker

So far, all we have done is add a claim to the user’s token.  We want to enable the scenario where someone can create a site, select Users and Groups, and use the People Picker to select our badge, allowing any users with that badge access to the site.  To do this, we need to implement a few more methods: FillSchema, FillSearch and FillResolve.  FillSchema tells the PeoplePicker what information we will display and in what format.  FillSearch accepts a string value that we use to find any badges that start with those characters.

        protected override void FillSchema(SPProviderSchema schema)
        {
            if (null == schema)
            {
                throw new ArgumentNullException("schema");
            }

            schema.AddSchemaElement(new SPSchemaElement(PeopleEditorEntityDataKeys.DisplayName,
                "Display Name", SPSchemaElementType.TableViewOnly));
        }

        protected override void FillSearch(Uri context, 
            string[] entityTypes, 
            string searchPattern, 
            string hierarchyNodeID, 
            int maxCount, 
            SPProviderHierarchyTree searchTree)
        {
            if (!EntityTypesContain(entityTypes, SPClaimEntityTypes.FormsRole))
            {
                return;
            }

            List<Badge> badges = BadgeHelper.GetBadgesByPartialName(searchPattern);
            if(null != badges)
            {
                foreach (var badge in badges)
                {
                    PickerEntity entity = CreatePickerEntity();
                    entity.Claim = CreateClaim(BadgeClaimType, badge.Title, BadgeClaimValueType);
                    entity.Description = badge.Title;
                    entity.DisplayText = badge.Title;
                    entity.EntityData[PeopleEditorEntityDataKeys.DisplayName] = badge.Title;
                    entity.EntityType = SPClaimEntityTypes.FormsRole;
                    entity.IsResolved = true;
                    searchTree.AddEntity(entity);
                }
            }
        }

Another way you can use the People Picker is to type a value in.  When you type the value instead of using search, this calls the FillResolve method of the registered claims providers.

        protected override void FillResolve(Uri context, 
            string[] entityTypes, 
            SPClaim resolveInput, 
            List<PickerEntity> resolved)
        {
            FillResolve(context, entityTypes, resolveInput.Value, resolved);
        }

        protected override void FillResolve(Uri context, 
            string[] entityTypes, 
            string resolveInput, 
            List<PickerEntity> resolved)
        {
            List<Badge> badges = BadgeHelper.GetBadgesByExactName(resolveInput);
            if (null != badges)
            {
                foreach (var badge in badges)
                {
                    PickerEntity entity = CreatePickerEntity();
                    entity.Claim = CreateClaim(BadgeClaimType, badge.Title, BadgeClaimValueType);
                    entity.Description = badge.Title;
                    entity.DisplayText = badge.Title;
                    entity.EntityData[PeopleEditorEntityDataKeys.DisplayName] = badge.Title;
                    entity.EntityType = SPClaimEntityTypes.FormsRole;
                    entity.IsResolved = true;
                    resolved.Add(entity);
                }
            }
        }

        protected override void FillHierarchy(Uri context, 
            string[] entityTypes, 
            string hierarchyNodeID, 
            int numberOfLevels, 
            SPProviderHierarchyTree hierarchy)
        {
            throw new NotImplementedException();
        }
    }
}

Registering Your Custom Claim Provider

In Visual Studio 2012, add a new farm-scoped feature and a feature receiver.  Change the type of the feature receiver to SPClaimProviderFeatureReceiver.  You only need to override a few properties here, no need to override FeatureActivating or FeatureDeactivating.

using System;
using System.Runtime.InteropServices;
using Microsoft.SharePoint.Administration.Claims;

namespace Microsoft.PFE.ClaimsProviders.Features.BadgeClaimFeature
{
    /// <summary>
    /// This class handles events raised during feature activation, deactivation, installation, uninstallation, and upgrade.
    /// </summary>
    /// <remarks>
    /// The GUID attached to this class may be used during packaging and should not be modified.
    /// </remarks>

    [Guid("99871a2c-f81a-443c-9625-d2465d2ec29c")]
    public class BadgeClaimFeatureEventReceiver : SPClaimProviderFeatureReceiver
    {


        public override string ClaimProviderAssembly
        {
            get { return typeof(BadgeClaimProvider).Assembly.FullName; }
        }

        public override string ClaimProviderDescription
        {
            get { return "A sample provider written by Kirk Evans"; }
        }

        public override string ClaimProviderDisplayName
        {
            get { return BadgeClaimProvider.ProviderDisplayName; }
        }

        public override string ClaimProviderType
        {
            get { return typeof(BadgeClaimProvider).FullName; }
        }
    }
}

Right-click the project and choose Publish.  This will package the solution into a WSP.  Once packaged into a WSP, you can add the solution and deploy it using PowerShell.

Add-SPSolution -LiteralPath C:\temp\claims\Microsoft.PFE.ClaimsProviders.wsp
Install-SPSolution microsoft.pfe.claimsproviders.wsp  -GACDeployment

Securing Content Based on a Badge

Now let’s put our new claim provider through its paces. I create a new site titled “MCM Only” with unique permissions.  I then determine who to add to the group “MCM Only Members”.  When I search for “Microsoft” I see a list of the badges that start with the search term (this called our FillSearch method).  When we hover over a result, a popup shows which claim provider the information came from.

Now let’s try our FillResolve method by typing the value in. 

When we hover over the underlined value, we choose the value that matches.

Once we choose a value, only users that have that claim will be granted access.  For those users that have access to the site because they match a claim, they will be granted access.  Notice that it shows in the top nav bar for a user who has the badge.

When I try to access the site, it succeeds.

Further, SharePoint is smart enough to security trim away the site that a user doesn’t have access to based on our claim provider!  Now I am logged in as Dan Jump, who does not have a badge.

And when I try to access the site by typing it in the URL, it fails just as we would expect.

Summary

Custom claim providers address a large number of scenarios that people commonly face with SharePoint, enabling the ability to add additional information into the user’s token and then using that information to secure content or assign work to users via the people picker.  There are quite a few interesting ideas on using custom claim providers.

Securing SharePoint Content using Profile Attributes

Leveraging Facebook for SharePoint Security

Creating a Claims Provider Based on SharePoint Audiences

Creating a Hierarchical Claim Provider Based on Favorite Basketball Teams

As I survey many of the implementations that have been done for various customers using a mix of HttpModules and code in master pages to provide less than elegant solutions to similar problems, I can see more and more scenarios that could be solved using custom claim providers.

I recently attained the Microsoft Certified Master for SharePoint 2010 certification.  I have long wanted to write about the experience, but honestly didn’t know how to begin.  After quite a few conversations with friends and colleagues, I thought I would share some insight about the process.  I am not writing this to dissuade you from pursuing MCSM… in fact, quite the opposite.  I think many more SharePoint professionals need to attend the program to receive the most amazing training that they will ever receive anywhere.  That said, you should also understand what it takes to achieve certification and what the rest of the Masters went through.  They didn’t just attend a training course and then get a certificate of completion, this is a true measure of your mastery of the technology.

IT WAS A LOT OF WORK

Mrs. Robertson, my high school English teacher, would scream at me for putting that heading in all caps, but I mean every single word of it: IT WAS A LOT OF WORK.  The MCM certification is just that, a certification.  You might be familiar with the Microsoft Most Valuable Professional (MVP) award which is given to recognize technical leaders who contribute to the community in a meaningful way.  I was lucky enough to be an MVP at one time and cannot speak highly enough about the hard work and dedication for these people.  Where MVP is an award that recognizes technical leadership within the community, MCM is a certification process that requires multiple tests.  The program requires that you pass 4 certification exams (two for IT Pro, and 2 for Developer) before you even qualify for an interview to determine if you are accepted into the MCM training program.  Once you complete the training program, you have to pass the mother of all written tests, by far the hardest written test I’ve ever taken (worse than my Calculus 3 final exam in college by far).  Finally, you sit for a qualification lab test that took every minute of 8.5 hours to complete, a test that I joke most consultants would bill at least a month of time to complete the same amount of work.  It was grueling, and worth every minute.  It was one of the most difficult and yet satisfying achievements of my career.

The Interview

My interview was with two people, both MCMs, and they grilled me on technical aspects of the product that I had never considered, let alone had any real world experience with.  I honestly don’t know of too many people who would have enough technical experience to approach that interview with confidence, let alone have much confidence left after the interview.  They asked me about some of the roughest edges in the product, how to solve problems on a global scale as well as how to solve very specific smaller complex problems.  I thought I knew a lot about the product, but I was seriously questioning my abilities once I got off the phone.  Somehow, I passed, and they gave me feedback on areas that I needed to focus on in order to be successful with the program going forward.  For me, unfortunately, the things that I needed to focus on were precisely the things that were emphasized and central components of the training program.

The Training

There are two formats for the technical training portion of the program: a three week immersive training course where you are on site the entire time, and a one week on site plus 10 week remote training course.  After talking with colleagues who had gone through the experience, they thoroughly recommended the three week course as the better option.  For me, I found that the three weeks on site provided me the best opportunity to succeed as I was able to work side by side with friends until the wee hours of the morning in the lab away from the distractions of work and family.  Yes, I said the distractions of family, as much as it pains me to say that this program requires that you set aside all personal responsibilities to focus.  I didn’t feel that I would be disciplined enough to attend the training virtually.

The material in the program covers a broad set of topics.  For instance, simply saying we covered “ECM” means that we covered document sets, holds, metadata navigation, document ID, retention policies, IRM, workflows, content organizer, and a bunch of other topics.  For each topic, we examined the timer jobs required, the interaction of processes (holds leverage search), the limitations of various capabilities, common issues that people have with various functions, and the recommended approach for applying it.  We covered things like the content migration API, discussing what information is retained and what is lost, and the implications of using this in various scenarios such as content deployment. 

Monday through Friday for three weeks we arrived in the lab around 7:00 am, took a few very short breaks and an hour for lunch, and were bombarded with incredibly deep technical training until around 6:00 pm, sometimes later.  The material is all 400 level, you are not introduced to things like search or user profiles, but rather it is assumed you already know how this stuff already works.  They teach you the deep inner workings of how things work, like a fantastic deep dive on user profiles by Spence Harbar or another incredibly deep dive on search by Neil Hodgkinson.  This training is an invaluable part of the program where you hear some of the most common issues and things that cause the most support cases for the product team.  You learn from people who know this stuff more than anyone else.

For each of the modules below, we went into detail about what service applications are used, the required permissions, the implication of topology choices, services required and where they should run, how to manage them and how to troubleshoot them.  Below is a summary of various topics covered, I am asked quite often what kind of information is covered in the course.  This is not an exhaustive list, I am probably missing a few and some of these are way deeper than I suggest in this list. 

• ECM – everything mentioned above plus WCM.
• Governance – creating a governance plan. 
• Planning and Information Architecture – planning topologies and how you might segment information based on requirements.  Did you know that TechNet has a bunch of information on planning?
• Multi Tenancy – how to configure multi-tenancy, why you shouldn’t do this for an on-premise farm, and design considerations.
• Service Applications – how they work, core components, how to create your own service application.
• User Profile – My Sites, social, architecture, provisioning, synchronization, troubleshooting, dependencies, interactions with managed metadata and search, enterprise considerations.
• Identity – Kerberos, FBA, constrained delegation, claims.
• Search – Everything from how to provision the search topology, what services are running where, what service accounts are used and what can be federated or delegated, how the crawler works, how search stores information and where, and what happens when you query.
• Search Customization – custom search pages, result refiners, extending search web parts, best bets, scopes, XSLT for core results, BDC results, people results, crawl extensibility.
• Business Continuity Management – recycle bin, list export and import, site backup and restore, farm backup and restore, mirroring, log shipping, SQL Server Always On.  How SQL backups work and what mirroring and log shipping are really doing, what databases can use log shipping and why.
• Upgrades – how to upgrade, best practices, things to watch out for, how to handle customizations and site definitions, how to troubleshoot.  How to patch, best practices for patching.
• Capacity Planning – how to determine the requirements for your farm and how to test your farm to determine if it can handle the load.  How to monitor the farm once deployed, and how to address constraints.
• Developer – workflows, data access technologies, schema-based deployment (declarative vs. imperative deployment), web templates, LINQ, REST, application lifecycle management, feature upgrades, management and monitoring custom solutions.
• Workflow – Developing, configuration, troubleshooting, performance tuning
• Business Connectivity Services – capabilities, APIs, PowerShell, security, search, augmenting user profiles, custom connectors, throttles and limits, performance tuning.

Pretty daunting list?  Admittedly, there are going to be topics you are strong at and topics you are not as strong at.  Prior to attending the training, you should spend quite a bit of time with the pre-reading materials.  We joked that it is basically, “read all of TechNet and all of Steve Peschka’s and Spence Harbar’s blog.”  From the list above, that’s not far from the truth.  The one thing that I didn’t write but is common to all of the modules: PowerShell.  You need to know PowerShell, and if you aren’t comfortable with it before training, you sure the heck will be afterwards.

Keep in mind that this was what we covered for SharePoint 2010 MCM, the list of materials has changed since then. 

• Latest Program Overview Webcast
• Current Course Topics List
• Program Home Page

I was a good student in college, I achieved Dean’s List for the last 2 years of college and generally was able to get by with taking notes, memorizing for tests, and then forgetting most of what I learned after finals.  This simply won’t work for this program: you must bring a broad set of experiences with you or the training will be over your head.  This is a test of experience and stamina.  Being attentive for 9-10 hours per day to deep technical content is exhausting and you get to the point where you start trying to figure out “do I really need to listen here or can I zone out for a minute, can this possibly be on the test?”  The answer is yes: this is probably on the test.  The information presented in class covers topics that you are expected to already know, just with more detail than you might already know.  For instance, you might already know a thing or two about ECM, but do you know all the services required to make everything work as expected?  The more familiar you are with search, user profiles, identity and authentication and claims, and service application federation before you attend the class, the more you will get out of the class.  If you are hearing a topic for the first time (admittedly, much of it was brand new to me), then you are going to struggle. The MCM certification is just as much a test of what you learned in class as it is your own experience, and it’s really difficult to master a concept you just learned about.

The Hands on Labs

You are given an environment to practice hands on labs to bolster the training that you receive during the day.  The training lectures runs Monday through Friday from around 8:00 am to 6:00 pm, so you spend as much time nights and weekends working on the labs as you can.  There are some labs that are relatively short, only taking a few hours to complete, while others may never be completed.  We were often in the lab until midnight or 1am working on problems, surviving on little sleep while knowing we needed to be alert during the training lectures.  Weekends were consumed with these labs as well as studying for the written test.  You have to balance your time practicing the things you think you know with the things you are not as strong with. 

These labs are not detailed walkthroughs, they are often just a description of a task that you need to accomplish that may seem deceptively easy yet take days to complete.  At first, I was frustrated at the lack of direction in the labs and how open-ended they were, but I came to love this about the training course.  Rather than show you one way to accomplish a task, you are forced to learn how to do the task on your own and understand various aspects of it.  The second-to-last sentence in the lab was usually “verify that a document shows up” or “verify that the search result shows this”.  The last sentence was usually a question like “what else could you do with this” or “how else could you accomplish this?” 

Here are the HOL topics that were covered. 

• Logical architecture and governance
• Topology
• ECM
• WCM
• Content migration
• Basic upgrades
• Complex upgrades
• Patching
• Business Continuity Management
• Capacity Planning and Performance
• Custom service applications
• Visual Studio workflows
• Authentication
• Platform security
• Schema based development
• Search topology
• Search customization
• Business Connectivity Services

My favorite lab was the Topology lab.  You are handed this lab on the first day, and wow is it trial by fire.  Given a set of servers, provision various services with a desired topology.  This lab took me almost two weeks to complete because I was stuck on one problem and couldn’t solve it. Time management is a huge part of this course, spending 2 weeks on that one lab was, in hindsight, stupid because I should have spent more time on the other labs making sure I was prepared for the qualification lab.  However, I now have a very firm grasp on user profiles and will never forget that the user profile service must be started before provisioning tenants!  You need to work through the hands on labs in order to prepare yourself to pass the written test or the qual lab, it is vital that you work through as much of the labs as possible.  Time management is key.

I rarely ate dinner out, opting to have food delivered just so that I could get more time in the lab.  On weekends, I would try to start a load of laundry while studying notes and then I’d go back to the lab so that I could work with peers, asking for help on some of the more difficult assignments.  It is a delicate balance of sleep deprivation and will power.  I attended during fall, during college football season, and I just wanted to go to a bar and watch an NFL game or the Georgia Bulldogs play but instead I buried my face in the glow of the computer monitor to keep practicing, keep studying.  Those who know how much I love football know this took tremendous willpower.

The Commitment

To put in the work required means you really have to put aside all other commitments.  You aren’t going to be effective if you are having to excuse yourself for a conference call when the instructor is going over material that you really need to hear.  You really need to make sure that you can excuse yourself from work during the course duration.  The hard part means that you need to stay off email, Facebook, and everything else.  I was lucky to have a very supportive group of friends at work who covered for me while I was gone, sent reassuring emails telling me that everything was going great and to keep working hard, and chastised me to get the heck off Facebook and Twitter.  I can’t thank you guys enough Chad and Eric.

I have a loving wife and two young children.  Every day with them is precious to me, and I fight to create work and life balance, although that often means that I am working from home and they know to find me in my home office.  I travel frequently, but usually only for a few days at a time.  For the MCM program, I got an apartment near campus and lived there for 3 weeks by myself.  There was a 2-hour time difference that meant when I woke up at 6:00 am (recall the sleep deprivation that I emphasized before), my kids were already at school.  When I was eating lunch, they weren’t yet home from school, and when training was done at night they were already in bed.  I missed my kids so much it hurt.  By the end of the second week, I damned near had a breakdown when I checked my voicemail and a tearful daughter said “I miss you daddy.”  It gets really hard to focus on listening to technical training all day and then working all night in a lab when all you want to do is to hear your family say I love you.  My wife was fantastic, she wouldn’t tell me about any of the times the kids got in trouble, when the oven broke, about a tax audit letter we received, she just wanted me to focus.  We usually spend so much time talking to each other every night that this was something we both very much missed while I was away at the program, she didn’t have me to share her highs and lows every day and that really made an impact on her.

On weekends, we tried to use Skype, and it was wonderful to see everyone’s faces and hear their voices… but it was only fleeting.  I have to admit that this was something I fully didn’t realize how much it would impact me.  I didn’t realize at all how much it would impact my family.

The commitment isn’t just you, it’s your family as well. 

The Written Test

Once the three weeks of training have concluded, there is a written test.  This is not like any other Microsoft certification test that I have ever taken. 

I have long questioned the value of Microsoft certification tests, there are so many stories of young kids passing tests or, worse, people passing tests who have never used the technology before.  A friend of mine decided to see how many certifications he could achieve within 2 months, most he had little or no experience with, and he achieved 6 of the 8 he attempted.  Simply put: this test is nothing like that.  During the lectures, I would often listen to the lecture and try to think of what questions could be asked on the test.  I made up my own set of questions that I thought could be fair game on the test.  Not even close.

The questions were way harder, and often required you to put multiple complex concepts together to arrive at an answer.  There should be a camera outside the door as people leave the written test just to see their reactions, many F bombs are dropped by even the most experienced and attentive students.  You are under NDA, so you cannot talk about what’s on the test to each other or compare notes, you can only acknowledge each others’ frustrations.  You can’t even blow off steam by going to a bar and drinking too many beers because you are now looking forward to the next day where you take the qualification lab. 

Many people do not pass the written test on the first try.  To me, this is the hardest one to retake because it requires that you study all of your notes and re-read TechNet and Peschka’s blog and Spence’s blog making sure you understand the finer points, realizing you aren’t getting the same test a second time.  One strategy is to do a brain dump of as many questions and topics that you saw and what you aren’t sure if you got right and make sure you study those concepts.  You really do not want to put off the retake for too long because it becomes increasingly hard to find the time to study like this.

My strategy during the program was to focus my attention towards studying for the written test as much as I could because I knew that would be harder for me to attempt a second time.  I passed the written test on the first try, and I was admittedly shocked to see that I passed it. 

The Qualification Lab

You are given a lab environment and a set of tasks to complete in 8.5 hours.  The test was open-internet, so you could research whatever you needed to, but honestly if you were looking it up in the first place your chances of passing are very slim.  It’s nothing that an average SharePoint professional couldn’t reasonably achieve in a month’s time, but you are given just 8.5 hours and are graded on your ability to complete the tasks as described. 

Let me emphasize here… a month’s worth of work in 8.5 hours, done accurately.  Either you know it or you don’t.  If you know it, you can do it quickly and accomplish it within the time limit, if you are unsure then you are going to waste a bunch of time trying to figure it out.  The clock is ticking.

A boxed lunch was served, although I didn’t eat much of it because I didn’t want to lose any focus on completing the tasks.  It is stressful, because we all know how easy it is to forget a seemingly small step only to spend hours troubleshooting something.  Forget one timer job, and you can find yourself beating your head on the keyboard.  This is where it is vitally important to have paid attention in class as well as having experience.

I failed the qualification lab.  Twice.  It was that hard, and from speaking with many others who have attempted the course, they agree that the combination of the written test and qualification lab are more than an accurate measure of the level of competency expected of an MCM.  It is stressful, realizing how many times you are trying to accomplish the task as written and fumbling on things you know how to do, things you do every day for work.  I remember thinking during my first attempt at the test that I had a reasonable shot at passing… yeah, not even close.  The three weeks of lecture and demos, the late nights in the lab, the 16+ hour days on weekends, all that bolsters what you are expected to already know, it doesn’t teach to the test, and there’s only so much partial credit given for incomplete tasks.  Simply put, it was by far the hardest test I have ever taken because you had to be both thorough and fast. 

The first time I took the test I honestly wasn’t prepared.  I did horrible, but that was my strategy the whole time was to focus on the written test and then plan to retake the qual lab.  The second time I took the qual lab, I failed by only a few points and it was completely time management that did me in.  I knew all the material, I just took too long to do some of the tasks and the clock ran away from me.

I finally passed the qualification lab on my third and final attempt.  The first time you take the qualification lab, you are in a room with the proctor.  Your retakes can be done remotely while a proctor monitors you. 

The Aftermath

Maybe it took more for me to accomplish this certification than it would for you, maybe you have more experience than I did going in and you wouldn’t struggle as much.  Plenty of people pass the written and qual labs during rotation.  You should be prepared for what happens if you don’t and understand that if you didn’t pass the first time, you are really going to have to apply yourself to be able to pass the second time.

Many people do not pass one or both of the tests on the first attempt.  Some of the MCMs that I look up to most (including some of the instructors) failed one or both parts on the first attempt.  Some of the SharePoint hardcore geeks that I look up to did not pass and decided not to even attempt a retake.  It’s hard to think about making such a huge time investment (pre-reading study, independent lab practice, three weeks of very long days of lectures and labs, two grueling tests) and then having to make yet another huge time investment afterward.  For some, they recognize that they don’t have it to give.  For me, I wasn’t about to give up. 

After attending the program in person, you have 2 retake attempts before you have to repeat the entire program.  I know quite a few people who attempted it and decided not to pursue it after failing the first time, recognizing the amount of work that it would take to prepare for a retake. 

Prior to attending, I spent quite a few long hours at my desk at work trying to understand search and user profiles.  I spent about a week (which is way too short) studying the pre-reading list.  After attending the training program, I studied nights and weekend for over a year.  I gained new experiences through working with the product day in and out with customers, bolstering my experience with the product.  I presented on some of the material at conferences, and made sure that I was able to cover the areas that I did poorly on.  I studied nights, weekends, holidays.  I was lucky that many of the customer issues I was handling for work were directly related to the material that I was studying and that gave me additional experience.  I felt like I lived, breathed, ate, slept, and dreamed of SharePoint.  I drove my family crazy by sitting in my office making up scenarios with the product and trying to solve them to make sure I knew how things worked instead of enjoying time with them outside.  On vacations I was distracted, I spent an absurd amount of time focusing on making sure that I was ready for my third and final attempt.  I rarely blogged, and when I did I tried to blog about the topics that I was studying.  Yes, I worked nights and weekends and busted my tail preparing for my 2nd qualification lab attempt.  For my 2nd attempt at the qualification lab, I thought I was ready… I barely failed, but it was still a failure.  That only strengthened my resolve that I was going to finish what I started.

For the third qualification lab attempt, I worked nights and weekends and holidays and on vacation.  The problem was that it was getting tougher to focus.  I still had a day job, and I was now focusing less on SharePoint 2010 and more on SharePoint 2013.  I was traveling all over the world teaching developers about the new features in SharePoint 2013 for the Ignite program.  I was recording videos for MSDN, editing them, reviewing them, and re-recording them and doing much of that during nights and weekends.  That stuff helped me study through experience, but it threatened to reduce the amount of studying that I could do to prepare for the third attempt.  I finally had to put everything else on hold and focus for a few weeks solid to prepare.  Yes, I took vacation just to study for MCM.

This is the part that many don’t plan on, the aftermath of the program.  If you don’t pass the first time, it takes a dedicated effort to make sure that you are ready for the next attempt.  It takes quite a bit of sacrifice.  It’s sometimes hard to explain to my non-technical friends that I sacrificed a significant amount of family time in order to pass a certification exam.  This program consumed my life for most of 18 months to the point that I questioned what possible benefit could I receive to justify all this hard work other than a title to add to my resume.  I thought several times about quitting, but I just couldn’t quit, that’s not who I am.  Instead I sacrificed quite a bit to accomplish a goal that I had set for myself.

I failed the first and second attempt at the qualification lab, and finally passed on my third and final attempt.  Funny enough, I am glad I had the opportunity to take the lab three times as I was able to see a wide number of real-world scenarios that made me go back to my lab at home and work on them to completely understand them for the next test attempt.  Even if I didn’t understand them enough to pass the test the first time, I sure the heck understand them now!  The level of experience that I took away from just the qual labs and studying is invaluable.

The Reward

This was a personal goal, a way to prove to myself that I could take my skills to the next level.  Looking back on where I was before I started this program, I can say it was definitely worth my time to better myself as a SharePoint professional.  I learned so much throughout the program and made so many friends through the experience that professionally it was worth the investment.  I can now speak much more confidently about how the product works.  I now have a network of peers who went through the same certification process and passed, and I have this incredibly rich network of professionals to draw from.  I can sit in a room with the lead architect of a Fortune 50 company and provide factual guidance on how something should or should not be implemented.  I gained the respect of my peers because I showed so much dedication to finishing what I started. 

For me, this was much more than trying to just get a promotion at work.  Actually, I haven’t yet realized any monetary rewards and I’m not sure that I will, but I have gained a few new opportunities that otherwise I would not have had.  For me, the real payoff was the sense of accomplishment, something I can show my kids that I didn’t give up and that I worked extremely hard to achieve a goal.  Not everyone will appreciate just what it took for me to accomplish this goal, but my family knows.

I did it.  I set a goal, I worked hard but not hard enough, then I buckled down and really pushed through.  I am proud of myself, and not at all ashamed to tell anyone how hard I worked towards this goal.  I don’t brag about being an MCM and don’t flaunt my credentials, in fact it was a tremendously humbling experience.  That said, I am not at all bashful about telling you how hard I worked to get there.  And I certainly am telling my kids.

There is nothing more valuable to me than being able to show my kids that it was tough, but if you work hard you can achieve your goals.  Once you accomplish your goals, sit back, reflect, and appreciate the hard work that you put into it and share that with others.

This post will show you how to inspect the SharePoint 2013 context token to better understand how OAuth is used in SharePoint 2013 apps.

First, Some Context

In order to use a context token with SharePoint 2013 apps, you will need to create a provider-hosted app that uses a client ID and a client secret.  This requires that the target SharePoint farm has a trust configured to Azure Access Control Services, or ACS.  Office365 automatically configures this trust for you, so if you create your app using Office365 then this works just fine.  If you create your app using your own SharePoint farm, then you will need to configure low-trust apps for on-premises deployments in order to establish a trust with Azure ACS for your SharePoint farm.

When you use a client ID and a client secret to build your app, you are using Azure ACS as the authentication server.  The OAuth flow looks like this:

This is what I jokingly refer to as “the scary slide”.  It’s not really that scary once you understand it.

1. User makes a request to SharePoint
2. SharePoint requests the context token from Azure ACS
3. The signed context token is returned
4. The HTML markup for the SharePoint page is sent to the client
5. If the SharePoint page contents contain a client app part, then a request is sent to the remote web server for the IFrame’s contents.  If the user simply clicked on the app to launch it, then a page in SharePoint called appredirect.aspx is used to redirect to the app’s start URL.  In either case, a context token is sent along with that page request in a form parameter called SPAppToken. 
6. The context token is a JWT token.  The JWT token is a set of base64 encoded claims.  One claim is named “refreshtoken” that has a base64 encoded value.  One of the claims is “appctx” which contains a child object, one of the properties is SecurityTokenServiceUri with a value https://accounts.accesscontrol.windows.net/tokens/OAuth/2.  Your app uses the TokenHelper.cs class to extract the refreshtoken and then send it to the SecurityTokenServiceUri to request an access token.
7. The access token is retrieved from ACS.
8. The access token is passed to the SharePoint site in the HTTP header Authorization that has a value beginning with “Bearer” and has your access token.  This token is passed in the HTTP header to the CSOM endpoint _vti_bin/client.svc/ProcessQuery when you use the CSOM.
9. If the app is authorized, SharePoint returns the data requested.
10. Finally, the app server returns the requested HTML markup as a response. 

The really important thing to understand is what’s passed to your app, the context token.  The context token contains a few key pieces of information necessary for the rest of the plumbing to function.

What’s In Your Wallet?

Now that we’ve walked through the steps of the OAuth dance, let’s take a look at what’s in that context token that is critical to the whole process.  When you use Visual Studio to create a provider-hosted app, the boiler plate code looks like this:

protected void Page_Load(object sender, EventArgs e)
{
    // The following code gets the client context and Title property by using TokenHelper.
    // To access other properties, you may need to request permissions on the host web.

    var contextToken = TokenHelper.GetContextTokenFromRequest(Page.Request);
    Response.Write(contextToken + "<br/>");
    var hostWeb = Page.Request["SPHostUrl"];

    using (var clientContext = TokenHelper.GetClientContextWithContextToken(hostWeb, contextToken, Request.Url.Authority))
    {
        clientContext.Load(clientContext.Web, web => web.Title);
        clientContext.ExecuteQuery();
        Response.Write(clientContext.Web.Title);
    }
}

Reading that code, you can see that the context token is passed to your app, but it is a base64 encoded string that is unintelligible.  I added a Response.Write statement to write the context token to the page, but you can also see it in Fiddler.  In the following screen shot, I highlighted the SPAppToken HTTP form variable where the context token is passed.

Once you have that context token, you have the base64 encoded value of the context token, which is a JWT token.  The JWT token format is pretty straightforward, and the TokenHelper.cs class in Visual Studio simply parses that JWT token for you.  However, for the curious who want to see what it looks like, you can use a tool like http://openidtest.uninett.no/jwt to inspect the JWT token.  Copy the context token string into the “Encoded JWT” text box and the client secret into the “Secret” text box, and then press decode. 

What is returned is the following JSON object, which is a JWT token that contains a set of claims.

{
    "aud": "4c2df2aa-3d14-4d84-8a79-5a75135e98d0/localhost:44346@d341a536-1d82-4267-87e6-e2dfff4fa325",
    "iss": "00000001-0000-0000-c000-000000000000@d341a536-1d82-4267-87e6-e2dfff4fa325",
    "nbf": 1365177964,
    "exp": 1365221164,
    "appctxsender": "00000003-0000-0ff1-ce00-000000000000@d341a536-1d82-4267-87e6-e2dfff4fa325",
    "appctx": "{\"CacheKey\":\"em1/saZohTOS4nOUZHXMb8QJgyNbkEO86TSe5j9WYmo=\",
\"SecurityTokenServiceUri\":\"https://accounts.accesscontrol.windows.net/tokens/OAuth/2\"}",
    "refreshtoken": "IAAAANc8bAVMWZceOsdfgsdfggbfm7oU_aM7D2qofUpQstMsdfgsdfgfYS0OtbZ-
eY9UQGvlYSl5kpPi913G1AwIVBMxoCux8-bhcCCiaGVo-vuFzrXetdhRGPftQdHh-
1rS5cvDuuQ_bw_mjySIyuHNGSavEs8HUgHY9BOVc3pTGZtZ_nS-
1NbDLYObjnznasdfasdfasdfQreLAeeOpVRY1PGsdfgsdfgOITA3BKhjJFz_40YJMubdHmY2OTS
nqwNnUe-rBBCtfvKt4xFWvdRzTzwfW",

    "isbrowserhostedapp": "true"
}

What are the claims in the context token?

The following shows the properties for the context token.

During Ignite training, I tell the attendees that the most important thing you can understand when building apps for SharePoint is OAuth.  You need to understand the OAuth dance, the reason for the context token, and it is invaluable to understand what the TokenHelper.cs is doing for you when it does things like GetClientContextFromContextToken.  This post goes into some of the details on why this is such an important piece to understand when troubleshooting apps.  You can also see why it is mandatory for apps to SSL due to the nature of information being sent in HTTP. 

You can find more information about the context token, calculating nbf and exp claim values, and links to additional articles at http://msdn.microsoft.com/en-us/library/office/apps/fp179932.aspx. 

For More Information

configure low-trust apps for on-premises deployments

JWT token format

Tips and FAQs: OAuth and remote apps for SharePoint 2013

I worked this weekend to prepare for a presentation this week at the Dallas SharePoint TechFest, “Access Services 2013 – Not Your Father’s Access!”  I am covering the new features of Access 2013 as an app designer for SharePoint 2013.  While presenting on Access, I frequently say that while Access has changed drastically, Access remains familiar because key parts of the UI have barely changed since 1993.  As proof, I installed Windows XP and Access 2.0 and took some screen shots for comparison.

Creating a table:

Creating a query:

Even the wizards to create tables have pretty much the same elements as they did back in Access 2.0 with the ability to choose from out of box templates.

The thing that is drastically different is the ability for end users to use Access 2013 with Access Services 2013 to create apps for SharePoint 2013.  That’s the part that I am very excited about because it provides the ability for end users to create composite applications without requiring developers.  Looking forward to speaking at the event!