Welcome to my blog, stay tunned :
Home | Blogs | Stephane Eyskens's blog

Leveraging AppFabric for custom caching in SharePoint 2013

Hi,

[Update] the scenario depicted in this blog post is not supported. I wrote it in 2012 with the public beta of SharePoint and I had doubts about supportability. Microsoft confirmed 1 year later that this is indeed not supported to use SharPoint's AppFabric for custom caching. You should create another cluster instance...

As you might have noticed, AppFabric is now one of the pre-requisites when installing SharePoint 2013. SharePoint makes use of it for workflows and activity feeds.

When I noticed that AppFabric was part of the pre-requisites, I directly thought of trying to figure out whether SharePoint would give developers new APIs to leverage AppFabric's distributed cache capabilities. As SharePoint developers, on the 2007/2010 versions, when it came to caching, we always had to either use ASP.NET cache (mostly inproc), either rely on third parties such as NCache or AppFabric but we had to install/configure those ourselves as they were not integrated to SharePoint.

Unfortunately, at the time of writing, Microsoft did not release any public API enabling us to benefit from a Farm-Wide caching system. Indeed, the classes leveraging AppFabric are all marked internal, so there is no public API. However, the good news is that we know AppFabric will be deployed on every SharePoint setup, therefore, we can start using it the context of SharePoint using the AppFabric client DLLS.

I built a wrapper that you can download here http://sptoolbasket2013.codeplex.com/ whose the purpose is to ease the use of the AppFabric layer in the context of SharePoint 2013. I should add that this is currently at an experimental stage since I've never used that for a real project and I don't even know whether this wrapper would be supported by Microsoft :).

I'll describe hereafter all the testings I made with this wrapper & AppFabric in SharePoint 2013.

Table of contents





Topology used for my testings

In order to test a distributed cache system, you really need to be in a farm configuration, stand-alone install is not sufficient. I built an environment on CloudShare which was made up of the following:

- A domain controller hosting the SharePoint services, WFE and SQL which I'll refer further to as C6208759806

- A WFE that I joined to the existing farm which I'll refer further to as C7763875784

That way, we'll really see if cache is indeed distributed :).

Services

In order to use distributed caching, you need to ensure that the Distributed Cache Service is started on both servers as shown below:

A corresponding windows service should also be up & running :

All the above is normally done automatically upon SharePoint installation but I prefer to mention it in case something went wrong during setup.

Checking the AppFabric configuration

The AppFabric Way

You can easily check if the AppFabric cluster is correctly configured by opening the PowerShell window labelled Caching Administration Windows. Then, you can type the following commands:

which will give you information about the AppFabric cluster configuration. One more command will give you information about SharePoint's usage of AppFabric:

As you can see from the above screenshots, AppFabric listens to specific ports other than 80/443. Firewalls must be configured accordingly

The SharePoint way

Open the SharePoint Management Shell, you'll find the AppFabric commands + some specific SharePoint cmdlets. They are listed in the below table:

Add-CacheAdmin Add-CacheHost Clear-CacheLogging Export-CacheClusterConfig
Get-Cache Get-CacheAllowedClientAccounts Get-CacheClusterHealth Get-CacheClusterInfo
Get-CacheConfig Get-CacheConnectionString Get-CacheHost Get-CacheHostConfig
Get-CacheRegion Get-CacheStatistics Get-SPDistributedCacheClusterInfoManager Grant-CacheAllowedClientAccount
Import-CacheClusterConfig Invoke-CacheGC New-Cache New-CacheCluster
Register-CacheHost Remove-Cache Remove-CacheAdmin Remove-CacheCluster
Remove-CacheHost Restart-CacheCluster Restart-CacheHost Revoke-CacheAllowedClientAccount
Set-CacheClusterSecurity Set-CacheConfig Set-CacheConfigurationLog Set-CacheConnectionString
Set-CacheHostConfig Set-CacheLogging Set-SPDistributedCacheClusterInfoManager Start-CacheCluster
Start-CacheHost Stop-CacheCluster Stop-CacheHost Stop-CacheHostShutdown
Stop-CacheNonUpdatedHosts Test-CacheClusterConnection Test-CacheConfigAvailability Unregister-CacheHost
Update-CacheConfigurationStore Update-CacheHostAllowedVersions Update-SPMicroblogFeedCache Update-SPMicroblogLMTCache

and the one you should always start with : Use-CacheCluster.

Description of the wrapper

As stated in the introduction, the wrapper is aimed at facilitating the usage of AppFabric within SharePoint. It's not making a complete abstraction of the underlying AppFabric layer but helps in creating factories according to the SharePoint config & topology.

It does the following:

  • Allows you to work also with SharePoint 2010 providing you have installed AppFabric. The wrapper comes up with a configuration page in order to let you specify which servers are in use
  • Detects which SharePoint servers are running the distributed cache service and add those when creating the factory
  • Makes use of SharePoint's default cache area that is created automatically upon SharePoint installation
  • Allows you to easily enable local cache
  • Allows you to easily enable local cache notifications providing the cluster is correctly configured

Note that you can specify explicitely to use another cache than SharePoint's default one. However, that cache must be created by an administrator using the PowerShell cmdlet New-Cache prior to consuming it from code.

Getting started with the wrapper

Using the wrapper, the simpliest way of adding an item to the cache is this one:

SPDistributedCacheWrapper.Cache.Add("key", "hello SharePointers");



You first need to add a reference to the following DLLs:

  • SilverIT.SharePoint.AppFabric
  • Microsoft.ApplicationServer.Caching.Client
  • Microsoft.ApplicationServer.Caching.Core

The wrapper can be used with the default settings that are calculated dynamically whenever a new factory gets instanciated. However, this operation is quite heavy. The wrapper offers two configuration modes :

  • Dynamic : you can make use of the SPCacheConfig class to specify the behavior (localcache, named cache, timeouts etc..)
  • Static : you can use a specific configuration page where you define once for all the parameters that the wrapper is supposed to use. This page looks like this: and what you specify there is persisted in a SPPersistedObject

Note that the latter approach takes priority over the dynamic one. For a full explanation of the wrapper and its configuration, go visit the documentation on the CodePlex project

Cache Invalidation

All the objects stored in the cache will expire after the time specified in parameter to the Add() method which has a few overloads and one of them allows you to specify the time to live for a specific object. So, if you add an object the following way:

SPDistributedCacheWrapper.Cache.Add("key", "hello SharePointers", new TimeSpan(0,5,0));

It will remain in cache during 5 minutes. If you add an object to the cache the following way:

SPDistributedCacheWrapper.Cache.Add("key", "hello SharePointers");

It will use the default TTL of the SharePoint cache.

Of course, if the objects get modified, they will also get invalidated.

Local Cache

AppFabric provides a way to enable local cache. When local cache is enabled, if a consuming process requests a specific object, it keeps it in its own memory for further uses. The advantage is that subsequent accesses to the cached objects are faster since you avoid an extra roundtrip to the cluster cache and you avoid the serialization process that occurs along with the roundtrip. To observe that behaviour, you can easily write a small console application, you enable local cache programmatically (see below) and you can for instance add 10 objects of +- 1MB each to the cache, after that, you read them 1 by 1 in a loop with a sleep of 2 seconds from another instance of that console app and you monitor the memory usage with the Task Manager. You will clearly see that every 2 seconds, the memory usage of this console app will increase of more or less 1MB.

If however, you do not enable local cache and do the same "exercise", you'll notice that your memory will increase/decrease/increase/decrease etc...and not constantly increase anymore.

In SharePoint, it's the worker process w3wp.exe that will store locally cached objects. To enable local cache using the wrapper, you just have to do the following:

SPCacheConfig.EnableLocalCache();

this will by default use Timeout based expiration policy. I actually created 5 overloads which are :


Timeout based expiration with default object count and default timeout.


Timeout based expiration with specific timeout.


Timeout based expiration with specific timeout & specific object count


Notification based poclicy with specific timeout & specific polling time


Notification based poclicy with specific timeout, specific polling time and specific object count

If you want to use notifcation based expiration, you can do this:

SPCacheConfig.EnableLocalCache(new TimeSpan(0, 10, 0),new TimeSpan(0, 5, 0));
 



With the above example, the system will check every 5 minutes whether there is a notification for locally cached objects. If nothing has changed, it will keep using the locally cached object until the timeout for the locally cached object is reached.
Note that you often see on the web timeout based or notification based. In reality, it's rather timeout based and notification based because whatever method you're using, you'll have to specify a local cache timeout even if you're using the notification based policy.

In the context of SharePoint, if you want to use the notification based expiration policy, you can enable it the following way with PowerShell:

Use-CacheCluster
Stop-CacheCluster
$f=Get-SPFarm
$id=$f.Id
Set-CacheConfig -CacheName "DistributedDefaultCache_$id" -NotificationsEnabled $true
Start-CacheCluster

Please note that this might not be supported by Microsoft and make sure you test carefuly this configuration on dev/integration/acceptance environments instead of rushing in production :). Note that when using Notification based policy, the timeout specified in the local cache config is still used by the system meaning that some items could be evicted from the local cache although no notification was triggered.



Different running contexts

As you might have noticed from the above examples, AppFabric can be used from non-web contexts.
This offers the following big advantages, in the context of SharePoint :

  • You can build time consuming cached object from within a Timer Job for instance
  • Your cached objects can be shared accross different SharePoint web apps
  • Your cached objects will resist to an application pool crash, IISRESET
  • The wrapper facilitates the usage of AppFabric
  • The SharePoint Infrastructure offers a good abstraction layer to manage AppFabric and facilitates admin's life
  • Providing you're running in a farm context and that you started the DistributedCache service on more than 1 server, you've got a HA cache system (read more on that below)

Distributed Cache Accross the SharePoint Farm

To perform the below tests, I wrote a very basic SharePoint WebPart that I added on a page served by my servers C7763875784 and C6208759806 having both the WFE role enabled.

Consider the following code (reusing the cache wrapper explained earlier) :

const string CacheKey = "DummyCachedItem";
const string CacheValue = "Dummy Value";
protected override void Render(System.Web.UI.HtmlTextWriter writer)
{
    string CachedItem = SPDistributedCacheWrapper.Cache.Get(CacheKey) as string;
    if (CachedItem == null)
    {
        writer.Write("Adding item to the cache<br/>");
        SPDistributedCacheWrapper.Cache.Add(CacheKey, CacheValue);
    }
    writer.Write("Got {0} from the cache", CachedItem);            
}  

in which I store a simple string in the cache, I first check whether it's already in the cache, if not I add it.

Cache Distribution

I went to the server C6208759806, browsed the home page and added the webpart which gave me the following result:

As expected, the string was not in the cache so the webpart has added it. After a page refresh, you get the following:

again, here, expected behaviour since now the item could be obtained successfully from the cache.

Then I went to server C7763875784 and just browsed the home page which returned this:

As you can see, the request has been served by C7763875784 not by C6208759806 where we inserted the item in the cache and the webpart could read the value without any problem. This simple basic test shows that the cache looks to be indeed correctly distributed. With the OOB ASP.NET caching, each w3wp.exe process would have had its own independent cache state on each server. Here, we clearly see that we get the same item out of the cache but is the system really distributed (see next section...)?

Testing HA

For this test, I went to Central Administration - System - Services on Server. Then I stopped the service Distributed Cache on C7763875784. I double-checked whether this was indeed stopped at AppFabric level:

Then I simply refreshed the page on C7763875784 and could still read the cache without any problem.

However, this was working because I stopped C7763875784 not C6208759806. If I repeat this scenario but if I stop C6208759806 instead of C7763875784, then my item cannot be found in the cache anymore and is re-cached on C7763875784...

What does this mean? It means that while C6208759806 was running, he was always used whenever SPDistributedCacheWrapper.Cache.Add() was called. I decided to go one step further to find out why the hell data was not replicated accross hosts that belong to the cache cluster.

I first read more carefully some AppFabric documentation :). Then, I found out that at the time of writing, SharePoint does not configure high availability automatically. It means that if you want the Cache-Cluster to replicate the cached items between hosts, you need to configure it yourself. Here is how to proceed, bear in mind that I'm just sharing the results of my findings, you should of course analyze the impact of such changes on your farm but at least, you can try this approach in a sandbox and decide later whether or not it is a good idea :).

To enable HA on the SharePoint Farm, you need to:

  • Ensure that your firewalls allow good AppFabric communication, see this link for more info http://msdn.microsoft.com/en-us/library/ee790914.aspx
  • Open a SharePoint PowerShell session
  • run Stop-CacheCluster
  • run the following :
  • Then run this command:
  • Set-CacheConfig DistributedAccessCache_8006e840-d454-4444-bd33-852445d36870 -Secondaries 1
    
  • Start-CacheCluster (or do it from CA)

From now on, any object cached with the wrapper is replicated accross hosts belonging to the cluster, giving you not only connectivity High Availability but also your cached items are Highly Available.

One step further

Locking

The AppFabric client DLLs deployed by the SharePoint setup provide two methods to handle locking. Consider the following code snippet:

public void UpdateItem<T>(T o) where T : ISharePointData<T>
{
    string key =o.CacheKey.ToString();
    string region = GetRegionName(o.GetType());
    DataCacheLockHandle CacheLock;
    SPDistributedCacheWrapper.Cache.GetAndLock(key, TimeSpan.FromMilliseconds(100), out CacheLock, region);
    SPDistributedCacheWrapper.Cache.PutAndUnlock(key, o, CacheLock,region);
}

Say an update method that receives an object as argument that you need to update. You can first call GetAndLock. This will try to lock the said object, in this case during 100ms. The lock handle is returned to CacheLock. The call of PutAndUnlock will update this object and explicitely unlock the object after the update is complete.

If you don't call the PutAndUnlock method explicitely, the lock will expire after the number of milliseconds (ticks/minutes/hours...) you have defined in GetAndLock. Note that GetAndLock will throw an exception is the object is already locked out by another process/thread.

Serialization

As I stated a few times earlier, with AppFabric, the objects you put in cache are serialized. Therefore, these objects must be decorated with the Serializable attribute. Objects such as SPWeb are not serializable, therefore, you can't cache an entire SPWeb as is.

That's anyway not a good idea since SPWeb is also not thread-safe. AppFabric makes use of NetDataContractSerializer to serialize objects. This serializer helper makes bigger cached objects than the default BinaryFormatter used by ASP.NET. On top of that, if you're used to work with the latter, pay attention not to encounter deserialization "conflicts".

Working with regions

AppFabric regions help representing a logical architecture that matches your application designs. In the context of SharePoint, you may want to cache User Profiles related objects in a separate region called "UserProfiles" and you may want to cache custom application data in "ApplicationDataxxx". Regions are created at runtime programmatically and have a small overhead in terms of space used (a few KG per region).

However, unless you create zillion of regions, you should not be afraid by the extra consumed space. When HA is enabled, regions are also replicated accross the cluster hosts. Regions also offer additional features such as Tagging (explained below).

It is very easy to create a region, you can just use the following line of code:

SPDistributedCacheWrapper.Cache.CreateRegion(args[0]);

If the region already exists, it doesn't recreate it. Another advantage I see when working with Regions is that you can reduce race conditions/multi-threading issues by storing individual objects instead of List or Dictionaries. If you have a List that's read/updated frequently, you might quickly run under race conditions since the entire List object will be either read, either updated for every operation. If, instead, you store each invidual object with a specific key in a region, you'll be able to update them individually, thus, reduce the number of concurrent threads that are reading/writing the same object.

Of course, that depends on the context, on the number of objects you have to deal with etc...If you're only reading or perform CRUD. All those aspects must be taken into account before taking a decision.

BulkGet

BulkGet allows developers to retrieve several cached items at once from a specific region. The following examples shows this in action:

[Serializable]
class CustomType1
{
    public CustomType1(){}
    public string SayHello()
    {
        return "Hello"; 
    }
}
[Serializable]
class CustomType2
{
    public CustomType2() { }
    public int Add(int a, int b)
    {
        return a + b;
    }

}

class Program
{
    static List<string> Keys = new List<string>();
    const string region="MyRegion";
    static void Main(string[] args)
    {
        SPDistributedCacheWrapper.Cache.CreateRegion(region);
        CreateCustomTypes();
        var CachedItems = SPDistributedCacheWrapper.Cache.BulkGet(Keys,region);
        foreach (KeyValuePair<string, object> CachedItem in CachedItems)
        {   
            if(CachedItem.Value is CustomType1)
                Console.WriteLine("Key : {0} says : {1}", CachedItem.Key,((CustomType1)CachedItem.Value).SayHello());

            if(CachedItem.Value is CustomType2)
                Console.WriteLine("Key : {0} computes : {1}", CachedItem.Key, ((CustomType2)CachedItem.Value).Add(1,1));

                                
        }

            
        Console.WriteLine("done");
           
    }
    static void CreateCustomTypes()
    {
        for (int i = 10; i < 20; i++)
        {
            Keys.Add(i.ToString());
            if ((i % 2) == 0)
                SPDistributedCacheWrapper.Cache.Add(
                    i.ToString(),
                    new CustomType1(),
                    region);
            else
                SPDistributedCacheWrapper.Cache.Add(
                    i.ToString(),
                    new CustomType2(),
                    region);
        }
    }
}

This gives the following output:

Cache Search

You can now search for any cached item matching certain tags. This can be particularly useful since you're free to associate any tag you'd like to your cached objects. It's a kind of keyword association.

Reusing the same CustomType classes than earlier, here is a code sample that shows how to make use of this search mechanism:

public enum CustomTags
{ 
    CustomType1,
    CustomType2
}

class Program
{
    static List<string> Keys = new List<string>();
    const string RegionName = "TaggedItems";
    static void Main(string[] args)
    {
        SPDistributedCacheWrapper.Cache.CreateRegion(RegionName);
        CreateCustomTypes();
        var CachedItems = SPDistributedCacheWrapper.Cache.GetObjectsByAnyTag(
            new List<DataCacheTag>() {new DataCacheTag(CustomTags.CustomType1.ToString()) }, RegionName);

            
        foreach (KeyValuePair<string, object> CachedItem in CachedItems)
        {                   
            Console.WriteLine("Key : {0} says : {1}", CachedItem.Key,((CustomType1)CachedItem.Value).SayHello());                                
        }

            
        Console.WriteLine("done");
           
    }
    static void CreateCustomTypes()
    {
        for (int i = 10; i < 20; i++)
        {
            Keys.Add(i.ToString());
            if ((i % 2) == 0)
                SPDistributedCacheWrapper.Cache.Add(
                    i.ToString(),
                    new CustomType1(),
                    new List<DataCacheTag>(){new DataCacheTag(CustomTags.CustomType1.ToString())},
                    RegionName);
            else
                SPDistributedCacheWrapper.Cache.Add(
                    i.ToString(),
                    new CustomType2(),
                    new List<DataCacheTag>(){new DataCacheTag(CustomTags.CustomType2.ToString())},
                    RegionName);
        }
    }
}

You can associate more than one tag to a cached object, so with a slight change to the previous example:

public enum CustomTags
{ 
    CustomType1,
    CustomType2,
    CommonTag
}

class Program
{
    static List<string> Keys = new List<string>();
    const string RegionName = "TaggedItems";
    static void Main(string[] args)
    {
        SPDistributedCacheWrapper.Cache.CreateRegion(RegionName);
        CreateCustomTypes();
        var CachedItems = SPDistributedCacheWrapper.Cache.GetObjectsByAnyTag(
            new List<DataCacheTag>{new DataCacheTag(CustomTags.CommonTag.ToString())}, RegionName);

            
        foreach (KeyValuePair<string, object> CachedItem in CachedItems)
        {                   
            Console.WriteLine("Key {0} of type {1} is at least mrked with the CommonTag ", 
                CachedItem.Key,
                CachedItem.Value.GetType());                                
        }

            
        Console.WriteLine("done");
           
    }
    static void CreateCustomTypes()
    {
        for (int i = 10; i < 20; i++)
        {
            List<DataCacheTag> Tags = new List<DataCacheTag>(); 
            if((i%3) == 0)
                Tags.Add(new DataCacheTag(CustomTags.CommonTag.ToString()));

            Keys.Add(i.ToString());
            if ((i % 2) == 0)
            {
                Tags.Add(new DataCacheTag(CustomTags.CustomType1.ToString()));
                SPDistributedCacheWrapper.Cache.Add(
                    i.ToString(),
                    new CustomType1(),
                    Tags,
                    RegionName);
            }
            else
            {
                Tags.Add(new DataCacheTag(CustomTags.CustomType1.ToString()));
                SPDistributedCacheWrapper.Cache.Add(
                    i.ToString(),
                    new CustomType2(),
                    Tags,
                    RegionName);
            }
        }
    }
}

where I also add a second tag to all items having a "modulo 3" key compliant, I can easily make a query to get all the items that were at least tagged with CommonTag. Here is the output of the above code:

Cache Exceptions

Some specific exceptions with self-explicit names can be thrown by SharePoint components using the distributed caching. They all start with SPDistributedCache....On top of those exceptions, if you're using the wrapper, you might encounter the regular AppFabric exceptions (similar to the SharePoint ones) + a specific exception from the wrapper which is SPDistributedCacheWrapperNoService.
This exception is raised if no server in the farm is running the Distributed Cache service.



Real World Scenario, caching User Profiles for end user search queries

A real world scenario could be that you're storing the User Profiles within the Cache because you need to perform a lof of queries such as "autocomplete search boxes", custom advanced search centers etc...and that you don't want to target the OOTB SharePoint search to avoid overloading it.

A while ago, I demonstrated a custom use of oData to perform such tasks, the post is available here. It would be possible to modify it to use AppFabric instead of the worker process to cache the items.

Conclusions

A lot of topics I talked about in this article might be subject to discussions. AppFabric is now required by SharePoint 2013 but the primary purpose is to serve SharePoint...In this article, I've showed how we can "deviate" from the expected OOTB behavior in order to use AppFabric for custom caching. I recall that I've not tried that approach on a real world project plus at few things I'm doing in this article might not be supported by Microsoft. These are :

  • Consuming the default SharePoint cache
  • Configuring the default SharePoint cache to be highly available
  • Configuring the default SharePoint cache to use notification based expiration

The two last bullets are actually completely optional. Also, we are still working with the preview and might expect Microsoft to deliver a public API or a specific SharePoint guidance to deal with the underlying AppFabric layer.



Happy Coding!

Comments

Sharing the cache with another ASP.NET application

Hi,

Thank you for your post and a very, very good one. I have a question - if I create certain cache variables/object using AppFabric, is it possible for a plain ASP.NET application (resides on another server) to configure AppFabric to the same set of servers and retrieve the same cached variables/objects?

Cache sharing

Hi,

Yes it is possible! You have two possible scenarios:

- you install AppFabric on your ASP.NET server and you add it as a node to the existing cluster
- you simply use the AppFabric client DLLs and you indicate where is the cache server.

In the context of SharePoint, I don't know if those scenarios will be supported or not by Microsoft.

Best Regards

Using AppFabric cluster for custom caches

Please keep in mind that using the SharePoint Cache Cluster to add custom caches or add non Distributed Cache Hosts to the SP Cluster is NOT supported.

Filip

Supportability

Hi,

Thanks for your comment but as I stated in my blog post which I wrote more than 2 years ago using the Beta of 2013, this scenario might not be supported and indeed, it's not. Now it is confirmed but they published the article 1 year after my blog post :). So, indeed, people should follow the instructions stated there:

http://technet.microsoft.com/en-us/library/jj219572.aspx#plandc

I never update my blog posts and that's why I often use the words "at the time of writing"...

Best Regards

Maryann Murphy

woot, thankyou! I finally came to a site where the webmaster knows what they're talking about. Do you know how many results are in Google when I search.. too many! It's so annoying having to go from page after page after page, wasting my day away with thousands of people just copying eachother's articles… bah. Anyway, thankyou very much for the info anyway, much appreciated.