Creating a UCMA 3.0 Trusted Application and Endpoint

This is a continuation of my previous post: Installing UCMA 3.0 and Creating a Communications Server “14” Trusted Application Pool.

Now that you have a trusted application pool set up, you can create and host UCMA 3.0 applications.  But first, you need to create a trusted application and application endpoint for your UCMA 3.0 application.

Create a UCMA 3.0 Trusted Application

Fire up the Communications Server Management Shell.  Use the New-CsTrustedApplication cmdlet to create a trusted application, for example:

New-CsTrustedApplication
-ApplicationId “MyUC14App”
-TrustedApplicationPoolFqdn “ts.fabrikam.com”
-Port 10607

where:

• ApplicationId is a unique name for your UCMA 3.0 application.
• TrustedApplicationPoolFqdn is the fully qualified domain name of your trusted application pool.
• Port is the port that Communications Server “14” will listen for your application on.

Note that to run these PowerShell cmdlets, the user you are logged in as needs to be a member of RTCUniversalServerAdmins, and also a local administrator on the server.

Create an Application Endpoint

To interact with your UCMA 3.0 trusted application, you need an application endpoint – or contact.  Use the New-CsTrustedApplicationEndpoint cmdlet to create an application endpoint, for example:

New-CsTrustedApplicationEndpoint
-ApplicationId “urn:application:MyUC14App”
-TrustedApplicationPoolFqdn “ts.fabrikam.com”
-SipAddress sip:myuc14app@fabrikam.com
-DisplayName “My UC 14 App”

where:

• ApplicationId is the id of trusted application created in the previous step – in the format urn:application:applicationid.
• TrustedApplicationPoolFqdn is the fully qualified domain name of your trusted application pool.
• SipAddress is the Sip address that you would like to assign to the application endpoint – in the format sip:username@domain.com.
• DisplayName is the display name of the application endpoint that shows up in Communicator.

Search for the contact in Communicator. 

Note that the presence of the contact is still Offline.  In the next post, I’ll show you how to publish the presence of the application endpoint.

HTML Presence Controls for Communications Server “14” – CodePlex Project

Showing Presence on the Web

If you’re running Office Communicator Server 2007 R2, you know that your only out-of-the-box option for showing presence on the web is to use the NameControl ActiveX control that ships as part of Office.  Being an ActiveX control, this obviously means that you’re limited to Internet Explorer. 

Also, nobody likes ActiveX controls…

In Microsoft Communications Server “14”, Microsoft will be shipping Silverlight (and WPF) controls for presence and other Communicator capabilities such as contact lists, contact card, and search.  Those controls are great for intranet based applications since they rely on Communicator running on the user’s machine.

What if you want to show the presence of users in a pure ASP.NET or HTML application and can’t assume that the user has Communicator installed – you need anASP.NET or HTML presence control. 

HTML Presence Controls for Microsoft Communications Server “14”

We recently worked with the UC team at Microsoft on a keynote demo for TechEd 2010 in New Orleans.  The demo was for a fictitious airline – Fabrikam Airlines – that wanted to show the presence of customer service and reservations agents on its website. 

Customers could also start an instant message conversation with the agents using a Silverlight web chat window that used WCF to communicate with the backend UCMA application.

We built HTML Presence Controls that use AJAX to poll a REST-based WCF service running in IIS and hosting a UCMA 3.0 presence subscription application.  

Microsoft has graciously allowed us to publish these on CodePlex so that the development community can benefit from them:  http://htmlpresencecontrols.codeplex.com/

We will be maintaining the CodePlex project as new builds of UCMA 3.0 become available.  Check out the project home page on CodePlex for some more in-depth details on how the controls are implemented.

ASP.NET Server Control Implementation

We’re providing an ASP.NET Server Control implementation that you can use stand-alone or in a GridView or Repeater
(or other layout control). 

The control has properties that allow you to control its appearance, e.g. you can choose whether or not to show the contact’s name or availability text.

You can also use the server control in a layout control such as a GridView by putting it in a TemplateColumn and binding to the Sip Uri in the data source.

Disclaimer

Once we started working on these, we realized why Microsoft hasn’t shipped such controls as part of the product.  There are some tradeoffs you have to be aware of when using these controls, here’s the high level.

Privacy

The backend UCMA 3.0 application that subscribes to presence of contacts runs as a trusted application and can thus retrieve the presence of any user in the organization. 

There’s currently no good way in UCMA to apply any privacy rules to ensure that the consumer of the presence controls has permission to see the presence of the contacts that the controls are bound to. 

Just to be absolutely crystal clear … These controls provide a way to query the presence of any user in the organization, regardless of the privacy relationship between the person consuming the controls and the contacts whose presence is being displayed.

We’re exploring options for a design pattern that would allow you to inject some privacy controls.  Keep in mind though that you would most likely be responsible for implementing this logic, as there is currently no functionality in UCMA that allows you to do that.

Polling the WCF REST Service

The controls poll the backend WCF service to retrieve the presence of contacts - you can control the refresh interval so that they poll less often.

We implemented a caching layer so that the WCF service is always communicating with a “presence cache” – it never communicates directly with Communications Server. 

For example, if your web page is showing the presence of sip:georged@fabrikam.com and 500 people have the page open, the presence cache only contains one instance of the subscription – Communications Server is not being polled 500 times for the presence of that contact.

Once the presence of a contact changes, it is updated in the cache. 

There are some server-based push mechanisms that would work nicely here, such as the one that Outlook Web Access 2010 uses.  Unfortunately we didn’t have time to explore these options.

Community Contribution

Take a look at the project Issue Tracker, there are a couple of things we can use some help with.  Shoot me a note if you’re interested in contributing to the project.

I’ll be Speaking at ILTA’s SharePoint for Legal Symposium on June 16th 2010

I’ll be speaking at the International Legal Technology Association’s “SharePoint for Legal” Symposium on June 16th 2010 at Microsoft’s offices in Downers Grove, IL. 

My talk will be about Building Public-Facing Websites with SharePoint 2010. 

SharePoint has quickly become a popular platform for companies to build their public-facing websites on.  I’ll go over the new features in SharePoint 2010 specific to web content management, and also discuss some best practices and lessons learned from our experience building internet sites with SharePoint.

The SharePoint for Legal Symposium is a two-day event with talks covering a variety of other topics such as:

• Enterprise Search Using SharePoint 2010 and FAST
• SharePoint as a Document Management System
• Content Classification in SharePoint 2010: Taxonomies, Folksonomies and More

I’m very interested in hearing from firms who have been testing SharePoint 2010 prior to RTM, particularly how they are taking advantage of the new features in SharePoint 2010, e.g. Managed Metadata.

I’ve made my presentation available in advance, check it out on SlideShare:

Adding User Profile Photos in Communicator “14”

Communicator "14" pulls profile photos from the thumbnailPhoto property (byte stream) of a user in Active Directory.  In this post, I'll show you how to set the profile photo for a user.

Note: Communicator now also allows you to also upload a photo manually in Tools >> Options >> Personal >> Display Photo.  I haven't tried this, but it appears to allow you to set your profile photo if you don't have a "corporate" (probably Active Directory) photo set.  I'm not sure if the thumbnailPhoto in Active Directory overrides this or vice versa.

Here's how you set the user's profile photo in PowerShell.

$firstName = $args[0]
$lastName = $args[1]
$profilePic = $args[2]

$objDomain = [ADSI]"" # returns a collection of properties on the domain
$dn = $objDomain.distinguishedname # returns a formatted DN

# Lookup the user in Active Directory using LDAP 
[string]$fullName = [string]$firstName + " " + [string]$lastName

# Create the user in an ou called Users - modify this as needed for your environment
$user = [ADSI]("LDAP://cn="+$fullName+",ou=users,ou=ous,dc=fabrikam,dc=com")

[byte[]]$file = Get-Content $profilePic -Encoding Byte

# clear previous image if exist
$user.Properties["thumbnailPhoto"].Clear()

# write the image to the user's thumbnailPhoto attribute by converting the byte[] to Base64String
$user.Properties["thumbnailPhoto"].Add([System.Convert]::ToBase64String($file))

# commit the changes to AD
$user.CommitChanges()

Save this PowerShell script, e.g. AddProfilePhoto.ps1, and run it as follows:

.\AddProfilePhoto.ps1 "George" "Durzi" "ProfilePics\George Durzi.png"

where "ProfilePics\George Durzi.png" is the file path to where the profile image is - note that the picture should be less than 100KB in size.

The last step is to run a couple of PowerShell cmdlets in the Communications Server Management Shell to apply the changes:

Update-CSUserDatabase cs-se.fabrikam.com
Update-CSAddressBook cs-se.fabrikam.com

where cs-se.fabrikam.com is the name of the Registrar Pool in your environment.

Next time you log out and back into Communicator, you should see the profile photo displayed next to the user's contact information.

Configuring Expert Search in Communicator “14” and SharePoint 2010

Communicator “14” provides functionality to be able to search for contacts not just by name, but by skill.  For example a customer service agent at an airline can search for other agents with Travel Advisory experience by typing the search criteria into the Communicator search box and performing a search by keyword. 

The search results will return users who have specified that skill in their profile on their SharePoint My Site.  This is actually pretty easy to configure, I’ll show you how.

Create Search and People Search Results Pages in SharePoint

Communicator “14” Expert Search works by using the SharePoint 2010 Search Service to search SharePoint for user profiles with matching keywords. 

This requires that you have an Enterprise Search site in your site collection which includes the search service and also the People Results pages.  The easiest way to do this is to create a Search Center site in your site collection.

Note: I get an error when trying to create an Enterprise Search site in a Team Site in the SharePoint 2010 RTM bits, so I created it as a site collection – that is evident in the URLs you see below.

In the screenshots below, you can see that the URL of the SharePoint search service in the Search site collection is http://sps2010/sites/search/_vti_bin/search.asmx, and the URL of the People Search Results page is http://sps2010/sites/Search/Pages/peopleresults.aspx.

Point Communications Server “14” to Search and People Search Results Pages

For Communicator “14” to be able to perform an Expert Search, you need to configure Communications Server “14” to point to the Search Service and People Search Results page URLs.

From a server with the OCS Core bits installed, fire up the Communications Server Management Shell and type Get-CsClientPolicy.

Scroll down to the bottom of the output, we’re interested in setting the values of:

• SPSearchInternalURL
• SPSearchExternalURL
• SPSearchCenterInternalURL
• SPSearchCenterExternalURL

SPSearchInternalURL and SPSearchExternalURL correspond to the internal and external URLs of the SharePoint search service in the Search site collection, while SPSearchCenterInternalURL and SPSearchCenterExternalURL correspond to the internal and external URLs of the people search results pages.

We’ll use the Communications Server Management Shell to set the values of these CS policy properties.

For simplicity, I’m only going to set the internal URLs here.

Set-CsClientPolicy –SPSearchInternalURL http://sps2010/sites/search/_vti_bin/search.asmx
    -SPSearchCenterInternalURL http://sps2010/sites/Search/Pages/peopleresults.aspx

Log out and back into Communicator. 

You can verify that these settings were applied by running the Get-CsClientPolicy cmdlet again from the Communications Server Management Shell.

However, there’s another super-secret ninja trick to verify that the settings were applied:

• Find the Communicator icon in the Windows System Tray
• Hold down the Ctrl button
• Click (left) the Communicator icon in the Windows System Tray – do not depress the Ctrl button

You should now see an extra menu item called Configuration Information, click it.

Scroll down and locate the Expert Search URL and SharePoint Search Center URL keys and verify that their values correspond to those you set using the Set-CsClientPolicy PowerShell cmdlet.

Configure a Sharepoint User Profile Import

I’m not going to provide detailed steps here except to say that you need to configure the SharePoint 2010 User Profile  Service Application to import user account details from Active Directory on a scheduled basis.

This is a critical step because there are several user profile properties – e.g. SipAddress – that are only populated by a user profile import. 

When performing an Expert Search, Communicator can only render results for users who have a SipAddress specified.

Add Skills to User Profiles

Navigate to your My Site and click on My Profile.  This page allows you to set many contact details that are searchable in SharePoint. 

We’re particularly interested in the Ask Me About property of a user’s profile.  Expert Search searches against this property to find users with matching skills.

Configure a SharePoint Search Crawl

Ensure that you have a scheduled job to crawl your Local SharePoint Sites content source. 

Depending on how you have this configured, it will also crawl the My Site site collection and add user properties such as Ask Me About to the search index.

That’s It!

SharePoint 2010 provides new social and collaboration features to help users find other users with similar skills or interests.

Expert Search extends this functionality directly into Microsoft Communicator “14”, allowing you to interact with the users directly from the search results.

Installing UCMA 3.0 and Creating a Communications Server "14" Trusted Application Pool

A lot of setup and administration tasks have gotten a lot easier in Communications Server “14”; one of them is building an application server to develop and run your UCMA 3.0 applications on.

In this post, I’ll walk you through installing the UCMA 3.0 Core SDK and creating a Trusted Application Pool on the server, thus adding it to the Communications Server “14” topology and allowing you to host and run UCMA 3.0 applications on it.

Note: These instructions will change slightly as the bits get updated for the eventual Beta release – I will update this post as soon as I get a chance to run this setup on a more recent build.

I’m doing the install on a simple Communications Server “14” topology consisting of the following Windows Server 2008 R2 Hyper-V images:

• DC – Domain Controller
• ExchangeUM – Exchange Server 2010
• CS-SE – Microsoft Communications Server 2010 Standard Edition
• TS – Development machine

I’ll walk through setting up UCMA 3.0 on the “TS” VM, which is a fully patched Windows Server 2008 R2 machine that is joined to the Fabrikam domain.   I’m also running Visual Studio 2010 on this VM because I intend to use it as a development machine.  In a future post, I’ll walk through installing just the UCMA 3.0 run time to build a true production UCMA application server.

I’m making a couple of assumptions here:

• You have an existing CS 2010 site and cluster configured(we’ll look at this in a future post)
• You’re starting with a fully patched Windows Server 2008 R2 machine
• The machine is joined to your domain

This walkthrough was done in my Fabrikam VM environment but can easily be modified for your own environment.

Installing the UCMA 3.0 SDK

Let’s start by installing the UCMA 3.0 SDK.  Run UcmaSdkWebDownload.msi to kick off the SDK installer package extract process.

The installed package is extracted to C: >> Program Files >> Microsoft UCMA 3.0 >> SDK Installer Package.  Browse there and run setup.exe.

Click Install to install the UCMA 3.0 Core SDK and Workflow SDK.

Install Communications Server Core Components

UCMA 3.0 introduces a new concept called Auto-provisioning, which is most easily explained from the developer point of view.  Remember what your app.config looked it in UCMA 2.0?  You had to store the application GRUU, the trusted contact SIP Uri, the port for your application, and the name of the certificate authority.

That’s all gone with auto-provisioning – all you need in your app.config is your ApplicationId, e.g.: urn:application:MyApplication.

How does CS 2010 do this? All of the application’s configuration data is associated with the application’s id.  UCMA also queries a replicated copy of the Central Management Database to retrieve the application’s configuration data and also the configuration data for any endpoints.

In this step, we’ll run Bootstrapper.exe to install the CS Core components, this checked for the following components and installs them if they are not already present:

• VcRedist
• Sqlexpress
• Sqlnativeclient
• Sqlbackcompat
• Ucmaredist
• OcsCore.msi

Open a command window at C: >> Program Files >> Microsoft Communications Server 2010 >> Deployment and run the following command:

Bootstrapper.exe /BootstrapReplica /MinCache /SourceDirectory:"%ProgramFiles%\Microsoft UCMA 3.0\
SDK Installer Package\Prereq\BootstrapperCache"

Create a New Trusted Application Pool

The next step is to create a new trusted application pool for the new server.  Fire up the Communications Server Management Shell from Start >> Microsoft Communications Server 2010 >> Communications Server Management Shell and enter the following PowerShell command:

New-CsTrustedApplicationPool -Identity <FQDN of Server> -Registrar <FQDN of CS Server> -Site <CS Site Name>

Verify that the new server was added to the CS topology by running the following PowerShell command:

(Get-CsTopology -AsXml).ToString() > Topology.xml

This created a file called Topology.xml in the directory that you ran the command from.  Open the file and find the Clusters section and look for a node for the new server.

The Cluster Fqdn is the name of your server, and note the name of the Site that this Cluster is a part of.

<Cluster Fqdn="appsrv.fabrikam.com" RequiresReplication="true" RequiresSetup="true">
  <ClusterId SiteId="UcMarketing2" Number="5" />
  <Machine OrdinalInCluster="1" Fqdn="appsrv.fabrikam.com">
    <NetInterface InterfaceSide="Primary" InterfaceNumber="1" IPAddress="0.0.0.0" />
  </Machine>
</Cluster>

Configure CS Management Store Replication

At this point, we have the CS Core components installed and the server configured as a trusted application pool.  We now need to set up replication so that the Central Management Store replicates down to the new server.

From the Communications Server Management Shell, run the following PowerShell command to enable the Replica service on the new server:

Enable-CSReplica

The Replica service is enabled, but hasn't done anything yet. This can be verified by running the following PowerShell command to check the replication status for the various servers in the topology:

Get-CSManagementStoreReplicationStatus

You can see in the screenshot below that the UpToDate property of the new server is still False

Run the following PowerShell command to force the replication to run:

Invoke-CSManagementStoreReplicationStatus

Run Get-CSManagementStoreReplicationStatus again to verify that the new service is now up to date

Request and Set a New Certificate

The last step in the process is to request a new certificate from the certificate authority on the domain and assign it to the new server.

From the Communications Server Management Shell, run the following PowerShell command to request a new certificate:

Request-CSCertificate -Action new -Type default -CA <Domain Controller FQDN>\<Certificate Authority>

Setting the -Verbose switch on the cmdlet creates an Xml file with its output. Open the Xml file and copy the thumbprint of the generated certificate.

<?xml version="1.0" encoding="utf-8"?>
<Action Name="Request-CsCertificate" Time="20100512T212258">
  <Action Name="Request-CsCertificate" Time="20100512T212258">
    <Info Title="Connection" Time="20100512T212258">Data Source=(local)\rtclocal;Initial Catalog=xds;Integrated Security=True</Info>
    <Action Time="20100512T212258">
      <Info Title="Certificate use" Time="20100512T212258">urn:certref:default</Info>
      <Info Title="Subject distinguished name" Time="20100512T212258">CN="appsrv2.fabrikam.com"</Info>
      <Info Time="20100512T212259">The certificate request is submitted to the Certification Authority dc.fabrikam.com\FabrikamCA.</Info>
      <Info Time="20100512T212259">The certificate was issued.</Info>
      <Info Time="20100512T212259">The certificate was imported with thumbprint AFC3C46E459C1A39AD06247676F3555826DBF705.</Info>
      <Complete Time="20100512T212259" />
    </Action>
    <Info Title="command status" Time="20100512T212259">Command execution processing completed</Info>
    <Action Name="DeploymentXdsCmdlet.SaveCachedItems" Time="20100512T212259">
      <Info Time="20100512T212259">0 updates</Info>
      <Complete Time="20100512T212259" />
    </Action>
    <Info Title="command status" Time="20100512T212259">Command has completed</Info>
  </Action>
</Action>

Run the following PowerShell command to set the certificate:

Set-CsCertificate -Type Default -Thumbprint <Thumbprint>

Wrapping Up

You now have a new UCMA 3.0 application server in your Communications Server 2010 server topology.  You can provision trusted applications and trusted application endpoints on the new server using the Communications Server 2010 Management Shell.  We’ll take a look at how to do that in another post.

Hyper-V for Developers – Part 1 – Internal Networks

Over the last year, we’ve been working with Microsoft to build training and demo content for the next version of Office Communications Server – code-named Microsoft Communications Server “14”. 

This involved building multi-server demo environments in Hyper-V, getting them running on demo servers which we took to TechEd, PDC, and other training events, and sometimes connecting the demo servers to the show networks at those events.  ITPro stuff that should scare the hell out of a developer!

It can get ugly when I occasionally have to venture into ITPro land.  Let’s leave it at that.

Having gone through this process about 10 to 15 times in the last year, I finally have it down.  This blog series is my attempt to put all that knowledge in one place – if anything, so I can find it somewhere when I need it again. 

I’ll start with the most simple scenario and then build on top of it in future blog posts.

If you’re an ITPro, please resist the urge to laugh at how trivial this is.

Internal Hyper-V Networks

Let’s start simple.  An internal network is one that intended only for the virtual machines that are going to be on that network – it enables them to communicate with each other.

Create an Internal Network

On your host machine, fire up the Hyper-V Manager and click the Virtual Network Manager in the Actions panel.

Select Internal and leave all the other default values.

Give the virtual network a name, and leave all the other default values.

After the virtual network is created, open the Network and Sharing Center and click Change Adapter Settings to see the list of network connections.

The only thing I recommend that you do is to give this connection a friendly label, e.g. Hyper-V Internal.  When you have multiple networks and virtual networks on the host machines, this helps group the networks so you can easily differentiate them from each other. 

Otherwise, don’t touch it, only bad things can happen.

Connect the Virtual Machines to the Internal Network

I’m assuming that you have more than 1 virtual machine already configured in Hyper-V, for example a Domain Controller, and Exchange Server, and a SharePoint Server.

What you need to do is basically “plug in” the network to the virtual machine.  In order to do this, the machine needs to have a virtual network adapter.  If the VM doesn’t have a network adapter, open the VM’s Settings and click Add Hardware in the left pane.  Choose the virtual network to which to bind the adapter to.

If you already have a virtual network adapter on the VM, simply connect it to the virtual network.

Assign IP Addresses to the Virtual Machines on the Internal Network

Open the Network and Sharing Center on your VM, there should only be 1 network at this time. 

Open the Properties of the connection, select Internet Protocol Version 4 (TCP/IPv4) and hit Properties.

In this environment, I’m assigning IP addresses as 192.168.0.xxx.  This particular VM has an IP address of 192.168.0.40 with a subnet mask of 255.255.255.0, and a DNS Server of 192.168.0.18.  DNS is running on the Domain Controller VM which has an IP address of 192.168.0.18.

Repeat this process on every VM in your environment, obviously assigning a unique IP address to each.  In an environment with a domain controller, you should now be able to ping the machines from each other.

What Next?

After completing this process, here’s what you still cannot do:

• Access the internet from any of the VMs
• Remote desktop to a VM from the host
• Remote desktop to a VM over the network

In the next post, we’ll take a look configuring an External network adapter on the virtual machines.  We’ll then build on top of that so that you can RDP into the VMs from the host machine and over the network.

Brian Harry – Microsoft Technical Fellow – Speaking at Clarity Consulting

Join us on Thursday, March 4th from 3 pm - 6 pm for Visual Studio 2010 and Team Foundation Server 2010, presented by Microsoft Technical Fellow, Brian Harry.

Attend this event to learn more about Microsoft’s vision for developer team solutions.  Brian will cover the key advances for developers in Visual Studio 2010 and Team Foundation Server 2010.

Attendees will have an opportunity to ask Brian Harry questions around Microsoft’s vision for developer tools and platforms.  This is a great opportunity to connect directly with a very senior and technical developer community leader within Microsoft.

Ryan Powers from Clarity Consulting will also discuss how Clarity’s project teams use Team Foundation Server as part of the software development lifecycle.

Registration and more details.

Upcoming TechEd and MSDN Events in the Midwest Region

I volunteer for INETA as the user group mentor for the Midwest Region (IL, IN, and WI).  One of the things I'm planning to do differently this year is to better help spread the word on my blog about events happening in the region, whether they're Microsoft or user group organized and hosted. 

If your user group is hosting an event such as a code camp, send me a short write up and some photos and I'll make sure it gets published in the INETA newsletter. 

Back to business ... Here's some info about some Azure and Visual Studio 2010 events that are coming up:

TechNet Events Present: 
Windows Azure, Hyper-V and 
Windows 7 Deployment - 8:30am – 12:00pm
Get the inside track on new tips, tools and technologies for IT pros. Join your TechNet Events team for a look at Windows Azure™ and learn the basics of this new online service computing platform. Next, we’ll explore how to build a great virtual environment with Windows Server® 2008 R2 and Hyper-V™ version 2.0. We’ll wrap this free, half-day of live learning with a tour of easy deployment strategies for Windows® 7.

TOPICS INCLUDE:
    •    The Next Wave: Windows Azure
    •    Hyper-V: Tools to Build the Ultimate Virtual Test Network
    •    Automating Your Windows 7 Deployment with MDT 2010

MSDN Events Presents:
Cloud Computing and Azure - 1:00pm – 4:30pm
Join your local MSDN Events as we take a deep dive into cloud computing and the Windows Azure Platform.

TOPICS INCLUDE:
    •    What is cloud computing?
    •    Running web and web service applications in the cloud
    •    Using the Windows Azure and local developer cloud fabric
    •    Getting started – tools, SDKs and accounts
    •    Writing applications for Windows Azure

MSDN Events Presents:
Visual Studio 2010, .NET Framework 4.0 and SharePoint Development  - 1:00pm – 4:30pm
Join your local MSDN Events team for a lively tour our latest version of Visual Studio and the .NET Framework. We’ll start with an overview the IDE and explore how you can use the latest features to develop great applications in a flash. Next, we’ll look at the newest changes to the .NET Framework and how you can leverage them today.  Finally, we end the day by looking at the new tools in Visual Studio that will supercharge your SharePoint development.

TOPICS INCLUDE:
    •    What’s new in Visual Studio 2010
    •    What’s new in .NET Framework 4.0
    •    SharePoint Development using Visual Studio 2010  

Speaking at the Central Illinois SharePoint User Group on 1/21/2010

I'll be speaking at the Central Illinois SharePoint User Group in Springfield, IL at 9:00AM on Thursday, January 21st 2010.

The topic will be Building Communications Enabled Applications with OCS and Exchange:

Office Communications Server 2007 R2 and Exchange 2010 enable the development of communication enabled applications more quickly and easily than ever before.  In this session you will learn about the core features included in the new platform SDKs and see a demo application built around a retail enterprise that utilizes each of the scenarios the platform enables.  You will also learn about opportunities for integrating OCS and Exchange into your SharePoint applications.

More details on location and registration areavailable at https://www.clicktoattend.com/invitation.aspx?code=144377

Querying Free/Busy Data with Exchange Web Services Managed API 1.0 SDK

This week at TechEd Europe 2009, Microsoft officially launched Exchange Server 2010 and the Exchange Web Services Managed API 1.0 SDK.  This API provides a managed wrapper around Exchange Web Services, eliminating the need to work directly with SOAP or with proxy classes generated when adding a web reference to your environment’s Exchange Web Services url.

The release version of EWSMA includes improved support for Exchange Web Services features such as:

• Free/Busy
• Autodiscover
• Timezones
• Impersonation
• Search

In this post, I’ll demonstrate using EWSMA to query the Exchange free/busy service to find a meeting time that is most suitable for a set of participants.

Referencing the Exchange Web Services Managed API 1.0 SDK

The Exchange Web Services Managed API 1.0 SDK is installed by default to C:\Program Files\Microsoft\Exchange\Web Services\1.0, you need to reference Microsoft.Exchange.WebServices.dll.

Improved Autodiscover

The Exchange Autodiscover service provides clients with a way to retrieve the url to use to connect to Exchange Web Services.  For a majority of deployments, there is one url for Exchange Web Services.  However, consider large, global Exchange deployments…the Autodiscover server can for example “auto discover” the most geographically appropriate web service endpoint for a client such as Outlook to connect to.

If you’ve worked with Exchange Web Services in the past, you know that  autodiscover’ing the url for Exchange Web Services involved a lot of code, this is what it looks like now:

   1:  ExchangeService service = new ExchangeService();

   2:  service.AutodiscoverUrl(String.Format(

   3:      "{0}@{1}.com", 

   4:      Environment.UserName, 

   5:      Environment.UserDomainName));

Set up Availability Options

Before querying the free/busy service, you need to specify some options that the service will use to retrieve a list of meeting time suggestions – this is represented by a class called AvailabilityOptions.

   1:  AvailabilityOptions availabilityOptions = new AvailabilityOptions();

   2:  availabilityOptions.MeetingDuration = 60;

   3:  availabilityOptions.MaximumNonWorkHoursSuggestionsPerDay = 4;

   4:  availabilityOptions.MinimumSuggestionQuality = SuggestionQuality.Good;

   5:  availabilityOptions.RequestedFreeBusyView = FreeBusyViewType.FreeBusy;

Check out the SDK documentation to explore other options you can set when creating an instance of AvailabilityOptions.  In this case, we’re looking for a 60 minute time slot, allowing some flexibility to schedule the meeting outside of working hours, setting the minimum suggestion quality to Good, and only requesting the FreeBusy view after querying the service.

Get Availability Results

First, create a List<AttendeeInfo> to store the list of contacts to include in the request to the free/busy service.  In this case, you can see how to add different attendee types to the list, e.g. the organizer, a required attendee, and a room resource.

   1:  List<AttendeeInfo> attendees = new List<AttendeeInfo>();

   2:   

   3:  attendees.Add(

   4:      new AttendeeInfo()

   5:      {

   6:          SmtpAddress = "rl@fabrikam.com",

   7:          AttendeeType = MeetingAttendeeType.Organizer

   8:      });

   9:   

  10:  attendees.Add(

  11:      new AttendeeInfo()

  12:      {

  13:          SmtpAddress = "sc@fabrikam.com",

  14:          AttendeeType = MeetingAttendeeType.Required

  15:      });

  16:   

  17:  attendees.Add(

  18:      new AttendeeInfo()

  19:      {

  20:          SmtpAddress = "boardroom@fabrikam.com",

  21:          AttendeeType = MeetingAttendeeType.Room

  22:      });

With the attendee list set up, create an instance of GetUserAvailabilityResults to store the meeting suggestions returned by the free/busy service. 

   1:  GetUserAvailabilityResults availabilityResults =

   2:       service.GetUserAvailability(

   3:          attendees,

   4:          new TimeWindow(DateTime.Now, DateTime.Now.AddDays(1)),

   5:          AvailabilityData.FreeBusyAndSuggestions,

   6:          availabilityOptions);

I want to point out a gotcha here.  When  AvailabilityOptions.RequestedFreeBusyView is set to FreeBusyViewType.FreeBusy, the time window for the call to GetUserAvailability needs to be at least 24 hours.  I’m not sure of the reasoning for this but if its set to a shorter time window, the call sometimes throws an exception about an invalid time window.

Explore Suggested Meeting Times

You can now explore the suggested meeting times  returned by the free/busy service. 

The results are in the form of a List<Suggestion> where Suggestion describes the date and quality of the suggestion, and a collection of individual TimeSuggestions.  A TimeSuggestion describes any conflicts at the suggested time, whether or not the suggestion occurs during working hours, the suggested time and quality of the meeting.

Unless you set a time window of more than 1 day, you’ll probably just want to grab availabilityResults.Suggestions[0] and pick the most suitable TimeSuggestion in availabilityResults.Suggestions[0].TimeSuggestions according to whatever criteria are suitable for your application.

Book the Meeting

It’s just as easy to book the appointment directly from your application.  Assuming that suggestion is an instance of a TimeSuggestion that you want to book:

   1:  // Create the appointment and set its properties

   2:  Appointment appointment = new Appointment(service);

   3:  // Set the appointment title

   4:  appointment.Subject = "Team Meeting";

   5:  // Set the appointment start and end time

   6:  appointment.Start = suggestion.MeetingTime;

   7:  appointment.End = suggestion.MeetingTime.AddMinutes(availabilityOptions.MeetingDuration);   

   8:  // Add the conference room as a meeting resource

   9:  appointment.Resources.Add("Board Room", boardroom@fabrikam.com);

  10:  // Set the meeting location to display name of the conference room

  11:  appointment.Location = "Board Room";

  12:  // Save the appointment

  13:  appointment.Save();

Summary

You’ve seen how easy it is to include Outlook-style functionality into your custom applications.  This didn’t use to be as simple, unless you loved working with WebDAV or CDO, or writing some sweet asynchronous WebRequest code…

Slides and Source Code from CSPUG Talk

I spoke today at the Chicago SharePoint User Group about Building Public Facing SharePoint sites. Thanks to Michael and Asif for inviting me, I'm looking forward to doing this again in the future.

You can download the presentation slides and the source code for the sample site I demo'd here.

Thanks to everybody who attended and asked great questions.

Building a Twitter Recommendation Engine for Gadfly

I recently got a chance to help out on Gadfly, a twitter client that @SteveHolstad, @leeiroth, and @eklimcz from Clarity are working on.  If you haven’t had a chance to check it out, I definitely recommend it, I’ve been waiting for a long while for a great web-based twitter client to show up!

Gadfly allows you to rate individual tweets from 1 to 5 stars.  Using this ratings data, we started looking into building a recommendation engine that would suggest other tweeple that you might find interesting.

How a Twitter Reputation Algorithm Needs to Work

As I was researching how I would build such a recommendation engine, @goodoldschu forwarded me an article titled How a Twitter Reputation Algorithm Needs to Work.  The author does a great job of describing what he thinks should and should not matter in a twitter reputation algorithm:

Should Matter:

• Co-follower rate:  when two people follow the same people, you can numerically represent how similar their tastes are.  This is a crucial concept for most recommendation algorithms.
• What your followers follow, and what they tweet about:  if you index the text of individual tweets, you can do some very interesting stuff such as clustering similar tweeple into groups, e.g. people who tweet about a certain technology.

Shouldn’t Matter:

• Follower counts
• When you started tweeting
• Frequency of tweets

Pretty straightforward… Your follower count, when you started tweeting, and how often you tweet don’t represent the quality of your tweets.  These metrics have no place in a twitter recommendation algorithm.

Practical Challenges

There are some practical challenges, however, when trying to incorporate the “should matter” metrics into a twitter recommendation algorithm; I’ll explain those as they relate to Gadfly in particular.

Gadfly is a web based application build in Silverlight; it uses isolated storage as a virtual file system to store application data on your machine.  We’re somewhat limited in how much data we can store in isolated storage, e.g. 1MB for an in-browser Silverlight application.

So why not store followers and tweet content in a central database?  If storage space weren’t an issue, this would certainly be feasible.  However, we didn’t want to get into the business of constantly querying follower and following lists and storing them in our database, let alone storing the content (raw or indexed) of tweets.  There’s also the matter of how quickly we’d eat into the twitter API rate limit with these operations.

Enter Ratings

The best way to explain this is to use the good old Netflix movie ratings example. 

Netflix allows users to rate movies that they have watched.  When new users join Netflix and begin rating movies, you can compare their preferences to existing users that have rated those same movies.  Based on this data, you can identify a set of existing users who are most “similar” to the new users and use these users’ ratings for other movies to fill in the gaps in the new user’s profile.  For example, a recommendation engine can predict that you would give The Bourne Supremacy a rating of 4 stars and place it on your recommended list.

This describes the KNN (K-Nearest Neighbor) algorithm in its simplest form.  This is the algorithm that Netflix uses (albeit with a ton of tweaks and optimizations) to recommend movies to users.

How does this apply to a twitter recommendation engine?  Instead of rating movies, you rate tweets.  Since you’re rating the same twitter user multiple times, we average out this rating to come up with a single rating.  This gives us a single figure that represents a Gadfly user rating a twitter user.  Over time, we’ll probably calculate the average based on more recent tweets, otherwise new ratings will have little to no effect on the calculation.

When other Gadfly users rate the same twitter user, we can compute how similar these users’ tastes are.  We can then predict what rating users will give to twitter users that they haven’t rated.  If the predicted rating is above an acceptable threshold that we set, we can recommend that twitter user as a Gadfly Pick.

Computing Similarity

The process that powers a KNN-based recommendation engine is always run in batch mode due to the computational intensity that is involved.

The first step is to calculate the similarity (or correlation) between every set of two users using a coefficient such as Pearson.   This is a number between 0 and 1, with 0 meaning that the users are not similar at all, and 1 meaning that they are as similar as they can be.

The key here though is that you can’t measure correlation between two users unless they have some ratings in common – we define this parameter to the algorithm as the Minimum Number of Ratings in Common.  Realizing that at first our data will be very sparse, we set this to 1 - we can tweak this as more Gadfly users enter ratings.  We can increase this to make the engine more selective as we get more ratings data.

To the left is example of some of the ratings data, this is all sample data so the similarity numbers are artificially high.

Generating Picks

Now that the system has computed all possible combinations of similarity among users with a minimum number of ratings in common, the next step is to generate the Gadfly Picks for each user.

When computing the picks for a particular user, e.g. UserId 67 in the above sample data, the K from KNN refers to the number of most similar users that the algorithm will use to predict ratings for twitter users that User 67 hasn’t rated.  If the predicted rating is greather than a specified threshold, we recommend that twitter user to User 67 and they appear in their Gadfly Picks timeline. 

So if we’re using a value of K=3 and the above sample data to put together the Gadfly Picks for User 67, the top 3 similar users are 68, 71, and 74 with Pearson values of 1.0, 1.0, and 0.992583334.  Let’s use those numbers to predict how User 67 would rate a particular twitter user, assuming the following information on how User 67’s top 3 most similar users have rated that twitter user:

Predicted Rating = (2*1 + 3*1 + 2*0.992583334) / (1 + 1 + 0.992583334) =6.985166668 / 2.992583334 = ~2.33

The predicted rating should be pretty self explanatory.  If the three Gadfly users that User 67 is most similar to gave this twitter user ratings of 2, 3, and 2, a predicted rating of 2.33 is right in line.

We currently have the value of Predicted Rating Threshold set to 3.5 – this user wouldn’t make the cut.

Right now we’re generating each user’s Gadfly Picks once a day, we can make this more frequent as usage increases.  It will be very interesting to see how much we have to tweak the backend SQL when it has to deal with much larger datasets.

Limitations and Opportunities for Improvement

I’m excited to get this rolled out in the next couple of days, but I can already identify several areas which I want to work on improving after we iron out the initial kinks.

Twitter Users in a Search Timeline have a Twitter User Id = 0

As I understand it, the twitter search API hasn’t yet been integrated into twitter’s core API.  The implication of this for us is that the Twitter User Id of a twitter user in a search timeline is always 0.  We’re thus not able to relate that user to an actual twitter user and can’t use that piece of ratings data.  Hopefully this will be a non-issue as the APIs continue to come together.

The Engine Recommends Me to Me

I thought it was funny when I saw myself as a Gadfly Pick, we need to tweak the algorithm so that I’m not recommended to myself!

Ability to Rate the Recommendations

The only way to truly measure the effectiveness of a recommendation algorithm is to compare the predicted rating to the actual rating provided by the user.  The functionality is currently there to rate the recommended picks.  We haven’t proved this out yet, but we suspect that the algorithm will automagically take care of things.

We’ll be rolling this out in the next day or two, I’m really interested in taking a look at the quality of the ratings as we get more realistic data.

Speaking at Chicago SharePoint User Group on September 10th 2009

I’ll be speaking at the Chicago SharePoint User Group’s monthly meeting on September 10th 2009. The meeting is scheduled from 1pm to 4pm at the Microsoft office in Downers Grove, IL.

Interested in seeing what it takes to build out a public facing site using SharePoint?  My talk will cover areas such as:

• Publishing workflow
• Server and network topology of a public facing SharePoint site
• Organizing your SharePoint solutions
• Publishing site definitions
• Branding
• Packaging the site’s assets
• Configuring content deployment
• Setting up anonymous access

I’ll also go over some special considerations for public facing SharePoint sites such as page payload size, accessibility, and search engine optimization.

More details and registration information should be available soon on the CSPUG site, hope to see you there!

Package Environment Specific Settings into a Silverlight XAP using Team Build

This one’s for you @SteveHolstad!

I was helping one of our teams ease the deployment of a Silverlight project into various server environments.  The project contained a ServiceReferences.ClientConfig file to reference some WCF services in the solution.  When deploying the project to a Development, Staging, or Production environment, the team had to:

• Un-XAP the Silverlight XAP
• Copy in an environment specific ServiceReferences.ClientConfig file (or edit)
• Re-XAP the Silverlight XAP

Here’s how to do this pretty easily with TFS Team Build.

I like to store environment specific settings in the source control branch that corresponds to that environment.  In the Dev branch (which translates to our Development Integration environment), I created a version of ServiceReferences.ClientConfig with the appropriate environment specific service bindings and placed it at. $\MyTeamProject\Dev\Env\Config\Services\ServiceReferences.ClientConfig.

In the Team Project’s build definition, you can now tap into the BeforeCompile target and do the following:

 <Target Name="BeforeCompile">

    <CreateItem Include="$(SolutionRoot)\Env\Config\Services\ServiceReferences.ClientConfig">

      <Output ItemName="ServiceReferences" TaskParameter="Include"/>

    </CreateItem>

    <Copy SourceFiles="@(ServiceReferences)"

      DestinationFolder="$(SolutionRoot)\Source\MySilverlightProject"

      OverwriteReadOnlyFiles="True"/>

  </Target>

The first thing that a Team Build does is get the latest code from source control, and place it in a working directory on the build agent.  The new BeforeCompile target replaces the ServiceReferences.ClientConfig in the working directory with the environment specific one. 

And it does it BEFORE COMPILE!  When TFS then compiles the MySilverlightProject project, the resulting XAP will contain the environment specific version of ServiceReferences.ClientConfig.

This of course assumes that you’re structured your source control in such a way where you can point different Team Build definitions at different branches or directories in your source tree.