These days I get to deal with variety of developers working on typical business solutions. I've had a lot of interesting discussions lately and I also heard some weird things about various technical things. So I thought about starting this "Myths" column to share my thoughts with rest of the community as I believe it will certainly be mutually beneficial.
Today I'm going to start off with is this Kerberos thing as I found it very interesting among everything else. Recently I ran into a very common application scenario. There are three parts in that.
- ASP.NET web application
- WCF Web Service
- SQL Server
ASP.NET application consumes WCF web service which will in turn interacts with the SQL database. Each of these parts runs in their own server boxes and they are nodes of a domain controller. IIS hosting the web app is configured to perform Windows authentication and users sign-in to the ASP.NET web application using their domain accounts.
In this application WCF web service implements the reusable business logic. However, it still relies on the web application to perform the authentication and authorization to access various business functionality :(. This is usually not what we would expect to see. The service itself should be able to authenticate and authorize the users and thus the security becomes feature of the service.
So that's essentially the story. Interesting part of that is "why" it's being implemented that way. Obviously the ASP.NET application can be configured to impersonate the client who logs in with her domain account. However, for ASP.NET application to access remote resources on behalf of the impersonated user (e.g. access the web service running in a different node), the hosting server must support delegation. And in order to support delegation Windows authentication process should have used Kerberos instead of NTLM. This is where the myth comes to the table. I managed to find a few blogs saying that you have to do number of things to enable Kerberos in Windows 2003.
But in reality Kerberos is the default authentication package used in Windows for domain accounts. But then why doesn't it work in the scenario described above? If you run all applications as network service (i.e. computer account) and access the resources using the standard host names everything should work as soon as you enable delegation (I will show you how to do that in a moment). However, this is not the case in most circumstances. You usually use custom host names to access your web application, web services and various other resources. As soon as you do this, you have configure something called Service Principal Names in order to make Windows use Kerberos package for authentication.
Instead of repeating the very few steps to enable delegation and configuring service principal names I would like point this wonderful Q & A paper written by Keith Brown few years ago. In that Keith talks about delegating the credentials to SQL server. But you do the same stuff in the aforementioned WCF scenario as well.
However, to summarize a few things that you may want to pay attention:
- If you are using host headers for ASP.NET application or Web service make sure you register SPNs for them. Also make sure that they are HOST entries (not CNAME) in your DNS records.
- Make sure you enable impersonation in the ASP.NET application (http://msdn.microsoft.com/en-us/library/aa292118(VS.71).aspx).
- When you are testing while changing these settings, always logout and log back into your client PC to flush the Kerberos ticket cache.
Last but not least I must thank my good friend Dominick for giving me a big hand to get my demo application up and running. His blog and Keith's blogs are must reads for all of those dealing with Security tidbits :).
Recently I looked at an application that leverages a lot of Workflow Runtime and Windows Communication Foundation bits shipped with .NET 3.0. It essentially had a state machine workflow which was hosted by a WCF service. Each operation invoked in the service fired an event to a local service which in turn resumed the execution of the state machine workflow. Also the state machine was considerably long running and therefore SQL server persistence service was used.
Just like any other typical WCF service, it was designed to service multiple users concurrently. Consequently two users could potentially send the same request at the same time which in turn fires the same event twice to the underlying workflow. But workflow runtime being very smart in this case does not allow two instances of the same workflow executing in parallel. Therefore the request that gets hold of the workflow goes through and the second one waits until the first one finishes its unit of work. But what happens if the first one transits the state of the workflow to another state (which is what was happening in the application I'm talking about BTW). When the second request tries to fire the event, workflow runtime detects that the event is not valid according new state and throws an EventDeliveryFailedException exception.
From the service's perspective it's important to let the clients know what went wrong. Specially in this case giving a clue would be helpful to the client to retry if that's required. At this point you might be thinking that it's not a big deal as you could catch EventDeliveryFailedException exception and translate it to a proper fault pretty easily. But in reality, EventDeliveryFailedException exception is too generic to detect the exact cause of the problem. For example, you might get an EventDeliveryFailedException exception if you are:
1. Trying to fire an event to the workflow which is invalid according to current workflow state.
2. Trying to fire an event that does not exist in the workflow (probably due to multiple versions running side by side)
3. Trying to fire an event whose event arguments are not marked as serializable (this would probably happen only when you are debugging).
4. Trying to fire an event while the workflow is owned by another thread (user) and your ownership timeout expires.
Therefore catching EventDeliveryFailedException exception itself does not seem to help. The answer lies within the InnerException property of EventDeliveryFailedException exception. Workflow runtime creates a distinguishable exception for each of those scenarios (at least for scenarios I was testing) and assigns it to InnerException property. For example, if the workflow could not deliver the event because of the current state of the workflow, the inner exception is set to System.Workflow.Runtime.QueueException exception. If it was because of the ownership timeout on the other hand, the inner exception is set to System.Workflow.Runtime.WorkflowOwnershipException exception. However, unfortunately, these exception types are marked as internal and therefore we cannot catch those exceptions directly. Consequently, if we want to send a more sensible fault to the client we would have to check the type names and create the appropriate fault to be returned from the service.
After playing around 3.0 bits I checked out how this scenario is handled by Workflow Services available in 3.5 bits. The 3.5 runtime automatically generates a generic fault saying "Operation is currently not available on the service". But I think it's always good to send custom fault messages which clients can specifically deal with under heavy concurrent environments (specially when retries are possible).
Wrapping up this post, I would love to see those inner exception types as public types so that the workflow developers can be consistent with the exception handling code. Also it's preferable to have a knob in 3.5 Workflow Services to hook up custom faults (a fault that appears in the contract) to notify concurrency issues.
Yeah, I know now there are some non-technical readers subscribed to my blog 
. So I just thought I would share something useful for them as well. Although heading down under was just great, leaving our parents, friends and relatives was not that easy for all of us. Therefore since very first day, I'm trying my best to keep up the close ties between us as much as possible. Here are some of the tools/services I find really useful.
Skype: Allows you to make voice calls to PCs (free) and phones. So far I've found that calling an SLT phone is a lot better than calling any other operator.
WOW SMS: Provides a free service to SMS some of Sri Lankan mobile carriers. Although there is a daily limit (15 messages per day) this is a fantastic way to exchange short messages. When you register, the service also provides an inbox number for you and your contacts can reply to that number. You just have to logon to the service to read incoming messages.
Kapruka: Great way to purchase and deliver gift items for them (Well... if you want to send something for us, you are welcome to order it from http://www.ebay.com.au/ for now 
. I will publish more links later).
BTW: I'm still trying to find a Sinhala webmail solution (AFAIK, gmail, yahoo and hotmail do not support this). If you know any, please drop me a note.
I invested a lot of time, money and gave a very careful thought when I wanted to move to Australia. After all it seemed to be the best option in mid and long term basis. The decision certainly affected my personal, professional and career life and I'm certainly enjoying the consequences now.
Over the past few years I was really missing a good broadband connection. Although I was fortunate to get access to MSDN library, Channel 9, .NET rocks, Codeplex and various other technical resources, being active in those spaces was a real bummer with my poor connection. But today I downloaded Windows Server 2008 RC1 (yeah with HyperV ) just in about 90 minutes (every time I tried something similar earlier it took nearly 40 hours or more). I don't think that I would wait for my MSDN DVDs to arrive any more.
So that's just another thing to be a happy man this Christmas.
A reader of my IOCP performance post recently sent me some interesting questions. I actually found them really, really interesting and thought I'd share my answers here instead of just replying to his email.
Q: I understand that at any given point there will be N number of threads associated with an IO Completion Port. Where N is number of Proccessors. If I call GetAvailbaleThreads it returns 25 for Worker Threads and 1000 IO threads. My question is why do we have such a high value for IO Threads as compared to worker threads, when Completion Port can only perform tasks in a sequential order in a queue.
A: You should not get confused with the numbers returned by GetAvailableThreads and the actual number of active IO worker threads. Numbers 25 and 1000 are merely tweaks made in the runtime. These numbers just represent the max number of non IO and IO worker threads that thread pool can spin up concurrently. The theory behind these numbers goes like this. Normal thread pool worker threads are not capable of preserving one thread per core per execution quantum invariant. Therefore if you queue 10 presumably long running items to the thread pool, all ten items will spin a new thread will be active in the thread pool. This also means if you queue CPU intensive work items, you will end-up with poor performance due to high amount of context switching etc. That's why runtime try to mitigate this by setting max number of threads in the thread pool to 25 by default. On the other hand, if you have 25 long running IO operations, you are out of luck unless you change the defaults. Your CPU will still be idle while bunch of threads in the thread pool are waiting for IO.
IO worker threads on the other hand make sure that there are only N number of threads active at any given time (thus preserve the one thread per core per execution quantum invariant). Where N is the number of processors/cores (It might be slightly above N in most circumstances as I explained in the original post, but that's absolutely negligible). So the thread pool can actually spin up as much as threads (1000 by default) if necessary to wait on the IOCP for the next work item. So let's say you queue 25 work items to IO worker threads to wait on some long running IO and then you queue another CPU intensive work. You still get your CPU intensive work done because all 25 IO worker threads are waiting for IO thus making the CPU idle.
Q: My understanding is that when a request is sent to Completion Port , it will finish that task first then move on to the next task. How would this scale if I make two request to download a 700mb files and third request to download 7mb file. Will the third request wait unless request 1 and 2 is finished? I hope not but my understanding is it will.
A: I think my answer to the first question explains this as well. Key thing here is, if you do some blocking work in an IO worker thread, Windows scheduler notifies the IOCP that one active thread went to inactive state thereby making the next thread waiting on the IOCP to go and pick up the next available work item. So if we put this to the sample context explained in the question, I would say, you can make IO worker threads download all 700MB, 700MB and 7MB files and whichever finishes first will probably finish its work first.
Q: How do we use Overlapped Object. I see we can specify offsets, and event handler but where do we specify the IO Handle based on which the operation should be made?
A: All this time I was talking about using IOCP to efficiently use CPU as opposed to how you would do this with regular thread pool work threads. In other words it can be thought as an efficient scheduling method. However, if you want to do asynchronous IO, you can just use the async operations in the .NET IO API (i.e. BeginXxx and EndXxx methods). Almost all of them use IOCP behind the covers. However, if you are still fiddling your keyboard to get your C program up and running, you can use CreateIoCompletionPort API and pass your IO handle to its first parameter . Then you can use IO API (ReadFileEx for example) to pass the necessary async IO information using OVERLAPPED structure.
HANDLE CreateIoCompletionPort(
HANDLE FileHandle,
HANDLE ExistingCompletionPort,
ULONG_PTR CompletionKey,
DWORD NumberOfConcurrentThreads
);
Q: Is IOCP is just maintaining a queue and after it deques it calls the return function with offsets to perform the operation?
A: The underlying Windows API actually does not use a function pointer per se. It's implemented in .NET framework thread pool (AFAIK). The original API uses a key (ULONG_PTR) based mechanism to correlate the work items when then return from IOCP.
Unlike the server application developers who used Winsock API directly in their code; now most of us rely on different abstractions like WCF to deal with Winsock API (and yes, this is a lot better). Each connection opened between the client and service consumes their resources (each tcp connection requires space for its kernel level structures for example). Therefore it's nice to have a good understanding of when they come and go if you want to build scalable solutions. Consequently I thought I would share some of my experience in connection management with you and this is my first attempt to discuss some subtleties in http connection management in services and clients.
When you use http transport WCF uses System.Net API in client and http.sys API in the server internally for the underlying communication. i.e. WCF does not directly deal with sockets like it does in tcp transport.
Let's first take a look at what happens in the client side. When you invoke a service method in the proxy class instance, underlying http transport channel uses the API in System.Net.HttpWebRequest class to send the http request to the server. It also creates a System.Net.ServicePoint to manage the connections to the specific server resource. You can think of this ServicePoint instance as a connection pool for http connections. Therefore the actual lifetime of a connection depends on the corresponding ServicePoint settings. For example, the first impression you get on the following code snippet is that the underlying connection is closed and all resources are claimed after the last line.
string uri = "http://localhost:8000/service";
BasicHttpBinding clientBinding = new BasicHttpBinding();
ChannelFactory<IFoo> cf = new ChannelFactory<IFoo>(clientBinding, uri);
IFoo f = cf.CreateChannel();
f.DoSomething();
((IClientChannel)f).Close();
cf.Close();
But if you take a closer look using a tool like tcpview.exe you will notice that the connection will remain open for little bit longer even after closing both channel and channel factory.
Reason for this is the connection pooling done by the corresponding ServicePoint as I mentioned before. By default ServicePoint sets the MaxIdleTimeout of a connection to 100000 milliseconds and the connections will be closed after 10000 milliseconds without any activity. So in some cases you may want to tweak this value to reach the optimum results. For example, if you have a client that connects to the service once in a blue moon and makes only one or two calls to the service, you might want close the connection and release the resources as soon as you are done. You can do this by setting the desired timeout value to ServicePointManager.MaxServicePointIdleTime property. However it's important to be aware of two things before hand. 1. You have to set this value prior to creating any channels or channel factories in your code. 2. this value will be used as the idle timeout for all ServicePoints created afterwards (even for different endpoints). One way to eliminate these two would be getting your hands on the specific ServicePoint as shown below. Then again, you must ensure that you call FindServicePoint method after calling the service method as runtime creates this ServicePoint during the first call to the service.
string uri = "http://localhost:8000/service";
BasicHttpBinding clientBinding = new BasicHttpBinding();
ChannelFactory<IFoo> cf = new ChannelFactory<IFoo>(clientBinding, uri);
IFoo f = cf.CreateChannel();
f.DoSomething();
ServicePoint sp = ServicePointManager.FindServicePoint(new Uri(uri));
sp.MaxIdleTime = 1000 * 10; // Set idle timeout to 10 seconds
((IClientChannel)f).Close();
cf.Close();
Having this redundant code would be probably painful. So I could compile it to a little WCF extension hooked up as a custom operation behaviour like this.
[AttributeUsage(AttributeTargets.Interface | AttributeTargets.Method)]
public class SetHttpConnectionIdleTimeoutAttribute : Attribute, IOperationBehavior
{
private int timeout;
public int Timeout
{
get
{
return timeout;
}
set
{
if (value < 0)
{
throw new InvalidOperationException("Timeout must be a positive integer.");
}
timeout = value;
}
}
public void AddBindingParameters(OperationDescription operationDescription,
BindingParameterCollection bindingParameters)
{
// nop
}
public void ApplyClientBehavior(OperationDescription operationDescription,
ClientOperation clientOperation)
{
ConnectionManager cm = AddOrFindConnectionManager(clientOperation.Parent);
cm.EnlistOperation(clientOperation, timeout);
}
public void ApplyDispatchBehavior(OperationDescription operationDescription,
DispatchOperation dispatchOperation)
{
// nop
}
public void Validate(OperationDescription operationDescription)
{
// nop
}
private ConnectionManager AddOrFindConnectionManager(ClientRuntime clientRuntime)
{
ConnectionManager cm = null;
foreach (IClientMessageInspector mi in clientRuntime.MessageInspectors)
{
if (typeof(ConnectionManager) == mi.GetType())
{
cm = (ConnectionManager)mi;
break;
}
}
if (cm == null)
{
cm = new ConnectionManager();
clientRuntime.MessageInspectors.Add(cm);
}
return cm;
}
}
internal class ConnectionManager : IClientMessageInspector
{
class RequestInfo
{
private string action;
private Uri remoteAddress;
public RequestInfo(string action, Uri remoteAddress)
{
this.action = action;
this.remoteAddress = remoteAddress;
}
public string Action
{
get { return action; }
set { action = value; }
}
public Uri RemoteAddress
{
get { return remoteAddress; }
set { remoteAddress = value; }
}
}
Dictionary<string, int> registeredOperations = new Dictionary<string, int>();
public void AfterReceiveReply(ref Message reply, object correlationState)
{
RequestInfo ri = (RequestInfo)correlationState;
ServicePoint servicePoint = ServicePointManager.FindServicePoint(ri.RemoteAddress);
if (servicePoint != null)
{
Debug.Assert(registeredOperations.ContainsKey(ri.Action));
int timeout = registeredOperations[ri.Action];
servicePoint.MaxIdleTime = timeout;
}
}
public object BeforeSendRequest(ref Message request, IClientChannel channel)
{
return new RequestInfo(request.Headers.Action, channel.RemoteAddress.Uri);
}
public void EnlistOperation(ClientOperation clientOperation, int timeout)
{
if (!registeredOperations.ContainsKey(clientOperation.Action))
{
registeredOperations.Add(clientOperation.Action, timeout);
}
}
}Now this extension can be used as an attribute in your operation contracts as shown below.
[ServiceContract]
interface IFoo
{
[OperationContract]
[SetHttpConnectionIdleTimeout(Timeout=5000)]
void DoSomething();
}If you want to close your connection as soon as you are done, make sure that you set your timeout to 0. But once again, keep in mind that reducing the timeout without a careful thought can surely lead into poor performance. Do it only if you really want to do it.
Alright... we talked about how we can manage the connections on the client side. So what about the service side? In some cases you may want to reduce the idle timeout in the server. This is sometimes essential when you are not controlling clients (the problem that drew my attention to this post was exactly that ). In the service side, I still could not find a way to get the hands on the underlying connections like we did in the client side. However, you could always configure the HTTP runtime using provided tooling. For example, in Windows Vista you can use following netsh command to reduce the timeout to 20 seconds from its 120 seconds default value.
netsh http add timeout timeouttype=idleconnectiontimeout value=20
Furthermore you could use the following command to view the existing timeouts.
It's a bit unfortunate that changing idle timeout using netsh causes the whole http runtime to change though. I really wish that there will be a feature in the future to control this per registered URL basis.
Last time I p
osted about my last day at thinktecture and today I proudly announce that I joined Readify last week. As a result I and my family arrived in Canberra last week. I'm truly honoured to be a part of one of the greatest consulting teams in this part of the world. I met a few guys over past few days and they are just fantastic and I'm quite convinced that there is a whole lot of fun ahead.
In most cases when you switch between jobs it's highly unlikely that you get to do what you want to do. However, I'm pretty delighted that I will get to continue my professional career in the areas that I'm mostly interested in. So you can definitely expect more and more ramblings on my WCF, concurrency and web services adventures. Other than that I'm also hoping to post remarkable things come along the way to settling in for a new life down under.
Last but absolutely not least I would like to thank Readify for making everything so comfortable for me. I never felt like an alien since I arrived in Sydney airport.
BTW: For those who know me, if you are planning to visit down under, try to buzz me before you come. We can probably catch up for couple of drinks together.
"Flames to dust... lovers to friends.... why do all good things come to an end... come to an end..." I just walked into my office humming one of my favorite Nelly Furtado songs and yes, one good (or best rather) thing in my life is about to end today. Although this is not a secret among my best friends, for all the others who've been reading this blog; today I'm retiring from thinktecture family. 
It has been a wonderful ride through out the past three years or so and I'm truly gifted to work with a fantastic team like this. At thinktecture for the first time in my professional life I could find things I like to live with.
However, just being a part of this rapidly changing world, lately I found that it's time for me to say good bye. So I'm leaving this virtual desk with lot of remarkable memories (And I only have good memories at tt ).
I wish all the best to thinktecture and I'm certainly looking forward to see them in our future encounters very soon .
Last but definitely not least during the past few years I and my family made really good family friends and that will remain unchanged for the rest of our life.
thinktecture Rocks!
PS: About next stop in my professional career: I really hope that it deserves a dedicated post in my new blog (http://geeksdiary.com) as it's also going be full of escapades!!!
WCF comes with handful of tracing and logging options. We can just enable it with a few lines in the config and we are good to go. Furthermore the SDK comes with a handy tool svctraceviewer.exe (for wimps ).
However, out-of-the-box trace output gives us access to the data/activities only in the WCF world. For example, at some point we might want to look at the HTTP headers sent/received by the application or we might want to check out the transport level frames are written properly in a custom transport. Although this is quite easy to do with a tool like WireShark or Ethereal, it still wants us to deploy the client and the service applications in two different machines (either virtual or physical).
I'm not done yet. In the end, under the covers, WCF also uses the well-known System.Net APIs at the transport level. Therefore we can just use System.Net tracing settings to capture the wire level traffic right from the dev box.
For example, to capture the http traffic we can use the following config settings.
<system.diagnostics>
<trace autoflush="true"/>
<sources>
<source name="System.Net.HttpListener">
<listeners>
<add name="FooNetTraceListener"/>
</listeners>
</source>
</sources>
<sharedListeners>
<add name="FooNetTraceListener"
type="System.Diagnostics.TextWriterTraceListener"
initializeData="C:\dev\src\lab\wcf\WireLevelTracing\Logs\FooNetTrace.log"
traceOutputOptions="None"/>
</sharedListeners>
<switches>
<add name="System.Net.HttpListener" value="Verbose"/>
</switches>
</system.diagnostics>If you want to capture the traffic for a built-in tcp transport or a custom transport using System.Net.Sockets API; change to trace source to System.Net.Sockets.
Dear fellow developer,
I would like to proudly announce that from this point onward this will be my home in the cloud. Ever since I started 
blogging I've shared my thoughts at two places (buddhike.net and my thinktecture blog respectively). Therefore please bear with me if you happen to end up at a broken link. However if you could take a minute to send me a note regarding anything like that, I'd be more than happy to help you to relocate (or may be re-post if that's really important) the missing content.
On the bright side, I really hope that geeksdiary.com will now be my permanent online residence 
.
In a lot of WCF streaming applications it's common to see a message contract like this.
[MessageContract]
public class DownloadFileRequest
{
[MessageHeader]
private string filename;
[MessageHeader]
private int length;
[MessageBodyMember]
private Stream fileStream;
public DownloadFileRequest()
{
}
}
This often brings up the question "How can I release the stream when the streaming is completed?". I just noticed a nice attempt by using a simple polling mechanism in this article http://www.codeproject.com/WCF/WCF_FileTransfer_Progress.asp. But, there is much simpler and elegant way to do this. You simply have to implement IDisposable interface in your message contract and clean up the stream in the Dispose method.
[MessageContract]
public class DownloadFileRequest : IDisposable
{
[MessageHeader]
private string filename;
[MessageHeader]
private int length;
[MessageBodyMember]
private Stream fileStream;
public DownloadFileRequest()
{
}
public void Dispose()
{
if(fileStream != null)
{
fileStream.Dispose();
fileStream = null;
}
}
}Once the message has been fully streamed the dispatcher runtime will call Dispose in all input/output parameter objects used to construct the message.
In to a little bit of internals like always ;). I looked up where exactly this happens in the reflector.
As far as I can understand this work is done in MessageRpc.DisposeParameterList method.
private void DisposeParameterList(object[] parameters)
{
IDisposable disposable = null;
if (parameters != null)
{
foreach (object obj2 in parameters)
{
disposable = obj2 as IDisposable;
if (disposable != null)
{
try
{
disposable.Dispose();
}
catch (Exception exception)
{
if (DiagnosticUtility.IsFatal(exception))
{
throw;
}
this.channelHandler.HandleError(exception);
}
}
}
}
}
So obviously if our Message contract implements IDisposable it will be perfectly disposed by this function.
Have fun!
Update: Scott Seely explained that this is by design. So I'm changing the title in my post. Also if you want to do this, use async methods as he pointed out.
Last night my friend Michele pointed me an interesting thing on the WCF client side runtime. Basically, she had a service method like as follows.
public void SendMessage(string message)
{
Console.WriteLine("SendMessage: {0}", message);
MessageBox.Show(String.Format("Received message: '{0}'", message));
}
(We know, we know, one should not display message boxes from a service method but here the idea was basically blocking the thread that executes the service method interactively.)
Further more the service was configured as PerCall/Multiple and was running on netTcp.
Then we tried to call this service with an instance of svcutil generated proxy in several threads simultaneously (see below).
private void button1_Click(object sender, EventArgs e)
{
MethodInvoker m = new MethodInvoker(CallService);
m.BeginInvoke(null, null);
}
private void CallService()
{
proxy.SendMessage(string.Format("Message {0}", ++counter));
}When we ran client and the service, we expected to see multiple messages boxes appearing in the service as we click on the button1 in the client. In theory this should work because, although we use a single tcp session, our service is configured for PerCall and Multiple. So the message pump in the service side ChannelHandler can use a new thread to pump the next message from the same tcp session and dispatch it to a new service instance object while the previous message is being processed.
However, things did not workout the way we want. When we clicked the button, first message box appeared in the service. But subsequent clicks did not do so until the first message box was closed (i.e. thread was released). However, when all pending messages are displayed, everything started to work as expected for the subsequent requests.
Let the fun begin! :)
Out of curiosity I got a snapshot of all threads while the client is blocked after the few very first calls. Call stack of one blocking service call revealed quite a lot about what's going on. The stack was like this (irrelevant frames and parameters are removed for simplicity sake).
[In a sleep, wait, or join]
mscorlib.dll!System.Threading.WaitHandle.WaitOne()
mscorlib.dll!System.Threading.WaitHandle.WaitOne()
System.ServiceModel.dll!System.ServiceModel.TimeoutHelper.WaitOne()
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.CallOnceManager.SyncWaiter.Wait()
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce()
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.EnsureOpened()
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call()
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call()
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.InvokeService()
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.Invoke()
mscorlib.dll!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke()
Aha! Interesting! On the client side WCF internally uses ServiceChannel.Call to send/receive messages from the service (regardless of whether you use svcutil generated proxy or manual ChannelFactory and IClientChannel). So it seems like the call is not going out of ServiceChannel.Call at all. It's blocking at a call to EnsureOpened method. Then I opened up Reflector hoping to find what's going on in this method (Lutz, I thank you every time I use it :)).
Refelctor demystified some of the interesting stuff WCF do in the client side. When we call service methods, we simply call them with just a single line (we hardly call Open explicitly). Therefore WCF has to make sure that the underlying channel is Open before sending the request. This opening is just one time action. So internally WCF uses a thing called CallOnceManager to make sure that actions like this are performed only once.
The ServiceChannel.EnsureOpened method calls the CallOnceManager which is responsible for calling Channel.Open once to open the underlying channel (look at the following code I extracted from Reflector).
internal void CallOnce(TimeSpan timeout, ServiceChannel.CallOnceManager cascade)
{
SyncWaiter item = null;
bool flag = false;
if (this.queue != null)
{
lock (this.ThisLock)
{
if (this.queue != null)
{
if (this.isFirst)
{
flag = true;
this.isFirst = false;
}
else
{
item = new SyncWaiter(this);
this.queue.Enqueue(item);
}
}
}
}
SignalNextIfNonNull(cascade);
if (flag)
{
bool flag2 = true;
try
{
this.callOnce.Call(this.channel, timeout);
flag2 = false;
}
finally
{
if (flag2)
{
this.SignalNext();
}
}
}
else if (item != null)
{
item.Wait(timeout);
}
}
CallOnceManager services the simultaneous requests trying to use it for the first time in a FIFO fashion using a queue. This queue is initialized in the ctor. When the CallOnce is invoked it checks whether this queue is available and if "yes", it checks whether this call is the first one. If it's NOT, a waitable object is created and placed in the queue and then CallOnce method blocks on this newly created waitable object.
So out of several simultaneous *first* calls to CallOnce the one that wins open the channel and returns back to the ServiceChannel.Call frame. After doing some more work it finally uses this newly opened channel to send the message. When the reply is returned, it calls CallOnceManager.SingnalIfNotNull method which in turn calls the CallOnceManager.SignalNext method to signal the next waitable object in the queue. When it's signaled the relevant CallOnce call that was waiting on it gets released and the next request is sent to the service.
So in our case first request did not return until we closed the message box. So the subsequent service method calls were hanging at the CallOnceManager.CallOnce method. Because first call has to return and call CallOnceManager.SingnalIfNotNull to get one of the waiting calls released.
IMO, this is too bad. This should essentially check whether the Open has called and if it has, it should just flow without blocking.
On the second part of the question. I was wondering why it was working after end all pending calls. I opened up the SignalNext method and the answer was there.
internal void SignalNext()
{
if (this.queue != null)
{
IWaiter state = null;
lock (this.ThisLock)
{
if (this.queue != null)
{
if (this.queue.Count > 0)
{
state = this.queue.Dequeue();
}
else
{
this.queue = null;
}
}
}
if (state != null)
{
IOThreadScheduler.ScheduleCallback(signalWaiter, state);
}
}
}When the SignalNext is invoked it dequeues the next waitable object from the queue and signals that. If the queue is empty (i.e. no more items striving to be first), it sets the queue to null. Therefore the next call to CallOnceManager.CallOnce just exists without doing anything because the queue is null.
IMO, this is a little bug we have there ;). Dear team, please correct me if I'm missing anything here or if this is by design please explain us why.
Meanwhile if someone is trying to get over the problem I've also found a nice solution. The CallOnceManager is used for automatic opening of channels. But if we call Open explicitly in our code we can get rid of it (Look at how ServiceChannel.EnsureOpened is called in ServiceChannel.Call method).
if (!this.explicitlyOpened)
{
this.EnsureDisplayUI();
this.EnsureOpened(rpc.TimeoutHelper.RemainingTime());
}So we can turn on this explicitlyOpened flag by calling Open *once* in our proxy or the channel before invoking the method.
After all I started to wonder why I did not spend little bit of more time to reflector the client side runtime. There is a lot of fun out there as well!!! :)
Writing my last post about WCF threading internals compelled me to reveal some of the tests I did sometime back. So today we are going to see how we can optimize the CPU utilization by using IO worker threads in CLR thread pool.
We are talking about CPU. Therefore I wrote this simple method to be used in my tests which does some CPU intensive operations. Note that I don't do any IO at all, not even a Console.WriteLine() and this is *very* important to assure the accuracy of the results.
In my CpuIntensiveMethod method I calculate the largest Fibonacci number that is below 100 and I run this calculation for int.MaxValue number of times (I will tell you why this number is preferred).
static void CpuIntensiveMethod(int testNumber)
{
Stopwatch sw = new Stopwatch();
sw.Start();
// Let's perform some math to keep our CPU busy here.
for (int i = 0; i < iterations; i++)
{
// Here we calculate the maximum Fibonacci number less than 100.
int fn = 0;
int preFn = 0;
while (fn <= 100)
{
if (fn < 2)
{
preFn = fn;
fn++;
continue;
}
int next = fn + preFn;
preFn = fn;
fn = next;
}
}
sw.Stop();
timings[testNumber, 0] = Thread.CurrentThread.ManagedThreadId;
timings[testNumber, 1] = sw.ElapsedMilliseconds;
timings[testNumber, 2] = sw.ElapsedTicks;
}In addition to the Fibonacci number calculation I also measure the time taken for computation using a StopWatch (newly available utility since .NET 2.0 bits) and record results in a multi-dimensional array. The reason is again, I did not want to do any IO in this method and thus stayed away from even using Console.WriteLine(). BTW: Since there is no interactive way to see whether the method has completed; I monitored the CPU usage using ProcessExplorer (or TaskManager) and then printed the values in the multi-dimensional array once the CPU usage spike is normalized.
In the first test, I simply ran CpuIntensiveMethod() in a single thread by simple calling this method from my main(). And the result was something like this.
Test 0 ran in managed thread id 1: Milliseconds 32942 Ticks 117920861
During the run I could see my CPU was at 100% which was quite expected. So at this point someone might say, that this is the optimal result that I should ever expect. I would definitely agree if I just want to use a single processor in my entire life. Apparently when I ran this test in my dual core box the process was consuming only 50% of my total processing power. This is the reason why that this model does not fit into server applications world. In fact to put this story in to server applications perspective, let's assume that CpuIntensiveMethod is a function in a server application which invoked by multiple clients. So if we had only one thread, we would take ~33 seconds to service one client. And everyone else would not be taken care of until we are done with current one.
So how can we solve this? We talked about one thread per client scenario in my last post (i.e. distributing work among multiple threads). Would it work? Sit back, we are about to try it now. To simulate this scenario I did my next test by running CpuIntensiveMethod method from several thread pool threads. But before talking about the test, let me tell you a little bit about some important CLR thread pool internals. CLR thread pool has worker threads (IO and non-IO) and a special thread called Gate thread. The gate thread is responsible for spinning up new worker threads. The algorithm it uses to determine when to spin a new thread is actually based on CPU utilization, GC frequency and worker queue size. So currently (to be more precise; as of last time I looked at win32threadpool.cpp back in 2006) this thread spins up for every 500ms. So if CpuIntensiveMethod method does not take too long it might get executed on the same thread pool thread and we would not be able to simulate concurrent clients as we originally intended. This is actually why I wanted to run my test int.MaxValue number of times (so, dear dev fellows, if you are running this code in a beefy box, please forgive me now :)). Getting back to our second test, I could successfully run my method in 10 thread pool threads concurrently and here are the numbers that I ended up with.
Test 0 ran in managed thread id 3: Milliseconds 116489 Ticks 416980652
Test 1 ran in managed thread id 4: Milliseconds 141677 Ticks 507140436
Test 2 ran in managed thread id 5: Milliseconds 149974 Ticks 536840900
Test 3 ran in managed thread id 6: Milliseconds 132707 Ticks 475034210
Test 4 ran in managed thread id 7: Milliseconds 156416 Ticks 559901108
Test 5 ran in managed thread id 8: Milliseconds 137661 Ticks 492765340
Test 6 ran in managed thread id 9: Milliseconds 149318 Ticks 534491916
Test 7 ran in managed thread id 10: Milliseconds 132840 Ticks 475507986
Test 8 ran in managed thread id 11: Milliseconds 136866 Ticks 489920458
Test 9 ran in managed thread id 12: Milliseconds 119222 Ticks 426763067
Looks horrible! Isn't it??? Each thread has taken roughly more than 110 seconds for the computation. This is more than 3 times of the time taken by first test. But don't panic. In fact I was happy the fact that it produced results I wanted to see. The reason is simple. In this test I had 10 threads, each striving to take 100% CPU. This means more work for the scheduler. Switcing contexts, swapping pages etc. cause the delay we are seeing in the above table. Consequently this proves that having one thread per client does not help us achieving the best CPU utilization.
In my final test I tried to service the same 10 concurrent calls that I had in the previous test but this time using thread pool's IOCP as shown below.
for (int i = 0; i < 10; i++)
{
unsafe
{
Overlapped overlapped = new Overlapped(0, 0, IntPtr.Zero, null);
NativeOverlapped* noverlapped = overlapped.Pack(new IOCompletionCallback
(IoThreadProc));
noverlapped->OffsetHigh = i;
ThreadPool.UnsafeQueueNativeOverlapped(noverlapped);
}
}
Before I mention my results I must show you some nits in the above code snippet. Creating a NativeOverlapped structure is an expensive operation. Therefore you should try to create only one and reuse it if that fits bill (wanna be happy? Then listen. This is what WCF IOThreadScheduler does). However, in my case I wanted to have a way to pass the test number to CpuIntensiveMethod method. Therefore I used a simple hack by passing it in the OffsetHigh filed in the NativeOverlapped instance. Also you must make sure that you release the memory block held by NativeOverlapped structure by calling Overlapped.Free() function. Otherwise you will notice a rapidly growing working set and eventually your program will end up with an OOM exception (but here that's fine as this is a demo and I have only 10 instances of NativeOverlapped structure). OK, moving to the test results, this is what I got after running test 3.
Test 0 ran in managed thread id 3: Milliseconds 33614 Ticks 120325281
Test 1 ran in managed thread id 4: Milliseconds 35301 Ticks 126363625
Test 2 ran in managed thread id 3: Milliseconds 32928 Ticks 117867839
Test 3 ran in managed thread id 4: Milliseconds 34606 Ticks 123877189
Test 4 ran in managed thread id 3: Milliseconds 33359 Ticks 119411572
Test 5 ran in managed thread id 4: Milliseconds 34528 Ticks 123596836
Test 6 ran in managed thread id 3: Milliseconds 33502 Ticks 119922606
Test 7 ran in managed thread id 4: Milliseconds 34434 Ticks 123260605
Test 8 ran in managed thread id 3: Milliseconds 34454 Ticks 123331465
Test 9 ran in managed thread id 4: Milliseconds 34645 Ticks 124013520
Pretty impressive! Isn't it??? Now each method has taken roughly around 33 seconds which was the same number we saw during test 1. So we were able to serve 10 concurrent calls while using our optimal processing power. Also notice that each method is executed on a thread with MT id either 3 or 4. What does this mean? When CLR thread pool creates its IOCP, it calculates the number of CPUs in the box and passes that value to NumberOfConcurrentThreads parameter of CreateIoCompletionPort function. So in my case, I ran these tests in a dual core box and thus IOCP only allowed only two threads to be active concurrently. That's why we see only manage thread ids 3 and 4.
Concluding this post, I hope that these tests will help you to understand the optimizations that could be achieved by using IO worker threads.
Wanna try out the tests yourself? Click here to download the sample code.
Have fun!
Along with its bunch of different features, WCF carries a lot of performance optimizations as well (you think I'm kidding? Then see it yourself http://msdn2.microsoft.com/en-us/library/bb310550.aspx). As a part of this, WCF has given a lot of thought about threading model that it uses behind the scenes.
At this point you might probably think;
"Well… I know it uses thread pool API. There is nothing so much about it. I just know that it performs well." Well you are correct but not really correct. I hope you want to go down to the metal. So please read on… :)
WCF uses CLR thread pool threads to do things asynchronously. However, interestingly it uses IO worker threads in the thread pool instead of the regular thread pool worker threads (don't fuzz if you are not aware of these two kinds of threads). The theory behind IO worker threads reveals a lot about why WCF use it. Therefore I thought I would dedicate this post to talk a little bit about it.
So before we actually dig in let me ask you a simple question. When do you consider that you are taking max out of your CPU? Is it when a single thread trying to take up 100% or multiple threads trying take up 100%? I know, you said single thread 100% case which is correct. When you have multiple CPU bound threads additional costs of things like context switching slow things down. Wanna see it yourself? Write a small lengthy loop. Measure the time it takes when you run it in a single thread. Then delegate it to several threads and again measure the time that each of them take to finish it and compare the values.
So… technically speaking, we can achieve the best CPU utilization only by sticking to "One tread per CPU per execution quantum" invariant.
I/O completion ports (IOCP) were introduce to Windows NT kernel to achieve this goal. Although it's a complex technology, the fundamental theory behind the scene is fairly straightforward.
Before the invent of IOCP; there were two major IO programming techniques. One thread for all IO and one thread per IO. In one thread for all IO model; a thread that is IO bound had to wait doing nothing. Also all other IO operation were blocked until the one going on is done. Although this model was fair enough for single threaded client apps, server applications did not fit in at all. For example, if a simple server was written in this way it will serve only one client at a time. One thread per IO model on the other hand spawn up a new thread for each client. But this eventually ended up with too many threads striving for CPU. So we either have too few threads or too many threads causing the trouble.
In order to solve this problem; IOCP work like a controller hub between two parties. In one end it receives IO completion packets saying that some work is available. On the other end it has some IO worker threads waiting for work. When an IOCP receives an I/O completion packet, it makes one of the waiting threads active and delegate the available work (thread is picked up in LIFO order to avoid potential context switching). So what? How does this model help to solve problem we addressed earlier. The secret lies within NumberOfConcurrentThreads parameter value we pass when we create the IOCP using CreateIoCompletionPort function. This parameter tells IOCP how many concurrent threads that we actually want to have active to process the incoming work. The preferred value for this number equals to the number of CPUs you have in the box. So no matter how fast the work is being queued, we only have desired number of threads concurrently processing them. The other cool thing about this is; when an active IO worker thread goes to wait state (may be it's doing some more IO work), windows scheduler tells the pertaining IOCP that one of its active threads are inactive now. So that the IOCP can make another thread active to perform some other work (Smart! Isn't it? ;)). Consequently the number of active threads in an IOCP at given time is usually a little bit higher than the provided NumberOfConcurrentThreads value.
OK. Now we know why IOCPs are so elegant. But how does WCF actually use it? Well... CLR thread pool has an IOCP associated with it. When thread pool creates its IOCP for the first time, it also creates IO worker threads which are waiting for work. So essentially, we can get an IO worker thread to do some work by sending an IO completion packet to this IOCP. To do that we can use ThreadPool.UnsafeQueueNativeOverlapped function (This method internally invokes the PostQueuedCompletionStatus function). Here is a little program to demonstrate how you could do that.
static void Main(string[] args)
{
unsafe
{
// Create an Overlapped structure and pack it with a pointer to the function
// that we want to invoke from the IO worker thread.
Overlapped overlapped = new Overlapped(0, 0, IntPtr.Zero, null);
NativeOverlapped* pOverlapped = overlapped.UnsafePack(
new IOCompletionCallback(OnIoCompletion), null);
// Send an IO completion packet to thrad pool's IOCP
ThreadPool.UnsafeQueueNativeOverlapped(pOverlapped);
}
}
static unsafe void OnIoCompletion(uint errorCode,
uint numBytes,
NativeOverlapped* pOverlapped)
{
Console.WriteLine("This is from an IO worker thread");
}WCF also basically follows the same concept. But it has an elegantly designed queue based IO thread scheduler. I would like to dedicate a separate post to talk about how exactly it works. But if you are reflector fan like me, take a look at System.ServiveModel.IoThreadScheduler class and you will see it in your own eyes.
So what does all this tell us? WCF uses this IoThreadScheduler to queue work items for IO worker threads. This way it preserves the "One thread per CPU per execution quantum" constant and achieves the best CPU utilization. It never (may be I should say I've never seen it but I have a lot of faith on the WCF team) uses ThreadPool.QueueUserWorkItem API and thus refrain from using regular thread pool worker threads (I'm sure now you see why WCF performs a lot better than ASMX runtime ;)).
OK, are you still not certain that WCF is working this way? Cool! I guess you don't have too much faith on me. Well.. Then get ready for a little exercise. Create a little service with a single operation. Make this operation do some lengthy CPU intensive work (perhaps a loop doing some math). And then try to invoke this operation from multiple clients simultaneously (or you can call it from multiple threads in the same client). How many requests that your server can service concurrently? Looking forward to hearing your results :)
During the past few days I came across some SOAP routing intermediary implementations (of course running on WCF ;)) and each of them were trying to route the WCF messages by changing the “To” WS-Addressing header in the message. In some cases this even required completely reconstructing the message by reading its body contents. This actually made me think about the basics once again. What actually happens in the real routers? For example, does Windows TDI driver for TCP/IP change the destination IP to your router’s IP after looking at the routing table? No! Instead, the transport transmits the traffic to the appropriate IP as specified in the routing table. Then the router reads the destination IP in the incoming TCP segments and forwards the traffic to the next network hop according to the routing table in the router itself. So essentially, the router is a device that simply forwards messages without tweaking them. Likewise this should be the theory behind the SOAP routers as well (of course SOAP works at a much higher level and you can divert this approach to meet your custom needs. But I’m talking about the routing in general).
So how exactly you can do this in WCF? The answer is hidden in an attribute that you might not pay too much attention in your everyday WCF adventures ;). When you configure an endpoint you can actually, specify the service address as well as the actual listening address as follows.
[ServiceEndpoint]
<endpoint address="http://localhost:8000/service/dummyendpoint.svc"
listenUri="http://localhost:8000/service/actualendpoint.svc"
contract="…"
binding="…"
bindingConfiguration="…" />
When you start your service the underlying transport actually uses the address specified in the listenUri to listen to the incoming traffic (if this is not specified, it uses the endpoint address by default). The service address on the other hand is the one which goes in the WS-Addressing “TO” header. This address is validated by service model layer to make sure that the messages that arrive at the endpoint are truly intended for this service (otherwise you’ll get the address filter mismatch error… remember that? ;)). So with these two attributes in our hands we can successfully model the aforementioned routing in the SOAP level as well. You can do it by having your router service actually listening on the endpoint address specified in the service (see below).
[Router endpoint]
<endpoint address="http://localhost:8000/service/dummyendpoint.svc"
contract="…"
binding="…"
bindingConfiguration="…" />
Then you can simply forward the messages to the actual service endpoints according your routing rules. For example, you can determine a message to the above service endpoint by looking at the action header of the incoming message and forward it to the service by making a channel to http://localhost:8000/service/actualendpoint.svc endpoint (which is the actual endpoint of our service). Also note that this way, no matter how many intermediaries the message passes through the WS-Addressing “TO” header remains consistent.
When I started writing this post I intended to provide a very rough sample that I created couple of months ago (in fact I gave up re-inventing the wheel after seeing Shy Cohen’s wonderful lossy router ;)). But then I realized that the SDK routing sample perfectly demonstrates this. So take a look at it to get a better picture on it.
Have fun!
More Posts
Next page »