To follow up on my last post, here are a few more debugging tips.
Attach, Attach, Attach
In the setup I described previously, I had a Silverlight app and several web services, all hosted on their own port in Cassini. This lead to some lovely cross domain communication issues. It also means that each of these components was running in its own process. So I would start up my Silverlight app, set a breakpoint in my web service and never hit that breakpoint.
Why not? Because the debugger was not attached to that web service’s process. The answer? One of my favorite Visual Studio niceties, attaching a debugger to a process (remote or local). Just go to Debug –> Attach to Process. In this case, find the right copy of WebDev.WebServer.EXE (Cassini) identified by the port number and Attach:
If it is not, this technique needs to be in your debugging toolbox. Especially for debugging a deployed application. If the application is on a remote server, and your attempt to attach fails, then you may need to activate the Remote Debugging service on that server.
I’ve used this technique extensively to debug ASP.NET apps that were behaving nicely locally, but not so on a test server. When you’re dealing with an app on IIS, just find the w3p.exe process (IIS) and attach. If you can’t see the w3p process on the remote server, check both checkboxes in the attach dialog to see all processes from all users.
I bring this up because sometimes I tell people that I debugged a deployed app and they assume I have some kind of deep code-fu. Really, don’t be intimidated by this technique, it is simple and has great benefits.
Another Candidate for Worst Exception Ever
The old NotFound exception. Which might lead you to believe that the endpoint (web service) was not found.
Well, I suppose it could mean that, but I’ve found it actually can indicate a problem with serializing your service’s return value. Or more accurately, I inferred that this could be the problem from reading about this error at: http://stackoverflow.com/questions/814627/silverlight-webservice-the-remote-server-returned-an-error-notfound and http://social.msdn.microsoft.com/Forums/en/wcf/thread/b9672bc6-5707-42e6-b6ce-621df78b037d.
WCF needs to be able serialize the data types it returns. In my case, one of my return types had a property that was an enumeration. No big deal, those work OK.
Except, we had just added a new value to the database table that backed this enumeration. We only used this new enumeration value in one isolated component, and did not update the enumeration object my component was using.
So, in our WCF service we cheerfully loaded a row from the database, casting one of the columns to this enumeration. The cast was successful, despite there not being a corresponding Enum value, but when WCF tried to serialize this half-Enum it threw up and out came “NotFound.”
Beyond an obscure exception message, this bug highlighted some issue with our design, particularly our WCF data objects. For more on this topic see this wonderful blog post: http://davybrion.com/blog/2010/05/why-you-shouldnt-expose-your-entities-through-your-services/
This post was born out of a long and frustrating process of trying to debug two Silverlight apps.
While you can apply these tips more generally as you choose, I can only promise they are relevant in the following situation:
- Your Silverlight application polls one or more WCF services that are not in the Silverlight web project
- You are developing on a shared server or otherwise would like to run the services and the Silverlight app in the Development Server (Cassini)
- You press F5 to debug your Silverlight application and you are bombarded with uncaught and seemingly untraceable WCF exceptions
- You have a feeling of anger, coupled with self loathing, that you cannot debug your own app
With this grim stage set, let’s follow along with how I made things better.
Unhandled Exceptions
Yuck, an unhandled exception is shocking; first that your code has any bugs, second that they dare to interrupt your debugging session. Of course, you want to fix the source of these exceptions, but first you’ve got to find them. Heading over to your app.xaml.cs file and let’s wire up the global unhandled exception handler.
In the constructor:
this.UnhandledException += this.Application_UnhandledException;
The handler code:
private void Application_UnhandledException(object sender, ApplicationUnhandledExceptionEventArgs e)
{
if (!System.Diagnostics.Debugger.IsAttached)
{
//define what your app should do when in PROD
}
else
{
//force break in vs debugger on error
System.Diagnostics.Debugger.Break();
//but can continue if like
e.Handled = true;
}
}
As you can see here, this code checks to see if there is a debugger attached to the process, and if there is, it forces a stop. The behavior you’ll see when debugging is that if you have an unhandled exception, all of the sudden you’ll jump to the else clause of this method and be able to inspect the exception object e for details.
My hope was that wiring this up would give me a chance to get more details about the following exception I kept seeing:
Scouting WCF Cross Domain Exceptions
Sadly, although I was now catching the above AsyncCallback exception, the exception details were useless. Which WCF service was erroring out and why? I cheated a bit, as I had previously had some cross domain troubles before, but if I hadn’t, here’s how I could have found out that this was the issue.
First off, whenever I have multiple WCF services in play, I disable all but one of them, run again and see if there is still an error. This should quickly lead you to which service is having trouble. At that point, assuming your services are running over HTTP, your next step should to be use an HTTP sniffer and take a look at the traffic going between the Silverlight application and the service.
My HTTP sniffer of choice is Fiddler, a wonderful little app that I install on any dev machine I touch. Unfortunately, Fiddler does not show localhost traffic by default when running in IE. This has something to do with IE ignoring proxy servers (including Fiddler’s) for localhost. There are some workarounds, such as specifying the machine name instead of localhost or doing localhost.:port#.
Instead, I chose to install the Web Development Helper, which is a toolbar for IE. I would describe it is as similar to Fiddler, except for it captures localhost traffic out of the box, is a bit harder to use and crashes a lot.
Whatever your tool, assume you can now capture localhost traffic. In my case, I was looking for traffic between my Silverlight application, for example localhost:50001/SLApp and a the service in question on localhost:50002/WCFService.
As I mentioned at the top, I was running everything in Cassini, so every application was on its own port. By definition, even though everything was on localhost, being on different ports made all calls cross domain ones. As a security measure, such cross domain calls are restricted in Silverlight.
There are two ways to enable cross domain calls. Either put a valid crossdomain.xml or clientaccesspolicy.xml at the root of the site in question. A crossdomain.xml is an Adobe Flash security measure that Silverlight also supports. Clientaccesspolicy.xml is more specifically for Silverlight. Either will work. What “root” means is straight forward on IIS, something like c:\inetpub\wwwroot. For a Cassini hosted app, I’ve found just including the clientaccesspolicy.xml in the project root directory works.
Getting back to HTTP traffic; when the Silverlight app tries to make a call to the WCF service app and the address is cross domain, it will first attempt to retrieve a crossdomain or clientaccesspolicy xml from the other domain to see if the call is allowed.
This will show up in the HTTP traffic log as a GET to otherdomain/crossdomain.xml. If that returns an HTTP code 200, that file was found and its contents will be parsed to see if the call is allowed. If that returns an HTTP code 404, not found, then next up you should see a GET to otherdomain/clientaccesspolicy.xml.
So if you see a 404 for both requests, your call will fail with a nasty exception and you should add a policy file. If you see a 200 for one of the requests and the call still fails, your file is probably not valid.
Wide open Clientaccesspolicy (for DEV only)
When you’re debugging, unless you’re testing security settings, you just want the thing to work. With that in mind, here’s a very permissive clientaccesspolicy.xml that allows all cross domain calls:
<?xml version="1.0" encoding="utf-8" ?>
<access-policy>
<cross-domain-access>
<policy>
<allow-from http-request-headers="SOAPAction">
<domain uri="http://*"/>
<domain uri="https://*" />
</allow-from>
<grant-to>
<resource include-subpaths="true" path="/"/>
</grant-to>
</policy>
</cross-domain-access>
</access-policy>
With this file in place, you should see a 200 OK response for the clientaccesspolicy.xml, followed by your WCF service url getting hit.
At this point, you may see exceptions from your service or that you’ve configured the wrong service address in your config, but at least you’ll be back in comfortable territory.
It doesn’t seem like too much, but after doing this for one app, I was able to easily take another tough to debug Silverlight app and get it playing nice with Visual Studio in no time. Hope this helps you all as well.
--- Link Roll ---
http://timheuer.com/blog/archive/2008/04/06/silverlight-cross-domain-policy-file-snippet-intellisense.aspx
http://projects.nikhilk.net/WebDevHelper
http://blogs.msdn.com/b/silverlightws/archive/2008/04/16/debugging-web-service-usage-in-silverlight-2.aspx (basically a more detailed and technical version of this post)
http://blogs.msdn.com/b/carlosfigueira/archive/2008/03/07/enabling-cross-domain-calls-for-silverlight-apps-on-self-hosted-web-services.aspx
http://timheuer.com/blog/archive/2008/04/09/silverlight-cannot-access-web-service.aspx
http://timheuer.com/blog/archive/2008/10/14/calling-secure-services-with-silverlight-2-ssl-https.aspx
http://www.adobe.com/devnet/articles/crossdomain_policy_file_spec.html
http://www.martijndevisser.com/blog/2005/why-crossdomainxml-is-a-good-thing/ (slideshow about why a cross domain access policy file is a good thing)
http://forums.silverlight.net/forums/p/18490/62275.aspx (difference between crossdomain and clientaccesspolicy xml files)
The subtitle of Daniel Coyle’s intriguing book The Talent Code is “Greatness Isn’t Born. It’s Grown. Here’s How.” The Talent Code proceeds to layout a theory of how expertise can be cultivated through specific practices that encourage the growth of myelin in the brain. Myelin is a material that is produced and wraps around heavily used circuits in the brain, making them more efficient.
Coyle uses an analogy that geeks will appreciate. When a circuit in the brain is used a lot (i.e. a specific action is repeated), the myelin insulates that circuit, increasing its bandwidth from telephone over copper to high speed broadband. This leads to the funny phenomenon of effortless expertise. Although highly skilled, the best players make it look easy.
Coyle provides some biological backing for the long held theory that it takes 10,000 hours of practice to achieve mastery over a given subject. 10,000 hours or 10 years, as in, Teach Yourself Programming in Ten Years and others. However, it is not just that more hours equals more mastery. The other factors that Coyle identifies includes deep practice, practice which crucially involves drills that are challenging without being impossible. Another way to put it is that every day you spend doing only tasks you find monotonous and automatic, you are literally stagnating your brain’s development!
Perhaps Coyle’s subtitle, needs one more phrase, “Greatness Isn’t Born. It’s Grown. Here’s How. And oh yeah, it’s not easy.” Challenging yourself, continuing to persist in the face of repeated failures, practicing every day is not easy.
As consultants, we sell our expertise, so it makes sense that we plan projects so that people can play to their strengths. At the same time, an important part of our culture is constant improvement, challenging yourself to be better. And the balancing contest ensues.
I just finished working on a proof of concept (POC) we did for a project we are bidding on. Completely time boxed, so our team naturally split responsibilities amongst ourselves according to who was better at what. I must have been pretty bad at the other components, as I found myself working on the user interface, not my usual strength.
The POC had a website frontend, and one thing I do know is HTML. After starting out in pure ASP.NET WebForms, I got frustrated as time was ticking, I knew what I wanted in HTML, but I couldn’t coax the right output out of the ASP.NET controls.
I needed two or three elements on the screen that were identical in layout, with different content. With a backup plan in of writing the HTML into the response by hand, I decided to challenge myself a bit and see what I could do in an hour or two using the Microsoft submitted jQuery micro-templating JavaScript library.
This “risk” paid off. I was able to quickly get the user interface up and running, responsive to the JSON data we were working with. I felt energized by the double win of getting the POC ready and learning something new. Opportunities specifically like this POC don’t come around often, but the takeaway is that while it won’t be easy, there are ways to generate your own opportunities to grow towards greatness.
Sometimes I come across a way of putting something that it is pithy good, not Hallmark trite, but an impactful and concise way of clarifying a previously obscure concept. A recent one of these happy occurrences was when I was reading the excellent Art of Unit Testing by Roy Osherove.
After going through the basics of why you’d want to test code and how to do it, Roy confronts a frequent objection to having unit tests, that it ends up changing how you design your components:
When we write unit tests for our code, we are adding another end user (the test) to the object model. That end user is just as important as the original one, but it has different goals when using the model. The test has specific requirements from the object model that seem to defy the basic logic behind a couple of object-oriented principles, mainly encapsulation. [emphasis added by me]
When I read this, something clicked for me. I used to find it persuasive that because unit tests caused you to change your design they were more disruptive than they were worth. The counter argument I heard is that the disruption was OK, because testable design was just obviously better. That argument was not convincing as it seemed like delusional arrogance to suggest that any one of type of design was just inherently better for the particular applications I was building.
What was missing was that I was not thinking of unit tests as an additional and equal end user to my design. If I accepted that proposition, than it was indeed obvious that a testable design was better because now all users of my component would be satisfied.
Have I accepted that proposition? I’d phrase it slightly different. I find more and more that having unit tests helps me write better, less buggy code before it gets to production or QA. As I write more unit tests, it gets easier to see how to create testable components, so I don’t feel like its taking me as much extra time up front. I pick and choose components that seem most likely to benefit from automated tests and it is working out nicely.
If you already implement Test Driven Development, this whole post was probably a waste of your time <g> If you hate the idea of unit tests, well, probably not a great value prop for you either. However, if you are somewhere in between, at least take a minute and check out a sample chapter from Roy’s book at: http://www.manning.com/osherove/.
As a consumer of geeky news it is hard to check my Google Reader without running into two or three posts about Apple’s iPad and in particular the changes to the developer guidelines which seemingly restrict developers to using Apple’s Xcode tool and Objective-C language for iPad apps.
One of the alternatives to Objective-C affected, is MonoTouch, an option with some appeal to me as it is based on the Mono implementation of C#. Seemingly restricted is the key word here, as far as I can tell, no official announcement has been made about its fate. For more details around MonoTouch for iPhone OS, check out Miguel de Icaza’s post: http://tirania.org/blog/archive/2010/Apr-28.html.
These restrictions have provoked some outrage as the perception is that Apple is arrogantly restricting developer’s freedom to create applications as they choose and perhaps unwittingly shortchanging iPhone/iPad users who won’t benefit from these now never-to-be-made great applications.
Apple’s response has mostly been to say they are concentrating on providing a certain user experience to their customers, and to do this, they insist everyone uses the tools they approve. Which isn’t a surprising line of reasoning given Apple restricts the hardware used and content of the apps already. The vogue term for this approach is curated, as in a benevolent museum director selecting only the finest artifacts for display or a wise gardener arranging the plants in a garden just so.
If this is what a curated experience is like it is hard to argue that consumers are not responding. My iPhone is probably the most satisfying piece of technology I own. Coming from the Razr, it really was an revolution in how the form factor, interface and user experience all tied together.
While the curated approach reinvented the smart phone genre, it is easy to forget that this is not a new approach for Apple. Macbooks and Macs are Apple hardware that run Apple software. And they’ve been successful, but not quite in the same way as the iPhone or iPad (based on early indications).
Why not? Well a curated approach can only be wildly successful if the curator a) makes the right choices and b) offers choices that no one else has. Although its advantages are eroding, the iPhone was different from other phones, a unique, focused, touch-centric experience. The iPad is an attempt to define another category of computing. Macs and Macbooks are great devices, but are not fundamentally a different user experience than a PC, you still have windows, file folders, mouse and keyboard, and similar applications.
So the big question for Apple is can they hold on to their market advantage, continuing innovating in user experience and stay on top? Or are they going be like Xerox, and the rest of the world says thank you for the windows metaphor, now let me implement that better? It will be exciting to watch, with Android already a viable competitor and Microsoft readying Windows Phone 7.
And to close the loop back to the restrictions on developing for iPhone OS. At this point the main target appears to be Adobe and Adobe Flash. Apple’s calculation is that a) they don’t need those developers or b) the developers they want will learn Apple’s stuff anyway. My guess is that they are correct; that as much as I like the idea of developers having more options, I am not going to buy a competitor’s product to spite Apple unless that product is just as usable. For a non-technical consumer, I don’t know that this conversation even factors into the buying decision. If it did, we’d be talking about how Microsoft is trying to retake a slice of market share from the behemoth that is Linux.
Pivotal Tracker is a free, online agile project management system. I’ve been using it recently to better communicate to customers about the current state of our project. In Pivotal Tracker, the unit of work is a story and stories are arranged into iterations or delivery cycles.
Stories can be any level of granularity you want, but the idea is to use stories to communicate clearly to customers, so you don’t want to write a novel. You especially don’t want to write a list of detailed programming tasks. A good story for a point of sale system might be: “Allow managers to override the price of an item while ringing up a customer.” A less useful story: “Script out the process of adding a manager flag to the user table and stage that script into the deploy directory.”
Stories are estimated using a point scale, by default 1, 2 or 3. Iterations are then automatically laid out by combining enough tasks to fill the point total for that period of time. You have to start with a guess on how many points your team can do in an iteration, then adjust with real data as you complete iterations.
This is basic agile methodology, but where Pivotal Tracker adds value is that it automatically and graphically lays out iterations for you on your project site. This makes communication and planning easy. Compiling release notes is no longer painful as it has been clear from the outset what work is going on.
While I much prefer Pivotal Tracker’s customer facing interface over what we used previously (TFS), I see a couple of gaps. First, I have not able to make much headway with the reporting tools. Despite my complaints about TFS, it can produce some nice reports. Second, it’s not clear where if at all, I’d keep track of purely internal tasks. I’m talking about server maintenance, cleaning up source control, checking back on some code which you never quite felt right about. There’s no purpose in cluttering up an iteration backlog with these items, but if you don’t track them, you lose them. I’m not sure what a good answer for that is.
One gap I thought I’d see, which I don’t, is more granular dev tasks. If I’m implementing a story, I’ll write out the steps and track my progress, but really, those steps aren’t useful to anybody but me. The only time I’ve found that level of detail really useful is when my tasks are defined at too high a level anyway or when I’m working with someone who needs more coaching and might not be able to finish a story in time without some scaffolding to get them going.
You can learn more about Pivotal Tracker at: http://www.pivotaltracker.com/learnmore.
--- Relevant Links ---
A good intro to stories: http://www.agilemodeling.com/artifacts/userStory.htm
If I were to rank the most consistently irritating parts of my work day, using TFS would come in first by a wide margin. Even repeated network outages this week seem like a pleasant reprieve from this monolithic beast.
This is not a reflexive anti-Microsoft feeling, that attitude just wouldn’t work for a consultant who does .NET development. It is also not an utter dismissal of TFS as worthless; I’ve seen people use it effectively on several projects.
So why? I’ll start with a laundry list of shortcomings. An out of the box UI for work items that is insultingly bad, a source control system that is confoundingly fragile when handling merges, folder renames and long file names, the arcane XML wizardry necessary to customize a template and a build system that adds an extra layer of oddness on top of msbuild.
I’m sure my legion of readers will soon point out to me how I can work around all these issues, how this is fixed in TFS 2010 or with this add-in, and how once you have everything set up, you’re fine. And they’d be right, any one of these problems could be worked around.
If not dirty laundry, what else? I thought about it for a while, and came to the conclusion that TFS is so irritating to me because it represents a vision of software development that I find unappealing. To expand upon this, let’s start with some wisdom from those great PSA’s at the end of the G.I. Joe cartoons of the 80’s: “Now you know, and knowing is half the battle.”
In software development, I’d go further and say knowing is more than half the battle. Understanding the dimensions of the problem you are trying to solve, the needs of the users, the value that your software can provide are more than half the battle. Implementation of this understanding is not easy, but it is not even possible without this knowledge.
Assuming we have a fixed amount of time and mental energy for any project, why does this spell trouble for TFS? If you think about what TFS is doing, it’s offering you a huge array of options to track the day to day implementation of your project. From tasks, to code churn, to test coverage. All valuable metrics, but only in exchange for valuable time to get it all working.
In addition, when you have a shiny toy like TFS, the temptation is to feel obligated to use it. So the push from TFS is to encourage a project manager and team to focus on process and metrics around process. You can get great visibility, and graphs to show your project stakeholders, but none of that is important if you are not implementing the right product. Not just unimportant, these activities can be harmful as they drain your time and sap your creativity away from the rest of the project.
To be more concrete, let’s suppose your organization has invested the time to create a template for your projects and trained people in how to use it, so there is no longer a big investment of time for each project to get up and running. First, I’d challenge if that template could be specific enough to be full featured and still applicable for any project. Second, the very existence of this template would be a indication to a project manager that the success of their project was somehow directly related to fitting management of that project into this format. Again, while the capabilities are wonderful, the mirage is there; just get everything into TFS and your project will run smoothly.
I’ll close the loop on this first topic by proposing a thought experiment. Think of the projects you’ve worked on. How many times have you been chagrined to discover you’ve implemented the wrong feature, misunderstood how a feature should work or just plain spent too much time on a screen that nobody uses? That sounds like a really worthwhile area to invest time in improving. How about going back to these projects and thinking about how many times you wished you had optimized the state change flow of your tasks or been embarrassed to not have a code churn report linked back to the latest changeset?
With thanks to the Real American Heroes, I’ll move on to a more current influence, that of the developers at 37signals, and their philosophy towards software development. This philosophy, fully detailed in the books Getting Real and Rework, is a vision of software that under does the competition. This is software that is deliberately limited in functionality in order to concentrate fully on making sure ever feature that is there is awesome and needed.
Why is this relevant? Well, in one of those fun seeming paradoxes in life, constraints can be a spark for creativity. Think Twitter, the small screen of an iPhone, the limitations of HTML for applications, the low memory limits of older or embedded system. As long as there is some freedom within those constraints, amazing things emerge. For project management, some of the most respected people in the industry recommend using just index cards, pens and tape. They argue that with change the constant in software development, your process should be as limited (yet rigorous) as possible.
Looking at TFS, this is not a system designed to under do anybody. It is a big jumble of components and options, with every feature you could think of. Predictably this means many basic functions are hard to use. For task management, many people just use an Excel spreadsheet linked up to TFS. Not a stirring endorsement of the tooling there. TFS as a whole would be far more appealing to me if there was less of it, but better. I’d cut 50% of the features to make the other half really amaze and inspire me.
And that’s really the heart of the matter. TFS has great promise and I want to believe it can work better. But ultimately it focuses your attention on a lot of stuff that doesn’t really matter and then clamps down your creativity in a mess of forms and dialogs obscuring what does.
--- Relevant Links ---
All those great G.I. Joe PSA’s are on YouTube, including lots of mashed up versions. A simple Google search will get you on the right track.
When you just use IE, Firefox or Chrome it can be easy to forget that HTTP is about more then just going to check the latest tech news at Engadget. It is a full and rich protocol, and a great way to experience that richness is the powerful command line utility cURL.
cURL has a lot of options, but the syntax starts out simple. You can retrieve the contents of a web page with a simple “curl http://blogs.claritycon.com/”. The results should be the full text of the web page, tags and all. From there, you can use –X to specify the HTTP verb to use, POST, PUT, DELETE, PATCH, etc and –d to specify the payload of a POST or PUT.
I have found cURL to be incredibly useful for two scenarios. First, as a good way to test basic web services. Second, while working a bit with CouchDB and another document based database, cURL has helped me learn more about RESTful APIs, including different verbs and response codes.
cURL is a mainstay in our environments and programming languages precisely because it is simple, powerful and discoverable. I encourage more .NET developers to take a look, bask on the command line for a while and enjoy the plain text of the web. And this excellent logo:
-- Relevant Links --
It’s not always the case with manuals, but the manual for cURL is quite useful: http://curl.haxx.se/docs/manual.html
To make your command line look a little nicer (and more powerful) on Windows, check out Console and add some transparency effects: http://sourceforge.net/projects/console/
A few days ago my attention was drawn to a tweet spat between Karl Seguin and Scott Hanselman around the relaunch of ASP.NET and the title element in HTML. Tempest in a teapot of course, but worthwhile as I did some googling on Karl and found his blog at codebetter.com.
From there it was a short jump to his free e-book, The Foundations of Programming. This short book is distinguished by its orientation, opinionated, its tone, mentoring and its honesty, which is refreshing. In Foundations, Karl covers what he considers the basics of programming and good design, including test driven development, dependency injection and domain driven design.
Karl is opinionated, as the topics suggest, and doesn’t bother to pretend that he doesn’t think what he’s suggesting is the better way, not just another way. He is aligned with ALT.NET, and gives an excellent overview of what that means; an overview more enlightening than the ALT.NET site. ALT.NET has its critics, but presenting a strong opinion grabbed my attention as a reader.
It is a short walk from opinionated to hectoring, but Karl held my attention without insulting me. He takes the time to explain, with examples, from the ground up, the problems that test driven development and dependency injection solve. So for dependency injection he builds it up from no DI, to a hand crafted approach, to a full fledged DI framework. This approach is more persuasive than just proscriptive and engaged me as the reader to follow along with his train of thought.
Foundations is not as pedantic as I am making it sound. The final ingredient in Karl’s mix is honesty. He acknowledges that sometimes unit testing does cost more up front and take more time. He admits that sometimes he designs something a certain way just to be testable. He also warns that focusing too much on DI and loose coupling can lead to the poor design you are trying to avoid. These points add depth to his argument as I could tell he’s speaking from experience, with some hard won lessons.
I enjoyed The Foundations of Programming. When I was done with it, I was amazed how much I got a lot out of its 80 some pages. It is a rarity to come across something worthwhile that is longer then a tweet, but shorter than a tome these days. Well done Karl.
-- Relevant Links --
The now titled and newly relaunched page in question: http://www.asp.net/
The pleasantly confusing ALT.NET homepage: http://altdotnet.org/
A longer review, with details, chapter listings and all that important stuff: http://accidentaltechnologist.com/book-reviews/book-review-foundations-of-programming-by-karl-seguin/
In my last post I discussed how some creative thinking on my part led me to redo large chunks of a task scheduling library. As promised, here are some of the major changes and what I learned from them.
Lush Verbiage
In the refactor, I expanded upon the concept of a timekeeper and extended the metaphor to the rest of they component. So a timekeeper, keeps time, or in this case, a timetable. A timetable is composed of entries, a combination of time slot and actions scheduled for that time. A timekeeper has a worker which “works” the timetable using a simple procedure. Find the next entry in the timetable, keep track of the passage of time until it is past that entry’s time, execute the actions for that entry, find the next entry, continue on.
The metaphor is strained at times, but by carefully choosing descriptive names for classes from the physical world, there was less mental friction when trying to implement or explain the component.
Design Simplicity
Dealing with time based code in .NET is harder than it should be. Or more accurately, harder then I’d want it to be. Even the conceptually simple idea of scheduling a task to happen every hour is fraught with peril. I started with a Timer that went off at the next hour and repeated with a period of 60 minutes. This mostly works, but that timer will sometimes fire before the hour is up. Only a fraction of a second before the hour, but that is time enough to wreck havoc on logic that looks for the next event chronologically or on other components that depend on the hourly events occurring on or after the hour. If not havoc, at least make it messier.
Ultimately, I could have gotten past that challenge, but I decided to simplify the code by making the main “tick” loop of the timekeeper (what looks for timetable entries that need to be executed) a brute force style thread.sleep, check, thread.sleep, check, loop. The idea of this approach grates on me as it is making the program spin more then it needs to, but a sleep of 100 ms was enough to not noticeably affect CPU utilization. In addition, this made it very easy to add support for a clean stop of the timekeeper worker:
internal class TimekeeperWorker : BackgroundWorker
{...
protected override void OnDoWork(DoWorkEventArgs e)
{...
while (!CancellationPending)
{
Thread.Sleep(100);
… “tick” loop logic
}
//cancellation pending, so exit out
e.Cancel = true;
Even after this latest refactor, there are more places I could simplify this code. However, just like optimization, there is an effort associated with simplification that isn’t always needed. At least not yet.
Daylight Savings Time
This is kind of a subset of Design Simplicity, but one of the concerns I had with this component was how it would handle Daylight Savings Time. When the clock jumps from 1:59AM to 1AM or 3AM for example. After some discussion with the team, the behavior we decided on was that in the case of jumping back, the timekeeper would fire the 1AM action twice, jumping forward, the timekeeper would fire the 3AM action, skipping any 2AM actions.
With that expectation set, how to implement that? I debated using the TimeZone and TimeZoneInfo classes to get to the DaylightTime objects and figure that all out. After scratching my head over that for a while without making headway, I decided to again take a bit of a brute force approach in that timekeeper worker’s “tick” loop:
Thread.Sleep(100);
timeNow = TimekeeperClock.Now();
if ( Math.Abs( (timeNow - lastTime).Ticks ) > TimeSpan.FromMinutes(1).Ticks)
{
//if the time changes more then one minute (plus or minus)
//after a 1/10 of a second sleep, then
//we've had some kind of major clock adjustment
//(potentially daylight savings time)
//so recalculate what the next entry would be
//have to back up the time a bit, in case it went from
//1:59:59am to 3:00:01am on dst change
_logger.InfoFormat("Clock drift detected from {0} to {1} in one cycle.",
lastTime, timeNow);
DateTime lookBackTime = timeNow.Add(TimeSpan.FromMinutes(-1));
_logger.InfoFormat("Recalculating next entry based on a time of {0}",
lookBackTime);
_next = _timetable.GetNextEntry(lookBackTime);
}
The worker knows the last time it got from the clock, so if the next time it gets is significantly different, something has happened. The worker assumes it is a daylight savings time adjustment and recalculates the next timetable entry from the new time. In order to actually catch the hour changing, that next entry is actually calculated from the new time minus a minute. A bit goofy, but workable, and easier then dealing with the .NET time zone classes.
Testing Time
Testing time based code is tough. I found it useful to just let the component run for days in our test environment and watch the log, but that’s not particularly efficient. So I wrote some automated tests for the timekeeper, for example, to test that the daylight savings time behavior was what I wanted:
[Test]
public void Should_Execute_One_AM_Task_Twice_On_Jump_Back_DST()
{
var tt = new Timekeeper();
int numberTimesExecuted = 0;
var incrementTimesExecuted = new Task
{
Execute = dt =>
{
numberTimesExecuted += 1;
return new TaskResult { Message = "Incremented" };
},
FriendlyName = "Increment Times Executed"
};
tt.ScheduleFor(new TimeSpan(1,0,0), incrementTimesExecuted);
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 0, 59, 59);
tt.Start();
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 0, 01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 59, 59);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 0,01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
Assert.AreEqual(2, numberTimesExecuted);
tt.Stop();
}
[Test]
public void Should_Execute_One_AM_And_3_AM_And_Not_2_AM_Task_On_Jump_Forward_DST()
{
var tt = new Timekeeper();
List<int> hoursExecuted = new List<int>();
var addHourToExecuted = new Task
{
Execute = dt =>
{
hoursExecuted.Add(dt.Hour);
return new TaskResult();
}
};
tt.ScheduleForEveryHour(addHourToExecuted);
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 0, 59, 59);
tt.Start();
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 0, 01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 59, 59);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 3, 0, 01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
Assert.Contains(1, hoursExecuted);
Assert.Contains(3, hoursExecuted);
Assert.False(hoursExecuted.Contains(2));
tt.Stop();
}
Pretty, these are certainly not. The lowlights include the numerous thread.sleep calls, which allow the timekeeper’s background worker to catch up and notice that the time has changed. The inline definition of tasks also ends up taking up a lot of screen real estate. Finally, the setting of method local variables from the action delegates seems clunky.
With all that said, these tests do make me feel more confident that the timekeeper will behave as expected, and furthermore express the expected behavior of the component to anybody else who reads them.
The reliance on using thread sleeps really bothers me; so far the only way around it that I could think of was to extract the main loop logic from worker into a pluggable algorithm, that is passed into the timekeeper as a dependency. Then I could test just that dependency in isolation. To put flesh on that, I’d have a property or constructor argument on the timekeeper that was an interface with a method that takes in the last known time, and a reference to the timetable, and then calculates what if anything to execute based on the current time.
Wrapping Up
This has been fun for me, and fixing up this component will be valuable to my current project in helping us reduce our day to day support load. I also got more experience with unit testing, and some experience using git to put the code up on github. Hopefully you learned something along the way as well. If you think this code would be useful to you (or part of it), check it out at http://github.com/phmiller/Timekeeper. And thanks for reading this far <g>
--- Relevant Links ---
The post wasn’t lengthy enough for you? Maybe try a dead-tree book, like Kent Beck’s XP Explained (2nd edition). Well worth your time:
http://www.amazon.com/Extreme-Programming-Explained-Embrace-Change/dp/0321278658/ref=dp_ob_title_bk
In the collection of interviews, Coders At Work, a common theme was the idea that when confronted with a difficult debugging problem, it can be better to just redo the troubled code rather then fix it.
This may seem like an invitation to put on your cape and do some hero-programming. This kind of creative destruction is only warranted when you are confident you can reproduce the functionality more coherently and with fewer bugs in less time than it would take to fully understand and tweak the existing code.
As I mentioned in a previous post, I’ve been working on a component to keep time and schedule tasks. The basic requirements of which are:
- Keep time in any time zone
- Provide a way to schedule tasks for every hour change, day change or arbitrary time
- Provide a mechanism to monitor the results of these tasks and/or any exceptions
The implementation I described in that post mostly worked, but would occasionally just fall over in our test environment. I couldn’t find the bug and looking through the code, I saw a lot of room for improvement. So after some creative destruction, I had a new and cleaner implementation:
public interface ITimekeeper
{
Action<Task, Exception> ExceptionHandler { get; set; }
Action<Task, TaskResult> CompletedHandler { get; set; }
void ScheduleFor(TimeSpan timeOfDay, Task task);
void ScheduleForEveryDay(Task task);
void ScheduleForEveryHour(Task task);
void Start();
void Stop();
}
public class Task
{
public Func<DateTime, TaskResult> Execute { get; set; }
public string FriendlyName { get; set; }
}
public class TaskResult
{
public bool Error { get; set; }
public string Message { get; set; }
}
Even just from a usage perspective this code reads much nicer to me:
var tk = new Timekeeper("Central Standard Time");
tk.ScheduleForEveryHour(task1);
tk.ScheduleForEveryHour(task2);
tk.ExceptionHandler +=
(task, ex) => System.Console.WriteLine(
"[Exception Handler] Exception for {0}: {1}",
task.FriendlyName, ex);
tk.CompletedHandler +=
(task, result) => System.Console.WriteLine(
"[Completed Handler] {0} completed with a message of {1}",
task.FriendlyName,result.Message);
tk.Start();
Under the hood, the changes were pervasive, and we’ll look at those changes in an upcoming post.
--- Relevant Links ---
Coders At Work was a fascinating read for the insights into how some of the giants of our field work. The homepage is also notable for an excellent salmon background:
Coders At Work Homepage
Can’t wait for the next post? Want to take a look at the code for yourself? View at your own risk:
Timekeeper on Github
For the time keeping component that I’ve mentioned before, one of the requirements was that it be able to run multiple actions or tasks at a given point in time. In case some of these tasks were particularly long running the idea was to run them in the background, rather than potentially block the main thread and miss a subsequent scheduled time. We also wanted to keep track of a task failing to execute properly and report back on that.
C# 3.0 provides some elegant and concise ways to do this. My exact implementation was pretty particular to our problem space, but for blogging’s sake, I abstracted out a basic and version of it for your convenience.
We’ll start from the top with usage:
class Program
{
static void Main(string[] args)
{
int i = 0;
List<Action> actionsToExecuteAsync = new List<Action>()
{
() => System.Console.WriteLine("executed action 1"),
() => System.Console.WriteLine("executed action 2"),
() => System.Console.WriteLine("exception time: {0}", 1/i),
() => System.Console.WriteLine("executed action 4")
};
AsyncUtil.ExecuteAsync(actionsToExecuteAsync, HandleException);
System.Console.ReadLine();
}
static void HandleException(Exception e)
{
System.Console.WriteLine("exception: {0}", e);
}
}
There is a heavy helping of syntactic sugar here as I create some anonymous delegates (Action’s) to execute and put them in a list using a collection initializer. Next up I call the ExecuteAsync method and wait to see the results. Which look something like this:
executed action 1
executed action 2
executed action 4
exception: System.DivideByZeroException: Attempted to divide by zero.
at AsyncWorker.Program.<>c__DisplayClass8.<Main>b__3() in C:\Users\Peter\src\
AsyncWorker\AsyncWorker\Program.cs:line 17
at AsyncWorker.AsyncUtil.<>c__DisplayClass3.<ExecuteAsync>b__0(Object state)
in C:\Users\Peter\src\AsyncWorker\AsyncWorker\AsyncUtil.cs:line 20
So we’ve got our tasks executing and an exception in one of them being handled by the HandleException method.
Looking behind the curtain:
public static class AsyncUtil
{
public static void ExecuteAsync(List<Action> actions, Action<Exception> exceptionHandler)
{
foreach (var action in actions)
{
var actionToExecute = action;
ThreadPool.QueueUserWorkItem(state =>
{
try
{
actionToExecute();
}
catch (Exception e)
{
exceptionHandler(e);
}
}, null);
}
}
}
Almost more whitespace and curly braces in there then code, but that won’t stop us. Again, with some sugary syntax, I’m throwing the action to a background thread through the call to ThreadPool.QueueUserWorkItem.
A gotcha was that I have to create the actionToExecute variable to ensure that all threads don’t just execute the last action in the list. Why? Because each thread would get a reference to “action” (from the foreach loop) and by the time they executed, “action” would be the final item in actions. Which is probably more confusing to you then it would be if you just took this code and tried it with and without the extra variable.
So take it for what it’s worth, but I found this snippet of code to be useful and a pleasant reminder of how expressive you can be in C#.
---Relevant Links---
If you care to learn anything about threading in .NET, read this whole threading guide by Joseph Albahari:
http://www.albahari.com/threading/
If you want to see what’s coming in .NET 4.0 that makes this even easier and more powerful:
http://www.codethinked.com/post/2010/01/25/NET-40-and-SystemThreadingTasks.aspx
Truly paying attention can be one of the hardest things to do. Even when you are doing something you want to do, say solving an interesting problem, your mind flits about. In a five minute interval you might answer an IM, check your email, check twitter and oh yeah, think about that problem you are working on.
There is a lot of literature out there around “flow” and the efficiency of focusing your attention on just one task. If you are working as an engineer or implementer on a project, common ways to help reduce these distractions are to put up a red flag (physically or with a do not disturb online status) or put on your headphones. You can also try to increase your focus by self imposing time boxes on your schedule, a la the Pomodoro Technique of 25 minute bursts of concentrated activity.
That’s at an individual level. To keep a software team focused, I’ve learned by counter example that the key is for the team to be highly focused on the smallest number of tasks you can get away with. Again, this is not easy to accomplish, but Agile methodologies have a lot to say on this.
From agile in general, comes the phrase “done done.” Meaning work on a task until the code works, is documented, tested, builds and deploys smoothly. This requires extra discipline on the part of the developer as making the code work is the fun part, whereas the rest just seems like extra effort. This also requires discipline on the part of the team lead to allow this “extra” work to be completed when the developer could go on to make something else work. The benefit is you don’t have to backtrack as much or have a stressful big bang at the end of a cycle to do all your integration, building, testing and documenting.
“Done done” talks about how to work on a task, which doesn’t help if you are overloaded with too many. Scrum addresses this problem through sprint planning, where tasks are given estimates and only X amount of work is allowed into a sprint. This should result in fewer tasks per developer, but not always. Again, discipline is the key as the scrum master and product owner have to engage in a give and take to keep scope manageable. If the product owner says you have to do everything this sprint, no matter what, you aren’t doing agile anymore, you’re just fire fighting.
Kanban or lean methodologies, deriving from production environments, focus on limiting the number of tasks going on at once. Every task goes through distinct stages and a new task cannot be added until another task exits the last stage. Progress is tracked visually on a kanban board. So there are not really sprints, so much as a continuous low flow of focused activity. I haven’t done kanban myself, but like scrum, it implies a give and take between the team and the other stakeholders to understand that doing less at once, does mean doing less for the project lifecycle.
So while Agile is becoming mainstream, and the concept of flow is almost passé at this point, I don’t get the sense that it is mainstream to follow through on their convergence of focusing more on fewer items at a time. This seems to be a general trend, where we all like the idea of being Agile, but think a light process = a free and easy process. After a rather unsuccessful attempt at Scrum with that concept in my mind, I can assure you it is not the case. Good process is difficult, but when did difficulty ever stop us before?
--Relevant Links--
The Pomodoro Technique and tomato timers:
http://www.pomodorotechnique.com/
If you like “Done done”, check out:
http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo
More on “done done”:
http://blog.phpdeveloper.org/?p=148
InfoQ’s stuff is pretty good, haven’t read this yet myself, but looks promising:
http://www.infoq.com/minibooks/kanban-scrum-minibook
Agile is now mainstream:
http://www.computerworld.in/articles/agile-software-development-now-mainstream
In a previous post, I mentioned working on a time keeping component. We used this component as the basis of a task scheduling application. Our tasks were represented by classes that implemented a common task interface:
public interface ICustomTask
{
CustomTaskResult Execute();
DateTime? ExecutionTime { get; }
TimeZoneInfo ExecutionTimeZone { get; }
string Description { get; }
}
public class CustomTaskResult
{
public bool WasSuccess { get; set; }
public string Message { get; set; }
}
So far, so good. Next challenge was how do we specify which of these tasks to run, and when? My initial thought was to put these details in an app.config or custom XML file. Task elements with the class name and time as attributes for example. Then additional elements for custom parameters that a task might need to initialize.
Fortunately enough, I work with some smart people, one of whom gently suggested that rather then create and manage my own custom config file, I instead leverage Castle Windsor (our dependency injection framework) and its configuration.
I’ll let the code do the talking; here’s a windsor section of an app.config for our task scheduler which will run two tasks at 9pm and the end of every day (the default if no time is specified) respectively:
<castle>
<components>
<component id="NightlyProcessing" lifestyle="transient"
service="Clarity.ICustomTask, Clarity.Task"
type="Clarity.Processing.NightlyTask, Clarity.Processing">
<parameters>
<timeZone>Eastern Standard Time</timeZone>
<processingTime>9:00 PM</processingTime>
</parameters>
<component id="RunSSISPackage" lifestyle="transient"
service="Clarity.ICustomTask, Clarity.Task"
type="Clarity.SSIS.PackageTask, Clarity.SSIS">
<parameters>
<description>Nightly Data Archiving</description>
<pathToPackage>c:\Data_Archive.dtsx</PathToPackage>
<pathToConfigFile>C:\DEV.dtsConfig</PathToConfigFile>
</parameters>
</component>
</components>
</castle>
The namespaces have been sanitized, but the point is that you can specify parameter elements that map to constructor arguments. So for example, the NightlyTask class has a constructor with a timeZone and processingTime argument that get mapped to the ExecutionTime and ExecutionTimeZone properties on the custom task.
--- Relevant Links ---
To take this to the next level (or if you just don’t like the angle bracket tax):
http://sheitm.blogspot.com/2009/05/fluent-castle-windsor-and-configured.html
As I continue to explore agile techniques at work, I had the opportunity to do some Test Driven Development, specifically Test First Development. The problem at hand was that our system is composed of several applications, all of which have various actions they have to take every hour or every day at a specific time. As the applications were developed, each application grew out its own time keeping infrastructure.
If you haven’t had the pleasure yet, working with time, dates and durations in .NET is not as easy as you’d like. It has gotten a lot better since framework 1.0, but it is still a great place for bugs to hatch. And not surprisingly, while each application was able to implement a system that worked most of the time, the code base was sprawling and behavior was unpredictable.
With the hindsight that comes after pushing a system to production, my team became convinced that we needed more then working most of the time, we wanted to prove that it worked all the time. Also, having abused the coding principle of (D)on’t (R)epeat (Y)ourself with multiple competing implementations we wanted to settle on one common implementation.
The approach we took was to throw out the old code, and first define an interface for our time keeper. Now it looks something like this:
public interface ITimekeeper
{
event EventHandler HourChanged;
event EventHandler DayChanged;
void Initialize(TimeZoneInfo timeZone, DateTime dayChangeTime);
void RegisterCustomEvent(DateTime time, TimeZoneInfo timeZone,
CustomEventCallback callback, object state);
void Start();
void Stop();
...
IClock Clock { get; set; }
}
Now is an important qualifier. We didn’t start out with the right interface, we took an educated guess and then started writing some tests. Importantly, and somewhat surprisingly to me, we got a lot of value out of just trying to write the tests themselves. It was a feedback loop as we’d try to write a test, find the interface clunky, refine the interface , write the test again and then move on.
We deliberately ignored the implementation at first and defined a reasonably wide set of tests that covered the basic functionality of our component. Then we subjected the interface to its sharpest critique by handing these tests off to another developer on the team and having him implement the code to make them pass.
In short, TDD drove our component level design to be more elegant, provided a natural way to break out tasks amongst the team, and of course provided us some confidence that the implementation would work.
I think I am taking a break from the bold prediction business, but this experience definitely has lead me to reconsider some of the points made over a year ago in this post of questionable origin:
http://blogs.claritycon.com/blogs/peter_miller/archive/2008/10/06/why-test-driven-development-is-a-hard-sell.aspx
<--- A questionable dude…
--- Related Links ---
Hat tip to Philipp Sumi, who implemented a nice scheduling library of his own (which I found after I finished my efforts) and provided me some good feedback and insight on TDD.
http://www.hardcodet.net/2010/01/lightweight-task-slash-job-scheduling-with-silverlight-support
Also, for the eagle eyed amongst you, yes we do have an interface for the system clock, IClock. See Ayende Rahien’s post on why this seeming non sequitur can be useful:
http://ayende.com/Blog/archive/2008/07/07/Dealing-with-time-in-tests.aspx
More Posts
Next page »