February 2010 - Posts
In my last post I discussed how some creative thinking on my part led me to redo large chunks of a task scheduling library. As promised, here are some of the major changes and what I learned from them.
Lush Verbiage
In the refactor, I expanded upon the concept of a timekeeper and extended the metaphor to the rest of they component. So a timekeeper, keeps time, or in this case, a timetable. A timetable is composed of entries, a combination of time slot and actions scheduled for that time. A timekeeper has a worker which “works” the timetable using a simple procedure. Find the next entry in the timetable, keep track of the passage of time until it is past that entry’s time, execute the actions for that entry, find the next entry, continue on.
The metaphor is strained at times, but by carefully choosing descriptive names for classes from the physical world, there was less mental friction when trying to implement or explain the component.
Design Simplicity
Dealing with time based code in .NET is harder than it should be. Or more accurately, harder then I’d want it to be. Even the conceptually simple idea of scheduling a task to happen every hour is fraught with peril. I started with a Timer that went off at the next hour and repeated with a period of 60 minutes. This mostly works, but that timer will sometimes fire before the hour is up. Only a fraction of a second before the hour, but that is time enough to wreck havoc on logic that looks for the next event chronologically or on other components that depend on the hourly events occurring on or after the hour. If not havoc, at least make it messier.
Ultimately, I could have gotten past that challenge, but I decided to simplify the code by making the main “tick” loop of the timekeeper (what looks for timetable entries that need to be executed) a brute force style thread.sleep, check, thread.sleep, check, loop. The idea of this approach grates on me as it is making the program spin more then it needs to, but a sleep of 100 ms was enough to not noticeably affect CPU utilization. In addition, this made it very easy to add support for a clean stop of the timekeeper worker:
internal class TimekeeperWorker : BackgroundWorker
{...
protected override void OnDoWork(DoWorkEventArgs e)
{...
while (!CancellationPending)
{
Thread.Sleep(100);
… “tick” loop logic
}
//cancellation pending, so exit out
e.Cancel = true;
Even after this latest refactor, there are more places I could simplify this code. However, just like optimization, there is an effort associated with simplification that isn’t always needed. At least not yet.
Daylight Savings Time
This is kind of a subset of Design Simplicity, but one of the concerns I had with this component was how it would handle Daylight Savings Time. When the clock jumps from 1:59AM to 1AM or 3AM for example. After some discussion with the team, the behavior we decided on was that in the case of jumping back, the timekeeper would fire the 1AM action twice, jumping forward, the timekeeper would fire the 3AM action, skipping any 2AM actions.
With that expectation set, how to implement that? I debated using the TimeZone and TimeZoneInfo classes to get to the DaylightTime objects and figure that all out. After scratching my head over that for a while without making headway, I decided to again take a bit of a brute force approach in that timekeeper worker’s “tick” loop:
Thread.Sleep(100);
timeNow = TimekeeperClock.Now();
if ( Math.Abs( (timeNow - lastTime).Ticks ) > TimeSpan.FromMinutes(1).Ticks)
{
//if the time changes more then one minute (plus or minus)
//after a 1/10 of a second sleep, then
//we've had some kind of major clock adjustment
//(potentially daylight savings time)
//so recalculate what the next entry would be
//have to back up the time a bit, in case it went from
//1:59:59am to 3:00:01am on dst change
_logger.InfoFormat("Clock drift detected from {0} to {1} in one cycle.",
lastTime, timeNow);
DateTime lookBackTime = timeNow.Add(TimeSpan.FromMinutes(-1));
_logger.InfoFormat("Recalculating next entry based on a time of {0}",
lookBackTime);
_next = _timetable.GetNextEntry(lookBackTime);
}
The worker knows the last time it got from the clock, so if the next time it gets is significantly different, something has happened. The worker assumes it is a daylight savings time adjustment and recalculates the next timetable entry from the new time. In order to actually catch the hour changing, that next entry is actually calculated from the new time minus a minute. A bit goofy, but workable, and easier then dealing with the .NET time zone classes.
Testing Time
Testing time based code is tough. I found it useful to just let the component run for days in our test environment and watch the log, but that’s not particularly efficient. So I wrote some automated tests for the timekeeper, for example, to test that the daylight savings time behavior was what I wanted:
[Test]
public void Should_Execute_One_AM_Task_Twice_On_Jump_Back_DST()
{
var tt = new Timekeeper();
int numberTimesExecuted = 0;
var incrementTimesExecuted = new Task
{
Execute = dt =>
{
numberTimesExecuted += 1;
return new TaskResult { Message = "Incremented" };
},
FriendlyName = "Increment Times Executed"
};
tt.ScheduleFor(new TimeSpan(1,0,0), incrementTimesExecuted);
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 0, 59, 59);
tt.Start();
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 0, 01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 59, 59);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 0,01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
Assert.AreEqual(2, numberTimesExecuted);
tt.Stop();
}
[Test]
public void Should_Execute_One_AM_And_3_AM_And_Not_2_AM_Task_On_Jump_Forward_DST()
{
var tt = new Timekeeper();
List<int> hoursExecuted = new List<int>();
var addHourToExecuted = new Task
{
Execute = dt =>
{
hoursExecuted.Add(dt.Hour);
return new TaskResult();
}
};
tt.ScheduleForEveryHour(addHourToExecuted);
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 0, 59, 59);
tt.Start();
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 0, 01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 1, 59, 59);
System.Threading.Thread.Sleep(200);//let timekeeper tick
TimekeeperClock.Now = () => new DateTime(2010, 3, 14, 3, 0, 01);
System.Threading.Thread.Sleep(200);//let timekeeper tick
Assert.Contains(1, hoursExecuted);
Assert.Contains(3, hoursExecuted);
Assert.False(hoursExecuted.Contains(2));
tt.Stop();
}
Pretty, these are certainly not. The lowlights include the numerous thread.sleep calls, which allow the timekeeper’s background worker to catch up and notice that the time has changed. The inline definition of tasks also ends up taking up a lot of screen real estate. Finally, the setting of method local variables from the action delegates seems clunky.
With all that said, these tests do make me feel more confident that the timekeeper will behave as expected, and furthermore express the expected behavior of the component to anybody else who reads them.
The reliance on using thread sleeps really bothers me; so far the only way around it that I could think of was to extract the main loop logic from worker into a pluggable algorithm, that is passed into the timekeeper as a dependency. Then I could test just that dependency in isolation. To put flesh on that, I’d have a property or constructor argument on the timekeeper that was an interface with a method that takes in the last known time, and a reference to the timetable, and then calculates what if anything to execute based on the current time.
Wrapping Up
This has been fun for me, and fixing up this component will be valuable to my current project in helping us reduce our day to day support load. I also got more experience with unit testing, and some experience using git to put the code up on github. Hopefully you learned something along the way as well. If you think this code would be useful to you (or part of it), check it out at http://github.com/phmiller/Timekeeper. And thanks for reading this far <g>
--- Relevant Links ---
The post wasn’t lengthy enough for you? Maybe try a dead-tree book, like Kent Beck’s XP Explained (2nd edition). Well worth your time:
http://www.amazon.com/Extreme-Programming-Explained-Embrace-Change/dp/0321278658/ref=dp_ob_title_bk
In the collection of interviews, Coders At Work, a common theme was the idea that when confronted with a difficult debugging problem, it can be better to just redo the troubled code rather then fix it.
This may seem like an invitation to put on your cape and do some hero-programming. This kind of creative destruction is only warranted when you are confident you can reproduce the functionality more coherently and with fewer bugs in less time than it would take to fully understand and tweak the existing code.
As I mentioned in a previous post, I’ve been working on a component to keep time and schedule tasks. The basic requirements of which are:
- Keep time in any time zone
- Provide a way to schedule tasks for every hour change, day change or arbitrary time
- Provide a mechanism to monitor the results of these tasks and/or any exceptions
The implementation I described in that post mostly worked, but would occasionally just fall over in our test environment. I couldn’t find the bug and looking through the code, I saw a lot of room for improvement. So after some creative destruction, I had a new and cleaner implementation:
public interface ITimekeeper
{
Action<Task, Exception> ExceptionHandler { get; set; }
Action<Task, TaskResult> CompletedHandler { get; set; }
void ScheduleFor(TimeSpan timeOfDay, Task task);
void ScheduleForEveryDay(Task task);
void ScheduleForEveryHour(Task task);
void Start();
void Stop();
}
public class Task
{
public Func<DateTime, TaskResult> Execute { get; set; }
public string FriendlyName { get; set; }
}
public class TaskResult
{
public bool Error { get; set; }
public string Message { get; set; }
}
Even just from a usage perspective this code reads much nicer to me:
var tk = new Timekeeper("Central Standard Time");
tk.ScheduleForEveryHour(task1);
tk.ScheduleForEveryHour(task2);
tk.ExceptionHandler +=
(task, ex) => System.Console.WriteLine(
"[Exception Handler] Exception for {0}: {1}",
task.FriendlyName, ex);
tk.CompletedHandler +=
(task, result) => System.Console.WriteLine(
"[Completed Handler] {0} completed with a message of {1}",
task.FriendlyName,result.Message);
tk.Start();
Under the hood, the changes were pervasive, and we’ll look at those changes in an upcoming post.
--- Relevant Links ---
Coders At Work was a fascinating read for the insights into how some of the giants of our field work. The homepage is also notable for an excellent salmon background:
Coders At Work Homepage
Can’t wait for the next post? Want to take a look at the code for yourself? View at your own risk:
Timekeeper on Github
For the time keeping component that I’ve mentioned before, one of the requirements was that it be able to run multiple actions or tasks at a given point in time. In case some of these tasks were particularly long running the idea was to run them in the background, rather than potentially block the main thread and miss a subsequent scheduled time. We also wanted to keep track of a task failing to execute properly and report back on that.
C# 3.0 provides some elegant and concise ways to do this. My exact implementation was pretty particular to our problem space, but for blogging’s sake, I abstracted out a basic and version of it for your convenience.
We’ll start from the top with usage:
class Program
{
static void Main(string[] args)
{
int i = 0;
List<Action> actionsToExecuteAsync = new List<Action>()
{
() => System.Console.WriteLine("executed action 1"),
() => System.Console.WriteLine("executed action 2"),
() => System.Console.WriteLine("exception time: {0}", 1/i),
() => System.Console.WriteLine("executed action 4")
};
AsyncUtil.ExecuteAsync(actionsToExecuteAsync, HandleException);
System.Console.ReadLine();
}
static void HandleException(Exception e)
{
System.Console.WriteLine("exception: {0}", e);
}
}
There is a heavy helping of syntactic sugar here as I create some anonymous delegates (Action’s) to execute and put them in a list using a collection initializer. Next up I call the ExecuteAsync method and wait to see the results. Which look something like this:
executed action 1
executed action 2
executed action 4
exception: System.DivideByZeroException: Attempted to divide by zero.
at AsyncWorker.Program.<>c__DisplayClass8.<Main>b__3() in C:\Users\Peter\src\
AsyncWorker\AsyncWorker\Program.cs:line 17
at AsyncWorker.AsyncUtil.<>c__DisplayClass3.<ExecuteAsync>b__0(Object state)
in C:\Users\Peter\src\AsyncWorker\AsyncWorker\AsyncUtil.cs:line 20
So we’ve got our tasks executing and an exception in one of them being handled by the HandleException method.
Looking behind the curtain:
public static class AsyncUtil
{
public static void ExecuteAsync(List<Action> actions, Action<Exception> exceptionHandler)
{
foreach (var action in actions)
{
var actionToExecute = action;
ThreadPool.QueueUserWorkItem(state =>
{
try
{
actionToExecute();
}
catch (Exception e)
{
exceptionHandler(e);
}
}, null);
}
}
}
Almost more whitespace and curly braces in there then code, but that won’t stop us. Again, with some sugary syntax, I’m throwing the action to a background thread through the call to ThreadPool.QueueUserWorkItem.
A gotcha was that I have to create the actionToExecute variable to ensure that all threads don’t just execute the last action in the list. Why? Because each thread would get a reference to “action” (from the foreach loop) and by the time they executed, “action” would be the final item in actions. Which is probably more confusing to you then it would be if you just took this code and tried it with and without the extra variable.
So take it for what it’s worth, but I found this snippet of code to be useful and a pleasant reminder of how expressive you can be in C#.
---Relevant Links---
If you care to learn anything about threading in .NET, read this whole threading guide by Joseph Albahari:
http://www.albahari.com/threading/
If you want to see what’s coming in .NET 4.0 that makes this even easier and more powerful:
http://www.codethinked.com/post/2010/01/25/NET-40-and-SystemThreadingTasks.aspx
Truly paying attention can be one of the hardest things to do. Even when you are doing something you want to do, say solving an interesting problem, your mind flits about. In a five minute interval you might answer an IM, check your email, check twitter and oh yeah, think about that problem you are working on.
There is a lot of literature out there around “flow” and the efficiency of focusing your attention on just one task. If you are working as an engineer or implementer on a project, common ways to help reduce these distractions are to put up a red flag (physically or with a do not disturb online status) or put on your headphones. You can also try to increase your focus by self imposing time boxes on your schedule, a la the Pomodoro Technique of 25 minute bursts of concentrated activity.
That’s at an individual level. To keep a software team focused, I’ve learned by counter example that the key is for the team to be highly focused on the smallest number of tasks you can get away with. Again, this is not easy to accomplish, but Agile methodologies have a lot to say on this.
From agile in general, comes the phrase “done done.” Meaning work on a task until the code works, is documented, tested, builds and deploys smoothly. This requires extra discipline on the part of the developer as making the code work is the fun part, whereas the rest just seems like extra effort. This also requires discipline on the part of the team lead to allow this “extra” work to be completed when the developer could go on to make something else work. The benefit is you don’t have to backtrack as much or have a stressful big bang at the end of a cycle to do all your integration, building, testing and documenting.
“Done done” talks about how to work on a task, which doesn’t help if you are overloaded with too many. Scrum addresses this problem through sprint planning, where tasks are given estimates and only X amount of work is allowed into a sprint. This should result in fewer tasks per developer, but not always. Again, discipline is the key as the scrum master and product owner have to engage in a give and take to keep scope manageable. If the product owner says you have to do everything this sprint, no matter what, you aren’t doing agile anymore, you’re just fire fighting.
Kanban or lean methodologies, deriving from production environments, focus on limiting the number of tasks going on at once. Every task goes through distinct stages and a new task cannot be added until another task exits the last stage. Progress is tracked visually on a kanban board. So there are not really sprints, so much as a continuous low flow of focused activity. I haven’t done kanban myself, but like scrum, it implies a give and take between the team and the other stakeholders to understand that doing less at once, does mean doing less for the project lifecycle.
So while Agile is becoming mainstream, and the concept of flow is almost passé at this point, I don’t get the sense that it is mainstream to follow through on their convergence of focusing more on fewer items at a time. This seems to be a general trend, where we all like the idea of being Agile, but think a light process = a free and easy process. After a rather unsuccessful attempt at Scrum with that concept in my mind, I can assure you it is not the case. Good process is difficult, but when did difficulty ever stop us before?
--Relevant Links--
The Pomodoro Technique and tomato timers:
http://www.pomodorotechnique.com/
If you like “Done done”, check out:
http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo
More on “done done”:
http://blog.phpdeveloper.org/?p=148
InfoQ’s stuff is pretty good, haven’t read this yet myself, but looks promising:
http://www.infoq.com/minibooks/kanban-scrum-minibook
Agile is now mainstream:
http://www.computerworld.in/articles/agile-software-development-now-mainstream