PAGES

26

Mar 13

Design for Software the book – available now!



Today is a big day, I’m pleased to announce that “Design for Software: A Playbook for Developers” is finally available to the public! It was a long journey taking nearly 14 months to complete, but it’s here and I couldn’t be more excited. Check it out at designforsoftware.com

A little background

Over the past ten years I’ve worked closely with designers and developers in the emerging technology industry. During that time I’ve realized that designing software isn’t like any other type of design. Designing software is unique. It combines many fields of design into a single medium – Motion, typography, color theory, interaction design, information architecture, ethnography, and more. And for many reasons it deserves its own design process.

That is the premiss for this book – a process for designing software. My goal is to make this unique discipline more accessible for those that don’t have expertise in UI/UX design. Throughout the book I attempt to break down design theory concepts and present them in a way that makes sense for application design.

This book is written for non-designers and tech-savvy artists looking to create best in class software. I believe anyone can design great software, regardless of background. It doesn’t matter if you have a design degree (although it might help) or a history of being a hard core developer. Many design principles can be broken down into frameworks that help take the guesswork out of an otherwise abstract field.

By the end of the book you’ll have effectively learned all my design “secrets”. I hope that the design process I’ve written about is as useful to you as it has been for me.

Book Extras

It’s pretty difficult to write about interactive design in a static medium..go figure. During the writing process there were many times that static images didn’t do the content justice, especially when trying to describe motion. So I created a bunch of complementary web content.

You don’t have to buy the book to reap the benefits of the extras, but you should. They will make more sense…and it will make my dog happy (not my actual dog).

Where can I buy it?

The publisher (Wiley) has done an amazing job making the book available from a variety of retailers and in a ton formats. Of course I would encourage you to pick up the physical book, but the digital versions are good too. As of now, you can purchase the book from the following retailers:

I’m looking forward to seeing it out in the wild!
Best,
Erik

5 comments , permalink


16

Dec 12

Freebie: Microsoft Surface RT PSD and Ai Template



We produce a ton of artwork for Windows 8 and when we present ideas to our clients I like the artwork to look as good as possible.

One way we do this is by placing the designs within a device frame. This gives the artwork some context, and a more premium positioning.

Mocking up artwork in a device frame is pretty commonplace for iOS and Android design, so I figured the Win8 community would benefit from these. Unfortunately, many of the MS Surface templates floating around the interwebs appear to be slightly inaccurate. They look as if a Samsung Series 7 and an iPad had an awkward kid. So I was compelled to have a legitimate go at making an accurate template. I’d like to think mine are at least 98.9% accurate.

Get the Goods

If you have Photoshop or Illustrator installed these should come in handy. And if you don’t…you can hand them off to your neighborhood friendly designer and I’m sure they’ll be pleased as punch…

Best

Erik Klimczak | eklimczak@claritycon.com | twitter.com/eklimcz

2 comments , permalink


12

Dec 12

Rally Keeper – Kinect based ping pong visualizer



Following up on my last post, I mentioned that we are creating a fun installation that involves tracking small fast-moving objects. So without further ado I give you Rally Keeper:

Overview

Before I get into how it works, I want to mention some of the inspiration for the project. A while back I saw this cool product concept for an interactive ping pong table. Inspired by the idea, the team and I spent some time pondering how we could take this “future” concept and turn it now concept.

Researching similar projects we examined a variety of approaches including audio triggers, triangulating sound waves, and color tracking. Then the ‘aHa’ moment came – we only needed to track the ball at a certain depth to capture it’s coordinates. Knowing that we could easily track depth with the Kinect, we quickly put together a prototype.

The Approach

The diagram above shows our approach..it’s actually quite simple:

  • Position the Kinect far enough away to capture the whole table
  • Slice Kinect’s depth image creating an “invisible” layer directly above the table surface
  • When the ball enters the tracking layer, use OpenCV to grab it’s coordinates
  • Store and paint the coordinates back onto the UI layer

Most of the tech details of how to do blob tracking can be found in my last post.

The UI Concept

When we overcame the major tech hurdles, it was time to give this thing a face-lift. The aesthetics were mostly inspired by the original concept. And the UI was implemented using WPF.

You can see more of the artwork here

Aside from a being purely a cool visualization, we also wanted to add some utility. After a typical battle session and a brutal defeat, we often exclaim “I can’t believe you beat me! How’d you do it!?” Convinced, that if we could see the ball patterns we could calculate a better strategy for the next bout. Additionally, analyzing data is cool. With the ball data we can calculate percentages, make predications, and see trends in our games.

What’s Next?

After messing around with this for a week we came up with dozens of additional ideas we’d love to explore…time permitting. Some of the highlights include:

  • Authenticating purely with Kinect gestures
  • Leverage the Kinect’s microphone array for automated scoring
  • Pushing game results up to a website for comparison and analysis
  • Use projection mapping to draw the visualization onto the table in real-time.
  • Use multiple Kinect cameras for even better tracking

Anybody else out there with any ideas? I’d love to hear them…

Best

Erik Klimczak | eklimczak@claritycon.com | twitter.com/eklimcz

Comments Off , permalink


22

Nov 12

Blob Tracking: Kinect + OpenCV + WPF



With the latest SDK release, the Kinect platform has become pretty robust – skeleton tracking, gesture support, and access to the raw infrared stream all come “out-of-the-box”. Which is great, because if you’ve ever used any of the opensource equivalents, things can get pretty hairy.

Hands and skeletons are cool and all, but unfortunately there isn’t an *easy* way to do blob or generic object detection.. Right now, we’re working on a fun project and ultra fast blob detection is one of the major tech hurdles we’re working through.

After googlin’ around for blob-tracking with Kinect I came up empty handed. So I decided to look into OpenCV for doing blob detection. Many curse words later, I finally got things working the way I wanted. Along the way I noticed a handful of forums with people looking to do similar object tracking with Kinect, so I figured I would share how I cobbled things together.

Demo Overview & Source Code

The demo below is showing a couple of key things:

  • Consuming the raw Depth image from the Kinect sdk
  • Slicing the Depth image to a particular range
  • Using OpenCV to perform object/blob tracking
  • Rendering OpenCV output with WPF

Before We Get Started

Kinect SDK

Be sure to download the latest Kinect SDK. There are also a bunch of handy samples that come with the toolkit. You can get it here

EMGU (.Net Wrapper for OpencV)

Emgu is a .net wrapper for OpenCV. It is very complete and does a good job of providing .net hooks into the common OpenCV functions. Unfortunately, getting it to work with WPF is a little tricky (More below ). Also, I wasn’t able to get the lastest (2.4) version of the code to work with .net 4.5 and Visual Studio 2012, but the 2.3 version works well. Get it here

Kiblob (optional)

This codeplex project is similar, but it’s using another computer vision library called AForge and only compiles with a beta version of the Kinect sdk. Regardless, there are some handy tid-bits of code in there. You won’t need it to build my sample but its a good resource to look through.

Setup

Ok, now that we have the dependencies installed, its time to start setting up our WPF project. There are similar instructions on the Emgu site, but I found them a little confusing, so I’ll give you the cliffnotes version here.

Add References

After you’ve installed Emgu you need to add some .dlls to your project. This is the standard Solution->right-click->Add Reference and then browse to the Emgu installation path to add the following .dlls

  • Emgu.CV.dl
  • Emgu.UI.dll
  • Emgu.Util.dll

The next step is a little odd. You need to add 3 .dlls to the root of your project. Right-click the Project fill and select “add existing item”, then navigate to the Emgu installation folder and add the following:

  • opencv_core231.dll
  • opencv_highgui231.dll
  • opencv_imgproc231.dll

The last thing, and this is important, is to set the build action to “copy always”. This will dump those dlls into your build folder when you build the project. When you’ve got all that done you’re solution should look something like this:

Getting OpenCV to Play Nice with WPF

Ok, now that we have all the references in place we need to make openCV (Emgu) play nice with WPF. The biggest thing to note is that OpenCV uses it’s own type of image objects (IImage) which don’t work by default with the native WPF image controls.

Luckily, there is a little helper class that comes with Emgu that will convert the IIMage to a BimapSource, which works nicely with the WPF image control. By adding this helper method, we now have a convenient way to plug an OpenCV image into a WPF image. See the helper class here

The Fun Part

Now we can get to the good stuff. When you do blob tracking it is very important to have a “clean” grayscale image that has a high contrast between the object you are tracking and the background. This is one of the best parts about Kinect – it provides a grayscale depth image that we can analyze and extract out the regions of interest relatively easily. Let’s looking at the key pieces of the solution.

Depth Image Slicing

Like I mentioned above, when you are doing generic object detection you want to have high contrast between your object and the background. In other words, you want to isolate the object the best you can and ignore everything else. By doing so, you can ensure the vision library, like OpenCV, does a reliable job of finding the blobs.

The beauty of using the Kinect for blob tracking is that it gives us a depth image that contains all of the information we need to “slice” the image into the parts that we’re interested in. While I was trying to figure out the best way to slice the depth image I came upon a Kinect depth data video that really opened my eyes to working directly with the depth byte stream coming from Kinect. Check it out here

By simply ignoring (coloring pixels black) data that is out of the defined depth range, I was able to get a nice high-contrast image of the objects the depth camera was picking up. You can imagine this being something like an invisible plane that excludes everything in front and in back of it, which coincidentally creates a nice interaction space where we can track objects. You can see the slicing code here. This is the output:

OpenCV Contour Finding

The last thing to do is to get the blob tracking working. At this point we have a very clean image that we can feed into OpenCV’s “Find Contours” method. The FindContours method analyzes a grayscale image and returns the number of blobs as well as their bounding rectangles. It was a little tricky getting this to work in WPF, you can see the main piece of the code here.

Once I was able to figure out some of the cryptic tracking mechanisms in OpenCV everything came together. The cool thing about OpenCV is that not only can you get blobs, but you can blobs within blobs..in a tree. Crazy..I know. And lastly, we use our handy helper method from above (ToBitmapSource(OpenCVImg)) and plug it into the Image control in the main window xaml. Which ends up looking something like this:

Wrap up

The last thing I added were a few sliders to the UI to adjust the slicing parameters and the max/min Blob size. The image below shows the finished application.

All said and done, I am happy with the performance and excited about the opportunities to use this slew of technologies in some upcoming installation work.

Ok, so what?

I know what you’re thinking…so you have some blobs, big deal. And you might be wondering what else you can do with blobs. Well, let me tell you. By using generic blob detection you can not only detect when objects are present, you can also track their size, shape and rotation. For example, its possible to track an open hand transitioning to a closed fist for a grab gesture, or a user holding a wand or prop. Or in our case, we want to very efficiently capture the position of small, fast moving objects that penetrate an “invisible” IR plane. There are a ton of uses for blob detection including surveillance, card counting, and even real-time projection mapping.

What’s Next

While I’m pretty happy with the performance of this solution, I’m curious if it would be better/faster if I rewrote everything with the C++ flavor of the Kinect sdk and coupled it with DirectWrite or XAML to render the UI. Anybody out there do anything like this yet? If so, shoot me a tweet or email.

We’re currently rolling this stuff into a fun side project that I’ll post about shortly…stay tuned.

Best

Erik Klimczak | eklimczak@claritycon.com | twitter.com/eklimcz

1 comment , permalink


2

Nov 12

Win8 and Flowing Authentication Credentials



The Problem

A couple months back I was developing a demo Windows 8 application (HTML/JS) and needed to authenticate against a web service on the corporate network. The credentials needed are the same as the users domain account and I hoped to make use of some of the capabilities of Windows 8 applications such as Enterprise Authentication. As a sometimes hacky developer, I tried enabling this and poking around various objects and interfaces looking for username and password. I had strong feelings that it would not be this easy and I should file a bug report if so. I did eventually find how to get the user’s principal name (domain name / username, close enough…) by using Windows.System.UserProfile.UserInformation.getPrincipalNameAsync(). Poking around the UserInformation object I found ways to find generic information about the user with nothing close to the type of authentication object I believed the web service could understand and make use of. As a result the project had to settle for an authentication prompt on startup.

Defeated, I gave in and posted a question on Stackoverflow lamenting the fact that I knew my user was logged in, but I was still prompted for user credentials…

Light at the end of the Authentication Tunnel

Just this week I finally received a potential answer to the situation. I should try adding the site to the Local intranet under internet options. It wasn’t until I started thinking about the implications of what I wanted the application to do that it started to make a bit of sense as to why this additional step was deemed necessary. I was telling the application: hey I know my user is logged in and I want his authentication credentials sent here because he/she needs to use this web service. Nowhere in that statement do I instill trust or security in what my goal is. I gave it a try and with much success the authentication prompt is no longer needed.

Along with this change I also had the following application capabilities set in the app manifest:

  • Enterprise Authentication
  • Internet (Client & Server)
  • Private Networks (Client & Server)

From the Dev’s Corner

Thinking about this from the developer point of view I should be able to send their credentials with permission and as of yet I’m not sure what I can do inside of the application to raise a UAC kind of prompt informing the user that the application would like to user their credentials to authenticate. The current manual solution does two things: it requires user/installer intervention to setup and it hides the fact that a logged in user’s credentials are being used. I’ll have to think about this problem a bit to see if there is a more elegant solution.

Comments Off , permalink


31

Oct 12

The SimpleMembershipProvider Plays Nice with EF Code First and Azure



We’ve been looking into some of the plumbing for a new project.  One part of the solution is an ASP MVC site hosted in Azure.  We’re plan to have our own membership database so we started looking at the Universal Providers which we’ve used successfully in the past with Azure hosted MVC solutions.  While some of the MVC templates are wired up to the Universal Providers, the MVC 4 Internet Application is configured to use the SimpleMembershipProvider

I thought I’d use this post to list some thoughts, notes and related resources about it. Based on my POC I found it plays nicely with CodeFirst and Azure and will walk through some changes I made from the default template’s code.

First I wanted to understand if this is the evolution of the ASP providers?  Beyond the fact that it is now the default in the template, I think this are article provides some background: SimpleMembership, Membership Providers, Universal Providers and the new ASP.NET 4.5 Web Forms and ASP.NET MVC 4 templates

Next I wanted to see how its data model interacted with my data model.  This is where the “Simple” comes in to play.  What’s nice  is that you can just create your own user profile table and “register” it with the SimpleMembershipProvider through the WebSecurity class.  Here you see my very simple data context and User model and configuration class:

public class MVCApp3Context : DbContext
{
    public MVCApp3Context()
        : base(ConnectionStringHelper.GetConnectionString())
    {
    }

    public DbSet<Customer> Customers { get; set; }
    public DbSet<User> Users { get; set; }

    protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);
        modelBuilder.Configurations.Add(new UserConfiguration());
    }
}

public class User
{
    public int UserID { get; set; }
    public string UserName { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

public class UserConfiguration : EntityTypeConfiguration<User>
{
    public UserConfiguration()
    {
        ToTable("User");
        Property(u => u.ID).HasColumnName("UserID").HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity);
    }
}

What’s nice is that it is using my model for user info and its own table for password info.  When a user is set up via the API you can just pass an anonymous object with the additional properties (i.e. the First and Last name from your registration model).  That means I can treat the calls for login as a black box, but I don’t have to do some hack to pull together the membership table and my own user table data to show the list of users in an Admin page.  This might include name, email, any other attribute I track about a user.  There is no Application table anymore, so tracking the tenant or app or other segregation of users would be another example.  One other note here is that UserID has to be an int.  I don’t personally like that because in general I use Guids for IDs, but it isn’t the end of the world.

To start getting everything hooked up, in Global.asax I set the initializer for my context and initialize it. 

protected void Application_Start()
{
    Database.SetInitializer<MVCApp3Context>(new MVCApp3ContextInitializer());
    using (MVCApp3Context db = new MVCApp3Context())
    {
        db.Database.Initialize(false);
    }
    ...
}

Then I restructured the code that default template generated, getting rid of the InitializeSimpleMembershipAttribute and instead make the call to WebSecurity to use my User table in the Seed of my initializer.  This will create the Membership tables after my model tables and also allows me to seed some users.  This provides a HUGE efficiency for a dev team and is why I love Code First.  I don’t like the prod or test environment tables being created or seeded like this, but during dev iterations this is great.

public class MVCApp3ContextInitializer : DropCreateDatabaseIfModelChanges<MVCApp3Context>
{
    protected override void Seed(MVCApp3Context context)
    {
        base.Seed(context);
        SeedMembership();
    }

    private void SeedMembership()
    {
        // Use the overload that allows for connection string to be passed in
        WebSecurity.InitializeDatabaseConnection(ConnectionStringHelper.GetConnectionString(), "System.Data.SqlClient", "User", "UserID", "UserName", true);
        WebSecurity.CreateUserAndAccount("jdoe99", "password99", new { FirstName = "John", LastName = "Doe" });
    }
}

Finally I was happy that with this provider that I didn’t have to rely of web.config for connection strings.  Unless something has changed, the Universal Providers expect a connection string name in code and then look it up based on what is defined in the connectionstrings section of your config.  This was annoying when using an Azure web role, because it requires some start up logic to swap out the connection string in the web.config that is in your package as you move environments instead of just keeping it in the Azure config file (less of a pain if you’re using the Azure Web Sites which are still only in Preview).  As you can see in the code above there is an overload that allows me to have my connection string come from wherever I want.

After making these changes I was able to deploy locally or to Azure and to local SQL or SQL Azure.

The only other thing that jumped out at me in my testing was that the PasswordSalt column was suspiciously empty in the db:

This explained how passwords are indeed being salted for hashing and I felt better: The SimpleMembershipProvider, Secure Passwords and the Crypto Helper

Comments Off , permalink


23

Oct 12

The Tech Behind a Contre Jour Snot



While the award winning game Contre Jour is amazing on many fronts, its stunning visuals are probably its most unique asset. As detailed in a colleague’s blog post the game creates its graphics through the OpenGL drawing framework and several elements are dynamically rendered in a more complex manner than by simply drawing images. One of these is the Snot (also known as the Tentacle). They are the most common tools used to help guide Petit through the perils of his epic journey. So sit back and relax as we take a deep dive into how they are constructed as well as how we overcame the pitfalls of translating one into HTML5

The Physics Magic

In order to understand how drawing snots works, it is important to understand the basic structure of one. Please note that Contre Jour uses the 2d physics engine Box2d for its physics. We aren’t going to go over much of the physics aspect of the snot in this blog, but if you’ve never dealt with a physics engine before a quick overview of the documentation may help you understand the next sentence. The talented programmers at Mokus Games designed the snot out of multiple Box2d bodies which are sequentially connected to each other with joints (with the type of joint depending on the type of snot). You can think of each body as being a hinge that flexes when forces are applied to it. Since each body is connected to one another, forces applied to one body will cause the others to react as well.  What is most important for us, however, is that the current location of each body is accessible. These control points will serve as the basis of all the upcoming drawing techniques.

Drawing a Snot

Now that we have a very basic understanding of the inner workings of a snot, we can take off our Box2d hats and think about how we are going to draw one. As shown in the picture below, there are two distinct sections of the snot that will require us to draw them using a different technique.

snot sections

The yellow sections contain the top and bottom portions of the snot and are comprised completely of images, so all we have to do is get the respective positions of the top and bottom Box2d bodies and draw the images at the appropriate location. This is easy to do in the HTML5 canvas and only requires the use of the drawImage method. The blue section that denotes the middle of the snot, however, is a bit more complex to draw. In order to precisely mimic the look and feel of the movement of the underlying bodies, a static image just won’t cut it. Instead, the game’s creators developed a formula that calculated the control points for a bezier curve from the snot’s bodies, calculated an array of intermediate points from the control points, and then gave the array to OpenGL to draw the smooth curve out of many triangles. At first glance, this may seem an impossible technique to translate to javascript. After all, we have no access to OpenGL. However, using the power of the HTML5, it is actually easier to implement the drawing of the body in the canvas than it is through native OpenGL! We simply took the control point calculation from the original code and drew quadratic curves through the canvas’s quadraticCurveTo function and filled the resulting path. The code and results are shown below:

// variables: ctx = HTML5 context, color = rgba color string,
// pairs: array of control point pairs, with each element containing a 
// first and second point that mirror each other along the body.
ctx.fillStyle = color;
ctx.beginPath();

start = pairs[0].first, control, anchor;
ctx.moveTo(start.x, start.y);

for (var i = 1; i < pairs.length; i += 2) {
   control = pairs[i].first;
   anchor = pairs[i + 1].first;
   ctx.quadraticCurveTo(control.x, control.y, anchor.x, anchor.y);
}

ctx.lineTo(pairs[pairs.length - 1].second.x, points[pairs.length -1].second.y);

for (var i = pairs.length - 2; i > 0; i -= 2) {
   control = pairs[i].second;
   anchor = pairs[i - 1].second;
   ctx.quadraticCurveTo(control.x, control.y, anchor.x, anchor.y);
}

ctx.lineTo(start.x, start.y);
ctx.closePath();
ctx.fill();

iOS snot on left, HTML5 snot on right

Drawing a Gradient Snot

While drawing a solid colored snot in HTML5 isn’t a big challenge compared to the native application, drawing the snots for the “Night” world (chapter 2) is a somewhat thornier issue. To modify the code to shade the snot with a gradient, the original version simply gives OpenGL a color array that it calculates based on the number of calculated curve points. When OpenGL draws the snot vertices, it uses the colors stored in the array to color the resulting shapes appropriately. Unfortunately, this is an area where canvas does not have the same flexibility. What canvas does have, however, is an API for creating linear gradients, which is what the snot body looks like. It isn’t as simple as creating a gradient and filling the snot body in one shot as we did with the solid black snot though. As the name implies, a linear gradient is defined as being, well, linear. The API expects a start point and an end point, which it takes and creates a gradient that goes straight between the two points. So if we simply create a gradient that goes from the snot head to its tail, the gradient will not fit correctly on the snot body unless it is in a straight line between the two points. Since a snot is very elastic, this will not always be the case. Luckily for us, simply segmenting how the snot is drawn solves this problem for us. Instead of drawing a path for the whole snot body at once, we break the control points up into groups that roughly correspond to the segments between the Box2d bodies and draw each segment individually and with a gradient that maps to the top and bottom of said segment.

visual representation of the gradient segments

While not 100% accurate, the behavior of the bodies connected by joints guarantees us that each segment will be fairly linear most of the time (except when a snot is bent at a hard angle). Though using phrases such as “not 100% accurate” and “fairly linear” are hardly confidence inspiring, the results of this technique are very close to the original.

// The colors parameter is an array of colors that represent
// the color the snot should be at each control point.
var point0, point1, point2, grd, color0, color1;
context.save();
var colorIndex = 0;

// Iterate through all the segments and draw each one individually.
for (var i = 1; i < points.length; i += 2) {
   point0 = points[i - 1];
   point1 = points[i];
   point2 = points[i + 1];
   color0 = colors[colorIndex];
   color1 = colors[colorIndex + 1];

   // In order to hide breaks it helps to draw the segments
   // with a slight overlap.
   if (i > 1) {
      Box2dUtil.b2Vec2Lerp(point0.first, points[i - 2].first, 0.05, point0.first);
      Box2dUtil.b2Vec2Lerp(point0.second, points[i - 2].second, 0.05, point0.second);
   }

   Box2dUtil.getCenter(point0.first, point0.second, DrawUtil.center);
   Box2dUtil.getCenter(point2.first, point2.second, DrawUtil.center2);

   // Create a gradient that runs from the top to the bottom of the segment
   // using the color value array and set the gradient as the fillStyle.
   grd = context.createLinearGradient(DrawUtil.center.x, DrawUtil.center.y, DrawUtil.center2.x, DrawUtil.center2.y);
   grd.addColorStop(0, color0);
   grd.addColorStop(1, color1);
   context.fillStyle = grd;

   // Fill in the segment.
   context.beginPath();
   context.moveTo(point0.first.x, point0.first.y);
   context.quadraticCurveTo(point1.first.x, point1.first.y, point2.first.x, point2.first.y);
   context.lineTo(point2.second.x, point2.second.y);
   context.quadraticCurveTo(point1.second.x, point1.second.y,point0.second.x, point0.second.y);
   context.lineTo(point0.first.x, point0.first.y);
   context.closePath();
   context.fill();
   colorIndex++;
}

context.restore();

iOS snot on left, HTML5 snot on right

Texturing the Rope

Despite the daunting task of porting all the OpenGL code to HTML5, so far we’ve done all right. The solutions we’ve come up with are fairly easy to understand and implement and stay pretty true to the original game. Sadly, there is always that one unruly individual who likes to rain on everyone’s parade and steal all the cookies. In our case this came to us in the form of the rope snot. In the iOS game, the striped pattern on the rope body is created by using a single texture:

rope texture

The texture is then mapped to the snot body shape by OpenGL using texture mapping in a repeat pattern along the length of the body. This technique ensures that the texture fits perfectly with no overlap or breaks between sections.

Now the million dollar question is how to emulate the results of texture mapping in HTML5 (which has no concept of true texture coordinates) without slowing the browser to a crawl in the process. We tried several techniques to get this to work, including linear gradients, canvas pattern fills and even experimented with trying to create true texture mapping in the canvas. None of these created an effect that looked at all like the original. Luckily, after all these options were exhausted there was one method left to be tried, a rough approximation of texture mapping. It’s really an extension of the gradient technique from the last section. First, the snot body was segmented in the same fashion as the night snot. These sections were then cut into two halves. For each half, we calculated a transform to apply to the texture that mapped as close as possible to the half size. The center of the half was used as the texture’s position, while the rotation was found through calculating the angle of the vector from the top of the segment to the bottom. The x and y scale values were both individually computed by calculating a proportion of the average length and width of the segment with the length and width of the texture. Then, the canvas clip method was used to ensure that any excess texture outside of the segment wouldn’t be drawn.

The big flaw with this method is that without anything else it produces a noticeable disjoint between the individual segments of the rope and the outer curve loses some of its smoothness.

disjointed texture sections

However, this can be minimized by drawing the entire body in with a solid color (like the normal snots) that matches the start and end of the texture pattern before drawing the texture segments. While it effectively hides the seams, it does create a small border on the outside of the snot that becomes more noticeable when the snot does a lot of twisting and turning. It actually creates a sort of 3D effect because of the color of the texture border, so we can chalk this one up as a “feature” and call it a day.

// backColor is a passed in rgba string that we can use to set the
// context color.
var point0, point1, point2;

// Fill in back with solid color to hide breaks using the method
// described in the solid snot section.
DrawUtil.drawBezier(context, points, backColor); 

// Fill the rest of the straight areas with texture.
// This method makes use of a custom Bitmap class to handle drawing
// the texture.
context.save();
for (var i = 1; i < points.length-1; i += 2) {
   point0 = points[i - 1];
   point1 = points[i];
   point2 = points[i + 1];

   if (i > 1){
      Box2dUtil.b2Vec2Lerp(point0.first, points[i - 2].first, 0.05, point0.first);
      Box2dUtil.b2Vec2Lerp(point0.second, points[i - 2].second, 0.05, point0.second);
   }

   // Draw segment path.
   context.beginPath();
   context.moveTo(point0.first.x, point0.first.y);
   context.quadraticCurveTo(point1.first.x, point1.first.y,  point2.first.x, point2.first.y);
   context.lineTo(point2.second.x, point2.second.y);
   context.quadraticCurveTo(point1.second.x, point1.second.y, point0.second.x, point0.second.y);
   context.lineTo(point0.first.x, point0.first.y);
   context.closePath();

   // Calculate a position, rotation, and scale to map half of the
   // texture with the top half of the segment. 
   context.save();

   // Set the texture to draw only the front half.
   texture.frame = 0;
 
   Box2dUtil.getCenter(point0.first, point0.second, DrawUtil.center);
   Box2dUtil.getCenter(point1.first, point1.second, DrawUtil.center2);
   // Calculate the rotation of the texture from the vector between the top and bottom centers of the top half segment.
   var direction = Box2dUtil.b2Vec2Subtract(DrawUtil.center2, DrawUtil.center);
   var rotation = Box2dUtil.atan2Vec(direction);

   // Calculate the y scale by dividing the segment's length by the
   // texture's length.
   var scaleY = direction.Length() / 16;

   // Calculate the x scale by averaging the widths of the top and
   // bottom of the segment and then divide that by
   // twice the width of the texture (because we are averaging two together).
   var point1Width = Box2dUtil.b2Vec2Distance(point1.first, point1.second);
   var scaleX =(Box2dUtil.b2Vec2Distance(point0.first, point0.second) + point1Width) / (32 * 2);
   // Calculate the center of this half of the segment.
    Box2dUtil.getCenter(DrawUtil.center, DrawUtil.center2, DrawUtil.center);  
   // Clip the context and use a draw method of the Bitmap to draw
   // the texture with the calculated values.     
   context.clip();
   texture.drawNoTransform(context, DrawUtil.center.x, DrawUtil.center.y, rotation, scaleX, scaleY); 
   context.restore();

   // Repeat for the bottom part of the segment.
   context.save();
   texture.frame = 1;
   Box2dUtil.getCenter(point2.first, point2.second, DrawUtil.center);
   var direction = Box2dUtil.b2Vec2Subtract(DrawUtil.center, DrawUtil.center2);
   var rotation = Box2dUtil.atan2Vec(direction);
   scaleY = direction.Length() / 16;
   scaleX = (point1Width + Box2dUtil.b2Vec2Distance(point2.first, point2.second)) / (32 * 2);
   Box2dUtil.getCenter(DrawUtil.center, DrawUtil.center2,DrawUtil.center);            
   context.clip();
   texture.drawNoTransform(context, DrawUtil.center.x, DrawUtil.center.y, rotation, scaleX, scaleY);
   context.restore();
}
context.restore();

iOS rope on left, HTML5 rope on right

As you can see, it took some creativity to replace OpenGL drawing with the canvas, but with a little elbow grease and the processing power provided by hardware-accelerated browsers such as IE10 we managed to create a pretty consistent look with the native application that managed not to slow it down to a crawl. So I encourage you to go out and experiment with HTML5 yourself. Maybe you’ll accidentally make the next big browser game.

Comments Off , permalink


9

Oct 12

Large Datasets in Windows 8: Part 2 – Syncing Datasets to SQLite



Screen shot

Note: This is part two of a two part series. Part one focuses on how to get data from a web service using a C# data layer and display it in a ListView and can be found here.

In my previous post, I showed how to page a ListView by making sequential calls to a web service. Let’s say that the application you’re developing will only have intermittent web access. In this situation, syncing data to a local SQLite database is a good option.


Syncing Data from a Web Service to SQLite

This post uses the same source code as my previous post, so we’ll once again be using the City of Chicago Data Portal, in particular, the Current Employees Names, Salaries, and Position Titles dataset. Since we are only concerning ourselves with the one table, our sync process is going to be very straightforward. We’ll pull down the data in batches, and either insert the row into our database or update the data if it already exists.

If you start to look at the code, you’ll see that the DataSyncService.cs file is the main entry point into the sync process. It is extensible allowing you to sync as many tables as you would like with a fine level of control. We’re going to skip over this class though and move on to the lower level guts of the process contained in DataSyncController.cs. Let’s take a look at the two main functions:

private async Task<bool> SyncTable<T, R>(Func<Task<List<T>>> getItems, Func<R, Task> insertOrUpdateItem)
{
    var items = await getItems.Invoke();
    if (items != null)
    {
        foreach (var item in items)
        {
            await insertOrUpdateItem.Invoke(ModelMapper.MapValues<T, R>(item));
        }
    }

    if (items.Count != BATCH_SIZE)
        return true;

    return false;
}

public async Task SyncEmployeeSalary()
{
    bool done = false;
    int count = 0;
    while (!done)
    {
        done = await SyncTable<ServiceModel.EmployeeSalary, LocalData.LocalModel.EmployeeSalary>(
            new Func<Task<List<EmployeeSalary>>>(() => _remoteDataProvider.GetEmployeeSalariesAsync((count * BATCH_SIZE).ToString(), BATCH_SIZE.ToString())),
            new Func<LocalData.LocalModel.EmployeeSalary, Task>(_localDataRepository.InsertOrUpdateEmployeeSalary));

        count++;
    }
}

SyncEmployeeSalary is called by the DataSyncService and in turn it calls SyncTable. The SyncTable function is a generic function that can be used to sync any table types. The first parameter passed to SyncTable tells it what web service call to make to get a list of objects. The second parameter tells the function what local repository function to call to insert or update SQLite data. Finally, the two types (T and R) provided to the function tell it to map ServiceModel.EmployeeSalary objects to LocalModel.EmployeeSalary objects. This is because the objects we receive from the web service are going to have a slightly different structure than what we store in our database. The while loop inside SyncEmployeeSalary simply batches web service calls. Since the web service has 32,000 rows, it is too much data for us to return all at once so we need to break up our calls. BATCH_SIZE is a local constant set to a default of 500. In your own work you may have to tinker with the number to suit your needs.

In Part 1 of this series, we covered getting data through web service calls, so we’re going to skip straight to the SQLite interaction in LocalDataRepository.cs:

public async Task InsertOrUpdateEmployeeSalary(EmployeeSalary employeeSalary)
{
    var connection = GetConnection();

    var existingEmployee = await (connection.Table<EmployeeSalary>()
        .Where(x => x.id == employeeSalary.id).FirstOrDefaultAsync() as Task<EmployeeSalary>);
    if (existingEmployee != null)
    {
        existingEmployee = ModelMapper.MapLocalModel<EmployeeSalary>(employeeSalary);
        await connection.UpdateAsync(existingEmployee);
    }
    else
    {
        await connection.InsertAsync(employeeSalary);
    }
}

This code is pretty straightforward, we first check to see if a record exists for our current Id. If it does, we update the record. If it doesn’t, we insert it.

In this sample application, I chose to sync data immediately on install and then every 30 days thereafter. The City of Chicago only updates this data quarterly, so syncing more frequently seems unnecessary especially given the time it takes to download 32,000 records. The sync process is kicked off in default.js if needed, and upon completion we navigate to our home page.


Displaying Data from SQLite

Displaying the SQLite data can be done in a very similar fashion to the way we were previously displaying data from the web service. I’ve added a new call to the Application Model, GetEmployeeSalariesLocalAsync, that targets local data instead of remote data. The call still takes in a single page, and since we are able to access SQLite using Linq we can do a Skip(start).Take(length) on the data to perform paging. For this example, this allowed me to give the user the ability to say whether they want to use remote data or local data as they page through the application. When you click the combo box in the lower left corner of the application, a boolean is flipped and the javascript calls the proper get function:

var cmboSelectionChanged = jsUI.eventHandler(function (e) {
    if (e.currentTarget.selectedIndex == 0) {
        isUsingWebService = true;
    } else {
        isUsingWebService = false;
    }
    getTeams();
});

var getTeams = function () {
    if (isUsingWebService) {
        WinJS.Promise.wrap(app.getEmployeeSalariesRemoteAsync(parseFloat(currentPage)).then(employeesLoaded));
    } else {
        WinJS.Promise.wrap(app.getEmployeeSalariesLocalAsync(parseFloat(currentPage)).then(employeesLoaded));
    }
};

Pretty basic. If you want a better explanation how the data is bound, you can find that in my previous post as well.

I tried to write this sync in a way that was extensible and straightforward. That”s why some of the code may seem extraneous in such a simple application but hopefully it gives you ideas of what can be done in more complicated scenarios.

Return to Part 1 – Displaying the Data

Source

Clarity.Samples.LargeDataset.zip

Note: The City of Chicago changed their variable names from last week to this week and I wouldn’t be surprised if they do it again. This may just happen quarterly when they update the data. If you run the project and a javascript error is thrown saying a value cannot be null, it just means the mapping needs to be redone. If you open Fiddler and get values back using the url above and open Clarity.Samples.LargeDataset.Services.RemoteData.ServiceModel.Model.cs you”ll see above the Name, JobTitle, Department, and Salary variables are name mappings. If you replaces those with the corresponding new numbers from Fiddler, everything should work just great again.

32 comments , permalink


9

Oct 12

Large Datasets in Windows 8: Part 1 – Displaying the Data



Screen shot

Note: This is part one of a two part series. Part two focuses on how to sync datasets to a local SQLite database and can be found here.

Windows 8 offers some great controls to display data. For this post, I’ll be focusing on the Win8 ListView control. The ListView gives you the ability to display items in a grid or a list with complete control of item templates. But first, let’s focus on getting data into the application.


Retrieving Data from a Web Service

For this example I am going to use the City of Chicago Data Portal, in particular, the Current Employees Names, Salaries, and Position Titles dataset. Even though the front end will be done in HTML and JavaScript, I recommend creating a C# data layer. While it is possible to call web services from JavaScript, I prefer moving this logic into C#. I have worked with both and found the C# data layer to be not only cleaner, but also easier to work with in general.

This dataset contains around 32,000 rows of information. Loading all that data into a ListView at once would bring performance to a grinding halt. So, we are going to page our data by calling the web service with a start value and a length value that determines how many records are returned. We see the main piece of this call in the below from RemoteDataProviderPartial.cs:

   
    public async Task<List<EmployeeSalary>> GetEmployeeSalariesAsync(
            string start, string length)
        {
            var response = await MakeHttpWebRequestAsync(
                RemoteDataOperationUrls.EmployeeSalary.EmployeeSalary_Get,
                "GET",
                null,
                new Dictionary<string, string>(){
                    {"ST", start},
                    {"LGTH", length}
                });
            return JsonHelper.Deserialize <List<EmployeeSalary>>(response);
        }

In this call I’m referencing the EmployeeSalary_Get url and deserializing the response into a List of EmployeeSalary objects. The full url that we are using is “https://data.cityofchicago.org/views/xzkq-xp2w/rows.json?method=getByIds&asHashes=true&start={ST}&length={LGTH}&meta=false”. You can put that url into Fiddler and see responses if you like, but be sure to replace the {ST} with a start value (or 0) and {LGTH} with how many records you would like returned. Now that we have data returned from the service, we need to tackle how we can make the above call asynchronously from javascript.


Displaying Data in a ListView

Without diving too far into the code, we initialize a sealed class (Locator.cs in the source code) in default.js that has static references to a few different properties. The only one that we are concerned about for now is the Application Model. In a more detailed application, the Application Model could be used to store current application data like active selections. For our simple app, we’ll just use it as an accessor for web service data through the call GetEmployeeSalariesRemoteAsync(int page).

By setting up our data layer this way, we can easily access the call through javascript. The following is a subsection of the home.js page:

    var app = Clarity.Samples.LargeDataset.Services.Locator.applicationModel;

    var onReady = function (element, options) {
        page = element;
        viewModel = jsBind.as(new homeViewModel());

        currentPage = 1;

        var promise = WinJS.Promise.wrap(app.getEmployeeSalariesRemoteAsync(parseFloat(currentPage)).then(employeesLoaded));
        promise = promise.done(function () {
		jsBind.processAll(page, viewModel);
                return markup.processAll(page);
        });
    };

    var employeesLoaded = function (employees) {
        viewModel.employees = new jsBind.List(employees.map(function (employee) { return new employeeViewModel(employee) }), { binding: true });
        var employeeList = document.querySelector("#employeesList").winControl;
        employeeList.itemDataSource = viewModel.employees.dataSource;
    };

The above code obviously won’t run if copy and pasted, but let’s step through it line by line. First we’re pointing the app variable at our C# Application Model namespace. In the onReady function, we’re creating our view model, setting our current page to 1, then calling our Application Model to get us the employee data from the web service. When that async call returns, we jump into the employeesLoaded function. In here we map each object in the list that was returned to a javascript employee view model and add it to our base home view model’s employee collection. We then get the ListView winControl by its HTML id and bind the employees to the ListView. The HTML for the ListView is shown below:

<div id="employeesList"
   data-win-control="WinJS.UI.ListView"
   data-win-options="{itemTemplate:select('#employeeTemplate'), layout:{type:WinJS.UI.GridLayout}, selectionMode:'none'}">
</div>

And the template:

<div id="employeeTemplate" data-win-control="WinJS.Binding.Template">
   <div class="tile employee">
      <header>
         <h2><span data-win-bind="textContent: name"></span></h2>
         <h3><span data-win-bind="textContent: jobTitle"></span></h3>
         <h3><span data-win-bind="textContent: department"></span></h3>
         <h3><span data-win-bind="textContent: salary"></span></h3>
      </header>
   </div>
</div>

As you saw before, the items of the ListView were bound in JavaScript. That means that each item is of type employeeViewModel which I skipped showing because of its simplicity. It simply contains the properties that are bound in the above template.

Since we only pulled back one page worth of data, I added some paging buttons to the bottom of the page. I won’t go through the implementation since its rather trivial, but whenever you click a button, it sends a page number to the Application Model. The Application Model has a constant telling it how many records to retrieve (12 is default) and calculates the start offset accordingly. For example, if the view sends “13″ down as the page number, the Application Model would contact the web service with a start value of 144 ((13-1) * 12 – remember you start at 0 for page 1) and a length of 12.

That’s all there is to it. If you’re interested in how to sync all the data from the web service into a SQLite database, you can check out part 2 here.

Continue to Part 2 – Syncing Datasets to SQLite


Source

Clarity.Samples.LargeDataset.zip

Note: The City of Chicago changed their variable names from last week to this week and I wouldn’t be surprised if they do it again. This may just happen quarterly when they update the data. If you run the project and a javascript error is thrown saying a value cannot be null, it just means the mapping needs to be redone. If you open Fiddler and get values back using the url above and open Clarity.Samples.LargeDataset.Services.RemoteData.ServiceModel.Model.cs you’ll see above the Name, JobTitle, Department, and Salary variables are name mappings. If you replaces those with the corresponding new numbers from Fiddler, everything should work just great again.

1 comment , permalink


9

Oct 12

Contre Jour’s Visuals, From Native to Web



  Contre Jour’s visuals are beautiful, unique, and conceptually mind boggling. The snot-like creatures that populate the game feel and look as if they’re made of liquid. The clay like ground molds like Play-doh; Even some of the background details feel as if they’re going to come to life and peel right off the screen. Contre Jour is memorable because of its organic look and feel. So when I started on the Contre Jour HTML5 port I was confident that we would be able to port the game logic perfectly, but uncertain we could do the visuals justice. And without the visuals remaining intact, the project would be considered a failure

  Porting the objective-c game logic to JavaScript was tedious, but there was a consistent pattern that could be followed in almost every instance. Conversely, porting the drawing logic was tricky because we needed to improvise and draw using canvas context commands, and not the OpenGL API that had been used in the original iOS code base. At times this was a boon, and others a curse. The most common operation needed during the rendering cycle was image drawing. Drawing images with canvas is considerably easier than drawing images through OpenGL, and trust me – that’s a very good thing! The canvas context gracefully replaces the tedious matrix stacks and vertex buffers used and abused by OpenGL aficionados. Fret not though, you can still apply the classic transformations of scale, rotation, and translation using the respective canvas commands ctx.scale(x,y), ctx.rotate(rotation), and ctx.translate(x,y). Typically to begin an image draw you’ll perform a ctx.save(), apply your transformations, perform a draw, and then clear out your transformations by executing a ctx.restore(); All together it ends up looking something like this.

var canvas = document.getElementById("exampleCanvas");
var ctx = canvas.getContext("2d");
ctx.save();
ctx.translate(x, y);
ctx.rotate(rotation);
ctx.scale(scaleX, scaleY);
ctx.drawImage(this.image);
ctx.restore();

  Those strung together commands are straightforward, require little background knowledge to use, and are extremely reusable when wrapped in a function call. The problems began when sections of the iOS code started applying effects on rendered images to give them a certain color hue, or other visual effect. The problem was that we couldn’t do it the traditional way, which was with shaders. Canvas doesn’t really have shaders to speak of. This prompted the first real struggle we faced during Contre Jour’s HTML5 transformation: Rendering the ground. In the iOS version, the ground has this delicate shading applied to it that really makes it (the design team might kill me for saying this) pop. That effect is accomplished using some custom shaders written in GLSL. The syntax is at first-sight so foreign that I’m not going to bother posting a sample of it. With the problem identified, I began snooping around online so that I could develop and experiment with different substitute shading techniques. I’ll describe a few of them for you.

  At first I attempted overlaying gradients on key locations of the ground, but the effect looked choppy and cheap. The next technique I tried was a pixel-by-pixel manipulation on the canvas’ pixel data. It was the best looking, but extremely slow. It wasn’t acceptable for a gaming environment. Though, the logic would be a pretty slick for adding effects to a photo.

Pixel-by-pixel Effect



Pixel-by-pixel manipulation snippet

shadeEdges:function(ctx, detailLevel, strength){
         if(strength === undefined){
             strength = 2;
         } 
         if(detailLevel === undefined){
             detailLevel = 20;
         } 
        var imageData = ctx.getImageData(0, 0, ctx.canvas.width, ctx.canvas.height );
        var data = imageData.data;
        var imageDataLength = data.length;
        var imageDataWidth = imageData.width;
        var pixelProximityLevel = 0;
        var pixelIterator = 0
        var widthTimesFour = imageDataWidth * 4;

        for (var i = 0; i < imageDataLength; i+=4) {
            if(data[i] === 0 && data[i + 3] !== 0)
            {
                pixelProximityLevel = 0;
                for (pixelIterator = 1; pixelIterator < detailLevel; pixelIterator++) {
                    if(data[i - 1 + (pixelIterator * 4)] === 0)    
                    {
                        pixelProximityLevel+= detailLevel - pixelIterator;
                        break;
                    }
                };
                for (pixelIterator = 1; pixelIterator < detailLevel; pixelIterator++) {
                    if(data[i + 3 - (pixelIterator * widthTimesFour)] === 0)
                    {
                        pixelProximityLevel+= detailLevel - pixelIterator;
                        break;
                    }
                }
                if(pixelProximityLevel > 0)
                {
                    data[i]   += pixelProximityLevel * strength;     
                    data[i+1] += pixelProximityLevel * strength;
                    data[i+2] += pixelProximityLevel * strength;
                }       
            }
        };
        ctx.putImageData(imageData,0,0);
    }

  My final attempt was something so elementary and easy to implement that I couldn’t believe it worked. That technique was this: simply stacking lines with decreasing width and intensity around the path that defined the grounds shape – essentially creating a form fitting gradient. The downside to the technique was that without producing hacky looking visuals, I was unable to limit the ground “shading” to only the top portion of the ground. The upside was that the technique looked great and could have its quality scaled dynamically. Keep in mind, if you’re about to examine the code, the path is created on the context before the context is passed as the first parameter into the function.

Stacked line effect



Stacked line snippet

shadePath: function(ctx, intensity, shadingWidth, passes) {
            intensity = intensity === undefined ? 1.3 : intensity;
            shadingWidth = shadingWidth === undefined ? 30 : shadingWidth;
            passes = passes === undefined ? 5 : passes;

            var decrementer = shadingWidth / passes;

            for (var i = shadingWidth; i > 0; i -= decrementer) {
                ctx.lineWidth = i;
                var strokeIntensity = Math.floor(i * intensity);
                ctx.strokeStyle = "rgb(" + strokeIntensity + "," + strokeIntensity + "," + strokeIntensity + ")";
                ctx.stroke();
            };
},

 The final look of the ground is visually different from the iOS version, but it still succeeds at giving the ground an organic look and feel. It’s also a huge improvement over what the HTML5 ground looked like to begin with.

What the ground looked like to start with


Some parting thoughts

 In my mind, there’s one major take away from this post. Canvas doesn’t offer the same spread of functionality that its native cousins do, but what it does offer is a low barrier of entry, affordability, and a simple means for green-horn and grizzled developers alike to develop games on the broadest platform available: The web.

If you’re interested in looking at and tinkering with a psychedelic version of the final “shader” in action, hit this link

1 comment , permalink