Recently I was poking around with some different ways to create a scalable image store for my project when I realized, I've done this before. I'm working on an enterprise ASP.NET e-commerce site that lives in a web farm. The site contains a large number of images, and I would like to make these images manageable and configurable.
Typically I have seen this done in one of two ways.
- Store the images on one or more resource servers that are shared among the web farm servers. This may or may not be a good option depending on a number of factors including the site's load, the rate that images are requested, etc.. There's a risk that the resource server(s) will bottleneck the site, regardless of how well the web servers are load balanced.
- Replicate the images on each of the servers in the web farm.
Both of these options require some meticulous file system management and syncing that I personally have limited patience for. Luckily, Microsoft has provided us other options. With LINQ and SQL Server 2008, we store images in a database without writing very much code. In versions before SQL Server 2008 it is possible to store files using the VARBINARY(MAX) data type (preserving transactional consistency). Using this approach however, data is stored as blobs directly in the database. This can cause the database to become very large and to perform poorly.
SQL Server 2008 introduces a new FILESTREAM attribute that can be added to a VARBINARY(MAX) column. This allows you, the database administrator, to choose a location on your NTFS file system where you would like images to be stored. Once this location is set, a directory structure will be created at your chosen path. These files are managed automatically by SQL Server and should not be modified manually. Data stored in the FILESTREAM field will follow the same consistency and backup rules as any other database field, giving users transparency. Filestream support is not enabled by default, so there is a little bit of initial set up on the database server. It's worth the effort.
So let's see it in action. In the following example, we will create a simple ASP.NET application that will allow us to upload an image and display a grid of images that currently stored. This example will use SQL Server 2008, ASP.NET 3.5 (LINQ-to-SQL to save and retrieve image data), Internet Information Services, and Visual Studio 2008.
I. Set up the database
My intention is to do as much as possible through the UI of the tools provided by SQL Server 2008. The application will be built back to front.
1. Open SQL Server Configuration Manager and right click the SQL Server 2008 instance you will be using for this example - by default it will be called SQL Server (MSSQLSERVER). Select the FILESTREAM tab and check Enable FILESTREAM for Transact-SQL access.
Click "Apply" and exit SQL Server Configuration Manager. You have just enabled file streaming on your database instance.
2. Next, open up SQL Server Management studio and log in to your database Instance. In my case it will be called KABIR-LITE. Once you are connected, right click on your database root and select Properties. A server properties window will pop open. Select Advanced from the list on the left of the window. The first option in the list of advanced server properties is "Filestream Access Level". The default setting for this is Disabled. Click on this to pop open a dropdown list with three options. Select Full access enabled.
Click OK. You will get a message saying that you need to restart your database instance for the changes to take effect. Open up SQL Server Configuration Manager again, right click on your instance, and click Restart.
Great. Now we are ready to create a database for our application to connect to.
3. If you already have a database that you would like to use for this example, you may skip this step. Create a database called "StoreImages". You can do this by right clicking on Databases under your database instance click selecting New Database.... The name of the database can be whatever you like, but I picked this name to be consistent with the name of my application. All other default values can remain the same during this creation.
4. Now that we have a filestream enabled database, we need to specify a location for SQL Server to create a directory structure to store our files. Right click on your database (StoreImages in my case), and select Properties.
Click on the Filegroups option from the list on the left. You will see two list boxes - Rows and Filestream. Click on Add from the Filestream listbox. This can be named anything you like, in my case I have named it "StoreImages_fg_fs". Make sure the Default checkbox is checked for this value.
From the same menu on the left, click Files. We will need to specify the physical location to dump the filestream data. Click Add and set the logical name of your file to "StoreImages_Filestream". Change the file type to Filestream Data and the filegroup to the name of the filegroup you created above (in my case "StoreImages_fg_fs". Select a path where you would like to store your images.
When you click OK, the database engine will create a directory with the name of your file directly under the path you just specified.
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
5. So you might be thinking, that was a heck of a lot of initial set up and you're right. Good news, that's all of it. Now we can go about creating our table schema and application just as we normally would. Let's create a very simple table with an ID and an image. Note that we cannot perform this step using the SQL Server Management Studio UI. Support for the FILESTREAM datatype has not yet made it into the designer.
Create a table called Image. Notice that our our VARBINARY(MAX) has a FILESTREAM attribute attached to it. This is required (and the reason we cannot create this table from the table designer).
CREATE TABLE Image (
ID UNIQUEIDENTIFIER ROWGUIDCOL NOT NULL PRIMARY KEY,
BLOB VARBINARY(MAX) FILESTREAM NULL
)
You might be thinking that this is a trivially bare bones example. That's true, but the purpose of this example is not to demonstrate the possible complexities of relational data modeling. My goal is to demonstrate how to plug image storage and retrieval into any relational data model, just as you would expect from a varchar or int datatype. Again, once you have specified a field as filestream, the data is treated and related to (even backed up) exactly the same way all other data in SQL Server.
II. Create a DBML data abstraction
1. Open Visual Studio 2008 and create a new ASP.NET Web Application and call it "StoreImages". Make sure .NET Framework 3.5 is selected in the dropdown list in the top right-hand corner of the New Project window.
2. The first thing we'd like to do is create some data abstraction that will allow us to retrieve images from the database we just created. Luckily, with LINQ this is a very fast process. Right click on the StoreImages project and click Add > New Item.
Click on Data from the categories menu and select LINQ to SQL Classes from the list of templates. I usually give my LINQ to SQL class file (DBML file) the same name as the database it will connect to. In this example I have named it "StoreImages.dbml". I know, shocker.
Click Add.
3. When your DBML file opens, there will be a linked tip asking you to open your server explorer and drag items onto the surface. Since we only have one item to add (our table named Images from dbo.StoreImages), let's go ahead and do that. Open up your server explorer and create a connection to your database.
Make sure to specify the database name (StoreImages) under "Connect to a database".
Click OK.
4. You should see a tree visualizer of your database. Expand it and click on Tables. Underneath Tables you should see the table "Images". Drag the images table onto the designer surface of your DBML file. Your DML file should now contain an item that looks like this:
III. Set up a generic handler to serve our image
1. Right click on the StoreImages project and click Add > New Item. Click on Generic Handler, call it "ServeImage.ashx", and click Add. You can remove the decorators that were automatically created for you at the class level. For the purpose of this example, we will not need support for a web service call into our handler. Your generic handler should look like this:
namespace StoreImages
{
public class ServeImage : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
context.Response.ContentType = "text/plain";
context.Response.Write("Hello World");
}
public bool IsReusable
{
get
{
return false;
}
}
}
}
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
Go ahead and remove the implementation of ProcessRequest (Delete the struck out section). We will be adding stuff here.
2. Use LINQ to get an image by its ID from the DBML data abstraction that we created. LINQ automatically creates a Binary field for database columns marked as VARBINARY.
Paste the following code in ProcessRequest.
//Declare a binary variable to hold image data
Binary image = null;
//Connect to our database and retrieve image data, if any exists for the ID in the querystrnig
using (var db = new StoreImagesDataContext())
{
image = (from img in db.Images
where img.ID == new Guid(context.Request.QueryString["id"])
select img.BLOB).SingleOrDefault();
}
//Write the image data out
if (image != null)
{
context.Response.BinaryWrite(image.ToArray());
}
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
The handler will now expect an "id" to be passed into the querystring. The handler will then ask the database for an image with that "id" and serve it back to the caller.
We now have a URL that we can give our ASP.NET image control (i.e. ServeImage.aspx?id=<ID>);
IV. Upload images and display them in a simple ASP.NET Gridview
1. Navigate to Default.aspx and click on Source at the bottom of the designer window. Create a Gridview with an Imagefield in the automatically created "<div>" . Point the Imagefield's DataImageUrlFormatString to ServeImage.ashx with a query string parameter set to variable {0}. The databound object will contain an "ID" property that we can replace the format strings variable with. Set the DataImageUrlField to "ID".
2. Navigate to the codebehind of Default.aspx. Create an OnPreRender event handler and set the datasource of our gridview to the entire list of Image Ids. The reason we do this in the OnPreRender event handler is so that any control events that need to fire will do their processing before we display our images.
protected override void OnPreRender(EventArgs e)
{
base.OnPreRender(e);
using (var db = new StoreImagesDataContext())
{
Gridview1.DataSource = db.Images.Select(i => new { ID = i.ID }).ToList();
Gridview1.DataBind();
}
}
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
The reason for the lambda expression selecting our ID into an object with one member is to demonstrate that this works outside the trivial case of using a list of native types.
3. Right now we have no images stored in the database, so we're not going to see anything in our grid. We need a way for us to be able to insert images into the database. Again, LINQ makes this nice and easy. Go back to Default.aspx and Drag a FileUpload control and a Button onto our surface above the Gridview. Switch to Design view and double click the new button to automatically generate a click handler in your codebehind. You will be redirected to the new handler.
4. Paste the following code into the button click handler
using (var db = new StoreImagesDataContext())
{
byte[] filebyte = FileUpload1.FileBytes;
var pageImage = new Image()
{
BLOB = filebyte,
ID = Guid.NewGuid()
};
db.Images.InsertOnSubmit(pageImage);
db.SubmitChanges();
}
.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, "Courier New", courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }
This will convert the specified file to a byte array and insert it into the database. Boom. That's should be all we need to do.
IV. Run it!
1. Hit F5. You should see a FileUpload control and a button. Click on browse and find an image on your file system. Click "Button". You should immediately see you image show up underneath your controls.
Here's my page after uploading two images:
2. Technically we're done, but if you want to prove to yourself that SQL Server is actually storing these files in the location that we went through the headache of specifying in part I....just take a look! Navigate to the file path, if you can remember what it is, that you specified in your default filegroup. I believe mine was c:\examples\storeimages.
There might be a whole mess of directories named as seemingly meaningless GUIDs, but if you drill down enough, you will see a list of files with....seemingly meaningless guids. Copy these files (DO NOT modify them in place) to your dsektop or some temporary location. Add the appropriate extensions to your files (in my case ".png" for the first image and ".jpg" for the second). You should now be able to open them up and view them in your default photo viewer.
At this point you might be thinking, aren't there security concerns with leaving my filestream data in raw on my file system? Good point, however keep in mind that the file path can be any location of your choosing. It can be on the database server itself, on a mapped network drive, or even on some external physical media. This gives you the option of storing this data in a secure location and the flexibility to change it whenever you like.
This is my first blog post and I've always wanted to say this....
Hope this helps