Building a File Service

Problem:

Managing files web applications should be quick and easy and most importantly, consistent. The traditional way to store files is on the file system or within a RDBMS (SQL SERVER, Oracle, MySql) or on SharePoint. However what if you were developing several applications that shared a million files and each was 5MB. Which solution would you choose? This is commonly the case for large content driven sites (Music Stores, video sharing sites, content management systems, etc).

Solution:

In providing a solution I would implement a SOA based File Service, and this article will guide you to making your own File Service. The reasons that I believe a File Service works are:

• Single point of storage for data
• Can be used by multiple applications
• Single source for security
• Easy to de-duplicate files
• Clear separation between binary data and metadata
• Efficient file system usage ensuring that director retrieval is fast
• Efficient IO usage using multiple disks (per share)

Data Structure:

Firstly we need to create a table to store our file information. Most columns on the table shown in Figure 1 are self explanatory and for possible enhancements see the "Future Enhancements" section below.


Figure 1

Solution Structure:

1. Create a new Solution in Visual Studio 2008, targeting the .NET Framework 3.5
2. Create a new Class Library called "FileServiceCore"
   1. Add a new class called "FileHandlerFactory.cs"
   2. Add a new class called "FileHandlerLinq.cs"
   3. Add a new class called "FileDistributor.cs"
   4. Add a new LINQ to SQL Classes called "FileService.dbml"
   5. Using the Server Explorer create a new connection to the database that contains the "Files" table described above. You should be able to drag and drop the table onto the designer.
3. Create a new Website called "FileService"
   1. Add a project reference to the FileServiceCore
   2. In the website root add a new generic handler called "ImageHandler.ashx"
   3. In the website root add a new generic handler called "DownloadHandler.ashx"
4. Create a new website called "SampleWebsite"
   1. Add a project reference to the FileServiceCore

Your solution should now look like figure 2 below.


Figure 2

Logic:

Now that you have created your Data Structure and Visual Studio solution we will need to add the application logic to each of the classes (e.g. FileHandlerFactory.cs).

FileDistributor.cs

The FileDistributor class is responsible for distributing files to the file system. It contains a single static method that calculates a network file path for a FileId.

namespace FileServiceCore

{

public static class FileDistributor

{

/**//// <summary>

/// We use a static variable to tell the system which path to use

/// This ensures the file distributor acts as similar to a Round robin DNS

/// </summary>

private static volatile int pathId = 0;

/**//// <summary>

/// Gets the distributed file path.

/// </summary>

/// <param name="fileId">The file id.</param>

/// <returns></returns>

public static string GetDistributedFilePath(int fileId)

{

// normally you would store these in a config file or a database, however in keeping this example

// simple I have hard coded the network share paths

string[] networkSharePaths = new string[] {

@"C:\Temp\FileService\Share1\",

@"C:\Temp\FileService\Share2\",

@"C:\Temp\FileService\Share3\"};

// we use an internal variable to ensure there are no multithreading issues

int innerPathId = pathId;

// check that we have a valid path

pathId += 1;

if (pathId >= networkSharePaths.Length)

{

pathId = 0;

}

// check that we have a valid path for the internal variable, as we don't want

// to have a situation where innerPathId doesn't reference a valid network share paths

// array index

if (innerPathId >= networkSharePaths.Length)

{

innerPathId = 0;

}

// check the root path exists and creates the directory if it does not

if (!System.IO.Directory.Exists(networkSharePaths[innerPathId]))

{

System.IO.Directory.CreateDirectory(networkSharePaths[innerPathId]);

}

// pad the fileId to 9 places which allows for about 999,999,999 files

string paddedFileId = fileId.ToString().PadLeft(9, '0');

// check main folder

string mainFolder = System.IO.Path.Combine(networkSharePaths[innerPathId], paddedFileId.Substring(0, 3));

if (!System.IO.Directory.Exists(mainFolder))

{

System.IO.Directory.CreateDirectory(mainFolder);

}

// check the next folder down

string fileFolder = System.IO.Path.Combine(mainFolder, paddedFileId.Substring(3, 3));

if (!System.IO.Directory.Exists(fileFolder))

{

System.IO.Directory.CreateDirectory(fileFolder);

}

// get the final file name (this will be in the xxx.extension format)

string finalFileName = paddedFileId.Substring(6);

// the result should be similar to the following

//  C:\Temp\FileService\Share1\

//      000

//          000

//              001.txt

//              002.txt

return System.IO.Path.Combine(fileFolder, finalFileName);

}

}

}

FileHandlerLinq.cs

The FileHandlerLinq class is a static class that each calling application can use to Insert/Select files from the Files table. For the purpose of this example update and deletes have been omitted.

Code
using System.Linq;
namespace FileServiceCore
{
public static class FileHandlerLinq
{
/**//// <summary>
/// Selects the specified file by a FileId.
/// </summary>
/// <param name="fileId">The file id.</param>
/// <returns><see cref="File"/></returns>
[System.ComponentModel.DataObjectMethod(System.ComponentModel.DataObjectMethodType.Select)]
public static File Select(int fileId)
{
using (FileServiceDataContext db = new FileServiceDataContext())
{
return (from z in db.Files
where z.FileId.Equals(fileId)
select z).FirstOrDefault();
}
}
/**//// <summary>
/// Inserts a record into the Files table and moves the file to a network share.
/// </summary>
/// <param name="fullFileName">Full name of the file.</param>
/// <param name="contentType">Type of the content.</param>
/// <returns><see cref="File"/></returns>
public static File Insert(string fullFileName, string contentType, System.IO.Stream inputStream)
{
using (FileServiceDataContext db = new FileServiceDataContext())
{
// get the file properties (so we can get the mime type, length, etc)
System.IO.FileInfo fileInfo = new System.IO.FileInfo(fullFileName);
File entity = new File();
entity.ContentType = contentType;
entity.ContentLength = (int)inputStream.Length;
entity.FileName = fileInfo.Name;
entity.CreatedDate = System.DateTime.Now;
// we set the network path to empty as it will be generated from 
// the database identity seed column "FileId"
entity.NetworkPath = string.Empty;
// inserting the record will update the entity object
db.Files.InsertOnSubmit(entity);
db.SubmitChanges();
// update the new path to be the correct location
entity = (from z in db.Files
where z.FileId.Equals(entity.FileId)
select z).FirstOrDefault(); ;
entity.NetworkPath = string.Concat(FileDistributor.GetDistributedFilePath(entity.FileId), fileInfo.Extension);
// convert the stream into a series of bytes
byte[] bytes = new byte[(int)inputStream.Length];
inputStream.Read(bytes, 0, (int)inputStream.Length);
// write the bytes to our network path
System.IO.File.WriteAllBytes(entity.NetworkPath, bytes);
// update the database with the new network path
// this will be in the format of \\FileShare\xxx\xxx\xxx.extension
db.SubmitChanges();
return entity;
}
}
}
}

FileHandlerFactory.cs

The FileHandlerFactory is a simple factory class that determines if the browser should handle the file or if the file should be downloaded by the client as an attachment. In this example the factory is simple and should be customized for your own needs.

using System;
using System.Web;
namespace FileServiceCore
{
public static class FileHandlerFactory
{
/**//// <summary>
/// Transmits the file.
/// </summary>
/// <param name="context"><see cref="HttpContext"/>.</param>
/// <param name="fileId">The file id.</param>
public static void TransmitFile(HttpContext context, int fileId)
{
TransmitFile(context, fileId, false);
}
/**//// <summary>
/// Transmits the file.
/// </summary>
/// <param name="context"><see cref="HttpContext"/>.</param>
/// <param name="fileId">The file id.</param>
/// <param name="forceAsAttachment">if set to <c>true</c> [force as attachment].</param>
public static void TransmitFile(HttpContext context, int fileId, bool forceAsAttachment)
{
File file = FileHandlerLinq.Select(fileId);
if (file != null)
{
// parse the file name and get the extension, we use a ToUpperInvariant 
// as there are some characters that cannot be converted to lower case
string extension = file.NetworkPath.Substring(
file.NetworkPath.LastIndexOf(
".", StringComparison.OrdinalIgnoreCase
)
).ToUpperInvariant();
if (forceAsAttachment)
{
extension = string.Empty;
}
// in keeping this example simple we simply parse the file extension to determine if
// we should set the file as an attachment or send it back to the browser
switch (extension)
{
case ".JPG":
case ".JPEG":
case ".PNG":
case ".GIF":
case ".BMP":
TransmitFileToBrowser(context, file);
break;
default:
TransmitFileAsAttachment(context, file);
break;
}
}
}
/**//// <summary>
/// Transmits the file as attachment.
/// </summary>
/// <param name="context"><see cref="HttpContext"/>.</param>
/// <param name="file">The file entity class.</param>
private static void TransmitFileAsAttachment(HttpContext context, File file)
{
context.Response.AddHeader("content-disposition", "attachment; filename=" + file.FileName);
context.Response.ContentType = file.ContentType;
context.Response.TransmitFile(file.NetworkPath);
}
/**//// <summary>
/// Transmits the file to browser.
/// </summary>
/// <param name="context"><see cref="HttpContext"/>.</param>
/// <param name="file">The file entity class.</param>
private static void TransmitFileToBrowser(HttpContext context, File file)
{
context.Response.ContentType = file.ContentType;
context.Response.TransmitFile(file.NetworkPath);
}
}
}

DownloadHandler.ashx and ImageHandler.ashx

The Generic ImageHander and DownloadHandlers are almost identical and their sole purpose is to parse the Uri for the FileId token. Once the FileId token is collected the generic handler calls for a file handler from the File Handler Factory.

<%@ WebHandler Language="C#" Class="DownloadHandler" %>
using System;
using System.Web;
public class DownloadHandler : IHttpHandler
{
/**//// <summary>
/// Enables processing of HTTP Web requests by a custom HttpHandler that implements
/// the <see cref="T:System.Web.IHttpHandler"/> interface.
/// </summary>
/// <param name="context">An <see cref="T:System.Web.HttpContext"/> object that provides references to the
/// intrinsic server objects (for example, Request, Response, Session, and Server) used to
/// service HTTP requests.</param>
public void ProcessRequest(HttpContext context)
{
int fileId = 0;
// the download handler expects a file id to be part of the URL QueryString
// I.E. http://someurl.com/DownloadHandler.ashx?FileId=1
if (int.TryParse(context.Request.QueryString["fileId"], out fileId))
{
FileServiceCore.FileHandlerFactory.TransmitFile(context, fileId, true);
}
}
/**//// <summary>
/// Gets a value indicating whether another request can use the <see cref="T:System.Web.IHttpHandler"/> instance.
/// </summary>
/// <value></value>
/// <returns>true if the <see cref="T:System.Web.IHttpHandler"/> instance is reusable; otherwise, false.</returns>
public bool IsReusable
{
get
{
return false;
}
}
}

Default.aspx

The below sample .aspx page shows a common usage to add data to the file service.

<%@ Page Language="C#" AutoEventWireup="true" CodeFile="Default.aspx.cs" Inherits="_Default" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head runat="server">

<title>Untitled Page</title>

</head>

<body>

<form id="form1" runat="server">

<div>

Upload File:

<br />

<asp:FileUpload runat="server" ID="UploadFileControl" /><br />

<asp:Button runat="server" ID="UploadFileButton" Text="Upload File"

onclick="UploadFileButton_Click" /><br />

</div>

</form>

</body>

</html>




Code Behind:
--------------------------------------------------------
using System;

using System.Globalization;

public partial class _Default : System.Web.UI.Page

{

protected void Page_Load(object sender, EventArgs e)

{

}

protected void UploadFileButton_Click(object sender, EventArgs e)

{

if (UploadFileControl.HasFile)

{

// get the acutal file name

string fileName = UploadFileControl.PostedFile.FileName;

fileName = fileName.Substring(fileName.LastIndexOf(@"\", StringComparison.OrdinalIgnoreCase) + 1);

// insert our file and redirect back to this page with the fileId appended to the querystring

FileServiceCore.File fileEntity = FileServiceCore.FileHandlerLinq.Insert(

fileName,

UploadFileControl.PostedFile.ContentType,

UploadFileControl.PostedFile.InputStream);

// redirect back to our page

Response.Redirect(string.Format(CultureInfo.InvariantCulture, "default.aspx?FileId={0}", fileEntity.FileId));

}

}

}

Future Enhancements:

The following list highlights some of the possible enhancements that can be made to the service, depending on your needs:

1. Public/Private flags for files
2. Cache headers for the ImageHandler
3. Resizing/cropping/effects for the ImageHandler
4. Store statistics for file usage (last accessed, times accessed, etc)
5. Demotion of non used files to another storage medium (tape, DVD, etc)
6. File locking, to ensure files aren't accidentally deleted.

Assumptions:

1. There is an available NAS/SAN with enough storage capacity for your needs
2. The IIS account running the File Service has appropriate permissions to read/write to the NAS/SAN

本文转载 kane-nina
http://www.kanebarton.com/Samples/FileService/Default.aspx