Working with memory mapped files in .NET 4
MSDN Utopia
- December 2011 (2)
- February 2011 (3)
- January 2011 (2)
- July 2010 (1)
- March 2010 (1)
- July 2009 (1)
- June 2009 (2)
- April 2009 (1)
- February 2009 (1)
- December 2008 (1)
- October 2008 (1)
- September 2008 (1)
- August 2008 (1)
- June 2008 (1)
- May 2008 (3)
- April 2008 (1)
- March 2008 (1)
- February 2008 (1)
- January 2008 (2)
- November 2007 (1)
- October 2007 (1)
- August 2007 (1)
- July 2007 (2)
- June 2007 (1)
- May 2007 (1)
- April 2007 (3)
Working with memory mapped files in .NET 4
Working with memory mapped files in .NET 4
SalvaPatuel
MSFT
I have been exploring different new features that come with the .NET 4, beyond the most popular ones like dynamic types and covariance; I was interested in performance enhancements. For this reason I am going to publish a couple of blog entries were I explore these different features.
Memory mapped files may sounds alien to the managed code developer but it has been around for years, what is more, it is so intrinsic in the OS that practically any communication model that wants to share data uses it behind the scenes.
So what is it? A memory mapped file allows you to reserve a region of address space and commit physical storage to a region (hmmm, sounds like virtual memory, isn’t it?) but the main difference is that the physical storage comes from a file that is already on the disk instead of the memory manager. I will say that it has two main purposes:
- · It is ideal to access a data file on disk without performing file I/O operations and from buffering the file’s content. This works great when you deal with large data files.
- · You can use memory mapped files to allow multiple processes running on the same machine to share data with each other.
The memory mapped file is the most efficient way for multiple processes on a single machine to communicate with each other. What is more, if we check other IPC methods we can see the following architecture:
Impressive, isn’t it? Now you have the power of this technology available on the System.IO namespace (instead of using the Pinvoke approach).
Now let’s quickly explore how it works. We have two types of memory mapped files models, one model is using a custom file, this can be any file that the application accesses it and the other one using the page file that we are going to share it with the memory manager (this is the model that most of the technologies above use).
Let’s explore the custom file model. The first thing that we need to do is to create a FileStream to the file that we are going to use, this can be an existing file or a new file (keep in mind that you should open this file as shared, otherwise no other process will be able to access it!). With the stream in place, we can now create the memory mapped file. Let’s see an example:
using System.IO.MemoryMappedFile;
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateFromFile(
new FileStream(@"C:\temp\Map.mp", FileMode.Create), // Any stream will do it
"MyMemMapFile", // Name
1024 * 1024, // Size in bytes
MemoryMappedFileAccess.ReadWrite); // Access type
I
have use one of the simplest constructor, we define the stream to use and we
provide a name. The object needs to know the size of it in bytes and the type of
access that we need. This will create the memory mapped file but to start using
it we will need a map view. We can create one using the following
syntax:
MemoryMappedViewAccessor FileMapView = MemoryMapped.CreateViewAccessor();
This map covers the file from the first byte until the end. If we need now to write or read information from it we just call the map view methods with the correct offset.
int Number = 1234;
FileMapView.Write(0, Number);
FileMapView.Write<Container>(4, ref MyContainer);
We can see that we can write built in types or custom types with the generic version. The good thing about the memory mapped files is persistence, as soon as you close it the contents will be dumped on the disk, this is a great scenario for sharing cached information between applications.
Now if we want to read from it, the other process needs also to create a memory mapped file, we can use the other static initialize that opens an existing one or creates one if it does not exist.
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateOrOpen(
new
FileStream(@"C:\temp\Map.mp",
FileMode.Create), // Any stream will do
it
"MyMemMapFile", // Name
1024 * 1024, // Size in bytes
MemoryMappedFileAccess.ReadWrite); // Access type
Create the map view and read it:
using (MemoryMappedViewAccessor FileMap = MemoryMapped.CreateViewAccessor())
{
Container NewContainer = new Container();
FileMap.Read<Container>(4, out NewContainer);
}
That’s it, really simple isn’t it? Now, there is a small drawback
with this approach and is related to the size of the memory mapped file. If you
don’t know in advance you will need to create a large file
“just in case”,
this can be a very large file with a lot of wasted space, it would be nice to
have the ability to reserve the space instead of committing all of it, isn’t
it?
In order to solve this issue you can use the page file, this has the advantage of allowing you to commit data on the fly but introduces another issue: you don’t own the file and the map will last until the last handle is destroyed. But think about it, for certain scenarios this is absolutely valid.
Now if we revisit the sample we will need to change some initialization parameters, for this particular one I will use the full constructor so I can introduce some other features that are also applicable to the custom files one.
MemoryMappedFileSecurity CustomSecurity = new MemoryMappedFileSecurity();
MemoryMappedFile PagedMemoryMapped = MemoryMappedFile.CreateNew(
@"Salvador", // Name
1024 * 1024, // Size
MemoryMappedFileAccess.ReadWrite, // Access type
MemoryMappedFileOptions.DelayAllocatePages, // Pseudo reserve/commit
CustomSecurity, // You can customize the security
HandleInheritability.Inheritable); // Inherit to child process
The memory mapped file security allows you to customize who or
which process can have access to the resource, this can be quite important when
you want to protect sensitive information and you don’t want other processes
changing the file map. You can explore that object and see all the different
settings that you can change. The reference is here. If we explore the
construction of the memory mapped file we can see that there is no stream, we
just name the resource. This will create an association between a section of the
file based on the size and the map name, this is how both processes will access
the file. Note as well that I am setting the property “DelayAllocatePages”, this
implements a pseudo reserve/commit model where we will use the space once is
needed. Finally, the other interesting parameter is the handle inheritance
model, this will allow to share this resource with a child process if is
required.
The access to the file uses the same syntax as the previous example, remember that if you close the memory mapped file this will be non accessible, this issue catches many developer.
Finally, another interesting area is the creation of multiple map views, these can work on the same memory mapped file accessing different areas of the files. This will allow you to properly protect the content and allowing you to synchronize the access.
You can do this defining different offsets and lengths when you create your map view.
MemoryMappedViewAccessor WriteMap =
MemoryMapped.CreateViewAccessor(0, 1024,
MemoryMappedFileAccess.ReadWrite);
MemoryMappedViewAccessor ReadMap =
MemoryMapped.CreateViewAccessor(1025, 1024,
MemoryMappedFileAccess.Read);
Now you can enjoy the power of this high performance data sharing model on your applications when you compile them with .NET 4.0! Note that this blog is based on beta 1 information, I will check the post once is released.
-
kurakuraninja10 Jun 2009 1:39 AM#
PingBack from http://blog.cwa.me.uk/2009/06/10/the-morning-brew-365/
-
kurakuraninja21 Jun 2009 5:58 AM#
.Net Framework 4.0 introduces memory mapped files. Memory mapped files are useful when you need to do
-
Cameron24 Jun 2009 11:07 AM#
Is it possible to use memory mapped files in VS2008, with the Net Framework 4 beta 1 installed on a box?
Thank you,
Great article
-
SalvaPatuel
SalvaPatuel
MSFT
4,795 Points 8 4 0Recent AchievementsBlog Conversation Starter Blogger II New Blog Rater24 Jun 2009 1:32 PM#Hi Cameron,
VS2008 does not know how to handle .NET 4 unfortunately. Therefore the IDE can not use the new compiler. You can write the code but you will need to manually call the VS2010 compiler to link it to .NET 4.
You can still use memory mapped files in VS2008 using PInvoke, but I imagine that is not the solution that you are looking after.
Regards,
Salvador
-
E. Scott Anderson20 Aug 2009 9:27 AM#
Salva,
Could you please elaborate more on how to "manually call the VS2010 compiler to link it to .NET 4." I am thinking that I would like to take the approach where I write a DLL in VS2010 .NET 4.0 to handle the reading-writing/caching of the data in the memory mapped file and reference this DLL in my VS2008 .NET 3.5 project. The reason is that my application is heavily dependent on MS Robotics Studio which is not 4.0 ready, but I need to process very large image data sets.
Thanks,
Scott
-
SalvaPatuel
SalvaPatuel
MSFT
4,795 Points 8 4 0Recent AchievementsBlog Conversation Starter Blogger II New Blog Rater24 Aug 2009 5:54 PM#Hi E. Scott Anderson
The short answer is you can't. You can do it the other way around as .NET 4 allows you to run side-by-side libraries (i.e running a .NET4 app and referencing a .NET 2/3/3.5 DLL). The problem is that the process running under CLR 2 (.net 2/3/3.5) does not know how to interpret a DLL compiled under CLR 4.
You can still access the memory mapped files using the PInvoke model, it works very well.
You can find more info here: http://msdn.microsoft.com/en-us/library/ms810613.aspx
Regards
-
John Ashman30 Mar 2010 2:01 AM#
Do you know if memory mapped files will work for a file on a distributed file system?
-
SalvaPatuel
SalvaPatuel
MSFT
4,795 Points 8 4 0Recent AchievementsBlog Conversation Starter Blogger II New Blog Rater30 Mar 2010 6:57 AM#Hi John,
No, it does not work with UNC or remote drives as is for same machine process intercom.
Regards
-
Jetha Lal9 Jul 2010 12:19 PM#
What is memory mapped file?
Is this new technology?
Never heard before.
-
Sunil K17 Oct 2010 6:29 AM#
Hi,
if MemoryMappedFile is created from <b>dbData.mdb</b> (1.5GB) file.
MemoryMappedFile MemoryMapped = MemoryMappedFile.CreateFromFile(FilePath);
MemoryMappedViewAccessor FileMap = MemoryMapped.CreateViewAccessor())
how do i put this in this connection string.
Microsoft.Jet.Oledb.4.0; Data Source=??
-
Tobias Persson21 Oct 2010 1:34 PM#
@Jetha
No, it's not really a new technique (though I haven't come across it before either). The MSDN article Salva linked to are more than 17 years old, so it has been around since the very first version of Windows NT! :-)
-
Microsoft Lover19 Jan 2011 5:07 PM#
So ,in memory maped file, can user updates in the file also?
If so, does it controls the versions?
-
Security???11 May 2012 10:12 PM#
What is the maximum size of the mapped file. Is it bounded or unbounded?
-
Chris31 Jul 2012 7:45 AM#
Does anyone know for sure if when you pull from MMF, does the local App process create it's own memory storage for the contents in the shared memory? or does it reference the memory directly?
I ask this question because I am in thinking about using a 4gb file in MMF, and if it pulls down a local memory for each app(process) that uses the resource, I will be looking at a lot of RAM. Any ideas?
-
Rama Naresh9 Jan 2013 11:20 AM#
1. I believe it is only sticking into memory chunks/pages of a file at a time. How does it release it back ? How does it know when we are done?
2. Should we still make a copy of the original file and put it in a temp location then read it into mmf from there or is it ok to just read it from the original now? (how does that play if someone overwrites the original as we are reading from it?
3. How do we ensure the mmf is always using the latest since we might constantly be updating a particular file?
4. What is the server hit of using this? I imagine its better than not using it but want to know more.
-
dzmitry.lahoda
dzmitry.lahoda
0 Points 5 0 0Recent AchievementsNew Blog Commentator New Blog Rater First Forums Reply13 Jan 2013 1:05 AM#I tested 2 apps - one sending WM_COPYDATA (Windows Messaging) and other with out of proc COM server consumed by VB script. Both from All in One Code Framework. I second case Process Explorer shows handle of "ALPC port" with name "\RPC Control\OLE44EE3FE7D3E04B7EA505A092387B". Seems it is en.wikipedia.org/.../Local_Procedure_Call. It differs from Named Shared Memory and "The memory mapped file is the most efficient way for multiple processes on a single machine to communicate with each other. " not always true. And COM/OLE and may be RPC uses not only shared memory but ALPC either.
