Going Native - Using the NT API for File I/O

If you asked a random collection of techno-dweebs to name the system service API that is native to Windows NT chances are the vast majority would say something that eventually translated to "The Win32 API". Those of you properly schooled in NT systems internals will know that this just is not the case. The Win32 API is implemented by a "Client-Side DLL" that is specific to the Win32 Subsystem. The Win32 Subsystem is just one of Windows NT’s Operating System (OS) Emulation Subsystems. All of NT’s OS Environment Subsystems (Win32, POSIX, OS/2, and DOS/WoW) utilize services provided by the Windows NT Executive. These services are accessed by the OS Environment Subsystems via Windows NT’s actual native system service API, which is called "the NT API".

A Special Purpose

 

From the ground up, Windows NT was designed to be an operating system that facilitates the emulation of other operating systems and their APIs. The NT OS itself works hard at providing a robust infrastructure, while not imposing undo constraints on its OS Environment Subsystem clients and their applications. For example, there is no uniform set of wildcards imposed on NT file systems. This allows one to build a file system that includes "*" or "?" in the names of files.

The native NT API was designed for the use of OS Environment Subsystems to provide services to their clients. Thus, when a program running under control of the Win32 OS Environment Subsystem wants to create a new process it uses the Win32 function call CreateProcess(...). The parameters to this function are designed to be easy to use and make sense within the Win32 OS Environment Subsystem’s framework. When the Win32 OS Environment Subsystem receives the call, it can check its own information on the calling process to see if (for example) the caller has the resources and privileges necessary to have the request granted. If all his internal requirements are met, the Win32 OS Environment Subsystem then issues a function call to the native NT API function NtCreateProcess(...) to request Windows NT to create a process on behalf of the user.

Not all client function calls are processed or even seen directly by the OS Environment Subsystem however. If NT’s native handling of a function is close enough to meet the OS Environment Subsystem’s requirements, and no additional protection or resource checks are required by the OS Environment Subsystem, the mapping between the subsystem’s function call and the native NT function call can be performed right in the client-side DLL. This is the case, for example, with file I/O requests to the Win32 subsystem. Win32 appears to rely for the most part on Windows NT’s native protection, quota, synchronization, and handle management schemes. Thus, a Win32 application’s ReadFile(...) call is translated within Win32’s client-side DLL to a call to the native NT function NtReadFile(...). The overhead of a call to the Win32 OS Environment Subsystem is thus saved for this very common, and performance critical, operation.

One of the most interesting things about the NT API is that it has never been comprehensively documented by Microsoft. This must make NT the world’s only commercially available operating system with an undocumented set of native system services! In one way, the existence of the NT API doesn’t really matter to users: Programs can do anything they legitimately need to by using the interface provided by their OS Environment Subsystem. On the other hand, it is through the native interfaces provided by an operating system that we get a feel for how the operating system actually works. Not to mention that in order to build your own reasonably efficient OS Environment Subsystem, you would have to know the native NT API.

Native NT File I/O

 

So, what does the NT API look like? Well, let’s take a look at a few of its most basic function calls.

First, NT’s native function to either access an existing file on disk or create a new such file is the NtCreateFile()function. The prototype for this function appears in Figure 1.

NTSYSAPI NTSTATUS NTAPI NtCreateFile(PHANDLE FileHandle,
        ACCESS_MASK DesiredAccess,
        POBJECT_ATTRIBUTES ObjectAttributes,
        PIO_STATUS_BLOCK IoStatusBlock,
        PLARGE_INTEGER AllocationSize,
        ULONG FileAttributes,
        ULONG ShareAccess,
        ULONG CreateDisposition,
        ULONG CreateOptions,
        PVOID EaBuffer,
        ULONG EaLength
        );

Figure 1 -- NtCreateFile(...)

Notice that the parameters for NtCreateFile(...) are identical to those for the ZwCreateFile(...) function, which is extensively and clearly documented in the Windows NT DDK. This is more than mere coincidence. Each of the native NT System Services comes in a ZwXxxx and NtXxxx variant.

According to NT’s Kernel-Mode Glossary (which contains a wealth of fascinating trivia, by the way) the NtXxxx functions check the supplied parameters and access modes for validity and explicitly set the previous mode to USER mode. The ZwXxxx function variants do not. Hence, NT Drivers call ZwCreateFile(...)when they are opening a file on their own behalf. OS Environment Subsystems (or, indeed native NT applications) would call NtXxxxx since they are calling from user mode.

Since the NT DDK is pretty clear on the meaning of the ZwCreateFile(...) function parameters, we’ll avoid describing each of them here. However, recall that this function is used to access lots of things other than files on disk. In addition to disk files, NtCreateFile() may be used to access a device, partition, directory, or even a socket, a pipe, or a mailslot.

One thing that might at first appear unusual about the NtCreateFile(...) function is that the name of the file to open is not one of the immediate parameters. Rather, the UNICODE_STRING specification for the file appears in the OBJECT_ATTRIBUTES structure supplied in the ObjectAttributes argument. This OBJECT_ATTRIBUTES structure is initialized with the function call InitializeObjectAttributes(...), which is also documented in the NT DDK. The prototype for InitializeAttributes(...) is shown in Figure 2.

VOID InitializeObjectAttributes(POBJECT_ATTRIBUTES InitializedAttributes,
        PUNICODE_STRING ObjectName,
        ULONG Attributes,
        HANDLE RootDirectory,
        PSECURITY_DESCRIPTOR SecurityDescriptor
        );

Figure 2 -- InitializeAttributes(...)

The ObjectName parameter takes a pointer to a UNICODE_STRING which contains either a fully qualified path specification for the file to be opened, or a partial file specification relative to a previously opened directory. If the latter, the RootDirectory contains the handle to that previously opened directory. Note that there is no default directory for the file being opened as there is in Win32.

Using the OBJECT_ATTRIBUTES structure, there are two possible ways to open the file "fred.txt" in the directory C:"temp. Either you build a UNICODE_STRING for ObjectName that contains the fully qualified file path specification, such as ""DosDevices"C:"Temp"fred.txt" or else you first open the directory ""DosDevices"C:"Temp"" via a CreateFile(...) request, obtaining a handle to this open (directory) file. You then build a UNICODE_STRING for ObjectName that contains just the file name portion "fred.txt" and pass it, along with the handle to the open directory to InitializeObjectAttributes(...).

Here, once again, NT shows it’s true colors as an operating system designed to facilitate the emulation of other operating systems. By not providing a default path specification, Windows NT allows it’s OS Environment Subsystems to the path for a file without intrusion by the underlying system. In addition, whether or not the file specification is case sensitive is determined by the Attributes parameter to the InitializeObjectAttributes(...) function. This allows the OS Environment Subsystem to supply its default behavior for that of Windows NT.

By the way, there is also a "short cut" API that can be used to access an already existing file, and hence dispenses with some of the lesser-used parameters. This is the NtOpenFile()function, which appears in Figure 3.

NTSYSAPI NTSTATUS NTAPI NtOpenFile(PHANDLE FileHandle,
        ACCESS_MASK DesiredAccess,
        POBJECT_ATTRIBUTES ObjectAttributes,
        PIO_STATUS_BLOCK IoStatusBlock,
        ULONG ShareAccess,
        ULONG OpenOptions
        );

Figure 3 -- NtOpenFile(...)

Note that things like a buffer for the ExtendedAttributes, and the CreateDisposition are absent from this call, making it quick and easy to code.

With the file opened, and the FileHandle returned, we can now proceed to issue read and write requests to the file.

NT’s native API to read from a file is the NtReadFile(...) function. The function to write to a file is the NtWriteFile(...) function. The prototypes for these two are functions are identical, differing only in name. The prototype for NtReadFile(...)is shown in Figure 4.

NTSYSAPI NTSTATUS NTAPI NtReadFile(HANDLE FileHandle,
        HANDLE Event,
        PIO_APC_ROUTINE ApcRoutine,
        PVOID ApcContext,
        PIO_STATUS_BLOCK IoStatusBlock,
        PVOID Buffer,
        ULONG Length,
        PLARGE_INTEGER ByteOffset,
        PULONG Key
        );

Figure 4 -- NtReadFile(...)

This function is also documented in the NT DDK in it’s ZwReadFile(...) variant. However, the documentation is not quite as extensive as it might be. Some notes on each of the parameters are shown in List 1.

Arguments to NtWriteFile(...)

FileHandle

The file handle returned from the CreateFile(...) call. If you have to ask, you’re reading the wrong publication;

Event

Handle to a previously created event, to use for synchronization. NTOS sets the state of this event to Signalled when the request is complete.

ApcRoutine

An optional pointer to a user Asynchronous Procedure Call (APC) function to be called when the request completes. This is what Win32 refers to as a FileIoCompletionRoutine(...). This function will only be called if the calling thread is in an alertable wait state. A wait is "alertable" if the Alertable argument to the wait function (such as KeWaitForSingleObject(...)) is set toTRUE.

ApcContext -- If ApcRoutine was supplied, above, this is a context argument passed to that function when it is called.

IoStatusBlock  -- A pointer to an IO_STATUS_BLOCK to receive the result for the operation. The returned data is not valid until the request completes.

Buffer -- Pointer to the user buffer for the operation;

Length -- Length in bytes of Buffer;

ByteOffset -- An optional pointer to a LARGE_INTEGER containing the byte offset into the file at which this operation is to begin.

Key -- An optional value for a key, corresponding to a previously granted lock taken out on the file.

List 1 -- Parameters

Beyond obviously writing to or reading from the file handle provided, the precise behavior of the NtReadFile(...) and NtWriteFile(...) call depends on the values supplied during the NtCreateFile(...). For example, if during the NtCreateFile(...) operation the "Synchronize" flag set in the DesiredAccess parameter, and one of the FILE_SYNCHRONOUS_IO_xxxx flags has been specified in the CreateOptions, calls to NtReadFile(...) and NtWriteFile(...) will complete synchronously and the I/O system will track the current read/write offset in the file.

There are two small differences between the NtReadFile(...)function and Win32’s ReadFile(...) and ReadFileEx(...) functions. While these differences are far from earth shattering, they still serve as an interesting example of how an OS Environment Subsystem customizes its interface to meet its users expectations or needs.

In NtReadFile(...) there is an explicit Key parameter. This Key is a value supplied by the user during a previously successful NtLockFile(...) function call. Supplying this Key allows the user to bypass a lock previously taken out on a specific region of the file. In the Win32 ReadFile(...) function, no Key value can be supplied. If a Win32 process has taken out a lock on a file, the Key for that lock is apparently automatically supplied by the Win32 Subsystem’s Client-Side DLL. Similarly, Win32 does not allow the user to specify the ApcContext, choosing to uniformly supply a pointer to its OVERLAPPED structure automatically instead.

A neat facility available through the native NT API but not through Win32 is the ability to cancel all I/O requests that you have outstanding on a particular file handle. The function to accomplish this is in Figure 5.

NTSYSAPI
NTSTATUS NTAPI NtCancelIoFile(HANDLE FileHandle,PIO_STATUS_BLOCK IoStatusBlock);

Figure 5 -- NtCancelIoFile()

If you’ve ever wondered how you accomplish an I/O cancel operation, now you know!

Finally, to close a previously opened file handle, the NtClose(...) function is used, also shown in Figure 6. 

NTSYSAPI NTSTATUS NTAPI NtClose(HANDLE HandleToClose);

Figure 6 – NtClose(...)

The operation of this call is similar to that of the Win32 function CloseHandle(...) function. Again, the differences between the native NT call and the Win32 function can be seen: The Win32 function returns a BOOLEAN and raises an exception (ugh!) if an invalid handle value is provided. The native NT function simply returns an NTSTATUS value that indicates the outcome of the call.

Building Native NT Programs

 

While all this new-found knowledge might be interesting, it would all be totally academic if you couldn’t actually USE it. So how do you actually build a native NT program? Well, the list below shows how we do it at OSR:

• You’ll need to define the function prototypes. Each of the NT API functions needs to be defined as shown above.
• When you compile, you’ll need to include ntddk.h, or else redefine the many types and structures that are found only there and are required for these functions. For example, the structure OBJECT_ATTRIBUTES appears to be defined in ntddk.h and nowhere else.
• You’ll need to link against ntdll.lib, which resolves your function references to calls into ntdll.dll.
• Want to free yourself completely of subsystem control? When you link your program, define the subsystem under which it runs to be "native" (using the /Subsystem:native linker option).

And that’s all there is to it! If you’d like to grab a very simple sample application that writes "hello world" to a file using the native NT API, you can download it from our web site.

Just Because You Can

 

So, is this the new way we should all write our user-mode code from now on? What does one achieve by "going native" and directly using the Windows NT system service API?

Well, you could certainly achieve a lot of hassles. Since the NT API is neither documented nor supported, there’s nobody at Microsoft you can complain to if the interface changes arbitrarily or doesn’t work as (not) advertised. Plus, with the absence of a default directory for your file opens you get to manually specify the path name to each of the files you want to open. Remember, the NT API really wasn’t designed to be an end-user interface.

Is there any good reason at all, then, to use the NT API directly? Well, of course. For one, there’s the ability to cancel I/O requests, which you can’t even get at from Win32. Playing with the NT API also gives you a window into the handiwork of the NT developers. So much of NT is buried underneath other stuff, much of which might be considered pretty ugly by comparison. Using the NT API helps you get a better "feeling" for what NT itself does as an operating system, and how it works. Also, we have in our travels found at least one real application that required the use of the NT API.