Pardon the Interruption -- Interrupt Handling Changes in Windows
For the first time in the history of NT, there are some big changes coming to interrupt handling. In this article, we’ll take a brief look at what some of those changes are planned to be. The timeframe for these changes taking effect is sometime after Windows Server 2003. When precisely? Your guess is as good as ours.
As with every other article in this issue of The NT Insider, we’ve tried to ensure that the information we provide here is correct as of press time. However, everything in this article is entirely speculative and totally subject to change. Don’t blame us if the future turns out differently. That’s the nature of the beast.
As a final caveat, please understand that we’re only able to scratch the surface of this topic in this issue of The NT Insider. We’ll have lots more to say on this topic. In the meantime, you can refer to the WinHEC paper that’s published on this topic. It’s pretty comprehensive.
The two most significant changes are the retirement of the classic function IoConnectInterrupt and addition of support for a new set of interrupt mechanisms. The new interrupt types are called Message Signaled Interrupts. Let’s look at each of these changes in turn.
IoConnectInterrupt – GONE!?
The first change we’ll discuss is likely to be a big surprise. The function IoConnectInterrupt will be deprecated, and a new function named IoConnectInterruptEx is taking its place. And yes, there’s also a corresponding IoDisconnectInterruptEx that takes the place of IoDisconnectInterrupt as well.
IoConnectInterrupt has long had the (dubious) honor of being one of Windows DDIs with the most parameters. And those parameters had often caused new Windows driver writers difficulties. For example, what do you pass as the KAFFINITY value to IoConnectInterrupt? Can you change the affinity bits that are passed to you in your interrupt’s CM_PARTIAL_RESOURCE_DESCRIPTOR (Answer: NO! NEVER!)? And how about that ShareVector parameter to IoConnectInterrupt? If you want to avoid interrupt vector sharing, can you just set this parameter to FALSE (the answer is, again NO!)?
An then there’s that lingering question that you’ve probably asked yourself while you’re deep into coding your driver’s interrupt handling paths: If the HAL/PnP Manager/Bus Arbiter gave me this resource information in the first place, and I can’t validly change any of it, why the heck do I have to collect it and then give all the same information back to him when I want to connect to the interrupt?
At least in part, in answer to these issues, Microsoft has given us the new DDI IoConnectInterruptEx. The prototype for this function is shown in Figure 1.
NTSTATUS
IoConnectInterruptEx(IN OUT PIO_CONNECT_INTERRUPT_PARAMETERS Parameters);
Figure 1 – IoConnectInterruptEx
Note the new function takes a single parameter as input. This parameter is a pointer to an IO_CONNECT_INTERRUPT_PARAMETERS data structure that’s been allocated by the driver and appropriately initialized. This data structure contains several individual data structures within a union. Each of these individual data structure describes a separate method of calling IoConnectInterruptEx. Figure 2 shows the format of the IO_CONNECT_INTERRUPT_PARAMETERS data structure.
typedef struct _IO_CONNECT_INTERRUPT_PARAMETERS {
IN OUT ULONG Version; // Max Version == 3
union {
// LINE_BASED Version
struct {
IN PDEVICE_OBJECT PhysicalDeviceObject;
OUT PKINTERRUPT *InterruptObject;
IN PKSERVICE_ROUTINE ServiceRoutine;
IN PVOID ServiceContext;
IN PKSPIN_LOCK SpinLock;
IN KIRQL SynchronizeIrql;
IN BOOLEAN FloatingSave;
} LineBased;
// MESSAGED_BASED Version
struct {
IN PDEVICE_OBJECT PhysicalDeviceObject;
union {
OUT PVOID *Generic;
OUT PIO_INTERRUPT_MESSAGE_INFO *InterruptMessageTable;
OUT PKINTERRUPT *InterruptObject;
} ConnectionContext;
IN PKMESSAGE_SERVICE_ROUTINE MessageServiceRoutine;
IN PVOID ServiceContext;
IN PKSPIN_LOCK SpinLock;
IN KIRQL SynchronizeIrql;
IN BOOLEAN FloatingSave;
IN PKSERVICE_ROUTINE FallBackSerivceRoutine OPTIONAL;
} MessageBased;
// FULLY_SPECIFIED Version
struct {
IN PDEVICE_OBJECT PhysicalDeviceObject;
OUT PKINTERRUPT *InterruptObject;
IN PKSERVICE_ROUTINE ServiceRoutine;
IN PVOID ServiceContext;
IN PKSPIN_LOCK SpinLock;
IN KIRQL SynchronizeIrql;
IN BOOLEAN FloatingSave;
IN BOOLEAN ShareVector;
IN ULONG Vector;
IN KIRQL Irql;
IN KINTERRUPT_MODE InterruptMode;
IN KAFFINITY ProcessorEnableMask;
} FullySpecified;
};
} IO_CONNECT_INTERRUPT_PARAMETERS, *PIO_CONNECT_INTERRUPT_PARAMETERS;
Figure 2 – IO_CONNECT_INTERRUPT_PARAMETERS Structure
It might look hideously obtuse and complicated at first, but it’s actually pretty easy. Let’s say you want to connect your Interrupt Service Routine (ISR) to a regular (line-based) PCI type interrupt. The code you’d use is shown in Figure 3.
RtlZeroMemory( ¶ms, sizeof(IO_CONNECT_INTERRUPT_PARAMETERS) );
// Which version of the structure we’re using...
params.Version = CONNECT_LINE_BASED;
// pointer to our driver’s PDO
params.LineBased.PhysicalDeviceObject = PhysicalDeviceObject;
// where to store the returned PKINTERRUPT Object
params.LineBased.InterruptObject = &devExt->IntObj;
// Pointer to our driver’s ISR
params.LineBased.ServiceRoutine = MyDeviceIsr;
// Context to pass into our driver’s ISR (our FDO)
params.LineBased.ServiceContext = FunctionalDeviceObject;
// spin lock pointer, synch IRQL, and floating save
params.LineBased.SpinLock = NULL;
params.LineBased.SynchronizeIrql = 0;
params.LineBased.FloatingSave = FALSE;
status = IoConnectInterruptEx(¶ms);
Figure 3 – Connecting an ISR to a Line-Based PCI-type Interrupt
There are a couple of interesting things to notice about the Figure 3 example. This example uses the CONNECT_LINE_BASED flavor of IoConnectInterruptEx. As a result, the driver sets the Version field to CONNECT_LINE_BASED and fills in the fields in the LineBased part of the larger IO_CONNECT_INTERRUPT_PARAMETERS data structure. Note that we don’t have to give IoConnectInterruptEx any description of our interrupt resource. You don’t need to do this because, and this is the second interesting thing to note, we supply a pointer to the PDO that’s associated with the interrupt. This pointer is now required for all calls made to IoConnectInterruptEx. Finally, note that IoConnectInterruptEx requires roughly half the input parameters as traditional IoConnectInterrupt. This is a good thing.
Did I hear you say that sometimes you need the same level of control as IoConnectInterrupt currently provides? That is, at certain times and in some drivers you still need the ability to connect to a specific interrupt resource? Well, fear not. In this case, you user the CONNECT_FULLY_SPECIFIED flavor and fill in the FullySpecified structure. Note that if you use CONNECT_FULLY_SPECIFIED, your driver will work all the way back to Windows 2000 (assuming you build it appropriately).
Message Signaled Interrupts
While the replacement of IoConnectInterrupt may be a surprise to long-time Windows driver writers, the interrupt architecture change that is likely to have the biggest impact on systems world-wide is support for Message Signaled Interrupts. Message Signaled Interrupts are a new interrupt, optional, mechanism described starting in the PCI V2.2 and later specifications.
Message Signaled Interrupts are a hardware facility that allows a device to generate interrupts by writing a pre-determined16-bit “message” to a specific memory location. Message signaled interrupts are intended for use as a replacement for typical line-based interrupts – A device may not use both Message Signaled Interrupts and traditional line-based interrupts simultaneously.
What makes Message Signaled Interrupts such a big deal? Well, there are actually quite a few reasons. Chief among them are:
- Each device may have multiple Message Signaled Interrupts. These messages can be distinguished by the driver by the MessageID parameter passed into the driver’s Interrupt Message Service Routine (IMSR – The IMSR is the Message Signaled Interrupt equivalent of an ISR).
- Using Message Signaled Interrupts eliminates interrupt sharing. In case you haven’t noticed, interrupt sharing can cause all sorts of annoying problems (from interrupt storms to unacceptable interrupt latency). See the March/April issue of The NT Insider, and the Microsoft White Paper cited in that issue, for more information.
Both of these features can be pretty important. Of course, a device and driver that support Message Signaled Interrupts don’t have to use multiple interrupts. Many devices will simply use Message Signaled Interrupts as a replacement for traditional line-based interrupts, to avoid interrupt sharing.
Because a device and driver can utilize multiple Message Signaled Interrupts, a device can use particular interrupts to signal back the occurrence of specific events to the driver. The driver’s IMSR runs, and the driver can determine what happened on the device (in some cases) without having to touch any of the device’s registers. This avoids the I/O bus serialization inherent in such an action.
Even more interesting about having multiple messages are the possibilities allowed by the type of Message Signaled Interrupt facility called MSI-X (first supported in V2.3 of the PCI spec via ECN). MSI-X allows each Message Signaled Interrupt to be targeted to a specific group of one or more processors. So, not only could a device generate an interrupt indicating that a specific condition occurred, but it could also generate that interrupt on an optimal group of processors.
It’s not hard to see real-life uses for the multiple message scenario. Consider a device that has three queues resident in host memory:
· An outgoing message queue
· An incoming message queue
· A buffer supply queue (that is, a queue of buffers that are awaiting to be filled with incoming data)
A device and driver using Message Signaled Interrupts could utilize different messages to signal each queue that needed service. A forth message might be used to indicate some sort of general error condition. Note that in many cases, the driver’s response to these interrupts might be nothing more than storing the MessageID that caused the interrupt and queuing a DPC. The driver could potentially accomplish this without any hardware access within its ISR.
Now, extend this scenario to a high-end NUMA machine with MSI-X support. In this case, the device could not only signal (for example) that an incoming message was queued, but it could also send that interrupt specifically to the NUMA node where the memory into which the message was stored is near (local).
Before leaving the general topic of Message Signaled Interrupts, it’s important for us to mention that not all system hardware has Message Signaled Interrupt support. Because support for Message Signaled Interrupt is an option, even systems that are PCI V2.2 or PCI V3.0 compliant might not support Message Signaled Interrupts. For example, as of this writing, we are not aware of any AMD-based systems that support Message Signaled Interrupts.
The reason this fact is important is that drivers that support Message Signaled Interrupts must always be written to properly “fall back” to supporting traditional line-based interrupts. Fortunately, the CONNECT_MESSAGE_BASED flavor of IoConnectInterruptEx makes it pretty easy to do this. We’ll discuss more about this in the next section.
Putting It Together
Obviously, an important motivation for replacing IoConnectInterrupt with IoConnectInterruptEx was support for Message Signaled Interrupts. Connecting to a Message Signaled Interrupt using IoConnectInterruptEx is almost identical to connecting to a traditional line-based interrupt. The main difference is that you specify a pointer to your driver’s Interrupt Message Service Routine (IMSR) instead of an Interrupt Service Routine. The prototype for the IMSR is:
BOOLEAN
MyISR(IN struct _KINTERRUPT *Interrupt,
IN PVOID ServiceContext,
IN ULONG MessageID);
Note that except for the addition of the MessageID parameter, the parameters are the same as an ISR. If only one Message Signaled Interrupt is used, the MessageID parameter will always be set to one.
There are several different ways that a driver can connect to Message Signaled Interrupts. The easiest is to use the CONNECT_MESSAGE_BASED flavor of IoConnectInterruptEx, and thus fill in the MessageBased part of the IO_CONNECT_INTERRUPT_PARAMETERS structure. If there are multiple Message Signaled Interrupts allocated to the device, this call connects the driver to all of them.
As previously mentioned, devices and drivers that support Message Signaled Interrupts also must be able to “fall back” to support standard line-based interrupts. This enables them to run on systems where the hardware doesn’t have Message Signaled Interrupt support. The CONNECT_MESSAGE_BASED flavor of IoConnectInterruptEx provides the ability to automatically fall back to a line-based interrupt – and connect a standard ISR to the line-based interrupt – in those situations. Figure 4 (reproduced, with permission, from the WinHEC Whitepaper on this topic) shows an example of calling IoConnectInterruptEx for a Message Signaled Interrupt.
RtlZeroMemory( ¶ms, sizeof(IO_CONNECT_INTERRUPT_PARAMETERS) );
// Flavor – We’re connecting to Message Signaled Interrupts
params.Version = CONNECT_MESSAGE_BASED;
//
// Provide the pointer to the PDO that identifies the
// device that is associated with this interrupt
//
params.MessageBased.PhysicalDeviceObject = PhysicalDeviceObject;
//
// Storage for pointer returned by IoConnectInterruptEx. Note
// that this could be either a pointer to an
// IO_INTERRUPT_MESSAGE_INFO structure or a pointer to a KINTERRUPT
// (interrupt object). Which one is returned depends on whether
// our device has be granted any Message Signaled Interrupts or
// if we fall back to using a single line-based interrupt.
//
// To accommodate the return of either one, we fill in the
// “generic” field, which is just a PVOID pointer. We’ll figure
// out what was returned based on the value returned in the
// Version field by IoConnectInterruptEx.
//
params.MessageBased.ConnectionContext.Generic =
&devExt->ConnectionContext;
//
// Pointer to driver’s Interrupt Message Service Routine (IMSR)
// and the context to be passed into that IMSR.
//
params.MessageBased.MessageServiceRoutine = MyDeviceMessageRoutine;
params.MessageBased.ServiceContext = FunctionalDeviceObject;
//
// In case we’re running on a system that doesn’t have support
// for Message Signaled Interrupts in its hardware, we also
// supply a pointer to a standard Interrupt Service Routine. This
// allows IoConnectInterruptEx to automatically fall-back to
// connecting our ISR to our line-based interrupt in this case.
//
// Note: If Message Signaled Interrupts ARE granted to our device,
// this ISR *is not used*
//
params.MessageBased.FallBackServiceRoutine = MyDeviceIsr;
//
// Because our device requests, and is likely to be granted,
// multiple messages, we use a single spin lock to serialize
// execution of all invocations of our IMSR.
//
// This lock is optional, and typically it's used if shared
// device registers or data structures are accessed within the
// IMSR. By not specifying a Synchronize IRQL,
// IoConnectInterruptEx will automagically compute the
// Synchronize IRQL for us, which is rather handy.
//
KeInitializeSpinLock(&devExt->IMSRLock);
params.MessageBased.SpinLock = &devExt->IMSRLock;
params.MessageBased.SynchronizeIrql = 0;
//
// No need to save the FP registers around our ISR
//
params.MessageBased.FloatingSave = FALSE;
//
// Connect to the interrupt
//
status = IoConnectInterruptEx(¶ms);
if (!NT_SUCCESS(status)) {
//
// The call to IoConnectInterruptEx failed.
// This is probably a fatal error, caused by
// a bug in our driver
//
... // clean up
return(status);
}
//
// The call to IoConnectInterruptEx SUCCEEDED
//
//
// Save the type of interrupt connected. We’ll use this
// and also the returned connection context later when we
// need to disconnect from the interrupt.
//
devExt->Version = params.Version;
// Check to see... Did we connect to any Message Signaled
// Interrupts, or did we have to fall back to using
// traditional line-based interrupts? IoConnectInterruptEx
// sets the Version field to indicate the interrupt type
// to which we actually were connected.
//
if(params.Version == CONNECT_MESSAGE_BASED) {
PIO_INTERRUPT_MESSAGE_INFO msgInfo;
//
// Because we succeeded in connceting to one or more Message Signaled
// Interrupts, the connection context that was returned was
// a pointer to an IO_INTERRUPT_MESSAGE_INFO structure.
//
msgInfo = (PIO_INTERRUPT_MESSAGE_INFO)devExt->ConnectionContext;
devExt->MessageUsed = TRUE;
//
// Get count of Message Signaled Interrupts we were granted
// Note: Even though our device requests more, we could be granted
// only 1 Message Signaled Interrupt in certain situations.
// Deal with that here...
//
devExt->MessageCount = msgInfo->MessageCount;
... // other processing here
} else {
ASSERT(params.Version == CONNECT_LINE_BASED);
devExt->MessageUsed = FALSE;
devExt->MessageCount = 0;
}
Figure 4 – Calling IoConnectInterruptEx For A Message Signaled Interrupt
There are a few important things that you should notice in the example in Figure 4. First, notice that both a pointer to an IMSR and a pointer to an ISR are specified. IoConnectInterruptEx will automatically connect all the device’s Message Signaled Interrupts to the IMSR if Message Signaled Interrupts were granted to the device, and will connect the device’s line-based interrupt to the ISR if Message Signaled Interrupts are not available. The IMSR pointer is filled into the MessageBased.MessageServiceRoutine field. The ISR pointer is filled into the MessageBased.FallbackServiceRoutine field.
Also notice that the driver checks the value in the Version field of the IO_CONNECT_INTERRUPT_PARAMETERS structure after calling IoConnectInterruptEx. This field is used as both an input and an output. Before returning, IoConnectInterruptEx sets this field to indicate whether the line-based interrupt or Message Signaled Interrupt was connected.
Allowing for this type of fall-back introduces an interesting problem in handling the data returned by IoConnectInterruptEx. When a line-based interrupt is connected, IoConnectInterruptEx returns a PKINTERRUPT. When a Message Signaled Interrupt is connected, IoConnectInterruptEx returns a PIO_INTERRUPT_MESSAGE_INFO. This is significant because (a) the driver needs the pointers to these structures to be able to call IoDisconnectInterruptEx, and (b) the IO_INTERRUPT_MESSAGE_INFO structure is not opaque (as is the KINTERRUPT Object) and has information that the driver may want to access.
To provide for this difference, IoConnectInterruptEx rather cleverly provides the ConnectionContext.Generic field, which is of type PVOID. The driver knows which data structure the returned pointer points to by examining the value returned in the Version field.
In Summary
Well, at the start of this article, we warned you that we would barely have the opportunity to scratch the surface of this topic. There’s a lot more to talk about and explain. You can be sure this is a topic that’ll get lots of covered in upcoming issue of The NT Insider.
