DSCP: Domain to Service Processor Communication Protocol
DSCP: Domain to Service Processor Communication Protocol
By Bob Hueston on May 16, 2007
Shared memory has long been a common method for inter-processor communication. Back in the 1980's I worked on embedded systems that used shared RAM to allow a host processor to communicate with digital signal processors (DSPs). The shared RAM was partitioned into mailboxes, with a pair of mailboxes per "application". The host processor would place a command in the incoming mailbox for a given DSP application and signal an interrupt to that DSP. The DSP interrupt service routine would check the mailboxes, find the new command, give it to the application for processing, and place the response in the outgoing mailbox, then interrupt the host processor.
Sun Fire 6800/6900: Shared RAM Mailbox
Years later, when I came to work at Sun I found the same approach was used to allow the embedded service processor (aka, system controller) to send commands to Solaris and get responses back. For example, the Sun Fire 6800/6900 family of servers used this approach. Each domain (a group of UltraSPARC CPUs running a single instance of Solaris) had a separate shared memory with the service processor (SP). The interface between the SP and one Solaris domain looked something like this:| SP | Domain(s) | |
| Application(s) | Application(s) | |
| | ioctls | | | ioctls | | |
| Mailbox Driver | Mailbox Driver | |
| | | | | |
| Hardware Shared RAM Interrupts | ||
While the architecture is efficient and effective, one weakness is that is doesn't scale with applications. As new applications are introduced (for example, Dynamic Reconfiguration or Fault Management), a new mailbox needs to be carved out, and a new protocol invented.
Sun Fire 15K/25K: Internal Ethernet
The Sun Fire 15000/25000 servers improved on the basic mailbox approach by running a separate Ethernet connection from the SP to each and every domain (actually, to every expander). The mailbox was used for low-level operation (running POST, booting to OpenBoot, simulating a serial console, etc). But the Ethernet connection, called the Maintenance Area Network, was used for application-level communication. This allowed applications such as Dynamic Reconfiguration to be developed using standard APIs such as sockets and multiplexed using TCP/IP, which helped ease their development. When a new application was rolled out, we didn't need to carve-out a new mailbox; we could just use a different TCP port. The one down side is the added cost and complexity of having a separate Ethernet subnet within the Sun Fire 15K chassis.Sun SPARC Enterprise Approach
Sun SPARC Enterprise M-series decided the application scalability of the Maintenance Area Network was a benefit, but we wanted to achieve it without having to run a separate Ethernet network. The result is what we called DSCP -- the Domain to Service Processor Communication Protocol. DSCP provides IP communication between the SP and the domain without any new hardware; it uses a single mailbox in the shared RAM, and using a pseudo serial driver on top of that mailbox, we enable PPP (the Point-to-Point Protocol). The DSCP stack looks like this:| SP | Domain(s) | |
| Application(s) | Application(s) | |
| | | | | |
| Sockets API | Sockets API | |
| | | | | |
| ipv4 | ipv4 | |
| | | | | |
| ppp | ppp | |
| | | | | |
| tty Driver | tty Driver (dm2s) | |
| | | | | |
| Mailbox Driver | Mailbox Driver (scfd) | |
| | | | | |
| Hardware Shared RAM Interrupts | ||
Configuring DSCP
The Sun SPARC Enterprise Server Administration Guide explains how to set up DSCP, but it is really quite simple. The easiest method is using the syntax:setdscp -i NETWORK -m NETMASKChoose a network address (be sure to pick a subnet that is not in use at your facility) and the corresponding netmask, and
setdscp will do the rest. For example, in my lab the subnet 192.168.244.0 is unused, so I do:
XSCF> setdscp -i 192.168.224.0 -m 255.255.255.0There are other ways to set up the DSCP network addresses, but this is really the best approach.
setdscp will assign an IP address to the SP, and reserve one IP address for every possible domain (the M9000-64 supports 24 domains, so a maximum of 25 IP addresses are reserved). A common question that's asked is, if you're running PPP between the SP and each domain, don't you need to two addresses for each domain, one for the domain and one for the SP? No, not really. Since routing is done based on the destination address, we can get away with using the same IP address for the SP on every PPP link. So technically speaking, the NETWORK and NETMASK are not defining a DSCP subnet; they are defining a range of IP addresses from which DSCP selects endpoint addresses. A subtle difference, but still a difference.
On the SP, showdscp will display the IP addresses assigned to each domain and the SP, for example:
XSCF> showdscp
DSCP Configuration:
Network: 192.168.224.0
Netmask: 255.255.255.0
Location Address
---------- ---------
XSCF 192.168.224.1
Domain #00 192.168.224.2
Domain #01 192.168.224.3
Domain #02 192.168.224.4
Domain #03 192.168.224.5
In Solaris, the
prtdscp(1M) command will display the IP address of that domain and the SP (
prtdscp is located in /usr/platform/SUNW,SPARC-Enterprise/sbin). You can get the same basic information from
ifconfig sppp0:
% /usr/platform/SUNW,SPARC-Enterprise/sbin/prtdscp
Domain Address: 192.168.224.2
SP Address: 192.168.224.1
% ifconfig sppp0
sppp0: flags=10010008d1 mtu 1500 index 3
inet 192.168.224.2 --> 192.168.224.1 netmask ffffff00
Benefits
Plumbing IP between the Solaris domain and the SP brings the obvious benefit of standards-based communication -- networking applications "just work". For example, you can configure the SP as an NTP server and configure the Solaris domains to use NTP to synchronize their time with the SP, all using the internal DSCP network. You can even use ssh to connect to the SP from a Solaris domain using the DSCP network. Since the SP does not have a hostname on the DSCP network, you need to get the IP address usingprtdscp, for example
ssh `/usr/platform/SUNW,SPARC-Enterprise/sbin/prtdscp -s`Personally, I create an alias sshsp with the above line.
On the SP side, you can't use ssh or scp directly -- they're not available in the XSCF shell. But you can use them indirectly. You can configure log archiving (see the setarchiving man page) to use one of the domains as an archive host:
XSCF> setarchiving -t rjh@192.168.224.2:/home/rjh/archive
XSCF> setarchiving enable
[I'm not sure it makes
sense to use a domain as a log archive host -- a catastrophic failure with the system means you also lose your log archive host -- but it is technically possible.]
And when you need to take a snapshot of the system for diagnosis purposes (see snapshot man page), you can specify one of the domains as the snapshot host using the -t option, for example:
XSCF> snapshot -l -t rjh@192.168.224.2:/home/rjh/snap
Downloading Public Key from '192.168.224.2'...
Public Key Fingerprint: 44:9a:ad:55:2e:33:99:2e:fd:b7:47:74:de:ad:be:ef
Accept this public key (yes/no)? yes
Enter ssh password for user 'rjh' on host '192.168.224.2':
Setting up ssh connection to rjh@192.168.224.2...
Collecting data into rjh@192.168.224.2:/home/rjh/snap/mymachine_10.4.55.144_2007-05-07T19-39-40.zip
Data collection complete
If your domain has internet access or a DVD burner, this might be the easiest way to get a snapshot back to a Sun Service Engineer.
Security
One of the most important security goals with the DSCP design was: Ensure that if one Solaris domain is compromised, that an attacker would not be able to affect the SP or another domain in the same chassis. This primary security requirement drove most of the DSCP design approach.Using PPP provides an added security benefit. Each shared RAM mailbox represents a single PPP connection between Solaris and the SP. This means there is no opportunity for one domain to snoop the traffic between another domain and the SP, and no way for one domain to directly attach another domain using the DSCP network. There is also no routing between DSCP networks (or from DSCP to Ethernet or vice versa) on the SP. The communication paths of each domain are physically isolated.
Most of the protocols used on the Sun SPARC Enterprise servers place the client on the SP and the server on the domain. This means that the SP does not need to open up well-known ports for incoming connections, reducing the opportunity for attacks. Furthermore, the severs running in Solaris use IPsec to authenticate that incoming connections are coming from the SP.
To prevent the domain from attacking the SP, several methods are used. First, all of the authentication and authorization protocols employed for Ethernet users are in place for the DSCP networks. There is no DSCP "back door", so to speak. Further, the SP employs a firewall that blocks all the ports on the DSCP networks except a couple -- ssh and ntp. There are additional features in place, for example, bandwidth limiting to prevent denial-of-service attacks.
We are seeing ipsec messages in the system log regarding the DSCP addresses. Is there an ipsec configuration option on the host that we should check? We did run JASS on this system. Would JASS change the ipsec configuration on the host O/S?
# dmesg |tail -1
Jan 14 14:38:42 sco01b ip: [ID 372019 kern.error] ipsec_check_inbound_policy: Policy Failure for the incoming packet (not secure); Source 192.168.037.026, Destination 192.168.037.028.
# /usr/platform/SUNW,SPARC-Enterprise/sbin/prtdscp
Domain Address: 192.168.37.28
SP Address: 192.168.37.26
Posted by Mike Beach on January 14, 2008 at 07:26 AM EST #
Mike,
I think, in general, you can ignore the ipsec_check_inbound_policy messages. They are probably happening whenever the SCF is rebooted. I'll try to post a new blog entry with a fuller explanation of what's happening.
-- Bob.
Posted by Bob Hueston on January 14, 2008 at 07:46 AM EST #