TCP, UDP, Unicast, Multicast? I thought this blog was supposed to be about portal[zz]

I promised I would write about the features of Oracle BPM and how they relate to the different features of modern BPMS suite. I haven't had the chance to complete writing the post for a variety of reasons. I should be able to finish it soon.

In this post, I'll instead talk about WebCenter Interaction or more precisely, how WCI makes use of network protocols. This post was originally posted by the great GeraldK on dev2dev, so I'm publishing it here to share some of his wisdom with you. Here we go.

This post gets pretty low-level, but I think this is important. As much as everything in our business relies on us abstracting away things at lower levels and specializing, abstraction only works up to a point. In some ideal, interfaces are clean and well-specified and focused; the application doesn't worry about how the OS works, and the OS doesn't worry about how the hardware works, and layer 4 of the OSI model doesn't worry about how layer 3 works. All you are supposed to do as a specialist at one level is assume the levels below you just work. But does that really work? If you're writing an application, it's better if you really know how the OS works and you do have to worry about how the database stores characters, and a good OS must worry about how the hardware works, and you're on crack if you think that TCP didn't worry about how IP was implemented.

So here, even though we're supposed to be about web applications and services that run up above OSI layer 7, we're going to go down n look at layer 4 (and even a bit lower) and see what's happening. We'll discuss what the differences are between TCP and UDP, what multicast is and how and why is works and doesn't work. Trust me, it'll be useful, and it all leads to some info about how PTSpy works in G6 and some ideas on how you can use it better.

So, let's start with HTTP. That's the protocol that matters directly for most of what we do here. HTTP, along with most of the rest of our application networking (SQL*NET, WCI Search) run at network layer 7, the application layer, on top of TCP which operates at layer 4. So what's TCP?

TCP (Transmission Control Protocol), along with UDP is one of the main underlying network protocols of the internet. Both of those are built on top of yet another one, IP (Internet Protocol). IP works one layer below TCP and UDP (layer 3). So to understand TCP, let's look at IP. We'll then work our way back up and talk about what TCP does on top of that.

IP: Internet Protocol
IP provides the basic ability to route a packet of data from one place to another. It does this by providing each "place" or "device on the network" with a specified address (the IP address) and specifying how such devices should move packets around based on the address. Now, the difference in functionality between IP and the protocol layer below it (layer 2, the data link layer, typically represented by Ethernet or WiFi) is that at layer 2, devices always know exactly how to send to every device on the network. At layer 2, either you know exactly how to get data to your destination (btw, usually represented by a "MAC address") or it's not possible to get it there. (For example, Ethernet and WiFi simply broadcast the entire packet onto the whole network, and the destination is expected to be listening for its MAC address and pick up the packet. If it's not there or not listening, Ethernet can't get it there. [And btw, network "sniffers" works by taking advantage of this broadcasting.] PPP, used in modem dial-up connections, can send everything to one destination only: the machine you dialed in to.)

What IP does is provide a way to connect separate networks so that devices on one network can send data to devices on another network without knowing where they are or how to get there exactly. Hence the appellation inter-net, "between networks". It does this by specifying routing rules that define what a network device does with a data packet with any destination address. Basically the rules are: if the destination is local (i.e. you know where it is because it's on the same network) send it there; otherwise, send it to one of a list of routers that vary according to the address. A router follows the exact same rules, except that it's assumed that it belongs to two or more different networks at the same time; and therefore has more local destinations, and also usually has a longer list of other routers for unknown destinations.

Very specifically now, IP does no more than give you the ability to send a single packet of data to a destination specified by a single IP address, and that's it. It can get that packet to any place on any network (unlike the lower level protocol), but that's about it. Significantly:

 

• It does not provide notification of delivery or receipt or failure

 

• It does not provide any notion of "port numbers" or other ways of segregating packets at the destination IP addresses

 

• It does not provide two-way communications

 

• It does not order or otherwise group multiple packets in any particular way

 

 

The simplest analogy is to think of IP like the postal service. You drop a postcard into the mail with an address on it, and somehow it makes it to the address you wrote on it. If it makes it, or if it doesn't, you don't know. When it gets to the house, you don't know if the roommate reads it. And if you want a reply, your recipient can't just write on the same card and hand it back to the mailman; they'd have to write out their own note, stamp and address it, and send it out again themselves.

TCP: Transmission Control Protocol
While IP does not provide any of those features, TCP does. If you looked at the list of the features that IP doesn't provide, and then you looked at the analogy of the postal service and said, "Huh? Of course you can have two-way communications. People used to write letters back and forth and have conversations all the time," or "Well, you could just request that either the recipient or the postal service send you back a return receipt," or "Come on, dummy, you can sequence your postcards by just writing numbers on them and telling the recipient to read them in order and let you know if any are missing," well, then, you're right. That's exactly what TCP does. It takes the basic IP (or postal service) and specifies how to use it and what to put in your message to get all those extra features.

So what TCP really winds up doing is implementing the concept of multiple reliable connections between IP devices. What does that mean? It means that you can send a sequence of messages (packets) about a specific conversation (port, or connection) and that the packets will be received in the same order they were sent, and that there won't be missing packets in between sent packets. (Also, there won't be duplicate packets.) It does this by attaching to every packet both a port number, to separate and identify multiple connections or conversations; and a sequence number, to let the recipient identify when data might have been lost in transmission. In addition, it specifies the recipient has to acknowledge every data byte that was received (not necessarily explicitly: it might simply say, "I've received everything up till byte 13456"; and it can also acknowledge selectively, "I've received everything between 845 and 13433") so that the sender can know if missing data needs to be resent. And finally, every connection is implicitly two-way, not just for acknowledgements, but also so that the recipient can just send data "back" rather than having to explicitly re-address (sort of the equivalent of sending a self-addressed reply envelope with every packet).

As you can see, if packets are getting lost or arrive out of order, there is actually quite a lot of work that TCP has to do. If we're continuing the postal service analogy, TCP is a bit like a personal assistant who collects your mail, groups it, sorts it in order, brings and reads it to you, takes your replies and organizes and sends them back. If the postal service is super-reliable, his job is easy, he's just a middleman passing paper around. If the postal service loses a lot of stuff, and/or you have a great volume of mail, he may have to do a lot of work, request that things be re-sent, and keep track of and store a lot of messages.

UDP: User Datagram Protocol
UDP on the other hand is a lot simpler. It does what IP does, but it adds the concept of a port, so that you can send messages to a specific recipient at one IP address. It doesn't have order, or connections, or two-way communication, or acknowledgements.

You might think that UDP is unreliable, because, you know, TCP is supposed to be the reliable one of the siblings. But in fact, over the same network segment, or over LANs with good quality gear and not excessive traffic, UDP is in practice very reliable. If there's no packet loss and packets arrive in order (which is almost always the case on a short LAN link), there's no need for any retransmissions of packets, so all the acknowledgements and waiting around of TCP is just a bunch of wasted overhead, creating latency. For applications that can tolerate packet losses (say, real-time audio and video), it's often a good choice even over not-so-good networks. It's also used for small messages and notifications. For example DHCP and DNS both use UDP. And notably, the Unix Network File System (NFS) used UDP over LANs. While you'd think a file system should have used a reliable TCP connection, the implementers of NFS felt they could get better performance using UDP and building their own mechanisms for ensuring reliability that were tuned to the application.

Btw, it's called "User Datagram Protocol" because it's was designed by operating system guys. "Datagram" is just another word for "packet". "User" in this case doesn't mean, like, you; it means a program on a computer that is not part of the operating system. The idea was that IP is low-level and it was used by people who write parts of the OS, while UDP provides mostly the same functionality (the datagram), but is intended for "user" (non-OS) programs.

Multicasting
I simplified the TCP/IP/UDP discussion up there a little when I implied that IP (and hence UDP and TCP) would route packets from one device on a network to another device. More accurately, IP routes packets from one IP address to another IP address. The trick with multicasting, or sending the same packets of data at the same time to many devices is that certain IP addresses can be designated as multicast address and assigned to many devices at the same time.

The first thing to know about IP multicasting is that there only exists UDP multicasting. There's no such thing as TCP multicast. Why not? See, the point of multicast is to be efficient with the network and send the same data packet to as many different, possibly unknown computers. But each TCP connection can potentially require retransmission of different lost packets, or different delays or ordered arrival and assembly of packets. So managing and sending all that for each possible device would be resource-intensive even if it were possible, which would defeat much of the point of using multicast. (And it's not possible anyway because multicast can't know where the outbound packets actually wind up.)

Oh, and via back-formation, regular non-multicast UDP (and TCP) messages are called unicast.

The next thing to know is that multicast will often not be sent over a router to another network. There are a few reasons why:

 

• A low TTL for most multicast packets: All IP packets have a time-to-live, or TTL. Unlike with DNS records, TTL on IP packets refers to the maximum number of network hops that a packet can make to get to its destination. Unicast packets typically are allowed to cross about 30 networks (i.e., get routed by, or "hop", 29 routers). Getting across the internet rarely takes more than 15 hops, so the limit is really just to kill packets if they get stuck in loops in badly configured networks. But most applications that send multicast set the TTL to a low number, often zero (where the message doesn't even leave the originating device), 

 

• one (so it will go only to computers on the local net), or two (where it will cross a single router). It's quite rare to have an application that is intended to multicast to unknown machines over an entire campus network, much less the entire Internet. 

 

• High TTL thresholds set on most routers: Many network routers, particularly WAN routers and internet gateway routers, have a higher TTL threshold set, such that they won't send out multicast packets whose TTL is below (say) 15. This is intended to prevent accidental leakage of multicast off a local network. 

 

• Routers are simply not configured to route multicast at all, or only configured for specific addresses, or otherwise blocking multicast packets.

 

 

UDP multicast might seem a bit exotic, but it's actually more common that you think. It's not used for web video sites like YouTube, since those need to send a video at a time each user demands, rather than at the same time for all users, and it's not used for VoIP traffic. However, it is used for a lot for discovery and automatic confguration in apps like Skype, iTunes, and uPnP. It's also used in a few places in the WCI portal.

Spy Logging
Since G6 was released, Logging Spy (formerly PTSpy) and the Plumtree Logging Framework (AL-something Logging Toolkit or whatever) have worked using UDP. The application that has messages to log just sends it out via UDP. Now, recall that UDP doesn't worry about things like actual delivery and order and error correction and flow control. The message can be sent out as "fire-and-forget", or "drop it in the outbox and forget about it forever". There's no record-keeping or waiting, so it's very cheap and quick for the program to just put send the message.

In fact, it costs pretty much the same for a program to just send out the log message via UDP as it would for it to check to see if the message should be filtered. So, the G6 portal (and collab, and the api and automation servers, etc.) doesn't bother checking; it just sends out the UDP log message unconditionally.

In contrast, 5.x and earlier portal versions logged positively: every message had to be written to a file (or the screen) before the program could proceed to the next operation. This impacted performance severely enough that you could only enable tracing when you were actively debugging, and even then had to try to limit what you logged. Throwing out the UDP packet causes so little delay that G6 traces a lot more, and it sends the trace messages the whole time the program is running.

Now, we also have a set of programs that read these UDP packets from the network. The most familiar one is Spy itself, which captures the packets, picks out the ones you told it you wanted to see, and displays only those. There is also the Logger Service, which works similarly, but reformats the messages and writes them to a text file.

You might not even actually know that the Logging Framework is using UDP to send log messages, because normally the applications are set to "local logging only" which sets the packet TTL to zero, which doesn't let it leave the originating computer (but which can be picked up by a destination on the same computer). At the same time, Spy is usually installed along with the application. When you start Spying, it just picks up the packets from the local application, and it looks like the old 5.x PTSpy connecting locally to the locally-installed application. But in fact, the application and the viewer for the Spy messages are now very loosely coupled over the network, and run independently of each other.

While UDP does offer great performance advantages, so much so that we now can have much more detailed logging available when necessary, there is the possibility that some log messages can be lost, just because of the unreliable nature of UDP. This isn't really a disaster, but it could be annoying if you're debugging and you don't know it may be happening. It may happen when there is a lot of activity on the portal (and hence a lot of messages being sent), the network is very busy so packets are lost, or the capture program is running slowly. In practice, the most common case in which messages are lost is when the Spy program is running slowly, maybe because the machine it's on is busy, or because of some other resource issues. We've found in particular that the X-Windows Spy, when writing to the screen, can miss a lot of messages. I usually recommend using the command-line logger on Unix anyway.

One more thing to note is that Spy is actually sent out via UDP multicast, not unicast. This means, first of all, it's possible to have several programs picking up and reading and recording log messages from a single application. For instance, while you are looking interactively at Spy, the Logging Service can also be recording the same or a different subset of messages to a file. Second, you can also monitor several applications remotely using a single (or multiple) programs on a remote machine. You simply need to make sure that the application "local logging only" option is set to false, and that your network and routers will route the packets from the WCI applications to the logging listeners. Then, just open up Spy, select View->Set Filters, then Edit->Add Message Sender. Spy will look for applications that are configured to log via UDP multicast over the network (and which can reach Spy) and allow you to choose to capture and record the messages.

Collab Clustering
The other place that we encounter UDP is when configuring Collaboration Server clustering. Collab clustering is used for cache invalidation among the Collab servers. When an object is updated on one Collab server, a message needs to be sent to all of them identifying the object. That's a small, simple message.

There are several configurations for setting up Collab to send the message. The two most common should be the "lan-cluster" configuration and the "lan-multicast-cluster" configuration. Both of them use UDP to send the messages. The unicast configuration requires you to list out every other server in the cluster in each server's config, but will work on any network where the servers are actually reachable to each other. The multicast setting requires less configuration and config maintenance, since you all you do is set the multicast listen address on each one to the same value, but you run the risk that the network may not support or route multicast among your servers. It's also hard to know if it's working without running network traces to look for the Collab cache invalidation messages, as for the most part Collab will run fine and no one single user is likely to notice inconsistencies in caches across different servers.

Well, I hope that was informative and interesting. And useful.