Linux Network: Receive side notes

At kernel init, each logical core has a thread called ksoftirq/%d running.

These threads are brought up by the initial function net_dev_init().

While probing newly installed device (nic) (also at kernel init), the nic has its hardware related interrupt handler registered. The interrupt handler's related host logical core can be changed. Sometimes nic support hardware level rss, mutiple hardware queues can register at different logical cores, thus having different cores splitting the load.

An pkt comes from outside, the hw interrupt handler is called, the handler notifies the soft interrupt handler to run(ksoftirq), and turns off the hw interrupt temporarily.

So now the ksoftirq on the same logical core preempts and runs net_rx_action(). It calls the driver registered poll() function. If the poll() funtion does not retrieve weight(64) packets this time, net_rx_action turns on the hw interrupt and goes out. (Be preempted)

The poll() function put retrieved pkt into skb rx_queue, filling the skb infos, packets dst-physaddr are compared to the physaddr of the device receives the packets. Determine if it is for local. (in eth_type_trans()) . 

Hand over to napi_gro_receive(). (receiveside offload: merging pkt of same header).

Once gro-ed pkts are sent to netif_receive_skb() for further network stack processing.(IP+) (Here in-between RPS/RFS may take place, if CONFIG_RPS is set)

RPS computes the hash of packet and determines the logical core for further processing of each packet. And each skb is enqueueed into specific struct softnet_data->sk_buff_head according to struct softnet_data->cpu.

Whatever RPS does, the packets arrive at specific cpu's netif_receive_skb().

Add packet type processing with dev_add_pack().

(L2)Packets each are processed by the registered function one by one until certain function recognizes the packet and deliver for further processing. If that is a ip packet, ETH_P_IP typed, ip_rcv() processes it. If it is an arp packet, the ip_rcv() does not recognize but arp_rcv() processes it.

(L3)First check netfilter pre-routing condition and deliver to ip_rcv_finish(). In ip_rcv_finish(), checks route table, and calls route table's registered input() function. If the packet targets local system (already probed by eth_trans_type() above), the ip_local_deliver() is called. The nefilter local-in condition is checked and calls ip_local_deliver_finish().

(L4)Registered ip protocol's handler() function is called. In tcp's term, tcp_v4_rcv(). In it the related socket/sock is allocated/found, the skb is related to the socket/sock, info filled into socket/sock. Its state is changed.