The Transport Layer

The Transport layer provides process-to-process communication — not just host-to-host like IP, but between specific applications running on those hosts. A server can simultaneously serve thousands of web clients, SSH sessions, and database queries on the same IP address; the Transport layer uses port numbers to distinguish them. Depending on which protocol you choose, it also provides reliability, ordering, and congestion control.

Ports, sockets, and multiplexing

A port is a 16-bit (0–65535) logical endpoint. IANA divides them into three ranges:

Linux uses 32768–60999 by default for ephemeral ports (configurable via /proc/sys/net/ipv4/ip_local_port_range). Windows uses 49152–65535.

A network flow (or connected TCP session) is uniquely identified by a 5-tuple: {protocol, source IP, source port, destination IP, destination port} (or a 4-tuple once the transport protocol is fixed). This allows a server on port 443 to have millions of simultaneous connections — each client has a different source IP or ephemeral port, making every 5-tuple unique.

Multiplexing and demultiplexing

The transport layer acts as an intermediary between network communication and local processes:

• Multiplexing (at the sender): Gathering data chunks from multiple application sockets, encapsulating each chunk with transport-layer header fields (including source and destination port numbers), and passing the resulting segments down to the network layer.
• Demultiplexing (at the receiver): Examining transport-layer header fields of arriving segments to steer the payload to the exact application socket awaiting it.

The fundamental distinction between UDP and TCP demultiplexing lies in which header fields are inspected:

• UDP demultiplexing (connectionless): A UDP socket is identified solely by a 2-tuple: (destination IP address, destination port number). If two UDP datagrams originate from different source IP addresses or different source ports, but share the same destination IP and destination port, both datagrams will be delivered to the same socket.
• TCP demultiplexing (connection-oriented): A TCP connection socket is identified by the full 4-tuple: (source IP address, source port number, destination IP address, destination port number). A server process (e.g. an HTTP daemon) initially binds a single listening socket to port 80. When a client initiates a connection, the operating system kernel accepts it and creates a separate, dedicated connection socket identified by the client's unique source IP and port. Two distinct clients connecting to the same server port 80 are routed to two different connection sockets.

UDP — User Datagram Protocol (RFC 768, 1980)

UDP is connectionless, unreliable, and unordered. The entire UDP header is just 8 bytes:

The IP pseudo-header and checksum calculation: Both UDP and TCP compute their 16-bit one's complement checksum over a 12-byte IP pseudo-header (Source IP, Destination IP, Protocol number [17 for UDP, 6 for TCP], and UDP/TCP segment length) in addition to the transport header and payload. By including these IP fields, the transport layer verifies that the segment was delivered to the correct IP destination host. This cross-layer dependency is also why NAT routers must recompute the transport-layer checksum whenever they translate an IP address or port number.

The maximum UDP payload over IPv4 is 65,507 bytes (65,535 − 20 IP header − 8 UDP header). Classic DNS over UDP was limited to 512 bytes; EDNS(0) (RFC 6891) allows larger messages, and resolvers commonly advertise about 1232 bytes to avoid fragmentation; larger answers fall back to TCP. UDP is used where:

• Low latency trumps reliability: DNS (1 query/response — retrying is the app's job), DHCP (broadcasts), SNMP monitoring.
• Timeliness over completeness: VoIP, live video streaming — a late audio packet is worse than a lost one.
• Application controls retransmission: QUIC, online games (custom reliability per packet type).
• Multicast and broadcast: Multicast and broadcast are only possible with connectionless UDP; TCP is strictly point-to-point.

TCP — Transmission Control Protocol (RFC 9293, updating RFC 793)

TCP provides a reliable, ordered, full-duplex byte stream between two endpoints. "Byte stream" means TCP delivers bytes in sequence with no message boundaries — the application reads bytes, not packets. TCP handles splitting data into segments, retransmitting lost ones, reordering, and pacing to avoid congestion, all transparently.

TCP header (20 bytes minimum)

Three-way handshake — with sequence numbers

The handshake establishes the connection and synchronises sequence numbers on both sides:

Client → Server:  SYN                 Seq=1000      Ack=0      (ISN_client = 1000)
Server → Client:  SYN-ACK             Seq=5000      Ack=1001   (ISN_server = 5000; acks client ISN+1)
Client → Server:  ACK                 Seq=1001      Ack=5001   (acks server ISN+1)

Connection ESTABLISHED. Client can now send data starting at Seq=1001; Server at Seq=5001.

SYN and FIN control flags each consume exactly one sequence number, even though they carry no application data. This explains why the server acknowledges Ack = ISN_client + 1 and the client responds with Ack = ISN_server + 1. If the server has no data to send when receiving a FIN, its ACK and FIN can be combined into a single segment, completing teardown in 3 segments instead of 4.

ISNs (Initial Sequence Numbers) are chosen randomly (RFC 6528 recommends a cryptographic ISN) to prevent sequence number prediction attacks. In Wireshark, "relative sequence numbers" starting at 0 are shown by default for readability.

Sequence numbers, ACKs, and reliable delivery

TCP numbers every byte. If the sender sends bytes 1–1000 and the receiver gets only bytes 1–500 (bytes 501–1000 were lost), the receiver sends ACK 501 (cumulative: "give me byte 501 next"). The sender retransmits from 501.

With SACK (Selective ACKnowledgement, RFC 2018), the receiver can inform the sender about non-contiguous received ranges: "I have 1–500 and 601–1000 but missing 501–600." The sender retransmits only the gap, rather than everything from 501. This is especially valuable on high-bandwidth, high-latency paths where many packets may be in flight.

Worked example (file slicing): Suppose an application sends a 500,000-byte file with MSS = 1000 bytes and Initial Sequence Number ISN = 0. TCP slices the byte stream into 500 segments:
• Segment 1 carries bytes 0–999 with Seq = 0.
• Segment 2 carries bytes 1000–1999 with Seq = 1000.
• Segment 3 carries bytes 2000–2999 with Seq = 2000, continuing up to Segment 500 with Seq = 499,000.
When the receiver successfully receives the first segment (bytes 0–999), it responds with cumulative acknowledgement Ack = 1000 (the next expected byte).

Delayed ACKs: Rather than acknowledging every single arriving packet immediately, TCP receivers typically delay ACKs to piggyback them onto return data or send one cumulative ACK for every second full-sized segment, bounded by a timer (up to 500 ms per RFC 9293; Linux defaults to ~40 ms).

Stop-and-Wait vs. Pipelining (GBN, SR, and TCP):

If a sender sends one packet and waits for an ACK before sending the next (Stop-and-Wait), sender link utilisation is bounded by the round-trip propagation time:

U_sender = (L / R) / (RTT + L / R)

versus Pipelined utilisation (allowing up to N unacknowledged in-flight packets):

U_pipelined = (N × L / R) / (RTT + L / R)

Worked comparison: For L = 1,000 bytes (8,000 bits), R = 1 Gbps (1,000,000,000 bps), and RTT = 30 ms:
• Transmission delay: L / R = 8,000 / 10^9 = 0.000008 s = 0.008 ms.
• Stop-and-Wait utilisation: U_sender = 0.008 / (30 + 0.008) = 0.008 / 30.008 ≈ 0.000267 (0.027%) — the link sits idle 99.97% of the time!
• By contrast, pipelining with window size N scales sender utilisation directly by N until the link is fully saturated (100%).

ARQ Protocol comparison:

• Go-Back-N (GBN): Uses cumulative ACKs. If a packet in the window is lost, the receiver discards all subsequent out-of-order packets. When the sender's timer for the oldest unACKed packet expires, the sender must retransmit all N packets in the window.
• Selective Repeat (SR): Uses individual ACKs for each correctly received packet. The receiver buffers out-of-order packets. The sender maintains a separate timer per unACKed packet and retransmits only the specific lost packet.
• TCP: A practical hybrid. Like GBN, TCP uses cumulative ACKs (acknowledging the next contiguous expected byte) and maintains a single retransmission timer. Like SR, modern TCP receivers buffer out-of-order segments within the receive window, avoiding unnecessary retransmissions when gaps are repaired.

Worked trace: Fast Retransmit with duplicate ACKs

When segments arrive out of order, TCP receivers immediately generate duplicate ACKs to alert the sender to the gap. Receiving 3 duplicate ACKs triggers Fast Retransmit without waiting for a retransmission timeout (RTO):

Client sends 5 consecutive 1,000-byte segments (MSS = 1,000 B):
1. Client → Server: Seg 1 (Seq=1001, Len=1000)  [Bytes 1001..2000]
2. Client → Server: Seg 2 (Seq=2001, Len=1000)  [Bytes 2001..3000]  --> LOST IN TRANSIT!
3. Client → Server: Seg 3 (Seq=3001, Len=1000)  [Bytes 3001..4000]
4. Client → Server: Seg 4 (Seq=4001, Len=1000)  [Bytes 4001..5000]
5. Client → Server: Seg 5 (Seq=5001, Len=1000)  [Bytes 5001..6000]

Receiver response and sender Fast Retransmit:
• Seg 1 arrives intact:  Server sends Ack=2001 (Original ACK: requests byte 2001)
• Seg 2 is LOST.
• Seg 3 arrives (gap!):  Server buffers Seg 3, sends Ack=2001 (1st Duplicate ACK)
• Seg 4 arrives:         Server buffers Seg 4, sends Ack=2001 (2nd Duplicate ACK)
• Seg 5 arrives:         Server buffers Seg 5, sends Ack=2001 (3rd Duplicate ACK)
  --> Sender sees 3 duplicate ACKs! Triggers Fast Retransmit of Seg 2 immediately:
• Client → Server: Seg 2 (Seq=2001, Len=1000) retransmitted!
• Seg 2 arrives at Server: Fills the missing byte gap (2001..3000).
  Server already has buffered Segments 3, 4, 5 (bytes 3001..6000).
• Server → Client: Cumulative Ack=6001 (acknowledges all bytes up through 6000 at once!)

Retransmission and RTO Beyond

The Retransmission Timeout (RTO) is computed from the smoothed RTT (SRTT) and RTT variance (RTTVAR) using the Jacobson/Karels algorithm (RFC 6298). Per RFC 6298, RTTVAR is updated before SRTT, using the previous SRTT value:

Initialisation (first RTT measurement R):
  SRTT   = R
  RTTVAR = R / 2
  RTO    = SRTT + max(G, 4 × RTTVAR)

Subsequent measurements (RTTVAR updated BEFORE SRTT):
  RTTVAR = (1 - β) × RTTVAR + β × |SRTT - RTT_sample|  (β = 1/4)
  SRTT   = (1 - α) × SRTT + α × RTT_sample             (α = 1/8)
  RTO    = SRTT + max(G, 4 × RTTVAR)                   (G = clock granularity, ≥ 1 ms)
  RTO    = max(RTO, 1 second)                          (RFC recommends 1 s minimum; Linux uses 200 ms)

Worked numeric example: Suppose current SRTT = 100 ms, RTTVAR = 20 ms, and a new sample RTT_sample = 140 ms arrives:
• RTTVAR = 0.75 × 20 + 0.25 × |100 − 140| = 15 + 10 = 25 ms
• SRTT = 0.875 × 100 + 0.125 × 140 = 87.5 + 17.5 = 105 ms
• RTO = 105 + 4 × 25 = 205 ms (assuming clock granularity G ≤ 100 ms).

Karn's algorithm: TCP never measures RTT samples from retransmitted segments (avoiding retransmission ambiguity).

On timeout, RTO is doubled (exponential backoff) up to an implementation-defined ceiling (RFC 6298 requires at least 60 s). After enough consecutive timeouts, TCP gives up and aborts the connection.

Flow control — the sliding window

The receiver advertises its available receive buffer in the Window Size field. The sender must not have more unacknowledged (in-flight) data than this value. When the window reaches zero, the sender sends periodic 1-byte window probes to check if the receiver has freed buffer space.

The receive window is computed dynamically by the receiving host based on its allocated socket buffer size (RcvBuffer) and the amount of buffered data waiting for the application process to read:

rwnd = RcvBuffer − (LastByteRcvd − LastByteRead)

Worked calculation: If a host allocates RcvBuffer = 64,000 bytes, has received and buffered up to byte LastByteRcvd = 20,000, and the local application has read up to byte LastByteRead = 5,000:
• Unread bytes occupying buffer: 20,000 − 5,000 = 15,000 bytes.
• Advertised receive window: rwnd = 64,000 − 15,000 = 49,000 bytes.

Example: RTT = 50 ms, window = 64 KB. Maximum throughput = 64 × 1024 × 8 bits ÷ 0.05 s = ~10 Mbps. With window scaling (RFC 7323), the window can reach 1 GB, enabling full utilisation of a 1 Gbps link with 10 ms RTT.

Congestion control — in depth

While flow control prevents the receiver from being overwhelmed, congestion control prevents the network from being overwhelmed. TCP infers congestion from packet loss signals:

This is AIMD (Additive Increase, Multiplicative Decrease), the principle that makes TCP fair: multiple flows competing for a bottleneck link converge to equal shares for flows with similar RTT; flows with a shorter RTT increase their window faster and claim a larger share of the bottleneck capacity.

Average throughput of an AIMD flow: Under periodic loss with AIMD, the congestion window ramps linearly from W / 2 up to W before dropping back to W / 2 upon loss. The average congestion window during this cycle is approximately (W + W/2) / 2 = 0.75 W. Therefore, the average throughput is approximated by:

Throughput_avg ≈ (3/4) × (W × MSS) / RTT

Worked example: Suppose a connection reaches a peak window of W = 40 MSS with MSS = 1500 bytes and RTT = 100 ms (0.1 s):
• Average window: 0.75 × 40 = 30 MSS = 45,000 bytes
• In bits: 45,000 × 8 = 360,000 bits
• Average throughput: 360,000 bits / 0.1 s = 3,600,000 bps = 3.6 Mbps.

Modern variants Beyond: TCP CUBIC (the default on Linux, verify with sysctl net.ipv4.tcp_congestion_control) grows cwnd as a cubic function of elapsed time since the last congestion event; BBR (Bottleneck Bandwidth and RTT) (Google, 2016) measures delivery rate and min-RTT directly rather than reacting to loss. BBR can perform much better in some conditions (high bandwidth-delay product, non-congestive loss), but BBR has had fairness issues against loss-based flows.

TCP Tahoe vs. TCP Reno: Fast Recovery and the cwnd sawtooth Historical background

Modern TCP congestion control evolved through two landmark BSD releases named after 4.3BSD-Tahoe (1988) and 4.3BSD-Reno (1990) Historical background, with core algorithms designed by Van Jacobson and standardised in RFC 5681 (which obsoletes RFC 2581, which obsoleted RFC 2001):

• TCP Tahoe (1988): Introduced Slow Start, Congestion Avoidance, and Fast Retransmit. However, Tahoe treated all packet loss identically: whether triggered by an RTO timeout or 3 duplicate ACKs, Tahoe halved ssthresh = cwnd / 2, collapsed cwnd = 1 MSS, and restarted full Slow Start from scratch.
• TCP Reno (1990): Recognised that receiving 3 duplicate ACKs implies that segments are still arriving at the receiver — the network pipe is not completely clogged. Reno introduced Fast Recovery:
  1. On 3 duplicate ACKs: sets ssthresh = cwnd / 2.
  2. Fast Retransmit sends the single missing segment immediately without waiting for RTO.
  3. Fast Recovery inflates cwnd = ssthresh + 3 MSS (crediting the 3 packets that successfully left the pipe). Each additional duplicate ACK during fast recovery inflates cwnd by 1 MSS (the pipe is draining).
  4. When the first new ACK arrives for the retransmitted segment, Reno deflates cwnd = ssthresh and proceeds directly in Congestion Avoidance (+1 MSS per RTT).
  5. Reno never resets to 1 MSS for duplicate ACKs (it resets to 1 only on a full retransmission timeout).

Worked example: 10-round cwnd progression after 3 duplicate ACKs: Consider a connection starting with cwnd = 1 MSS and ssthresh = 8 MSS. During rounds 1–8, cwnd progresses through Slow Start (1, 2, 4, 8) and Congestion Avoidance (9, 10, 11, 12). At the end of round 8, 3 duplicate ACKs arrive. The new threshold is set to ssthresh = 12 / 2 = 6 MSS. Under standard textbook/exam conventions (Slow Start doubles cwnd each RTT up to ssthresh then increments by +1; Reno resumes from ssthresh in the first round after loss), here is the progression for the subsequent 10 RTTs (rounds 9–18):

If instead a retransmission timeout occurs at the end of round 8 (cwnd = 12), both variants react identically: both set ssthresh = 12 / 2 = 6 MSS, reset cwnd = 1 MSS, and enter Slow Start:

Comparing Tahoe, Reno, and NewReno: Over time, TCP refined its recovery handling to address packet losses occurring within the same transmission window:

Modern loss recovery: SACK-based loss recovery (RFC 6675 Beyond) extends NewReno by using TCP SACK option blocks to selectively retransmit all missing segments in a single RTT during Fast Recovery.

# 1. Check your operating system's default congestion control algorithm
$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic

# 2. Inspect active socket internals (cwnd, RTT, ssthresh)
$ ss -ti dst <server-ip>
ESTAB  0  0  192.168.1.50:54218  93.184.216.34:443
     cubic wscale:7,7 rto:204 rtt:4.125/0.850 ato:40 mss:1460 rcvspace:14600
     cwnd:10 ssthresh:16 bytes_acked:142010 segs_out:112 segs_in:98 data_segs_out:105

# 3. View the cwnd sawtooth in Wireshark:
# Capture a file download, select a TCP packet, then navigate to:
# Statistics > TCP Stream Graphs > Time Sequence (Stevens)
# to visualise the cwnd sawtooth ramp-up and drops.

Connection teardown (four-way)

Either side can initiate teardown. It's a four-way handshake because each side's FIN must be independently acknowledged:

Client → Server:  FIN  Seq=1500         (client done sending)
Server → Client:  ACK  Ack=1501         (server acks client FIN)
                  [server may still send data]
Server → Client:  FIN  Seq=6000         (server done sending)
Client → Server:  ACK  Ack=6001         (client acks server FIN)

The active closer (the side that initiates termination) enters TIME_WAIT for 2×MSL.
RFC 9293 defines 2×MSL with MSL = 2 minutes (4 min); Linux uses a fixed 60 s,
so TIME_WAIT is typically 60 s to 4 min.
Purpose: absorb delayed duplicate packets and ensure final ACK reached remote peer.

Half-close: a TCP connection can be half-closed — one side finishes sending (sends FIN) while the other continues transmitting data. A half-close is done with shutdown(SHUT_WR) (e.g. netcat, some request/response tools).

RST (Reset): immediately terminates a connection when something is wrong — e.g. a packet arrives for a port with no listener, or an OS crashes and the socket is gone. RST is not a graceful close.

TCP state machine (simplified)

CLOSED
  │ (passive open: server calls listen())
  ▼
LISTEN ◄─────────────── waiting for SYN
  │ (SYN received)
  ▼
SYN_RCVD ─── send SYN-ACK ──► ESTABLISHED (when ACK received)
  
CLOSED ─── (active open: client calls connect())
  │ (send SYN)
  ▼
SYN_SENT ─── SYN-ACK received, send ACK ──► ESTABLISHED

ESTABLISHED ─── send FIN ──► FIN_WAIT_1 ─── ACK received ──►
FIN_WAIT_2 ─── FIN received, send ACK ──► TIME_WAIT ─── 2×MSL ──► CLOSED

ESTABLISHED ─── FIN received ──► CLOSE_WAIT ─── send FIN ──►
LAST_ACK ─── ACK received ──► CLOSED

The side that closes first (active close) ends up in TIME_WAIT. Sockets in TIME_WAIT are normal; inspect them with ss -tan state time-wait | wc -l. Many sockets stuck in CLOSE_WAIT indicate the local application failed to call close(); many in SYN_RCVD may indicate a SYN flood attack (see Chapter 7).

Key TCP options (negotiated in SYN/SYN-ACK)

TCP vs. UDP comparison

QUIC (RFC 9000) Beyond

QUIC was developed by Google (original implementation 2012, standardised by IETF in 2021 as RFC 9000) and is the transport protocol underlying HTTP/3. It runs over UDP, commonly on port 443, but reimplements most of TCP's reliability features with several modern design improvements:

• Connection setup latency: Fresh connection: TCP + TLS 1.2 = 3 RTT before application data (1 TCP + 2 TLS), TCP + TLS 1.3 = 2 RTT (1 TCP + 1 TLS), QUIC = 1 RTT (transport and crypto handshake combined). Resumed connection: a RESUMED connection can use 0-RTT data, sending application payload alongside the very first packet.
• No head-of-line blocking at transport layer: each stream is an independent QUIC stream. A lost packet for stream A doesn't block stream B. (HTTP/2 over TCP suffers HOL blocking because all streams share one TCP byte stream.)
• Connection migration: QUIC connections are identified by a 64-bit connection ID rather than the socket 4-tuple. If your client switches network interfaces (e.g. Wi-Fi to LTE), the connection continues without renegotiating.
• Integrated TLS 1.3: QUIC encrypts the payload and most header fields; there is no unencrypted plain-text QUIC mode.
• Improved loss recovery: QUIC uses monotonically increasing packet numbers (never reused on retransmission) instead of TCP's sequence numbers, cleanly solving the retransmission ambiguity problem (Karn's algorithm).