ACKs, resends and waiting

Segments that arrive are answered with an acknowledgement (a receiver may wait briefly and answer every second one, and the bench answers every one), and the number in it is a position in the stream. An acknowledgement of X means that all bytes up to, but not including, X have been received. It is cumulative, and it says which byte the receiver wants next. A single acknowledgement of 800 covers everything before byte 800, so a lost acknowledgement is often harmless: the next one covers it.

One strip missing

Suppose the segment holding bytes 200 to 299 is dropped on the way. The ones behind it arrive. The receiver cannot move its acknowledgement past 200, because byte 200 is not there, so it answers each late segment at once with the same number. The standard says it should send that immediate duplicate acknowledgement when a segment arrives out of order.

The receiver keeps the late segments, as the standard says it should, and the lwIP stack on the ESP32 does by default. What it cannot do is hand them over. The program reads the stream in order, so it reads up to byte 199 and then waits, with bytes 300 to 799 already in memory. That is head-of-line blocking, and it is the price of an ordered stream: one gap stops everything behind it, however much has arrived.

Finding the loss

The sender has two ways to learn that a segment is gone.

The quick way is counting. Three duplicate acknowledgements in a row make the sender send the missing segment again at once, without waiting for its timer. A single duplicate can come from a segment that was only reordered, which is why the rule waits for three.

The slow way is the timer. If no acknowledgement comes within the retransmission timeout, the sender sends the oldest unacknowledged segment again and doubles the timeout. It has to be the timer when the lost segment was the last one, because nothing behind it can cause a duplicate. Before a round trip has been measured, the standard suggests one second to start. The Arduino core for the ESP32 builds lwIP with three seconds, and the ESP-IDF default is 1.5 seconds. The bench uses a fixed number of steps for its timer. Real stacks measure the round trip and adjust.

Each failed try doubles the wait, which is deliberate. A sender that retries eagerly into a struggling path makes it worse. After a limited number of tries the stack gives up, and in lwIP that is twelve resends of the same data, which takes minutes.

What an acknowledgement does not say

It says the receiver's TCP has the bytes. It does not say the program has read them or done anything with them. For that, the application has to send an answer of its own.

Loss also does not look like loss to the program. It sees the stream pause, and then continue. A resend costs waiting, not data, which is why a slow connection on a bad link looks like a stall and not like an error. Where this meets a board is HTTP from a board, at the end of the course.

Common mistakes

• Reading an acknowledgement as "the program got it". It means the receiver's TCP has the bytes up to that number.
• Setting an application timeout shorter than the first resend. A connect to a silent host waits about three seconds before the first SYN resend on the Arduino core, so a one second application timeout gives up before the stack has tried again.
• Treating a pause as a dead connection. A gap makes the stream wait for the resend, and a repeated loss makes each wait twice as long as the one before.