Nasty Winsock Overlapped I/O Gotcha
Today at work I ran into a nasty gotcha with network socket I/O in Windows.
To achieve maximal network throughput in Replay, we use I/O Completion Ports (IOCPs), which are kernel objects which aggregate asynchronous queuing of multiple I/O operations. Due to the way we’ve written our network transfer code, when doing a transfer we don’t know in advance how many read operations will be needed to complete the transfer, so I keep some n number of socket reads outstanding at all times, then once we get the EOF, I use CancelIO on the socket object to cancel any reads left outstanding. That way, they still come back from GetQueuedCompletionStatus and thus the memory associated with them can be freed, plus we maximize throughput by keeping plenty of read buffers available to the TCP/IP stack at all times.
This works fine except in two isolated, non-reproducible cases in which the call to CancelIO resets the TCP socket, resulting in a RST packet being sent back to the source, and both sides failing subsequent send/recv calls with WSAECONNRESET. I can only conjecture that the CancelIO implementation has some sort of failure case wherein it must reset the connection and give up. Highly fucking lame.
As a result, I had to rearchitect our transfer code to make sure we do all the writes we need to do before CancelIO, since I now have to assume CancelIO is a death sentence for whatever socket I call it on. Lame.