TCP protocol is the heart of internet. Good tcp understanding is basic requirement for system and network administrators for resolving fastly network problems.
Why do we receive RST? Why is the connection in time out? The server is down or the traffic is blocked by firewall? Is there some routing asymmetric? We can fastly answer with a good tcp understanding.
The goal of this article is to explain how to use simple network utility to find immediately the reason of tcp network problem.
Following the laboratory made with Oracle virtual box.
The client’s ip address is 192.168.1.108. There is a virtual IP 192.168.1.211 natted from pfsense firewall to server 192.168.2.11. A apache server is listening on port 80 on the server.
We start to analyze the three TCP phases: connection establishment, transmission data, close establishment.
TCP Connection Establishment
This phase is used for establishing a connection between client and server. The way to open connection is called three way handshake.
Let me show what happens every time a tcp connection is opened. We use telnet for that:
[root@client ~]# telnet 192.168.1.211 80
Trying 192.168.1.211…
Connected to 192.168.1.211.
Escape character is ‘^]’.
With tcpdump on client system we can analyze tcp packets beewten client and server.
Here packets description:
- The first packet is SYN. it’s sent from client to request to server to open a new TCP connection. The sequence number is 1357872344: it’s the starting point number in the flow bytes sent by clients, and it’s used for error control. The windows size is 16400: it’s how many bytes the client can receive without fulling his buffer and it’s used for flow control. It’s important to have random sequence number in order to avoid to hackers to get control of a tcp connection predicting its value. Firewall generally change it to new value ( remember it, it’s important).
- The client receive the second tcp packet: SYN ACK. It’s sends a ACK equals to client’s sequence number plus 1. A new server windows size is sent equals to 3963647727.
- The client answer with ACK equal to server’s sequence number plus 1 (tcpdump shows only 1, you should put -v option). The connection in the client is now in ESTABLISHED. Before receiving SYN ACK the connection is in SYN SENT.
During this phase two kind of errors can happened: timeout and reset. Let’s analyze both.
TCP Timeout Error
The connection goes in time out every time that client doens’t receive any SYN ACK from server.
[root@client ~]# telnet 192.168.1.211 80
Trying 192.168.1.211…
[root@client ~]# netstat -an |grep 192.168.1.211
tcp 0 1 192.168.1.108:51289 192.168.1.211:80 SYN_SENT
It means that the server has not answered to SYN. We have these possible solutions:
- Some Firewall has blocked the traffic.
- Some router in the path doesn’t have route to reach the destination.
- Destination server doesn’t have route to reach the sender.
A special case is that: remote server is not reachable (let’s suppose down) and it’s in the same network. We got it:
[root@client ~]# telnet 192.168.1.123 22
Trying 192.168.1.123…
telnet: Unable to connect to remote host: No route to host
What does No route to host mean? The remote is in the same vlan client. The comunication is direct: no need default gateway. The client before sending packet to remote has to resolve the server’s mac address; the remote is down and no mac address is available. The client sends a icmp unreachable to itself and it is the root cause.
[root@client ~]# tcpdump -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
17:49:25.994606 IP 192.168.1.108 > 192.168.1.108: ICMP host 192.168.1.123 unreachable, length 68.
This ICMP packet can be sent from router too, if it doesn’t have any route to reach the destination.
If the remote server default gateway doesn’t have any route to reach the server, what does it happen? We get the connection in SYN_RECV in the server. I simulate it changing the default gateway of server. The telnet goes in time out in the client.
This state could alarm because it could hide DOS attack.
After investigating about time out issues, we can analyze another kind of error: reset.
TCP Reset Error
Leet’s suppose that the httpd service is down. Linux sytem, according to RFC 793, sends RST packet. This is what happened.
The client that receives the RST can almost be sure that the service is down. I say almost, because can happen that the RST is sent from Firewall. This is not very probable because generally the FW block the traffic without sending any RST, but simply ignoring it. This is the output in the client:
[root@client ~]# telnet 192.168.1.211 80
Trying 192.168.1.211…
telnet: connect to address 192.168.1.211: Connection refused
Another scenario is asymmetric routing. The sender receive the final ACK but someting is wrong: the ACK is referred to another sequence number or the destination port is different. It happens for example because the SYN went through a Firewall that changed the sequence number, the SYN ACK went back from another way: the sequence number is different and the client anwers with RST. I describe it in the picture below. I simulated the asymmetric routing adding another interface to server directly to 192.168.1.0/24 vlan.
The server does’t answer to SYN because it understands that there is asymmetric routing: no packet is sent. Only after putting down the second interface down, the connection is established:
In the client the connection is in time out until SYN ACK is received.
TCP Transmission Data
For trasmitting data the connection must be in established state. If the tcp connection goes in established correctly, everything should be ok in this phase. Data can be transferred as well.
Following a http request to server by curl and the meaning of tcpdump output.
[root@client ~]# curl -v http://192.168.1.211
* About to connect() to 192.168.1.211 port 80 (#0)
* Trying 192.168.1.211…
* Connected to 192.168.1.211 (192.168.1.211) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.1.211
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 16 Feb 2016 14:16:04 GMT
< Server: Apache/2.4.6 (CentOS) PHP/5.4.16
< X-Powered-By: PHP/5.4.16
< Set-Cookie: JSESSIONID=p45KJF23JKJ1EKJ21213KJ
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host 192.168.1.211 left intact
Here the output of tcpdump:
15:16:05.360735 IP client.38608 > 192.168.1.211.http: Flags [S], seq 2473414591, win 14600, options [mss 1460,sackOK,TS val 11318107 ecr 0,nop,wscale 6], length 0
15:16:05.365457 IP 192.168.2.11.http > client.38608: Flags [S.], seq 2879155514, ack 2473414592, win 14480, options [mss 1460,sackOK,TS val 12134602 ecr 11318107,nop,wscale 6], length 0
15:16:05.368394 IP client.38608 > 192.168.1.211.http: Flags [.], ack 2879155515, win 229, options [nop,nop,TS val 11318115 ecr 12134602], length 0
15:16:05.370364 IP client.38608 > 192.168.1.211.http: Flags [P.], seq 2473414592:2473414669, ack 2879155515, win 229, options [nop,nop,TS val 11318117 ecr 12134602], length 77
15:16:05.373865 IP 192.168.2.11.http > client.38608: Flags [.], ack 2473414669, win 227, options [nop,nop,TS val 12134610 ecr 11318117], length 0
15:16:05.379230 IP 192.168.2.11.http > client.38608: Flags [P.], seq 2879155515:2879155745, ack 2473414669, win 227, options [nop,nop,TS val 12134615 ecr 11318117], length 230
15:16:05.379394 IP 192.168.2.11.http > client.38608: Flags [.], ack 2879155745, win 229, options [nop,nop,TS val 11318115 ecr 12134602], length 0
This is the meaning:
- The first 3 packets are for established TCP connection. The client set sequence number equal to 2473414591, the server to 2879155514. They send ACK equal to remote’s sequence number plus 1.
- The fourth packet sent from client contains the http request. The data lenght is 77. The new sequence number becomes 2473414592. The next sequence number is 2473414592+77=2473414669. The ACK is always 2879155515.
- The fifth packet is sent from Server to client. It acknowledges the preview packet sending ACK equal to remote’s next sequence acknowledges 2473414669.
- The sixth sent from server to client contains the http response. The data lenght is 230. The new sequence number becomes 2879155515 (preview plus 1)The next sequence number is 2879155515+230=2879155745.
- The seventh packet is sent from client to server. It acknowledges the preview packet sending ACK equal to remote’s next sequence acknowledges 2879155745.
After sending data, the connection can be closed. Here the third phase.
TCP Close Establishment
The tcp closing phase is called four way handshaking. Client request to server to close the connection and viceversa.
Every tcp connection is twice opened. It means that every one of this half connection must be closed. Request close connection is done by TCP FIN message.
A good tutorial about the closing of connection is in http://ssfnet.org/Exchange/tcp/tcpTutorialNotes.html.
Important thing in this phase is understanding that:
- Time Wait is a transitory state. The connection persist in this state for a fixed timeout: it is to avoid that another tcp connection can be opened with the same port number. No alarm if netstat -an shows a lot of connection in TIME_WAIT
- Close wait means that the remote client has close its half connection and it waits for closing the other half. The remote state is in FIN_WAIT_2, and the local is in CLOSE_WAIT. The only way to close this connection is to wait the the close api is called to tcp socket. If the server has a lot of connection, it is dangerous: something strange can happen. Why does the server leave these connections in half opened for a long time? A restart of the process owned of tcp connection could be necessary. No timeout exists about CLOSE_WAIT state.
I said that CLOSE_WAIT could be dangerous. I’m going to explain a real case of this issue.
Let’s suppose to have a library that manages tcp connection, for example HTTP Client Library java. Let’s suppose to set to 10 the maximum connection number to server. Under stress the software opens a maximun of 10 tcp connections for sending http request.
If for a strange reason, one of this connection goes to CLOSE_WAIT state, the system continue to consider this socket as opened, because it’s always opened but close from remote system. It’s cannot be used for sending traffic, but it’s always tcp open connection. The system has only 9 tcp connections available: it can cause degrade performance.
Following the four way handshake filtered by tcpdump.
14:38:26.960139 IP 192.168.1.211.http > 192.168.1.108.12709: Flags [F.], seq 3247013079, ack 2360296249, win 457, length 0
14:38:26.960613 IP 192.168.1.108.12709 > 192.168.1.211.http: Flags [.], ack 3247013080, win 64, length 0
14:38:26.970265 IP 192.168.1.108.12709 > 192.168.1.211.http: Flags [F.], seq 2360296249, ack 3247013080, win 64, length 0
14:38:26.970327 IP 192.168.1.211.http > 192.168.1.101.12709: Flags [.], ack 2360296250, win 457, length 0
This the meaning:
- The first packet is sent from from server to client for request to close the tcp half connection. The connection in the server from ESTABLISHED goes to FIN_WAIT_1.
- When the server receive the ACK to FIN the connection goes in FIN_WAIT_2. In the client the connection goes in CLOSE_WAIT.
- The server receive the FIN from client. It answers with ACK and its connection goes to TIME_WAIT.
- When the client send its FIN the connection goes from CLOSE_WAIT to LAST_ACK. When the ACK is received the connection goes to CLOSED.
You should see all the tcp states by netstat -an, but the tcp transition state is very fast and it’s difficult to see them.
I hope that you now have tcp understanding better.