In this article I am going to demonstrate how to restrict incoming connections to specific port in Ubuntu Linux. Then we will see how to check the logs on the server side for rejected client connection attempts and how to troubleshoot this issue from clients perspective by analysing TCP packets.
By the end you should be more familiar with UFW(Uncomplicated Firewall), lsof, tcpdump and TCP Three-way handshake analysis, Wireshark, nc, rsyslog, telnet, vagrant.
The Article assumes you have VirtualBox and vagrant installed, if you haven’t google it, it is very easy to setup.
First thing first, you will need to setup 2 VMs, alternatively you can use the host as the client, given you can
use tcpdump on it:
cat Vagrantfile
Vagrant.configure(2) do |config|
config.vm.synced_folder ".", "/vagrant"
config.vm.define "sensu" do |m|
m.vm.box = "ubuntu/trusty64"
m.vm.hostname = "sensu"
m.vm.network "private_network", ip: "192.168.2.212"
m.vm.provider "virtualbox" do |v|
v.memory = 1024
end
end
config.vm.define "sensuclient" do |m|
m.vm.box = "ubuntu/trusty64"
m.vm.hostname = "sensuclient"
m.vm.network "private_network", ip: "192.168.2.213"
m.vm.provider "virtualbox" do |v|
v.memory = 512
end
end
end %
I have two servers now, sensu will be the server side and sensuclient will be the one connecting to sensu server.
I now have got two VMs, lets start them and then connect to server first:
vagrant up
Bringing machine 'sensu' up with 'virtualbox' provider...
Bringing machine 'sensuclient' up with 'virtualbox' provider...
==> sensu: Checking if box 'ubuntu/trusty64' is up to date...
==> sensu: A newer version of the box 'ubuntu/trusty64' is available! You currently
==> sensu: have version '20170505.0.0'. The latest is version '20171115.1.2'. Run
==> sensu: `vagrant box update` to update.
==> sensu: Clearing any previously set forwarded ports...
==> sensu: Clearing any previously set network interfaces...
==> sensu: Preparing network interfaces based on configuration...
sensu: Adapter 1: nat
..
....
...... <span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>
vagrant ssh sensu
We are going to use UFW(Uncomplicated Firewall for Ubuntu), once on the server, first let’s make sure it is installed:
vagrant@sensu:~$ ufw version The program 'ufw' is currently not installed. To run 'ufw' please ask your administrator to install the package 'ufw' vagrant@sensu:~$
vagrant@sensu:~$ sudo apt-get update && sudo apt-get install ufw -y
Now let’s check it’s status:
vagrant@sensu:~$ sudo ufw status Status: inactive
Before enabling let’s make sure port 22 is open so we still be able to connect to our server:
vagrant@sensu:~$ sudo ufw allow ssh/tcp Rules updated Rules updated (v6) vagrant@sensu:~$
Let’s enable it now:
vagrant@sensu:~$ echo 'y' | sudo ufw enable Command may disrupt existing ssh connections. Proceed with operation (y|n)? Firewall is active and enabled on system startup vagrant@sensu:~$ vagrant@sensu:~$ sudo ufw status verbose Status: active Logging: on (low) Default: deny (incoming), allow (outgoing), disabled (routed) New profiles: skip To Action From -- ------ ---- 22/tcp ALLOW IN Anywhere 22/tcp (v6) ALLOW IN Anywhere (v6) vagrant@sensu:~$
We can also check config files here:
vagrant@sensu:~$ sudo grep "### RULES ###" /lib/ufw/user.rules -A 5 ### RULES ### ### tuple ### allow tcp 22 0.0.0.0/0 any 0.0.0.0/0 in -A ufw-user-input -p tcp --dport 22 -j ACCEPT ### END RULES ### vagrant@sensu:~$
One thing you should know is UFW is mere user friendly frontend for more complex iptables, so let’s check what
our changes did to iptables:
vagrant@sensu:~$ sudo iptables -L | grep ssh ACCEPT tcp -- anywhere anywhere tcp dpt:ssh vagrant@sensu:~$
In order to check it’s logs we need to make sure logging is enabled:
vagrant@sensu:~$ sudo ufw logging ON Logging enabled vagrant@sensu:~$
If we assume UFW is using rsyslog as logging backend, a popular logging tool for linux, then it’s config should be somewhere down here:
vagrant@sensu:~$ grep -R UFW /etc/rsyslog.d/ /etc/rsyslog.d/20-ufw.conf:# Log kernel generated UFW log messages to file /etc/rsyslog.d/20-ufw.conf::msg,contains,"[UFW " /var/log/ufw.log /etc/rsyslog.d/20-ufw.conf:# Doing this will stop logging kernel generated UFW log messages to the file vagrant@sensu:~$
Assumption was right, plus we can see logs forwarded to /var/log/ufw.log. Let’s open another connection
from vagrant to check the logs, in fact, we can even check the logs without being logged into the VM, here is a trick you
can do with vagrant:
vagrant ssh sensu -c "sudo tail -f /var/log/ufw.log"
We should be able to see the logs, once they appear, but we will do it a bit later, first we will check the connection from client to server actually works:
Connect to second box, senssuclient:
vagrant ssh sensuclient
We now going to run a process on a server which will listen to commands on port 8080:
vagrant@sensu:~$ nc -l -p 8080
Let’s also disable firewall for a sake of connection test:
vagrant@sensu:~$ sudo ufw disable Firewall stopped and disabled on system startup
And once it is done, let’s try to connect to our server from client:
vagrant@sensuclient:~$ telnet 192.168.2.212 8080 Trying 192.168.2.212... Connected to 192.168.2.212. Escape character is '^]'. hello ^] telnet> close Connection closed.
All good, connection is working, now enable firewall back again:
vagrant@sensu:~$ echo 'y' | sudo ufw enable Command may disrupt existing ssh connections. Proceed with operation (y|n)? Firewall is active and enabled on system startup vagrant@sensu:~$
Once enabled, let’s try to connect once again:
vagrant@sensuclient:~$ telnet 192.168.2.212 8080 Trying 192.168.2.212...
As you can see connection hangs until timeout. As soon as we do that, our ufw logs should start to appear on the server:
➜ sensu vagrant ssh sensu -c "sudo tail -f /var/log/ufw.log" Nov 25 22:15:01 sensu kernel: [ 4273.462992] [UFW BLOCK] IN=eth1 OUT= MAC=08:00:27:9b:e5:a2:08:00:27:09:cb:4c:08:00 SRC=192.168.2.213 DST=192.168.2.212 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=19229 DF PROTO=TCP SPT=48623 DPT=8080 WINDOW=29200 RES=0x00 SYN URGP=0 Nov 25 22:15:02 sensu kernel: [ 4274.464543] [UFW BLOCK] IN=eth1 OUT= MAC=08:00:27:9b:e5:a2:08:00:27:09:cb:4c:08:00 SRC=192.168.2.213 DST=192.168.2.212 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=19230 DF PROTO=TCP SPT=48623 DPT=8080 WINDOW=29200 RES=0x00 SYN URGP=0 Nov 25 22:15:04 sensu kernel: [ 4276.470821] [UFW BLOCK] IN=eth1 OUT= MAC=08:00:27:9b:e5:a2:08:00:27:09:cb:4c:08:00 SRC=192.168.2.213 DST=192.168.2.212 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=19231 DF PROTO=TCP SPT=48623 DPT=8080 WINDOW=29200 RES=0x00 SYN URGP=0 Nov 25 22:15:11 sensu kernel: [ 4283.601921] [UFW BLOCK] IN=eth1 OUT= MAC=08:00:27:9b:e5:a2:08:00:27:0
As can see ufw blocks the incoming connection. We can see multiple records, why? Because this is how TCP works, it will try until
it times out. But let’s assume client doesn’t know what is happening and wants to troubleshot this matter.
Before we do any deep investigation on the client side to understand the cause of the issue we probably would simply run ping:
vagrant@sensuclient:~$ ping 192.168.2.212 PING 192.168.2.212 (192.168.2.212) 56(84) bytes of data. 64 bytes from 192.168.2.212: icmp_seq=1 ttl=64 time=0.504 ms 64 bytes from 192.168.2.212: icmp_seq=2 ttl=64 time=0.245 ms 64 bytes from 192.168.2.212: icmp_seq=3 ttl=64 time=0.323 ms ^C
Everything look OK, meaning server is available. We can go further now and try lsof:
vagrant@sensuclient:~$ lsof -i TCP COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME telnet 25846 vagrant 3u IPv4 290409 0t0 TCP sensuclient:48652->192.168.2.212:http-alt (SYN_SENT) vagrant@sensuclient:~$
As we can see telnet has successfully sent a ‘SYN_SENT’, we probably want to see ‘ESTABLISHED’, as otherwise it means
something preventing to establish the connection. For even deeper analysis we can next use TCP dump:
vagrant@sensuclient:~$ telnet 192.168.2.212 8080 Trying 192.168.2.212... ^C vagrant@sensuclient:~$ sudo tcpdump -U -w tcpdump_error -i eth1 'port 8080' tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
I am using ‘-U’ flag to make sure packets will be written to file as soon as they appear, without buffering.
Now, if you don’t want to open another session to run telnet from, you can send tcpdump to background, by pressing CTR+z, and then typing ‘bg’:
vagrant@sensuclient:~$ sudo tcpdump -U -w tcpdump_error -i eth1 'port 8080' tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes ^Z [2]+ Stopped sudo tcpdump -U -w tcpdump_error -i eth1 'port 8080' vagrant@sensuclient:~$ bg [2]+ sudo tcpdump -U -w tcpdump_error -i eth1 'port 8080' &
We can proceed now with telnet, run it for couple seconds, then terminate and bring back tcpdump and terminate it as well:
vagrant@sensuclient:~$ telnet 192.168.2.212 8080 Trying 192.168.2.212... ^C vagrant@sensuclient:~$ fg sudo tcpdump -U -w tcpdump_error -i eth1 'port 8080' ^C0 packets captured 8 packets received by filter 8 packets dropped by kernel
Now we will have the file in current directory which we need to copy to /vagrant directory which is synchronised with host machine, so we can open it on the hot with wireshark:
vagrant@sensuclient:~$ ls -lH tcpdump_error -rw-r--r-- 1 root root 384 Nov 25 22:40 tcpdump_error vagrant@sensuclient:~$ cp tcpdump_error /vagrant/ vagrant@sensuclient:~$ exit logout Connection to 127.0.0.1 closed.
As you see I copy the file, then exit VM, once on host, I can see the file:
➜ sensu ls -ls tcpdump_error 8 -rw-r--r-- 1 kayanazimov staff 384 25 Nov 22:42 tcpdump_error ➜ sensu
Now time to open it with Wireshark, if you don’t have it, you can install it on mac with brew:
brew install wireshark
Let’s open it:
wireshark tcpdump_errorv
As you can see 192.168.2.213, which is our client, sends SYN, to start synchronisation, but as it never receives any response,
it then sends TCP Retransmission, until connection times out.
If you have tail from server /var/log/ufw.log open you can see why, as our packets being dropped, and they never reach the server process.
Now let’s disable firewall and rerun tcpdump to better understand how tcp 3-way handshake works:
vagrant@sensuclient:~$ sudo tcpdump -U -w tcpdump_ok -i eth1 'port 8080' & [1] 30721 vagrant@sensuclient:~$ tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes vagrant@sensuclient:~$ telnet 192.168.2.212 8080 Trying 192.168.2.212... Connected to 192.168.2.212. Escape character is '^]'. hi ^] telnet> close Connection closed. vagrant@sensuclient:~$ fg sudo tcpdump -U -w tcpdump_ok -i eth1 'port 8080' ^C8 packets captured 8 packets received by filter 0 packets dropped by kernel vagrant@sensuclient:~$ cp tcpdump_ok /vagrant/ vagrant@sensuclient:~$ exit logout Connection to 127.0.0.1 closed. ➜ sensu wireshark tcpdump_ok
So what is going on here:
1 – client sends ACK
2 – server responds SYN, ACK
3 – client responds ACK
at this stage 3 way tcp handshake is finished, and next data exchange starts:
4 – client sends PSH, which means push all data
5 – server sends ACK
if server responded with some data as well, then we would have multiple PSH, ACK/ACK combinations as in the next example:
vagrant@sensuclient:~$ sudo tcpdump -U -w tcpdump_ok -i eth1 'port 8080' & [1] 32673 vagrant@sensuclient:~$ tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes vagrant@sensuclient:~$ telnet 192.168.2.212 8080 Trying 192.168.2.212... Connected to 192.168.2.212. Escape character is '^]'. 1 2 ^] telnet> close Connection closed.
On the server side type ‘2’ and ‘enter’, so data is sent to client:
vagrant@sensu:~$ nc -l -p 8080 1 2
Finally last 3 steps are
– client FIN, ACK, meaning client terminates connection
– server FIN, ACK, meaning server terminates connection too
– client ACK, last acknowledgement.