Let us suppose that you have a firewall set up in the traditional manner: an interface to the inside, one to the outside, perhaps some NAT for the IPv4 masquerading and straightforward firewalling for the IPv6. (v6 is not on the menu for today, do not worry.) There is a default deny for incoming NEW (SYN) packets, a default accept for related inbound packets, specific services are whitelisted (HTTPS? DNS? IMAPS? SSH, of course). And you default-accept outbound packets, because it is too much of a pain to do a default deny and then go through every outbound connection ever.

What are you sending out, though? What are you connecting to? If you feel paranoid enough, you capture your outbound traffic and pretty soon you need extra terabyte drives for your logs.

Go into your iptables rules set and find where you allow outbound traffic, probably something like this:

iptables -A FORWARD -i $INSIDE -j ACCEPT

Before that line, we can log the packets that start new connections:

iptables -A FORWARD -i $INSIDE -m state --state NEW -j LOG --log-level 4 --log-prefix 'NEW-OUTBOUND '

There is a space inside the log prefix quoted string, because it is a very literal prefix.

Go over to your syslog configuration and copy kernel.warn messages – that would be what our log rule will emit – to a useful file, perhaps /var/log/iptables-newoutbound.log,

Reload your syslogd, reload your iptables rules. Take a look at the logfile and make sure it has some content.

Let it sit for a while. I would recommend 24 hours or more, depending on what you think normal behavior is on your network. If you are at a business, you might want to run a Friday through Monday sweep, assuming this is not a holiday weekend. At home, any typical weekday might do, or a weekend day if everyone is in the house.

Comment out the iptables line and reload the rules again.

Time for some rude log analysis. Our tools will be awk, grep, sort, wc and uniq.

Run wc -l on your logfile. Now you know the total number of connections you captured. head will show you the first few lines. Figure out which fields (spaces are field delimiters) are interesting to you. You can print them out by piping your file through

awk '{ print $1" "$12" "$13" "$16 }'

$1 is the first field. You print spaces in quotes because awk is very literal, too.

Mar  5 14:59:59 gw kernel: [158144.383443] NEW-OUTBOUND IN=eth1 OUT=eth4 MAC=a0:37:9f:01:9a:91:d2:3d:73:93:33:1a:0c:01 SRC=192.168.0.102 DST=69.171.239.11 LEN=79 TOS=0x00 PREC=0x00 TTL=63 ID=52128 PROTO=UDP SPT=36268 DPT=53 LEN=59

A typical line will start out with a date and time stamp, the hostname, an indication that is a kernel message, another, high-resolution timestamp, and then the log prefix you had configured.

Then we get: inbound interface, outbound interface, MAC addresses; source IP, destination IP, length, TOS field, PREC, time to live, ID, protocol, source port, destination port, and finally protocol-relevant information like TCP window number, flags, or payload length.

That is a lot.

Things which I usually think are interesting: the source IP, the destination IP, the destination port, and the protocol type. For example, the packet logged above is UDP port 53, which is DNS. We made a request to 69.171.239.11, which is a nameserver owned by Facebook.

We can separate out various protocols into their own files. To get them out of var/log/iptables-newoutbound.log, we can do a two-step dance:

grep DPT=53 /var/log/iptables-newoutbound.log > iptdns
grep -v DPT=53 /var/log/iptables-newoutbound.log > ipt-dns

The first file contains only the DNS entries. The second files contains everything else, which we will continue to break down:

grep DPT=123 ipt-dns > iptntp
grep -v DPT=123 ipt-dns > ipt-dns-ntp

NTP is common. Our new files contain the NTP connections in iptntp and things which are neither DNS nor NTP in ipt-dns-ntp.

We can get HTTP and HTTPS out, too:

grep DPT=80 ipt-dns-ntp > ipthttp
grep -v DPT=80 ipt-dns-ntp > ipt-dns-ntp-http
grep DPT=443 ipt-dns-ntp-http > ipthttps
grep -v DPT=443 ipt-dns-ntp-http > ipt-dns-ntp-http-https

If you add SSH (DPT=22) and anything else you use a lot – SMTP (DPT=25, 467 and 587), IMAP (143 and 993) you should discover that you have relatively few lines left.

Run wc -l ipt* to see the relative frequencies.

Eventually you will get all the common protocols separated – really, I would expect only a few more than this – and have some leftovers.

You can sort them and see how many duplicates you have:

sort ipt-dns-ntp-http-https-smtp-imap-ssh |uniq -c > mess.counted

Did you know that uniq -c takes a sorted file and outputs the number of times each line appears, followed by a single copy of the line? I only saw that option a few weeks ago.

Grovelling through mess.counted will inevitably come up with some things you do not recognize. You can look for protocols in /etc/services or by plugging it into a helpful site such as SpeedGuide. whois can tell you who owns a particular IP, which is often of use.

And then, when you finally decide that there is no good reason for one of your machines to have reached out to that odd machine on that weird port, you can write a new iptables rules to drop those packets. I recommend that you watch carefully for a few days – nobody will be pleased if you cannot revert the change swiftly so that they can play their game again.