Rachel Kroll
I ran into a troubleshooting scenario the other day which ended up adding to the list of things that I need to check on when trying to figure out why packets seem to be disappearing. It went like this.
I showed up at a site where I'm running some weather station sensors and jumped on the console of one of the Linux boxes. My visit was about adding some sensors to some new areas, and I wanted to see how things were going. In particular, I wanted to see how the receiver on the local machine was doing, and what it had managed to log of late.
(Just imagine the port number is 1234 here.)
$ thermo_cli ::1 1234 rpc error: deadline exceeded while awaiting connection
... what? That made no sense. The thing was running.
I looked in 'ss' to make sure it was listening on that port and specifically was ready for IPv6 connections. It was.
LISTEN 0 1024 0.0.0.0:1234 0.0.0.0:* users:(("thermo_server",pid=1141761,fd=4))
LISTEN 0 1024 [::]:1234 [::]:* users:(("thermo_server",pid=1141761,fd=5))
I tried netcat instead... same thing. Instead of connecting, it just hung there. I looked in ip6tables... nothing. This host has no rules at the moment: nothing in 'filter', 'nat', 'mangle', etc. This should Just Work.
This site isn't running IPv6 natively due to a dumb ISP, but there are still ULA and link-local addresses, so I tried one of those from another host on the same network. That also went nowhere.
Looking in tcpdump, it was pretty clear: SYN comes in, nothing returns.
But, at the same time, I could change from that port to something like 22 and it would work fine. I'd get the usual banner from sshd.
Assuming it was something stupid I had done with my own code somehow, I ran another program as a test, then tried connecting to it over v6. It worked fine.
I straced it to make sure it was handing the IPv6 listener fd to poll(). It was. It wasn't getting any sort of notification from the kernel. But, actually, hold up, no SYN/ACK happened, so there's no way it would have gotten anywhere near userspace.
I stopped the service with systemctl and put up a netcat listening on the same TCP port (nc -6 -l -p 1234) ... and then I could connect to that just fine. So, it's not something magic about the port, somehow. It's just that port when it's going to the service which normally runs on it.
I started making a list to see what the patterns were. This box, this program, talking to ::1? Bad. Another box at this site, same program, also talking to ::1? Same problem.
Was it because this site has no v6 routing to the outside world? That makes no sense as to why ::1 wouldn't work, but, hey, one more thing to discard. I invented a fake route. Nothing happened (fortunately).
Next I started up a Debian VM on my laptop, hooked one of the radios to it, and started the receiver program on it by hand. It ran just fine, and accepted traffic over IPv6 to that same port without any trouble. It's on the same v6-route-less network as the other hosts, so what's up with that, right?
Maybe I did something stupid with the config file for the program, so I copied that across verbatim from one of the site hosts instead of just making a fresh one for testing. It didn't change things.
What if I dump the v4 listener on that port and just run the v6 listener? Nothing. What if I add a listener on another port? Nothing. Now that port also drops packets when I try to connect to it that way.
I don't know what it was about this last point, but somewhere around here, a couple of ideas finally connected in my head and I went "uh, systemd".
The failing instances were both running as systemd services. The successful instances (whether the thermo_server program, or my other test stuff) were just me doing ./foo from a shell.
That's when I thought about the hardening work I'd been applying to my systemd services of late. I've been taking away all kinds of abilities that they really don't need.
One of the newer tricks in systemd is that you can do "IPAddressDeny=" and then "IPAddressAllow" and keep a program from exchanging traffic with the rest of the world. For a program that's only ever supposed to talk to the local network, this was a good idea.
That's when I saw it: I had 127.0.0.0/8 and the local RFC 1918 networks on the Allow line, but *not* ::1, never mind the ULA prefix or the link-local v6 stuff. Adding ::1 and doing the usual daemon-reload && restart <service> thing fixed it.
Here's the deal: systemd implements that by injecting bpf program(s) when you ask it to filter traffic by IP addresses in the .service file. When this thing rejects traffic, it just drops it on the floor. It does this past the point where ip[6]tables would match it, and well before the point where it would generate a SYN/ACK or whatever else.
There are no counters associated with this, and it doesn't generate any messages in the syslog or whatever. The packets just disappear.
It's effectively equivalent to an ip[6]tables rule of "-j DROP", but at least those rules have byte and packet counters that you'd see incrementing when you're smashing your head against it. This just makes the packets disappear and nobody has any idea what's going on.
So, if you ever see traffic effectively being blackholed to the port or ports which are bound for a particular systemd-run service without anything showing up in your iptables (or let's face it, nftables) stuff, you'd better check to see if there are IPAddress* rules in the .service file. That might just explain it.
Hopefully you'll remember this and not waste a bunch of time like I just did.