Hi there, hoping to find some help with a naive networking question.
I recently bought my first firewall appliance, installed Opnsense and am going to use it with my ISP modem in bridge mode, but while I’m learning I added it to my existing LAN with a 192.168.0.0/24 address assigned to the WAN port by my current DHCP. On the firewall’s LAN port I set up a 10.0.0.0/24 network and am starting to build up my services. So far so good, but there’s one thing I can’t get to work: I can’t port forward the firewall’s WAN IP to a service on the firewall’s LAN network and I can’t figure out why.
To illustrate, I would like laptop with IP 192.168.0.161 to be able to reach service on 10.0.0.22:8888 by requesting firewall WAN IP 192.168.0.136:8888.
Private IPs and bogons are permitted on the WAN interface and I have followed every guide I can find for the port forwarding, but the closest I have come to this working is a “connection reset” browser error.
Hope my question is clear and isn’t very dumb. Thanks for the help or any explanation why I might be struggling to get this to work. Am I missing something obvious?
UPDATE The thread is all over the place, but I have made some progress:
- RDR rule gets triggered when requesting 192.168.0.136:8888 from 192.168.0.123
- Apache logs show
2024-02-09T17:39:17.056208857Z 192.168.0.123 - - [09/Feb/2024:17:39:17 +0000] "GET / HTTP/1.1" 200 161 - a tcpdump (in spoiler below) on the apache container looks inconspicuous to my untrained eye, with the exception of checksum errors in some packets from the docker container (172.20.0.2). The last five lines, after the second GET request (why is there a second GET request?) appear in tcpdump after a delay of about five seconds.
tcpdump
192.168.0.123.54120 > 172.20.0.2.80: Flags [S], cksum 0xfdc5 (correct), seq 4106772895, win 64240, options [mss 1460,sackOK,TS val 1485594466 ecr 0,nop,wscale 7], length 0
17:45:14.918207 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.20.0.2.80 > 192.168.0.123.54120: Flags [S.], cksum 0x6d68 (incorrect -> 0x2fd7), seq 3999845366, ack 4106772896, win 65160, options [mss 1460,sackOK,TS val 1469298770 ecr 1485594466,nop,wscale 7], length 0
17:45:14.924098 IP (tos 0x0, ttl 62, id 63128, offset 0, flags [DF], proto TCP (6), length 52)
192.168.0.123.54120 > 172.20.0.2.80: Flags [.], cksum 0x5b30 (correct), ack 3999845367, win 502, options [nop,nop,TS val 1485594472 ecr 1469298770], length 0
17:45:14.924102 IP (tos 0x0, ttl 62, id 63129, offset 0, flags [DF], proto TCP (6), length 134)
192.168.0.123.54120 > 172.20.0.2.80: Flags [P.], cksum 0x70f5 (correct), seq 4106772896:4106772978, ack 3999845367, win 502, options [nop,nop,TS val 1485594472 ecr 1469298770], length 82: HTTP, length: 82
GET / HTTP/1.1
Host: 192.168.0.136:8888
User-Agent: curl/7.74.0
Accept: */*
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /</title>
</head>
<body>
<h1>Index of /</h1>
<ul></ul>
</body></html>
17:45:14.924119 IP (tos 0x0, ttl 64, id 34500, offset 0, flags [DF], proto TCP (6), length 52)
172.20.0.2.80 > 192.168.0.123.54120: Flags [.], cksum 0x6d60 (incorrect -> 0x5ad1), ack 4106772978, win 509, options [nop,nop,TS val 1469298776 ecr 1485594472], length 0
17:45:14.924407 IP (tos 0x0, ttl 64, id 34501, offset 0, flags [DF], proto TCP (6), length 364)
172.20.0.2.80 > 192.168.0.123.54120: Flags [P.], cksum 0x6e98 (incorrect -> 0x0a74), seq 3999845367:3999845679, ack 4106772978, win 509, options [nop,nop,TS val 1469298776 ecr 1485594472], length 312: HTTP, length: 312
HTTP/1.1 200 OK
Date: Fri, 09 Feb 2024 17:45:14 GMT
Server: Apache/2.4.58 (Unix)
Content-Length: 161
Content-Type: text/html;charset=ISO-8859-1
17:45:14.929077 IP (tos 0x0, ttl 61, id 0, offset 0, flags [DF], proto TCP (6), length 40)
192.168.0.123.54120 > 172.20.0.2.80: Flags [R], cksum 0x1833 (correct), seq 4106772978, win 0, length 0
17:45:15.138862 IP (tos 0x0, ttl 62, id 63130, offset 0, flags [DF], proto TCP (6), length 134)
192.168.0.123.54120 > 172.20.0.2.80: Flags [P.], cksum 0x701e (correct), seq 4106772896:4106772978, ack 3999845367, win 502, options [nop,nop,TS val 1485594687 ecr 1469298770], length 82: HTTP, length: 82
GET / HTTP/1.1
Host: 192.168.0.136:8888
User-Agent: curl/7.74.0
Accept: */*
17:45:15.138872 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
172.20.0.2.80 > 192.168.0.123.54120: Flags [R], cksum 0xb48d (correct), seq 3999845367, win 0, length 0
17:45:19.995097 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.20.0.1 tell 172.20.0.2, length 28
17:45:19.995161 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.20.0.2 tell 172.20.0.1, length 28
17:45:19.995164 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.20.0.2 is-at 02:42:ac:14:00:02, length 28
17:45:19.995164 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.20.0.1 is-at 02:42:b8:07:c2:99, length 28```
UPDATE 2 I see the exact same behaviour with a second VM and apache directly installed on it instead of in a docker container.
UPDATE 3
Thank you everybody for coming up with ideas. And thank you most of all to @maxwellfire@lemmy.world: The culprit was the Filter rule association in my Port Forward settings which I had as Add associated filter rule but needs to be Pass. As soon as that is set, everything works.
The full solution is a NAT Port forwarding rule with filter rule “pass”, an outbound NAT rule for hairpinning, and everything related to reflection turned off in Settings > Advanced. It’s that easy! 😵💫
Post a screenshot of your NAT > Port Forward rule if you can, that will be the easiest way to help I think
The docker01 alias is a host alias with 10.0.0.22 and there’s an apache test container running on port 8888.
I have created a pass any in rule on WAN (just until I figure out what’s wrong)
In firewall > settings > advanced, I have set “reflection for port forwards” and “automatic outbound Nat for reflection” although I’m not sure if that is needed.
Is there any other info I can provide?
Your filter rule association is set to ‘rule’. What is that associated rule, and do things work if you change it to ‘pass’?
https://www.reddit.com/r/opnsense/comments/puty62/correct_option_for_filter_rule_association_when/
Son of a gun!!! Thank you so much! I spent HOURS changing every setting except this one and actually came to the conclusion that it must be something to do with my ISP’s modem or DNS or something.
The rule is the “associated filter rule” OPNsense automatically creates (interfaces are WAN and LAN) and it triggers as a “pass” just fine when I send a request. (I’m attaching another screenshot from the live log below.)
You don’t happen to have a clue WHY this rule breaks everything?
Associated filter rule
Live log with associated filter rule active (leads to
curl: (56) Recv failure: Connection reset by peer)Here’s some more: From behind the firewall (i.e. from a 10.0.0.x IP) the port forward works (which would be a reflection, I suppose?).
From in front of the firewall, I get “connection reset”, which I interpret as somewhat working but then breaking somewhere else. Does that make sense?
Instead of connecting with a web browser, can you try using curl or telnet just to check if you’re getting through at the TCP/IP connection level?
Like this?
~$ curl 192.168.0.136:8888 curl: (56) Recv failure: Connection reset by peerAnd here’s what this request looks like in the firewall log:
The rules and the log looks find to me, so I suspect something else is going on. Since you’re seeing the request hit apache.
Have you tried another browser just to see if that’s the issue?
Please take a look at my updated original post. I have added some information and a tcpdump.
Further digging: The request reaches the docker container, which returns
200 OK.my-apache-app | 2024-02-09T12:53:22.925676854Z 192.168.0.123 - - [09/Feb/2024:12:53:22 +0000] "GET / HTTP/1.1" 200 161What is going on here? Do I need some rules in the other direction, on top of “Automatic outbound NAT rule generation”?