Debugging Example: 502 Bad Gateway From a Load Balancer

The load balancer answers, the targets are running, and every request still fails.

Troubleshooting bottom up behind a layer 7 load balancer finds a port and protocol mismatch here. A release moved the application from plain HTTP on port 8080 to HTTPS on port 8443. Its health endpoint still answers on a separate management port, 8081, so every target is healthy, and every request returns 502. The output is described, not captured.

Each check from inside the network, its result, and the layer it clears

InputApplication load balancer returns 502; target group sends HTTP to port 8080; health checks pass on 8081

Target health in the console
1. healthycleared
2. the instances run; layer 3 reachability works
Healthy only means port 8081 answered.
nc -vz 10.0.1.25 8080
1. Connection refusedfault
2. the host answers; nothing listens on 8080
A refusal clears the route and the security group, which would drop silently.
nc -vz 10.0.1.25 8443
1. succeededcleared
2. the application listens on 8443
The new release is running, on the new port.
curl -v http://10.0.1.25:8443/
1. Empty reply from serverfault
2. plain HTTP sent to a TLS port
The server expected a TLS ClientHello and closed the connection.
curl -vk https://10.0.1.25:8443/
1. 200 OKcause
2. TLS and HTTP work on the new port
-k skips the certificate check, since the address is not the certificate's name.

ResultThe target group still sends plain HTTP to port 8080. Changing it to HTTPS on port 8443 ends the 502s.

Why the load balancer said 502. It accepted the client's request, layer 7 on the client side, then failed to get a usable reply from the target: its connection to port 8080 was refused. A refusal or a broken reply is a 502; had the security group dropped the traffic instead, the load balancer would have waited and returned a 504.

Why the health checks did not help. They tested a different port from the one carrying traffic. A health check proves only what it checks, so it should hit the traffic port, over the same protocol.

Why bottom up here. The 502 is the load balancer's own page, so the client side was already fine, and everything worth testing lay between the load balancer and its targets. Each check, run from inside the network, cleared one layer.

Both this example and the shop's DNS fault were found the same way, one layer at a time, though they share no command beyond curl.

← → move through sections and cards