AWS ALB + express 502 ramdom

it is our env

in recent, we found our client called api, the server side return 502 randomly (bad gateway)

we try to find out the root cause in different way, including

• memory usage
• cpu usage
• server crash

but we found it is ok to service our client

util I google the answer, I found there many users has this problem

keyword: alb 502 express

this article tells us this detail, and the recent

Dealing with Intermittent 502's between an AWS ALB and Express Web Server

it is my understanding:

• node js has keep alive timeout(default 5 sec)
• loading balance has alive idle time(ex: 15 sec)

loading balance sent request to node js server , and create a connection between loading balance and node js server

2. next request will reuse the connection if node js keep alive and loading balance does not time out

it is great to reuse the connection to improve the performance

the problem happen in :

when loading send to request and node js send a package to loading balance close the connection (nearly at the same time), loading balance get close package, and return to client 502 bad gateway to client

resolve this problem:

let loading balance idle time < node js keep alive time

ensure node js does not send close package to loading balance when loading balance receive the client request and back post to server

Conclusion:

1. 502 bad gateway due to loading balance send start request package and node js send the close package to loading balance in the same time
2. to resolve this problem, seting

node js headersTimeou > node js keepAliveTimeout > alb idle time

sample:

const server = app.listen(8080)
server.keepAliveTimeout = 61 * 1000
server.headersTimeout = 65 * 1000

Ref:

Dealing with Intermittent 502's between an AWS ALB and Express Web Server

Regression issue with keep alive connections · Issue #27363 · nodejs/node