Posts

About

RDS Connections in a Serverless World

March 25, 2020

Introduction

In January I gave a talk called From Dynamodb to RDS and Back: Database Considerations in a Serverless World at an Indy DevOps meetup. My talk broadly covered topics such as DynamoDB, RDS, cold starts, and databases connections in serverless. In this post I would like to take a deeper dive into RDS connections with Lambda and how that compares to traditional connection pooling. I will also talk about how cold starts affect RDS connections.

Traditional Connection Pooling

connectionPooling

Most applications use some type of database to persist data for users. Servers traditionally run the application’s code which will create connections to the database to access and save data. Creating a connection to the database is often expensive and takes time. Once the connection is established though, data can flow back and forth from the server and database quickly. Application servers are usually multithreaded and can handle multiple requests at once. Each of these requests might need a connection to the database. For both of these reasons, the concept of connection pooling was created. A connection pool is usually created when the server is started. The pool creates some number of databases connections that it shares across requests that need access to the database. This approach keeps requests running quickly by not having to create new connections and also reduces the load on the database since connections can be shared.

Lambda and RDS Connections

lambdaConnection

With serverless and lambda, connections to RDS are a different. A lambda function has some container that runs your application code. A single container only handles a single request at a time. This means that connection pooling doesn’t really make sense. A single container only needs a single connection to the database at once. This database connection can be created in one of two places:

  1. Outside your handler code
  2. Inside your handler code.
// outside handler code

module.exports.handler = async event => {
  // inside handler code
}

If it is in the global space outside of the handler code, then the connection will be maintained between requests and kept around until the container is killed. This means you only have to take penalty of estabilishing the connection to RDS once when the container is intialized. When multiple requests come in at the same time, new containers are spun up to handle them. For every concurrent lambda container you have running, you also have a single connection to the database. This is a huge problem because you can quickly exceed the connection limit on an RDS instance.

If the connection is created inside the handler code then it will not be maintained between requests. Just be sure to close the connection at the end of the request so there aren’t zombie connections hanging around. This approach has the benefit that connections are closed meaning N number of concurrent lambdas might not be N number of connections to the database. However the time penalty of establishing a connection to RDS happens on every request.

How Cold Start Affect Connections

coldStartAndConnections

In my last post I talked in depth about cold starts. They can have a really adverse affect on the number of RDS connections which can exhaust the connections to a RDS cluster. The image above is a good example of this. A lambda function using RDS can have cold start times of 4 or more seconds. This is because the lambda has to be inside of a VPC for security purposes and the 1-2 seconds time to initialize the connection to RDS. In the example above we assume that we have 1 request per second to the lambda function. At 0 seconds the first connection comes in and there are not any warm lambda containers to service that request. A new container has to be spun up which takes 4 seconds to cold start and then quickly service the request. At 1 second a new request comes in. The first container is still trying to spin up so a new container must be spun up. At 2 and 3 seconds the same thing happens and two more containers are created. At 4 seconds the first container is finally done and can start serving more request coming in. It is now warm so it can handle any new requests coming in. This shows how cold starts caused four containers to be created each having 4 connections to RDS. If more lambdas are using the same RDS cluster, you can imagine how colds starts can quickly cause problems.


Written by Jacob Oakes
I am a software architect who enjoys learning new things, clean code, and automated tests.