As we're into the last quarter of 2024, we have seen an explosion of new dev tools, companies, and programming models transforming how developers approach data storage and retrieval. Many of these databases offer high availability, geographic scaling, and seamless integrations with other cloud services, but they come with challenges.

Costs can also scale quickly, especially with large datasets, and control over data can be limited, which might be a concern for developers, founders and companies.

In this post, I've shared my overview of the databases that pair exceptionally well with modern app architectures, focusing on serverless.

💡 This blog post is inspired by Lee Robinson's 2023 State of Databases for Serverless & Edge and focuses on 2024, incorporating my personal experience working with various developer tools startups.

Criteria

To keep this overview focused, I'll concentrate on transactional workloads and three primary criteria:

• Services that pair well with serverless.
• Solutions compatible with JavaScript and TypeScript.
• Who should use and who should avoid Serverless?

Disclaimer: I recently did consulting work for Neon serverless database, and it blew my mind to learn how easy it is to go serverless these days. I have also used many of these tools for my projects and do consulting work for other developer tools.

What is Serverless?

Before we get started, let's talk about the term "serverless" which should never have become a thing because there's always a server, and it is a marketing term that doesn't refer to any specific computer term.

So maybe the term serverless became a thing because configuring your server is a huge hassle and a headache and not a great idea for many beginners.

As a user, you and I do not need to worry about a server which means no provisioning, scaling, upgrading, or updating the server.

Serverless just means not hosting your own infrastructure + boxes. Amazon Lambda is a good example where you run arbitrary scripts on their infrastructure on demand without having always-available servers. So, being serverless means you don't have to think about servers. Of course there are servers hard at work handling your application's requests, but that's their problem, not yours. They do all the work behind the scenes of allocating, configuring, and maintaining the servers. Instead of paying for servers, you and I pay for the requests that your application makes to the database and the storage that your data consumes.

so "Serverless" means:

1. Never having to patch another OS.
2. Never being paged when a rack goes down.
3. Never needing to rollout software on a hundred nodes.
4. Never paying for idle infrastructure.
5. Never losing business because you can't scale.

A New Programming Model

Relational databases have been the backbone of application development for over 25 years, but serverless computing demands a shift in how we interact with these databases. The new programming model emphasizes:

• Connectionless Protocols: Traditional databases use stateful protocols, which can be inefficient for serverless environments where functions scale to zero. New solutions offer connectionless access via HTTP APIs or WebSockets, abstracting connection management away from developers.

• Web-Native Interfaces: Modern databases are increasingly designed to work seamlessly with browser-based APIs and lighter runtime environments, making them ideal for edge computing. This shift makes databases more compatible with web-centric development, where data is fetched using standard web protocols.

• Lightweight Client Libraries: The burden of managing complexity is shifted to the database vendor to simplify development. This allows client libraries to be thin and easy to integrate, with the database provider handling tasks like connection pooling and caching.

• Type Safety: TypeScript's popularity has driven demand for databases and libraries that support type-safe interactions, ensuring better developer experience and reducing runtime errors.

Example: Consider solutions like Supabase and Neon. Supabase offers a Postgres-based solution with an HTTP API, making it serverless-friendly. Neon, on the other hand, uses a WebSocket-based approach to manage Postgres connections efficiently in serverless and edge environments.

For type-safe development in serverless environments, several tools and databases stand out. Here are some examples:

1. Prisma: Prisma is a popular ORM for TypeScript and Node.js, providing a type-safe interface for interacting with databases like PostgreSQL, MySQL, and MongoDB. It automatically generates TypeScript types from your database schema, ensuring that your queries are type-safe. This makes it easier to catch errors during development rather than at runtime.

2. Supabase: An open-source alternative to Firebase, Supabase provides a complete backend as a service, including a Postgres database, authentication, and storage. It's designed with TypeScript support, ensuring type safety across the board, from database interactions to API requests.

3. Neon: Neon is a serverless PostgreSQL database focusing on providing a type-safe experience for front-end developers. It offers features like branching and scaling, making it a powerful tool for projects that require a scalable and resilient database.

4. Turso: Built on SQLite, Turso is an edge-hosted, distributed database that integrates well with TypeScript. It allows developers to define APIs in TypeScript, providing a type-safe interface handy for serverless and edge computing applications.

5. Xata: Xata is a serverless database that integrates relational and search functionalities. It offers type-safe APIs and has built-in AI features, making it a powerful tool for modern applications requiring strong type safety and advanced data handling capabilities.

Trends in 2024

The shift in the programming model has given rise to several key trends:

1. Databases as Comprehensive Data Platforms: Databases are no longer just for storing and retrieving data; they're evolving into comprehensive data platforms that integrate search, analytics, auth, and more.

2. Decoupled Storage and Compute: Inspired by solutions like Snowflake, more databases separate storage and compute, allowing cost-effective scaling and improved performance in serverless environments.

3. Automated Scalability: Developers increasingly seek databases that can scale automatically without manual intervention. Solutions like DynamoDB and Upstash are leading the way by offering auto-scaling capabilities.

4. Global Data Distribution: While global replication remains challenging, new approaches like specialized data APIs and user-specific data stores are emerging to handle the complexities of edge computing.

Serverless Database

Let's look at some of the established database solutions with code examples.

Firebase Firestore

Firestore has been a reliable choice for developers building real-time applications, especially in mobile contexts. It offers seamless integration with Firebase Authentication, making it an excellent choice for cross-platform applications.

Example: Retrieving Documents from Firestore

Copy
1import { getFirestore, doc, getDoc, DocumentData } from "firebase/firestore";
2
3const db = getFirestore();
4const docRef = doc(db, "users", "userId123");
5const docSnap = await getDoc(docRef);
6
7if (docSnap.exists()) {
8  const data: DocumentData = docSnap.data();
9  console.log("Document data:", data);
10} else {
11  console.log("No such document!");
12}

Learn more about Firebase Firestore

Firestore's real-time capabilities and ease of use make it a strong contender for applications that require real-time data synchronization across multiple clients.

MongoDB Atlas Serverless

MongoDB's serverless offering, Atlas, has been a game-changer, providing a full-featured NoSQL database with serverless scalability.

Example: Querying MongoDB Atlas Serverless

Copy
1import { MongoClient } from 'mongodb';
2
3const uri = process.env.MONGODB_URI as string;
4const client = new MongoClient(uri);
5
6async function run() {
7  try {
8    await client.connect();
9    const database = client.db('mydb');
10    const collection = database.collection('users');
11    const user = await collection.findOne({ name: 'Alice' });
12    console.log(user);
13  } finally {
14    await client.close();
15  }
16}
17
18run().catch(console.dir);

Explore MongoDB Atlas Serverless

MongoDB Atlas's serverless model allows for flexible scaling while maintaining the robust feature set that MongoDB is known for, including full-text search and analytics capabilities.

PlanetScale

PlanetScale, built on Vitess, offers MySQL with a serverless twist. One of its standout features is the ability to branch databases, similar to how you might branch code in Git, making it easier to manage schema changes and collaborate on database development.

Example: Querying PlanetScale with Kysely

Copy
1import { Kysely, sql } from 'kysely';
2import { PlanetScaleDialect } from 'kysely-planetscale';
3
4interface Database {
5  users: {
6    id: number;
7    name: string;
8    email: string;
9  }
10}
11
12const db = new Kysely<Database>({
13  dialect: new PlanetScaleDialect({
14    host: process.env.DATABASE_HOST,
15    username: process.env.DATABASE_USERNAME,
16    password: process.env.DATABASE_PASSWORD,
17  }),
18});
19
20async function getUsers() {
21  const users = await db.selectFrom('users').selectAll().execute();
22  console.log(users);
23}
24
25getUsers();

PlanetScale's serverless capabilities and powerful features like branching make it a top choice for modern web applications.

AWS Aurora Serverless

AWS Aurora Serverless offers a robust, fully managed PostgreSQL and MySQL-compatible database with the ability to automatically scale based on application demand.

Example: Connecting to Aurora Serverless with Node.js

Copy
1import { Client } from 'pg';
2
3const client = new Client({
4  host: process.env.AURORA_HOST,
5  user: process.env.AURORA_USER,
6  password: process.env.AURORA_PASSWORD,
7  database: process.env.AURORA_DB,
8});
9
10async function queryDatabase() {
11  try {
12    await client.connect();
13    const res = await client.query('SELECT NOW()');
14    console.log(res.rows[0]);
15  } catch (err) {
16    console.error(err);
17  } finally {
18    await client.end();
19  }
20}
21
22queryDatabase();

Aurora Serverless is ideal for applications that need to scale in and out automatically without manual intervention, offering the performance and reliability of traditional RDBMS systems.

Rising Serverless Database Solutions

In addition to the established players, several rising stars are pushing the boundaries of what's possible with serverless databases.

Neon: Serverless PostgreSQL with a Twist

Neon is a serverless PostgreSQL offering that uniquely separates storage and compute and enables instant scaling and cost-effective operation, especially in edge environments. Neon also supports database branching, making it a powerful tool for modern development workflows.

Example: Using Neon's Serverless Driver

Copy
1import { Client } from '@neondatabase/serverless';
2
3const client = new Client(process.env.DATABASE_URL);
4
5async function getTime() {
6  await client.connect();
7  const { rows: [{ now }] } = await client.query('SELECT NOW();');
8  await client.end();
9  return now;
10}
11
12getTime().then(console.log);

Discover Neon Serverless PostgreSQL

For more details on how Neon's serverless driver works and its impact on the developer experience, check out Neon's blog post.

Supabase: The Open-Source Firebase Alternative

Supabase is an open-source alternative to Firebase, built on PostgreSQL. It offers a full suite of backend services, including database, authentication, and storage.

Example: Querying Supabase with TypeScript

Copy
1import { createClient } from '@supabase/supabase-js';
2
3const supabaseUrl = 'https://xyzcompany.supabase.co';
4const supabaseKey = process.env.SUPABASE_KEY as string;
5const supabase = createClient(supabaseUrl, supabaseKey);
6
7interface Country {
8  id: number;
9  name: string;
10  code: string;
11}
12
13async function getCountries() {
14  const { data: countries, error } = await supabase
15    .from<Country>('countries')
16    .select('*');
17
18  if (error) {
19    console.error('Error fetching countries:', error);
20    return;
21  }
22
23  console.log(countries);
24}
25
26getCountries();

Explore Supabase

I've written a detailed comparison of Supabase vs Neon.

Other Solutions

In addition to the above, several other solutions deserve mention for their unique approaches to data storage and management in serverless and edge environments:

• Xata: Combining database, search, and analytics, Xata offers an intuitive, spreadsheet-like UI that appeals to both developers and non-developers alike.

• EdgeDB: Challenging the traditional ORM/database separation, EdgeDB offers an optimized approach to querying and managing data.

• Hasura: With its ability to connect multiple data sources and expose them via a GraphQL API, Hasura simplifies the process of building scalable, serverless applications.

Who should go serverless, and who should avoid going serverless?

Who Should go serverless:

Many people might say, "We can optimize it later" for a scale-up of orders of magnitude, but this is nearly always wrong.

Reddit is a perfect example of running into scaling issues recently but is locked into architecture with few escape hatches built in; when a bridge fails, it'll just fail. When software runs into a scaling limit, it'll degrade and fall on its arse constantly and be terrible as long as it takes for the software team to completely rework its entire architecture, often having to learn entirely new technologies. I don't know the relevant details about Reddit, but the assumption is that the early Reddit people could have quickly built something more scalable, yet there are tech reasons why the later people, with far more resources, can't.

Reddit: Lessons Learned from Mistakes Made Scaling to 1 Billion Users

The amount of data collected worldwide is growing faster than the number of database engineers. So the question isn't whether you have 10k req/s now but whether you expect to in the future.

• If you are designing a blog, then yeah, you probably don't need to worry about it.
• If you are starting a social network or SAAS business app, then you probably do.

Who Shouldn't go serverless:

Tradeoffs are tradeoffs. For example - "by running a k8s cluster when you don't need one, a cost you pay is the overhead." but 👇

but make sure your know all the rules of the game otherwise...you'll see the huge bills once your app goes viral.

Nowadays, there's a ton of idling compute capacity running on the client side that could be used to spin up a personal cloud environment with all core services if one has the proper knowledge. For anyone that has a core to spare and a few GBs of memory, which should be easy for most modern midrange hardware, I would recommend they save that $5 paying for a third party VPS, go deploy an open source hypervisor such as KVM, and run the virtual server on your hardware.

Comparison of free tiers: https://paul.totterman.name/posts/free-clouds/

Final Thoughts

There are many other databases; I just named several, and there are hundreds more for every possible niche, but DBaaS seems to be rapidly evolving. It looks like there will be a lot of new developments in the next few years to help reduce the overall dev effort.

🔥 Serverless is my secret weapon for hobby projects. When my lambdas are snoozing and S3's collecting dust, it's like having a VIP pass to free cloud computing. This blog is? Powered by Neon and Vercel, and my wallet's pretty happy too. Only splurged on the domain name 🚀💸

And last, maybe I'm in my own bubble in "database/backend", but there have been many new startups in this space, which is fun to see 👊

It's also interesting to see more database services support HTTP APIs and as a person living in the EU, an unpopular saying is that the current situation in the EU makes these SaaS offerings almost irrelevant.

Companies and governments are very reluctant to pick any cloud services, and It's a NIGHTMARE to sell SaaS to European Govs if it's hosted with a US company (except, of course, you are one of said US companies. Then you can sell whatever you want. Yes, it's bizarre).

I'm not against or in favour of cloud services, but I wish more people understood their costs upfront and picked based on merits. Some teams benefit greatly from cloud services, but usually if they're not cost sensitive.

In my last job, we had everything on AWS because we'd never scaled to somewhere where it'd become expensive and convenient. The moment the bill starts to bite, people ought to at least price out alternatives and consider hybrid setups.

Interesting Videos, Reads, Tweets on Serverless

— The business case for serverless

"Serverless promised to abstract away all the infrastructure management and allow developers to focus solely on writing code. However, the reality has been somewhat different. The complexity of deploying and managing serverless applications has led to frustrations, particularly in areas like debugging, monitoring, and vendor lock-in."

— Why the Serverless Revolution Has Stalled

"Serverless is an alluring marketing term that suggests a world where developers don’t have to worry about servers. But the reality is that it often adds more complexity and removes control from developers, creating a dependency on cloud providers that can be hard to break."

— Don’t Be Fooled by Serverless

Serverless was a big mistake... says Amazon

Check out this video by Fireship for more insights:

And here are some interesting takes from industry experts:

For more serverless stories, check out Serverless Horrors.