With the growing popularity of Serverless, I wanted to explore how to to build a (Big) Data platform using Amazon’s serverless services. In this article, we will look into what is a data platform and the potential benefits of building a serverless data platform. We will also look at the architectures of some of the serverless data platforms being used in the industry.

What is a Data Platform?

Over the last decade, software applications have been generating more data than ever before. As computation and storage have become cheaper, it is now possible to process and analyze large amounts of data much faster and cheaper than before.

Introduction

A lot of companies don’t fully utilize Spot Instances because they are worried about making their service unreliable. AWS Spot Fleet is a service that lets you create a fleet of EC2 instances that can help make your application more reliable and significantly cheaper to run. A fleet is composed of both Spot and On-Demand Instances. In this article, we will learn more about Spot Fleet, how they are different from Spot Instances and how they can be used to make your infrastructure cheaper and more reliable.

Spot Instances

Before we go deeper into Spot Fleet, it is useful to understand how Spot instances work. AWS has three tiers of EC2 instances. They are:

• On-Demand Instances: This is generally the default option when you create an EC2 instance. Most of the time, when you see any EC2 pricing online, it refers to the On-Demand pricing. On-Demand instances are the most expensive because you have no commitment to AWS. You can terminate the instance at any time and only pay for how long you use the instance.

• Reserved Instances (RIs): RIs are a billing construct instead of being an actual EC2 instance tier. RIs can be significantly (40-60%) cheaper than On-Demand instances. However, in return for the discount, you have to make a long term commitment (1-year or 3-year).

• Spot Instances: AWS has a lot of spare compute capacity (unused EC2 instances) which they offer as Spot Instances. Spot Instances are generally the cheapest option (50%-80% lower than On-Demand) amongst the three. However, since Spot Instance availability is predicated upon the demand, Amazon provides no guarantee that once you get a Spot Instance you can run it indefinitely. This is the biggest difference between On-Demand and Spot Instances.

Introduction

Amazon recently announced that it’s Elastic Kubernetes Service (EKS) now supports running Kubernetes pods on AWS Fargate. Amazon is putting a lot of resources into their serverless products and this new feature allows us to run Kubernetes in a “serverless” mode. I am pretty excited about being able to run Kubernetes pods in Fargate as this reduces the overhead to run Kubernetes applications as there is no need to run any EC2 worker nodes anymore.

This article will cover the following topics:

• Differences between running a traditional EKS cluster and running EKS on AWS Fargate
• How to bring up a new EKS cluster which can run pods on Fargate

Amazon made a bunch of new product announcements in the recently concluded re:Invent conference. I was looking forward to seeing what new products and features they were going to introduce in their serverless services and there are quite a few announcements. In this article, I have tried to summarize some of the announcements which I thought were the most interesting.

Provisioned Concurrency for Lambda

Amazon introduced Provision Concurrency for Lambda functions. This feature allows you to keep your Lambda functions initialized and ready to respond to any requests in milliseconds. This new feature should help with the Cold Start problem in Lambda.

The pricing for Provisioned Concurrency is slightly different. More information on pricing is available here.

Have you ever been confused by which AWS tool to use for your startup? AWS currently has more than 100 services listed on its platform across 20 categories. As Amazon continues adding new services to its platform at an incredible rate, it is becoming impossible, especially for startups and engineers to know which AWS service they should be using.

In this article I’ll breakdown the evolution of a startup into three stages and talk about which AWS tools in particular are useful in each of these stages. This article should give you a good mental model for choosing between various AWS tools. The different stages we are going to talking about are:

• Early days / Founding Stage
• Product/Market Fit
• Achieving Scale