Build a serverless API like a senior developer, Part 2
Part 2: App configuration and setting up local testing
Link to Part 1 here.
There are already many, many tutorials out there that talk about getting baby’s first API endpoint up and running with AWS Lambda and AWS API Gateway. I want to go beyond the scope of other tutorials and lay everything out in a way that is easy to understand and follow.
We are going to create a simple API with a GET method that simply retrieves information about video courses from a database table. However, in the course of this series, we are going to go over ideas such as infrastructure as code, automated testing, CI/CD, runtime dependency injection, error alerting, local AWS testing, and more.
In the second part of the tutorial series here, I will go over finishing API configuration and setting up local testing. Our API will be written in C# and this part of the tutorial series will take us from having a basic code setup to having everything we need to test and execute this lambda function locally.
Remember, there is much more to software than just your code :)
Preface
This tutorial is written in great detail, perhaps more detail than a more experienced developer will need. In the areas where I reference basic fundamental ideas, I have either placed links to other areas where those fundamental ideas are explained or I have placed links to skip ahead to the more advanced ideas.
If you are at a section for which you already understand the material, there will be an opportunity for you to skip ahead.
Prerequisites
You will need the following applications installed from Part 1.
In addition, you will need the following applications ready for Part 2.
In Part 1, we created the basic structure of our code, along with a unit test project, so that we can easily extend and expand our code to fulfill all of our needs.
A brief explanation of DynamoDB
Feel free to skip ahead to the next section if you don’t need an explanation of DynamoDB.
DynamoDB is a cloud-managed NoSQL database in AWS. It’s particularly good for data where the structure is unknown, data where relationships between different types aren’t strictly required, and data where the primary focus is not on columns of data but key-value pairs.
DynamoDB identifies unique objects by what’s known as a primary key. This is similar to how a SQL table defines a unique object, including how you can make a composite primary key out of two or more attributes on an object. In DynamoDB, a composite primary key is composed of a partition key and sort key. For the purpose of this tutorial, only a partition key is necessary, since the data is not currently structured in such a way where it makes sense to have a composite primary key.
When you create a new DynamoDB table, you don’t specify a table size, but instead you specify the reads and writes per second that you need, as well as when to scale up or down that capability.
We plan to use this instead of a SQL database because with a NoSQL database, we have more control and flexibility over the content we put into this application. Since this API we are creating would merely be the first building block for a larger scale application, it wouldn’t be a good idea to create a database with unchanging tables and a specific capacity that will need more effort to scale and modify.
Setting up the local database
Let’s set up a local DynamoDB table to which we can connect from our code. To do this, first start up Docker Desktop. After a little bit, you should be able to open up Docker Desktop and see that it’s running.
At this point, you can now open a Terminal (or Command Prompt/Powershell) window. Navigate to the directory where you downloaded the DynamoDB JAR executable.
To start the local DynamoDB database, run the following command.
docker run -p 8000:8000 amazon/dynamodb-local -jar DynamoDBLocal.jar -inMemory -sharedDbI’m not going to go over Docker or this command in much detail, since all we really need it for is to get the local DynamoDB instance running. It suffices to say that 8000:8000 represents the port to which you want to bind the instance, -inMemory means that all data will be lost once you exit the instance (which is fine for us), and amazon/dynamodb-local represents the name of the Docker container we need to run a local instance of DynamoDB.
If you get an error saying something about not being able to find a logging implementation, you can safely ignore it, as your DynamoDB instance will continue to run.
Now, open a separate terminal window and type the following command.
aws dynamodb list-tables --endpoint-url http://localhost:8000You should see an empty array, like so.
This tells you that your local AWS can indeed identify the local DynamoDB instance, connect to it, and recognize that no tables have been created.
Next, enter the following command in that same window.
aws dynamodb create-table --table-name Course --attribute-definitions AttributeName=title,AttributeType=S --key-schema AttributeName=title,KeyType=HASH --billing-mode PAY_PER_REQUEST --endpoint-url http://localhost:8000Before we go ahead and execute it, let’s unpack all of that by the parameters --table-name , --attribute-definitions , --key-schema , --billing-mode , and --endpoint-url . For further detail, you can check out the AWS CLI documentation for the dynamodb create-tablecommand.
--table-name: The name of the table.
--attribute-definitions: The names and types of each of the known attributes in the object. Each of these values is separated by spaces and defined with an AttributeName and an AtributeType. AttributeName is just the name of the attribute, and AttributeType corresponds to a type code as defined by Amazon. Type codes are shown below.
We only have one attribute defined here because we only need to define those attributes for the primary key at the outset. Once we have that, we can then add whatever other attributes we need, those by which we don’t intend to query.
--key-schema: This describes the structure of the primary key. If you only have one attribute composing the key, you will need exactly one AttributeName and KeyHash value, with the KeyHash equal to HASH .
--billing-mode: This is required to determine how to provision (create) the table with respect to how many read/writes per second the table handles, and therefore how much it costs to run it. In our case, it will be free to run this table since it’s not actually running in our AWS account, but we will set PAY_PER_REQUEST to allow the table to scale up or down as needed.
--endpoint-url: This is simply the URL to call out to the database, which for us is http://localhost:8000 since we set the DynamoDB instance to run on port 8000 and we are running it locally.
Now that we understand what we’re doing, we can go ahead and execute the command.
Okay awesome, the table is active now.
Now we need to add some records. To do this, we’re actually going to create a YAML file and load records from that into our test table. I’m setting up my YAML file as such. Feel free to change the instructor name, or really any attribute, as you desire.
RequestItems:
Course:
- PutRequest:
Item:
title:
S: 'Intro to Serverless APIs'
instructor:
S: 'Marwan Nakhaleh'
description:
S: 'Learn how to build a rest API running on Lambda served through API Gateway.'
category:
S: 'API'
- PutRequest:
Item:
title:
S: 'Intro to Angular'
instructor:
S: 'Marwan Nakhaleh'
description:
S: 'Learn the key concepts of Javascript front-end framework development by example with Angular 12.'
category:
S: 'frontend'
ReturnConsumedCapacity: INDEXES
ReturnItemCollectionMetrics: NONEI’ve named it batch_put.yaml. We are sending a batch request because we intend to add more than one record. To send the request to put those items into our table, we will run the following command.
aws dynamodb batch-write-item --cli-input-yaml file://batch_put.yaml --endpoint-url http://localhost:8000
If your CapacityUnits says 1.0, that’s fine too.
Now you can query your table for the values you entered.
Create a new file called expression_criteria.json and add the following to it.
{
":v1": {"S": "Intro to Serverless APIs"}
}Save the new file. Type the following command into the terminal.
aws dynamodb query --table-name Course --key-condition-expression "title = :v1" --expression-attribute-values file://expression_criteria.json --return-consumed-capacity TOTAL --endpoint-url http://localhost:8000You’ll get a response that looks something like this.
Okay, excellent, we have submitted our values to our table and we can retrieve them. Now we can connect to the table from our code.
Add database configuration in the code
Open up our SampleAPI solution in Visual Studio and open appsettings.Development.json . Edit the file contents to look like this.
{
"AWS": {
"Region": "us-east-1",
"DynamoDB": {
"ServiceURL": "http://localhost:8000",
"TableName": "Course"
}
}
}We’re going to allow our code to set variables based on configuration files for each environment we run. Create a new folder in your main project within the solution called Configurations.
Your folder structure should now look like this.
Inside of that Configurations folder, create a new class ConfigurationKeys.cs . Inside of that new class, write the following.
public static string DynamoDBServiceURLKey = "AWS:DynamoDB:ServiceURL";
public static string DynamoDBTableNameKey = "AWS:DynamoDB:TableName";Your final class should now look like this. You can delete the unused dependencies at the top.
namespace SampleAPI.Configurations
{
public class ConfigurationKeys
{
public static string DynamoDBServiceURLKey = "AWS:DynamoDB:ServiceURL";
public static string DynamoDBTableNameKey = "AWS:DynamoDB:TableName";
}
}Notice how these values correspond to the JSON structure in the appsettings.Development.json file? We’ll be connecting those real soon.
Let’s create another new class inside the Configurations folder called DbConfiguration.cs . In there we will add the following.
public string ServiceURL { get; set; }
public string TableName { get; set; }When we extract the values for the DynamoDB URL and the table name from the appsettings.<environment>.json files, they will be stored in these variables. To do this, we will create a new method in our startup file that will get the current database configuration for the environment and inject a new DbConfiguration object into our application on runtime.
Let’s head over to our Startup.cs file. Create a new private static method in the class called GetDbConfiguration that returns an object of type DbConfiguration .
The method will look like this.
private static DbConfiguration GetDbConfiguration()
{
DbConfiguration dbConf = new DbConfiguration()
{
ServiceURL = Configuration.GetValue<string>(ConfigurationKeys.DynamoDBServiceURLKey),
TableName = Configuration.GetValue<string>(ConfigurationKeys.DynamoDBTableNameKey)
};
return dbConf;
}You’ll get a few red squigglies when you write up this method. Hover over one of them for DbConfiguration , select “Show potential fixes” and select using SampleAPI.Configurations; .
Now we will need to add this to our ConfigureServices method. Find this line in the method.
services.AddAWSService<IAmazonDynamoDB>();Add this underneath it.
services.addSingleton(GetDbConfiguration());Querying the database
Now, all we need to do is modify our Repository implementation of BaseRepository to query an actual database.
Add the following at the beginning of the class declaration in Repository.cs .
private readonly string _dbServiceUrl;
private readonly string _dbTableName;Now, locate your constructor declaration. Change it to this.
public Repository(DbConfiguration dbConf, ILoggerFactory logger) : base(logger)The red squigglies will appear under DbConfiguration , and once again we shall resolve that by adding using SampleAPI.Configurations at the top.
Now, inside of the constructor, add the following lines.
_dbServiceUrl = dbConf.ServiceURL;
_dbTableName = dbConf.TableName;Your Repository.cs class should now look like this.
using Microsoft.Extensions.Logging;
using SampleAPI.Configurations;
using SampleAPI.Domain;
using System.Collections.Generic;namespace SampleAPI.Data.Repositories
{
public sealed class Repository : BaseRepository
{
private readonly string _dbServiceUrl;
private readonly string _dbTableName; public Repository(DbConfiguration dbConf, ILoggerFactory logger) : base(logger)
{
_dbServiceUrl = dbConf.ServiceURL;
_dbTableName = dbConf.TableName;
_logger = logger.CreateLogger("Repository");
}
public override List<Course> GetCourses()
{
_logger.LogInformation("returning course information");
return _courses;
}
}
}
Now we want to add a new private method in Repository.cs where we set up our database client. This will return an object type AmazonDynamoDBClient .
private AmazonDynamoDBClient GetDbClient()
{
if (!String.IsNullOrEmpty(_dbServiceUrl))
{
AmazonDynamoDBClient client;
AmazonDynamoDBConfig ddbConfig = new AmazonDynamoDBConfig()
{
ServiceURL = _dbServiceUrl
};
client = new AmazonDynamoDBClient(ddbConfig);
return client;
}
else
{
throw new Exception("No database URL has been provided!");
}
}What all is going on here? First, we check if we have a value provided in the service URL. If we do not, we throw an exception with a message. If we do, we create a new client variable, then we create a client config object of type AmazonDynamoDBConfig .
Among the properties that can be configured for an object of that type is a ServiceUrl of type string. From there, we finally create the client object and return it, passing in the config object as a parameter.
Now, we need to update GetCourses() to use this new method. At the beginning of the method, add this code.
AmazonDynamoDBClient client;
try
{
client = GetDbClient();
}
catch(Exception e)
{
_logger.LogCritical(e.Message);
return _courses;
}Here, we’re creating a client with the new GetDbClient() method, and then if it fails for whatever reason (i.e. if we have to throw an exception), we will log a critical message.
A brief explanation of logging levels
Feel free to skip ahead to the next section if you don’t need an explanation of logging levels.
Logging is one of the easiest things you can do to understand what happens in your code as it is being executed. Different code paths will have different meanings and different levels of urgency, therefore it is necessary to be able to separate these messages by level. The levels will allow you very easily to see what is going on in the code, how bad the event is, and decide what you may need to do (if anything) to remediate an issue.
In most logging library implementations, there are five log levels; they are debug, info, warning, error, and critical.
Debug: This is generally only for debugging your app, i.e. when you need to see every little detail of what your app is doing. These are very verbose logs.
Info: This is generally used for app events, such as a successful authentication or a successful API call.
Warning: This can be used when some non-critical functionality does not work as expected, specifically something that the application can lose without the loss affecting core requirements.
Error: This is used when there is something wrong in the app, such as an inability to retrieve information or display something correctly to a user.
Critical: This is used when something fails that essentially breaks the whole app.
With a logging library, you can also enable what’s called a default level in your app. What that means is that anything below the severity of the default level will not show up in your logs. For example, if your default logging level is set to “warning”, that means you will see warning, error, and critical log messages in your logs but you won’t see info or debug.
Querying the database
Now we’re going to create a new method that simply returns all objects from our Course table. In this case, we’re going to scan the database to retrieve all objects, instead of querying the table to retrieve specific objects. Using scan operations on DynamoDB tables gets very slow and costly if not used carefully, though it is okay on small datasets.
Create a new private method called ScanAllItems() that returns a ScanResponse object and takes in a AmazonDynamoDBClient object.
private ScanResponse ScanAllItems(AmazonDynamoDBClient client)
{
}Fill it in with the following code.
ScanRequest request = new ScanRequest
{
TableName = _dbTableName,
};try
{
var response = client.ScanAsync(request);
ScanResponse result = response.Result;
return result;
}
catch (Exception e)
{
_logger.LogError("Unable to scan items: " + e.Message);
return null;
}
Here, we are simply creating a new ScanRequest and passing in the table name we provided on Repository instantiation. Since it will just return all records in the table, no additional query parameters are needed.
After that, we will run a ScanAsync call against the client with that ScanRequest information. In C#, the code will wait on this action to execute and it will return a Task of type ScanResponse . The result of the Task can then be extracted and finally we can return that ScanResponse.
If for whatever reason something fails in that process, we log an error and return a null value. We will check later in the code that the ScanResponse object is not null.
Adding retrieved course to repository course list
Next, it’s time to add the retrieved courses to the _courses list. Create a new private method returning void called AddCoursesToCourseList , passing in a parameter of type ScanResponse.
private void AddAllCoursesToCourseList(ScanResponse response)
{
}Fill it with the following code.
if(response != null)
{
foreach (Dictionary<string, AttributeValue> item in response.Items)
{
Course course = new Course(item["title"].S, item["category"].S, item["instructor"].S, item["description"].S, new List<Video>());
_courses.Add(course);
}
}
else
{
throw new Exception("No retrieved courses to add.");
}A successful ScanResponse will have an Items attribute from which we can retrieve information about each record’s attributes. Each attribute on the record can be accessed by the attribute name as well as its primitive data type value. So where you see, for example, item["title"].S , we are getting the title attribute on the record, and then the string stored for the attribute’s value. We use .S specifically for the String value on the record, as based on the DynamoDB data types table above.
Returning all the courses
Finally, we can simply override the method from the base abstract class to get courses. Since we’ve already written methods that handle each part of the course retrieval process, we can put them all together in the GetCourses() method.
public override List<Course> GetCourses()
{
try
{
AmazonDynamoDBClient client = GetDbClient();
ScanResponse result = ScanAllItems(client);
AddAllCoursesToCourseList(result);
}
catch(Exception e)
{
_logger.LogError(e.Message);
} return _courses;
}
Simple! All we needed to do was use our existing methods to get the database client, scan all items, and add those courses to the course list. This code structure makes development changes a breeze.
Final API testing
Once again, you can hit the IIS Express play button at the top center, wait for the browser page to pop up to say “Welcome to running ASP.NET Core on AWS Lambda”, copy that URL into Postman, and make a request against /api/v1/course .
Flexibility
In real-world projects, there are often at least a few major changes that happen between when development begins and when the product is shipped to the user, and even after. Therefore, it is crucial to have flexibility in your code and in your architecture.
If we wanted to change our whole database client, we would not have to rewrite all of the code. We would only need to modify the appsettings files, the configuration keys, the retrieval of the info from appsettings in Startup.cs , and the GetDbClient() method.
If we wanted to change the way in which we retrieved all items, we would only need to change the ScanAllItems() method, and perhaps the parameter type in the AddAllCoursesToCourseList() method.
Improvements (maybe coming in a future part)
In fact, we may want to change the way we retrieve all items in the future to prevent attempting to scan a table with tons of courses. What can be done, for example, is to create a sort key (described above) in our Course DynamoDB table, create a second table called CourseCategories or something like that, and then scan the CourseCategories table, and query the Course table by each of those categories. That would only require the CourseCategories table addition, the update to the Course table, changing which table we scan all items from from the Course table to the CourseCategories table, and then creating a new method that queries the Course table by each of those categories and then adds those items to the _courses list.
In addition, our unit tests leave something to be desired. Each of the methods we used inside of the GetCourses() method in our Repository class were private, but we could consider making them public to test each of those individually. We could also create unit tests for the actual controller method or the query method that massages the _courses from the Repository into something to be returned by the API.
