Alarm system from application logs by CloudWatch Metrics [Step1]
In the past, I did not used to care much about loggings. When I released a new application for production, I checked the server and making sure that new application was running fine. After production release was completed, new app was working fine for a while. However, every now and then, my client called me saying there seemed to be some issue. Fixes were usually simple, but I wished that I had been able to proactively find issues and fix them before my client noticed an issue.
In this article, I would like to share my way of handling this issue by centralizing logs from .Net Core console application to AWS CloudWatch.
The setup and configuration in CloudWatch to create alerts from logs is covered in this article.
Why logging though?
If our goal is only to send an alert from CloudWatch, yes, we can just send metrics from our application to CloudWatch and send an alert. However, I believe that logging is an important piece of application not just because we can make sure that your applications are working correctly, but also it helps you troubleshoot any issues when something goes wrong.
One can argue that we can just debug the application. That’s true, but when we do not have good data to debug in development environment, we have to start creating the data to debug.
Also in a microservice architecture, an application is divided into small pieces, which makes it difficult to debug the issue since you may not know which application you should look into.
With the popularity of a microservice architecture, I think loggings is more important than ever.
Centralize logs
In an environment where you have multiple instances of same applications, it is not ideal to write to local hard drive in each servers since we cannot know which server has logs we are looking for.
We could simply point the log location from local drive to network drive so that we can have logs in one place, but it could affect the application performance because there is network cost and multiple workers write to the same single file. I had an experience with apps running slow, and once I switched the log location from network drive to local drive, apps started running fine…
One way to solve this issue is to write to local drive and then install an agent which will push to centralized location. For example, in ELK stack, LogStash can push local logs to ElasticSearch. If ElasticSearch is already set up, then that’s great. If not, maybe you can look for other solutions since the learning curve for ElasticSearch can be high.
In my opition, AWS CloudWatch is an easy way to accomplish the centralized loggings. We just need to have AWS account and create a Log Group, which is like a folder where you want to send your logs. In .Net Core, there is a library called AWS.Logger.AspNetCore which we can install from NuGet. With that, we just need a few lines in startup code to send logs to CloudWatch.
Get Started to log to CloudWatch
Let’s take a look at the code. Here is the simple console app which just writes random numbers to console. In order to write to CloudWatch, we need to first set up AWS user profile.
Get Access key ID & Secret access key
First of all, sign into AWS and go to IAM
Create a user to use from application. Need to check “Programmatic access” since this user won’t be used to sign into AWS.
Select “CloudWatchLogsFullAccess” policy.
Once user is created, you can view “Access key ID” and “Secret access key”.
Create profile file
Since “Access key ID” and “Secret access key” need to be secured, we do not want to put them even in source control. One way to securely use them is to use a profile file. Create a text file like this and save it locally. For example, I put it in C:\AWS Credentials\kagawa.txt. In .Net core app, we just need to tell where this file is when we run the app.
Install AWS.Logger.AspNetCore in NuGet
AWS.Logger.AspNetCore allows an app to add CloudWatch for log destination. Once it is installed, we can add “ConfigureLogging” to register AWS.Logger.AspNetCore. Regarding AWS config, it looks for it under Loggings section in appsettings.json by default.
Finally, we need to tell the application what AWS access key & secret to use, which can be configured in appsettings.json.
The red square needs to be under the “Logging”. Also, please note that “ProfilesLocation” at line 13 is where AWS access key & secret are saved. “Profile” at line 12 is the name of the profile in a file.
That’s it! Logs will be written to the LogGroup specified in appsettings.
Structured logs
Instead of plain text logs, we can structure logs so that it can be queried by fields. It is very important when it comes to create alerts. I believe Serilog is the most popular choice for structured loggings, but we can achieve the same result by serializing object to Json.
When logs are written in structured way, we can query specific logs. For example, if we want to find logs whose number is greater than 60, we can do this using filter pattern. This will be useful when we create metrics and send alerts.
Next step
Since now we have logs in CloudWatch, we can use query to determine the threshold for when we want to send alert. The next article covers how to set up Alert in CloudWatch, but I believe centralized logs in CloudWatch can provide lots of values to developers to save time troubleshoots and make sure apps are running correctly. I hope you enjoyed my article. You can find the full code in my repo here.
