Creating a Scalable Website Analysis Tool using Lambda and API Gateway and S3

Welcome all my new follow_ers, it’s a pleasure to have you aboard and I hope you enjoy this latest attempt at me becoming a millionaire!_

The Week’s Million Dollar Idea

So last weeks’ million dollar idea of creating a highly scalable website monitor seems to have somewhat failed.

Creating A Highly Scalable Website Monitoring Service in half an hour using AWS Lambda_Lambda functions are a truly awesome bit of technology, if you haven’t read up about my theory that serverless and more…_hackernoon.com

I don’t think it was “disruptive” enough for these billionaire angel investors but whatever… This time round we are going to crack the nut and I am going to buy myself a big yacht.

This week, we are going to pivot on our initial idea, we are going to create a service that, get this, analyses websites. Sounds cool right? The best part about this new service is going to meet the following requirements:

• it’s going to be done on the cheap. Our frontend will be hosted in S3. Unless we hit the truly big time our lambda’s will not cost a dime.
• It’s going to be highly scalable! This will be ready for the prime time once we IPO.
• It’s going to be stable and extendible — I can’t have the platform falling over if new requirements come in.

If it meets these requirements it’s sure to be a hit!

So, in order to achieve these hefty requirements we are going to be leveraging AWS Lambda, the API Gateway and S3. We are going to be creating a series of API endpoints that will be hit by our frontend written in Vue.JS. These endpoints will then go on to hit a number of different Lambda functions written in any language we wish! I’m going to be leveraging Go, Python and Node because, why not? it’s all a bit of fun!

The Frontend — In Vue.JS on S3

So first things first, we need a flashy looking frontend. If it doesn’t look cool, the investors won’t be enticed to part ways with their hard earned money.

This will be a very simplistic frontend that will take in a webpage URL through a simple input and it will then hit the 3 distinct endpoints that we are going to be defining in our API Gateway. These 3 endpoints are going to calculate things like, the total page load speed, the first time to byte, and the number of links and 404s present on that page.

The finished product for this looks a little like this:

Inspired By: https://dribbble.com/shots/3203637-Dashboard-Report/attachments/683409

Full Source code for the frontend as well as all of the following Lambda Functions can be found here: https://github.com/elliotforbes/website-analyzer

Time to First Byte — In Go

So, the first Lambda function will utilize the very recently announced Go runtime! If you’ve never written a Go Lambda before then I recommend you check out my very brief introductory Youtube video on how to do so.

(Remember to like and subscribe to support the channel! :D)

In our Lambda function we are going to be taking in a URL through our Request object and then trying to accurately calculate the time to first byte.

Our finished version of our Go based endpoint will eventually look like this:

This essentially parses the URL that is passed in through the request and then attempts to establish a TCP connection with that URL on port 80. We then send a GET HTTP 1.0 request down this TCP connection and we calculate the length of time it takes for 1 byte to travel back. This calculated time value is our time-to-first-byte.

Once it’s calculated this time to first byte it returns this back to the caller of our Lambda function.

Page Speed — In Python

The second lambda function we’ll be defining is the one that calculates our page load speed. This is going to be rather crude and will simply calculate the time taken to perform a urllib.request.urlopen() call and return that to the caller.

You’ll notice that there is no fancy error handling to check if the url passed in is valid, this could be done in the future!

404 Checker — In Node

Finally, we come to our last Lambda function. Our 404 checker endpoint will simply take in the same URL as the other two functions and it will utilize the broken-link-checker node module in order to adequately test our page for links and broken links.

The broken-link-checker module made this endpoint ridiculously simple to implement. I would have saved a lot of time just assuming such a node module already existed as opposed to trying to come up with my own implementation…

Key Takeaway — If you are writing anything in node, chances are there is already a package out there that does what you want.

Our API — Using API Gateway

So now that we have our 3 distinct Lambda functions we need to define a way to access these through a HTTP Endpoint. The best way to do this is through the API Gateway!

Essentially this allows you to pair up paths and HTTP methods with their respective lambda functions. It’s incredibly simple and once you are happy with your endpoints, you simply click the ‘actions’ button and click deploy.

Deployments are done to particular stages, so if you wish to deploy to a test environment you have to deploy to a test stage, for production you deploy to a prod stage and so on.

After a little bit of tweaking and enabling things like the dreaded CORS, I had a working API that I could point my frontend at.

Once I was happy with my frontend locally, I pushed it up to a public S3 bucket and made it accessible by the public. The URL for this newly deployed site can be found here: https://s3-eu-west-1.amazonaws.com/website-analyzer/index.html

Note: I’m positive that with a little tinkering you could break this in some way, making it production-worthy will be done after I’ve gone public!

Conclusion

And there you have it folks! We now have a fully functioning website analysis tool that we can demonstrate to investors and hopefully attract investments!

My and my team after our first seed round.

This website is not only incredibly resilient, it’s also incredibly cheap. The frontend is hosted on a service that features a ridiculously high SLA in terms of up time and the backend is hosted on Lambda functions that are not only massively scalable, but also incredibly resilient.

There isn’t much that could bring down this site that easily. This will not only insure that our customers don’t see any major impact, it will also keep management and investors happy as they have a product that can be relied upon.

Hopefully you found this article super useful as well as somewhat entertaining! I also hope that it’s demonstrated to you just how powerful the likes of Lambda and the API Gateway can be when used in conjunction with one another!

If you enjoyed this then please feel free to let me know on twitter: Elliot Forbes. I’m also on LinkedIn should you wish to connect: https://linkedin.com/in/elliotforbes. I’m also currently working on a new book titled “An Introduction to Cloud Development” — if you wish to support me and learn more about the cloud then check it out here:

An Introduction to Cloud Development_This book provides a gentle introduction to the world of cloud development. We will be taking an in-depth look at…_leanpub.com