(Note: after seeing stern feedback on Hacker News and reddit, Iâm changing my test approach to hire 10+ contractors in popular cities around the globe to compare CDN download times. And if you disagree with my analysis in any way, please cordially do so đ. Iâm on twitter at @SpeedTestDemon)
Iâve learned Cloudinary basically does image and video transformations on top of the Fastly CDN. As such, I feel this is basically a review of the Fastly CDN platform. Fastly costs minimum $50/month, so itâs not free CDN Iâm interested in analyzing. However, Cloudinary has a free tier going up to 25 GB of storage or bandwidth, so hereâs âAnalysis #2 of a Free CDNâ.
Cloudinary was SUPER impressive. Cloudinary was way better than AWS CloudFront. They had a secret sauce: dramatic image transformation. Cloudinary somehow compressed a 703 KB image down to 37 KB.
Before I present the simplified summary, here are some links relevant to this article.
As a very simplified summary, based only on hot cache results
- Cloudinary was faster than CloudFront by 3x (0.133 vs 0.397 seconds, or difference of 264 milliseconds)
- Even after controlling for image size, Cloudinary was still faster by 13% (0.11 vs 0.126)
Cloudinary had better performance for several reasons:
- How did Cloudinary compress that image 95% smaller? (~700 KB image to ~36 KB).
- Much more common DNS path (res.cloudinary.com vs d20zaq59cm4c4j.cloudfront.net), which allows for significantly faster DNS lookup. In one case, 0.11 vs 0.008 seconds.
- Uses Fastly CDN underneath. A lot of big companies (including Amazon) have adopted Fastly, probably for a good reason.
- CloudFront typically degrades a file from hot cache to warm cache very rapidly, whereas Cloudinary (i.e. Fastly) doesnât do this untilâŚweeks? (itâs a long time whatever it is).
What follows is a much more detailed technical analysis.
I had to do 2 speed tests. The 1st speed test was not apples-to-apples because Cloudinary had compressed the image by nearly 95%. The 2nd speed test, I controlled for file size, and this time Cloudinary came out ahead very slightly by 6% (although the main credit should be due to Fastly).
In the first speed test, Cloudinary really crushed CloudFront.
At first sight, I had to double-check for errors. Cloudinary was nearly 3x faster â on the hot cache! â and it didnât make sense. Turns out the test was correct. Cloudinary simply compressed the image 95% lower (700 KB to 36 KB).
I had to do a second speed test to make it more apples-to-apples between CloudFront and Cloudinary (masquerading as Fastly). CloudFront did a lot better in the hot cache. Still sucked at the other two metrics.
There were several reasons for Cloudinary performing so well:
OK Cloudinaryâs free tier is pretty impressive. The only limitation I can think of is you canât host your whole site on it like you can with Cloudfront.
Now one momentâŚyou do âget what you pay forââŚwhile Cloudinary has great performance, it is limited in many other ways. You canât update a file on Cloudinary easily, unless you wholesale rename the file (very annoying). The amazing performance applies mainly to images. JS and CSS wonât have the same amazing image compression, although you will see Cloudinary perform on par with CloudFront.
See the conclusion for my thoughts on the Cloudinary vs CloudFront decision.
It should be noted that not every free CDN performs as well as Cloudinary. I found a couple free CDNs that performed really poorly.
Why is it interesting to test Cloudinary?
Honestly, it was just because they had a free tier CDN and my goal here was just to test every free CDN.
I didnât know anything about Cloudinary before writing this article.
But wow, the results have blown my mind. Good job Cloudinary guys! These guys have a good product.
After doing some more research on Cloudinary, I found out these guys have won some interesting accolades
- Inc. Magazine called Cloudinary the âgold standardâ of image management on the web (according to Wikipedia at least).
- Actually has rather large usage (unusual that Iâve never heard of them even though Iâve been in the industry for many years). Usage stats are 1 million developers and 7500 companies (Wikipedia).
OK, so itâs got a reputation for great image management. I can see that actually. They did insanely well in this test.
Why is it interesting to test CloudFront as the gold standard in CDN speed tests?
Because Amazon is Amazon and theyâre the biggest cloud provider out there (they are #1 at 32% of the market). But Cloudinaryâs impressive speed test results here have me really questioning why Iâm using CloudFront. Well Cloudinaryâs premium tier is very expensive at $100/month, so itâs not an obvious switch. CloudFront is only at $2/month for the equivalent network out.
Brief Summary Of The Metrics Being Measured
We are interested in the hot, cold, and warm cache speeds because CDNs are essentially a cache layer. If youâre interested in reading more, see the analysis here: metrics measured in Jetpack vs CloudFront and my CDN testing methodology.
Finding #1: Cloudinaryâs DNS lookup is significantly faster than CloudFrontâs DNS lookup.
This was similar to what I found in Jetpack CDN vs Cloudfront. Cloudinary is able to achieve a significantly faster lookup time because itâs DNS path is more likely to be cached by the intermediate routers. Cloudinary DNS lookup path is âres.cloudinary.comâ which seems to be the shared DNS path between most Cloudinary users, so that DNS lookup path is receiving a lot of traffic.
While CloudFrontâs DNS lookup path is a very customized âd20zaq59cm4c4j.cloudfront.netâ.
Whatâs very interesting is Jetpack CDNâs DNS lookup times were always nearly as fast as the hot cacheâs DNS lookup time (even the cold cache 0.00223 seconds was so close to hot cache 0.0016 seconds), while Cloudinaryâs DNS lookup times took a small hit (hot cache 0.00175 seconds to warm cache 0.0236 seconds). I think this can be attributed to Jetpack CDNâs immense popularity which means its DNS lookup path âi0.wp.comâ is going to be cached much more frequently than Cloudinary.
Finding #2: Cloudinary had a mind blowingly good image compression algorithm
Iâm not exactly sure Cloudinary got it so good. The same 703 KB image was somehow compressed down to 36 KB. Thatâs just really good and explains how Cloudinary was able to get much faster Data Transfer times than CloudFront.
Other interesting findings
I already talked about these in previous essays ( Jetpack CDN vs CloudFront) so I didnât want to write more paragraphs for these.
- Generally, the TCP handshake and SSL handshake times are the same regardless of file size or file type. Most CDNs and servers are going to be equally fast on these metrics, unless they are using a slower CPU to do SSL computations.
- CloudFrontâs warm cache (a request 30 minute after the most recent) is about 2x to 4x slower than hot cache. I found this a lot when testing Jetpack CDN. CloudFront likes to degrade a file from RAM to disk (NVMe or SSD) if itâs not seen active requests recently.
The first time I ran this comparison test, I was blown away by how good Cloudinary was. I thought Jetpack CDN was a good Image CDN. I was wrong. Cloudinary was way better.
- Cold cache: OK so Cloudinary was 5x faster than CloudFrontâŚ.because Cloudinary doesnât really have a cold cache? Because there is no expiration on the files (unless you get Cloudinaryâs premium tier).
- Hot cache: I guess this is the metric that really matters, since itâs the most common case that a CDN is going to contain the file. And boy did Cloudinary impress. By compressing the file from ~700 KB down to 36 KB, Cloudinary beat CloudFront by 3x.
- Warm cache: Cloudinary did even better, beating CloudFront by 4x. Cloudinary hot cache didnât degrade even after 30 minute wait time, while CloudFront once again degraded the file so it performed 33% worse (same thing I found in Jetpack CDN vs CloudFront).
Cold cache measurement and analysis (speed test #1)
CloudFront really lagged horribly here at 2.355 vs 0.395 seconds for Cloudinary.
Sharp readers will notice that CloudFront showed a much worse time (2.355 seconds) in this speed test comparison than in the Jetpack CDN vs CloudFront analysis (1.199 seconds). Itâs because I just ran this CloudFront download at a different time than when I wrote the Jetpack CDN analysis.
Wait a second. CloudFront took 2.35 seconds while Cloudinary was only 0.39 seconds? Holy moly. Where did all that time go?
The two big time consumers were: 1) Request Latency, and 2) Data Transfer.
Regarding Request Latency: CloudFront took 1.022 seconds while Cloudinary took 0.258 seconds. Whatâs impressive is Cloudinary must have done two things during the cold cache lookup: it must have had the file still cached on a server despite my asking for the file only several days after getting that CDN setup. Cloudinary certainly did not fetch the file again, or it wouldnât have gotten such a fast time. Thatâs really really good.
Cloudinaryâs cold cache is not as fast as its hot cache. Most of the time went in Request Latency. For cold cache, itâs likely they had the file stored somewhere on disk, while for hot cache, itâs probably in RAM. Actually, since Cloudinary is built on top of Fastly, I would place a strong bet that Fastly has the file in RAM or else why are people buying Fastly?
Regarding Data Transfer: yeah this was really incredible. Cloudinary only needed 0.0069 seconds while CloudFront took 1.098 seconds. Thereâs only one main factor for this and itâs Cloudinaryâs insane image compression algorithm which means Cloudinary only has to send 36 KB while CloudFront has to send 703 KB. Itâs just not even close. See more analysis in Finding #2.
Regarding SSL handshake: honestly not sure why CloudFront took much longer at 0.091 seconds vs 0.056 seconds but the absolute difference is only 0.035 seconds, so itâs a small contributor to the overall delta of nearly 2 seconds.
Regarding TCP handshake: this is just the same for any connection. So thereâs not a big absolute time delta.
Regarding DNS lookup: not the biggest factor for the large overall time delta but itâs still worth mentioning because on a relative basis, Cloudinary outperformed by over 100% (0.0494 seconds vs 0.124 seconds). See Finding #1 analysis for why Cloudinaryâs DNS lookup is generally much faster than CloudFront.
Hot cache measurement and analysis (speed test #1)
This hot cache measurement was the one that really blew my mind.
Cloudinary smokes Cloudfront, nearly 3x faster (0.133 vs 0.397 seconds).
Wow! How did Cloudinary do that?
(Reminder: these times are an average of 10 curl calls).
Looks like all the times are pretty similar except for Data Transfer. In fact, Data Transfer accounts for 0.234 seconds of the overall time difference of 0.263 seconds. Literally about 90%.
Itâs really shocking that CloudFront took so much longer to transfer the file, even in a hot cache state.
The reason is Cloudinary compressed the original 700 KB file down to 36 KB. Thatâs nearly a 20x reduction in file size. So no wonder Cloudinary ended up smoking the pants off CloudFront.
Notes:
Warm cache measurement and analysis (speed test #1)
Ouch. CloudFront nearly 4x slower than Cloudinary: 0.465 vs 0.126 seconds.
What happened?? (reminder: warm cache is download taken exactly 30 minutes after hot cache)
First off, a monstrous Data Transfer difference: Cloudfront 0.198 seconds vs Cloudinary 0.009 seconds. Absolute difference of 0.19 seconds, contributing to the overall time difference of 0.34 seconds.
Second, a monstrous DNS lookup difference: Cloudfront 0.115 seconds vs Cloudinary 0.0236 seconds. Absolute difference of 0.09 seconds, contributing to the overall time difference of 0.34 seconds. If you thought DNS lookup speeds donât matter â well think again. See âFinding #1â for why CloudFrontâs DNS lookup is so much slower than Cloudinary.
Data Transfer and DNS lookup combine for 0.28 seconds out of the 0.34 seconds difference.
Other notes:
- As said in Finding #2, CloudFrontâs Data Transfer was so much slower because it was transferring ~700 KB of data while Cloudinary was transferring ~36 KB. Cloudinary compressed the image really well.
- Dang, Cloudinaryâs warm cache faster than the hot cache time (0.126 vs 0.133 seconds). Thatâs impressive.
- I donât know why CloudFrontâs SSL handshake is always slower than Fastlyâs SSL handshake. Iâm still thinking about it. (Cloudinary is built on Fastlyâs CDN).
- Yeah, CloudFrontâs âRequest Latencyâ was 2x slower. Iâve noticed CloudFront tends to degrade the file from RAM to NVMe or SSD.
Speed Test #1 conclusion and analysis
Cloudinary beat CloudFront in hot cache, cold cache, and warm cache, due mainly to the image automatically compressed to 36 KB and CloudFront still trying to transfer a 720 KB PNG.
Impressively, the hot cache was a staggering 3x faster: 0.133 vs 0.397 seconds.
What if we even the odds and make CloudFront and Cloudinary transfer the same file? Surely CloudFront should beat Cloudinary now? Itâs not that simple. See the next section for Speed Test #2.
I re-ran the speed test to make sure CloudFront and Cloudinary were downloading the exact same file. To do this, I just uploaded the Cloudinary 36 KB file to CloudFront CDN.
Notes:
- Cold cache: I guess Cloudinary keeps the file in hot cache for an absurdly long time (this is a good thing for the end-consumer. It does not cost extra money for this feature). it takes a long time for Cloudinaryâs cache to turn coldâŚso Iâm skipping this test.
- Hot cache: Cloudinary still slightly faster (12%). Absolute difference is only 0.016 seconds. Not a big deal I guess.
- Warm cache: after waiting 30 minutes, Cloudinary didnât degrade the cache at all, while CloudFront faced a 2x slow down.
Cold cache measurement and analysis (speed test #2)
SoâŚ.I found out Cloudinary doesnât really have a cold cache? Even if I try running the test a few days after my last fetchâŚit is still pretty much the exact same as the hot cache resultâŚ.
For example, I ran the Cloudinary test again, and I noticed the cold cache was only 0.02 seconds slower than the hot cacheâŚnearly all of which was in the DNS lookup.
The coldest Iâve ever seen Cloudinary was when I waited a couple weeks to test Cloudinary. If you test only a few days after, it is pretty much the same as hot cache.
This is actually a good thing for the consumer. It means Cloudinary doesnât degrade the file down to cold cache until a couple weeks. It is kind of incredible.
Iâm not going to bother writing much of an analysis for this.
Cloudinaryâs cold cache is pretty much as fast as the hot cache, with the only difference of the DNS lookup (was not cached by intermediate routers).
CloudFrontâs cold cache is just the same as last time except for a significantly faster Data Transfer phase since we are making CloudFront download 36 KB.
Hot cache measurement and analysis (speed test #2)
Much closer where we can say that perhaps CloudFront is as fast as Cloudinary. Cloudinary holds a slight 12% faster time. Cloudinary still 12% faster than CloudFront. Although the absolute difference is only 0.016 seconds so perhaps not a big deal.
DNS lookup, TCP handshake, SSL handshake, Data Transfer were all pretty close (these are the same operations for any file size or type).
The main difference of 0.010 seconds comes from Request Latency, where CloudFront takes 50% long (0.030 vs 0.20 seconds). Honestly not sure but perhaps Cloudinary has a more efficient file cache read than CloudFront. Cloudinary is built on Fastly, which has significant market share among enterprise companies.
So Cloudinary is faster, but itâs not really worth stressing about it at this point. The difference is small enough.
Warm cache measurement and analysis (speed test #2)
Same thing that happened in Jetpack CDN vs AWS CloudFront. CloudFront warm cache degraded significantly (relative to hot cache) â 2x slower (0.25 seconds vs 0.126 seconds). Cloudinary warm cache pretty much the same as hot cache.
Regarding Request Latency*: the big rock was Request Latency: 0.076 vs 0.0168 seconds. Cloudinaryâs Request Latency was the same in both hot and warm cache, so Cloudinary must be keeping the file in RAM. CloudFront definitely downgraded the file from RAM down to disk. This is very consistent behavior that Iâve seen in the* Jetpack CDN vs Cloudfront comparison as well.
(reminder: warm cache is download taken exactly 30 minutes after hot cache)
CloudFrontâs speed performance is once again significantly slower as found when I was doing Jetpack CDN vs CloudFront. Cloudfrontâs slowdown came in 3 areas: DNS lookup, SSL handshake, and Request Latency.
Regarding DNS lookup: Cloudfront is slower here because the DNS name d20zaq59cm4c4j.cloudfront.net is entirely unique to my account so it is even more unlikely to be cached in routers than the Cloudinary DNS name res.cloudinary.com (which appears to be shared with multiple users).
Interestingly, Cloudinaryâs DNS lookup was significantly slower when going from the hot cache to the warm cache (0.035 seconds vs 0.0016 seconds).
Another interesting point is Jetpack CDNâs DNS lookup was only 0.0092 seconds. This happened because the Jetpack CDN DNS name i0.wp.com is heavily used across millions of wordpress installations and Cloudinary has significantly smaller install base, so the Jetpackâs DNS name is way more likely than Cloudinary to be cached in routers.
Regarding SSL handshake: it is really unusual that Cloudfrontâs SSL handshake takes nearly 2x as long as Cloudinary (0.084 vs 0.045 seconds), because the TCP handshake occurred with equal speed on both CDNs, which indicates the problem is not the internet speed.
There are 2 possibilities: 1) for some reason the certificate authority is taking a long time to respond, and 2) the CloudFront is serving the traffic from a cheap CPU server so the SSL calculations are taking much longer?
I guess possibility #2 is more likely. Still weird.
My thoughts on this are extremely mixed. I would say it highly depends on your circumstances whether you should use Cloudinary.
Reasons to use Cloudinary
Cloudinary is a highly performant CDN built on top of Fastly CDN, very widely used among tech companies such as Reddit and even Amazon itself (which is weird because Amazon has its own CDN called CloudFrontâŚ).
Cloudinary has some very impressive image compression capabilities. The 700 KB test image was compressed down to 36 KB. Itâs possible to do the same thing in CloudFrontâŚbut itâs more involved. Youâd need a WordPress plugin that was as good at compressing images. Actually, Jetpack CDN was already compressing, but only down to 216 KB. Again, really impressive stuff from Cloudinary. I guess nobody is compressing the image that wellâŚ(âŚyet. I will probably build this WordPress plugin).
Cloudinary keeps files in a hot cache state for weeks (much longer than CloudFront which degrades the file in less than 30 minutes).
Cloudinary also boasts video CDN delivery. I have not tested using videos so I canât make an evaluation here. AlthoughâŚitâs likely the video CDN performs as well as the image CDN, although we still need to test it.
Even in the scenario that CloudFront is delivery the same compressed file as Cloudinary, Iâm seeing Cloudinary (i.e. Fastly) is performing better on cold, hot, and warm cache.
- 12% faster on hot cache
- 2x faster on warm cache
- Cloudinaryâs cold cache is a couple weeks, not 30 minutes.
SoâŚCloudinary has been extremely impressive indeed.
Reasons not to use Cloudinary
However, the free tier is extremely limited to only 25 GB/month of data transfer, while the premium tier is very expensive (starting at $99/month). Cloudinary is by far the most expensive CDN Iâve tested so far. To put the $99/month in perspective: CloudFront only charges $0.08/GB, and Cloudinary has a free tier of 25 GB and premium tier of 225 GB. Run the numbers: 1) for the free tier, CloudFront would cost only 25 GB * $0.08/GB = $2.00. 2) for the premium tier, itâs 225 GB * 0.08/GB = $18. AndâŚCloudinaryâs charging $99/month? Wow.
On the plus side, if youâve got a low or medium traffic website and youâre sure youâre transferring less than 25 GB a monthâŚby all means go for Cloudinary. Since youâre staying in the free tier.
Another limitation is you canât host a static HTML page on Cloudinary like you can with CloudFront and CNAME Aliases. You can host static assets like CSS, JS, images, and videosâŚbut not HTML. For example, this entire blog is hosted off CloudFront, even the initial HTML request. I canât do that with CloudinaryâŚjust offload the images/videos/CSS/JS, thatâs it.
As an aside: thereâs a good reason why Cloudinary is so expensiveâŚit is built on top of Fastly CDN. Cloudinaryâs primary value-add are image and video transformations. And Fastly is already fairly expensive (minimum $50/month).
Final Analysis: to use or not use Cloudinary?
In the final analysis, what youâre paying Cloudinary for are the âon-the-flyâ image and video transformations. Cloudinary contains some bells and whistles that makes it a powerful image & video CDN. Is this worth $99/month to you? i.e. $1188/year.
Also remember Cloudinary is basically a reseller of the Fastly CDN. Cloudinaryâs primary value-add is in the image and video transformations. Are you the kind of person or business that needs Cloudinaryâs advanced features? It costs $99/monthâŚ
What kind of person or business are you?
- Hobby blog: since youâre probably a low-traffic site, you could easily stay within Cloudinaryâs free tier of 25 GB.
- High traffic blog: in my opinion, you can easily compress the images yourself to get it as good as Cloudinary. Then for fast delivery, either use Fastly directly or use CloudFront. Since using Cloudinary would simply be using Fastly indirectly.
- Your website earns $1000/month: well $99/month is 10% of your profits. Not really worth it.
Software engineer building complex app: Cloudinary has some complex image transformations like cartoonify, brightness, rotate. Thatâs neat. Using Cloudinary as a REST API could save you some time (and developer time is always the most expensive).
Python CDN Speed Test Script: I use this to automate the collection of metrics about CDNs.
APPENDIX: Wanna reproduce our test results? Hereâs how to setup Cloudinary
- Login or Sign up in Cloudinary.
- Install and Activate Cloudinary Plugin in WordPress