Excluding costs, there are 2 main things that makes AWS great: large number of services offered and integration between those services. And for the last category, a good example is the integration between AWS S3 and Glacier. With only few clicks in the S3 console, you can create a lifecycle rule that automatically moves objects from S3 to Glacier. And this rule can be applied on the whole bucker or per key prefix or per tags. Sounds great, no? Yes, it is.
But there is a very important aspect you should consider before setting such a rule. Do you remember that long time ago we published an article where we demonstrated that for small objects, S3 is more expensive than DynamoDB. If you missed it, you can find it here. Basically, most of the cost is given by the price of PUT calls and not by the price of storage.
And now it comes our question: do you know how expensive are PUT requests in Glacier? If you don’t know, then we can say it’s 11x more expensive than S3 PUT cost.
Now you can anticipate which is the conclusion: Glacier is definitely not for small objects. And by small objects we understand objects that have at least 1Mb.
If your objects are smaller than 1Mb, then for sure there is a solution: fetch content from S3, merge it into larger files and then upload them in Glacier. But the goal of this post was not to come up with a solution for this problem, but rather to make you aware you understand that an inoffensive lifecycle management rule could cost a lot of money. The calculation is very simple: it costs 55$ to upload 1M objects in Glacier, which maybe is not a lot, but if you have 10M, the amount gets bigger. Also, if 1M objects has 10Gb (10Kb per object), then it’s weird to pay 0.4$ per month per storage, but 55$ to upload them.
On the same time, we consider that it would be very nice to have a rough estimate in the console when you add a lifecycle rule. At least to advice you to double check the Glacier costs before, because as you can see the initial cost could be bigger and this service is promoted as a very cheap one.
If you enjoyed this article, share it with your friends and colleagues!