A Bucket Full Of Objects — Amazon S3

Amazon is not just about books anymore.  Since 2001, Amazon Web Services (AWS) has found ways of deriving income from the excess capacity of the network infrastructure used to run their own e-commerce business.  One of these products, the Simple Storage Service (S3) allows users to store their files, known as “objects” online.  As of March 2010, it was estimated that S3 contained over 102 billion objects.

Objects are referenced by their “keys,” which consist of an optional “pseudo folder” (directory) path name, followed by the name of the object. The keys “His-Stuff/test.txt” and “Her-Stuff/test.txt” refer to 2 separate instances of the  “test.txt” object.  Because the “folder” name part of the keys are unique, so are the object instances.  The term “pseudo folder” is used because S3 does not really store objects in folders; the same way that Windows, OS-X or Linux does.  The entire object key is considered (by S3) to be the equivalent of a file name.

Object keys are stored in “buckets.”  Bucket names are unique across the entire S3 system.  Each bucket is owned by an AWS account.  AWS accounts can claim ownership of many buckets, but buckets can only be owned by one AWS account at a time.

Buckets are assigned to geographical “regions,” in which objects contained in the bucket are stored.  Currently, there are five regions; US East (N. Virginia), US West (N. California), EU West (Ireland), Asia Pacific Southeast (Singapore), and Asia Pacific Northeast (Tokyo).  Objects within a bucket are stored across multiple data centers within each region.  A sixth umbrella region called US Standard refers to either US East or West.  With the recent announcement of the Asia Pacific (Tokyo) region, it is expected that an umbrella “Asia Pacific Standard” region will also be created.  Unless explicitly specified, network mapping is used to automagically determine which region newly created buckets are assigned.

There are three reasons why you would explicitly assign a bucket to a particular region; to reduce latency, for regulatory reasons, and to avoid regional volatility.  If you are in an EU country, then by storing your objects in the EU region you reduce the amount of time needed to retrieve your data.  Similarly, for reasons of compliance with EU privacy laws, objects containing customer information may have to be kept in buckets assigned to the EU region.  Finally, if you have objects containing sensitive information and you are concerned about the ability of data centers within your default region to protect them, you might create buckets that are assigned to other more stable or data friendly regions.

Objects can be public or private (default).  Public objects are accessible by anyone who knows the bucket and key names of the object.  Private objects can only be accessed by the bucket’s owner, by explicit permission, or by expiring URL request.  It is the power of the expiring URL request that e-commerce products like WP eStore use, to enhance the download experience of digital products.  Once a download has been authorized, expiring URL requests are generated and users are redirected to S3 to download their digital product.

Expiring URL requests grant access to specific private objects within a bucket, for a small period of time.  Bookmarking of the URL request, and knowledge of the bucket and object names, are ineffective once the URL request has expired.

By redirecting users to the S3 network for downloads, you dramatically reduce the resource burden associated with large downloads from your own server; to a more faster and more reliable one.  This is analogues to the concept of having physical products kept and shipped to customers from a warehouse location that is separate from the retail location.

Redirection of downloads to S3 is not the same as simply using S3 to store files and then copying them through your server, to the user.  Without redirection, your server is not only burdened with having to act as a middleman during the download process, but with the additional bandwidth required to first copy the data to your own server and then resending it to the user.  That is why it is important to look for native Amazon S3 integration when considering e-commerce products.

The cost of using S3 is quite low and there is no monthly minimum.  Monthly cost is dependent upon the amount of data stored, the number of times an object is uploaded/downloaded, and the amount of bandwidth used.  Amazon offers a Free Usage Tier that allows you to try out all AWS products, including S3, for one year.

About the Author: This post was written by TheAssurer who writes for Tips and Tricks HQ and helps out with plugin development.

Comments (14 responses)

  1. Web says:

    Only 14 cents for their first month! That is really inexpensive. (Taking notes) Man. Going to your site feels like shopping for new stuffs…!

  2. Daisey says:

    After reading this article I tried S3 and I’m very happy with it. Thanks a lot for sharing

  3. Esmalte says:

    My tool Bucket Explorer it helps you a lot to learn Amazon S3 more efficiently!!

  4. Trace002 says:

    Its really good for newbies like me to learn and i am using Bucket Explorer is really easy and good tool to work with. Its so basic and user friendly. Its documentation also help me to learn about Amazon S3.
    Simplly good to be an Amazon S3 user

  5. Robert says:

    amazon web services and the cloud is a nice feature and we are currently working on moving our VoIP/SIP support application to the Cloud.

  6. Amazon S3 is a great service facilitating people world wide…try My tool Bucket Explorer it helps you a lot to learn Amazon S3 more efficiently

  7. Great article for those who are new to Amazon S3! Don’t miss S3 Browser Freeware – easy to use windows client for Amazon S3!

  8. I’m genuinely enjoying my every click on your site. Amazon S3 appears to be the answer to slow downloads…:) Wow! Only 14 cents for their first month! That is really inexpensive. (Taking notes) Man. Going to your site feels like shopping for new stuffs. Thanks a heap!!!

  9. Grace004 says:

    Nice article about Amazon S3. I tried Bucket Explorer command line as well as ui, it supports all the features provided by Amazon S3. Supports Windows, Mac OS X & Linux platform to work with it.

  10. Peter says:

    Amazon S3 does look like a very cheap but reliable and robust solution. Thanks for sharing this article.

  11. I always enjoy learning what other people think about Amazon Web Services and how they use them. Check out my very own tool CloudBerry Explorer that helps manage S3 on Windows . It is freeware. http://s3.cloudberrylab.com/

  12. Gary Smith says:

    Amazing timing. A client of mine just uploaded a huge amount of files to S3 and was billed about 14 cents for their first month! I’ve started looking into moving my downloadables (about 1 gig) onto the service. Thanks for the nice summary.


  13. Robert says:

    My customer used to have problem downloading large files from my site. My server couldn’t keep up with that. Then I switched to Amazon S3 with eStore. Now there is no download issue anymore and it’s really cheap.

    By the way, Thanks for the article.

  14. Jamie says:

    Nice explanation… I have been using the WP eStore plugin for a while. I think I will go with the Amazon S3 integration option as it sounds more robust.

Speak Your Mind