Posted by: akolk | February 7, 2009

Using Amazon Cloudfront

Last year, a customer was running into trouble with static product images on their website. The images were store in an OCFS2 Filesystem and every so often OCFS2 would hang and the site became slowly unresponsive. We had talked a number of times about using a Content Delivery Network (CDN) to also improve the download streams to the client. The number of concurrent downloads from a domain is limited and different per browser. So increasing the number of domains for your site will help to improve the concurrent download. While we were discussing this (and after another problem with OCFS2) Amazon AWS sent out an email about the availability of Cloudfront. Here you can store static content and it will be cached in different servers around the world that are the closest to the browsers that request content from them. So we decide to implement this.  

Step 1 was to signup for the Amazon AWS service. Then we had to create an S3 bucket and upload the static content. There you run into of one performance issues with the Cloud. Uploading large amounts of data to the S3 (European) bucket is limited by your internet UPLOAD speed. Most people use ADSL connection with high DOWNLOAD speeds but with lower UPLOAD speeds. So uploading 30 GB of data will take some time (depending on your upload speed). An S3 bucket is basically a raw datastore. You have to tell what objects in that datastore are directories or files. Uploading the data was done with the JetS3 program. This has a commandline interface (CLI) and is written in Java so it can run on different platforms. After 3 days(a weekend) all the data was online and available from the Cloudfront. We implemented a monitor service with ZABBIX to see the peformance and availability of the Cloudfront service. It hasn’t been down but we noticed a couple of performance degradations, but the service has been available 100%. 

So was it worth all the effort that was put into it? The reason we did it was that OCFS2 seem to have a performance and stability problem. Well 2 months later we discovered the real reason for the problem. There was a firewall between the Read-Only nodes an the one Read-Write node of the OCFS2 cluster. This firewall was doing some housekeeping and lost control of that . So certain connections and messages between the OCFS2 got lost. That caused hangs and performance degradations. So the reason for switching has been fixed, but we haven’t switched back to the old implementation.

This Amazon Cloudfront service turned out to be a nice solution to serve static content. 

More to follow later.

About these ads

Responses

  1. I always enjoy learning how other people employ Amazon S3 and CloudFront. I am wondering if you can check out my very own tool CloudBerry Explorer that helps to manage S3 and CloudFront. It is a freeware. http://cloudberrylab.com/Would be interested what the author of the blog thinks!

    • Thanks Andy,

      I tried to download the program and I noticed a problem. This seems to be Windows only version. I tried to install on my Mac and it of course complained. Can I download a Mac and/or Linux version? In the meantime I will fire up my windows VM and tried to use it from there.

      Thanks,
      Anjo.

  2. Unfortunately there is no Mac or Linux version yet. Mac version is something we are looking at now. Thanks for trying the product out!

    Regards,
    Andy

    • I just installed Cloudberry in my Windows VM. Installed registered/installed my Amazon S3 account and found that it generated the following error while opening some buckets:

      An item with the same key has already been added.

      Could it be that the length of the key is too short? I can open it without any problems in the S3 firefox extension.

      Anjo.

  3. sorry, for not getting back to you – we are checking the possible cause of the error with the dev team. Thanks for trying the product out!

  4. Could you tell me the name of your bucket that causing the error? Is there any files that might have some adverse impact in your mind?

    please send me an email at andy at cloudberrylab dot com
    Thanks for your help!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: