Ramana Lokanathan

Random Ramblings of a Techie

By

AMBER Alert system needs critical Improvements

amber_active

Those of you in California would have woken up to that crazy sounding AMBER Alert (abbreviation for “America’s Missing: Broadcast Emergency Response”) on Monday Aug 5th at 10:52PM. The alarm was worse than a fire alarm in a big industrial building.

That AMBER Alert (officially known nation-wide as IPAWS — Integrated Public Alert and Warning System, or even more generally as WEA — Wireless Emergency Alert system) was to notify the millions of users in CA about the abduction of a teen Hannah Anderson in San Diego, CA. More information about that case can be found here.

I was at home trying to wind down for the day when I heard that alert. It certainly was “alarming” to say the least… I thought that the alert was for an early warning for a Tsunami, an Earth quake, or a Nuclear attack, until I actually read that cryptic message.

Without having any expectations about the seriousness of the AMBER Alerts, and without any knowledge that these Amber Alerts have been activated by default in my smartphone, my imagination went wild for a little while. I even thought of getting outside the house assuming that the alert was for an earthquake. I then went online and figured out what was going on. As expected, I noticed mixed reactions to this alert. A series of questions came to my mind:

daphne-missing

  • Who determines the criticality of the issue, when to send alerts, and to whom?
  • Why did the alert not go for Daphne Viola Webb, a 1-year old who went missing in Oakland, CA on July 10th, or for Jensi Curtis who went missing in Las Vegas, NV on Aug 3rd?

    jensi-missing

  • Can we scale as a community by sending alerts to millions of users in 5 different states on a regular basis for these sort of crimes?

    Anyways, this Alert got me thinking about the pros and cons of the service, and also about how it can be improved. For starters, I am all in favor of the Amber Alert technology. It has helped recover many under age kids from perpetrators. My thoughts are more about making it a better service for people.

    1. Alert technology:

      The AMBER Alert technology has to improve A LOT! If some local authority decides that “we don’t know where the suspect is heading, so lets alert all the users in bottom 48 states”, then this service will fail miserably. There are many ways to target the alerts based on a specific geographic location, for ex:

      • just alert the coastal cities in pacific north west for a potential tsunami
      • just alert the users in the flight path dynamically every few mins, if there is an air threat (similar to 9-11)

        and so on… The current technology allows these capabilities — we just have to be smart about using them.

    2. Alert levels:

      Coming from the technology world, these sort of alerts are very common. Systems and servers go down frequently, and automatic/manual alerts are issued to corresponding engineering teams on a regular basis. But, there is a method to the madness. Alerts are heavily categorized based on the severity level, and response times for these alerts are agreed upon in advance. For ex:

      • A Severity zero (Sev 0) alert means hell broke loose, and all-hands are needed on the table in less than 30mins (hacker attack, site went down, etc).
      • In contrast, a Severity four (Sev 4) alert is mostly a warning that needs to be attended in 1 to 3 business days (hard disk might get full in few days, etc)

      Just like the Tech world, and just like the infamous “Homeland Security National Terrorism Alert levels”, there should be some categorization of these alerts based on severity, the # of users to target based on the severity, and the expected response from the general public. For ex:

      • A Sev-0 Alert might be similar to a “Presidential Alert” that can be issued any time of the day, and could require immediate attention.
      • A Sev-1 Alert could be issued for big natural disasters such as Tsunami, Earth quake, Nuclear Attack, etc, and might require the general public to act fast.
      • A Sev-2 Alert might be similar to AMBER Alert which can only be issued between 8AM and 10PM every day, and does not expect general public to drop everything and attend to this alert. Also, this alert will not be accompanied with any loud, shrieking alarm.

      And so on…

      Having various alert levels, and setting expectations on both the gov authorities, and the general public will avoid a LOT of these confusions.

    3. Education about these Alerts

      The last but more important way to improve this service is by educating the general public. No state-wide alert has been issued over cell phones in CA before Aug 5th, and most people (including me) did not know that our cellphones are enabled with these alerts by default (opt-out nature of these alerts).

      It is the responsibility of both the gov agencies, and the mobile/tv/satellite/radio network providers to educate the users about:

      • the various wireless alert services
      • the features and limitations of these alerts
      • expectations about the response for these alerts from general public

    Without the above mentioned changes, these alerts are in danger of losing their viability. The CA state government is already facing a humongous task of getting all those hundreds of thousands of users who opted-out that day after receiving that crazy shrieking AMBER alert.

    In summary, I believe that, a focused effort to address the above-mentioned areas will definitely make these these wireless alert systems from being just a “better-than-nothing” service to “a very important, and timely service that potentially saves the lives of the citizens”.

    I welcome your comments.

     

    Related links:

  • By

    Amazon AWS Summit 2013, San Francisco, CA – Summary

    aws-summit-2013-sf

    Last week, I attended the Amazon Web Services (AWS) Summit in San Francisco, CA. The AWS Summit is a one day event designed to broadcast the latest and the greatest in the world of Amazon AWS. More than AWS itself, I learnt a lot about the rapidly growing ecosystem of products and services built by both startups, and established companies around AWS. The event was well-organized and informative. I also got an opportunity in this summit to meet some of my old colleagues and make new connections. Given below is a quick summary of the AWS SF Summit for those who couldn’t make it.

    The summit started with a keynote address given by Andy Jassy – SVP of AWS. Some interesting anecdotes from his keynote:

    • AWS is the fastest growing division in Amazon, both in revenue and in size
    • The infrastructure is now setup in 9 regions, 25 availability zones (AZ) and 39 edge locations around the world.
    • Amazon Elastic MapReduce (EMR) currently holds 5.5Million clusters across data centers
    • AWS customers now span 190 countries, 2000+ academic institutions, and 500+ government organizations
    • AWS has now given 31 price reductions since 2006
    • Just this year alone, AWS has launched more than 70 features, and counting
    • They now have a Direct Connect feature where enterprises can directly connect from their own data centers to Amazon Virtual Private Cloud (VPC)
    • Amazon Glacier (offline backup & archival storage space) costs as low as $.01/GB/Month.
    • The newly launched Amazon RedShit, a data warehouse solution on the cloud, is the fastest growing service this year. It can handle 100GB to PB or more, and costs just less $1000/TB/Yr

    Andy continued to talk about the value proposition of adopting cloud computing model, and how customers in various industries, small and large, are saving time and money by moving to AWS.

    He then brought in four of his customers to the stage to talk about their AWS experiences. One was a startup while other was aimed at established companies.

    The first customer was MailBoxApp.com – a Mobile/Tablet based Mail collaboration tool. Sean, the Engineering Lead of MailBox explained how they used AWS to quickly identify, prototype, iterate and execute various product features to meet the exponential growth projection (100MM emails/day).

    In the second segment, Simone Brunozzi, a Tech Evangelist from AWS spoke about how enterprises are moving host of their services to AWS to save Capex and Opex. He touched upon the following areas:

    • Authentication to AWS components using a company’s own Active Directory credentials
    • AWS Storage premises for backup & recovery
    • Move Disk images or snapshots to offline storage in multiple regions to meet the compliance demands

    In the third segment, Greg Johnson, Head of Analytics for Nokia spoke about how they used Amazon RedShift Data Warehouse solution from AWS, and achieved at least 50% cost benefit and more than 2 times the time benefit.

    In the last segment, Ryan Tudhope, Founder of Atomic Cloud, a visual effects and animation company based in San Francisco gave a nice overview of how intensive and how uneven computing needs are in the animation industry, and why moving to Amazon AWS made total sense for them.

    In case you missed the keynote speech, you can watch it here.

    After the keynote, the crowd split into multiple breakout sessions that covered a breadth and depth of AWS topics for the rest of the day. These breakout sessions were OK – you can only learn so much from a 45min session.

    I did attend the one-day technical bootcamp session a day earlier, and found it very useful. I will recommend the bootcamp sessions to anyone who wants to get a jumpstart on a specific service/component of AWS.

    During Lunch, I spent some time at the Partner & Solutions Expo. I spoke to various companies and learnt about the products/solutions they offer. I was pleasantly surprised by this growing ecosystem around AWS. Here are some of the companies that I visited (there were many) in no particular order:

    (Please go to their website to learn more about these companies. I won’t be able to do justice by writing in two sentences about all the good work they do)

    Here are my general observations from this summit:

    • Amazon is working hard to deal with amazing growth of its AWS business – its a great problem to have at hand
    • AWS is now focusing on bringing in more Government customers to its cloud. Not only will it bring a healthy revenue stream, but also provide legitimacy in terms of security, stability, etc.
    • Along the same lines, Amazon is trying hard to crack the nut of bringing big Enterprise customers to its cloud. I will say that they are getting successful in doing so, since the cost-benefit speaks for itself.
    • In terms of the length and breadth of the functionalities, Amazon AWS is THE leader in the Cloud computing business today, by a large margin. Competitors need to work hard to catch up with AWS.
    • I was impressed to see the products/partners/services ecosystem that is mushrooming around AWS. The larger this ecosystem, the healthier AWS gets. Many of them are small startups that meet a specific need for AWS customers (logging, billing, database security, etc), but some are also big enterprises (Citrix, NetApp, etc).

    Thats it for now folks. I hope this summary is useful for those who missed the AWS Summit 2013 in San Francisco.

    By

    Cloud Tech IV – Cloud Computing Conference Summary

    600_226655692

    photo credit: a Meetup member

    Last Saturday (Apr 20th), I attended the Cloud Tech IV, a conference meant to learn, share and discuss the latest and greatest in the Cloud Computing Technologies. The conference was organized by the Silicon Valley Cloud Computing Group, with support from some big players in this field including but not limited to Amazon AWS, RackSpace, VMWare, Scalr, HP, etc. (most notably, Microsoft, Google, and other big cloud service providers were missing in this conference).

    As with other Meetup groups, there were a mix of people in the attendees list including developers, ops guys, employers, marketers, venture capitalists, and job seekers, and the underlying motives are many. This event was partially successful in achieving those objectives. As for me, It was useful. Given below, I have summarized my thoughts on few of the talks that interested me.

    Amazon AWS Discussion:

    The AWS talk was given by Apolak Borthakur, the head of AWS Bay Area development center. He touched upon few web scale challenges unique to Amazon AWS, such as the need for developing custom software based routing software, the need to customize virtualization softwares, and the reason why the standard Database scaling practices didn’t work for them. I was a bit disappointed by the fact that the speaker couldn’t provide more details due to confidentiality issues.

    Some interesting snippets from the AWS talk:

    • AWS has real, serious web scale issues to solve — to the point where they require custom OS, softwares, VMs, Network, DNS, and even physical hardware such as Routers!
    • The S3 storage has grown to more than 2 Trillion+ objects lately (1.3T on Nov ’12, 1T on Jun ’12).
    • AWS infrastructure can handle 1.1 million reqs/sec at Peak!
    • The virtualization overhead is less than 5% (i.e., minimal, but not stellar)

    It became clear to me that in spite of all the world-class infrastructure and service excellence, the real brain behind AWS success is its softwares, especially the way they add intelligence to commodity hardware. Amazon knows it very well! The company is a big believer of building technologies from scratch. It makes sense — from my Yahoo! experience, i can say that the standard out-of-the box softwares only works to certain extent.

    The big reason behind AWS sponsoring this event (in my opinion) is to let Bay Area know that Amazon has opened a new silicon valley development center for AWS (Amazon already has presence in SF bay area through companies such as A9, Lab126, Amazon Music, etc.). This is good news for Bay Area engineers who have been interested to work on great engineering challenges faced by Amazon AWS.

    FYI, the speaker is not planning to post the slides from this talk online.

    How Box.com implemented MySQL Sharding

    This talk was presented by Tamar Bercovici, Engineer at Box.com, and it was about how to handle exponential Database growth using Sharding. The concept of Sharding is not new — it reminded me of my days at Y! Personals where we used sharding to split the Mailbox data into multiple “farms” based on the unique UserID hash. Back then we used MySQL 4.x and ran into various operational issues due to immature replication technology. Sharding is used successfully in many Y! Properties such as Flickr, Y! Finance, and Y! Sports. Facebook is known for taking MySQL performance to extreme levels using Sharding (among other techniques). The latest number I saw on the web is that MySQL infrastructure at Facebook can handle 60Million qps peak reads. (In 2010, it was close to 13MM qps peak)

    Whats interesting about this talk was how the presenter put together a complete picture from analysis, design, implementation, testing, phased deployment, and finally, the learnings from this process.

    Two things to point out here:

    • It is easy to come up with an exotic design to solve a pressing problem, but seeing it all the way through finish line is not easy – it requires lot of planning, perseverance and creative problem solving. In my opinion, this is how I differentiate a normal engineer vs. an EXCEPTIONAL engineer.
    • In almost all cases, you have to rollout a major new design WITHOUT ANY interruption to the existing site functionality/usage/revenue/etc. Sort of like changing the tires of a car that is running at 60miles/hr on a freeway. EXCEPTIONAL engineers acknowledge this difficulty and plan well ahead of time.

    I thought the incremental testing/implementation/rollout approach taken by the Box.com team was smart. It helped handle the above two scenarios very well. If you are interested in how they did it, check out the presentation (might not be the same, but close).

    AirBNB – Using Chronos to orchestrate ETL Jobs

    This talk was presented by Florian Leibert of AirBNB.com. It was about the challenges that AirBNB encountered in scheduling and running massive data processing jobs that require complex dependency management. They ended up building a new fault-tolerant scheduler called “Chronos” to replace plain old cron jobs, and also open sourced the project (thats very nice of them). Chronos is built on top of “Mesos”, which is a cluster manager software from Apache.org, which I am not familiar with.

    The presenter clearly summarized the pain points of dealing with cron jobs, i.e., lack of dependency management, retries, and synchronous flow. Chronos can work on multiple frameworks on a same cluster, and is also agnostic about the env it runs on (VMs or bare metal).

    I still didn’t understand why AirBNB did not use Apache’s Oozie workflow scheduler — its something that I need to spend time to understand (Y!, Cloudera and few other Hadoop distributions use Oozie as a standard mostly). Overall, I learned a good bit from this talk.

    Real-time analytics at Facebook

    Jun Fang, an Engineer from Facebook, along with Eddie Ma, Manager at Facebook, presented how they extended Facebook’s analytics infrastructure to deliver near real-time analytics. Facebook’s problem has been two-fold. One one hand, the data volume is growing exponentially, while on the other hand the Business/Analysts are demanding complex data requirements. Getting the metrics asap was key to business success, and Facebook solved it by providing near real-time analytics solution.

    The solution involved building a custom “Subscriber” (a pub-sub model) layer which streamed the Web logs real-time to a HBase merge-table. The data is then processed using a custom Hive-HBase handler, and then moved/appended to the Data Warehouse, and finally made available to Hive using a read queue. It might sound complex but easy to understand with a diagram. Unfortunately, I don’t have the diagram handy.

    Real-time analytics is fascinating. Lets consider a practical use case where a Product owner (PM) needs to A/B test a feature before going prime time, and has 25 unique variations/sequences to A/B test. Say, If there is a 1-day delay in gathering full metrics, then it will take a month minimum to make a feature decision! Not good. Fast forward to real-time analytics, and now, the PM can make the same decision in 1 day. That is the power of real-time analytics.

    I have to say though, I was under-whelmed with the presentation. The speaker didn’t communicate well on the stage. Also, the panel discussion was a waste of time, mainly because the conversation was not audible, and the speakers swallowed words during conversation.

    Percona XtraDB – the MySQL Cluster

    Vadim, the CTO/Co-Founder of Percona Software presented the Percona’s XtraDB Cluster product and its features. Since I have no working knowledge of XtraDB product, I was blown away with its features and capabilities. I didn’t know that such a feature rich product for MySQL exists in the market today.

    Basically XtraDB is the next level up after InnoDB. It solves some of the inherent shortcomings of InnoDB, and more. While InnoDB provides ACID compliance, and crash recovery capabilities, it sacrifices on performance and synchrony. If you implement a multi-master replicated database setup, there is no guaranty that the data in system A is fully in-sync with data in system B. With XtraDB, you get Virtual Synchrony.

    I have not tried any of Percona’s products yet, but I plan to do so in future. If you are interested in this topic, here is the slide deck from this talk (might not be the same but close).

    Unfortunately, I didn’t get an opportunity to listen to the remaining talks, so I can’t comment on them. But here are the links to the presentations if you are interested:

    I hope this summary helps those in the Cloud Computing community who couldn’t make it to the conference last weekend.

    By

    Running MapReduce jobs on Amazon Elastic MapReduce (EMR)

    aws-logo-1

    Following my initial exploration of Amazon AWS Infrastructure including EC2, S3, and RDS, I continued to spent more time on AWS, but this time looking in to the Elastic MapReduce (EMR) service. Using EMR, we can create and run MapReduce jobs at ease using 1 or even 1000 instances with a click of a button. EMR service takes care of all the overhead related to procuring servers, provisioning cluster, installing softwares, upgrading, and optimizing Hadoop. This means developers and companies can focus on “core development work” rather than dealing with maintenance and capex overheads.

    The EMR uses the EC2 computing platform for core computation, along with other services such as S3 for storage, and Dynamo DB. Lately, they have added support for HBase and MapR edition of Hadoop too. Also, all the good features associated with EC2 such as elasticity, multiple instance types, geographical redundancy, and reliability comes packaged with EMR service.

    To begin with, I followed the Getting Started with EMR documentation to the dot (primarily using the command line interface), and it was a smooth learning experience. The basic steps I followed are:

    • Setup an AWS account (if not already)
    • Create New Keys and get SSH access
    • Download, Install, and Configure EMR Ruby Client
    • Create a Job Flow (using PIG or Hive) either as a Streaming Flow or using Script
    • Run the job flow, and monitor the output
    • Navigate to Amazon S3 to review the output of the job flow
    • Terminate the Environment

    The diagram below illustrates the steps:

    amazon-emr-workflow

    Source: Amazon AWS

    I tried creating both simple PIG scripts and Hive Interactive Queries. Both options worked well. I also tried to use the EMR Web Console to launch jobs, and it was not bad. I have not tried writing web service APIs to manage the MapReduce jobs yet.

    Overall, i am impressed with this service. We have come a long way in the past 5 years with respect to running MapReduce jobs. In the earlier days of Hadoop at Yahoo!, it was not easy to setup your own Hadoop instance. We had to rely on the company’s central infrastructure, and there were lots of restrictions for access, usage, and data storage. Now, in a matter of 30mins or less, we can get setup a MapReduce instance and compute huge data set using services such as EMR.

    The picture is not all rosy. Amazon EMR is not for everyone. I found out that the data upload costs could get hefty, if we rely on S3. And, if we use cheap instance types, the performance will be poor. Also, EMR instances does not come with other related open source tools such as oozie, avro, flume, etc. in case you need them. Depending on the data security requirements, data volume, scale and performance requirements, your company might look into running your own custom Hadoop setup. As one user summarized concisely, its a compromise of “Control and customization” vs “Automated and Managed Service”

    Now that I’ve got a pretty neat MapReduce framework setup using EMR, EC2, Hadoop, Hive, S3, etc., I plan to explore these services further by creating few MapReduce jobs using this setup.

    Related Links:

    By

    Setting up an Amazon EC2 Instance (AWS) for Application Development

    aws-logo-2

    I have been playing around with Amazon Web Services (AWS) Cloud account recently, and I am loving it! Amazon provides a FREE Micro Instance for one year trial period for testing its AWS services. I think the free account is a smart approach because it takes a while for developers to setup a decent size installation, and try out the various services provided by AWS in order to decide if AWS is right for them. Amazon has been generous with the FREE account by allowing developers to use many of its AWS services without too many restrictions. In other words, developers will get first-hand knowledge of how things will work in the AWS cloud before making the critical decision of moving a company’s IT infrastructure to AWS cloud.

    I tried it in the context of installing a custom WordPress instance for my personal site. This involves setting up Apache, MySQL, PHP & WordPress softwares. The WordPress setup process requires the installation of Apache, MySQL, PHP, Perl softwares, and while it is basic, it gave me a good hands-on overview of how to get going with a straight forward LAMP based website development on Amazon EC2. Now that I am up and running on the EC2 Micro instance, I would like to share the steps I followed. I assume that you are novice in your area of expertise (backend/api/frontend/dba/big data/etc), and also that you know the infrastructure/platform/software needs/requirements related to your area of work.

    Step: 1 – Getting to know the Amazon AWS Cloud & its Building Blocks

    I further divide this step in to the following 2 sections:

    Amazon AWS Cloud and its value proposition

    AWS is the big umbrella name for the collection of remote cloud computing services offered by Amazon.com. At the core of it, the Amazon’s version of Cloud Computing service promises the following advantages:

    1. Replace Capex with Opex
    2. Lower Overall Costs
    3. Elastic capacity
    4. Agility / Speed / Innovation
    5. Shift focus to Differentiation
    6. Ability to go Global in Minutes

    Some key stats (as of 2012 second half, collected from various sources):

    • More than 450K physical servers
    • Can scale up to 300K+ request/sec
    • More than 900 billion objects in the S3 data store
    • 8+ Regions and 19+ CloudFront (CDN) locations world wide
    • As low as $.02/hr computing charges (~$15/month)
    • Decent availability (99.95%+)
    • Many high profile customers (Netflix, NASA, Nasdaq, US Gov, AirBNB, etc)

    To get up-to-speed on the Amazon AWS value proposition quickly, check out this 7+min video about Amazon Cloud Computing (AWS).

    AWS Building Blocks

    AWS comprises of about 25+ services each of which provides a certain service needed for the Cloud Computing. Some of the key services include:

    Global Infrastructure

    • Physical Infrastructure – Servers, Network, Storage, Security, etc. in multiple regions (called AZs)
    • Amazon Machine Image (AMI) – a pre-configured OS (that supports Linux/Windows/OpenSolaris/etc) used to create a virtual machine that serves as a base unit of deployment in the Amazon Cloud.

    Foundation Services

    Application Services

    • Elastic Map-Reduce – Managed Hadoop Framework
    • CloudFront – Global Content Delivery Network (CDN)
    • SNS – Push Notification Service
    • SQS – For large scale systems where you need to buffer jobs between various stages

    Deployment & Management

    • Elastic BeanStalk – Simplified Deployment environment for Applications developed on Amazon AWS. You create an application (say, a .var file), and upload it. Amazon will take care of deployment, scaling and management.
    • IAM – Authentication and Authorization Management
    • CloudWatch – to collect monitoring data centrally throughout the cloud

    There are more… The Applications that we write will sit on top of these layers and will make direct (or indirect use) of these services as needed. To see how these services work together to form the Amazon AWS cloud, check out the high-level Architecture diagram below:

    amazon-aws-cloud-diagram

    Credit: Amazon AWS.

    I found this 1 hour Video Tutorial very handy in order to get up-to-speed on the various building blocks of Amazon AWS.

    For more details on specific service, follow the links as mentioned above. I found the AWS documentation to be very comprehensive.

    Step: 2 – Sign up for Amazon AWS & Get an Instance running

    Now that we are familiar with the Amazon AWS, its time to try it out. As I mentioned earlier, the best way to try out is by starting with the Micro instance, which is FREE for up to 1 year. In order to get the free account, you need to create an AWS account and provide your credit card info. Amazon won’t charge you.

    1. Go to the Amazon AWS site and click on the “Get Started for Free” link to complete this step. It takes 1 to 24hrs to provision your account. You will get an email when your account is ready for use.
    2. Go to the AWS management dashboard, and click on the “EC2″ service. You will now see the EC2 service console.
    3. Now click the “Launch Instance” button. It is a wizard that will take you through the following steps:
    4. Choose an Instance type from available options. Amazon has 15+ standard instance types that spans many OSes such as Linux (Amazon’s own version, Ubuntu, RHEL, etc), Windows and Open Solaris. The cool thing is that you can choose a “community supported” AMI too, and there are 2000+ versions of them! More info can be found in the Amazon AMI service page.
    5. Choose the Instance Details such as number of instances (1 to 1000+), CPU (1 to 16+ cores), Memory (600MB to 60GB+), and availability zones (8+ regions). You can also decide if you want to go with “Spot Instances”. More on this topic later.
    6. Create (or) Choose key pair. This is an one-time process. You will need this key in order to SSH to the box later on. You will get a *.pem file which has the private key. Please store it in a safe location.
    7. Configure Firewall. This step allows us to open specific ports that are needed for communication with that instance. Go with the default configuration for now, and we will tweak it in Step:3

    Once you launch the instance, it takes few minutes for it to be available online. You can see the details of the instance in the dashboard, and trap the hostname (which will look something like this: ec2-..compute.amazonaws.com) from the description.

    I tried this setup few times on both Amazon’s own AMI Linux image (which I believe is RHEL like enterprise friendly version), and the Ubuntu Linux image, and then I extended the setup further with S3, and RDS services.

    Step: 3 – Login to the instance and Install the Softwares

    Now that we have an instance up and running, we are ready to login and install softwares. To ssh to the box, I copied my key file to the ~/.ssh/ folder and used the ssh -i option to login.

    
    ssh -i .ssh/mykey.pem ec2-user@ec2-54-243-121-257.us-west-2.compute.amazonaws.com
    

    If you run into issues, follow the link here for more details.

    Once you login, you will notice that you have root privileges to the box (sudo -i in Amazon Linux, and sudo -su in Ubuntu Linux). Awesome!. I took few mins to setup my bash profile, passphrase less login, etc., and now I am all set to install packages.

    I chose Amazon’s version of Linux, which uses “yum” for package management. Some basic commands to install apache, php, and mysql.

    
    yum install httpd php mysql mysql-server php-mysql
    service httpd restart
    service mysqld start
    

    To install WordPress, follow the steps below:

    
    wget http://wordpress.org/latest.tar.gz
    tar -zxvf latest.tar.gz
    mv wordpress /var/www/
    chown -R apache:root /var/www/wordpress
    

    Step: 4 – Configure and Go Live

    1. Make sure that the security group that this instance belongs to has access to the key ports we need, such as Port 22, 80, 3306, etc. My Security group looks like below:

      aws-security-group-1

    2. For more details on Security Group configurations, Click here.
    3. Lets check to see if Apache is running by looking up the hostname in a browser. If Apache is running, you will see the default Index page.
    4. Then check if PHP is working fine by creating a simple php file in the Web Home dir:
      
      root@ip-10-204-91-183 html # pwd
      /var/www/html
      root@ip-10-204-91-183 html # cat info.php
      <?php
        echo "PHP Version: " . phpversion();
        phpinfo();
      ?>
      root@ip-10-204-91-183 html #
      
    5. View the above page in your web browser and make sure if it works fine.
    6. Now, we need to setup MySQL server. I am more comfortable with the command line setup, so I didn’t install any GUI such as PHPMyAdmin. I started with running the “mysql_secure_installation” command to setup the MySQL root password.
    7. Test to see if the connection is working:
      
      mysql -u root -p
      
    8. Assuming you are connected to MySQL server, then create a database, and database user for WordPress:
      
      create database wp
      create user &#8216;wp&#8217;@'localhost&#8217; identified by &#8216;some-super-secret-password-&#8217;;
      grant all privileges on wp.* to &#8216;wp&#8217;@'localhost&#8217;;
      flush privileges;
      
    9. Now, login to your newly created WordPress DB to see if it works:
      
      mysql -u wp -p -A wp
      
    10. Now we are ready to setup WordPress. In order to do so, go to the wordpress folder under your web root, copy the wp-config-sample.php to wp-config.php, then edit the file, and add the Database user name and password details. When it is done, restart the Apache web server.
      
        cd /var/www/wordpress/
        cp wp-config-sample.php wp-config.php
        chown -R apache:root wordpress
        vi wordpress/wp-config.php
        service httpd restart
      
    11. Now, go to your WordPress site (http://your-amazon-instance-name/wordpress) from a Browser and complete the setup process. You can now view the WordPress Admin Dashboard by going to: http://your-amazon-instance-name/wordpress/wp-admin
    12. One last step. Amazon has a neat feature, called Elastic IP where you can associate a static IP to your instance, so that you can access the instance using a specific IP address. This IP address stays the same as long as that instance is running. Once you terminate the instance, this IP address will go away. To setup this up, click the “Elastic IPs” link on the left panel of the EC2 Dashboard, and follow the simple steps. I used this Elastic IP to setup a DNS A-record for one of the many domain names that I own (by using the console access provided by my web domain hosting provider), so now I am able to access this EC2 instance using a user-friendly domain name!

    And, thats it!

    I didn’t go into the details of the steps above because there are plenty of instructions available on the web. My favorite is this short video tutorial, which goes over the above process for Amazon Linux instance type.

    If you are into Ubuntu version of Linux, here is my another favorite Video tutorial which goes over the same details in the Ubuntu Linux instance.

    I briefly looked at Microsoft’s Windows Azure and Google App Engine, but I have not tried it hands-on in order to compare those services with Amazon AWS. I plan to do so in future. As of now I should say that I really like AWS, its variety of service offerings, and ease of use.

    Now that I am up and running on Amazon EC2, I am now ready to explore the other services in the Cloud. So far, I have looked at S3, Route 53, and Amazon RDS services. My next steps are to:

    • Understand the Pricing & Economics of using AWS (how cheap or expensive it is?),
    • Try out the Elastic BeanStalk for PHP Application development, and
    • Setup an Elastic Hadoop framework, and run few data processing jobs

    I hope this post helps you to get started on Amazon AWS.

    Related Links:

    By

    Responsive Web Design (RWD) – Part 2

    Since last year, I have been reading about Responsive Web Design (RWD). I believe it is an important and timely paradigm shift to deal with:

    In this blog, I wanted to spend more time on the nuts and bolts of RWD. For Starters, I recommend reading this formative article by Ethan Marcotte that explains the need for Responsive Web Design succinctly.

    Then check out this site to get a visual grasp of RWD principles.

    At the core of it, the RWD is about the following the three fundamental concepts:

    • Fluid Layouts
    • Media Queries
    • Flexible Media

    The recent advancements in HTML5, CSS3 and JavaScript languages have made the development of responsive websites a bit easier. Lets look at these concepts briefly:

    Fluid Layouts:

    Responsive websites use Fluid layouts that work seamlessly in multiple device screen & orientations.

    Using “Viewport” meta tag is an effective way to control the layout for many screens. Here is a short example of how to control the Viewport:

    1. Using viewport meta tag (or)

    
      <head><meta name="viewport" content="width=device-width, initial-scale=1"></head>
    

    2. @viewport CSS rule

    
      @viewport {
        width: 480px;
        zoom: 1;
      }
    

    Learn more about Viewport here, and about the Syntax here.

    Check out this site that demonstrates fluid layout beautifully.

    We also need to understand how pixels work in various devices. For example both iPhone 3 & iPhone4 have the same screen size of 3.5inches, but iPhone3 has 320×480 resolution whereas iPhone 4 has 640×960. How do we make a picture (or text) look the same in a screen that has twice the resolution of the other? Thankfully, CSS provides this feature called “reference pixel” (aka CSS pixel) which establishes an optical standard for the length of the pixel which is independent of the “hardware pixel”.

    Learn more about Hardware Pixel and Reference Pixel here

    Flexible Media:

    Using Vector images instead of Bitmap images gives you more flexibility, as they are resolution independent. Keep in mind that Vector file sizes are larger. Also, Vector images get re-drawn with each refresh. Use it as circumstance permits. You can learn more about Vector graphics here, and here.

    Another way to control images is by using CSS based Graphics. These are ideal for Buttons, Background images, etc.

    Media Queries:

    Media queries allows you to write styles that respond to different styles or orientations. The “media” tags have been around since CSS 2.x, but CSS3 has extended its features in a meaningful way.

    Here is a short example:

    
    <link rel="stylesheet" href="global-small.css" media="screen and (min-width: 480px) and (max-width: 767px)">
    

    Media queries allows us to create “break points” not just based on width, but also based on (device-)width, (device-)height, orientation, color, resolution, etc.

    We can also use media query to surround a specific style, as shown below:

    
    @media only screen and (max-width: 480px) {
            div.swapimg { background: url(&#8220;lowres-bg.png&#8221;); }
        }
    

    The above method allows us to use a specific background image for any screen less than 480px, and also to download it only as needed (as a background image).

    The W3C Media Queries Doc is an excellent reference for advanced features. I also recommend reading this article.

    While I mentioned just the basic concepts above, there is lot more to Responsive Design, and especially when it comes to making the design work effectively in various browser/os/screen/device combinations. I am no way an expert on CSS or Responsive Design techniques, but I have learned a lot. My perspective is that, there are many valid use cases where forking the main codebase to create a native Android, iOS, and Blackberry app becomes expensive to build and maintain. And in many cases, that might not be enough (iPad, Kindle, etc). In comparison, a well thought out responsive website will go a long way in meeting the business needs in a cost effective way.

    I am not advocating that every website should quit developing native Mobile Apps and should go the responsive Mobile Web design route. Facebook famously decided to build native apps for each Mobile OS, after they realized that the Mobile Web site was not meeting their needs. Each business use case is different, but for majority of the Long Tail use cases, a Responsive Web site will do magic.

    I recommend checking out my favorite sites to take note on good responsive design techniques.

    My Favorite Responsive Sites:

    Hope it helps!

    By

    My Website is fairly Responsive!

    responsive-design

    (Image credit: cyberchimps wordpress theme creator)

    When we developed websites in the early 2000s, we only cared about Desktop/Laptop experience because that was the source of almost all of the Web traffic. One of the major concern back then was Bandwidth, so we focused heavily on optimizing the site for Modem users vs. Broadband users. The Old Yahoo.com home page used to be less than 10K and was famous for its loading speed.

    In the Desktop/Laptop world, the Viewport (the available area for display on the screen) is flexible, i.e., you can change your Browser window to any size that you please. This feature gave headache to Web Designers in providing consistent look and feel to a website. Hence the concept of Fixed Width website came up.

    The idea is simple. You create a fixed size layout, and center the page to the browser window, so that no matter how big your screen is, the site will look consistent. To see how a fixed-width site looks like, go to say the current Yahoo.com or Aol.com or CNN.com sites and try to reduce the window size. You got my point right? As you reduce the screen size, you will notice that the site becomes unusable, since the fixed-width layout does not change based on the window size. I mean, a majority of the sites even now follow the fixed-width site design model.

    In the meantime, the screen landscape has changed dramatically over the past 7 to 10 years. These days, when designing websites, we have to worry about how the site will look like in Tablets, Mobile Phones, TVs, Automobiles, etc., on top of the standard Desktop/Laptop experience. If predictions are right, by late 2013 we will reach the milestone where the Mobile web users will surpass the Desktop web users.

    Mobile Users are projected to surpass Desktop Users this year (2013):

    mobile-vs-desktop-web-users

    (Image credit: Mary Meeker, Morgan Stanley)

    The top online sites counter this trend by “forking” multiple versions of the website for Mobile vs. Tablet vs. Desktop. If you look at HuffingtonPost.com or NYTimes.com in Desktop vs. Mobile phone, you will see a totally different experience. To take this further, some sites (such as Facebook.com) will look different even in different mobile phones (Android vs. iOS vs. Blackberry). While this approach might work for the top sites, it is not a good solution for all.

    So, How do we handle the explosion of these screen types?

    The answer in many cases is Responsive Web Design (RWD). It is a practice of building highly flexible digital experiences that works seamlessly depending on the device, screen size, resolution, operating system, and orientation. The motto is, Build once, and use it in many screens! I know it sounds a LOT, and it is! RWD is a new paradigm of Web Design, and it is constantly evolving. You can find plenty of articles/discussions about it on the Web. According to Wikipedia, 2013 will be the Year for RWD.

    Check out this Interactive Demo to visually understand the RWD.

    I will write more about RWD soon, since there is a lot to talk about the Responsive Design techniques. I brought up this topic because, when I decided to move my Personal site from a custom, hand-coded site to WordPress based Blog, one of my top requirement is to create a responsive site that is flexible enough to render in screen with multiple form factors.

    Surprisingly, I found many highly responsive page templates in WordPress that I can readily use. Almost all of them use a combination of fluid layouts, viewport tag, and Media queries, along with help from Modernizr, JQuery, and other useful libraries. The one I personally chose was the DesignFolio theme (free version), but there are plenty of great themes available. Search for “responsive” in the WordPress themes section.

    Check out how my site appears in different screens:

    Desktop:
    my-site-desktop-1

    Tablet – Horizontal Orientation:

    my-site-ipad-h

    Tablet – Vertical Orientation:

    my-site-ipad-v

    Mobile:

    my-site-iphone-v

    Pretty cool, isn’t it? Thats it for now. Check back later for more on Responsive Web Design.

    By

    How to move WordPress to its own Directory?

    By now you know that I am a big fan of WordPress. These days, I run multiple websites using WordPress, all using one Web Hosting service. When running multiple WordPress sites, a requirement is to move each WordPress site under its own directory under your “www” directory. The WordPress site has a nice article on how to get this done. Check it out here.

    The steps in that article works mostly except one part where we need to copy the .htaccess and index.php file to both the website root dir and your WordPress dir. Assuming your website root is /home/user/www and your WordPress home dir is /home/user/www/blog/, then here is how the .htaccess in both those did should look like below:

    
    For your: /home/user/www/.htaccess file :
    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteRule ^index\.php$&; &#8211; [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /index.php [L]
    </IfModule>
    # END WordPress
    For your: /home/user/www/blog/.htaccess file :
    # BEGIN WordPress
    <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /blog/
    RewriteRule ^index\.php$&; &#8211; [L]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . /blog/index.php [L]
    </IfModule>
    # END WordPress
    

    Once you make the above minor changes to the .htaccess file, you will be all set. Your site http://yourdomain.com/ will now display the WordPress blog from http://yourdomain.com/blog

    Hope it works!

    While we are on the topic of .htaccess file, you should know by now that when we deal with hosting providers where we don’t have our own web servers and root access, then, the .htaccess file is the only way to deal with making Apache web server configurations. This site provides some basic directives that you can add to the .htaccess file for bread-and-butter configurations such as URL Re-writes, Cache-Control, Directory Browsing, etc.

    Have fun.

    By

    My evolution from building custom Personal Web Sites to using WordPress

    Back in the mid 90s, I used to create my personal website from scratch. It was lot of fun! Since I was a student, and since we had access to servers, it was a great way to learn.

    When I got out of school, there was no cost-effective way to host my own website. In those days, there were not many fancy website building platforms. The closest thing to having your own site without too much time and money was to use GeoCities. Yes, i tried that too for a little while…

    In early 2000, I settled on using my own Windows Desktop to host the site (with just Apache web server and CGI). Since I did not have a static IP address, I used one of those free Dynamic IP services to point my domain name to my Windows desktop. As you can imagine, the IP address changed very often, and hence my website will go down frequently. It was not fun.

    Around that time, Broadband internet became affordable, and Linux became popular as a server. This led me to experiment with running my own web server using Linux box and a Static IP. It was great! I used Debian Linux, Apache, CGI, PHP, and MySQL. I also installed my own MovableType instance for Blogging. I learnt a lot from this experience. But, it turned out to be more expensive and I wasn’t making any good use of it.

    So around Mid 2000′s I started looking into hosting providers and found them to be relatively cheap. For about $10/month, you get your own hard disk space, Linux, Apache, MySQL and PHP (LAMP), and many other site building softwares. I ended up moving my site to a hosting provider. It worked out very well, i.e., I was abstracted from the pains of day-to-day site management. This approach allowed me to use all the good softwares I like, with some minor compromises on Security, Computing Space/resources, and root privileges.

    I was still creating my own web pages from scratch…

    I did not like the fact that my site got stale, and that it took time to make any serious site updates (changing layout, adding new features, etc). I looked at some of my friends’ websites, and liked the simplicity of using a blog as a primary web site. After all, the purpose of a website is to share things with the world.

    So, recently, while I was looking into cheaper hosting companies, I decided to look at using blogging softwares as my primary website. I played with WordPress.com, Blogger.com, etc., and quickly realized that I need my own Blogging software to do all those fancy customizations, and to fiddle with the code. As a tech geek, I felt too restricted using a hosted blog platform. After reading few reviews, I quickly settled on WordPress software – the swiss-army knife of Blog softwares. I found a web hosting company that is cheap ($4/month), and provides ready to install WordPress software, among others.

    I have been using WordPress for couple of weeks now, and it is fun! It is very easy to change themes, add pages, and update content. I am now rejuvenated to Blog again! I wish I had done this early on. I plan to stick with this approach until I move to the next step, which is hosting my site using Amazon EC2 (more on it soon…).

    So friends, TWO THUMBS UP for WordPress.Org. I highly recommend setting up your own instance of WordPress software and use it as your Personal Website.

    By

    My First Post!

    Hi All,

    I just moved my domain to a new ISP.  Please bear with me while I setup my blog.  Thank you and please visit again.

    -Ramana