Amazon Web Services’ yearly shindig, re:Invent, turned out to be yet another whirlwind of information and product releases. Here’s the skinny on all of the important new services, in the order of what I see as the most impactful. Like all major cloud providers these days, AWS is launching these services in beta or preview mode. But in AWS parlance, that simply means that the services don’t yet have all features they’d like to see in production. All of the services below are available for immediate use.
Amazon QuickSight – QuickSight is a pay-as-you-go, cloud-based data analytics and visualization platform. With it, AWS is taking direct aim at the lucrative business intelligence (BI) visualization market currently monopolized by a handful of players like Tableau, MicroStrategy, QlikView and others. It includes an in-memory analytics engine, collaboration capabilities, easy data discovery and report recommendation/creation for common tasks like Web log analysis, out of the box integration to AWS’ data sources, and a SQL interface. QuickSight claims to be a 10th of the cost of traditional visualization tools.
AWS IoT – With this service, AWS addresses one of the biggest challenges in creating an Internet of Things (IoT) architecture: how to scale your back-end system to deal with millions or trillions of disparate devices, each of which is sending or receiving only small packets of data on an unknown or sporadic schedule. It provides MQTT or REST interfaces; a responsive, highly scalable message broker bus; and uses a SQL-based real-time rules engine to transform messages and route part or all of the data to AWS data stores. Many partners like Intel have also released $100– $200 IoT starter kits to allow users to easily experiment with building IoT applications and interfacing with the AWS IoT back-end.
AWS Import/Export Snowball – One of the biggest challenges in using AWS has always been securely and cost-effectively migrating data into or out of the platform. To move vast amounts of data, customers typically had to drop that data onto dozens or hundreds of 1-2TB hard disks and then FedEx the disks back and forth. Snowball provides customers with a 50TB durable, easy-to-configure, secure, completely self-contained shipping crate. It’s about as simple as you can get, and only costs $200 per snowball.
AWS Config Rules – AWS Config is a service that lets users inventory and track changes to AWS-side resources. It’s been around for some time, but was limited to only reporting static configurations and histories. With Config Rules, users can now set actions to be taken if/when configurations go out of spec. Users can also programmatically ensure compliance with tagging taxonomies, security guidelines and other best practices. The service hooks into Lambda, which now supports even Python scripts, to call functions that can do anything from terminating servers and rolling back configurations to raising alerts.
AWS Database Migration Service – One of the most complicated and nerve-wracking steps in any migration is cutting over the database layer – a process that normally involved some downtime and/or overtime. With AWS Database Migration Service (DMS), users can now push-button migrate a running database. All data synchronization and cutovers happen transparently behind the scenes, allowing systems to remain up and running during the migration. The DMS also includes a Schema Conversion Tool which automagically assesses and converts data formats, syntax, and stored procedures/functions from the source to the target. This greatly simplifies migration from costly, proprietary engines like Oracle or SQL Server to lower cost, open source alternatives like MySQL and PostgreSQL.
Amazon Kinesis Firehose – Kinesis, launched only two years ago, is a highly scalable managed service to ingest huge amounts of streaming data. One of the biggest limitations to the service, however, was that performing a simple load of streamed data into AWS required a fleet of expensive EC2 instances which you had to set up and maintain yourself. Firehose completely obviates the need for these EC2 fleets behind Kinesis. If you’re looking to load that data into S3 or Redshift, Firehose can complete that operation with just a few clicks in the console. AWS also alluded to Kinesis Analytics – a soon-to-be-announced service to run standard SQL queries against live streams of Kinesis data, again removing the need for EC2 instances. AWS also upped the maximum retention period of Kinesis data from 24 hours to seven days.
Amazon Elasticsearch Service – Elasticsearch (ES) is a document-oriented, popular open source text search engine for uploading, indexing, and querying massive document stores like journals, web pages, or Web logs. It does everything Apache Lucene can do, plus more. With ES, AWS offers the functionality of Elasticsearch as a scalable, fault tolerant, pay-as-you-go managed service — greatly reducing the time and costs required to create and maintain the infrastructure behind Elasticsearch. At first glance, ES directly competes against another AWS service called CloudSearch, and it will be interesting to see how, or if, they choose to differentiate the two.
Some of the other interesting announcements were a host of changes for Lambda, including Python support; Web Application Firewall as a Service (AWS WAF); Amazon Inspector , an automated security assessment service; RDS support for MariaDB; and a swath of changes around EC2, Container support, and Mobile Hub.
Although we’ve presented what we feel are the most exciting announcements to come out of AWS re:Invent, a full list of all service announcements can be found on AWS’ New Announcements page. At any rate, the announcements at re:Invent 2015 signal that AWS is only pushing the gas pedal harder to put yet more distance between themselves and the competition.
Related Training
AWS Training