AWS always seems to amaze me how they can come up with new things every year (in fact continuously all year, they just wait for re:Invent for the really fun stuff). Well actually I guess it’s really us and you, their customers who come up with the ideas, AWS just figures out cool ways to solve them and Andy Jassy today’s keynote speaker (CEO of AWS) did not disappoint with todays keynote at re:Invent 2017.
Elastic Container Service for Kubernetes (EKS)
AWS released ECS about 3 years ago into an ecosystem filled with many different container orchestration engines. Over the last 18+ months Kubernetes has really started to become the winner in this race, and today it’s great to see that AWS are now adding Kubernetes to their ECS offering.
ECS for Kubernetes will be completely managed and all the community tools that you use with Kubernetes today will be supported. And of course in true AWS fashion, it will be built with security 100% in mind and integrate with other AWS services, e.g. IAM for access control.
EKS is in preview today, but don’t let that hold you back. If you are interested in building a micro-services architecture based on Kubernetes or migrating to Kubernetes, get in touch and we can help you in your journey.
Of course no cloud provider would be complete without offering a way to “just run” containers. AWS have had support for doing this in a loose way with the likes of Elastic Beanstalk, but now it’s going to be tied into ECS/EKS with AWS Fargate.
All you do is package your app with Docker, push it to a registry, define some networking, cpu, memory and IAM policies and you’re away. Easy!
Of course with AWS you have options. You can also have some control of the underlying cluster. Meaning you can manage the ECS/EKS cluster, but without the full control that ECS or EKS gives you. This allows you to schedule your containers in a particular way and also customise (to some degree) the environment these containers run on. Win!
Currently in Aurora you have a single writer and many readers. If the writer failed, then a reader would be promoted. This could take 10secs of seconds to complete and isn’t as fast as having another writer ready to go. With todays announcement you will be able to have multiple readers as well as multiple writers in a cluster with-in one region.
This feature is currently in preview.
It was also mentioned that a multi-master multi-region option will be in preview in 2018. It will be very interesting to find out if this allows you to run your workload in 2 regions and how replication will happen and if it will be fast enough to support this type of workload. I can’t wait to give these features a test.
If you can make compute serverless (and really it’s just a function), why then can’t you do this with databases. Really?! Mind-blown!
What this means in reality is that you have a ‘defined’ database and at the backend AWS will turn on/off resources and add them to your pool as requested. You then pay for the storage and cpu on a per second basis. This was made possible because of the design of Aurora.
Aurora Serverless is in preview today and expected to be generally available in 2018.
DynamoDB Global Tables and Backup/Restore
DynamoDB is AWS’s answer to NoSQL. It’s a document store that we see utilised for all types of workloads.
Global Tables will really allow you to take your application and make it extremely highly-available and ensure that users are having the best experience possible as latency to the services and specifically the database will be dramatically reduced. Global Tables does this by supporting multiple writers across multiple regions, not just two.
From today, you will now be able to perform on-demand backups of your DynamoDB tables. Something that has been tricky in the past. You’ll be able to restore these backups into new DynamoDB tables. This feature is being rolled out on an account-by-account basis. If you are currently a DynamoDB user and want to make use of this feature, please get in contact with us and we can help you setup your backup plan.
Graph databases have started to become popular recently especially when you are trying to build queries around complex relationships and using a traditional RDBMS is complex or not performant enough.
There are a number of popular Graph databases out there, but now AWS have their answer for a managed Graph Database, Amazon Neptune.
Amazon Neptune will support the following graph models: Property Graph and W3C’s RDF. Both of their query languages, TinkerPop and SPARQL, will be supported. Neptune is currently in preview.
S3 and Glacier Select
S3 is often the basis for many data lake implementations. With tools like Athena, Redshift and EMR that can utilise S3 this makes it really great for querying and consuming data. What is sometimes annoying, is you just need a tiny bit of data from large objects in your S3 bucket.
With S3 select this is now easy.
But why stop there? The team have also made this available to Glacier as well. Of course it will still take time to retrieve the data, but you’ll get back only what you need.
Again this is currently only available in preview.
This got us quite a way through the key note, but there’s still so many more things to come.
If you want to try and get started on Machine Learning, but not sure where or how to start, then SageMaker is for you. Amazon SageMaker is AWS’s answer to helping you build, train and deploy Machine Learning for your application. Because it is modular it also means you can swap out any part and BYO if you like.
This really now brings Machine Learning to smaller customers who couldn’t afford the infrastructure or the specialist in the past. I’m really excited to see what people will be creating.
And if you are still not sure where to start with your Machine Learning journey, then there is always DeepLens. You can be up and running within 10 minutes all for the low price of USD$249 (pre-order now, but US only at the moment). You can deploy your ML model to the device (think Greengrass) and test it out.
If you get bored with that, I’m sure you can run your favourite video conferencing software on it as it’s running Ubuntu and has a really good camera.
Amazon Rekognition Video and Amazon Kinesis Video Streams
If you have been following along we recently built an app that utilises Rekognition for image processing. Well now, we can take that to the next level and interpret video with Amazon Rekognition Video. That’s pretty awesome. And how do you get all that video into AWS I hear you ask, well Kinesis Video Streams of course!
Amazon Transcribe; Amazon Translate; Amazon Comprehend
And for all those audio and text processing fans, there are new text and audio processing tools to build an entire solution for what ever project you are looking at.
Transcribe allows you to perform automatic speech recognition on files that have been uploaded to S3 and return a text file of the speech transcribed.
Translate is a neural machine translation engine which tries to make the translation more natural compared to traditional methods of translation. It can be used for bulk translation of large blocks of text.
Comprehend is a natural language processing service that uses machine learning to comprehend language in text and provide information about the text such as people, places brands and events.
If you are still reading, congratulations! I was starting to get a bit twitchy by this time as I wanted to get in and find out more about some of the products, but let’s keep going and find out what new services and features have been released around IoT.
If you are working in the IoT space then there are lots of cool things here to keep you busy over the holiday period.
AWS IoT 1-Click allows simple devices to trigger Lambda functions. e.g. to open doors, call technical support, etc. It’s really simple to get started, you can download the app from Google Play or Apple App store to your mobile device, configure the devices and select the Lambda function to trigger. You then deploy the 1-Click to your device and you are ready to go.
AWS Device Manager allows you to manage the full life-cycle of all your IoT devices, from onboarding to decommissioning and everything in between. This will allow you to keep on top of firmware updates for example to avoid attacks similar to those we’ve seen in recent past.
AWS Device Defender extends Device Manager to ensure your fleet is as secure as possible on an ongoing basis. It helps detect abnormal device behaviour and can trigger alerts with recommended mitigation strategies.
AWS IoT Analytics helps you organise all the data that is coming back from your IoT fleet. It’ll help you store, collate, query and report on any of the data, metrics or meta information that you are sending.
Amazon FreeRTOS is Amazons answer to building everything you need for your IoT device ready to go. It’s based on the FreeRTOS kernel and provides all the libraries for building applications to integrate with AWS.
Greengrass was launched last year, and a new feature has now been added. You can run Machine Learning inference right on the device instead of needing to rely on an Internet connection and the cloud being available.
This is only the 5th re:Invent. Andy and AWS just don’t stop giving. The rate of innovation is amazing and the team just keeps on going. It’s a really exciting time to be involved in an amazing industry.