DigitalOcean was where I began; its USD5/month is extremely competitive and the overall experience as highly user-friendly.

However, their offerings were lacking and integrating with other resources I had on AWS was getting more costly (due to transfer costs on AWS). Eventually I moved the entire project off DO's Droplets and onto AWS's EC2.

One may initially find the cost (w/o free tier) and interface of AWS daunting however with good planning you can achieve highly cost-efficient systems with savings plans, spot instances, etcetera.

Do not dive into AWS head-first! Seriously, don't. Stand back and read pricing documentation thoroughly. You can, not to the fault of AWS, easily go way overbudget. Your first action upon getting your AWS account should be to set up billing alarms for estimated and current bill totals.

We first selected Google Cloud Platform about five years ago, because HIPAA compliance was significantly cheaper and easier on Google compared to AWS. We have stayed with Google Cloud because it provides an excellent command line tool for managing resources, and every resource has a well-designed, well-documented API. SDKs for most of these APIs are available for many popular languages. I have never worked with a cloud platform that's so amenable to automation. Google is also ahead of its competitors in Kubernetes support.

GCE is much more user friendly than EC2, though Amazon has come a very long way since the early days (pre-2010's). This can be seen in how easy it is to edit the storage attached to an instance in GCE: it's under the instance details and is edited inline. In AWS you have to click the instance > click the storage block device (new screen) > click the edit option (new modal) > resize the volume > confirm (new model) then wait a very long time. Google's is nearly instant.

• In both cases, the instance much be shut down.

There also the preference between "user burden-of-security" and automatic security: AWS goes for the former, GCE the latter.

We offer our customer HIPAA compliant storage. After analyzing the market, we decided to go with Google Storage. The Nodejs API is ok, still not ES6 and can be very confusing to use. For each new customer, we created a different bucket so they can have individual data and not have to worry about data loss. After 1000+ customers we started seeing many problems with the creation of new buckets, with saving or retrieving a new file. Many false positive: the Promise returned ok, but in reality, it failed.

That's why we switched to S3 that just works.

Most bioinformatics shops nowadays are hosting on AWS or Azure, since they have HIPAA tiers and offer enterprise SLA contracts. Meanwhile Heroku hasn't historically supported HIPAA. Rackspace and Google Cloud would be other hosting providers we would consider, but we just don't get requests for them. So, we mostly focus on AWS and Azure support.

To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

#BigData #AWS #DataScience #DataEngineering