If you just want the basic orchestration between a set of defined hosts, go with Docker Swarm. If you want more advanced orchestration + flexibility in terms of resource management and load balancing go with Kubernetes. In both cases, you can make it even more complex while making the whole architecture more understandable and replicable by using Terraform.

Mortie Torabi
DevOps Architect
7 upvotes·179.6K views
From my experience, I can confirm that Elasticsearch is a highly flexible and powerful NoSql database that you can use for such purposes. It is actually used even for scenarios with much more pressure like logging systems that collect lots of logs per second. We used Elasticsearch in our company to collect logs of 90 different microservices in parallel each generating a huge amount of logs and it worked very nicely. In addition, if you want to stay with the AWS Stack, there is an Amazon service called "Amazon Elasticsearch Service" exactly for your case.
3 upvotes·3.3K views