Also, for database I'm using mongo for my this huge personal project but I checked Aerospike (more like redis + mongo) you may use this instead of mongo for better performance. Or if you plan on going on with mongo try Tokumx by http://percona.com/. It literally says MongoDB on steriods (I tried but didn't go with my setup). Percona is really known for pumping databases like MySQL for better performance.

Ajay M

I'd recommend using CloudTrail, it helped me a lot. But depending on your situation I'd recommed building a custom solution(like aws amazon-ssm-agent) which on configuration change makes an API call and logs them in grafana or kibana.
Go with RabbitMQ over redis. Not that I had any problem with redis but I used (without any problem) RabbitMQ with over 10 million records on a 1GB system and 3 cpu with 25+ (50 workers each node) nodes to handle scraping for our project. I'm sure redis can also handle that but it may need more ram.

I don't think you can achieve parallelism with rails as I know ruby also got GIL similar to python. I'd recommend going with system languages, I'd some luck with Go to auto deploy scrapy scrapers. Or try java even though it's slow compared to other system languages but it's robust.