We have some lambdas we need to orchestrate to get our workflow going. In the past, we already attempted to use Airflow as the orchestrator, but the need to coordinate the tasks in a database generates an overhead that we cannot afford. For our use case, there are hundreds of inputs per minute and we need to scale to support all the inputs and have an efficient way to analyze them later. The ideal product would be AWS Step Functions since it can manage our load demand graciously, but it is too expensive and we cannot afford that. So, I would like to get alternatives for an orchestrator that does not need a complex backend, can manage hundreds of inputs per minute, and is not too expensive.
I think the problem is that you need some kind of "realtime orchestration" – and thus completely different tools designed for that. It seems like an product development problem – probably you need to use some message queue (RabbitMQ, Kafka) and own processing gateway service (maybe another Lambda) that'll react to those inputs.
