Hello, I am building an analytical application that uses several third-party APIs, combines the data, performs heavy computations, sorts data, and finally serves over a RESTful API which is being used by Frontend tools to visualize the results.
The problem is computation. It's a computation hungry app which has to work on tones of data before it can return a chunk to be visualized. Moreover, the third-party APIs which this application relies on actually provide data in reports format, not the RAW data, which means this data cannot be normalized to be saved in any database format.
The backend, which does computation, is PHP based serving results over a RESTful API.
What do you think is the best approach to design such an application? What tools, infrastructure, and technologies I should consider for making sure that the computation is fast for huge data, and what should I use to cache/store this kind of data for faster retrievals?
wrong question. lol
python is a language, lambda is faas.
I think your question should be php vs python which can help me to get better computation? should I use lambda for my computation hungry app.
the answer is.
- use python if you can, python is more flexible than php, especiaslly for data processing (the restful api does not return your data, you need some processing for it)
- you can use lambda, it will be better to not use lambda alone, lambda has limit like 15 minutes for the maximum running time, my suggestion is to design an async solution, use AWS SQS as the queue and lambda as the worker, at front-end, you can either polling from the queue through api gateway or use graph ql subscribe or websocket.