Hi everyone! I'm starting a personal project that I've been postponing for a little while and I'm starting with the backend first.
It's an app that needs to query open APIs (It could go up to a hundred APIs) and do filtering and other manipulations onto the data and then store it in my own database. So basically, the end result medium/long term is a lot of data!
Besides filtering/storing data from other APIs, the API will interact with my client application. I should note that the client app attempts to give real-time information.
Question: I would want to launch the API within 6 months as I'm also planning to make it available for other devs through platforms like RapidAPI. With all being said, my question is twofold:
Which backend framework would allow me to do these operations with the best response time possible?
Which framework would be the easiest to deal with deployment on AWS?
Background:
I have a background in Spring since I'm using it at work, I have none in Rails and Django but I don't mind trying something new as long as the learning curve isn't crazy. I know python but not Ruby. Beginner on AWS and choose that one after some research.
Thank you very much, sorry it was a bit long. Cheers
The most relevant part is the central architecture more than the specific backend. For me, it makes sense to go for an event-driven solution. In this case, an event is emitted on every new data, and one or more listeners react to it. In AWS, you can easily get this by storing your data in DynamoDB, which is pretty fast, and have as many lambdas working on the received data (here you can find more details). This way, you keep your worker(s) busy fetching data while others consolidate it. I wouldn't get too crazy on the lambdas, and I'd start with one as there will be only one client by your description. My only concern is how similar the APIs you're consuming are. Depending on that, it may be easier to have different lambdas by target API. My only note on the backend language is that if you want low latency, I'd discard Python and Ruby and choose Go (I know that is not in the list, but given that you're open to learning a new language, you could give it a try). It is easy to learn, has excellent performance, and is the only Lamdba runtime with an X (1.X) in the supported language, making it a great choice.
I am a business analyst with many years of experience. I agree with j-Velasco suggestion. I would add to the stack Apollo server so your API will get the benefits of GraphQL. Apollo server can be deployed with AWS Lambda.
