Hi
I want to build a tool to check asset availability (video, images, etc.) from third-party vendors. These vendors have APIs. However, this process should run daily basis and update the database with the status. This is a kind of separate process. I need to know what will be the good approach and technology for this?
Piusha, I can tell you how I implement this on my projects. I have the message queue and task scheduler the same daemon on my micrioservices which is controlled by REST API , has limitations on requesting stuff, etc. It generates the events, and another my service onboard, the API gateway, has the request fired at that same moment. The scheduler doesn't know how to authorize on vendor's API, what info is needed from there and which is not; but API gateway knows that, performs the job and has the info back to the scheduler. The next great feature of the scheduler is that it sends that same info it received on that same API gateway (or wherever I want, as it's a part of its' particular job description) but to another endpoint - to be stored then.
Among other things such a scheduler is able to repeat the job up to the limit of tries for the case if the latest request fired by that job was unsuccessful.
This comprehenses alot my other approaches, e. g., by Cron daemon built in OS.
The name of the scheduler/message-queue software is: Agenda-Rest, it can be found on github. You also may want to use Bree instead, it should be even better.
Thank you for your valuble idea. let me explore what you gave as an idea
The major advantage of Go is that you can run queries in parallel. Fire off a Go thread for each vendor and each thread can check the availability of assets from a specific vendor and update the database. Go supports hundreds of threads with ease.
You can do that with any language... The main difference with Go is that it implements Green Threads rather than OS threads, allowing for parallel code on a single CPU core.