"Splitting" big cron jobs (tasks) or functions/API

Lukasz Kwiatkowski

Hello Guys, I have three use cases that make think how we can execute tasks or API calls in a "background" without blocking all computing power or to avoid external API rate limits:
1. My integration partner API has a rate limit of 6000 hits per minute so basically if my user bases reaches it 3k as I need two calls per user, it will hit the rate limit if I re-sync every day for example. Now one of the options is putting an index in the loop and "pausing" for 60 minutes every time index reaches multiple of certain value. Does that make sense? or there any other ways of doing this?

2. Second use case is a task that will simply clean my DB and re-order data. Let's say, it would take 5 minutes to process everyday which creates certain risk of flooding server processing power. I would definitely prefer to distribute this tasks to run for example 10 seconds every 5 minutes or sth like this, given the potential future scale and lack or urgency. That solution to execute certain number of loops and pause all seems feasible here. What do you guys think?

3. 3rd use case is a user-facing function. the difference with two others is that has to return response to user (cannot be just paused for 30 min) but it could process only initial part of data, return success response and let the rest run in the "background", any idea solution on how to handle this?