[celery] Configuring Servers in Batches with Time Limit and Revocation /u/ParticularAward9704 Python Education

I have a task where we need to configure multiple remote servers in batches of 4. The user can specify any number of servers to configure, but we can only configure 4 servers concurrently. Once a batch of 4 finishes, we proceed with the next batch (if available).

There are two key things to consider:

1. Strict Time Frame (Start Time and End Time):
   • I want to set a start time using eta (which works fine).
   • For end time, I plan to check the time elapsed after each batch finishes. If we exceed the available time (e.g., if configuring the second batch takes too long), I don’t want to schedule any subsequent batches.
   • For example, if I have 3 hours to configure 16 devices:
     • The first batch takes 1 hour, leaving 2 hours.
     • The second batch takes 1.5 hours, totaling 2.5 hours elapsed.
     • The average time per batch is 2.5/2 = 1.25 hours, which exceeds the available time for the remaining batches (if any). Hence, no further batches should be scheduled.
2. Revocation of Future Batches:
   • Let’s say I have 16 devices. The first batch runs successfully. However, if the second batch fails, I want to be able to revoke the execution of all subsequent batches (e.g., from server 8-16).
   • In the UI, if the user sees that the second batch failed, they can choose to stop the next batches from executing.

Here’s the current structure of my Celery task:

@app.task(bind=True, name='conf_servers') def conf_servers(self, servers): for idx, server in enumerate(servers): if idx % 4 == 0: if not can_proceed_next_batch(): return configure(server) 

However, this approach doesn’t run the servers in parallel within each batch, and it doesn’t address how to stop future tasks if the user requests revocation.

How can I configure the servers in parallel within each batch, and how can I implement revocation of further tasks if the user chooses to stop them?

I have a task where we need to configure multiple remote servers in batches of 4. The user can specify any number of servers to configure, but we can only configure 4 servers concurrently. Once a batch of 4 finishes, we proceed with the next batch (if available).

There are two key things to consider:

1. Strict Time Frame (Start Time and End Time):
   • I want to set a start time using eta (which works fine).
   • For end time, I plan to check the time elapsed after each batch finishes. If we exceed the available time (e.g., if configuring the second batch takes too long), I don’t want to schedule any subsequent batches.
   • For example, if I have 3 hours to configure 16 devices:
     • The first batch takes 1 hour, leaving 2 hours.
     • The second batch takes 1.5 hours, totaling 2.5 hours elapsed.
     • The average time per batch is 2.5/2 = 1.25 hours, which exceeds the available time for the remaining batches (if any). Hence, no further batches should be scheduled.
2. Revocation of Future Batches:
   • Let’s say I have 16 devices. The first batch runs successfully. However, if the second batch fails, I want to be able to revoke the execution of all subsequent batches (e.g., from server 8-16).
   • In the UI, if the user sees that the second batch failed, they can choose to stop the next batches from executing.

Here’s the current structure of my Celery task:

@app.task(bind=True, name='conf_servers') def conf_servers(self, servers): for idx, server in enumerate(servers): if idx % 4 == 0: if not can_proceed_next_batch(): return configure(server) 

However, this approach doesn’t run the servers in parallel within each batch, and it doesn’t address how to stop future tasks if the user requests revocation.

How can I configure the servers in parallel within each batch, and how can I implement revocation of further tasks if the user chooses to stop them?