Module Queue

9 views Markdown

The nine endpoints that watch, retry and clear the work modules do in the background.

Overview

Provisioning, suspending or cancelling a service means the module has to talk to a provider. To keep the request from waiting, that conversation goes into a queue and runs in the background. These nine endpoints watch that queue and step into it.

A job gets a set number of tries; on reaching it the job fails and is never tried again on its own. Putting a failed job back in line, or running it without waiting, is what these endpoints are for.

Reference

Listing the Queue

get/api/v1/admin/tools/module-queue
Tools/GetModuleQueue admin paged

Returns the jobs handed to modules to run in the background.

Query parameters 6
statusstringFilters by status.
module_namestringFilters by module name.
actionstringFilters by operation.
pageintDefaults to 1.
limitintDefaults to 25, maximum 100.
searchstringSearches the records.
Response fields data[] — 17
idintId of the queue item.
module_typestringThe module type.
module_namestringThe module name.
actionstringThe operation to run.
statusstringpending is queued, processing is running, completed finished, failed did not.
service_idintThe service the operation concerns.
service_namestring | nullThe service name.
user_idintThe client id.
user_full_namestring | nullThe client's name.
addon_idint | nullThe add-on id.
addon_namestring | nullThe add-on name.
server_idint | nullThe server id.
attemptsintHow many times it was tried.
max_attemptsintHow many tries it gets. On reaching it the item fails and is not retried on its own.
created_atstring | nullWhen it entered the queue.
updated_atstring | nullWhen it last changed.
next_retrystring | nullWhen the next try is due.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl -G 'https://panel.example.com/api/v1/admin/tools/module-queue' \
  -H "Authorization: Bearer $API_KEY" \
  -d status=failed
const url = new URL('https://panel.example.com/api/v1/admin/tools/module-queue');
url.searchParams.set('status', 'failed');

const res  = await fetch(url, { headers: { Authorization: `Bearer ${apiKey}` } });
const body = await res.json();
$url = 'https://panel.example.com/api/v1/admin/tools/module-queue?' . http_build_query(['status' => 'failed']);

$ch = curl_init($url);
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The list carries no logs: read an item's detail to see why a job failed.
$failed = Api::Tools()->GetModuleQueue([], ['status' => 'failed'])['data'];

The Queue Counters

get/api/v1/admin/tools/module-queue/stats
Tools/GetModuleQueueStats admin

Returns how many jobs are in the queue, counted by status.

Response fields data — 5
totalintTotal jobs in the queue.
pendingintHow many are queued.
processingintHow many are running.
completedintHow many finished.
failedintHow many failed.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/tools/module-queue/stats' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/stats', {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/stats');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// This is the cheapest endpoint to watch: it gives the queue's health without pulling the list.
$stats = Api::Tools()->GetModuleQueueStats()['data'];

$stuck = $stats['failed'] > 0 || $stats['pending'] > 100;

Item Detail

get/api/v1/admin/tools/module-queue/{id}
Tools/GetModuleQueueItem admin the logs are here

Returns one job together with the record of what was exchanged with the provider.

Response fields data — 19
idintId of the queue item.
module_typestringThe module type.
module_namestringThe module name.
actionstringThe operation to run.
statusstringpending is queued, processing is running, completed finished, failed did not.
service_idintThe service the operation concerns.
service_namestring | nullThe service name.
user_idintThe client id.
user_full_namestring | nullThe client's name.
addon_idint | nullThe add-on id.
addon_namestring | nullThe add-on name.
server_idint | nullThe server id.
attemptsintHow many times it was tried.
max_attemptsintHow many tries it gets. On reaching it the item fails and is not retried on its own.
created_atstring | nullWhen it entered the queue.
updated_atstring | nullWhen it last changed.
next_retrystring | nullWhen the next try is due.
api_logsarrayThe requests and responses exchanged with the provider. Not in the list; it is heavy, so only here.
process_logsarrayThe job's own step-by-step record.
Errors 3
invalid_id422The id is not valid.
not_found404No such queue item.
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/tools/module-queue/101' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/101', {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/101');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Read the logs BEFORE retrying: with the same condition in place the job fails at the same point.
$item = Api::Tools()->GetModuleQueueItem(['id' => 101])['data'];
$last = end($item['api_logs']);

Retrying an Item

post/api/v1/admin/tools/module-queue/{id}/retry
Tools/RetryModuleQueueItem admin queues it

Puts a failed job back in the queue and resets its try counter.

Body —
——No body is needed. The job is addressed by the path parameter; send an empty body.
Response fields data
dataobjectThe job as it stands after the reset. Same shape as the item detail endpoint.
Errors 3
not_found404No such queue item.
not_failed422Only failed jobs can be retried.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/tools/module-queue/101/retry' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/101/retry', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/101/retry');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// A retry does not run the job NOW: it queues it and the background worker takes over.
// To see it without waiting, use the run endpoint.
Api::Tools()->RetryModuleQueueItem(['id' => 101]);

Running an Item Now

post/api/v1/admin/tools/module-queue/{id}/run
Tools/RunModuleQueueItem admin runs immediately

Runs the job there and then, without waiting for the background worker, and returns the outcome.

Body —
——No body is needed. The job is addressed by the path parameter; send an empty body.
Response fields data — 3
task_successboolWhether the job succeeded.
task_messagestringThe error message when it did not.
itemobjectThe job as it stands after the run.
Errors 3
not_found404No such queue item.
not_runnable422Only queued or failed jobs can be run.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/tools/module-queue/101/run' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/101/run', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/101/run');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The request answers 200 even when the job FAILED: success is in 'task_success'.
$result = Api::Tools()->RunModuleQueueItem(['id' => 101])['data'];

if (!$result['task_success']) {
    $why = $result['task_message'];
}
Response 
{
  "data": {
    "task_success": false,
    "task_message": "Provider refused: quota exceeded.",
    "item": {
      "id": 101,
      "status": "failed",
      "attempts": 3,
      "max_attempts": 3
    }
  }
}

Deleting an Item

delete/api/v1/admin/tools/module-queue/{id}
Tools/DeleteModuleQueueItem admin hook-guarded

Takes a job out of the queue. The work itself stays undone.

Response fields data — 2
deletedboolWhether the delete succeeded.
idintId of the deleted item.
Errors 3
not_found404No such queue item.
blocked_by_gate422The gate:module.queue_intervene hook vetoed the operation.
insufficient_scope403The key lacks the required scope.
Request 
curl -X DELETE 'https://panel.example.com/api/v1/admin/tools/module-queue/101' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/101', {
  method: 'DELETE',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/101');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'DELETE',
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Deleting CANCELS the job: a service waiting to be provisioned keeps waiting.
Api::Tools()->DeleteModuleQueueItem(['id' => 101]);

Deleting in Bulk

post/api/v1/admin/tools/module-queue/bulk-delete
Tools/BulkDeleteModuleQueue admin hook-guarded

Takes several jobs out of the queue.

Body 1
idsint[]requiredIds of the items to delete.
Response fields data — 3
deletedboolWhether the delete ran.
countintHow many ids were accepted. It is the length of the list below, not a fresh count from the database.
idsint[]The ids that were accepted for deletion. Ids that no longer exist come back too, so this is not a confirmation of what was removed.
Errors 3
ids_required422No id was given.
blocked_by_gate422The gate:module.queue_intervene hook vetoed the operation.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/tools/module-queue/bulk-delete' \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"ids":[101,102,103]}'
const res = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/bulk-delete', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${apiKey}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ ids: [101, 102, 103] }),
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/bulk-delete');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => [
        'Authorization: Bearer ' . $apiKey,
        'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS => json_encode(['ids' => [101, 102, 103]]),
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The id list comes back as you sent it: ids that no longer exist are echoed too, never dropped.
$response = Api::Tools()->BulkDeleteModuleQueue(['ids' => [101, 102, 103]]);

$echoed = $response['data']['ids'];

Retrying Everything That Failed

post/api/v1/admin/tools/module-queue/retry-all-failed
Tools/RetryAllFailedModuleQueue admin the whole queue

Puts every failed job in the queue back in line.

Body —
——No body is needed. The action covers the whole queue and cannot be narrowed; send an empty body.
Response fields data — 1
retriedboolWhether it ran. It does not say how many were queued.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/tools/module-queue/retry-all-failed' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/retry-all-failed', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/retry-all-failed');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The endpoint does not say how many were queued; measure it with the counters either side.
$before = Api::Tools()->GetModuleQueueStats()['data']['failed'];
Api::Tools()->RetryAllFailedModuleQueue();
$after  = Api::Tools()->GetModuleQueueStats()['data']['failed'];

Clearing What Finished

post/api/v1/admin/tools/module-queue/clear-completed
Tools/ClearCompletedModuleQueue admin

Deletes the finished jobs from the queue.

Body —
——No body is needed. The action covers the whole queue and cannot be narrowed; send an empty body.
Response fields data — 1
clearedboolWhether the clear ran.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/tools/module-queue/clear-completed' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/tools/module-queue/clear-completed', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/tools/module-queue/clear-completed');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The clear takes what FINISHED; failed jobs stay put and wait to be looked at.
Api::Tools()->ClearCompletedModuleQueue();

Pitfalls

A failed run comes back inside a 200

The run endpoint answers 200 even when the job failed; whether it worked is a field in the response. A client reading only the status code counts a failed provisioning as a success. The error message comes back in the same response.

Retrying and running are not the same

A retry queues the job: the try counter resets and the background worker runs it when its turn comes. The run endpoint does the work there and then and returns the outcome. Use the second when you want to see what happened.

Deleting cancels the work

Deleting a queue item does not merely tidy a record: that job is never done. A service waiting to be provisioned keeps waiting and nothing reminds you. That is why the delete is guarded by a hook, and your installation may refuse the request through it.

Why it failed is not in the list

The record of what was exchanged with the provider is heavy, so it is left out of the list and appears only on a single item. Read it before retrying a job, because with the same condition in place it fails at the same point again.

The bulk endpoints give no count

The retry-all and clear-completed endpoints say only that they ran; they do not report how many jobs they touched. To measure the effect, read the counters before and after.

Was this helpful?

Thanks for your feedback!

Still Need Help?

Our support team is here around the clock for anything you can't find above.

Contact Us Open a Ticket

We use cookies to keep you signed in, remember your language and currency, and improve the site. Cookie Policy