Queue Maintenance

7 views Markdown

The five endpoints that clear the queues, measure them and handle the restore warning.

Overview

Queues grow over time. These five endpoints deal with maintenance: clearing old jobs, seeing how much room the tables take, and handling the warning that appears when the database is restored.

The clean-up comes in two steps. The preview counts and removes nothing, while the run takes away for good the jobs that outlived the retention. The measure is the retention days in the automation settings.

The restore warning arises when the automation notices it has been silent for a long while. That usually means the database came back from an older backup, and the queue may hold work that will run again or that is long past.

Reference

Previewing the Clean-Up

get/api/v1/admin/automation/cleanup/preview
Automation/PreviewAutomationCleanup admin it removes nothing

Shows how many jobs the retention rules would remove.

Response fields data — 4
completedintHow many completed jobs would go.
cancelledintHow many cancelled jobs would go.
failedintHow many failed jobs would go.
retentionobjectThe retention days the counts rest on.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/automation/cleanup/preview' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch('https://panel.example.com/api/v1/admin/automation/cleanup/preview', {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const { data } = await res.json();

const willGo = data.completed + data.cancelled + data.failed;
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/cleanup/preview');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// This endpoint COUNTS rather than removes; always look here before a clean-up.
$plan = Api::Automation()->PreviewAutomationCleanup()['data'];
if ($plan['completed'] + $plan['cancelled'] + $plan['failed'] > 0)
    Api::Automation()->RunAutomationCleanup();

Running the Clean-Up

post/api/v1/admin/automation/cleanup/run
Automation/RunAutomationCleanup admin a permanent delete

Removes for good the jobs that outlived the retention.

Body —
——No body is needed, send an empty one. The retention comes from the automation settings, not from the call, so there is no field to widen or narrow the sweep with.
Response fields data — 2
deletedintHow many jobs were removed.
breakdownobjectThe breakdown by status: completed, cancelled and failed.
Errors 2
blocked_by_gate422A hook refused the clean-up.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/cleanup/run' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/cleanup/run', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const { data } = await res.json();
console.log(data.deleted, data.breakdown);
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/cleanup/run');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// What goes never returns: the error messages on failed jobs go along with them.
$r = Api::Automation()->RunAutomationCleanup()['data'];

Reading the Table Sizes

get/api/v1/admin/automation/table-stats
Automation/GetAutomationTableStats admin

Returns the row count and the average data size of the three queue tables.

Response fields data — 3
cronjob_queueobjectThe scheduled-task queue: its row count and average data size.
module_queueobjectThe module queue: its row count and average data size.
notification_queueobjectThe notice queue: its row count and average data size.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/automation/table-stats' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch('https://panel.example.com/api/v1/admin/automation/table-stats', {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/table-stats');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Growth is the row count TIMES the average size rather than the count; read both.
$t = Api::Automation()->GetAutomationTableStats()['data'];
$approx = $t['cronjob_queue']['rows'] * $t['cronjob_queue']['avg_payload'];

Dismissing the Restore Warning

post/api/v1/admin/automation/restore/dismiss
Automation/DismissAutomationRestore admin

Takes away the restore-suspicion warning.

Body —
——No body is needed, send an empty one. The endpoint takes nothing: it clears the suspicion marks kept in the settings.
Response fields data — 1
dismissedboolWhether the warning was taken away.
Errors 2
config_write_failed422The settings file could not be written.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/restore/dismiss' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/restore/dismiss', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/restore/dismiss');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Dismissing does not remove the REASON; find out where the silence came from first.
$s = Api::Automation()->GetAutomationSettings()['data'];
if ($s['restore']['suspected_at'] !== null)
    Api::Automation()->DismissAutomationRestore();

Starting a Restore Reconciliation

post/api/v1/admin/automation/restore/trigger
Automation/TriggerAutomationRestore admin the panel asks for a password

Marks the queue for reconciliation after a restore.

Body —
——No body is needed, send an empty one. The endpoint takes nothing: the suspicion time it writes is the moment of the call.
Response fields data — 1
triggeredboolWhether the mark was set.
Errors 2
config_write_failed422The settings file could not be written.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/restore/trigger' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/restore/trigger', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/restore/trigger');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// In the panel this asks for the ADMIN PASSWORD; over the API the key's scope is the only gate.
Api::Automation()->TriggerAutomationRestore();

Pitfalls

Count first, remove after

The clean-up run cannot be undone and does not say in advance how many records will go. The preview exists for exactly that: it counts by the same measure and touches nothing. After changing the retention days, running the clean-up without looking can take far more than you expected.

A removed job takes its error with it

The clean-up removes failed jobs too, and their error messages go with them. Running it while looking into a problem can take away the very evidence you need. Keeping the retention on failed jobs longer than on completed ones is worth doing for that reason.

The row count alone does not tell the size

The table figures give both the row count and the average data size. A queue with few rows carrying large data can take more room than a busy one carrying little. When watching growth, multiply the two; counting rows alone misleads.

Dismissing the warning leaves the cause

Dismissing the restore warning only clears the mark and changes nothing about why the queue fell silent. When the warning really follows a restore, work long past sits in the queue and runs all at once when things resume. Look at the dates on the recent jobs before dismissing it.

The panel's password gate is not here

Starting a restore reconciliation asks for the administrator password in the panel, while here the key's scope is the only gate. A key carrying that scope can do outright what the panel guards with a second check. Weigh that when handing keys out.

Was this helpful?

Thanks for your feedback!

Still Need Help?

Our support team is here around the clock for anything you can't find above.

Contact Us Open a Ticket

We use cookies to keep you signed in, remember your language and currency, and improve the site. Cookie Policy