Queue Maintenance

2.3k visitas Markdown

The six endpoints that clear the queues, measure them and handle the restore warning.

Overview

Queues grow over time. These six endpoints deal with maintenance: clearing old jobs, seeing how much room the tables take, and handling the warning that appears when the database is restored.

The clean-up comes in two steps. The preview counts and removes nothing, while the run takes away for good the jobs that outlived the retention. The measure is the retention days in the automation settings.

The restore warning arises when the automation has been silent for a long while. A second trace kept outside the database then shows whether the database came back from an older backup. When it does, or when the two cannot be told apart, the waiting queue jobs are held instead of run. The scheduled tasks are postponed too, so nothing is repeated. A plain outage, where the trace stops together with the database, raises no warning.

Reference

Previewing the Clean-Up

get/api/v1/admin/automation/cleanup/preview
Automation/PreviewAutomationCleanup admin it removes nothing

Shows how many jobs the retention rules would remove.

Response fields data — 4
completedintHow many completed jobs would go.
cancelledintHow many cancelled jobs would go.
failedintHow many failed jobs would go.
retentionobjectThe retention days the counts rest on.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/automation/cleanup/preview' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch('https://panel.example.com/api/v1/admin/automation/cleanup/preview', {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const { data } = await res.json();

const willGo = data.completed + data.cancelled + data.failed;
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/cleanup/preview');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// This endpoint COUNTS rather than removes; always look here before a clean-up.
$plan = Api::Automation()->PreviewAutomationCleanup()['data'];
if ($plan['completed'] + $plan['cancelled'] + $plan['failed'] > 0)
    Api::Automation()->RunAutomationCleanup();

Running the Clean-Up

post/api/v1/admin/automation/cleanup/run
Automation/RunAutomationCleanup admin a permanent delete

Removes for good the jobs that outlived the retention.

Body —
——No body is needed, send an empty one. The retention comes from the automation settings, not from the call, so there is no field to widen or narrow the sweep with.
Response fields data — 2
deletedintHow many jobs were removed.
breakdownobjectThe breakdown by status: completed, cancelled and failed.
Errors 2
blocked_by_gate422A hook refused the clean-up.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/cleanup/run' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/cleanup/run', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const { data } = await res.json();
console.log(data.deleted, data.breakdown);
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/cleanup/run');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// What goes never returns: the error messages on failed jobs go along with them.
$r = Api::Automation()->RunAutomationCleanup()['data'];

Reading the Table Sizes

get/api/v1/admin/automation/table-stats
Automation/GetAutomationTableStats admin

Returns the row count and the average data size of the three queue tables.

Response fields data — 3
cronjob_queueobjectThe scheduled-task queue: its row count and average data size.
module_queueobjectThe module queue: its row count and average data size.
notification_queueobjectThe notice queue: its row count and average data size.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/automation/table-stats' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch('https://panel.example.com/api/v1/admin/automation/table-stats', {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/table-stats');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Growth is the row count TIMES the average size rather than the count; read both.
$t = Api::Automation()->GetAutomationTableStats()['data'];
$approx = $t['cronjob_queue']['rows'] * $t['cronjob_queue']['avg_payload'];

Dismissing the Restore Warning

post/api/v1/admin/automation/restore/dismiss
Automation/DismissAutomationRestore admin

Marks the restore suspicion as a false alarm. The warning goes, the held queue jobs are released and the postponed tasks return to their previous run times.

Body —
——No body is needed, send an empty one. The endpoint takes nothing: it clears the suspicion marks, the queue hold and the task postponement.
Response fields data — 3
dismissedboolWhether the warning was taken away.
releasedobjectEach held queue (cronjob_queue, module_queue, notification_queue) with the number of its held jobs still waiting at release time. Empty when nothing was held.
resumed_tasksintHow many postponed tasks went back to their previous run time. A task run by hand since the warning keeps its new time and is not counted.
Errors 2
config_write_failed422The settings file could not be written.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/restore/dismiss' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/restore/dismiss', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/restore/dismiss');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Dismissing releases the held jobs and the postponed tasks; check the restored data first.
$s = Api::Automation()->GetAutomationSettings()['data'];
if ($s['restore']['suspected_at'] !== null)
    Api::Automation()->DismissAutomationRestore();

Releasing the Held Jobs

post/api/v1/admin/automation/restore/release
Automation/ReleaseAutomationRestore admin

Lets cron take the queue jobs held by a suspected restore. The warning and the task postponement stay.

Body —
——No body is needed, send an empty one. The endpoint takes nothing: it lifts the queue hold only.
Response fields data — 1
releasedobjectEach held queue (cronjob_queue, module_queue, notification_queue) with the number of its held jobs still waiting at release time. Empty when nothing was held. The next cron run processes them.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/restore/release' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/restore/release', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/restore/release');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Release only once the restored data has been checked; the postponement stays until the warning is dismissed.
$s = Api::Automation()->GetAutomationSettings()['data'];
if (array_sum($s['restore']['frozen']) > 0)
    Api::Automation()->ReleaseAutomationRestore();

Raising the Restore Warning

post/api/v1/admin/automation/restore/trigger
Automation/TriggerAutomationRestore admin the panel asks for a password

Raises the restore warning by hand. No queue job is held and no task is postponed.

Body —
——No body is needed, send an empty one. The endpoint takes nothing: the suspicion time it writes is the moment of the call.
Response fields data — 1
triggeredboolWhether the mark was set.
Errors 2
config_write_failed422The settings file could not be written.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/restore/trigger' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/restore/trigger', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/restore/trigger');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// In the panel this asks for the ADMIN PASSWORD; over the API the key's scope is the only gate.
Api::Automation()->TriggerAutomationRestore();

Pitfalls

Count first, remove after

The clean-up run cannot be undone and does not say in advance how many records will go. The preview exists for exactly that: it counts by the same measure and touches nothing. After changing the retention days, running the clean-up without looking can take far more than you expected.

A removed job takes its error with it

The clean-up removes failed jobs too, and their error messages go with them. Running it while looking into a problem can take away the very evidence you need. Keeping the retention on failed jobs longer than on completed ones is worth doing for that reason.

The row count alone does not tell the size

The table figures give both the row count and the average data size. A queue with few rows carrying large data can take more room than a busy one carrying little. When watching growth, multiply the two; counting rows alone misleads.

Dismissing the warning releases everything

Dismissing the restore warning treats it as a false alarm. In the same call the held jobs are released and the postponed tasks go back to their previous run times. When the database really came back from an older backup, check the restored data first. Or call the release endpoint alone: it lifts the hold and keeps the postponement.

The panel's password gate is not here

Raising the restore warning asks for the administrator password in the panel, while here the key's scope is the only gate. A key carrying that scope can do outright what the panel guards with a second check. Weigh that when handing keys out.

¿Le ha resultado útil?

¡Gracias por sus comentarios!

¿Todavía necesita ayuda?

Nuestro equipo de soporte está disponible las 24 horas para ayudarle con lo que no encuentre aquí.

Contactar Abrir una solicitud

Usamos cookies para mantener su sesión iniciada, recordar su idioma y moneda y mejorar el sitio. Política de cookies