Automation Jobs

7 views Markdown

The ten endpoints that watch the jobs in three queues and handle them one by one or in bulk.

Overview

The automation carries three queues: the scheduled tasks, the module calls and the notices. In each, job rows wait, run, complete or fail, and these ten endpoints work with those rows.

The dashboard gives everything in one call: the worker's health, the queue counts, the task list, the recent jobs and the last twenty-four hours as a chart. Reach for it when writing monitoring rather than making five calls.

Four endpoints touch one job (retry, cancel, clear the lock, delete), and three reach the whole queue. Those last three move hundreds of jobs with one call and cannot be undone.

Reference

Reading the Dashboard

get/api/v1/admin/automation/dashboard
Automation/GetAutomationDashboard admin

Returns the whole state of the automation in one call.

Query 3
chart_rangestringThe span the chart covers.
chart_stagestringLimits the chart to one stage.
chart_taskstringNarrows the chart to a single task.
Response fields data — 5
statusobjectThe worker as it stands: its health, its last run, how long ago that was, how long to the next, which task is next and how many jobs wait.
statsobjectThe queue and system counts: per queue the waiting, running, completed today and failed; across the system the open jobs, the waiting jobs, the longest wait and the gap between runs.
tasksarrayThe task list: its name, its shown name, its frequency, whether it is off, how many jobs wait and run, how many completed and failed today, and its last error.
jobsobjectThe last fifty job rows, the total job count and the state of the restore suspicion.
chartarrayThe last twenty-four hours broken down by type into successes and failures.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/automation/dashboard' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch('https://panel.example.com/api/v1/admin/automation/dashboard', {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const { data } = await res.json();

if (data.status.status !== 'ok') console.warn('worker', data.status.status);
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/dashboard');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// ONE call is enough for monitoring; the dashboard carries what five endpoints would.
$d = Api::Automation()->GetAutomationDashboard()['data'];
$healthy = $d['status']['status'] === 'ok';

Listing the Jobs

get/api/v1/admin/automation/jobs
Automation/GetAutomationJobs admin cursor paging

Returns the queue feed, newest first.

Query 2
limitintHow many rows to return. Clamped between ten and two hundred.
before_idintFetches rows older than this id. This is how you page backwards.
Response fields data — 4
jobsarrayThe job rows: id, type, title, status, label, time, attempt count and estimated start.
totalintHow many rows the queue holds. It comes back on the first feed alone.
restore_activeboolWhether a restore is suspected. It comes back on the first feed alone.
has_moreboolWhether older rows remain. It comes back on a cursor call alone.
Errors 1
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/automation/jobs?limit=50' \
  -H "Authorization: Bearer $API_KEY"
const url = new URL('https://panel.example.com/api/v1/admin/automation/jobs');
url.searchParams.set('limit', '50');

const res  = await fetch(url, {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const { data } = await res.json();

// Daha eskisi: url.searchParams.set('before_id', data.jobs.at(-1).id)
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs?' . http_build_query(['limit' => 50]));
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The total and the restore mark come on the FIRST call alone; a cursor call carries neither.
$first = Api::Automation()->GetAutomationJobs()['data'];
$older = Api::Automation()->GetAutomationJobs([], [
    'before_id' => end($first['jobs'])['id'],
]);

Reading One Job

get/api/v1/admin/automation/jobs/{queue}/{id}
Automation/GetAutomationJob admin

Returns a job's whole row with its payload and logs.

Path 2
queuestringWhich queue: cron, module, notification.
idintThe job id.
Response fields data
idintThe job id.
typestringThe job type.
statusstringThe job status.
attemptsintHow many times it was tried.
created_atstringWhen it joined the queue.
completed_atstringWhen it finished.
payloadobjectThe data given to the job. It arrives decoded.
process_logsobjectThe logs kept while it ran.
result_payloadobjectWhat the job left behind.
result_summaryobjectA readable summary of the result. It comes back on the scheduled-task queue.
Errors 4
invalid_queue422The queue key is not recognised.
invalid_id422The job id is not valid.
not_found404No such job.
insufficient_scope403The key lacks the required scope.
Request 
curl 'https://panel.example.com/api/v1/admin/automation/jobs/cron/9001' \
  -H "Authorization: Bearer $API_KEY"
const res  = await fetch(`https://panel.example.com/api/v1/admin/automation/jobs/cron/${id}`, {
  headers: { Authorization: `Bearer ${apiKey}` },
});
const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/' . $id);
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The fields differ BY QUEUE; read past the common ones without assuming they are there.
$job = Api::Automation()->GetAutomationJob(['queue' => 'cron', 'id' => $id])['data'];
$logs = $job['process_logs'] ?? [];

Retrying a Job

post/api/v1/admin/automation/jobs/{queue}/{id}/retry
Automation/RetryAutomationJob admin

Puts a job back among the waiting.

Body —
——No body is needed. The queue and the id in the path name the job; send an empty body.
Response fields data — 3
retriedboolWhether the call ran.
queuestringThe job's queue.
idintThe job id.
Errors 3
invalid_queue422The queue key is not recognised.
invalid_id422The job id is not valid.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/jobs/cron/9001/retry' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch(`https://panel.example.com/api/v1/admin/automation/jobs/cron/${id}/retry`, {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/' . $id . '/retry');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The attempt counter is CLEARED: a job that fails again looks as though it started afresh.
Api::Automation()->RetryAutomationJob(['queue' => 'cron', 'id' => $id]);

Cancelling a Job

post/api/v1/admin/automation/jobs/{queue}/{id}/cancel
Automation/CancelAutomationJob admin

Marks a job cancelled.

Body —
——No body is needed. The queue and the id in the path name the job; send an empty body.
Response fields data — 3
cancelledboolWhether the call ran.
queuestringThe job's queue.
idintThe job id.
Errors 3
invalid_queue422The queue key is not recognised.
invalid_id422The job id is not valid.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/jobs/cron/9001/cancel' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch(`https://panel.example.com/api/v1/admin/automation/jobs/cron/${id}/cancel`, {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/' . $id . '/cancel');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Cancelling does not stop a job ALREADY RUNNING; it changes the queue row's status.
Api::Automation()->CancelAutomationJob(['queue' => 'cron', 'id' => $id]);

Clearing a Stuck Lock

post/api/v1/admin/automation/jobs/{queue}/{id}/reclaim
Automation/ReclaimAutomationJob admin risks a double run

Clears the lock a crashed worker left on a job.

Body —
——No body is needed. The queue and the id in the path name the job; send an empty body.
Response fields data — 3
reclaimedboolWhether the call ran.
queuestringThe job's queue.
idintThe job id.
Errors 3
invalid_queue422The queue key is not recognised.
invalid_id422The job id is not valid.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/jobs/cron/9001/reclaim' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch(`https://panel.example.com/api/v1/admin/automation/jobs/cron/${id}/reclaim`, {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/' . $id . '/reclaim');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// If the worker IS STILL ALIVE the job runs twice; check the longest wait first.
$d = Api::Automation()->GetAutomationDashboard()['data'];
if ($d['status']['status'] === 'dead')
    Api::Automation()->ReclaimAutomationJob(['queue' => 'cron', 'id' => $id]);

Deleting a Job

delete/api/v1/admin/automation/jobs/{queue}/{id}
Automation/DeleteAutomationJob admin

Removes a job row from the queue.

Response fields data — 3
deletedboolWhether the delete ran.
queuestringThe job's queue.
idintThe id of the job removed.
Errors 3
invalid_queue422The queue key is not recognised.
invalid_id422The job id is not valid.
insufficient_scope403The key lacks the required scope.
Request 
curl -X DELETE 'https://panel.example.com/api/v1/admin/automation/jobs/cron/9001' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch(`https://panel.example.com/api/v1/admin/automation/jobs/cron/${id}`, {
  method: 'DELETE',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/' . $id);
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST  => 'DELETE',
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Cancel rather than delete: the record stays and the job is never picked up again.
Api::Automation()->CancelAutomationJob(['queue' => 'cron', 'id' => $id]);

Deleting Many Jobs

post/api/v1/admin/automation/jobs/{queue}/bulk-delete
Automation/BulkDeleteAutomationJobs admin

Removes the job ids you give in one call.

Body 1
idsint[]reqThe job ids to remove. Zero and below are dropped, and repeats are folded.
Response fields data — 3
deletedboolWhether the call ran.
queuestringThe queue the jobs were in.
countintHow many were removed.
Errors 3
invalid_queue422The queue key is not recognised.
ids_required422No job id was given.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/jobs/cron/bulk-delete' \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"ids":[9001,9002,9003]}'
const res = await fetch('https://panel.example.com/api/v1/admin/automation/jobs/cron/bulk-delete', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${apiKey}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ ids: [9001, 9002, 9003] }),
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/bulk-delete');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => [
        'Authorization: Bearer ' . $apiKey,
        'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS => json_encode(['ids' => [9001, 9002, 9003]]),
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The count can come back BELOW what you sent: ids from another queue are not removed.
$n = Api::Automation()->BulkDeleteAutomationJobs([
    'queue' => 'cron', 'ids' => $ids,
])['data']['count'];

Retrying Everything That Failed

post/api/v1/admin/automation/jobs/{queue}/retry-all-failed
Automation/RetryAllFailedAutomationJobs admin the whole queue

Puts every failed job in a queue back among the waiting.

Body —
——No body is needed. The queue comes from the path and the set cannot be narrowed: every failed job in it is taken. Send an empty body.
Response fields data — 2
retriedboolWhether the call ran.
queuestringThe queue worked on. How many jobs it touched is not returned.
Errors 2
invalid_queue422The queue key is not recognised.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/jobs/cron/retry-all-failed' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/jobs/cron/retry-all-failed', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/retry-all-failed');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// Hundreds that failed FOR ONE REASON all run again at once and fail again.
// Read the reason from one job's detail first, then make this call.
Api::Automation()->RetryAllFailedAutomationJobs(['queue' => 'cron']);

Cancelling Everything Waiting

post/api/v1/admin/automation/jobs/{queue}/cancel-all-pending
Automation/CancelAllPendingAutomationJobs admin the whole queue

Cancels every job waiting in a queue.

Body —
——No body is needed. The queue comes from the path and the set cannot be narrowed: every pending job in it is cancelled. Send an empty body.
Response fields data — 2
cancelledboolWhether the call ran.
queuestringThe queue worked on. How many were cancelled is not returned.
Errors 2
invalid_queue422The queue key is not recognised.
insufficient_scope403The key lacks the required scope.
Request 
curl -X POST 'https://panel.example.com/api/v1/admin/automation/jobs/cron/cancel-all-pending' \
  -H "Authorization: Bearer $API_KEY"
const res = await fetch('https://panel.example.com/api/v1/admin/automation/jobs/cron/cancel-all-pending', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}` },
});

const body = await res.json();
$ch = curl_init('https://panel.example.com/api/v1/admin/automation/jobs/cron/cancel-all-pending');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST           => true,
    CURLOPT_HTTPHEADER     => ['Authorization: Bearer ' . $apiKey],
]);

$body = json_decode(curl_exec($ch), true);
curl_close($ch);
// The waiting jobs are INVOICES, NOTICES and provider calls; they all drop at once.
Api::Automation()->CancelAllPendingAutomationJobs(['queue' => 'cron']);

Pitfalls

The queue-wide endpoints cannot be undone

Retrying everything failed and cancelling everything waiting move hundreds of jobs in one call, and neither says which. Cancelling the waiting drops invoices not yet issued, notices not yet sent and provider calls not yet made, all at once. Look at the feed first to see what is there.

Clearing a lock can run a job twice

The reclaim endpoint assumes the worker died. If it is still alive the job gets picked up a second time and the same invoice can be issued twice. Read the worker health on the dashboard before using it; on a live worker a job that looks stuck is usually only slow.

Cancelling does not stop a running job

The cancel endpoint changes the queue row's status and does not kill a running process. Cancelling a job in flight does not cut it short; when it ends it writes its own result. When something truly has to stop, switching the cron off is the only way.

A retry clears the attempt counter

Retrying a job clears its attempt counter. A job that failed three times looks as though it is running for the first time, and failing again starts the count at one. A check that sifts troubled jobs by their attempt count will miss the difference.

A bulk delete is bound to its queue

A bulk delete looks only in the queue named in the path, and ids from another are passed over quietly. A count below what you sent is no error; those ids live elsewhere. When removing a mixed list, group the ids by queue and make separate calls.

Was this helpful?

Thanks for your feedback!

Still Need Help?

Our support team is here around the clock for anything you can't find above.

Contact Us Open a Ticket

We use cookies to keep you signed in, remember your language and currency, and improve the site. Cookie Policy