# Writing a Storage Module

https://dev.wisecp.com/es/writing-a-storage-module

Add a backup destination by implementing six transfer methods. The operator can register as many accounts on it as they like.

## Overview

A Storage module is where backup archives go. It is not a global choice: an operator registers destinations as rows. Credentials do not live in `config.php`; they arrive in the constructor, per destination, decrypted.

There are two base classes. `StorageModule` connects with credentials the operator typed; `CloudStorageModule` adds the OAuth authorization code flow.

Six modules ship. Failure is a thrown `StorageException`; success returns nothing.

## Prerequisites

- A destination that can store a large file, read it back, list a directory and report a byte size.
- For a cloud provider, a client id and secret plus the redirect address.
- [Module Anatomy](https://dev.wisecp.com/en/module-anatomy) and [Module Configuration](https://dev.wisecp.com/en/module-configuration).

## Structure

```bash
coremio/modules/Storage/AcmeVault/
├── AcmeVault.php     class AcmeVault extends StorageModule       (or CloudStorageModule)
├── config.php        ['meta' => [...], 'defaults' => [...]]      no credentials here
└── lang/
    ├── en.php
    └── tr.php
```

- **StorageModule**: The plain base: five abstract transfer methods plus overridable defaults.
- **CloudStorageModule**: Adds three abstract OAuth methods, a callback handler and state helpers.
- **StorageException**: The failure channel. Narrowed by `StorageConnectionException` and `StorageAuthException`.
- **Backup::buildStorage()**: Destination row to instance.
- **Backup::decodeStorageConfig()**: Decrypts the fields `encryptedFields()` named; `encodeStorageConfig()` encrypts on save.
- **coremio/modules/Storage**: Your folder goes here.

## Walkthrough

### 1. Pick the Base Class

1. Credentials the operator types: extend `StorageModule`, like FTP.
2. The operator signs in and you hold a refresh token: extend `CloudStorageModule`, like GoogleDrive.
3. The destinations screen shows the connect button only for a cloud subclass.

### 2. Declare Form and Secrets

1. Implement `configuration()`. Each field name becomes a key of the config array you receive later.
2. Implement the static `encryptedFields()` and name every secret. A forgotten field is stored in clear text.
3. Put non-secret defaults in `config.php` under `defaults`; the row is merged on top.

### 3. The Transfer Methods

1. Write `test()` so it proves writability: upload a probe, read its size, delete it.
2. Write `upload()`, `download()`, `delete()`, `list()`, `ensureDirectory()`. All return void, throw on failure.
3. Override `remoteSize()` if your provider has a cheap size call; the default derives it from `list()`.
4. Attach the progress callback in long transfers, or the watchdog reaps the job.

### 4. Add the OAuth Half

1. Implement `authorizationUrl()`, `exchangeCode()`, `refreshToken()` and `revokeTokens()`.
2. Do not write a callback route. The shared one validates the state, calls `exchangeCode()` and posts the result back.
3. Declare the token fields as hidden entries in `configuration()`: access token, refresh token, expiry, account.
4. Refresh lazily: check the expiry before each API call.

## Reference

### StorageModule

```php
// Config arrives per destination: defaults merged with the row's decrypted credentials.
public function __construct(array $storageConfig = []);

// ── abstract: you must implement all five ──────────────────────────────
abstract public function test(): void;
abstract public function upload(string $localPath, string $remoteName): void;
abstract public function download(string $remoteName, string $localPath): void;
abstract public function delete(string $remoteName): void;
abstract public function ensureDirectory(string $path): void;

// Each entry: ['name' => string, 'size' => int, 'mtime' => int|null]
abstract public function list(string $prefix = ''): array;

// ── virtual: sensible defaults, override when you can do better ────────
public function remoteSize(string $remoteName): ?int;                       // derived from list()
public function downloadStream(string $remoteName, $outputStream): void;    // via a temp file
public function getPresignedDownloadUrl(string $remoteName, int $ttlSeconds = 300): ?string;  // null
public static function encryptedFields(): array;                            // []
public function configuration(): array;                                     // []
public function controller_test(): array;                                   // wraps test() in try/catch

// ── inherited plumbing, do not reimplement ─────────────────────────────
public function setProgressCallback(?callable $cb): void;
public function getProgressCallback(): ?callable;
protected function attachProgressCallback(\CurlHandle $ch, ?int $intervalSec = null): void;

const DEFAULT_FOLDER_PATH   = '/wisecp-backup';
const HEARTBEAT_INTERVAL_SEC = 5;
```

- **remoteSize() returning null**: Means "the provider could not answer", not a failure. The backup is kept as unverified; zero would fail verification.
- **getPresignedDownloadUrl()**: A short-lived direct URL; the download redirects to it, so the archive never transits this installation. Null when unavailable.
- **downloadStream()**: Override when your client can stream; otherwise the archive is written to a temp file first.
- **controller_test()**: Already written: calls `test()`, returns what the form expects.

### CloudStorageModule

```php
// $state is produced by generateSignedState() and must be echoed to the provider.
abstract public function authorizationUrl(string $state): string;

// Returns ['access_token' => string, 'refresh_token' => string,
//          'expires_at' => int, 'account_email' => ?string]
// Throws StorageAuthException on failure.
abstract public function exchangeCode(string $code): array;

// Mutates $this->config['access_token'] and ['expires_at'] in place.
abstract public function refreshToken(): void;

// Default is a no-op. Override to hit the provider's revoke endpoint.
public function revokeTokens(string $refreshToken): void;

// Already implemented: validates state, calls exchangeCode(), builds the popup payload.
public function callback_handle(): array;

// HMAC-signed state that carries the provisional credentials, so the callback needs no
// session and no database lookup. Default lifetime is ten minutes.
public static function generateSignedState(array $cfg = [], int $ttlSeconds = 600): string;
public static function decodeSignedState(string $state): array|false;
```

The callback route exists at `coremio/modules/Storage/router.php`. It maps a slug onto a module name, so a new provider needs one entry:

```php
// api/system/backup/{slug}/callback
$providers = ['google-drive' => 'GoogleDrive', 'onedrive' => 'OneDrive', 'yandex-disk' => 'YandexDisk'];

$module = Modules::getInstance("Storage", $providers[$slug], [[]]);
$result = $module->callback_handle();   // {status: connected|error, access_token, refresh_token, ...}
// The result is posted back to the window that opened the popup.
```

### configuration() Descriptor

A flat list of arrays; only `type` and `name` are required.

| Key | Accepts | What it does |
| --- | --- | --- |
| `type` | `text`, `number`, `password`, `checkbox`, `select`, `hidden`, `section`, `redirect_uri`, `oauth_connect` | The control. The last three are layout: heading, address, sign-in button. |
| `name` | string | The config key, read as `$this->config['name']`. |
| `label` | string | Visible label, from `$this->lang` with a literal fallback. |
| `width` | 1 to 12 | Grid columns. |
| `required` | bool | Marks the field mandatory. |
| `encrypted` | bool | Marks a secret for the interface. It does **not** encrypt: `encryptedFields()` does, and both must agree. |
| `value`, `checked`, `placeholder`, `description` | mixed | Initial value, tick state, hint, help. |
| `step`, `doc_url`, `doc_label` | int, string, string | On a `section`: its number and a console link. |

### Where Credentials Live

- **backup_storage row**: One row per destination: name, module name as `type`, `config` as encrypted JSON.
- **the constructor merge**: Reads `meta` and `defaults`, then merges the passed config over them into `$this->config`.
- **the third factory argument**: `Modules::getInstance("Storage", $type, [$config])`. The nesting is the constructor argument list.
- **the mask**: Encrypted fields read back as five asterisks; posting that means "unchanged".
- **revalidation on save**: An update re-runs `test()` only when `folder_name`, `folder_path`, `base_path` or `remote_directory` changed.

### Upload Verification

The engine does not trust a clean return from `upload()`.

```php
$module = Backup::buildStorage($storageId);          // row -> decrypt -> getInstance

// Heartbeat: a slow upload otherwise exceeds the queue's stale window and gets reaped.
$module->setProgressCallback(function ($bytes = 0, $total = 0) use ($queueId, $backupId, $row): void {
    if ($queueId > 0) CronJobQueue::touch($queueId);
    $tot = (int) ($total > 0 ? $total : ($row['file_size'] ?? 0));
    $pct = $tot > 0 ? (int) min(100, round(((int) $bytes) * 100 / $tot)) : 0;
    Backup::updateProgress($backupId, 'upload', $pct, (int) $bytes, $tot);
});

$module->upload($finalArchive, $finalName);

// Then: prove the bytes landed. A clean return is not proof; an interrupted transfer can
// leave a correctly named but truncated file, and that is discovered only when someone
// needs the backup. null means "could not answer" and is recorded as unverified, not failed.
$remoteSize = $module->remoteSize($finalName);
if ($remoteSize !== null && $remoteSize !== $localSize)
    throw new Exception("Upload verification failed: the destination holds {$remoteSize} bytes.");
```

## Example

A plain module, then the two sides that read it back.

```php
<?php
namespace WISECP\Modules\Storage;

use StorageModule;
use StorageException;
use StorageConnectionException;
use Utility;

class AcmeVault extends StorageModule
{
    // Named here, encrypted outside the class on the way into the destination row.
    // A secret missing from this list is stored in clear text.
    public static function encryptedFields(): array
    {
        return ['api_key'];
    }

    public function configuration(): array
    {
        return [
            ['type' => 'text',     'name' => 'bucket',  'width' => 8, 'label' => $this->lang['field-bucket'] ?? 'Bucket', 'required' => true],
            ['type' => 'text',     'name' => 'region',  'width' => 4, 'label' => $this->lang['field-region'] ?? 'Region', 'value' => 'eu-central'],
            ['type' => 'password', 'name' => 'api_key', 'width' => 12, 'label' => $this->lang['field-api-key'] ?? 'API Key', 'required' => true, 'encrypted' => true],

            // Naming a directory field one of the path-like names makes an edit to it
            // re-run test() on save. Anything else is saved without revalidating.
            ['type' => 'text', 'name' => 'remote_directory', 'width' => 12,
             'label' => $this->lang['field-remote-directory'] ?? 'Remote directory',
             'value' => self::DEFAULT_FOLDER_PATH, 'placeholder' => self::DEFAULT_FOLDER_PATH],
        ];
    }

    public function test(): void
    {
        $this->ensureDirectory('');

        // Reachability is not writability. Write a probe, read its size back, remove it.
        $name  = '.wisecp-storage-test-' . bin2hex(random_bytes(4));
        $probe = 'wisecp';
        $tmp   = tempnam(sys_get_temp_dir(), 'wcp-acme-');
        file_put_contents($tmp, $probe);

        try {
            $this->upload($tmp, $name);

            $size = $this->remoteSize($name);
            $this->delete($name);

            // null means the provider could not answer, which proves nothing either way.
            if ($size !== null && $size !== strlen($probe))
                throw new StorageException(sprintf(
                    $this->lang['error-upload-incomplete'] ?? 'Upload incomplete: the server stored %s of %s bytes',
                    $size, strlen($probe)
                ));
        }
        finally {
            @unlink($tmp);
        }
    }

    public function upload(string $localPath, string $remoteName): void
    {
        if (!is_file($localPath)) throw new StorageException("Local file not found: {$localPath}");

        $handle = fopen($localPath, 'rb');
        if (!$handle) throw new StorageException("Local file not readable: {$localPath}");

        $ch = curl_init($this->endpoint($remoteName));
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_UPLOAD         => true,
            CURLOPT_INFILE         => $handle,
            CURLOPT_INFILESIZE     => filesize($localPath),
            CURLOPT_HTTPHEADER     => $this->headers(),
        ]);

        // Without this the watchdog sees no activity and can reap a slow upload as stale.
        $this->attachProgressCallback($ch);

        $body = curl_exec($ch);
        $code = (int) curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $err  = curl_error($ch);
        curl_close($ch);
        fclose($handle);

        if ($err !== '')            throw new StorageConnectionException($err);
        if ($code < 200 || $code > 299) throw new StorageException("Upload failed (HTTP {$code}): " . (string) $body);
    }

    public function download(string $remoteName, string $localPath): void
    {
        $out = fopen($localPath, 'wb');
        if (!$out) throw new StorageException("Cannot write to {$localPath}");

        try     { $this->downloadStream($remoteName, $out); }
        finally { fclose($out); }
    }

    public function delete(string $remoteName): void
    {
        $this->request('DELETE', $remoteName);
    }

    public function list(string $prefix = ''): array
    {
        $data = Utility::jdecode($this->request('GET', $prefix), true);

        $out = [];
        foreach (($data['objects'] ?? []) as $item)
            $out[] = [
                'name'  => (string) ($item['key'] ?? ''),
                'size'  => (int) ($item['bytes'] ?? 0),
                'mtime' => isset($item['modified']) ? (int) strtotime((string) $item['modified']) : null,
            ];

        return $out;
    }

    // Cheaper than the inherited version, which reads a whole directory listing.
    public function remoteSize(string $remoteName): ?int
    {
        try     { $head = Utility::jdecode($this->request('HEAD', $remoteName), true); }
        catch   (\Throwable $e) { return null; }

        return isset($head['bytes']) ? (int) $head['bytes'] : null;
    }

    public function ensureDirectory(string $path): void
    {
        $this->request('MKCOL', $path);
    }

    private function endpoint(string $name): string
    {
        $base = trim((string) ($this->config['remote_directory'] ?? self::DEFAULT_FOLDER_PATH), '/');

        return 'https://' . rawurlencode((string) ($this->config['region'] ?? '')) . '.acmevault.example/'
             . rawurlencode((string) ($this->config['bucket'] ?? '')) . '/' . trim($base . '/' . $name, '/');
    }

    private function headers(): array
    {
        // The key arrives already decrypted: decoding happened when the row was read.
        return ['X-Api-Key: ' . (string) ($this->config['api_key'] ?? '')];
    }

    private function request(string $method, string $name): string
    {
        $ch = curl_init($this->endpoint($name));
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_CUSTOMREQUEST  => $method,
            CURLOPT_HTTPHEADER     => $this->headers(),
            CURLOPT_TIMEOUT        => 30,
        ]);
        $body = curl_exec($ch);
        $code = (int) curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $err  = curl_error($ch);
        curl_close($ch);

        if ($err !== '')            throw new StorageConnectionException($err);
        if ($code < 200 || $code > 299) throw new StorageException("{$method} failed (HTTP {$code})");

        return (string) $body;
    }
}
```

```php
<?php
return [
    'meta' => [
        'name'        => 'AcmeVault',
        'version'     => '1.0',
        'description' => 'Object storage over HTTPS',
        'icon'        => 'bi bi-hdd-rack',
    ],
    // Merged UNDER the destination row, so a row value always wins. Credentials never
    // appear here: they belong to the destination, not to the module.
    'defaults' => [
        'region'           => 'eu-central',
        'remote_directory' => '',
    ],
];
```

```php
// 1. The destinations screen builds the provider list and your form from the module itself.
$module = Modules::getInstance("Storage", $name, [[]]);   // empty config: metadata only
$entry  = [
    'key'         => $name,
    'name'        => $module->lang['name'] ?? $name,
    'description' => $module->lang['description'] ?? ($module->meta['description'] ?? ''),
    'oauth'       => is_subclass_of("WISECP\\Modules\\Storage\\{$name}", "CloudStorageModule"),
    'fields'      => $module->configuration(),
];

// 2. Saving encrypts exactly the fields encryptedFields() named, then stores the JSON.
WDB::insert("backup_storage", [
    'name'   => $displayName,
    'type'   => $name,
    'config' => Backup::encodeStorageConfig($name, $config),
    'status' => 'enabled',
]);

// 3. Running a backup goes the other way: read the row, decrypt, instantiate, transfer.
$module = Backup::buildStorage($storageId);
$module->upload($finalArchive, $finalName);
```

## Pitfalls

> **A secret missing from encryptedFields() is readable**
> 
> The `encrypted` flag on a form field only styles the input; encryption is driven by the static list. The failure is silent: the value sits readable in the database.

> **Never report zero when you cannot measure**
> 
> `remoteSize()` separates "could not answer" (null) from "the file is empty" (zero). Zero for an unanswerable check discards a good archive.

> **A silent upload is reaped as stale**
> 
> Uploads run inside a queued job with a stale window that a large transfer can outlast. Without the progress callback the job is killed mid-flight.

> **The factory nests the config one level deeper**
> 
> The third argument of `Modules::getInstance("Storage", $type, [$config])` is the constructor's argument list; an unwrapped config spreads across parameters.

> **Revoke on disconnect**
> 
> Deleting a cloud destination calls `revokeTokens()` first, best effort: the row is removed either way. Without it a removed destination leaves a live refresh token.

## Related Articles

- [Module Anatomy](https://dev.wisecp.com/en/module-anatomy)
- [Module Configuration](https://dev.wisecp.com/en/module-configuration)
- [Adding a Scheduled Task](https://dev.wisecp.com/en/adding-a-scheduled-task)
- [Error Handling](https://dev.wisecp.com/en/error-handling)
- [Writing a Social Login Provider](https://dev.wisecp.com/en/writing-a-social-login-provider)
- [Security Practices](https://dev.wisecp.com/en/security-practices)