Writing an Import Module

3 views Markdown

Move another billing platform into this one, in resumable chunks, by writing a class the migration wizard drives step by step.

Overview

An Imports module is a one-way migration: it reads someone else's database and writes this installation's tables. Three modules ship: WHMCS, Blesta and WISECP.

There is no base class and no interface. The wizard instantiates your class by name, assigns four properties, then calls methods it probes for.

The chunk loop matters most. A migration cannot finish inside one request, so the wizard calls one data type at a time and your module returns a row count and a done flag. The id map is persisted between calls.

Prerequisites

  • Read access to the source database: host, database, user, password.
  • The source's encryption key, if it stores secrets encrypted.
  • Domain Helpers: an import writes through them, not with raw inserts.
  • A restore point; the wizard offers to queue one.

Structure

file layout 
coremio/modules/Imports/AcmeBill/
├── AcmeBill.php      class AcmeBill  in namespace WISECP\Modules\Imports
├── hooks.php         optional; only if imported rows need runtime help afterwards
├── lang/
│   ├── en.php        name + area-* (the platform card) + one -desc per data type
│   └── tr.php
└── pages/
    └── index.php     the connection form, rendered inside the wizard's platform card
AdminTools::import() The operation every step posts to. Resolves your class, assigns properties, calls the step method.
Modules::Load() Discovery: the platform name, or All for the platform list.
Modules::getPage() Puts pages/index.php in your platform card, called from area().
Config::setd() Where the id map is persisted, keyed by platform name plus a connection token.
coremio/modules/Imports Your folder goes here.

Walkthrough

1. Scaffold the Platform

  1. Create coremio/modules/Imports/AcmeBill/AcmeBill.php in the WISECP\Modules\Imports namespace.
  2. Declare the four properties the wizard assigns.
  3. Write lang/en.php and lang/tr.php; the area-* keys build the platform card.
  4. Add one <type>-desc key per data type.

2. Connect and Declare Types

  1. Write pages/index.php, inputs named AcmeBill[db_host]. The wizard hands the group to you as one array.
  2. Implement connect(): validate, connect, build a token from the details. That token is half the map's storage key.
  3. Filter each credential by kind; passwords use the pass-through filter.
  4. Implement data_types(): a name, a description and a required list per entry.

3. Write the Chunk Loop

  1. Implement pull_data(): once per type, it only counts rows.
  2. Implement transfer_data(): load the map, dispatch per type, save the map either way, return the row count and a done flag.
  3. In a type method, select rows above the highest mapped id, limited to the chunk size, and record every new id.
  4. Compute done by asking the source what is left after the last mapped id.

4. Awkward Parts

  1. Import prerequisites at the top of the type method: a type can be selected alone.
  2. Respect the mode: clean carries source ids and refuses a non-empty table; enrich adds alongside.
  3. Implement disconnect(), called after every step.
  4. For anything that cannot work yet, add a hooks.php; Blesta rewrites unreadable passwords on first login.

Reference

What the Wizard Calls

No interface exists. These names are probed before each call.

the contract 
// ── assigned onto the instance by the wizard, declare all four ──────────
public array  $lang;        // your lang file, already resolved to the admin's language
public string $area_link;   // the wizard's own address, for links inside your page
public string $name;        // your module name, e.g. 'AcmeBill'
public object $controller;  // the admin controller running the request

// ── read by the transfer step when the property exists ──────────────────
public array  $selected_data_types = [];   // everything selected in this run
public string $import_type         = 'enrich';   // 'enrich' | 'clean'
public int    $records_per_request = 10;         // chunk size chosen by the operator

// ── methods, each guarded by method_exists() before it is called ────────
public function area(): void;                  // renders your connection form
public function connect($info = []): void;         // $info = the posted credential group, untyped in all three shipped modules
public function disconnect(): void;                // called at the end of every step

// ['type' => ['name' => string, 'description' => string, 'required' => string[]]]
public function data_types(): array;

// ['status' => 'successful', 'count' => int, 'type' => string]
public function pull_data(string $type = ''): array;

// ['processed' => int, 'done' => bool]  - one chunk, called repeatedly
public function transfer_data(string $type): array;

Step Machine

Every action posts to one operation with a step field.

stepWhat the operation doesWhat it calls on your module
validationStores the backup preference, filters the types.connect(), data_types()
pull_dataAsks for a row count.connect(), pull_data($type)
selectValidates prerequisites, remembers mode and selection.connect() only
backup_statusRestore point, before the module loads.nothing
transferRuns a gate hook, transfers one chunk.connect(), transfer_data($type)
connect() runs on every step Except backup_status. Make it cheap: once per chunk.
the credentials come from the session Posted once, kept encrypted in the session, replayed into connect().
filter:module.import_data_types Runs over your type list at validation. Return the filtered list; an addon can add or hide a type.
gate:module.import_run Runs before each chunk with platform, type and mode. Returning a non-empty string stops the run.
action:import.completed Fires after every chunk with your result array; the return is ignored.

The Id Map

One structure, three jobs.

load, use, persist 
// Shape: ['users' => [sourceId => newId, ...], 'orders' => [...], 'language' => 'english']
// Stored under "import_{$this->name}_{$this->token}", where the token is a hash of the
// connection details - so two source databases never share one run's progress.

public function initialize_data($predefined_data = []): void
{
    if ($this->data === null) {
        $data = Config::getd("import_" . $this->name . "_" . $this->token) ?: [];
        if (!$data || !is_array($data)) $data = [];
    }
    else $data = [];

    $this->data = array_replace_recursive($predefined_data, $data);
}

public function save_data(int|array $overwrite_data = -1): void
{
    $data = $this->data ?? [];
    if ($overwrite_data !== -1) $data = $overwrite_data;

    Config::setd("import_" . $this->name . "_" . $this->token, $data);
}

// The resume point is simply the highest source id already recorded for that type.
public function last_id($array): int
{
    return $array ? max(array_keys($array)) : 0;
}
resume The next chunk starts after the highest mapped source id.
deduplicate The same filter stops a second run importing twice, which is why the map can be cleared.
resolve foreign keys A service row needs the new client id for its old one, and only this map has it.
saved even when a chunk throws The dispatcher catches, saves the map and rethrows, so a retry does not duplicate rows.

Enrich and Clean

enrich (default) Add rows alongside existing ones. New ids are local; the map remembers the pairing.
clean For an empty target: source ids carry across, so invoice numbers stay recognisable.
the clean-mode guard On the first chunk, throw if the target table already holds a row.
the guard runs only on chunk one Clean mode and an empty map. From chunk two on, the target rows are your own.

Example

The skeleton plus one data type, then the wizard side.

AcmeBill.php 
<?php
namespace WISECP\Modules\Imports;

use Config;
use Database;
use Exception;
use Filter;
use Language;
use Modules;
use User;
use Utility;
use Validation;
use WDB;

class AcmeBill
{
    // Assigned by the wizard. A typed property that is not declared here throws on write.
    public array  $lang;
    public string $area_link;
    public string $name;
    public object $controller;

    // Read by the transfer step when present.
    public array  $selected_data_types = [];
    public string $import_type         = 'enrich';
    public int    $records_per_request = 10;

    public string $token = '';
    private ?Database $db  = null;
    private ?array $data   = null;

    public function __construct()
    {
        @set_time_limit(0);
    }

    public function area(): void
    {
        echo Modules::getPage("Imports", $this->name, Filter::route($_GET["page"] ?? "index"), [
            'module' => $this,
        ]);
    }

    public function connect($info = []): void
    {
        $host = Filter::html_clear($info["db_host"] ?? '') ?: 'localhost';
        $user = Filter::html_clear($info["db_username"] ?? '');
        $name = Filter::route($info["db_name"] ?? '');

        // Pass-through on purpose: every other filter strips the characters that make
        // a password strong, and the connection then fails for a reason nobody can see.
        $pass = Filter::password($info["db_password"] ?? '');

        if (Validation::isEmpty($user)) throw new Exception(Language::gc("admin/tools/error6"));
        if (Validation::isEmpty($pass)) throw new Exception(Language::gc("admin/tools/error7"));
        if (Validation::isEmpty($name)) throw new Exception(Language::gc("admin/tools/error8"));

        try {
            $this->db = new Database("mysql", "pdo", $host, 3306, $user, $pass, null, $name, "utf8mb4", "utf8mb4_unicode_ci");
        }
        catch (Exception $e) {
            throw new Exception(Language::gc("admin/tools/error9", ['{message}' => $e->getMessage()]));
        }

        // Half of the progress key: a different source database gets a different run.
        $this->token = md5($host . "+++" . $user . "+++" . $pass . "+++" . $name);
    }

    public function disconnect(): void
    {
        $this->db->disconnect();
    }

    public function data_types(): array
    {
        return [
            'users' => [
                'name'        => Language::gc("admin/tools/import-result-users"),
                'description' => $this->lang["users-desc"] ?? '',
                'required'    => [],
            ],
            'invoices' => [
                'name'        => Language::gc("admin/tools/import-result-invoices"),
                'description' => $this->lang["invoices-desc"] ?? '',

                // Invoices carry a client id, and only the users map can translate it.
                // A type listed here that is not selected too is refused before transfer.
                'required'    => ['users'],
            ],
        ];
    }

    public function pull_data(string $type = ''): array
    {
        $table = $type === 'users' ? 'clients' : 'invoices';
        $count = $this->db->select("COUNT(id) as total")->from($table)->build();

        return [
            'status' => "successful",
            'count'  => $count ? (int) $this->db->getObject()->total : 0,
            'type'   => $type,
        ];
    }

    public function transfer_data(string $type): array
    {
        $this->initialize_data();

        try {
            $result = match ($type) {
                'users'    => $this->users(),
                'invoices' => $this->invoices(),
                default    => [],
            };
        }
        catch (Exception $e) {
            // Persist before rethrowing: rows already written keep their mapping,
            // so the retry continues instead of importing them a second time.
            $this->save_data();
            throw $e;
        }

        $this->save_data();

        return $result;
    }

    public function users(): array
    {
        $processed = 0;
        $last_id   = $this->last_id($this->data["users"] ?? []);

        $rows = $this->db->select()->from("clients");
        $rows->where("id", ">", $last_id);
        $rows->order_by("id ASC");
        $rows->limit($this->records_per_request);
        $rows = $rows->build() ? $rows->fetch_assoc() : [];

        foreach ($rows as $row) {
            $userId = (int) User::create([
                'type'          => "member",
                'status'        => ($row["status"] ?? '') === 'Active' ? "active" : "passive",
                'name'          => Filter::html_clear($row["firstname"] ?? ''),
                'surname'       => Filter::html_clear($row["lastname"] ?? ''),
                'full_name'     => Filter::html_clear(trim(($row["firstname"] ?? '') . ' ' . ($row["lastname"] ?? ''))),
                'email'         => $row["email"] ?? '',
                'creation_time' => $row["created_at"] ?? null,
            ]);
            if (!$userId) continue;

            // The map entry is the resume point, the duplicate guard and the key the
            // invoice type will use to find this client again. Write it immediately.
            $this->data["users"][(int) $row["id"]] = $userId;
            $processed++;
        }

        // Ask the source what is left rather than comparing counts: a row deleted at the
        // source mid-run would otherwise keep the loop going forever.
        $last_id = $this->last_id($this->data["users"] ?? []);
        $done    = !$this->db->select("id")->from("clients")->where("id", ">", $last_id)->limit(1)->build();

        return ['processed' => $processed, 'done' => $done];
    }

    public function invoices(): array
    {
        $processed = 0;
        $last_id   = $this->last_id($this->data["invoices"] ?? []);

        // Clean mode carries source ids across, so it needs an empty table. The check runs
        // only on the first chunk: from the second on, the rows found are the ones we wrote.
        if ($this->import_type === "clean" && $last_id === 0) {
            if (WDB::select("id")->from("invoices")->limit(1)->build())
                throw new Exception(Language::gc("admin/tools/import-type-clean-error"));
        }

        $rows = $this->db->select()->from("invoices");
        $rows->where("id", ">", $last_id);
        $rows->order_by("id ASC");
        $rows->limit($this->records_per_request);
        $rows = $rows->build() ? $rows->fetch_assoc() : [];

        foreach ($rows as $row) {
            // Foreign keys are resolved through the map, never through the source id.
            $ownerId = (int) ($this->data["users"][(int) ($row["client_id"] ?? 0)] ?? 0);
            if (!$ownerId) continue;   // owner not in this run; the prerequisite check prevents it

            $data = [
                'owner_id' => $ownerId,
                'total'    => (float) ($row["total"] ?? 0),
                'status'   => ($row["status"] ?? '') === 'Paid' ? "paid" : "unpaid",
                'cdate'    => $row["created_at"] ?? null,
            ];

            // Only clean mode preserves the source numbering.
            if ($this->import_type === "clean") $data["id"] = (int) $row["id"];

            WDB::insert("invoices", $data);
            $this->data["invoices"][(int) $row["id"]] = (int) WDB::lastID();
            $processed++;
        }

        $last_id = $this->last_id($this->data["invoices"] ?? []);
        $done    = !$this->db->select("id")->from("invoices")->where("id", ">", $last_id)->limit(1)->build();

        return ['processed' => $processed, 'done' => $done];
    }

    public function initialize_data($predefined_data = []): void
    {
        $data = $this->data === null
            ? (Config::getd("import_" . $this->name . "_" . $this->token) ?: [])
            : [];
        if (!is_array($data)) $data = [];

        $this->data = array_replace_recursive($predefined_data, $data);
    }

    public function save_data(int|array $overwrite_data = -1): void
    {
        $data = $this->data ?? [];
        if ($overwrite_data !== -1) $data = $overwrite_data;

        Config::setd("import_" . $this->name . "_" . $this->token, $data);
    }

    public function last_id($array): int
    {
        return $array ? max(array_keys($array)) : 0;
    }
}
pages/index.php 
<?php if (!defined("CORE_FOLDER")) die("403 Forbidden"); ?>
<div class="cred-stack">
    <div>
        <label for="AcmeBill_Host" class="form-label"><?php echo Language::gc("admin/tools/import-db-host"); ?></label>
        <!-- Every input is namespaced by the platform: the whole group arrives as
             one array and is handed to connect() as $info. -->
        <input id="AcmeBill_Host" type="text" class="form-control" name="AcmeBill[db_host]" value="">
    </div>
    <div>
        <label for="AcmeBill_Name" class="form-label"><?php echo Language::gc("admin/tools/import-db-name"); ?></label>
        <input id="AcmeBill_Name" type="text" class="form-control" name="AcmeBill[db_name]" value="" data-role="db-name">
    </div>
    <div>
        <label for="AcmeBill_User" class="form-label"><?php echo Language::gc("admin/tools/import-db-username"); ?></label>
        <input id="AcmeBill_User" type="text" class="form-control" name="AcmeBill[db_username]" value="">
    </div>
    <div>
        <label for="AcmeBill_Pass" class="form-label secret-input"><?php echo Language::gc("admin/tools/import-db-password"); ?></label>
        <input id="AcmeBill_Pass" type="password" class="form-control" name="AcmeBill[db_password]" value="">
    </div>
</div>
wizard side, do not copy 
// operations/AdminTools.php - how your instance is built and driven, reduced.
$load = Modules::Load("Imports", $platform);
if (!$load) throw new Exception("The platform is not supported. #1");

$instanceKey = "\\WISECP\\Modules\\Imports\\" . $platform;
if (!class_exists($instanceKey)) throw new Exception("The platform is not supported. #2");

$instance             = new $instanceKey();
$instance->lang       = $load["lang"] ?? [];
$instance->area_link  = LinkGenerator::admin("tools-1", ["imports"]);
$instance->name       = $platform;
$instance->controller = $this;

if (method_exists($instance, "connect")) $instance->connect($platform_info);

// Prerequisites are enforced at the select step, before a single row moves.
foreach ($data_Types as $selected_type)
    foreach (($data_types[$selected_type]["required"] ?? []) as $needed)
        if (!in_array($needed, $data_Types, true))
            throw new Exception(Language::gc("admin/tools/import-missing-prerequisite", [
                '{type}'   => $data_types[$selected_type]["name"] ?? $selected_type,
                '{needed}' => $data_types[$needed]["name"] ?? $needed,
            ]));

// The transfer step, once per chunk.
if (property_exists($instance, "selected_data_types")) $instance->selected_data_types = $data_types_s;
if (property_exists($instance, "import_type"))         $instance->import_type         = $import_type;
if (property_exists($instance, "records_per_request")) $instance->records_per_request = $records_per_request;

$result = $instance->transfer_data($type);       // ['processed' => int, 'done' => bool]

if (method_exists($instance, "disconnect")) $instance->disconnect();

Pitfalls

A stale id map makes a rerun import nothing

The chunk query skips everything already mapped, so a second run finds no rows and looks like it worked. Clear the map for "start over", never for "resume".

Declare every prerequisite

A type that resolves a foreign key through another type's map must name it in required. Leave it out and the map is empty at lookup time: rows land with no owner and nothing throws.

Filter each credential by its kind

One text filter over the whole group mangles the password and the encryption key, and the failure looks like wrong credentials. Use pass-through for secrets.

Do not compute done by counting

Comparing processed rows against the starting total breaks when the source changes mid-migration. Ask what remains past the last mapped id.

Write through the helpers

Clients, orders, services and invoices have side effects raw inserts skip. Import the global classes explicitly: inside the module namespace an unqualified helper name does not resolve.

Was this helpful?

Thanks for your feedback!

Still Need Help?

Our support team is here around the clock for anything you can't find above.

Contact Us Open a Ticket

We use cookies to keep you signed in, remember your language and currency, and improve the site. Cookie Policy