Historian & Trend Import

The historian is Consystence’s time-series store and the primitive that site servers sync upward into a cell. Unlike a conventional historian — one value per (tag, timestamp) — the Consystence historian carries provenance on every sample: each value records how it came to exist (measured by the plant, computed by the platform, imported from a file, produced by a simulator, and so on). This page covers reading the historian, the multi-source provenance model, the CSV trend-data import contract, and how a site server syncs its historian to the cloud.

flowchart TB
  PLC[Controller / edge<br/>measured] --> H
  SIM[Simulator<br/>simulated] --> H
  CSV[CSV trend import<br/>imported] --> H
  H[(Multi-source historian<br/>tag + source_class + source_name + timestamp)]
  H -->|default read<br/>measured-current| UI[Server-driven UI / AI tools]
  H -->|explicit multi-source query| MS[Multi-source consumers]
  H -->|site-to-cloud sync<br/>samples + retention deletions| CELL[Cell / cloud historian]

Caution

The trend-data import surface is code-complete and wired but not yet deployment-verified, and is part of the early onboarding path rather than a generally available feature. At its first shipped wave it is cloud-only, and the upload UI and AI tool handler are deferred. Treat endpoint behaviour described here as the contract, not a deployed guarantee. Cell-mediated site→cloud sync likewise depends on the account control plane and cells, which are live in the public beta.

Reading the historian

Reads return measured-current data by default. Every existing tag-read path — the server-driven UI, AI tool surfaces, alarm and state-machine evaluation, scene rendering — sees the live PLC reading for the device’s current site source and nothing else, unless it explicitly opts in. This default is non-negotiable: it stops simulated, replayed, or workspace data leaking into operator views.

# Default — returns the measured/current site-PLC value
read tag "Speed"
    → one value (the current measured source)

# Explicit multi-source — caller names the source class(es) and a name pattern
read tag "Speed"
    classes      = [measured, simulated]
    name pattern = "%/study/commissioning/%"
    → one value per matching (class, source name)

The shape is illustrative; the load-bearing rule is that multi-source callers do something different from the default call and never receive other classes implicitly. The read default is also server-mode aware: a Site server defaults to measured, while a Cloud/cell server defaults to measured-plus-default-simulated so that cloud demos can animate simulated trends honestly. A per-tenant source-filter override applies in both server modes — it replaces whichever server-mode default is in force; absent an override the server-mode default applies.

Note

When AI tools read trends, they read through the same historian seam as any other consumer — they observe and explain values, they do not write them and they never command equipment. The platform’s AI is advisory-only; deterministic control stays on the controller and hard interlocks live in IO modules and safety relays.

Historian values are queryable over one public REST endpoint in the rendered reference: GET /api/sites/{siteId}/instances/{instanceId}/trend returns a bounded, instance-scoped trend window (driven from the CLI as csy instance trend with --last or --from/--to and --max-points), resolved through the same server-side tag-read seam as every other consumer. The trend-data import routes are described below. Site-to-cloud sync is server-to-server and has no client endpoints (see Site to cloud sync).

Source-class provenance

The historian’s primary key is the 4-tuple (tag, source_class, source_name, timestamp). Multiple values for the same tag at the same instant coexist when they differ by source. Two columns carry the provenance:

source_class	Meaning	Retention
measured	Direct sensor or PLC reading from physical hardware. The plant told us this.	Raw window, then rollups to the tenant window
inferred	Derived/computed value — power factor from V×I, state inferred from tag combinations, AI-derived health scores. The platform computed this.	Raw window, then rollups to the tenant window
imported	Externally supplied — operator corrections, third-party systems, CSV imports from legacy historians. A human or external system told us this.	Raw window, then rollups to the tenant window
simulated	Output of a committed simulator run — commissioning studies, validation, training data, compliance evidence. The model told us this and we kept it.	Raw window, then rollups to the tenant window
replayed	Historical playback through a fresh pipeline for forensic/what-if analysis; may be annotated post-replay.	Raw window, then rollups to the tenant window
workspaced	Transient exploratory data from any pipeline — scratchpads, agent what-ifs, test scenarios. The actor is exploring, not committing.	7 days (raw only)

The classes are organised by epistemic origin (how a value came to exist), not by purpose, which keeps the taxonomy stable as use cases evolve. At the current ship state, producers exist for measured (real PLC and edge), simulated (the device-simulator path), and imported (CSV import, below); inferred, replayed, and workspaced are schema-ready without producers.

Source names

source_name is a string column with hierarchical conventions per class — documented and enforced by code review and typed builders, not by schema constraints, so prefix queries work (e.g. source_name LIKE 'measured/riverbend/pump1/%'):

measured/{site_id}/{device_instance_id}/vplc
measured/{site_id}/{device_instance_id}/edge/{edge_id}
inferred/{site_id}/{device_instance_id}/{calc_id}
imported/{site_id}/csv/{import_batch_id}
simulated/{site_id}/{device_instance_id}/study/{study_id}/{run_id}
replayed/{originating_class}/{originating_source_name}/{replay_session_id}
workspaced/{user_id}/{session_id}/{...}

Per-source-name addressing makes deletion atomic: removing a batch or a replay is a single DELETE … WHERE source_name = '…'.

Retention

Raw samples are bounded. On a cloud cell, raw samples are kept for a raw window of at least 30 days, sized to absorb late-arriving edge store-and-forward data; older raw data is then dropped. Older history survives at coarser fidelity in a rollup tier maintained cloud-side: analogue signals as 1-minute rollups (min/max/avg/first/last/count, with the worst quality in each bucket) and boolean/state signals as transition rows. The rollup tier is kept for the tenant’s retention window (395 days by default — roughly 13 months). Trend reads that reach past the raw window are served from the rollup tier, stitched to raw samples for the recent part.

A Site server keeps a 30-day raw working set and no rollups. workspaced data is raw-only, is never rolled up, and is swept at 7 days. Imported history is tiered like every other class: its raw rows expire with the raw window, and its rollups are kept to the tenant’s window, keyed on each sample’s own (historical) timestamp. A partner importing years of legacy trend data therefore keeps only the most recent retention window, at rollup fidelity.

A per-row retention_hint can shorten a row’s life below its class default but never extends beyond the tenant’s retention policy; tenant-level policies may impose tighter limits.

CSV trend-data import

The trend-data import contract covers bringing months-to-years of existing trend data (Citect, OSIsoft PI exports) into a tenant. Imported rows land as source_class = imported, source_name = imported/{site_id}/csv/{batch_id}, so they remain distinguishable from measured and simulated rows in cross-source queries.

Two-file upload: narrow CSV + tag mapping

Import takes two files. The trend data uses a narrow format (timestamp,tag_id,value[,quality]):

timestamp,tag_id,value,quality
2026-05-20T14:30:00+10:00,PP-01.MotorCurrent,45.7,Good
2026-05-20T14:30:01+10:00,PP-01.MotorCurrent,45.8,Good

The quality column is optional per file, not per row: the header is either the exact 3-column or the exact 4-column form, and every row must match the header’s width (under a 4-column header, a 3-cell row is rejected).

A separate mapping CSV resolves the customer’s external tag IDs to the platform’s canonical synthetic instance IDs and device-type tag names:

external_tag_id,instance_id,tag_name
PP-01.MotorCurrent,dev_BVSWSZ1Q,MotorCurrent
PP-01.Running,dev_BVSWSZ1Q,Running

[quality_mapping]
external_quality_value,canonical_quality
192,Good
0,Bad
64,Uncertain

Every mapping entry is validated pre-import (the instance must exist; the tag_name must exist on the resolved device type). The mapping is reusable — prepared once per site, reused for subsequent monthly imports — and the optional [quality_mapping] section translates customer quality codes to the canonical Good / Bad / Uncertain (with Questionable aliased to Uncertain). Wide-format CSV is out of scope at this stage.

Timestamps must carry an explicit offset

Timestamps must be ISO-8601 with an explicit timezone offset (2026-05-24T10:30:00+10:00 or …Z). Naive timestamps are rejected per row. This is deliberate: silently assuming UTC for a naive AEST timestamp produces a 10-hour shift indistinguishable from a real process anomaly. Storage normalises to UTC (TIMESTAMPTZ).

Validation, threshold, and rollback

Validation is two-tier with a hard gate:

• Structural failures in the trend file (malformed header, missing required columns, non-UTF-8, empty file, wide-format detected) fail the whole batch immediately — status = Failed, no rows written. A structurally invalid mapping file is rejected at upload with 422 and per-line errors — no batch is created at all.
• Row-level failures (unparseable/naive timestamp, unmapped external_tag_id, value not coercible to the resolved DataType, unmapped quality, future-dated

  24h, older than the configured minimum) are tolerated per row and recorded with row_number, original_row, and error_reason.

• Threshold gate — rows are written to a staging area and swapped into the live samples table only on success. If the reject ratio exceeds 5%, the batch is rolled back (DELETE … WHERE source_name = 'imported/{site}/csv/{batch}') and marked Failed. At or below 5% the batch completes as Completed (zero rejects) or PartiallyFailed (one or more rejects).

The threshold is not configurable per tenant.

Idempotency

Each trend file is hashed (SHA-256). A partial unique index over (site_id, file_hash) for non-superseded batches means the same content can be active in at most one batch per site. A duplicate upload returns 409 Conflict with the existing batch ID and status; re-uploading with the force form field set to true supersedes the prior batch (its superseded_by is set and the supersession is recorded in server logs). This closes the re-upload loophole that the 4-tuple PK alone cannot — a fresh batch_id would otherwise mint a fresh source_name and double-ingest.

Async upload and polling

Upload returns 202 Accepted immediately with a batch_id (a GUID) and status = Pending; a background service processes the batch and recovers Processing rows to Pending on restart. Clients poll for status. An upload can also return 429 when the per-tenant active-batch cap is hit (retry after the Retry-After interval) or 413 over the multipart size limit.

Status: Pending → Processing → Completed | Failed | PartiallyFailed. A batch replaced by a forced re-upload before it started ends Superseded. Deleting a batch moves it to Deleting, then Deleted.

The endpoint surface (see the note below for what the published reference covers):

Method	Path	Purpose
POST	/api/historian/import-batches	Upload (multipart/form-data, fields below) → 202
GET	/api/historian/import-batches?site={site_id}&limit={n}	One site’s batches for your organisation, newest first (site required; limit 1–200, default 50)
GET	/api/historian/import-batches/{batchId}	Status and summary
GET	/api/historian/import-batches/{batchId}/errors	Row-reject report (CSV download)
GET	/api/historian/import-batches/{batchId}/mapping-diff?compare-to={priorBatchId}	Mapping diff between two batches
DELETE	/api/historian/import-batches/{batchId}	Delete one batch’s imported rows and their rollups. Returns 202 until the deletion has run (poll the status until Deleted) and 200 once it is Deleted. The batch record is kept as the audit trail, and deleting does not refund the import quota.

These six routes are the import surface. csy trend-import delete <batchId> drives the delete.

The upload is a multipart/form-data POST carrying individual form fields — there is no JSON metadata part:

Form field	Value
site_id	Target site identifier (required)
format	Trend-file format — CsvNarrow (required)
trend_csv	The narrow trend CSV file (required)
mapping_csv	The tag-mapping CSV file (required)
force	Optional — true supersedes an existing batch with the same content (see idempotency)

The per-batch mapping is persisted as JSONB, which powers the mapping-diff endpoint: a customer asking “why are my imports getting more rejects this month?” gets the added/removed/modified mappings between two batches in one call rather than diffing files by hand.

What the published reference covers

Of the import routes, the rendered reference lists the multipart upload (POST /api/historian/import-batches), the per-batch status (GET /api/historian/import-batches/{batchId}) and the batch delete (DELETE /api/historian/import-batches/{batchId}), each for a person’s sign-in only. The batch list, row-reject report and mapping-diff routes are not in it. This page is authoritative for all six routes.

All of these endpoints, including the read-only status, list, errors and mapping-diff GETs, require an org owner or admin signed in as a person (csy auth login); a platform admin also passes. API keys are refused on every import route, whatever their role, so a --token csk_… run of csy trend-import is refused. A plain member gets 403 even for status polling. See the auth model for the schemes and roles for the membership model (owner / admin / member). An import AI tool surface is deferred (see the caution above) — it is planned as read-only status reporting, and agents cannot trigger imports.

Site to cloud sync

A site server linked to the cloud syncs its historian up to its tenant’s cloud cell. This is server-to-server sync, not an API for clients: the site server authenticates with the site→cloud sync key it received at site activation.

Each synced sample carries full provenance — sourceClass and sourceName travel alongside the value, timestamp, quality, and typed payload — so the cloud historian preserves the same multi-source attribution the site recorded. Retention runs at the site: when a site sweeps aged rows (for example a 7-day workspaced run), it sends the deletions to the cloud first and deletes locally only after that succeeds, so a failed send leaves the site’s rows in place for the next sweep. Site-model sync is opt-in during the beta (Sync:ModelSyncEnabled, default off). Site-to-cloud sync depends on the cell topology, which is live in the public beta.

See also

• API overview and the rendered reference
• Authentication model and API keys
• Device tags — the model tag_name resolves against
• Cell topology — where synced data lands
• Controller communication — the measured-class source