Historian & Trend Import

The historian is Consystence’s time-series store and the primitive that site servers sync upward into a cell. Unlike a conventional historian — one value per (tag, timestamp) — the Consystence historian carries provenance on every sample: each value records how it came to exist (measured by the plant, computed by the platform, imported from a file, produced by a simulator, and so on). This page covers reading the historian, the multi-source provenance model, the CSV trend-data import contract, and the site→cloud ingest endpoints.

flowchart TB
  PLC[Controller / edge<br/>measured] --> H
  SIM[Simulator / DSI<br/>simulated] --> H
  CSV[CSV trend import<br/>imported] --> H
  H[(Multi-source historian<br/>tag + source_class + source_name + timestamp)]
  H -->|default read<br/>measured-current| UI[Server-driven UI / AI tools]
  H -->|explicit multi-source query| MS[Multi-source consumers]
  H -->|sync + deletion blocks<br/>POST /api/cloud/ingest/historian| CELL[Cell / cloud historian]

Caution

The trend-data import surface is code-complete and wired but not yet deployment-verified, and is part of the early onboarding path rather than a generally available feature. At its first shipped wave it is cloud-only and PostgreSQL-only (the staging write/swap/delete path throws on SQL Server), and the upload UI and AI tool handler are deferred. Treat endpoint behaviour described here as the contract, not a deployed guarantee. Cell-mediated site→cloud sync likewise depends on the account control plane and cells, which are live but not yet GA.

Reading the historian

Reads return measured-current data by default. Every existing tag-read path — the server-driven UI, AI tool surfaces, alarm and state-machine evaluation, scene rendering — sees the live PLC reading for the device’s current site source and nothing else, unless it explicitly opts in. This default is non-negotiable: it stops simulated, replayed, or workspace data leaking into operator views.

# Default — returns the measured/current site-PLC value
read tag "Speed"
    → one value (the current measured source)

# Explicit multi-source — caller names the source class(es) and a name pattern
read tag "Speed"
    classes      = [measured, simulated]
    name pattern = "%/study/commissioning/%"
    → one value per matching (class, source name)

The shape is illustrative; the load-bearing rule is that multi-source callers do something different from the default call and never receive other classes implicitly. The read default is also server-mode aware: a Site server defaults to measured, while a Cloud/cell server defaults to measured-plus-default-simulated so that cloud demos can animate simulated trends honestly. A per-tenant source-filter override applies in both server modes — it replaces whichever server-mode default is in force; absent an override the server-mode default applies.

Note

When AI tools read trends or classify behaviour, they read through the same historian seam as any other consumer — they observe and explain values, they do not write them and they never command equipment. The platform’s AI is advisory-only; deterministic control stays on the controller and hard interlocks live in IO modules and safety relays.

Historian values are queryable over one public REST endpoint in the rendered reference: GET /api/sites/{siteId}/instances/{instanceId}/trend returns a bounded, instance-scoped trend window (driven from the CLI as csy instance trend with --last or --from/--to and --max-points), resolved through the same server-side tag-read seam as every other consumer. The rest of the published historian REST surface covers the trend-data import upload and per-batch status (below); the site→cloud ingest endpoints are server-to-server and sit outside the CLI-scoped reference.

Source-class provenance

The historian’s primary key is the 4-tuple (tag, source_class, source_name, timestamp). Multiple values for the same tag at the same instant coexist when they differ by source. Two columns carry the provenance:

source_class	Meaning	Retention
measured	Direct sensor or PLC reading from physical hardware. The plant told us this.	Bounded (retention window)
inferred	Derived/computed value — power factor from V×I, state inferred from tag combinations, AI-derived health scores. The platform computed this.	Bounded (retention window)
imported	Externally supplied — operator corrections, third-party systems, CSV imports from legacy historians. A human or external system told us this.	Bounded (retention window)
simulated	Output of a committed simulator run — commissioning studies, validation, training data, compliance evidence. The model told us this and we kept it.	Bounded (retention window)
replayed	Historical playback through a fresh pipeline for forensic/what-if analysis; may be annotated post-replay.	Bounded (retention window)
workspaced	Transient exploratory data from any pipeline — scratchpads, agent what-ifs, test scenarios. The actor is exploring, not committing.	7 days (raw only)

The classes are organised by epistemic origin (how a value came to exist), not by purpose, which keeps the taxonomy stable as use cases evolve. At the current ship state, producers exist for measured (real PLC and edge), simulated (the device-simulator path), and imported (CSV import, below); inferred, replayed, and workspaced are schema-ready without producers.

Source names

source_name is a string column with hierarchical conventions per class — documented and enforced by code review and typed builders, not by schema constraints, so prefix queries work (e.g. source_name LIKE 'measured/riverbend/pump1/%'):

measured/{site_id}/{device_instance_id}/vplc
measured/{site_id}/{device_instance_id}/edge/{edge_id}
inferred/{site_id}/{device_instance_id}/{calc_id}
imported/{site_id}/csv/{import_batch_id}
simulated/{site_id}/{device_instance_id}/study/{study_id}/{run_id}
replayed/{originating_class}/{originating_source_name}/{replay_session_id}
workspaced/{user_id}/{session_id}/{...}

Per-source-name addressing makes deletion atomic: removing a batch or a replay is a single DELETE … WHERE source_name = '…'.

Retention

Retention is bounded, not indefinite. Current behaviour keeps raw samples to the retention window regardless of class: the tenant’s configured window on a cloud cell (395-day default — roughly 13 months), and a 30-day raw working set on a Site server. workspaced is the exception: it is raw-only and swept at 7 days. Imported history is bounded like every other durable class, keyed on each sample’s own (historical) timestamp — a partner importing years of legacy trend data keeps the most recent retention window, and older imported data is deliberately bounded out.

Storage tiering is committed design, not yet live

The committed design replaces the class-blind window with tiered retention: raw samples kept for a bounded recent window (roughly 30 days, sized to absorb late-arriving edge store-and-forward data), with older data surviving at coarser fidelity — analog signals as 1-minute rollups (min/max/avg/first/last/count with worst-case quality per bucket) and boolean/state signals as transition rows, both retained to the tenant’s window, maintained cloud-side. No rollup job runs today; until it ships, the raw-to-window behaviour above is what you will observe.

A per-row retention_hint can shorten a row’s life below its class default but never extends beyond the tenant’s retention policy; tenant-level policies may impose tighter limits.

CSV trend-data import

The trend-data import contract covers bringing months-to-years of existing trend data (Citect, OSIsoft PI exports) into a tenant. Imported rows land as source_class = imported, source_name = imported/{site_id}/csv/{batch_id}, so they remain distinguishable from measured and simulated rows in cross-source queries.

Two-file upload: narrow CSV + tag mapping

Import takes two files. The trend data uses a narrow format (timestamp,tag_id,value[,quality]):

timestamp,tag_id,value,quality
2026-05-20T14:30:00+10:00,PP-01.MotorCurrent,45.7,Good
2026-05-20T14:30:01+10:00,PP-01.MotorCurrent,45.8,Good

The quality column is optional per file, not per row: the header is either the exact 3-column or the exact 4-column form, and every row must match the header’s width (under a 4-column header, a 3-cell row is rejected).

A separate mapping CSV resolves the customer’s external tag IDs to the platform’s canonical synthetic instance IDs and device-type tag names:

external_tag_id,instance_id,tag_name
PP-01.MotorCurrent,dev_BVSWSZ1Q,MotorCurrent
PP-01.Running,dev_BVSWSZ1Q,Running

[quality_mapping]
external_quality_value,canonical_quality
192,Good
0,Bad
64,Uncertain

Every mapping entry is validated pre-import (the instance must exist; the tag_name must exist on the resolved device type). The mapping is reusable — prepared once per site, reused for subsequent monthly imports — and the optional [quality_mapping] section translates customer quality codes to the canonical Good / Bad / Uncertain (with Questionable aliased to Uncertain). Wide-format CSV is out of scope at this stage.

Timestamps must carry an explicit offset

Timestamps must be ISO-8601 with an explicit timezone offset (2026-05-24T10:30:00+10:00 or …Z). Naive timestamps are rejected per row. This is deliberate: silently assuming UTC for a naive AEST timestamp produces a 10-hour shift indistinguishable from a real process anomaly. Storage normalises to UTC (TIMESTAMPTZ).

Validation, threshold, and rollback

Validation is two-tier with a hard gate:

• Structural failures in the trend file (malformed header, missing required columns, non-UTF-8, empty file, wide-format detected) fail the whole batch immediately — status = Failed, no rows written. A structurally invalid mapping file is rejected at upload with 422 and per-line errors — no batch is created at all.
• Row-level failures (unparseable/naive timestamp, unmapped external_tag_id, value not coercible to the resolved DataType, unmapped quality, future-dated

  24h, older than the configured minimum) are tolerated per row and recorded with row_number, original_row, and error_reason.

• Threshold gate — rows are written to a staging area and swapped into the live samples table only on success. If the reject ratio exceeds 5%, the batch is rolled back (DELETE … WHERE source_name = 'imported/{site}/csv/{batch}') and marked Failed. At or below 5% the batch completes as Completed (zero rejects) or PartiallyFailed (one or more rejects).

The 5% default is configurable per tenant (via the tenant policy accessor).

Idempotency

Each trend file is hashed (SHA-256). A partial unique index over (site_id, file_hash) for non-superseded batches means the same content can be active in at most one batch per site. A duplicate upload returns 409 Conflict with the existing batch ID and status; re-uploading with the force form field set to true supersedes the prior batch (its superseded_by is set and the supersession is recorded in server logs). This closes the re-upload loophole that the 4-tuple PK alone cannot — a fresh batch_id would otherwise mint a fresh source_name and double-ingest.

Async upload and polling

Upload returns 202 Accepted immediately with a batch_id (a GUID) and status = Pending; a background service processes the batch and recovers Processing rows to Pending on restart. Clients poll for status. An upload can also return 429 when the per-tenant active-batch cap is hit (retry after the Retry-After interval) or 413 over the multipart size limit.

Status: Pending → Processing → Completed | Failed | PartiallyFailed.

The endpoint surface (see the note below for what the published reference covers):

Method	Path	Purpose
POST	/api/historian/import-batches	Upload (multipart/form-data, fields below) → 202
GET	/api/historian/import-batches/{batchId}	Status and summary
GET	/api/historian/import-batches/{batchId}/errors	Row-reject report (CSV download)
GET	/api/historian/import-batches/{batchId}/mapping-diff?compare-to={priorBatchId}	Mapping diff between two batches

These four routes are the whole surface today — a collection listing (GET /api/historian/import-batches) and a per-batch hard delete (DELETE /api/historian/import-batches/{batchId}) are not yet implemented; poll a batch by the batch_id the upload returned.

The upload is a multipart/form-data POST carrying individual form fields — there is no JSON metadata part:

Form field	Value
site_id	Target site identifier (required)
format	Trend-file format — CsvNarrow (required)
trend_csv	The narrow trend CSV file (required)
mapping_csv	The tag-mapping CSV file (required)
force	Optional — true supersedes an existing batch with the same content (see idempotency)

The per-batch mapping is persisted as JSONB, which powers the mapping-diff endpoint: a customer asking “why are my imports getting more rejects this month?” gets the added/removed/modified mappings between two batches in one call rather than diffing files by hand.

What the published reference covers

The rendered reference is CLI-scoped: the multipart upload (POST /api/historian/import-batches) and the per-batch status (GET /api/historian/import-batches/{batchId}) are published there. The row-reject report and mapping-diff GETs are not yet CLI-exposed and so are not published — the contract above is authoritative for those.

All of these endpoints — the read-only status, errors, and mapping-diff GETs included — require an org owner or admin (the OrgAdmin policy; a platform admin also passes). A plain member gets 403 even for status polling. See the auth model for the schemes and roles for the membership model (owner / admin / member). An import AI tool surface is deferred (see the caution above) — it is planned as read-only status reporting, and agents cannot trigger imports.

Cloud ingest (site → cloud sync)

A site server pushes historian data up to its cell over the ingest endpoints. These are server-to-server sync primitives, not operator APIs:

Method	Path	Body	Purpose
POST	/api/cloud/ingest/historian	HistorianSyncBlock	A block of samples for a window from a site
POST	/api/cloud/ingest/historian/delete	HistorianSyncDeletionBlock	A retention deletion block (cloud-≥-site coordination)
POST	/api/cloud/ingest/model	ModelSyncPayload	A model/configuration payload from a site — the site-side sender is opt-in during beta (Sync:ModelSyncEnabled, default off)

The model family also exposes read-back endpoints — delivery status and custody snapshot listings — that a site uses to verify what the cloud holds.

Each synced sample carries full provenance — sourceClass and sourceName travel alongside the value, timestamp, quality, and typed payload — so the cloud historian preserves the same multi-source attribution the site recorded. The deletion block exists because retention runs at the site: when a site sweeps aged rows (for example a 7-day workspaced run), it pushes the deletions to the cloud before deleting locally, keeping the cloud copy a superset of the site (an idempotent delete on the matching 4-tuple). Site-to-cloud sync depends on the cell topology, which is not yet GA.

Administrative deletion

DELETE /api/admin/tenants/{slug}/historian-data wipes a tenant’s historian data and requires the request body { "confirm": "DELETE-{slug}-HISTORIAN" } — the literal token with the tenant slug; any other value is refused with 400. This platform-admin operation is currently the only deletion endpoint for imported rows — a per-batch hard delete (removing only the rows under one import batch’s source name) is not yet implemented. As an operator-internal endpoint it sits outside the CLI-scoped rendered reference; this page is the contract.

See also

• API overview and the rendered reference
• Authentication model and API keys
• Device tags — the model tag_name resolves against
• Cell topology — where ingest blocks land
• Controller communication — the measured-class source