Designing a Minimal, Trust‑Bounded Audit Logging Layer in Clarity 5.x

Problem Statement

Enterprise systems must record user actions and data changes with cryptographic integrity so that audits can prove compliance. Clarity 5.x offers a native audit‑logging module that captures every mutation with immutable timestamps. However, without a careful design, logs can become a single point of failure, grow uncontrollably, or expose sensitive data. This note sketches a minimal, trust‑bounded architecture that satisfies audit requirements while keeping operational complexity low.

Requirements

• Clarity version 5.0 or newer with the audit_log module enabled.
• Append‑only storage for audit entries (e.g., a dedicated table with a write‑once constraint).
• Immutable timestamps and optional cryptographic hash of each record.
• Retention policy that meets regulatory deadlines.
• Automated checks for storage capacity, hash integrity, and write failures.
• Rollback strategy only when the logger writes to a mutable store.

The Smallest Suitable Design

To keep the audit subsystem isolated, we wrap the logger in a lightweight service boundary. The service receives events from Clarity’s internal event bus, serialises them, and writes them to a dedicated append‑only table. The design has three layers:

1. Event Producer – Clarity’s core emits audit events (user ID, action, payload).
2. Audit Service – A stateless process that listens to the bus, validates payloads, adds a UTC timestamp, computes a SHA‑256 hash, and writes the record to the audit_log table.
3. Append‑Only Store – The audit_log table is defined with INSERT ONLY permissions and a unique constraint on the primary key. No UPDATE or DELETE operations are permitted.

Because the service is stateless, it can be scaled horizontally and restarted without losing the audit trail. All state is persisted in the immutable store, so a crash only drops in‑flight events, not historical data.

Trust and Data Boundaries

The audit boundary is defined by:

• Write‑Only Access – Only the audit service can insert into audit_log. All application code must route through the service.
• Hash Verification – Each entry includes a hash_sha256 column calculated over the concatenated string of user_id|action|payload|timestamp. External auditors can recompute the hash to verify integrity.
• Immutable Timestamps – Clarity’s internal clock is used; the service records CURRENT_TIMESTAMP without allowing overrides.
• Retention Enforcement – A scheduled job deletes rows older than the policy window (e.g., 365 days) but never modifies existing rows.

Operational Checks

Production environments require continuous monitoring. Below is a checklist of automated tasks:

1. Storage Capacity Check

   # Run daily via cron
   SELECT pg_size_pretty(pg_total_relation_size('audit_log')) AS size;
   # Alert if size > 80% of allocated quota.

2. Hash Integrity Scan

   # Detect tampering (should never happen in append‑only mode)
   SELECT id FROM audit_log
   WHERE hash_sha256 <> encode(digest(user_id||action||payload||timestamp, 'sha256'), 'hex');
   # Alert if any rows returned.

3. Write‑Failure Alert

   # Audit service logs failures to a separate error table.
   # Monitor the error count per hour and trigger an alert if > 0.
   SELECT COUNT(*) FROM audit_errors WHERE error_time > NOW() - INTERVAL '1 hour';

4. Version Verification

   # Ensure the running Clarity instance is 5.x
   SELECT version FROM clarity_system WHERE component = 'clarity';
   # Expect: 5.0 or higher.
   

Failure Modes and Mitigation

• Logger Process Crash – The stateless service can be restarted by the orchestrator. In‑flight events are lost but historical logs remain intact. A fallback buffer (e.g., a local disk queue) can be enabled to retry writes after restart.
• Storage Quota Exhaustion – If the append‑only table reaches disk capacity, new writes will fail. The service should emit an error event that triggers an alert. Implement a retention job that runs before capacity is hit to purge the oldest records.
• Misconfiguration of Output Path – If the audit service is pointed to a non‑write‑only table or a shared database, sensitive data could be exposed. Enforce schema validation on startup: check that audit_log has the required columns and constraints.
• Hash Collision or Corruption – Extremely unlikely with SHA‑256, but the integrity scan will detect any mismatches. In that case, the audit trail is considered compromised and a forensic audit is required.

When the Design Must Change

Consider revising the architecture if any of the following conditions arise:

• Compliance regulations require signed audit signatures instead of hashes.
• Audit volume exceeds 1TB per week, necessitating sharding or a dedicated log store (e.g., ElasticSearch).
• The business demands real‑time audit visibility in dashboards; a streaming layer (Kafka) may be added between the service and the store.
• The Clarity version drops support for the built‑in module; an external log aggregation pipeline must be introduced.

Concrete Example: Enabling and Testing Audit Logging

1. Enable the Module

   # In clarity-config.yaml
   audit_log:
     enabled: true
     retention_days: 365
     output_table: audit_log
   

2. Verify Table Schema

   CREATE TABLE audit_log (
     id BIGSERIAL PRIMARY KEY,
     user_id VARCHAR(64) NOT NULL,
     action VARCHAR(128) NOT NULL,
     payload JSONB NOT NULL,
     timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
     hash_sha256 CHAR(64) NOT NULL,
     CONSTRAINT audit_log_append_only CHECK (false)
   );
   -- The CHECK (false) ensures no UPDATE/DELETE via SQL.
   

3. Simulate a User Action

   # Assume Clarity exposes an API endpoint /api/submit
   curl -X POST http://clarity.local/api/submit \
     -H 'Content-Type: application/json' \
     -d '{"user_id": "alice", "action": "create_order", "payload": {"order_id": 12345}}'
   

4. Check the Log Entry

   SELECT id, user_id, action, payload, timestamp, hash_sha256
   FROM audit_log
   WHERE user_id = 'alice' AND action = 'create_order'
   ORDER BY timestamp DESC LIMIT 1;
   

   Verify that the timestamp is recent, the hash_sha256 matches the computed hash, and the row is present. If any field is missing, the audit logging is misconfigured.

Limitations & Practical Checks

• Log volume can outpace default storage limits. Monitor pg_total_relation_size('audit_log') and adjust retention_days or add archival storage.
• Older Clarity releases (<5.0) lack append‑only guarantees; in those cases, integrate an external log aggregator (e.g., Fluentd) to achieve similar trust.
• Misconfiguring the audit output path can leak data. Validate the output table on service start and enforce a read‑only role for application code.

Conclusion

By isolating the audit logger behind a stateless service, writing to an immutable table, and automating capacity, integrity, and failure checks, you can meet compliance requirements with minimal operational overhead. The design is intentionally simple: a single write‑only table and a lightweight service. If future constraints demand more complexity—such as real‑time dashboards or higher throughput—you can extend the pipeline without re‑architecting the core audit boundary.