Dyalog Component Files: Architecture for Persistent APL Arrays

The Challenge: Storing Nested Arrays Durable in APL

APL applications often need to persist complex nested arrays—trees of mixed data types—without introducing a full database. Dyalog's component files provide a lightweight, single-node storage mechanism, but using them reliably requires understanding their constraints and failure modes.

Requirements Driving the Design

• Persist nested APL arrays with component-level read/write access
• Verify data integrity via checksums
• Bound file growth to prevent runaway storage
• Avoid external dependencies like SQL or Redis

Smallest Suitable Design

A minimal implementation uses one component file per logical aggregate. Key operations:

1. ⎕FCREATE – Create file with explicit size limit and checksum enabled
2. ⎕FAPPEND – Add new version of array as a new component
3. ⎕FREAD – Retrieve array by component number
4. ⎕FREPLACE/⎕FDELETE – Update or remove components
5. OS permissions – Restrict access to the service's OS user

Trust and Data Boundaries

The Dyalog process OS user is the trust boundary. Component files contain no authentication, encryption, or metadata beyond what APL exposes. All security must be enforced at the filesystem level—user isolation, directory permissions, and potentially encrypted disks.

Operational Checks for Production

Log every component number and the result code from ⎕FREAD. Monitor file size against the limit set in ⎕FCREATE. On every read, validate the checksum status before using the returned data. A simple wrapper function can automate this:

∇ result←ReadComponent fileName compNum
  result←⎕FREAD(fileName, compNum)
  ⍝ Check checksum status via ⎕FI or error handling
  ⍝ Log compNum and success/failure
∇

Failure Modes to Anticipate

• Checksum mismatch – Indicates file corruption; treat as data loss
• Partial last component – May occur after a crash during append; truncate or rebuild
• Lost updates – Concurrent writers without coordination can overwrite each other
• Write failure – Disk full or quota exceeded; handle gracefully

When This Design Breaks

Switch to a database or Dyalog SQL interface if you need:

• Multi-node shared access
• Encryption at rest within the storage layer
• Ad-hoc queries beyond component number lookup
• File sizes approaching Dyalog-specific limits

Practical Verification Steps

1. Create a scratch file: ⎕FCREATE 'test.fc' 1000000 1
2. Append a nested array: data←(1 2 3) (,⊂'hello' 42) ⋄ ⎕FAPPEND 'test.fc' data
3. Read it back: result←⎕FREAD('test.fc' 1)
4. Compare: result≡data should return 1

Testing Corruption Detection

After confirming basic operations, corrupt a byte in the file using an external tool (e.g., dd or a hex editor), then attempt to read the affected component. Dyalog should report a checksum error if the corruption is detected.

Version-Specific Limits

Check ⎕VERSION and consult Dyalog documentation for your release. Maximum component count, file size limits, and checksum defaults vary between versions. Do not assume portability across Dyalog versions without testing.

Conclusion

Component files offer a pragmatic way to persist APL data durably in single-node services. By respecting their boundaries—trust delegated to the OS, no built-in concurrency control, and version-dependent limits—you can build reliable storage with minimal moving parts.