Using Dyalog APL Component Files for Persistent Array Storage

Problem: Needing reliable, random‑access storage for APL arrays

When an APL workspace ends, any arrays you created disappear unless you explicitly save them. Workspace saves are coarse‑grained and reload the whole environment, which is inconvenient when you only need to persist a few arrays or share data between concurrent sessions. Dyalog APL provides a built‑in component file system that lets you store individual arrays as numbered components, read or replace them without loading the whole file, and control access with simple locking.

Thesis: Component files give efficient, versioned persistence with minimal overhead

By treating each array as a component identified by an integer index, you can append new data, update existing entries, and read any component directly. The underlying file uses binary format optimized for fast random access, and the system serializes write operations to prevent corruption.

Creating and writing components

To start, you create a component file with ⎕FCREATE. The second argument is an access code; 0 means read/write for the owner, 1 read‑only for others, and 2 read/write for all (adjust according to your OS permissions).

⍝ Run in a Dyalog session; you need write permission in the current directory
file ← 'mydata'
⎕FCREATE file 0   ⍝ creates an empty component file named mydata

Once the file exists, you can add components with ⎕FAPPEND. Each call stores the supplied array as the next sequential component number.

⍝ Store a simple integer vector as component 1
⎕FAPPEND file ⍳5   ⍝ appends 1 2 3 4 5
⍝ Store a 2×3 matrix as component 2
mat ← 2 3 ⍴ ⍳6
⎕FAPPEND file mat  ⍝ appends the matrix

If you need to replace an existing component, use ⎕FREPLACE with the target component number.

⍝ Replace component 1 with a different vector
⎕FREPLACE file 1 ⍳10   ⍝ now component 1 holds 1..10

Reading components and verifying integrity

Reading is performed with ⎕FREAD, which returns the exact array as it was written, preserving shape, datatype, and nesting.

⍝ Read component 2 (the matrix) back
readMat ← ⎕FREAD file 2
⍝ Check that it matches the original using match (≡)
readMat ≡ mat   ⍝ should return 1 (true)

You can also query the number of components with ⎕FNUMS and the size of each component with ⎕Fsize if you need to manage storage.

Concurrent access and limitations

Multiple APL processes can open the same file in read‑only mode simultaneously. Write access is serialized through a file‑level lock: only one process may hold the lock for writing at any time. If a process crashes while holding the write lock, the file may be left in an inconsistent state; recovery requires deleting the damaged file and restoring from a backup.

Component files are platform‑specific binary formats. Moving a file between different CPU architectures (e.g., from x86‑64 Linux to ARM‑based Raspberry Pi) will cause read errors. To transfer data across architectures, export the arrays to a portable format (such as CSV or JSON) or use ⎕FCOPY with conversion options provided by Dyalog.

Worked example: a simple configuration store

Suppose you want to persist a set of configuration parameters that can be read and updated by different scripts.

1. Create the file (once):

   ⎕FCREATE 'config' 0

2. Store an initial configuration as component 1 (a namespace of settings):

   cfg ← ⎕NS ''
   cfg.host ← 'example.com'
   cfg.port ← 8080
   ⎕FAPPEND 'config' cfg

3. Later, a script reads the config, makes a change, and writes it back:

   current ← ⎕FREAD 'config' 1
   current.port ← 9000   ⍝ modify a setting
   ⎕FREPLACE 'config' 1 current   ⍝ update component 1

4. Verify the update:

   updated ← ⎕FREAD 'config' 1
   updated.port   ⍝ should display 9000

Throughout this example, you only need read/write permission on the file; no special privileges are required beyond standard filesystem access.

Actionable closing

Component files are a lightweight way to give APL programs durable storage without the overhead of a full database. Use them for logs, configuration, or intermediate results that multiple sessions need to share. Remember to:

• Keep a backup strategy, since a crashed writer can corrupt the file.
• Export data to a neutral format when moving between hardware architectures.
• Monitor component counts with ⎕FNUMS to avoid unexpected growth.

With these practices, you can rely on Dyalog’s component file system for fast, reliable persistence in your APL projects.