Designing a Trust‑Separated Storage Layer in OCaml with Functors

Requirements

We need a storage component that can be plugged into different back‑ends (in‑memory hash table, file‑based key/value store, remote service) while keeping the storage logic oblivious to low‑level details. The design must guarantee that a faulty or malicious back‑end cannot break the storage’s invariants, and any inconsistency must be reported through a well‑defined error path.

Minimal Design

The core idea is to express the storage policy as a functor that receives a concrete key/value module as a parameter. The functor only uses the abstract interface supplied by that module, so the storage logic never sees the backend’s internal representation.

Signature for the key/value backend

module type KEY_VAL = sig
  type key
  type t
  val empty : t
  val add   : t -> key -> string -> t
  val find  : t -> key -> string option
  val remove: t -> key -> t
  (* optional validation hook – returns true if the store is consistent *)
  val check : t -> bool
end

The check function is supplied by the backend and lets the storage layer verify its own invariants after each mutation.

Functor defining the storage API

module Storage(Make : KEY_VAL) = struct
  type error = Storage_Error of string
  exception Storage_Error of string

  type t = Make.t

  let init () = Make.empty

  let set store k v =
    let new_store = Make.add store k v in
    if not (Make.check new_store) then raise (Storage_Error "set failed validation");
    new_store

  let get store k = Make.find store k

  let delete store k =
    let new_store = Make.remove store k in
    if not (Make.check new_store) then raise (Storage_Error "delete failed validation");
    new_store
end

The functor Storage is instantiated with a concrete backend, producing a module that offers init, set, get, and delete operations. All mutations go through the backend’s check function; a failure raises Storage.Error.

Trust and Data Boundaries

The trust boundary lies at the functor argument: the storage policy trusts only the functions exposed by KEY_VAL. It does not depend on how t is represented, nor does it expose any mutable state owned by the backend. Consequently, even if a backend corrupts its internal representation, the storage layer will detect the inconsistency via check and raise an exception before the corrupted state propagates outward.

Operational Checks

After each mutating operation (set or delete) the functor calls the backend’s check. If the function returns false, the storage module raises Storage.Error. Callers can catch this exception to handle failures explicitly, ensuring that error conditions are visible and contained.

Failure Modes

• Backend bug: returns stale or incorrect data. The subsequent check fails, raising Storage.Error.
• Malicious backend: attempts to bypass validation by exposing internal mutable state. Because the storage layer never accesses that state directly, the only way to affect the API is through the exposed functions; any attempt to break invariants will be caught by check.
• Missing validation: if a backend omits check or always returns true, the trust guarantee is weakened. This is a configuration error detectable during code review or unit testing.

When the Design Would Change

If the application requires:

• Sharing a single mutable storage instance across many modules without recompiling a new functor instance each time,
• Higher‑order combinators that take storage modules as arguments (e.g., map_storage : ('a -> 'b) -> storage -> storage),
• Dynamic swapping of back‑ends at runtime without recompilation,

then a simple functor may become insufficient. In those cases one might consider OCaml objects or first‑class modules to enable runtime polymorphism, or adopt a dependency‑injection pattern where the storage policy receives a backend value rather than a module.

Concrete Example

Below are two backend implementations that satisfy KEY_VAL. The first uses OCaml’s built‑in Hashtbl; the second stores data in a plain file line‑by‑line.

In‑memory backend

module HashtblKV = struct
  type key = string
  type t = (key, string) Hashtbl.t
  let empty = Hashtbl.create 16
  let add tbl k v = Hashtbl.replace tbl k v; tbl
  let find tbl k = Hashtbl.find_opt tbl k
  let remove tbl k = Hashtbl.remove tbl k; tbl
  let check _tbl = true   (* hash table is always internally consistent *)
end

File‑based backend (simplified)

module FileKV = struct
  type key = string
  type t = string * string   (* filename, temporary file for atomic ops *)
  let empty = ("/tmp/kv.db", "/tmp/kv.tmp")
  let add (db, tmp) k v =
    (* append line "key value\n" to tmp, then replace db atomically *)
    let oc = open_out tmp in
      output_string oc (k ^ " " ^ v ^ "\n");
      close_out oc;
      Unix.rename tmp db;
    (db, tmp)
  let find (db, _) k =
    let ic = open_in db in
      try
        while true do
          let line = input_line ic in
            let kv = Str.split (Str.regexp " +") line in
              if List.hd kv = k then begin
                close_in ic;
                return (Some (List.nth kv 1))
              end
        done;
        close_in ic;
        None
      with End_of_file ->
        close_in ic;
        None
  let remove (db, tmp) k =
    (* rewrite db without the line for k into tmp, then swap *)
    let ic = open_in db
        and oc = open_out tmp in
          try
            while true do
              let line = input_line ic in
                let kv = Str.split (Str.regexp " +") line in
                  if List.hd kv <> k then output_string oc (line ^ "\n")
            done
          with End_of_file ->
            close_in ic; close_out oc;
            Unix.rename tmp db;
    (db, tmp)
  let check (db, _) =
    (* verify that the file is well‑formed: each line has exactly two fields *)
    let ic = open_in db in
      try
        while true do
          let line = input_line ic in
            if List.length (Str.split (Str.regexp " +") line) <> 2 then
              close_in ic; false
        done;
        close_in ic; true
      with End_of_file ->
        close_in ic; true
end

Instantiating the storage functor:

module MemStore = Storage(HashtblKV)
module FileStore = Storage(FileKV)

Both modules expose the same API; swapping the backend only requires changing the functor argument and recompiling.

Build and Test Commands

Assuming a Dune project:

1. Place the signature and functor in storage.ml.
2. Place the two backends in hashtbl_kv.ml and file_kv.ml.
3. Create a test file storage_test.ml that uses Expect or Alcotest to assert:

let () =
  let store = MemStore.init () in
  let store2 = MemStore.set store "foo" "bar" in
    assert (MemStore.get store2 "foo" = Some "bar");
    (* introduce a bug in the backend to see error propagation *)
    let bad_store = HashtblKV.add (HashtblKV.empty) "foo" "baz" in
      (* corrupt the hash table by manually inserting a non‑string value – impossible without exposing internals, so we simulate by returning false from check *)
      let module Bad = struct
        include HashtblKV
        let check _ = false
      end
      let module BadStore = Storage(Bad) in
        try
          let _ = BadStore.set BadStore.init () "x" "y" in
            failwith "expected error"
        with Storage.Storage_Error _ -> ()

Run the test:

dune runtest --force

Expected outcome: the test passes, demonstrating that a backend that fails validation triggers Storage.Error. No special privileges are required; a regular user can execute dune in the project directory.
Limitations and Practical Verification
Because functors are evaluated at compile time, changing the backend necessitates recompiling the functor instantiation and any modules that depend on it. This can slow inner‑loop development if back‑ends are swapped frequently. To mitigate, keep backend changes isolated to their own files; Dune will only rebuild those files and the functor.
To verify that the trust boundary holds, deliberately modify a backend to violate its signature (e.g., expose a mutable field or return a value of the wrong type from find) and recompile. The compiler will reject the mismatched signature, proving that the abstraction enforces the contract.
Another practical check is to measure compile time: time dune build after touching only a backend file should show that only that backend and the functor instantiation are rebuilt, confirming modularity.
Conclusion
Using an OCaml functor to parameterize a storage layer cleanly separates policy from mechanism, enforces trust boundaries through an explicit signature, and provides a clear error path via a validation hook. The design satisfies the requirements for pluggable back‑ends while containing failures. When runtime flexibility or higher‑order composition becomes necessary, consider evolving to objects or first‑class modules.