fopen path-length failure: ENAMETOOLONG reliance on libc implementation

Direct Answer

On POSIX-conforming systems (glibc, musl, BSD libc), fopen sets errno to ENAMETOOLONG when the kernel's open(2) returns that error for a pathname exceeding the system limit. The C standard itself does not require fopen to set errno for any failure; a conforming implementation may leave errno unchanged. Non-POSIX runtimes (Windows CRT, some embedded newlib configurations) often return NULL without modifying errno or substitute a different code such as EINVAL.

Confirmed Behavior by Implementation

• glibc / musl / FreeBSD / NetBSD / OpenBSD libc: pass the pathname to open(2); if the kernel returns ENAMETOOLONG, that value is copied directly into errno and fopen returns NULL.
• Windows CRT (Universal CRT, MSVCRT): validates the path length internally before calling the OS; on overflow it typically returns NULL with errno set to EINVAL, not ENAMETOOLONG.
• Embedded newlib / minimal libc: may omit errno assignment entirely for this failure mode.

Portability of Pre-Checking Against PATH_MAX

Pre-checking strlen(path) >= PATH_MAX is not universally portable:

• PATH_MAX may be undefined (no fixed limit) on some POSIX systems, causing compilation errors.
• Even when defined, the effective limit can be lower due to filesystem-specific constraints (NFS, CIFS, encrypted volumes), so the check can pass while the kernel still returns ENAMETOOLONG.
• On Windows, PATH_MAX is not a standard constant; the CRT uses _MAX_PATH (260) or extended-length prefixes, which differ from POSIX semantics.

Verification Steps

1. Compile and run the test program below on each target platform.
2. Observe whether errno is set to ENAMETOOLONG, another value, or left unchanged after fopen returns NULL.

#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <limits.h>
#include <stdlib.h>

int main(void) {
    #ifdef PATH_MAX
    size_t len = PATH_MAX + 10;
    #else
    size_t len = 5000; /* arbitrary long path */
    #endif
    char *path = malloc(len + 1);
    if (!path) return 1;
    memset(path, 'a', len);
    path[len] = '\0';

    errno = 0;
    FILE *f = fopen(path, "r");
    if (f) { fclose(f); }
    printf("fopen returned %s, errno = %d (%s)\n",
           f ? "non-NULL" : "NULL",
           errno,
           errno ? strerror(errno) : "(unchanged)");
    free(path);
    return 0;
}

Missing Diagnostic Detail

Which C library and OS version are you targeting? The recommendation changes if you must support Windows CRT or a specific embedded libc.