Apparently unlimited != unlimited in the Linux kernel for maximum number of open files. After some extensive digging, I finally found the actual maximum to the nofile setting in /etc/security/limits.conf. Yes I was searching in the context of Oracle (Imagine that) for a maximum number of procs / files. The Linux kernel has a hard upper limit of 1024*1024 (1048576 – a magical number I won’t soon forget).
Consider the following:
mrsmith soft nofile unlimited
mrsmith hard nofile unlimited
Trying to su to mrsmith suddenly casts you aside:
could not open session
Displaying the open file setting for mrsmith shows something odd:
open files (-n) 1024
Why would our number of open files be 1024? This apparently is the default setting for users without a custom nofile set based on the following bugzilla.
Then what is our actual upper bound for nofile in /etc/security/limits.conf? This isn’t well documented anywhere I’ve seen, but based on some research it appears to be 1024*1024. Setting nofile to 1048577 produces the same “could not open session” error as mentioned previously. 1048576 however seems to work just fine. Although you can set your nofile ulimit to this, the effects of such are unknown and I would highly recommend testing any such setting.