Linux › Section H · Processes and resource utilization › Topic H.2
Open files, file descriptors, and lsof
To the kernel, a file, a pipe, a terminal and a network connection all look alike once a process has opened them: each is a small number called a file descriptor. Knowing how descriptors work, and how lsof lists them, solves some of the most puzzling problems an administrator meets.
What a file descriptor is
When a process opens something, the kernel hands back a file descriptor (fd): a small non-negative integer that the process uses in every later read, write or close. The kernel always gives the lowest number not already in use. By convention every process starts with three: 0 is standard input, 1 standard output and 2 standard error, so a program's first open() usually returns 3. Descriptors are not only for regular files. Pipes, terminals, device files and network sockets are all opened and used through the same calls, which is the practical meaning of the Unix idea that "everything is a file".
Descriptors, open files and redirection
Each process has its own descriptor table. An entry points to an open file description inside the kernel, which records the current offset and the access mode (read, write, append). Two descriptors can share one description, and that is how shell redirection works: for ls > out.txt, the shell opens out.txt in a child process, uses the dup2() call to copy that descriptor onto number 1, and then runs ls. ls writes to fd 1 as usual and never knows that it now leads to a file. open() takes flags such as O_RDONLY, O_CREAT and O_APPEND; read() and write() return the number of bytes moved, possibly fewer than asked, and read() returns 0 at end of file. Written data usually reaches only the kernel's page cache until fsync() forces it to disk. A pipe is a pair of descriptors, one for writing and one for reading, which is how cmd1 | cmd2 joins two processes.
Seeing descriptors in /proc
Every process's descriptors appear as symbolic links in /proc/PID/fd, and /proc/PID/fdinfo/N shows the offset (pos) and flags for descriptor N. In an illustrative test, a shell that opened a log file on fd 3, opened /etc/hostname on fd 4 and then deleted the log showed this with ls -l /proc/$$/fd (trimmed): 0 -> /dev/null, 1 -> pipe:[9460834], 3 -> /tmp/app.log (deleted), 4 -> /etc/hostname. Two lessons are visible. Standard output was a pipe, not a terminal. And a deleted file stays alive while a descriptor still refers to it: its name has gone, but its data and disk space remain until the last descriptor closes.
Limits: too many open files
Descriptors are a limited resource. Each process has a soft limit, which it may raise up to a hard limit; ulimit -n shows the soft limit in the current shell and ulimit -Hn the hard one. Many distributions set the soft limit to 1024 for login shells, although containers and services often have far higher values. When a process reaches its limit, open() fails with the error EMFILE, which programs report as "Too many open files". A busy server that leaks descriptors, opening connections or files without closing them, will eventually hit this. For a systemd service the limit is set with LimitNOFILE= in its unit file.
lsof: listing open files
lsof (list open files) shows every open file and the process holding it. Run with no arguments it prints thousands of lines, so you normally narrow it. As an ordinary user you see full details only for your own processes; run it with sudo to see everything. Each line has these columns: COMMAND (the program name), PID, USER, FD, TYPE, DEVICE (major and minor numbers of the device holding the file), SIZE/OFF (size, or the current offset), NODE (the inode number) and NAME (the path, or a description such as a socket address). An illustrative line: nginx 812 www-data 5w REG 259,2 48213 1835 /var/log/nginx/access.log.
Reading the FD and TYPE columns
The FD column either names a special use or gives a descriptor number with a mode letter. Special uses include cwd (current working directory), rtd (root directory), txt (the program's own code) and mem (a memory-mapped file, usually a shared library). A number is followed by r for read, w for write or u for both, so 5w means descriptor 5 open for writing. TYPE describes the object: REG is a regular file, DIR a directory, CHR a character device such as a terminal, FIFO a pipe, IPv4 or IPv6 a network socket and unix a Unix domain socket. In the nginx line above, the server holds its access log on fd 5 for writing.
Narrowing the output
Common selections: lsof -p 812 lists one process; lsof -u alice one user; lsof -c ssh processes whose names begin with ssh; lsof /var/log/syslog the processes using one file; lsof +D /srv/data everything open under a directory, searched recursively (slow on large trees); lsof +L1 open files whose link count is below 1, which means deleted but still held open. -t prints only PIDs, which is handy in scripts. Beware one rule: when you give several selections, lsof combines them with OR, so lsof -u alice -c ssh lists everything of alice's plus every ssh process. Add -a to require all of them: lsof -a -u alice -c ssh. Network selections with -i are covered in the topic "Diagnostic tools and remote procedure calls".
Worked example: a filesystem that will not unmount
sudo umount /mnt/usb fails with "target is busy". 1. Ask who is using it: sudo lsof +D /mnt/usb (or sudo fuser -vm /mnt/usb, from the psmisc package). 2. Read the result (illustrative): bash 2301 alice cwd DIR 8,17 4096 2 /mnt/usb and less 2398 alice 4r REG 8,17 91327 15 /mnt/usb/notes.txt. 3. Interpret: alice's shell has its working directory there (cwd), and less holds notes.txt open for reading on fd 4. 4. Fix it gently: quit less and cd out of the directory in that shell, rather than killing processes that may have unsaved work. 5. Retry the unmount; it now succeeds.
Worked example: a full disk that du cannot explain
df -h reports /var at 100% used, but du -sh /var/* adds up to far less. 1. Suspect deleted files still held open: du measures names in the directory tree, while df measures blocks in use. 2. Look: sudo lsof +L1 shows (illustrative) java 1477 app 9w REG 259,3 6442450944 0 2621 /var/log/app/debug.log (deleted), where the 0 is the NLINK column that +L adds. 3. Read it: PID 1477 still writes to a 6 GiB log that someone removed with rm, so the space cannot be freed. 4. Fix it the safe way: restart the service (sudo systemctl restart app), which closes the descriptor and releases the space. 5. Prevent a repeat: rotate logs with logrotate, which tells programs to reopen their files, instead of deleting live logs.
Common mistakes to avoid
First, deleting a large log file that a process is still writing and expecting the space back; it returns only when the descriptor closes. Second, forgetting that lsof combines selections with OR unless you add -a. Third, running lsof without sudo and concluding that no process uses a file, when you simply cannot see other users' processes. Fourth, killing every process lsof lists without checking for unsaved work. Fifth, assuming fd 1 is always the terminal; it may be a pipe or a file.
Key takeaways
A file descriptor is the lowest free small integer the kernel returns from open(); 0, 1 and 2 are standard input, output and error. Descriptors point to open file descriptions holding the offset and mode, and dup2() and pipes build redirection and pipelines. /proc/PID/fd shows a process's descriptors, and a deleted file lives on while it is open. ulimit -n shows the per-process limit, and EMFILE means it was reached. lsof lists open files; FD values such as cwd, txt, mem and 5w and TYPE values such as REG and FIFO describe each one. -p, -u, -c, +D, +L1, -t and -a are its everyday options.
Flashcards
Flashcard
Small integer a process uses to refer to an open file
Test your understanding
Question 1 of 5
A process has descriptors 0, 1, 2 and 4 open. It calls open() on a new file. Which descriptor does it get?
Question 2 of 5
df shows a filesystem full, but du finds much less data. Which command most directly shows the likely cause?
Question 3 of 5
In lsof output, what does an FD value of txt mean?
Question 4 of 5
You want only the files that user alice has open in processes named ssh. Which command is correct?
Question 5 of 5
A web server starts failing with "Too many open files" after running for several days. What is the most likely explanation?
