Linux › Section J · Network applications and services › Topic J.3
Diagnostic tools and remote procedure calls
"The service is down" can mean a dozen different things: the program is not running, it listens on the wrong address, a firewall drops the packets, or the client is talking to the wrong port. A short set of tools lets you check each possibility in turn instead of guessing.
Work up the layers
When a client cannot reach a service, check from the bottom up and from the server outwards. Is the server process running (systemctl status)? Is it listening, and on which address and port (ss)? Which process owns that port (ss -p or lsof -i)? Do the client's packets actually arrive, and does anything answer (tcpdump)? Can a simple client connect at all (nc or curl)? Which ports are reachable from another machine (nmap, on networks you are allowed to scan)? Each question has a tool, and each answer rules out part of the problem. ping and name lookups are in the topic "Basic ICMP and DNS tools".
ss: sockets at a glance
ss (socket statistics, from the iproute2 package) is the modern tool for listing sockets; netstat, from net-tools, is its legacy predecessor with similar options. -t selects TCP and -u UDP, -l shows listening sockets and -a all of them, -n prints numbers instead of names (faster, and still works when DNS is broken), -p names the owning process (needs sudo for other users' processes), and -4 or -6 restricts the address family. sudo ss -tulpn is the usual first look at a server. ss -tn state established lists open TCP connections, ss -s prints summary counts, and filters such as ss -tn '( dport = :443 )' select by port. The Local Address column is the key detail: 127.0.0.1:5432 is reachable only from the machine itself, while 0.0.0.0:5432 or [::]:5432 accept connections on every interface.
lsof for network connections
lsof, met in the topic "Open files, file descriptors, and lsof", also lists sockets, because sockets are open files. sudo lsof -i lists every network socket; -n skips reverse DNS lookups of addresses and -P keeps port numbers instead of names from /etc/services, and both make it much faster. Filters follow the pattern -i[46][protocol][@host][:port]: lsof -i :22 shows anything using port 22, lsof -iTCP:443 only TCP port 443, and lsof -i6 only IPv6. sudo lsof -nP -iTCP -sTCP:LISTEN lists listening TCP servers with their PIDs and users. UDP sockets have no listening state, so use -iUDP to see them.
Worked example: a service nobody else can reach
A new API works with curl on the server itself, but other machines get "connection refused". 1. Check it is running: systemctl status api shows active (running). 2. Check where it listens: sudo ss -tlnp shows (illustrative) LISTEN 0 4096 127.0.0.1:8080 0.0.0.0:* users:(("api",pid=1840,fd=7)). 3. Read the Local Address: 127.0.0.1 means loopback only, so connections arriving on the network interface find nothing listening, and the kernel answers with a TCP reset, which clients report as refused. 4. Fix the cause in the service's own configuration: make it listen on the server's LAN address, or 0.0.0.0 if every interface is intended. 5. Restart it, confirm the new address with ss, and make sure the firewall allows port 8080 only from the hosts that need it.
tcpdump: seeing the packets
tcpdump captures packets on an interface and prints a line per packet; it needs root and puts the interface into promiscuous mode, so it may also see frames addressed to other hosts. Useful options: -i eth0 picks the interface (-i any for all), -n skips name lookups, -c 20 stops after 20 packets, -w file.pcap saves packets for Wireshark and -r reads them back. Filters, written in the pcap-filter(7) language, keep output manageable: primitives such as tcp, udp, icmp, host 10.0.0.5, net 10.0.0.0/24 and port 443, joined with and, or and not, and grouped with parentheses, as in sudo tcpdump -n -i eth0 'tcp port 8080 and host 10.0.0.7'.
Worked example: do the packets arrive?
Clients still cannot connect to port 8080 after the fix above. On the server, run sudo tcpdump -n -i eth0 tcp port 8080 while a client at 10.0.0.7 tries again. 1. Case A: lines such as 10.0.0.7.51544 > 10.0.0.2.8080: Flags [S] repeat every second or so with no reply. The SYNs arrive but nothing answers, so a firewall rule on the server is probably dropping them; check the firewall, as the topic "Firewalls" explains. 2. Case B: each [S] is answered by Flags [R.], a reset, so packets get through but no process listens on that address; go back to ss. 3. Case C: no lines at all, so the packets never reach this interface; suspect routing or a firewall upstream. 4. Case D: [S], then [S.] and a final ACK, so the TCP handshake works and the problem is in the application.
netcat: a general-purpose test client
nc (netcat) connects standard input and output to a network socket. nc -v host 8080 opens a TCP connection and reports success or failure; -u switches to UDP; -z with a port or range, as in nc -zv host 20-25, checks whether ports accept connections without sending data. nc -l 9000 listens on port 9000 and prints what arrives, so two machines can test a path through a firewall with no real service installed. Beware that several versions exist: the OpenBSD netcat used by Debian and Ubuntu writes nc -l 9000, the traditional version needs nc -l -p 9000, and Fedora and RHEL ship ncat from the Nmap project. Check man nc on your system.
Port scanning with Nmap, with permission
Nmap probes many ports and reports which are open, closed or filtered. nmap 192.168.1.20 scans the most common 1,000 TCP ports of one host, nmap -p 1-65535 host scans every port and nmap -6 scans an IPv6 address. Administrators use it to audit their own systems: scanning from the machine itself and again from another network shows what the firewall really exposes. Scan only systems you own or have written permission to test. Scanning other people's networks may break their acceptable-use rules or the law, and intrusion-detection systems flag scans, so organisations often block the source.
Remote procedure calls and rpcbind
A remote procedure call (RPC) lets a client call a function that runs on a server as if it were local. Sun's ONC RPC, which classic NFS and NIS use, identifies each server program by a number rather than a fixed port, so it needs a directory service, rpcbind, listening on port 111, to tell clients which port each program is using. rpcinfo -p localhost lists the programs registered. On a machine that serves no NFS or NIS, rpcbind is usually unnecessary; NFS version 4 needs only TCP port 2049 and does not depend on it. Disable rpcbind where it is not used and never expose port 111 to the internet. Modern RPC systems such as gRPC run over HTTP/2 and need no portmapper.
Common mistakes to avoid
First, checking only that a service is running, not which address it listens on; 127.0.0.1 is unreachable from outside. Second, leaving off -n, so every address waits for a slow or broken DNS lookup. Third, running tcpdump without a filter on a busy server and drowning in output. Fourth, assuming the same nc options work everywhere. Fifth, scanning a network you do not control. Sixth, reading "connection refused" as a firewall problem: a refusal is a reset, while a silently dropped packet usually ends in a timeout.
Key takeaways
Diagnose layer by layer: process, listening socket, owning process, packets, client connection, external view. ss -tulpn (legacy netstat) lists listening sockets; the local address shows who can reach them. lsof -nP -i, with filters such as -iTCP:443 and -sTCP:LISTEN, maps sockets to processes. tcpdump -n -i with pcap filters shows whether SYNs arrive and what answers. nc tests connections and listens on ports, in several dialects. Nmap audits open ports, only with permission. RPC services register with rpcbind on port 111; rpcinfo -p lists them.
Flashcards
Flashcard
TCP and UDP listening sockets, numeric, with processes
Test your understanding
Question 1 of 5
ss -tlnp shows a database listening on 127.0.0.1:5432. A client on another machine cannot connect. Why?
Question 2 of 5
tcpdump on the server shows repeated Flags [S] packets from the client and nothing sent back. What is the most likely cause?
Question 3 of 5
Which command lists listening TCP servers with their PIDs, without slow name and port lookups?
Question 4 of 5
What does rpcbind do?
Question 5 of 5
A colleague suggests running Nmap against a partner company's servers to check their security. What is the correct response?
