How Nmap OS Detection Works: TCP/IP Stack Fingerprinting Explained

That decoy is running a Windows Server deception profile. The service banners are consistent, nothing in the application layer pointed to Linux, but Nmap still identifies it as a Linux machine.

Nmap is one of the most widely used network reconnaissance tools in offensive security. It can discover live hosts, enumerate open ports, identify running services and their versions, and infer the operating system of a target.

The reason is that Nmap’s OS fingerprinting engine does not rely on banners. It observes how the TCP/IP stack behaves when responding to carefully crafted packets.

In this article, we will explore how Nmap OS fingerprinting works in detail and what it takes to build a believable operating system impersonation. Along the way, we will cover the challenges we faced while developing Anantis TrapEye’s OS deception engine.

How is Fingerprinting Possible if Every OS Follows the TCP RFCs?

Because the RFCs leave room for implementation choices.

The TCP RFCs define how the protocol should behave, but they do not specify every detail. A TCP segment, for example, must advertise a receive window, but the RFCs do not define a specific value. RFCs also describe how to handle valid packets, but leave undefined cases such as packets with no TCP flags set, because those are not part of normal traffic.

Whenever the specification leaves room for interpretation, the operating system developers had to make a choice. Those choices may have been made by Microsoft decades ago or in an old Linux kernel commit. Once deployed, they tend to remain unchanged because modifying TCP behavior can introduce compatibility issues with existing systems.

Nmap takes advantage of these differences, which are often a much more reliable fingerprint than any service banner that can be changed with a configuration file.

TTL and Window Size

When people think about OS fingerprinting, they often think about two signals: TTL and TCP window size.

TTL

Every IP packet carries a Time To Live field: a counter that is decremented by one at each router it crosses. When it reaches zero, the packet is dropped. Its purpose is to prevent routing loops: without TTL, a misrouted packet could circulate forever between routers.

The RFC does not define the initial TTL value, leaving operating systems free to choose their own defaults:

Example: a packet arriving with a TTL of 57 most likely started at 64 and crossed seven routers That’s a reasonable indication that the remote host is running a Unix-like system.

However, TTL is only a weak signal. There are only a few common initial values, and Nmap compares fingerprints against a database containing thousands of signatures.

Window Size

The TCP window is the amount of data a host is willing to receive before requiring an acknowledgment. It’s flow control that stops a fast server from drowning a slow client in data it can’t absorb.

The advertised value depends on the receive buffer configuration chosen by the operating system. Windows historically used values such as 8192 and later 65535. Linux commonly advertises values around 64240 or 65160 depending on the kernel version and MSS. Network appliances such as FortiGate devices have their own patterns.

Compared to TTL, the window size provides more information because it reflects TCP stack behavior rather than just a hop-count estimate.

However, it remains a single value in a packet header. Like TTL, it can be modified easily, for example with a simple sysctl setting.

Nmap Behavior

One common misconception is that OS fingerprinting is based on a single packet.

nmap -O sends sixteen probes and decomposes the responses into roughly fifteen groups of attributes. For best results, it expects both an open TCP port and a closed one. Indeed, the difference between how a host accepts and rejects connections is itself part of the fingerprint.

Sequence Numbers

When a host accepts a connection, it chooses an Initial Sequence Number (ISN). That number must be unpredictable for a good reason: in 1994, Kevin Mitnick compromised Tsutomu Shimomura’s systems by predicting the ISNs generated by a TCP stack. If you can correctly predict an ISN, you can inject packets into a foreign TCP connection.

Nmap collects ISNs from these replies and computes several statistics. One of them is the greatest common divisor (GCD) of the differences between successive sequence numbers. Some stacks increment their ISNs by multiples of a constant, and the GCD reveals that immediately.

Two pseudo-random number generators can be equally unpredictable while remaining statistically distinguishable. Nmap is not testing whether the sequence numbers are random.

For example, one generator might increment an internal counter by a fixed amount before adding random noise, while another draws each value independently from a cryptographically secure PRNG. Both are unpredictable enough to prevent TCP hijacking, but each has a distinct statistical personality that can be fingerprinted.

IP ID

Every IP packet has a 16-bit identifier used for fragment reassembly. The way this value is generated depends on the implementation. Older Windows versions used a single global counter, incremented by one for every outbound packet across all connections. Modern Linux often sets it to zero for packets that are not fragmented, while other systems use random values or per-flow counters.

Nmap analyzes these behaviors (TI for TCP, II for ICMP) and looks at how the values evolve over multiple packets.

Timestamps

TCP timestamps are another source of information for OS fingerprinting. Each TCP segment can include a timestamp value, generated from an internal kernel counter. The option is mainly used for mechanisms such as PAWS (Protection Against Wrapped Sequence numbers) and round-trip time (RTT) measurement.

Nmap measures how much the timestamp value advances between its probes and uses the elapsed time to estimate the clock frequency behind it. Older Linux systems commonly used a 100 Hz tick rate, while newer Linux and FreeBSD systems often use 1000 Hz. Some embedded systems use much lower frequencies.

With only a few packets, Nmap can infer characteristics of the clock used by the remote TCP stack.

TCP timestamp support is also useful as a negative signal. Windows historically disabled TCP timestamps by default, so the absence of the option becomes part of the fingerprint. Nmap records this as TS=U (unsupported) and scores it like any other attribute.

TCP Option Order

TCP options are another example of implementation-specific behavior. A SYN-ACK can include options such as MSS, SACK, timestamps, and window scaling. Most modern systems support the same options, but the order in which they are placed in the packet differs between implementations.

Linux:

1. MSS
2. SACK
3. Timestamp
4. NOP
5. WScale

Windows:

1. MSS
2. NOP
3. WScale
4. SACK
5. Timestamp

Nmap records this ordering across multiple probes (O1 to O6). Since the RFCs do not mandate a specific order, each TCP stack developed its own convention.

Malformed Probes

After the SYN probes, Nmap sends packets that normal applications would rarely generate: segments with no flags set, or ACK packets without an existing connection.

These packets are useful because RFCs define normal behavior, but leave many invalid cases unspecified. Different TCP implementations made different choices when handling them. Some reply with a RST, some remain silent, and others produce subtle variations in their responses.

Also, certain stacks historically left non-zero values in fields such as the urgent pointer even when the corresponding flag was not set. These behaviors are harmless but become highly distinctive fingerprints.

How Nmap Scores a Fingerprint

The fingerprint is compared against the nmap-os-db database using a weighted scoring system.

The weighting shows why some common OS fingerprinting assumptions are misleading. Indeed, TTL/Window size are easy to observe and modify but have relatively little impact on the final score.

Defeating Nmap OS Fingerprinting

Once you understand how Nmap builds a fingerprint, defeating it becomes a matter of controlling those signals.

Some parts of the fingerprint are simple packet-level attributes: TTL, TCP window size, or TCP option ordering can be modified directly. Others are the result of internal kernel behavior: sequence number generation, timestamp progression, IP ID allocation, and error handling. These cannot be changed reliably without reproducing the state that normally exists inside the operating system.

There are two main approaches to building an OS deception engine.

Userspace Network Stack Emulation

The first approach is to replace the kernel TCP/IP stack entirely with a userspace implementation. Every packet is generated by the deception engine itself, providing complete control over all observable network behavior. This is the model used by projects such as Honeyd.

The advantage is maximum control over the fingerprint. The deception engine can make a system appear to be any operating system from a network perspective.

The trade-off is that the kernel TCP stack is no longer involved. The engine is responsible for implementing TCP behavior that is normally provided by the operating system. If the goal is to expose a realistic SSH, SMB, or HTTP service behind a custom TCP stack, the userspace stack cannot simply forward traffic to an existing service. The kernel does not know about the emulated TCP connection, so a normal application cannot attach to it using a standard socket.

Native TCP/IP Stack with Packet Modification

The second approach is to let the operating system handle the connection normally and modify packets before they leave the machine. Applications continue running normally, while selected parts of the fingerprint are rewritten on the fly. The challenge is that not every attribute is equally easy to modify.

This approach provides less control than a full userspace TCP stack, because some kernel-generated behaviors are difficult or impossible to rewrite consistently. However, it preserves the realism of native services and avoids rebuilding the entire networking stack.

For a deception platform, keeping real services while controlling the most relevant fingerprinting signals is often the better engineering trade-off. It preserves realistic interactive services while the exposed network fingerprint can be manipulated without replacing replacing the native TCP/IP stack.

About Anantis

Anantis TrapEye is our advanced deception platform and OS masquerading is one layer of its deception model: our decoys present a coherent, unremarkable target that an attacker engages with long enough to reveal its intent.

Learn more

References

https://nmap.org/book/osdetect-methods.html https://raw.githubusercontent.com/nmap/nmap/refs/heads/master/nmap-os-db https://github.com/DataSoft/Honeyd