Chapter 2.2 - IDS/IPS: Signatures, Anomaly Detection & Evasion

Module 2: Traffic Analysis & Intrusion Detection Prerequisites: Chapter 2.1 (Packet Analysis & Protocol Dissection)

---

Table of Contents

1. Detection Philosophy: IDS vs IPS
2. Deployment Architectures
3. Signature-Based Detection
4. Anomaly & Behavioral Detection
5. Snort: Rules, Preprocessors & Tuning
6. Suricata: Multi-Threading, EVE JSON & Lua Scripting
7. Evasion Techniques
8. Detection Engineering Workflow
9. MITRE ATT&CK Mapping

---

1. Detection Philosophy: IDS vs IPS

An Intrusion Detection System (IDS) is a passive observer - it mirrors traffic, inspects it, and raises alerts without modifying the traffic flow. An Intrusion Prevention System (IPS) sits inline; it can drop, reset, or modify packets in real time. The choice is not purely technical - it's also organizational risk tolerance. A misconfigured IPS that generates false positives becomes a self-inflicted denial-of-service. Most enterprise environments start with IDS, validate rules with low false-positive rates, then promote to IPS mode on high-confidence signatures only.

Both paradigms rely on the same detection engines underneath. What changes is the enforcement hook: IDS reads from a tap or SPAN port, while IPS inserts itself into the forwarding path using techniques like NFQUEUE (Linux Netfilter), inline interface pairs, or bump-in-the-wire network appliances.

NIDS vs HIDS:

• Network IDS (NIDS) operates on raw traffic - sees everything traversing a network segment, but has no visibility into host state (process memory, syscalls, file changes).
• Host IDS (HIDS) runs as an agent on the endpoint - deep host visibility, but sees only that host's traffic and is subject to tampering if the host is compromised.

In mature environments these layers are complementary. A NIDS detects lateral movement across the wire; a HIDS catches the resulting process execution on the target host.

---

2. Deployment Architectures

Tap vs SPAN

Method	Pros	Cons
Network TAP (hardware)	Passive, zero packet loss, cannot be detected or disrupted	Hardware cost, limited to physical links
SPAN / Mirror Port	Cheap, flexible, software-configurable on managed switches	Can drop packets under load, limited to switch-local traffic
Inline (bump-in-wire)	Active blocking possible, full packet access	Single point of failure, latency added, fail-open/fail-closed decision required
Virtual TAP / vSwitch	Works in virtualized/cloud environments	Hypervisor overhead, vendor-specific config

Placement Strategy

Placing a single NIDS at the perimeter misses east-west traffic entirely. The modern approach segments sensors across trust boundaries:

Internet -->[Perimeter Firewall] --> DMZ Sensor
                               --> [Core Switch] --> Internal Sensor
                                                 --> [DC Fabric] --> Data Center Sensor

Each sensor reports to a central SIEM. Correlation across sensors is where sophisticated multi-stage attacks become visible - a port scan from the perimeter sensor followed by authentication anomalies on the internal sensor is a kill-chain signal invisible to either sensor alone.

---

3. Signature-Based Detection

Signatures match known-bad patterns: byte sequences, protocol field values, behavioral fingerprints. Detection is fast and precise for known threats. The weaknesses are equally well-known: zero-days produce no signature hits; minor payload mutations break exact matches.

How Signatures Work

A signature encodes:

1. Protocol context - TCP, UDP, ICMP, application layer (HTTP, DNS, TLS)
2. Direction - client-to-server, server-to-client, or bidirectional
3. Content match - byte patterns, regex, protocol field values
4. Threshold / frequency - detect on N occurrences in T seconds

False Positive vs False Negative Tradeoff

	Alert fires	Alert does not fire
Attack present	True Positive	False Negative
No attack	False Positive	True Negative

Signature sensitivity is tuned by adjusting content specificity and threshold. A signature matching User-Agent: Mozilla fires on virtually all HTTP traffic (useless). A signature matching a specific 16-byte shellcode sequence is highly precise but misses polymorphic variants.

---

4. Anomaly & Behavioral Detection

Anomaly detection establishes a statistical baseline of "normal" and alerts when observed behavior deviates beyond a threshold. It is the only mechanism with a theoretical chance of catching novel attacks.

Statistical Methods

• Volume-based: bytes/sec, packets/sec, flows/sec per host or segment. A host suddenly sending 10x its baseline egress is anomalous.
• Protocol conformance: RFC-compliant parsers flag malformed headers. A TCP segment with both SYN and FIN set simultaneously is RFC-illegal - either a scanner or an evasion attempt.
• Entropy analysis: DNS query labels with high entropy (e.g., a3f9c2b1d4e7.evil.com) suggest DNS tunneling or DGA (Domain Generation Algorithm) traffic. Shannon entropy:

H = -Sum p(x) * log2(p(x))

Normal hostnames score 2.5-3.5 bits/char. DGA domains typically score 3.8-4.2.

• Flow behavior profiling: src_port, dst_port, byte_count, packet_count, duration tuples per source IP. NetFlow / IPFIX data feeds ML models (isolation forests, autoencoders) that flag outlier flows without inspecting payload.

Behavioral Detection

Rather than detecting a specific exploit, behavioral rules fire on patterns of activity:

• Horizontal scan: one IP contacts >N unique IPs on the same port in T seconds
• Vertical scan: one IP contacts >N unique ports on the same destination in T seconds
• Beaconing: periodic outbound connections at fixed intervals (C2 check-in)
• Exfiltration spike: upload-to-download ratio reversal for a host that normally consumes more than it produces

Behavioral detection produces fewer actionable alerts than signature detection but catches evasive, slow-burn campaigns that deliberately stay below per-signature thresholds.

---

5. Snort: Rules, Preprocessors & Tuning

Snort is the reference NIDS implementation. Its rule language is widely cloned (Suricata accepts Snort-compatible rules). Understanding Snort rule anatomy is foundational.

Rule Structure

action proto src_ip src_port direction dst_ip dst_port (options)

Full anatomy of a real rule:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (
    msg:"ET WEB_SERVER SQL Injection attempt -- select from";
    flow:established,to_server;
    content:"select"; nocase; http_uri;
    content:"from"; nocase; http_uri; distance:0;
    pcre:"/select.{0,30}from/Ui";
    classtype:web-application-attack;
    sid:2006445;
    rev:9;
    metadata:affected_product Web_Server_Applications,
              attack_target Web_Server,
              mitre_tactic TA0001;
)

Field	Meaning
alert	Generate alert (other actions: drop, reject, pass, log)
tcp	Match TCP protocol
$EXTERNAL_NET	Variable defined in snort.conf; typically !$HOME_NET
flow:established,to_server	Match only on packets within an established TCP session, in the client->server direction
content:"select"; nocase; http_uri	Case-insensitive byte match in the HTTP URI buffer (post-normalization)
distance:0	Second content must appear immediately after first
pcre:"/select.{0,30}from/Ui"	Perl-compatible regex; U = HTTP URI buffer, i = case-insensitive
classtype	Assigns priority class for alerting
sid	Unique rule ID

Preprocessors / Inspectors

Raw TCP reassembly and application-layer normalization happen in preprocessors before rules execute. This is critical: a content match against a fragmented or out-of-order stream would miss attacks split across segments.

Key preprocessors:

• stream5 - TCP/UDP stream reassembly and session tracking
• frag3 - IP defragmentation (prevents fragment overlap attacks)
• http_inspect - HTTP normalization: decodes %20, //, ..%2F, chunk encoding, etc.
• dcerpc2 - SMB/DCERPC parsing (detects EternalBlue-style exploits)
• sfportscan - Detects horizontal/vertical scan patterns

# Run Snort in IDS mode on interface eth0, log to /var/log/snort
snort \
  -i eth0 \                          # capture interface
  -c /etc/snort/snort.conf \         # main config (includes rules, preprocessors, vars)
  -A fast \                          # alert format: fast (one-line), full, unified2
  -l /var/log/snort \                # log directory
  -D \                               # daemonize
  -u snort -g snort                  # drop privileges after startup

# Test a rule file for syntax errors without capturing live traffic
snort -T -c /etc/snort/snort.conf

# Replay a pcap against Snort rules (offline analysis)
snort \
  -r capture.pcap \                  # read from pcap instead of live interface
  -c /etc/snort/snort.conf \
  -A console \                       # print alerts to stdout
  -l /tmp/snort_replay

Tuning: Suppression and Thresholding

Untuned Snort on a busy network produces thousands of alerts/hour, most of which are false positives or low-value noise. Suppression and thresholding are the primary tools:

# threshold.conf -- suppress a specific sid from a specific source entirely
suppress gen_id 1, sig_id 2006445, track by_src, ip 10.10.10.5

# Rate-limit: only alert once per 60 seconds per source IP
threshold gen_id 1, sig_id 2006445, type limit, track by_src, count 1, seconds 60

# Alert only when threshold is exceeded: 10 hits in 5 seconds
threshold gen_id 1, sig_id 1000001, type threshold, track by_src, count 10, seconds 5

---

6. Suricata: Multi-Threading, EVE JSON & Lua Scripting

Suricata is the production-grade evolution of the Snort model. Key differentiators:

Feature	Snort 2.x	Snort 3.x	Suricata
Multi-threading	No (single-threaded)	Yes	Yes (worker threads per CPU)
Protocol detection	Port-based	Port-based	Port-agnostic (deep protocol ID)
TLS inspection	Limited	Limited	JA3/JA4 fingerprinting, cert extraction
Output	unified2 binary	unified2 / JSON	EVE JSON (rich structured logs)
Lua scripting	No	Limited	Yes - full Lua rule engine
File extraction	No	No	Yes (HTTP, SMB, FTP, NFS)

EVE JSON Output

Suricata's EVE (Extensible Event Format) JSON log is the primary integration point with SIEMs and Elastic Stack. Every event type (alert, flow, dns, http, tls, files) emits a structured JSON record:

{
  "timestamp": "2024-11-15T14:23:01.882342+0000",
  "event_type": "alert",
  "src_ip": "192.168.1.50",
  "src_port": 54821,
  "dest_ip": "203.0.113.44",
  "dest_port": 4444,
  "proto": "TCP",
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2100498,
    "rev": 7,
    "signature": "GPL ATTACK_RESPONSE id check returned root",
    "category": "Potentially Bad Traffic",
    "severity": 2
  },
  "flow": {
    "pkts_toserver": 6,
    "pkts_toclient": 9,
    "bytes_toserver": 472,
    "bytes_toclient": 1841
  },
  "payload": "aWQKdWlkPTAocm9vdCkgZ2lkPTAocm9vdCkK",
  "payload_printable": "id\nuid=0(root) gid=0(root)\n"
}

Suricata CLI

# Start Suricata in IDS mode; AF_PACKET for high-performance packet capture
suricata \
  -c /etc/suricata/suricata.yaml \   # main config
  --af-packet=eth0 \                 # AF_PACKET mode (kernel bypass lite)
  -D \                               # daemonize
  --pidfile /var/run/suricata.pid

# Replay pcap offline (useful for testing new rules)
suricata \
  -c /etc/suricata/suricata.yaml \
  -r suspicious_traffic.pcap \       # offline pcap
  -l /tmp/suricata_output            # EVE JSON written here

# Live-tail alerts from EVE JSON
tail -f /var/log/suricata/eve.json \
  | jq 'select(.event_type == "alert") | {ts: .timestamp, sig: .alert.signature, src: .src_ip}'

# Update Emerging Threats ruleset
suricata-update                      # pulls latest ET Open rules, rebuilds rule index
suricata-update --reload-command "kill -USR2 $(cat /var/run/suricata.pid)"

Lua Scripting Example

Lua rules execute arbitrary detection logic beyond what the rule keyword language supports. Useful for stateful detection across multiple packets:

-- detect_base64_dns_tunnel.lua
-- Fires if a DNS query name is >40 chars and entropy > 3.8

function init(args)
    local needs = {}
    needs["payload"] = tostring(true)  -- request access to packet payload
    return needs
end

function match(args)
    local payload = args["payload"]
    if not payload then return 0 end

    -- extract DNS QNAME (simplified: skip 12-byte header)
    local qname = payload:sub(13)
    if #qname < 40 then return 0 end

    -- Shannon entropy calculation
    local counts = {}
    for i = 1, #qname do
        local c = qname:sub(i, i)
        counts[c] = (counts[c] or 0) + 1
    end
    local entropy = 0
    for _, count in pairs(counts) do
        local p = count / #qname
        entropy = entropy - p * math.log(p) / math.log(2)
    end

    if entropy > 3.8 then return 1 end  -- 1 = match, 0 = no match
    return 0
end

-- Rule referencing the Lua script
alert dns any any -> any 53 (
    msg:"Possible DNS Tunnel - High Entropy Query";
    lua:detect_base64_dns_tunnel.lua;
    sid:9900001;
    rev:1;
)

---

7. Evasion Techniques

Evasion is the adversary's answer to signature detection. Understanding evasion is mandatory for writing effective rules and for understanding the hard limits of signature-based systems.

Architecture of Evasion

TCP Fragmentation & Reassembly Desync

The classic Ptacek-Newsham evasion: send the same byte range in two overlapping fragments with different content. The IDS sees fragment A (benign), the endpoint reassembles using fragment B (malicious) based on the OS's overlap policy.

# fragroute -- intercept and fragment outbound packets
# /etc/fragroute.conf:
# ip_frag 8            # fragment into 8-byte chunks
# tcp_seg 1            # segment TCP into 1-byte chunks

fragroute -f /etc/fragroute.conf 192.168.1.100

# scapy -- craft overlapping fragments manually
python3 << 'EOF'
from scapy.all import *

target = "192.168.1.100"
payload_a = b"GET /safe"          # fragment 1: benign
payload_b = b"GET /exploit"       # fragment 2: same offset, malicious content
# Note: endpoint OS reassembly policy determines which wins

p1 = IP(dst=target, flags="MF", frag=0) / TCP(dport=80) / payload_a[:8]
p2 = IP(dst=target, frag=0) / TCP(dport=80) / payload_b[:8]  # overlaps frag 0
send([p1, p2], verbose=False)
EOF

Defense: Modern preprocessors (frag3 in Snort, defrag engine in Suricata) implement per-OS reassembly policies and resolve overlaps consistently with the target OS, eliminating the desync window - provided the target OS is correctly identified in the preprocessor config.

HTTP Evasion

Web application attacks are particularly susceptible to normalization evasion because HTTP allows many equivalent representations:

/admin/../admin/         -> /admin/           (path traversal collapse)
/%61dmin/                -> /admin/           (%61 = 'a' in hex)
/ADMIN/                  -> /admin/           (case normalization)
/%252F                   -> /%2F -> /          (double-encoding)
/admin%09/               -> /admin/           (tab character)

A signature matching content:"/etc/passwd" misses /etc%2Fpasswd, /%65%74%63/passwd, /etc/./passwd, etc. IDS HTTP preprocessors must normalize before matching.

# Test normalization coverage with nikto
nikto \
  -h http://192.168.1.100 \
  -evasion 1 \    # random URI encoding
  -evasion 2 \    # directory self-reference /./
  -evasion 4      # premature URL ending

Insertion Attacks

Send packets the IDS processes but the endpoint rejects - inserting "noise" into the stream to break content matches:

• TCP segments with invalid checksums (endpoint discards them; some IDS engines process them)
• Packets with expired TTLs (reach IDS but not endpoint)
• TCP RSTs mid-stream (endpoint closes connection; IDS may continue tracking)

# scapy: send a packet with TTL=1 (dies at first hop, never reaches target)
# If IDS is behind that hop, it processes the packet; target never sees it
from scapy.all import *
p = IP(dst="192.168.1.100", ttl=1) / TCP(dport=80, flags="A") / b"BADCONTENT"
send(p)

Defense: IDS sensors must be placed as close to the target as possible to minimize TTL manipulation windows. Suricata's stream-reassembly engine validates checksums; configure checksum-validation: yes in suricata.yaml.

Slow / Low-and-Slow Scans

# nmap slow scan: 1 probe per ~15 minutes, randomized order
nmap \
  -sS \                     # SYN scan
  -T0 \                     # paranoid timing: 5 min between probes
  --randomize-hosts \       # randomize target order
  --data-length 15 \        # append random data to packets (breaks length signatures)
  --ttl 64 \
  192.168.1.0/24

# hping3: craft custom timing
hping3 \
  -S \                      # SYN flag
  -p 22 \                   # destination port
  -i u1000000 \             # interval: 1 second between packets
  192.168.1.100

Threshold-based detection (fire after N probes in T seconds) has a direct bypass: reduce probe rate below the threshold. Behavioral detection with longer time windows and per-source tracking is required.

Encryption as Evasion (T1573)

TLS 1.3 with forward secrecy renders payload inspection impossible without a TLS inspection proxy. Attackers route C2 traffic over HTTPS/443 to legitimate-looking domains:

• Domain Fronting: route traffic through a CDN (Cloudflare, AWS CloudFront); the SNI/Host header points to a legitimate domain while the actual content is proxied to the C2
• HTTPS C2: Cobalt Strike, Covenant, Sliver all support full TLS C2 channels
• Encrypted DNS: DNS-over-HTTPS (DoH) routes DNS queries over HTTPS, bypassing DNS-based detection

Without payload access, detection falls back on metadata:

• JA3/JA4 fingerprinting: hash of TLS ClientHello parameters (cipher suites, extensions, elliptic curves) - fingerprints TLS client library independently of payload
• Certificate analysis: self-signed certs, short validity windows, newly registered domains
• Flow timing analysis: beacon periodicity, jitter patterns

# Extract JA3 fingerprints from pcap using zeek
zeek -r capture.pcap /opt/zeek/share/zeek/policy/protocols/ssl/ja3.zeek
cat ssl.log | zeek-cut ja3 ja3s server_name

# ja3 via python-ja3
pip install pyja3
python3 -m ja3 --json capture.pcap | jq '.[] | select(.ja3_digest == "51c64c77e60f3980eea90869b68c58a8")'
# Known Cobalt Strike default JA3: 51c64c77e60f3980eea90869b68c58a8 (changes with malleable profiles)

---

8. Detection Engineering Workflow

Writing and deploying rules is not a one-time task - it is a continuous engineering loop. Alert quality degrades as the environment changes; rules written for last year's attack patterns may be tuned into silence while new techniques go undetected.

Building a Test Lab for Rule Validation

# Replay attack PCAPs against Suricata offline -- no production risk
# 1. Download known-malicious PCAP
wget https://malware-traffic-analysis.net/2024/01/15/2024-01-15-Formbook-infection-traffic.pcap.zip
unzip -P infected 2024-01-15*.zip

# 2. Run Suricata against it
suricata \
  -c /etc/suricata/suricata.yaml \
  -r 2024-01-15-Formbook-infection-traffic.pcap \
  -l /tmp/test_output \
  -S /etc/suricata/rules/emerging-malware.rules  # load specific ruleset only

# 3. Check what fired
cat /tmp/test_output/eve.json \
  | jq 'select(.event_type=="alert") | .alert.signature' \
  | sort | uniq -c | sort -rn

# 4. Check for missed events (what a competing ruleset catches that yours doesn't)
# Compare against ET Pro or Snort VRT results

Rule Performance Profiling

# Suricata rule profiling -- identifies slowest rules
# In suricata.yaml, enable:
#   profiling:
#     rules:
#       enabled: yes
#       filename: rule_perf.log
#       append: yes
#       sort: avgticks

# After a run, examine top CPU consumers
cat /var/log/suricata/rule_perf.log | head -30
# Slow rules are usually PCRE-heavy; optimize by adding fast_pattern content match first
# fast_pattern tells Suricata to use the specified content for the multi-pattern matcher
# before evaluating the full rule

alert http any any -> any any (
    msg:"Slow PCRE - optimized with fast_pattern";
    content:"cmd.exe"; fast_pattern; http_uri;  # MPSE matches this first
    pcre:"/cmd\.exe(\s|%20|\+)\/[cC]/U";        # PCRE only runs if fast_pattern hit
    sid:9900002;
)

---

9. MITRE ATT&CK Mapping

Technique	ATT&CK ID	Detection Method	Key Rule/Logic
Network Service Scanning	T1046	Behavioral (port scan thresholds)	sfportscan preprocessor; flow count per src
Exploit Public-Facing App	T1190	Signature (HTTP/SQL injection patterns)	ET rules 2006445 family
C2 over HTTPS	T1071.001	TLS metadata (JA3, cert analysis)	JA3 blacklist; beacon periodicity
DNS Tunneling (C2)	T1071.004	Entropy analysis; query length	Lua entropy script; dns_query length threshold
Protocol Tunneling	T1572	Protocol anomaly	Non-HTTP on port 80/443
Traffic Signaling (covert C2)	T1205	Behavioral (timing analysis)	NetFlow jitter analysis
Obfuscated Files / Info	T1027	Content inspection (base64, compression headers in unexpected contexts)	file_data buffer inspection
Indicator Removal: Clear Logs	T1070	HIDS (file integrity monitoring)	Out of scope for NIDS
Defragmentation Evasion	T1599	Preprocessor normalization	frag3 / defrag engine
Encrypted Channel	T1573	TLS fingerprinting; cert inspection	JA3; Suricata TLS app-layer