16 Troubleshooting Fundamentals

Troubleshooting BIG-IP means knowing what normal application traffic looks like — HTTP codes, TCP behavior, resource usage — and having a method. This note covers HTTP status codes, TCP connection problems, packet captures and their limits, resource utilization, and a step-by-step method for “the app is down/slow”.

HTTP status codes

Class	Meaning	Key codes
1xx	Informational	100 Continue — server got the headers, client should send the body
2xx	Success	200 OK
3xx	Redirection	301 Moved Permanently · 302 Found (used by the fallback host) · 304 Not Modified (unchanged since the last visit — use the cache)
4xx	Client error	400 Bad Request · 401 Unauthorized · 403 Forbidden · 404 Not Found · 405 Method Not Allowed (e.g. upload to a read-only site)
5xx	Server error	500 Internal Server Error · 502 Bad Gateway · 503 Service Unavailable · 504 Gateway Timeout · 511 Network Authentication Required

401 vs 403 vs 511

• 401 — the web server needs you to authenticate (the usual “you haven’t logged in”).
• 403 — you’re not allowed, authenticated or not.
• 511 — a network (captive portal, hotel Wi-Fi) requires login before giving access. The course quiz uses 511 for “access something that requires authentication”; know both.

• Who generated the code? BIG-IP can answer itself (fallback host, HTTP::respond, WAF blocking page). Compare client-side and server-side captures. A 5xx from the server = application problem; a reset or no answer = network, monitor or BIG-IP configuration.

HTTP headers worth knowing

Header	Meaning
Host	Requested domain (name-based virtual hosting, SNI-like routing in iRules)
User-Agent	Client browser or application
Cookie / Set-Cookie	Session state; BIG-IP’s persistence cookie BIGipServer…
Accept-Encoding / Content-Encoding	Compression negotiation (gzip, br)
Connection: keep-alive / close	Reuse or close the TCP connection
Content-Length / Transfer-Encoding: chunked	Body size
Location	Target of a 3xx redirect
X-Forwarded-For	Original client IP behind a proxy or SNAT
Cache-Control, ETag, If-Modified-Since	Caching (leads to 304 responses)

TCP connection problems

• FIN = “I’m done sending” → graceful close: FIN → ACK from each side.
• RST = abort immediately.
• Any HTTP participant (client, server, proxy) can close a TCP connection at any time without warning.
• Pipelining (several requests without waiting for answers) breaks when a connection closes mid-stream: pending requests are lost and must be retried.
• Improper closure leaves idle/half-open connections consuming memory until the idle timeout (TCP profile, default 300 s) removes them.

Failure to establish a connection

• Start with the simplest causes and go up the stack: physical → L2 → L3 → L4 → application.
• Ping works but the website doesn’t → ICMP is fine; suspect a filter/firewall on the port, the service being down, or the app itself (or a return path that bypasses BIG-IP).
• RST from BIG-IP → tmsh show /net rst-cause (connection limit exceeded, no pool member available, port denied, no flow found for ACK…).

Statistics first

Before capturing, read the counters:

Counter	Tells you
VS current/total connections	Clients are arriving (Total not growing → traffic doesn’t reach BIG-IP)
VS bits/packets in vs out	Traffic flows both ways (“In” only → servers aren’t answering)
Member current connections	How load is distributed; 0 may mean down/disabled
Interface errors/drops/collisions	Physical or duplex problems
Availability column	Monitor status

Statistics exist per virtual server, pool and each member. Reset them before a test to read clean numbers.

Packet captures (tcpdump)

BIG-IP includes tcpdump, the Linux capture tool. Key options:

Option	Meaning
-i <iface>	Interface/VLAN: 0.0 (all), external, internal, 1.1
-i 0.0:nnn	Adds TMM “noise” info (VS, flow); :nnnp also shows the peer (server-side) flow
-w file.pcap	Write a binary capture for Wireshark
-r file.pcap	Read a capture
-nn · -s0	No name/port resolution · capture the full packet
Filters	host 10.10.101.103 and port 80

Limits

• Best effort — under heavy load tcpdump can miss packets, and it adds load (it’s a Linux process receiving copies of packets from TMM). On a busy BIG-IP, captures can slow things down further; keep them short and filtered.
• FastL4 + PVA traffic is accelerated in hardware and doesn’t fully pass through TMM → captures are incomplete, not “faster and more accurate”.
• Encrypted traffic needs decryption (keys / --f5 ssl on recent versions).
• Save captures in /var/tmp or /shared/tmp.

Other analyzers

• Wireshark — open the .pcap; filters like http, tcp.flags.reset == 1, tcp.analysis.retransmission, tls.handshake.type == 1.
• ssldump — on BIG-IP, decodes the TLS handshake (and can decrypt RSA key exchange with the private key). Useful to see which version/cipher failed.
• curl -v / openssl s_client — test HTTP and the TLS handshake from the command line.

Reading a capture for performance problems

You see	Likely meaning
Big gap between request and response on the server side	Slow server/application
Gap between BIG-IP receiving the response and sending it to the client	BIG-IP processing (iRules, compression, SSL)
Retransmissions / duplicate ACKs	Packet loss on that segment
Zero window	Receiver can’t keep up
Slow TLS handshake	Key size, CPU, OCSP/CRL checks
SYN without SYN-ACK	Port closed/filtered, member down, routing

Resource utilization

• Control plane problems (high mcpd, bigd, tcpdump, GUI CPU) → slow GUI, flapping monitors. Data plane problems (high TMM CPU) → slow traffic.
• TMM memory is pre-allocated, so it always looks high — watch trends and allocation errors.
• High TMM CPU causes: heavy SSL, complex iRules, compression, DoS, too many monitors.
• Low disk space → failed hotfix installs, logging stops, slowness. Clean old images, QKviews, cores and UCS files.

Symptom	Check
Slow GUI, monitors flapping	top, bigd, disk
All apps slow	tmsh show sys performance system, interfaces, licensed throughput
One app slow	tmsh show ltm virtual (CPU per VS), its members, its iRules
Install/logging failures	df -h

Network-level causes of slowness

Duplex/speed mismatch, bad cables/optics, MTU problems, licensed throughput reached (e.g. 10 Mbps lab license), wrong TCP profile for the path, upstream congestion or asymmetric routing.

Method: “the app is down/slow”

1. Scope — one user or everyone? one app or all? since when? what changed?
2. VS status (Network Map) and VS counters: is traffic arriving?
3. Pool/members — which monitor failed? /var/log/ltm.
4. Client side — DNS resolves to the right VIP? firewall? certificate valid?
5. Server side — BIG-IP reaches the members? return path / SNAT correct?
6. Persistence, iRules, policies changing the path?
7. Resources and limits — CPU, memory, disk, connection limits, rst-cause.
8. Capture both sides and compare.
9. Escalate with evidence: QKview + captures + timeline.

Self-check

Q1 — Which HTTP class is reserved for redirection?

Respuesta

3xx.

Q2 — Which code tells the client the content hasn't changed since its last visit?

Respuesta

304.

Q3 — Which code would you expect when uploading a file to a read-only site?

Respuesta

405.

Q4 — Which TCP flag says "I'm ready to close and have nothing more to send"?

Respuesta

FIN.

Q5 — Which tcpdump flag writes a binary capture file, and which selects the interface?

Respuesta

-w writes the file; -i selects the interface.

Q6 — With FastL4 and PVA, do captures become faster and more accurate?

Respuesta

No. Accelerated traffic doesn’t fully pass through TMM, so captures are incomplete.

Q7 — A hotfix install failed and users report slow apps. Most likely cause?

Respuesta

Low disk space.

Q8 — Which commands show per-process usage, CPU per virtual server, and system CPU + memory?

Respuesta

top, tmsh show ltm virtual, tmsh show sys performance system.