Realm of Chaos - networking
Welcome to my notebook! I’m a Site Reliability Engineer who is curious, loves to learn and discover the nature of things.
Zola
2022-03-22T00:00:00+00:00
https://realmofchaos.xyz/tags/networking/atom.xml
Application Performance Analysis
2022-03-22T00:00:00+00:00
2022-03-22T00:00:00+00:00
Thomas Cuthbert
https://realmofchaos.xyz/tech/incident-response/
<p>In this post I will detail the investigation process I followed while responding to a request backlog alert.</p>
<p><em><strong>High level overview of the alerting service</strong></em></p>
<pre class="mermaid">
flowchart TD
HAProxy --> App --> Storage
</pre>
<script src="https://realmofchaos.xyz/js/mermaid-init.js"></script>
<p>I’ve redacted any identifiable information, however the methodolgy remains the same.</p>
<h2 id="frontend-haproxy">Frontend (HAProxy)</h2>
<p>The <code>halog</code> tool can generate lots of useful reports. Adjust the time slice so you have a decent buffer before and after the time of the incident.</p>
<p>The snippet below will generate a series of reports:</p>
<ul>
<li>URL averages</li>
<li>URL total times OK averages</li>
<li>Connect/response histogram <sup><a href="javascript:;" onclick="document.location.hash='#fn1';">1</a></sup></li>
</ul>
<p>The <code>-e</code> flag filters by errors.</p>
<pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="plain"><span class="giallo-l"><span>sudo cat /var/log/haproxy.log|halog -time "$(date '+%s' -d '6 hour ago')" -H -ua|awk 'NR==1;NR>1{print $0 | "sort -rnk4"}'|head -n25|column -t</span></span>
<span class="giallo-l"><span>sudo cat /var/log/haproxy.log|halog -e -time "$(date '+%s' -d '6 hour ago')" -H -uto|awk 'NR==1;NR>1{print $0 | "sort -rnk4"}'|head -n25|column -t</span></span>
<span class="giallo-l"><span>sudo cat /var/log/haproxy.log|halog -time "$(date '+%s' -d '6 hour ago')" -pct -q</span></span>
<span class="giallo-l"><span>sudo cat /var/log/haproxy.log|halog -time "$(date '+%s' -d '6 hour ago')" -H -srv|awk 'NR==1;NR>1{print $0 | "sort -rnk12"}'|column -t</span></span></code></pre><h2 id="application-server">Application server</h2>
<p>Capture 5 minutes worth of TCP header data. Limit pcap file size to 500M. Absolute sequence numbers <sup><a href="javascript:;" onclick="document.location.hash='#fn2';">2</a></sup> .</p>
<pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="plain"><span class="giallo-l"><span>intf=</span></span>
<span class="giallo-l"><span>timeout 300s tcpdump -i "${intf?}" -vvvSs 1500 -C500 -w cap_5m.pcap 'tcp && net 10.0.0.0/16'</span></span></code></pre>
<p>Analyse the data with <a rel="external" href="https://www.wireshark.org/docs/man-pages/tshark.html">tshark</a></p>
<pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="plain"><span class="giallo-l"><span>> cap.analysis \</span></span>
<span class="giallo-l"><span> tshark -r cap_5m.pcap -t ud \</span></span>
<span class="giallo-l"><span> -z \</span></span>
<span class="giallo-l"><span> endpoints,tcp \</span></span>
<span class="giallo-l"><span> -z \</span></span>
<span class="giallo-l"><span> expert,comment,tcp \</span></span>
<span class="giallo-l"><span> -T \</span></span>
<span class="giallo-l"><span> fields \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> _ws.col.Time \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> ip.src \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> ip.dst \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> tcp.dstport \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> _ws.expert.message \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> _ws.expert.severity \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> _ws.expert.group \</span></span>
<span class="giallo-l"><span> -e \</span></span>
<span class="giallo-l"><span> _ws.col.Info</span></span></code></pre>
<p>Identify any problematic flows from the <code>cap.analysis</code> file tshark generated. In my case, I observed a high rate of TCP Window Full responses from the server, indicating that the application was pushing too much data to it, or the server may be having systemic overload issues.</p>
<p>You can confirm your theory by monitoring tcp statistics over time.</p>
<p><strong>System level tcp stats.</strong></p>
<pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="plain"><span class="giallo-l"><span>while sleep 300; stdbuf -o0 nstat -p; done tee nstat.sout</span></span></code></pre>
<p><strong>Endpoint level statistics</strong></p>
<p><code>ip -s tcpmetrics</code> provides similar data, I prefer the socket stats tool <code>ss</code>.</p>
<pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="plain"><span class="giallo-l"><span>for i in {0..300}; do</span></span>
<span class="giallo-l"><span> echo === ; ss -in state synchronized 'dst 10.0.0.10 '; echo ===</span></span>
<span class="giallo-l"><span> sleep 1</span></span>
<span class="giallo-l"><span>done | tee sockets2.sout</span></span>
<span class="giallo-l"><span>awk '/ESTAB/&&/10.0.0/&&and($4,$5){print "";print;print "";getline;l=patsplit($0,f,/[^:[:blank:]]*/);for(i=1;i<=l;i+=2) printf "%24s %24s\n", f[i],f[i+1]}' sockets2.sout</span></span></code></pre>
<p>From the output I observed the following:</p>
<p>The application server misses too many acknowledgements and backs off from sending data (<code>backoff 1</code>).</p>
<pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="plain"><span class="giallo-l"><span> rto 208</span></span>
<span class="giallo-l"><span> rtt 4.714/0.643</span></span>
<span class="giallo-l"><span> ato 40</span></span>
<span class="giallo-l"><span> mss 1374</span></span>
<span class="giallo-l"><span> pmtu 1426</span></span>
<span class="giallo-l"><span> rcvmss 536</span></span>
<span class="giallo-l"><span> advmss 1374</span></span>
<span class="giallo-l"><span> cwnd 143</span></span>
<span class="giallo-l"><span> send 333.4Mbps</span></span>
<span class="giallo-l"><span> pacing_rate 400.1Mbps</span></span>
<span class="giallo-l"><span> delivery_rate 257.2Mbps</span></span>
<span class="giallo-l"><span> busy 11672ms</span></span>
<span class="giallo-l"><span> rwnd_limited 11388ms(97.6%)</span></span>
<span class="giallo-l"><span> unacked 126</span></span>
<span class="giallo-l"><span> retrans 0/12</span></span>
<span class="giallo-l"><span> reordering 74</span></span>
<span class="giallo-l"><span> notsent 1051110</span></span>
<span class="giallo-l"><span> minrtt 0.993</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> rto 216</span></span>
<span class="giallo-l"><span> backoff 1</span></span>
<span class="giallo-l"><span> rtt 14.491/16.568</span></span>
<span class="giallo-l"><span> ato 40</span></span>
<span class="giallo-l"><span> mss 1374</span></span>
<span class="giallo-l"><span> pmtu 1426</span></span>
<span class="giallo-l"><span> rcvmss 536</span></span>
<span class="giallo-l"><span> advmss 1374</span></span>
<span class="giallo-l"><span> cwnd 143</span></span>
<span class="giallo-l"><span> send 108.5Mbps</span></span>
<span class="giallo-l"><span> pacing_rate 130.2Mbps</span></span>
<span class="giallo-l"><span> delivery_rate 3.3Mbps</span></span>
<span class="giallo-l"><span> busy 12680ms</span></span>
<span class="giallo-l"><span> rwnd_limited 12360ms(97.5%)</span></span>
<span class="giallo-l"><span> retrans 0/16</span></span>
<span class="giallo-l"><span> reordering 74</span></span>
<span class="giallo-l"><span> notsent 1023662</span></span>
<span class="giallo-l"><span> minrtt 0.993</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> rto 208</span></span>
<span class="giallo-l"><span> rtt 5.519/0.312</span></span>
<span class="giallo-l"><span> ato 40</span></span>
<span class="giallo-l"><span> mss 1374</span></span>
<span class="giallo-l"><span> pmtu 1426</span></span>
<span class="giallo-l"><span> rcvmss 536</span></span>
<span class="giallo-l"><span> advmss 1374</span></span>
<span class="giallo-l"><span> cwnd 217</span></span>
<span class="giallo-l"><span> send 432.2Mbps</span></span>
<span class="giallo-l"><span> pacing_rate 518.6Mbps</span></span>
<span class="giallo-l"><span> delivery_rate 254.7Mbps</span></span>
<span class="giallo-l"><span> busy 15700ms</span></span>
<span class="giallo-l"><span> rwnd_limited 15324ms(97.6%)</span></span>
<span class="giallo-l"><span> unacked 133</span></span>
<span class="giallo-l"><span> retrans 0/17</span></span>
<span class="giallo-l"><span> reordering 74</span></span>
<span class="giallo-l"><span> notsent 1719216</span></span>
<span class="giallo-l"><span> minrtt 0.993</span></span></code></pre>
<p>We can get richer statistics by analysing the packet capture with tshark. The output is easily grep’d. My favourite statistic report is the io,stat table, you pass it packet fields to aggregate over and produce a configurable histogram. In my example below I created a per second packet rate and window size histogram.</p>
<pre class="giallo" style="color: #F8F8F2; background-color: #272822;"><code data-lang="plain"><span class="giallo-l"><span>tshark -r cap_5m.pcap -q -z http_srv,tree</span></span>
<span class="giallo-l"><span>tshark -r cap_5m.pcap -q -z http_srv,tree -z conv,tcp</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>grep -P 'Window Full|PUT' cap.analysis|head;</span></span>
<span class="giallo-l"><span>tshark -n -q -r *.pcap -z io,stat,1,"COUNT(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport == 8080","MIN(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport == 8080","MAX(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport == 8080","AVG(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport == 8080" |grep -v ' 0 '</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>2022-03-22 09:45:08.845938 10.0.1.100 10.0.0.10 8080 PUT /api/resource HTTP/1.1\r\n 2097152 33554432 PUT /api/resource HTTP/1.1</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.116693 10.0.1.100 10.0.0.10 8080 PUT /api/resource HTTP/1.1 [TCP segment of a reassembled PDU]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.245875 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.250035 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.250133 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.253985 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.255907 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.257952 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.262310 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"><span>2022-03-22 09:45:09.268328 10.0.1.100 10.0.0.10 8080 TCP window specified by the receiver is now completely full 6291456 33554432 [TCP Window Full] Continuation[Packet size limited during capture]</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>======================================================================================================</span></span>
<span class="giallo-l"><span>| IO Statistics |</span></span>
<span class="giallo-l"><span>| |</span></span>
<span class="giallo-l"><span>| Duration: 270.938461 secs |</span></span>
<span class="giallo-l"><span>| Interval: 1 secs |</span></span>
<span class="giallo-l"><span>| |</span></span>
<span class="giallo-l"><span>| Col 1: COUNT(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport |</span></span>
<span class="giallo-l"><span>| == 8080 |</span></span>
<span class="giallo-l"><span>| 2: MIN(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport == |</span></span>
<span class="giallo-l"><span>| 8080 |</span></span>
<span class="giallo-l"><span>| 3: MAX(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport == |</span></span>
<span class="giallo-l"><span>| 8080 |</span></span>
<span class="giallo-l"><span>| 4: AVG(tcp.window_size_value)tcp.window_size_value and ip.addr==10.0.0.10 && tcp.dstport == |</span></span>
<span class="giallo-l"><span>| 8080 |</span></span>
<span class="giallo-l"><span>|----------------------------------------------------------------------------------------------------|</span></span>
<span class="giallo-l"><span>| |1 |2 |3 |4 | |</span></span>
<span class="giallo-l"><span>| Interval | COUNT | MIN | MAX | AVG | |</span></span>
<span class="giallo-l"><span>|--------------------------------------------| |</span></span>
<span class="giallo-l"><span>| 9 <> 10 | 898 | 508 | 65142 | 651 | |</span></span>
<span class="giallo-l"><span>| 10 <> 11 | 849 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 11 <> 12 | 5 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 24 <> 25 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 25 <> 26 | 51 | 507 | 65142 | 3042 | |</span></span>
<span class="giallo-l"><span>| 32 <> 33 | 6 | 486 | 62314 | 10791 | |</span></span>
<span class="giallo-l"><span>| 33 <> 34 | 91 | 486 | 62314 | 1844 | |</span></span>
<span class="giallo-l"><span>| 34 <> 35 | 5 | 485 | 486 | 485 | |</span></span>
<span class="giallo-l"><span>| 38 <> 39 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 47 <> 48 | 121 | 508 | 65142 | 2110 | |</span></span>
<span class="giallo-l"><span>| 48 <> 49 | 80 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 56 <> 57 | 4 | 508 | 65142 | 16667 | |</span></span>
<span class="giallo-l"><span>| 57 <> 58 | 22 | 507 | 65142 | 6383 | |</span></span>
<span class="giallo-l"><span>| 82 <> 83 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 83 <> 84 | 43 | 507 | 65142 | 3514 | |</span></span>
<span class="giallo-l"><span>| 98 <> 99 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 99 <> 100 | 42 | 507 | 65142 | 3585 | |</span></span>
<span class="giallo-l"><span>| 100 <> 101 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 102 <> 103 | 7 | 508 | 65142 | 18975 | |</span></span>
<span class="giallo-l"><span>| 103 <> 104 | 164 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 105 <> 106 | 13 | 508 | 65142 | 15424 | |</span></span>
<span class="giallo-l"><span>| 106 <> 107 | 34 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 115 <> 116 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 117 <> 118 | 556 | 508 | 65142 | 740 | |</span></span>
<span class="giallo-l"><span>| 118 <> 119 | 5 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 127 <> 128 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 128 <> 129 | 4 | 508 | 65142 | 16667 | |</span></span>
<span class="giallo-l"><span>| 129 <> 130 | 501 | 508 | 65142 | 637 | |</span></span>
<span class="giallo-l"><span>| 131 <> 132 | 5 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 159 <> 160 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 161 <> 162 | 4 | 508 | 65142 | 16667 | |</span></span>
<span class="giallo-l"><span>| 162 <> 163 | 149 | 507 | 65142 | 941 | |</span></span>
<span class="giallo-l"><span>| 187 <> 188 | 4 | 508 | 65142 | 16667 | |</span></span>
<span class="giallo-l"><span>| 188 <> 189 | 2 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 225 <> 226 | 10 | 508 | 65142 | 13435 | |</span></span>
<span class="giallo-l"><span>| 226 <> 227 | 76 | 507 | 65142 | 1358 | |</span></span>
<span class="giallo-l"><span>| 240 <> 241 | 6 | 508 | 65142 | 11280 | |</span></span>
<span class="giallo-l"><span>| 242 <> 243 | 444 | 508 | 65142 | 799 | |</span></span>
<span class="giallo-l"><span>| 243 <> 244 | 7 | 508 | 65142 | 18975 | |</span></span>
<span class="giallo-l"><span>| 244 <> 245 | 898 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 245 <> 246 | 496 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 246 <> 247 | 516 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 247 <> 248 | 496 | 508 | 65142 | 638 | |</span></span>
<span class="giallo-l"><span>| 248 <> 249 | 482 | 508 | 65142 | 776 | |</span></span>
<span class="giallo-l"><span>| 249 <> 250 | 485 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 250 <> 251 | 602 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 251 <> 252 | 629 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 252 <> 253 | 664 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 253 <> 254 | 519 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 254 <> 255 | 719 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 255 <> 256 | 646 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 256 <> 257 | 764 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 257 <> 258 | 376 | 508 | 508 | 508 | |</span></span>
<span class="giallo-l"><span>| 260 <> 261 | 11 | 507 | 65142 | 6383 | |</span></span>
<span class="giallo-l"><span>| 263 <> 264 | 51 | 508 | 65142 | 4310 | |</span></span>
<span class="giallo-l"><span>| 264 <> 265 | 19 | 507 | 508 | 507 | |</span></span>
<span class="giallo-l"><span>| 267 <> 268 | 3 | 509 | 65142 | 22053 | |</span></span>
<span class="giallo-l"><span>| 268 <> 269 | 7 | 508 | 65142 | 9741 | |</span></span>
<span class="giallo-l"><span>| 269 <> 270 | 80 | 507 | 65142 | 2123 | |</span></span>
<span class="giallo-l"><span>| 270 <> Dur | 17 | 507 | 65142 | 8112 | |</span></span>
<span class="giallo-l"><span>======================================================================================================</span></span></code></pre><h2 id="next-steps">Next steps</h2>
<p>At this point it’s best to raise a bug with the developers. In the mean time, spawning additional application workers to spread the queue around may help ungrease the wheel a bit.</p>
<p>Longterm we should confirm the retransmissions aren’t caused by poor network conditions. The application could be patched to establish a long lived connection while pushing data into object storage. It currently creates a new session per push. Longer connection life-times would ensure the TCP congestion algorithm heuristics have time to stabilise.</p>
<p>Footnotes</p>
<p>1.</p>
<p><code><percentile> <request count> <Request Time*> <Connect Time**> <Response Time***> <Data Time****></code></p>
<p>2.</p>
<p>TCP sequence numbers are generally unhelpful when capturing packets from a server. This is due to segment offloading to the NIC.</p>