You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 15:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Wed Aug 27, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350970,0irq/25-eth10-21swapper/306:41:303
315524839,0sleep20-21swapper/206:41:472
301223525,0sleep10-21swapper/106:40:141
289323423,0sleep00-21swapper/006:38:530
339970,0migration/318018-21munin-node10:13:123
33349960,0cyclictest1451-21diskmemload09:13:282
33349960,0cyclictest1451-21diskmemload09:13:282
33349960,0cyclictest0-21swapper/209:55:242
33349960,0cyclictest0-21swapper/207:03:142
33279965,0cyclictest10606-21cut08:48:081
33279960,0cyclictest7885-21ssh11:49:531
33279960,0cyclictest1451-21diskmemload08:45:451
33269965,0cyclictest16058-21ps09:33:060
339960,0migration/39205-21sh11:12:573
339960,0migration/34275-21munin-node12:08:083
339960,0migration/318745-21kthreadcore11:18:063
339960,0migration/315515-21sh08:48:163
339960,0migration/315456-21sh11:33:423
28415060,0irq/27-eth0-tx-0-21swapper/211:55:592
28415060,0irq/27-eth0-tx-0-21swapper/211:38:282
28415060,0irq/27-eth0-tx-0-21swapper/211:35:232
28415060,0irq/27-eth0-tx-0-21swapper/211:22:292
28415060,0irq/27-eth0-tx-0-21swapper/211:07:232
28415060,0irq/27-eth0-tx-0-21swapper/210:55:382
28415060,0irq/27-eth0-tx-0-21swapper/210:51:112
28415060,0irq/27-eth0-tx-0-21swapper/210:43:412
28415060,0irq/27-eth0-tx-0-21swapper/210:39:462
28415060,0irq/27-eth0-tx-0-21swapper/210:06:162
28415060,0irq/27-eth0-tx-0-21swapper/209:49:152
28415060,0irq/27-eth0-tx-0-21swapper/209:24:162
28415060,0irq/27-eth0-tx-0-21swapper/209:18:192
28415060,0irq/27-eth0-tx-0-21swapper/209:06:082
28415060,0irq/27-eth0-tx-0-21swapper/208:53:082
28415060,0irq/27-eth0-tx-0-21swapper/007:28:120
28415060,0irq/27-eth0-tx-0-21swapper/007:05:280
28405060,0irq/26-eth0-rx-31817-21cut08:38:103
28405060,0irq/26-eth0-rx-31574-21awk09:22:503
28405060,0irq/26-eth0-rx-18985-21kthreadcore11:58:063
28405060,0irq/26-eth0-rx-17902-21perl11:38:053
249960,0watchdog/218103-21cut08:03:082
249960,0watchdog/20-21swapper/210:34:202
149960,0migration/06826-21proc_pri12:08:120
149960,0migration/029187-21taskset09:38:570
149960,0migration/024821-21kernelversion06:58:080
33349955,0cyclictest0-21swapper/211:59:382
33349955,0cyclictest0-21swapper/211:24:292
33349955,0cyclictest0-21swapper/209:33:102
33279955,0cyclictest0-21swapper/112:03:081
33279955,0cyclictest0-21swapper/109:04:491
33279955,0cyclictest0-21swapper/108:33:081
33279950,0cyclictest0-21swapper/112:09:551
33279950,0cyclictest0-21swapper/111:43:261
33269955,0cyclictest0-21swapper/011:28:480
33269955,0cyclictest0-21swapper/010:33:460
33269955,0cyclictest0-21swapper/009:07:040
33269955,0cyclictest0-21swapper/008:53:110
33269950,0cyclictest0-21swapper/011:36:410
33269950,0cyclictest0-21swapper/010:56:290
33269950,0cyclictest0-21swapper/010:17:510
339950,0migration/36642-21munin-node07:58:023
339950,0migration/34431-21latency_hist07:07:503
339950,0migration/33645-21munin-node06:43:033
339950,0migration/33493-21kthreadcore08:18:073
339950,0migration/330247-21kthreadcore11:43:083
339950,0migration/32955-21kthreadcore07:28:073
339950,0migration/329292-21runrttasks06:58:143
339950,0migration/327791-21kthreadcore08:13:073
339950,0migration/326313-21kthreadcore07:23:133
339950,0migration/326130-21hwlatdetect11:23:043
339950,0migration/325387-21cpuspeed_turbos08:58:153
339950,0migration/325072-21taskset07:45:093
339950,0migration/323375-21fschecks_count09:38:053
339950,0migration/318630-21munin-node07:18:063
339950,0migration/318095-21munin-node08:53:053
339950,0migration/317338-21kthreadcore09:53:093
339950,0migration/31686-21sh08:42:573
339950,0migration/316810-21munin-node09:13:043
339950,0migration/316810-21munin-node09:13:043
339950,0migration/31515-21fschecks_count11:48:043
339950,0migration/314531-21runrttasks09:32:063
339950,0migration/311308-21kthreadcore09:08:083
28415050,0irq/27-eth0-tx-9418-21munin-node08:48:152
28415050,0irq/27-eth0-tx-8015-21ssh12:10:142
28415050,0irq/27-eth0-tx-7508-21ssh11:48:512
28415050,0irq/27-eth0-tx-6372-21iceccd08:22:430
28415050,0irq/27-eth0-tx-6372-21iceccd08:03:120
28415050,0irq/27-eth0-tx-3561-21munin-node06:43:050
28415050,0irq/27-eth0-tx-27411-21kthreadcore11:43:072
28415050,0irq/27-eth0-tx-0-21swapper/212:05:122
28415050,0irq/27-eth0-tx-0-21swapper/211:14:502
28415050,0irq/27-eth0-tx-0-21swapper/211:08:532
28415050,0irq/27-eth0-tx-0-21swapper/210:58:042
28415050,0irq/27-eth0-tx-0-21swapper/210:22:562
28415050,0irq/27-eth0-tx-0-21swapper/209:59:492
28415050,0irq/27-eth0-tx-0-21swapper/209:07:562
28415050,0irq/27-eth0-tx-0-21swapper/008:33:060
28405050,0irq/26-eth0-rx-7733-21open_inodes10:48:173
28405050,0irq/26-eth0-rx-6547-21runrttasks09:37:463
28405050,0irq/26-eth0-rx-4375-21irqcore10:28:073
28405050,0irq/26-eth0-rx-25772-21needreboot10:18:133
28405050,0irq/26-eth0-rx-14395-21cut10:53:073
28405050,0irq/26-eth0-rx-13500-21sa109:52:493
28405050,0irq/26-eth0-rx-0-21swapper/312:06:253
28405050,0irq/26-eth0-rx-0-21swapper/311:55:363
28405050,0irq/26-eth0-rx-0-21swapper/311:27:493
28405050,0irq/26-eth0-rx-0-21swapper/310:43:013
28405050,0irq/26-eth0-rx-0-21swapper/310:38:093
28405050,0irq/26-eth0-rx-0-21swapper/310:23:123
28405050,0irq/26-eth0-rx-0-21swapper/309:58:083
28405050,0irq/26-eth0-rx-0-21swapper/309:43:493
28405050,0irq/26-eth0-rx-0-21swapper/309:23:073
28405050,0irq/26-eth0-rx-0-21swapper/309:03:163
28405050,0irq/26-eth0-rx-0-21swapper/307:38:043
259950,0migration/211408-21kthreadcore07:58:082
249950,0watchdog/222504-21kthreadcore07:43:082
249950,0watchdog/20-21swapper/211:28:122
249950,0watchdog/20-21swapper/210:28:362
249950,0watchdog/20-21swapper/210:08:002
249950,0watchdog/20-21swapper/209:47:482
249950,0watchdog/20-21swapper/209:39:122
249950,0watchdog/20-21swapper/208:58:162
249950,0watchdog/20-21swapper/208:43:562
249950,0watchdog/20-21swapper/208:28:162
249950,0watchdog/20-21swapper/208:14:322
249950,0watchdog/20-21swapper/207:48:082
249950,0watchdog/20-21swapper/207:28:082
249950,0watchdog/20-21swapper/206:53:002
149950,0migration/09552-21aten_r1power_po08:48:040
149950,0migration/07022-21sendmail_mailqu11:08:110
149950,0migration/04863-21kthreadcore08:43:080
149950,0migration/03512-21kthreadcore07:53:090
149950,0migration/032157-21taskset11:04:180
149950,0migration/031571-21kthreadcore10:43:080
149950,0migration/031219-21kthreadcore10:03:060
149950,0migration/029687-21ntp_offset09:18:110
149950,0migration/027854-21df_abs08:38:050
149950,0migration/027410-21kthreadcore11:23:070
149950,0migration/026960-21kthreadcore08:58:190
149950,0migration/025340-21sh11:42:590
149950,0migration/02491-21kthreadcore09:43:080
149950,0migration/024002-21kthreadcore09:58:060
149950,0migration/022266-21kthreadcore08:08:080
149950,0migration/022198-21missed_timers06:53:100
149950,0migration/020963-21kthreadcore11:18:070
149950,0migration/018390-21kthreadcore09:13:080
149950,0migration/018390-21kthreadcore09:13:080
149950,0migration/017959-21kthreadcore09:53:100
149950,0migration/014753-21kthreadcore07:38:070
149950,0migration/011433-21latency09:48:100
149950,0migration/010353-21irqcore11:53:050
33349943,0cyclictest3425-21lldpd07:33:272
33349943,0cyclictest3425-21lldpd07:12:552
33349943,0cyclictest3425-21lldpd06:58:262
33349943,0cyclictest20845-21tail08:08:072
33279944,0cyclictest0-21swapper/110:34:031
33279943,0cyclictest6537-21awk07:33:061
33279943,0cyclictest6445-21snmpd07:17:121
33279943,0cyclictest3425-21lldpd11:14:171
33279943,0cyclictest3425-21lldpd10:30:321
33279943,0cyclictest3425-21lldpd10:09:191
33279943,0cyclictest3425-21lldpd09:29:491
33279943,0cyclictest3425-21lldpd08:13:591
33279943,0cyclictest3425-21lldpd06:47:561
33279943,0cyclictest30245-21awk10:03:051
33279943,0cyclictest29064-21sadc10:02:491
33279943,0cyclictest27546-21awk10:43:051
33279943,0cyclictest26054-21awk08:58:171
33279943,0cyclictest25252-21kworker/1:109:40:311
33279943,0cyclictest18813-21awk07:18:071
33279943,0cyclictest18165-21awk11:58:041
33279943,0cyclictest17333-21awk09:13:061
33279943,0cyclictest17333-21awk09:13:061
33279943,0cyclictest14023-21awk08:03:051
33279943,0cyclictest10964-21sadc07:12:501
33279943,0cyclictest10857-21awk10:53:041
33279943,0cyclictest10154-21awk11:53:041
33269944,0cyclictest0-21swapper/011:13:020
33269944,0cyclictest0-21swapper/010:41:200
33269944,0cyclictest0-21swapper/010:25:350
33269943,0cyclictest5638-21awk10:08:080
33269943,0cyclictest5475-21awk07:08:070
33269943,0cyclictest24226-21sadc07:22:500
339940,0migration/33829-21kthreadcore11:08:073
339940,0migration/316558-21ntp_states10:33:093
339940,0migration/314241-21irqcore08:03:073
28415040,0irq/27-eth0-tx-25672-21telnet07:48:020
28415040,0irq/27-eth0-tx-13606-21grep10:13:072
28415040,0irq/27-eth0-tx-0-21swapper/008:29:080
28405040,0irq/26-eth0-rx-0-21swapper/311:04:353
28405040,0irq/26-eth0-rx-0-21swapper/310:58:053
28405040,0irq/26-eth0-rx-0-21swapper/310:07:593
28405040,0irq/26-eth0-rx-0-21swapper/310:04:043
28405040,0irq/26-eth0-rx-0-21swapper/308:33:073
28405040,0irq/26-eth0-rx-0-21swapper/308:25:453
28405040,0irq/26-eth0-rx-0-21swapper/307:53:033
259940,0migration/23479-21latency_hist06:42:502
249940,0watchdog/219513-21kthreadcore07:18:082
249940,0watchdog/20-21swapper/210:19:282
249940,0watchdog/20-21swapper/209:28:082
249940,0watchdog/20-21swapper/208:35:202
249940,0watchdog/20-21swapper/208:23:482
249940,0watchdog/20-21swapper/208:19:482
249940,0watchdog/20-21swapper/207:53:522
249940,0watchdog/20-21swapper/207:38:042
249940,0watchdog/20-21swapper/207:25:282
249940,0watchdog/20-21swapper/207:07:522
249940,0watchdog/20-21swapper/206:47:562
179940,0migration/131960-21kthreadcore08:38:101
179940,0migration/131074-21sh09:21:101
179940,0migration/126156-21kthreadcore06:58:081
179940,0migration/120836-21sh09:33:121
179940,0migration/119285-21kthreadcore08:28:081
169940,0watchdog/1958-21kthreadcore07:03:081
169940,0watchdog/18297-21ssh10:48:281
169940,0watchdog/17146-21irqcore09:48:081
169940,0watchdog/14250-21missed_timers07:28:081
169940,0watchdog/130545-21kthreadcore07:48:081
169940,0watchdog/124992-40yum-updatesd-he11:22:441
169940,0watchdog/12411-21ssh10:26:161
169940,0watchdog/122505-21kthreadcore07:43:081
169940,0watchdog/121753-21kthreadcore08:53:081
169940,0watchdog/119380-21grep06:53:081
169940,0watchdog/115549-21kthreadcore10:13:081
169940,0watchdog/10-21swapper/111:28:081
169940,0watchdog/10-21swapper/111:23:081
169940,0watchdog/10-21swapper/110:58:121
169940,0watchdog/10-21swapper/109:43:081
169940,0watchdog/10-21swapper/109:08:081
169940,0watchdog/10-21swapper/107:38:081
149940,0migration/09882-21sh10:30:090
149940,0migration/04841-21kthreadcore09:23:100
149940,0migration/018508-21irqrtprio11:58:060
149940,0migration/013265-21kthreadcore07:13:080
33279933,0cyclictest3425-21lldpd08:23:291
33279933,0cyclictest22357-21kworker/1:011:37:491
33279933,0cyclictest22357-21kworker/1:011:08:011
33279932,0cyclictest24438-21cat07:22:501
33279932,0cyclictest1553-21awk08:18:051
33279931,0cyclictest18851-21munin-node10:38:041
33269932,0cyclictest26246-21awk12:03:040
33269932,0cyclictest13170-21grep10:13:040
33269932,0cyclictest13144-21sensors08:23:130
33269930,0cyclictest7136-21perl07:58:040
33269930,0cyclictest4328-21cut11:48:080
33269930,0cyclictest26474-21munin-node08:13:060
33269930,0cyclictest24469-21munin-node07:23:090
33269930,0cyclictest17688-21diskstats11:38:040
33269930,0cyclictest11852-21grep09:28:080
33269930,0cyclictest0-21swapper/010:58:110
33269930,0cyclictest0-21swapper/007:43:110
33269930,0cyclictest0-21swapper/007:32:560
33269930,0cyclictest0-21swapper/006:48:100
28405030,0irq/26-eth0-rx-0-21swapper/308:28:013
28405030,0irq/26-eth0-rx-0-21swapper/308:10:413
28405030,0irq/26-eth0-rx-0-21swapper/307:49:183
28405030,0irq/26-eth0-rx-0-21swapper/307:37:063
28405030,0irq/26-eth0-rx-0-21swapper/307:13:113
28405030,0irq/26-eth0-rx-0-21swapper/307:02:523
28405030,0irq/26-eth0-rx-0-21swapper/306:53:033
28405030,0irq/26-eth0-rx-0-21swapper/306:52:313
249930,0watchdog/2426-21ntpq08:38:122
19130,0ktimersoftd/120214-21sh10:17:551
179930,0migration/16644-21aten_r1power_po07:58:021
179930,0migration/1343-21fschecks_time07:53:051
169930,0watchdog/113939-21cut09:53:041
169930,0watchdog/10-21swapper/111:32:521
169930,0watchdog/10-21swapper/111:03:081
169930,0watchdog/10-21swapper/109:23:041
169930,0watchdog/10-21swapper/108:08:081
169930,0watchdog/10-21swapper/106:43:081
33269920,0cyclictest0-21swapper/010:48:050
33269920,0cyclictest0-21swapper/009:07:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional