You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 16:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Nov 15, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
886228281,0sleep00-21swapper/006:38:110
291350480,0irq/25-eth10-21swapper/306:36:083
883323526,0sleep10-21swapper/106:37:481
864322824,0sleep20-21swapper/206:35:372
28415070,0irq/27-eth0-tx-0-21swapper/209:34:332
249970,0watchdog/20-21swapper/209:02:302
90819960,0cyclictest0-21swapper/209:25:092
90809966,0cyclictest0-21swapper/111:26:271
90809966,0cyclictest0-21swapper/110:19:361
90809960,0cyclictest1636-21diskmemload10:08:441
90809960,0cyclictest0-21swapper/111:33:521
4160,0ktimersoftd/031286-21iostat_ios07:20:080
4160,0ktimersoftd/00-21swapper/011:24:270
35160,0ktimersoftd/30-21swapper/311:36:173
339960,0migration/332498-21sh11:46:433
339960,0migration/319473-21sh10:35:303
339960,0migration/317413-21sh09:13:113
339960,0migration/310048-21sendmail_mailst10:50:133
28415060,0irq/27-eth0-tx-23250-21ssh09:35:372
28415060,0irq/27-eth0-tx-0-21swapper/211:45:242
28415060,0irq/27-eth0-tx-0-21swapper/211:40:422
28415060,0irq/27-eth0-tx-0-21swapper/211:16:492
28415060,0irq/27-eth0-tx-0-21swapper/211:08:052
28415060,0irq/27-eth0-tx-0-21swapper/211:08:052
28415060,0irq/27-eth0-tx-0-21swapper/210:51:532
28415060,0irq/27-eth0-tx-0-21swapper/210:41:462
28415060,0irq/27-eth0-tx-0-21swapper/210:39:232
28415060,0irq/27-eth0-tx-0-21swapper/210:33:372
28415060,0irq/27-eth0-tx-0-21swapper/210:27:512
28415060,0irq/27-eth0-tx-0-21swapper/210:19:132
28415060,0irq/27-eth0-tx-0-21swapper/210:10:132
28415060,0irq/27-eth0-tx-0-21swapper/209:54:292
28415060,0irq/27-eth0-tx-0-21swapper/209:48:322
28415060,0irq/27-eth0-tx-0-21swapper/209:09:502
28415060,0irq/27-eth0-tx-0-21swapper/208:55:122
28415060,0irq/27-eth0-tx-0-21swapper/208:47:262
28405060,0irq/26-eth0-rx-7264-21ps11:50:133
28405060,0irq/26-eth0-rx-30405-21cut08:35:103
28405060,0irq/26-eth0-rx-2078-21sh11:07:193
28405060,0irq/26-eth0-rx-2078-21sh11:07:193
28405060,0irq/26-eth0-rx-0-21swapper/311:11:593
28405060,0irq/26-eth0-rx-0-21swapper/311:11:593
27160,0ktimersoftd/20-21swapper/211:24:272
169960,0watchdog/10-21swapper/109:45:101
90819955,0cyclictest0-21swapper/212:05:292
90819950,0cyclictest0-21swapper/207:45:122
90809955,0cyclictest0-21swapper/110:29:511
90809955,0cyclictest0-21swapper/108:29:531
90809953,0cyclictest25450-21kworker/1:009:42:431
90799955,0cyclictest1636-21diskmemload08:50:100
90799955,0cyclictest0-21swapper/011:17:530
90799955,0cyclictest0-21swapper/010:05:130
90799955,0cyclictest0-21swapper/010:02:510
90799954,0cyclictest7596-21munin-node08:20:130
90799954,0cyclictest11216-21awk09:10:100
90799950,0cyclictest0-21swapper/012:02:020
8225050,0irq/24-0000:00:0-21swapper/207:14:512
339950,0migration/39709-21kthreadcore08:20:083
339950,0migration/39413-21munin-node07:00:183
339950,0migration/34984-21kthreadcore08:40:093
339950,0migration/3468-21taskset08:13:573
339950,0migration/33906-21munin-node07:20:163
339950,0migration/33482-21latency_hist06:59:513
339950,0migration/33446-21kthreadcore10:05:113
339950,0migration/328144-21munin-node06:50:143
339950,0migration/327677-21munin-node09:00:063
339950,0migration/323747-21cpuspeed_turbos10:00:063
339950,0migration/323661-21munin-node07:35:173
339950,0migration/323614-21kthreadcore08:05:103
339950,0migration/323046-21fschecks_time10:20:073
339950,0migration/322783-21kthreadcore10:40:103
339950,0migration/322587-21munin-node09:35:143
339950,0migration/321948-21taskset09:55:443
339950,0migration/321494-21fschecks_time08:30:063
339950,0migration/320156-21kthreadcore12:00:113
339950,0migration/320086-21kthreadcore06:45:093
339950,0migration/31974-21sh10:45:123
339950,0migration/319595-21taskset08:03:413
339950,0migration/316237-21kthreadcore07:30:093
339950,0migration/316098-21kthreadcore10:15:103
339950,0migration/314398-21aten_r1power_vo08:25:053
339950,0migration/312446-21cpuspeed_turbos10:55:063
339950,0migration/310888-21kthreadcore09:30:113
339950,0migration/310096-21idleruntime06:40:063
28415050,0irq/27-eth0-tx-31828-21ssh09:43:202
28415050,0irq/27-eth0-tx-0-21swapper/211:38:152
28415050,0irq/27-eth0-tx-0-21swapper/211:34:412
28415050,0irq/27-eth0-tx-0-21swapper/210:55:402
28415050,0irq/27-eth0-tx-0-21swapper/210:45:032
28415050,0irq/27-eth0-tx-0-21swapper/210:20:232
28415050,0irq/27-eth0-tx-0-21swapper/209:57:542
28415050,0irq/27-eth0-tx-0-21swapper/209:15:172
28415050,0irq/27-eth0-tx-0-21swapper/209:10:062
28415050,0irq/27-eth0-tx-0-21swapper/208:52:492
28415050,0irq/27-eth0-tx-0-21swapper/208:44:142
28415050,0irq/27-eth0-tx-0-21swapper/008:10:110
28415050,0irq/27-eth0-tx-0-21swapper/006:44:320
28405050,0irq/26-eth0-rx-8879-21cut10:30:133
28405050,0irq/26-eth0-rx-7826-21iostat_ios10:10:103
28405050,0irq/26-eth0-rx-588-21fschecks_count09:45:063
28405050,0irq/26-eth0-rx-4526-21kthreadcore09:25:103
28405050,0irq/26-eth0-rx-31380-21sed09:20:113
28405050,0irq/26-eth0-rx-16592-21kthreadcore08:50:113
28405050,0irq/26-eth0-rx-15283-21cut11:15:103
28405050,0irq/26-eth0-rx-0-21swapper/312:09:233
28405050,0irq/26-eth0-rx-0-21swapper/311:40:103
28405050,0irq/26-eth0-rx-0-21swapper/311:34:453
28405050,0irq/26-eth0-rx-0-21swapper/310:25:263
28405050,0irq/26-eth0-rx-0-21swapper/309:44:163
28405050,0irq/26-eth0-rx-0-21swapper/309:05:143
28405050,0irq/26-eth0-rx-0-21swapper/308:57:473
259950,0migration/25440-21kthreadcore11:10:092
259950,0migration/25440-21kthreadcore11:10:092
259950,0migration/230225-21kthreadcore11:25:152
259950,0migration/223130-21latency12:00:122
259950,0migration/221652-21gltestperf08:30:072
249950,0watchdog/27375-21munin-node07:55:062
249950,0watchdog/2582-21kthreadcore10:05:102
249950,0watchdog/230066-21kthreadcore09:20:102
249950,0watchdog/222724-21cut11:00:142
249950,0watchdog/216493-21cut08:00:102
249950,0watchdog/21582-21kthreadcore07:20:102
249950,0watchdog/20-21swapper/210:00:182
179950,0migration/17538-21munin-node08:40:131
179950,0migration/14694-21kthreadcore11:10:091
179950,0migration/14694-21kthreadcore11:10:091
179950,0migration/124957-21cpuspeed_turbos07:40:051
179950,0migration/122506-21kthreadcore10:40:101
179950,0migration/121949-21kthreadcore07:10:111
179950,0migration/113510-21missed_timers09:50:131
179950,0migration/112096-21munin-node07:55:121
169950,0watchdog/129001-21sh10:22:381
169950,0watchdog/126127-21kthreadcore07:15:101
169950,0watchdog/124267-21ssh09:38:581
169950,0watchdog/114694-21kthreadcore11:15:101
169950,0watchdog/114512chrt09:01:581
169950,0watchdog/10-21swapper/108:46:341
149950,0migration/0719-21diskmemload11:05:200
149950,0migration/0719-21diskmemload11:05:200
149950,0migration/012656-21fschecks_count10:55:070
90819943,0cyclictest6445-21snmpd08:20:172
90819943,0cyclictest6445-21snmpd07:05:162
90819943,0cyclictest3425-21lldpd08:39:272
90819943,0cyclictest3425-21lldpd07:51:572
90819943,0cyclictest3425-21lldpd07:34:382
90819943,0cyclictest3425-21lldpd06:40:022
90819943,0cyclictest27502-21awk08:10:082
90819943,0cyclictest25558-21awk07:40:072
90819943,0cyclictest24506-21awk07:15:082
90819943,0cyclictest15167-21awk08:25:082
90809943,0cyclictest7775-21awk10:10:091
90809943,0cyclictest6445-21snmpd11:50:041
90809943,0cyclictest6445-21snmpd11:00:331
90809943,0cyclictest6445-21snmpd07:53:171
90809943,0cyclictest6445-21snmpd07:05:311
90809943,0cyclictest32148-21kworker/1:109:17:011
90809943,0cyclictest32148-21kworker/1:109:08:191
90809943,0cyclictest24341-21awk10:00:081
90809943,0cyclictest16576-21awk09:55:091
90809942,0cyclictest32148-21kworker/1:108:05:511
90809940,0cyclictest0-21swapper/106:49:551
90799944,0cyclictest0-21swapper/011:47:030
90799944,0cyclictest0-21swapper/011:36:200
90799944,0cyclictest0-21swapper/009:51:010
90799944,0cyclictest0-21swapper/009:36:500
90799943,0cyclictest6555-21awk10:30:110
90799943,0cyclictest6445-21snmpd10:22:480
90799943,0cyclictest6445-21snmpd09:49:020
90799943,0cyclictest3425-21lldpd11:50:350
90799943,0cyclictest3425-21lldpd10:42:520
90799943,0cyclictest3425-21lldpd10:26:210
90799943,0cyclictest3425-21lldpd09:43:500
90799943,0cyclictest3425-21lldpd07:56:570
90799943,0cyclictest3425-21lldpd07:11:050
90799943,0cyclictest3347-21sadc06:59:510
90799943,0cyclictest26873-21awk12:05:110
90799943,0cyclictest26864-21awk09:20:070
90799943,0cyclictest21729-21awk08:30:070
90799943,0cyclictest19083-21awk09:15:090
90799943,0cyclictest1724-21awk08:15:090
90799943,0cyclictest12223-21awk07:30:070
90799943,0cyclictest11196-21awk07:05:070
90799943,0cyclictest11081-21sadc07:29:510
90799943,0cyclictest10063-21sadc09:09:510
90799943,0cyclictest10049-21awk09:30:090
90799940,0cyclictest0-21swapper/008:35:070
8225040,0irq/24-0000:00:4750-21kthreadcore11:50:092
339940,0migration/36051-21taskset07:52:073
339940,0migration/33752-21kthreadcore07:45:103
339940,0migration/327728-21kthreadcore07:40:103
339940,0migration/325233-21diskmemload11:21:143
339940,0migration/312602-21kthreadcore07:05:093
28415040,0irq/27-eth0-tx-1636-21diskmemload08:40:090
28415040,0irq/27-eth0-tx-0-21swapper/211:55:132
28415040,0irq/27-eth0-tx-0-21swapper/008:25:090
28415040,0irq/27-eth0-tx-0-21swapper/007:40:120
28405040,0irq/26-eth0-rx-6547-21runrttasks08:45:503
28405040,0irq/26-eth0-rx-0-21swapper/311:55:163
28405040,0irq/26-eth0-rx-0-21swapper/311:24:573
28405040,0irq/26-eth0-rx-0-21swapper/311:00:403
28405040,0irq/26-eth0-rx-0-21swapper/309:51:393
28405040,0irq/26-eth0-rx-0-21swapper/309:15:093
28405040,0irq/26-eth0-rx-0-21swapper/307:55:033
28405040,0irq/26-eth0-rx-0-21swapper/307:12:193
259940,0migration/29522-21timerwakeupswit07:00:182
259940,0migration/25383-21kthreadcore08:15:112
259940,0migration/227999-21munin-node06:50:132
249940,0watchdog/230050-21hddtemp_smartct06:55:062
249940,0watchdog/224779-21grep08:05:102
249940,0watchdog/20-21swapper/207:35:102
179940,0migration/1931-21kthreadcore07:45:091
179940,0migration/18031-21fschecks_count08:20:051
179940,0migration/16737-21timerandwakeup08:15:161
179940,0migration/128952-21kthreadcore12:05:131
179940,0migration/124079-21sh11:40:361
179940,0migration/112269-21kthreadcore09:30:111
179940,0migration/111644-21kthreadcore11:35:071
169940,0watchdog/126967-21diskstats08:10:061
169940,0watchdog/126916-21ps11:45:101
169940,0watchdog/126584-21perl09:20:061
169940,0watchdog/124676-21switchtime11:20:181
169940,0watchdog/117717-21ntpdc10:55:141
169940,0watchdog/11583-21kthreadcore07:20:101
169940,0watchdog/112702-21kthreadcore08:50:101
169940,0watchdog/10-21swapper/112:00:101
169940,0watchdog/10-21swapper/111:54:541
169940,0watchdog/10-21swapper/110:49:541
169940,0watchdog/10-21swapper/110:35:141
169940,0watchdog/10-21swapper/110:30:261
169940,0watchdog/10-21swapper/109:25:101
169940,0watchdog/10-21swapper/109:10:021
169940,0watchdog/10-21swapper/108:55:181
169940,0watchdog/10-21swapper/108:35:101
169940,0watchdog/10-21swapper/108:25:101
169940,0watchdog/10-21swapper/108:00:101
169940,0watchdog/10-21swapper/107:35:101
169940,0watchdog/10-21swapper/107:31:461
169940,0watchdog/10-21swapper/106:55:061
149940,0migration/09917-21sh08:46:090
149940,0migration/08951-21proc_pri11:10:140
149940,0migration/08951-21proc_pri11:10:140
149940,0migration/07633-21kthreadcore10:50:090
149940,0migration/04197-21fschecks_time07:00:090
149940,0migration/026515-21taskset08:58:390
149940,0migration/024298-21runrttasks07:38:110
149940,0migration/021225-21ntp_offset09:55:140
149940,0migration/020775-21kthreadcore11:40:110
149940,0migration/018103-21kthreadcore06:45:090
149940,0migration/016938-21kthreadcore10:35:090
149940,0migration/01231-21latency_hist09:24:510
90819932,0cyclictest5486-21awk07:25:072
90809933,0cyclictest9108-21kworker/1:006:45:131
90809933,0cyclictest21877-21kworker/1:210:44:531
90799933,0cyclictest0-21swapper/010:47:370
90799932,0cyclictest6445-21snmpd08:00:100
90799932,0cyclictest4021-21cut11:30:110
90799932,0cyclictest3425-21lldpd08:06:580
90799932,0cyclictest32240-21awk07:45:070
90799932,0cyclictest15604-21awk10:15:090
90799932,0cyclictest15431-21sensors11:55:140
90799932,0cyclictest10187-21tr10:10:110
90799931,0cyclictest0-21swapper/007:15:100
90799930,0cyclictest27359-21perl11:25:110
90799930,0cyclictest0-21swapper/009:00:160
90799930,0cyclictest0-21swapper/006:50:050
8225030,0irq/24-0000:00:0-21swapper/206:44:512
4607233,0sleep19108-21kworker/1:007:24:551
28405030,0irq/26-eth0-rx-0-21swapper/308:17:313
28405030,0irq/26-eth0-rx-0-21swapper/307:25:153
28405030,0irq/26-eth0-rx-0-21swapper/307:15:143
179930,0migration/16459-21kthreadcore07:00:121
169930,0watchdog/10-21swapper/111:04:541
169930,0watchdog/10-21swapper/111:04:541
169930,0watchdog/10-21swapper/106:39:541
149930,0migration/06271-21runrttasks07:54:380
90799921,0cyclictest19961-21aten_r1power_cu11:00:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional