You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-09 - 01:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sun Nov 09, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350530,0irq/25-eth10-21swapper/318:39:433
1798125241,0sleep00-21swapper/018:37:340
1808623531,0sleep10-21swapper/118:38:471
1785623228,0sleep20-21swapper/218:36:042
28415070,0irq/27-eth0-tx-0-21swapper/020:40:250
182799976,0cyclictest4365-21aten_r1power_en22:55:172
4160,0ktimersoftd/00-21swapper/022:59:570
339960,0migration/35475-21sh22:14:573
339960,0migration/33831-21sh22:54:283
339960,0migration/332281-21sh20:49:343
339960,0migration/327918-21sh22:25:563
339960,0migration/327417-21sh23:08:373
339960,0migration/321496-21sh22:02:553
339960,0migration/315735-21sh20:58:423
339960,0migration/310352-21sh23:16:533
28415060,0irq/27-eth0-tx-7110-21munin-node19:20:300
28415060,0irq/27-eth0-tx-0-21swapper/223:43:442
28415060,0irq/27-eth0-tx-0-21swapper/223:39:102
28415060,0irq/27-eth0-tx-0-21swapper/223:25:012
28415060,0irq/27-eth0-tx-0-21swapper/223:15:322
28415060,0irq/27-eth0-tx-0-21swapper/222:34:302
28415060,0irq/27-eth0-tx-0-21swapper/222:27:132
28415060,0irq/27-eth0-tx-0-21swapper/222:20:582
28415060,0irq/27-eth0-tx-0-21swapper/221:51:332
28415060,0irq/27-eth0-tx-0-21swapper/221:45:372
28415060,0irq/27-eth0-tx-0-21swapper/221:42:522
28415060,0irq/27-eth0-tx-0-21swapper/221:03:212
28415060,0irq/27-eth0-tx-0-21swapper/200:04:552
28415060,0irq/27-eth0-tx-0-21swapper/020:25:270
28415060,0irq/27-eth0-tx-0-21swapper/019:30:220
28415060,0irq/27-eth0-tx-0-21swapper/018:49:530
28405060,0irq/26-eth0-rx-9533-21kthreadcore22:15:233
28405060,0irq/26-eth0-rx-4720-21df_inode22:55:183
249960,0watchdog/20-21swapper/222:16:442
182799966,0cyclictest0-21swapper/223:03:482
182749960,0cyclictest0-21swapper/023:03:490
149960,0migration/024970-21sh23:46:210
149960,0migration/013513-21unixbench_multi21:35:310
8225050,0irq/24-0000:00:25883-21latency_hist19:10:062
4150,0ktimersoftd/00-21swapper/020:46:440
339950,0migration/39763-21kthreadcore22:35:243
339950,0migration/38253-21timerwakeupswit20:05:293
339950,0migration/38065-21latency_hist19:45:063
339950,0migration/37128-21kthreadcore20:30:223
339950,0migration/36743-21munin-node19:40:253
339950,0migration/36225-21sensors_volt20:50:293
339950,0migration/3408-21kthreadcore22:30:233
339950,0migration/332241-21fschecks_count18:50:213
339950,0migration/331872-21runrttasks21:09:013
339950,0migration/330968-21timerwakeupswit18:45:293
339950,0migration/329856-21kthreadcore19:35:263
339950,0migration/324121-21sendmail_mailqu18:40:273
339950,0migration/32351-21taskset23:33:573
339950,0migration/323025-21kthreadcore19:55:243
339950,0migration/321688-21munin-node22:45:203
339950,0migration/319947-21fschecks_count19:05:233
339950,0migration/31973-21kthreadcore21:50:243
339950,0migration/318633-21timerwakeupswit19:00:323
339950,0migration/318517-21irqrtprio00:05:243
339950,0migration/318113-21kthreadcore22:40:263
339950,0migration/318040-21sendmail_mailtr23:00:283
339950,0migration/314353-21latency20:10:243
28415050,0irq/27-eth0-tx-3161-21ssh23:14:142
28415050,0irq/27-eth0-tx-26618-21ssh23:28:142
28415050,0irq/27-eth0-tx-21413-21munin-node19:55:200
28415050,0irq/27-eth0-tx-19459-21munin-node19:05:320
28415050,0irq/27-eth0-tx-0-21swapper/223:53:342
28415050,0irq/27-eth0-tx-0-21swapper/223:45:122
28415050,0irq/27-eth0-tx-0-21swapper/223:30:162
28415050,0irq/27-eth0-tx-0-21swapper/222:50:442
28415050,0irq/27-eth0-tx-0-21swapper/222:11:302
28415050,0irq/27-eth0-tx-0-21swapper/221:55:462
28415050,0irq/27-eth0-tx-0-21swapper/221:29:242
28415050,0irq/27-eth0-tx-0-21swapper/221:20:462
28415050,0irq/27-eth0-tx-0-21swapper/221:16:242
28415050,0irq/27-eth0-tx-0-21swapper/221:11:562
28415050,0irq/27-eth0-tx-0-21swapper/220:58:292
28415050,0irq/27-eth0-tx-0-21swapper/220:50:512
28415050,0irq/27-eth0-tx-0-21swapper/200:10:012
28415050,0irq/27-eth0-tx-0-21swapper/019:35:270
28405050,0irq/26-eth0-rx-27032-21perl23:50:183
28405050,0irq/26-eth0-rx-24370-21python23:45:313
28405050,0irq/26-eth0-rx-21915-21cut19:30:233
28405050,0irq/26-eth0-rx-16693-21aten_r1power_vo21:00:183
28405050,0irq/26-eth0-rx-15871-21kthreadcore23:20:263
28405050,0irq/26-eth0-rx-14581-21if_err_eth022:20:203
28405050,0irq/26-eth0-rx-11766-21cat20:35:223
28405050,0irq/26-eth0-rx-0-21swapper/323:59:013
28405050,0irq/26-eth0-rx-0-21swapper/323:27:183
28405050,0irq/26-eth0-rx-0-21swapper/321:41:443
28405050,0irq/26-eth0-rx-0-21swapper/321:25:393
28405050,0irq/26-eth0-rx-0-21swapper/320:43:083
27150,0ktimersoftd/20-21swapper/223:05:412
27150,0ktimersoftd/20-21swapper/222:40:092
27150,0ktimersoftd/20-21swapper/221:06:202
259950,0migration/28522-21kthreadcore20:30:222
259950,0migration/28148-21fschecks_time21:35:212
259950,0migration/230776-21unixbench_singl20:45:312
259950,0migration/228696-21kthreadcore19:35:262
249950,0watchdog/20-21swapper/222:37:162
249950,0watchdog/20-21swapper/222:05:242
249950,0watchdog/20-21swapper/221:30:362
249950,0watchdog/20-21swapper/219:05:282
249950,0watchdog/20-21swapper/218:56:282
19150,0ktimersoftd/10-21swapper/122:59:571
182799950,0cyclictest0-21swapper/222:46:462
182799950,0cyclictest0-21swapper/220:20:092
182759955,0cyclictest0-21swapper/123:32:521
182759955,0cyclictest0-21swapper/122:49:221
182759955,0cyclictest0-21swapper/120:44:301
182759954,0cyclictest21814-21ssh21:41:251
182759950,0cyclictest0-21swapper/122:40:511
182759950,0cyclictest0-21swapper/122:24:271
182759950,0cyclictest0-21swapper/100:00:181
179950,0migration/19296-21sh23:35:561
179950,0migration/16695-21sh21:33:291
179950,0migration/14673-21kthreadcore20:05:231
179950,0migration/132713-21taskset23:51:021
179950,0migration/131673-21latency_hist18:50:061
179950,0migration/120929-21kthreadcore00:05:251
169950,0watchdog/10-21swapper/119:30:201
149950,0migration/0833-21kthreadcore21:30:230
149950,0migration/08085-21kthreadcore22:15:230
149950,0migration/06977-21kthreadcore20:05:240
149950,0migration/05957-21sh21:53:530
149950,0migration/04483-21gltestperf23:15:210
149950,0migration/04317-21kthreadcore21:10:250
149950,0migration/03302-21sh22:52:550
149950,0migration/032660-21taskset23:50:490
149950,0migration/03149-21kthreadcore23:55:240
149950,0migration/030472-21sh21:26:380
149950,0migration/021750-21kthreadcore23:05:230
149950,0migration/021154-21kthreadcore21:20:240
149950,0migration/017565-21kthreadcore19:50:240
149950,0migration/016510-21kthreadcore22:00:230
149950,0migration/016432-21latency_hist21:00:060
149950,0migration/01406-21irqcore20:50:230
149950,0migration/013869-21kthreadcore23:40:270
149950,0migration/013788-21munin-node22:40:230
149950,0migration/013553-21gltestperf19:00:230
149950,0migration/013298-21kthreadcore21:15:240
149950,0migration/012601-21kthreadcore00:00:240
149950,0migration/010588-21latency_hist23:40:050
4140,0ktimersoftd/00-21swapper/023:20:120
339940,0migration/37588-21if_eth021:55:233
339940,0migration/3633-21kthreadcore20:00:243
339940,0migration/329514-21diskmemload21:45:533
339940,0migration/323043-21fschecks_time22:05:203
339940,0migration/31306-21if_eth119:15:213
28415040,0irq/27-eth0-tx-1712-21ssh23:55:092
28415040,0irq/27-eth0-tx-0-21swapper/222:00:062
28415040,0irq/27-eth0-tx-0-21swapper/020:30:220
28405040,0irq/26-eth0-rx-894-21smartctl21:10:213
28405040,0irq/26-eth0-rx-0-21swapper/323:40:183
28405040,0irq/26-eth0-rx-0-21swapper/323:37:133
28405040,0irq/26-eth0-rx-0-21swapper/323:10:073
28405040,0irq/26-eth0-rx-0-21swapper/321:22:073
28405040,0irq/26-eth0-rx-0-21swapper/319:28:343
28405040,0irq/26-eth0-rx-0-21swapper/318:55:263
28405040,0irq/26-eth0-rx-0-21swapper/300:02:253
259940,0migration/25784-21munin-node19:15:242
259940,0migration/223584-21kthreadcore19:30:232
259940,0migration/211881-21kthreadcore20:10:232
249940,0watchdog/24826-21kthreadcore19:40:242
249940,0watchdog/21352-21smartctl20:00:282
249940,0watchdog/20-21swapper/220:40:202
249940,0watchdog/20-21swapper/220:36:242
249940,0watchdog/20-21swapper/220:26:082
249940,0watchdog/20-21swapper/219:25:082
249940,0watchdog/20-21swapper/219:02:082
249940,0watchdog/20-21swapper/218:53:202
249940,0watchdog/20-21swapper/218:40:282
182799943,0cyclictest6445-21snmpd20:15:242
182799943,0cyclictest6445-21snmpd19:50:092
182799943,0cyclictest6445-21snmpd19:46:092
182799943,0cyclictest6445-21snmpd19:22:392
182799943,0cyclictest3255-21awk20:05:212
182799943,0cyclictest22345-21awk19:55:232
182759944,0cyclictest22353-21kworker/1:121:15:231
182759944,0cyclictest22353-21kworker/1:121:05:191
182759944,0cyclictest0-21swapper/122:16:151
182759943,0cyclictest9430-21awk20:55:221
182759943,0cyclictest6445-21snmpd23:41:561
182759943,0cyclictest6445-21snmpd23:29:261
182759943,0cyclictest6445-21snmpd20:25:091
182759943,0cyclictest3425-21lldpd20:16:321
182759943,0cyclictest3425-21lldpd20:03:021
182759943,0cyclictest3110-21irqbalance22:37:101
182759943,0cyclictest3110-21irqbalance22:00:201
182759943,0cyclictest3110-21irqbalance20:50:101
182759943,0cyclictest3110-21irqbalance19:55:101
182759943,0cyclictest3110-21irqbalance19:50:201
182759943,0cyclictest3110-21irqbalance19:45:101
182759943,0cyclictest3110-21irqbalance19:06:001
182759943,0cyclictest3110-21irqbalance19:00:301
182759943,0cyclictest3110-21irqbalance19:00:001
182759943,0cyclictest3110-21irqbalance18:40:101
182759943,0cyclictest30450-21sh22:30:201
182759943,0cyclictest24944-21awk21:25:191
182749944,0cyclictest7860-21irqrtprio21:55:240
182749944,0cyclictest0-21swapper/022:13:310
182749943,0cyclictest6445-21snmpd19:45:090
182749943,0cyclictest6445-21snmpd19:12:390
182749943,0cyclictest3425-21lldpd23:26:090
182749943,0cyclictest3425-21lldpd22:47:080
182749943,0cyclictest3425-21lldpd19:28:310
182749943,0cyclictest3110-21irqbalance22:24:200
182749943,0cyclictest28381-21awk23:30:210
182749943,0cyclictest25465-21awk21:05:220
182749943,0cyclictest23320-21awk20:20:250
182749943,0cyclictest2280-21awk19:40:220
182749943,0cyclictest19337-21awk18:40:210
182749943,0cyclictest18128-21awk00:05:220
182749943,0cyclictest1216-21awk19:15:200
182749940,0cyclictest6143-21munin-node18:55:260
182749940,0cyclictest31547-21grep23:10:240
179940,0migration/19053-21kthreadcore21:55:251
179940,0migration/16404-21latency_hist21:55:061
179940,0migration/16178-21kthreadcore23:15:241
179940,0migration/14060-21kthreadcore19:40:241
179940,0migration/130986-21fschecks_time22:10:191
179940,0migration/130601-21timerandwakeup20:45:311
179940,0migration/128611-21latency_hist22:50:051
179940,0migration/127221-21munin-node19:35:211
179940,0migration/125471-21kthreadcore23:05:241
179940,0migration/125373-21kthreadcore22:25:241
179940,0migration/125077-21kthreadcore21:45:251
179940,0migration/123364-21if_eth1.320:20:251
179940,0migration/113134-21kthreadcore23:20:251
179940,0migration/112759-21fschecks_count23:00:201
169940,0watchdog/18618-21grep19:20:241
169940,0watchdog/18249-21ssh23:56:401
169940,0watchdog/16124-21wc19:15:281
169940,0watchdog/132124-21df18:50:201
169940,0watchdog/115419-21missed_timers20:35:241
169940,0watchdog/114665-21ssh21:38:481
169940,0watchdog/10-21swapper/122:05:201
169940,0watchdog/10-21swapper/121:20:201
149940,0migration/0727-21kthreadcore18:50:250
149940,0migration/032635-21kthreadcore20:00:240
149940,0migration/015273-21kthreadcore20:35:230
28405030,0irq/26-eth0-rx-0-21swapper/321:35:183
28405030,0irq/26-eth0-rx-0-21swapper/321:30:193
28405030,0irq/26-eth0-rx-0-21swapper/321:15:173
28405030,0irq/26-eth0-rx-0-21swapper/320:27:363
28405030,0irq/26-eth0-rx-0-21swapper/320:20:073
28405030,0irq/26-eth0-rx-0-21swapper/320:15:063
28405030,0irq/26-eth0-rx-0-21swapper/319:54:533
28405030,0irq/26-eth0-rx-0-21swapper/319:10:183
249930,0watchdog/20-21swapper/218:45:082
182759933,0cyclictest3425-21lldpd19:28:011
182759932,0cyclictest9659-21tune2fs20:10:201
182759932,0cyclictest6445-21snmpd21:10:171
182759932,0cyclictest27068-21awk19:10:211
182759930,0cyclictest7167-21cut20:30:221
182759930,0cyclictest28095-21cut23:10:051
182749932,0cyclictest7067-21kthreadcore22:35:230
182749932,0cyclictest6445-21snmpd20:55:140
182749932,0cyclictest27418-21cut21:45:260
182749932,0cyclictest16612-21awk20:15:210
182749930,0cyclictest16670-21irqstats21:40:220
182749930,0cyclictest1532-21tr22:30:230
182749930,0cyclictest0-21swapper/020:10:240
169930,0watchdog/117350-21ssh21:00:201
169930,0watchdog/10-21swapper/123:45:321
149930,0migration/029847-21diskmemload22:09:370
28405020,0irq/26-eth0-rx-0-21swapper/319:20:213
182749921,0cyclictest6547-21runrttasks22:25:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional