You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-14 - 02:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Tue Oct 14, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2591826058,0sleep10-21swapper/118:36:171
291350540,0irq/25-eth10-21swapper/318:40:283
3192823726,0sleep20-21swapper/218:38:462
3187523726,0sleep00-21swapper/018:38:080
28415070,0irq/27-eth0-tx-0-21swapper/221:55:392
339960,0migration/329264-21taskset21:31:183
339960,0migration/324780-21sh23:30:343
339960,0migration/313111-21gltestperf21:41:193
339960,0migration/311430-21sh22:00:443
339960,0migration/311031-21sh21:17:013
339960,0migration/310790-21sh21:37:183
321979960,0cyclictest0-21swapper/022:15:400
321979960,0cyclictest0-21swapper/022:15:400
28415060,0irq/27-eth0-tx-0-21swapper/223:59:312
28415060,0irq/27-eth0-tx-0-21swapper/223:52:082
28415060,0irq/27-eth0-tx-0-21swapper/223:42:552
28415060,0irq/27-eth0-tx-0-21swapper/223:36:532
28415060,0irq/27-eth0-tx-0-21swapper/223:33:472
28415060,0irq/27-eth0-tx-0-21swapper/223:17:452
28415060,0irq/27-eth0-tx-0-21swapper/223:14:042
28415060,0irq/27-eth0-tx-0-21swapper/222:56:072
28415060,0irq/27-eth0-tx-0-21swapper/222:55:302
28415060,0irq/27-eth0-tx-0-21swapper/222:50:482
28415060,0irq/27-eth0-tx-0-21swapper/222:43:052
28415060,0irq/27-eth0-tx-0-21swapper/222:31:132
28415060,0irq/27-eth0-tx-0-21swapper/222:23:102
28415060,0irq/27-eth0-tx-0-21swapper/222:16:192
28415060,0irq/27-eth0-tx-0-21swapper/222:13:212
28415060,0irq/27-eth0-tx-0-21swapper/222:13:212
28415060,0irq/27-eth0-tx-0-21swapper/222:07:072
28415060,0irq/27-eth0-tx-0-21swapper/222:02:122
28415060,0irq/27-eth0-tx-0-21swapper/221:47:532
28415060,0irq/27-eth0-tx-0-21swapper/221:41:372
28415060,0irq/27-eth0-tx-0-21swapper/221:30:142
28415060,0irq/27-eth0-tx-0-21swapper/221:22:472
28415060,0irq/27-eth0-tx-0-21swapper/220:53:082
28415060,0irq/27-eth0-tx-0-21swapper/220:48:322
28415060,0irq/27-eth0-tx-0-21swapper/019:11:180
28405060,0irq/26-eth0-rx-8360-21mdstat19:56:243
28405060,0irq/26-eth0-rx-20836-21df20:46:133
249960,0watchdog/20-21swapper/223:02:542
249960,0watchdog/20-21swapper/222:36:142
249960,0watchdog/20-21swapper/220:59:022
149960,0migration/026634-21sh22:30:340
149960,0migration/025149-21kthreadcore20:46:190
149960,0migration/013588-21aten2.4-expect18:51:160
149960,0migration/011011-21idleruntime-cro22:21:030
8225050,0irq/24-0000:00:0-21swapper/220:26:032
4150,0ktimersoftd/00-21swapper/023:57:310
4150,0ktimersoftd/00-21swapper/023:10:040
339950,0migration/39878-21irqcore23:41:203
339950,0migration/39010-21sh22:59:013
339950,0migration/38414-21fschecks_time00:01:173
339950,0migration/36436-21munin-node22:16:213
339950,0migration/35361-21aten_r1power_en20:31:113
339950,0migration/331073-21switchtime23:31:273
339950,0migration/330741-21munin-node20:21:173
339950,0migration/329869-21kthreadcore19:01:233
339950,0migration/327927-21taskset21:50:483
339950,0migration/327655-21latency_hist22:11:043
339950,0migration/327655-21latency_hist22:11:043
339950,0migration/327489-21kthreadcore20:16:173
339950,0migration/322918-21taskset23:46:573
339950,0migration/322124-21taskset00:08:163
339950,0migration/322066-21irqcore19:21:233
339950,0migration/321941-21irqcore21:06:193
339950,0migration/321082-21munin-node21:26:173
339950,0migration/320896-21kthreadcore22:26:253
339950,0migration/320770-21taskset19:20:473
339950,0migration/319921-21taskset20:45:123
339950,0migration/319030-21runrttasks21:22:053
339950,0migration/317438-21runrttasks22:01:243
339950,0migration/316808-21munin-node22:21:253
339950,0migration/316550-21irqrtprio19:41:233
339950,0migration/314927-21taskset19:38:403
339950,0migration/313214-21irqrtprio20:36:193
339950,0migration/310956-21cpuspeed_turbos22:41:173
321989955,0cyclictest0-21swapper/121:53:571
321989955,0cyclictest0-21swapper/121:24:381
321989955,0cyclictest0-21swapper/121:02:051
321989950,0cyclictest0-21swapper/123:05:221
321989950,0cyclictest0-21swapper/122:16:301
321989950,0cyclictest0-21swapper/121:33:091
321989950,0cyclictest0-21swapper/100:01:021
321979955,0cyclictest0-21swapper/020:53:220
321979950,0cyclictest22076-21munin-node19:46:230
321979950,0cyclictest0-21swapper/021:38:470
28415050,0irq/27-eth0-tx-6770-21cut21:16:232
28415050,0irq/27-eth0-tx-2700-21ssh21:11:302
28415050,0irq/27-eth0-tx-0-21swapper/223:47:202
28415050,0irq/27-eth0-tx-0-21swapper/223:06:232
28415050,0irq/27-eth0-tx-0-21swapper/221:59:302
28415050,0irq/27-eth0-tx-0-21swapper/221:08:462
28415050,0irq/27-eth0-tx-0-21swapper/200:04:242
28415050,0irq/27-eth0-tx-0-21swapper/020:41:210
28415050,0irq/27-eth0-tx-0-21swapper/020:36:140
28415050,0irq/27-eth0-tx-0-21swapper/020:26:290
28415050,0irq/27-eth0-tx-0-21swapper/019:56:170
28405050,0irq/26-eth0-rx-7759-21munin-node19:11:243
28405050,0irq/26-eth0-rx-29098-21smartctl21:51:203
28405050,0irq/26-eth0-rx-2554-21ssh20:51:583
28405050,0irq/26-eth0-rx-22076-21munin-node19:46:213
28405050,0irq/26-eth0-rx-18836-21if_eth023:06:183
28405050,0irq/26-eth0-rx-1840-21cat19:31:033
28405050,0irq/26-eth0-rx-1702-21kthreadcore21:11:213
28405050,0irq/26-eth0-rx-0-21swapper/323:36:203
28405050,0irq/26-eth0-rx-0-21swapper/323:22:343
28405050,0irq/26-eth0-rx-0-21swapper/323:18:033
28405050,0irq/26-eth0-rx-0-21swapper/323:12:403
28405050,0irq/26-eth0-rx-0-21swapper/322:50:123
28405050,0irq/26-eth0-rx-0-21swapper/322:06:183
28405050,0irq/26-eth0-rx-0-21swapper/321:01:153
28405050,0irq/26-eth0-rx-0-21swapper/318:46:163
259950,0migration/28274-21kthreadcore20:31:162
259950,0migration/26307-21kthreadcore21:36:202
259950,0migration/224133-21kthreadcore19:21:242
259950,0migration/222166-21timerandwakeup20:06:282
259950,0migration/219038-21latency_hist22:26:042
249950,0watchdog/20-21swapper/223:26:062
249950,0watchdog/20-21swapper/223:24:062
249950,0watchdog/20-21swapper/221:32:022
249950,0watchdog/20-21swapper/221:01:142
249950,0watchdog/20-21swapper/220:41:262
249950,0watchdog/20-21swapper/220:22:542
249950,0watchdog/20-21swapper/219:41:302
249950,0watchdog/20-21swapper/219:36:262
249950,0watchdog/20-21swapper/219:31:142
249950,0watchdog/20-21swapper/219:20:342
249950,0watchdog/20-21swapper/219:02:022
249950,0watchdog/20-21swapper/200:06:102
19150,0ktimersoftd/118463-21diskmemload22:04:471
179950,0migration/131674-21kthreadcore22:51:221
179950,0migration/125959-21needreboot20:46:201
179950,0migration/125954-21timerwakeupswit18:56:271
179950,0migration/12271-21sh22:35:501
179950,0migration/121125-21kthreadcore20:06:231
179950,0migration/112564-21kthreadcore00:01:221
179950,0migration/110761-21sh20:56:431
169950,0watchdog/125730-21grep23:31:181
149950,0migration/09424-21aten_r1power_fr23:21:160
149950,0migration/08252-21munin-node22:36:230
149950,0migration/07538-21sh23:16:290
149950,0migration/03460-21sh23:36:210
149950,0migration/032269-21kthreadcore19:51:240
149950,0migration/031693-21sh23:55:050
149950,0migration/031373-21ntp_offset23:11:240
149950,0migration/031064-21switchtime23:31:270
149950,0migration/030894-21kthreadcore19:26:220
149950,0migration/029341-21fschecks_time21:31:180
149950,0migration/02861-21kthreadcore20:21:210
149950,0migration/028012-21sh21:51:020
149950,0migration/027013-21latency_hist22:31:040
149950,0migration/026431-21runrttasks22:51:030
149950,0migration/024696-21vmstat22:46:270
149950,0migration/024129-21sh23:28:360
149950,0migration/023924-21cpuspeed_turbos20:16:120
149950,0migration/02309-21kernelversion19:06:200
149950,0migration/021412-21gltestperf21:26:180
149950,0migration/020989-21aten_r1power_po21:06:150
149950,0migration/019068-21sh21:01:580
149950,0migration/017947-21ksmtuned22:45:010
149950,0migration/017849-21kthreadcore19:41:240
149950,0migration/016320-21latency_hist20:06:030
149950,0migration/015057-21kthreadcore22:01:210
149950,0migration/012931-21kthreadcore22:21:220
149950,0migration/012114-21sh21:00:210
149950,0migration/011366-21kthreadcore23:41:210
8225040,0irq/24-0000:00:0-21swapper/220:01:032
339940,0migration/38480-21sh22:36:253
339940,0migration/34781-21kthreadcore19:06:213
339940,0migration/33408-21kthreadcore19:31:213
339940,0migration/330605-21kthreadcore22:31:213
339940,0migration/322660-21runrttasks20:09:403
339940,0migration/315011-21kthreadcore18:51:213
339940,0migration/312588-21kthreadcore20:01:213
321989944,0cyclictest0-21swapper/123:52:311
321989943,0cyclictest9630-21awk19:36:211
321989943,0cyclictest7589-21awk18:46:191
321989943,0cyclictest6445-21snmpd23:24:411
321989943,0cyclictest6445-21snmpd20:33:401
321989943,0cyclictest6445-21snmpd19:02:251
321989943,0cyclictest31685-21kworker/1:123:26:151
321989943,0cyclictest31685-21kworker/1:121:46:251
321989943,0cyclictest31685-21kworker/1:121:44:451
321989943,0cyclictest21609-21awk21:26:191
321989943,0cyclictest20806-21sadc19:21:031
321989943,0cyclictest1641-21awk23:36:191
321989943,0cyclictest11502-21awk22:41:191
321979944,0cyclictest0-21swapper/023:04:320
321979943,0cyclictest8662-21awk00:01:190
321979943,0cyclictest6445-21snmpd21:16:100
321979943,0cyclictest6445-21snmpd19:38:550
321979943,0cyclictest2941-21awk22:56:190
321979943,0cyclictest2856-21awk19:31:190
321979943,0cyclictest13207-21awk21:41:200
321979940,0cyclictest0-21swapper/018:46:240
28415040,0irq/27-eth0-tx-0-21swapper/020:01:270
28415040,0irq/27-eth0-tx-0-21swapper/018:41:160
28405040,0irq/26-eth0-rx-0-21swapper/323:57:153
28405040,0irq/26-eth0-rx-0-21swapper/323:51:273
28405040,0irq/26-eth0-rx-0-21swapper/323:01:583
28405040,0irq/26-eth0-rx-0-21swapper/322:51:233
28405040,0irq/26-eth0-rx-0-21swapper/320:56:583
28405040,0irq/26-eth0-rx-0-21swapper/318:41:113
259940,0migration/227729-21munin-node19:26:162
259940,0migration/224823-21kthreadcore18:56:222
249940,0watchdog/222075-21taskset19:46:142
249940,0watchdog/20-21swapper/220:36:262
249940,0watchdog/20-21swapper/220:17:502
249940,0watchdog/20-21swapper/220:11:062
249940,0watchdog/20-21swapper/219:56:222
249940,0watchdog/20-21swapper/219:51:222
249940,0watchdog/20-21swapper/219:11:222
249940,0watchdog/20-21swapper/219:06:302
249940,0watchdog/20-21swapper/218:55:102
249940,0watchdog/20-21swapper/218:41:222
179940,0migration/18524-21munin-node19:11:191
179940,0migration/15884-21timerwakeupswit18:41:271
179940,0migration/15522-21irqrtprio21:36:201
179940,0migration/15350-21kthreadcore21:56:211
179940,0migration/128456-21taskset19:50:391
179940,0migration/12565-21ksmtuned20:52:001
179940,0migration/124808-21kthreadcore19:21:241
179940,0migration/123092-21runrttasks20:12:021
179940,0migration/121815-21kthreadcore22:06:201
179940,0migration/120757-21kthreadcore23:46:211
179940,0migration/119223-21switchtime18:51:271
179940,0migration/118430-21kthreadcore19:41:241
179940,0migration/118345-21fschecks_count23:06:161
179940,0migration/110438-21gltestperf20:01:181
169940,0watchdog/19625-21hddtemp_smartct23:41:181
169940,0watchdog/18544-21ssh23:19:341
169940,0watchdog/18510-21python22:36:261
169940,0watchdog/16547-21runrttasks20:00:061
169940,0watchdog/16445-21snmpd19:51:221
169940,0watchdog/15915-21kthreadcore22:56:221
169940,0watchdog/130894-21cut22:11:221
169940,0watchdog/130894-21cut22:11:221
169940,0watchdog/130668-21kthreadcore19:26:221
169940,0watchdog/129652-21tr23:11:221
169940,0watchdog/119761-21tune2fs22:26:221
169940,0watchdog/118463-21diskmemload21:09:501
169940,0watchdog/112516-21irqrtprio22:21:221
149940,0migration/031607-21kthreadcore21:51:240
149940,0migration/014473-21kthreadcore21:21:220
35130,0ktimersoftd/30-21swapper/319:54:373
321989933,0cyclictest5543-21awk21:16:201
321989933,0cyclictest31685-21kworker/1:122:46:131
321989933,0cyclictest31685-21kworker/1:120:36:251
321989933,0cyclictest31685-21kworker/1:120:26:101
321989933,0cyclictest31685-21kworker/1:120:21:091
321989933,0cyclictest31685-21kworker/1:120:16:051
321989933,0cyclictest31685-21kworker/1:119:31:071
321989933,0cyclictest31685-21kworker/1:119:06:111
321989933,0cyclictest31685-21kworker/1:100:06:291
321989932,0cyclictest31685-21kworker/1:121:11:131
321979933,0cyclictest3425-21lldpd21:11:110
321979932,0cyclictest9196-21cut20:31:160
321979932,0cyclictest23612-21grep22:06:210
321979932,0cyclictest16397-21awk00:06:210
321979930,0cyclictest4304-21diskstats21:56:170
321979930,0cyclictest14415-21munin-node19:16:190
321979930,0cyclictest0-21swapper/018:56:120
28415030,0irq/27-eth0-tx-0-21swapper/019:21:190
28405030,0irq/26-eth0-rx-0-21swapper/320:26:143
28405030,0irq/26-eth0-rx-0-21swapper/318:57:113
249930,0watchdog/20-21swapper/218:46:102
169930,0watchdog/10-21swapper/120:41:421
149930,0migration/0646-21runrttasks19:05:000
321979922,0cyclictest6445-21snmpd20:11:260
321979920,0cyclictest16587-21latency_hist23:46:030
28405020,0irq/26-eth0-rx-0-21swapper/320:11:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional