You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-23 - 09:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Tue Dec 23, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
210324938,0sleep00-21swapper/019:05:380
230623627,0sleep30-21swapper/319:08:013
210023525,0sleep20-21swapper/219:05:362
239823127,0sleep10-21swapper/119:09:031
2469991919,0cyclictest20517-21kworker/2:021:59:372
2469991818,0cyclictest20517-21kworker/2:021:23:322
24699983,0cyclictest27381-21awk21:24:192
24699976,0cyclictest0-21swapper/223:15:242
24699972,0cyclictest5592-21awk19:59:212
24699972,0cyclictest29185-21awk23:39:202
24699971,0cyclictest0-21swapper/223:09:172
24699970,0cyclictest22656-21munin-node00:14:182
24699970,0cyclictest0-21swapper/221:50:122
24699970,0cyclictest0-21swapper/221:17:282
339960,0migration/38572-21kthreadcore00:24:213
339960,0migration/329692-21runrttasks00:16:343
339960,0migration/327685-21sh23:38:203
339960,0migration/323635-21sh00:33:483
29715060,0irq/26-eth0-rx-18290-21diskstats21:19:163
24699966,0cyclictest0-21swapper/223:34:192
24699963,0cyclictest0-21swapper/200:24:182
24699962,0cyclictest23848-21aten2.4-expect19:49:172
24699961,0cyclictest32214-21ssh22:43:482
24699960,0cyclictest4234-21latency_hist21:49:042
24699960,0cyclictest3588-21grep23:24:162
24699960,0cyclictest22401-21cat00:14:042
24699960,0cyclictest21261-21munin-node23:54:172
24699960,0cyclictest20017-21munin-node21:39:212
24699960,0cyclictest16994-21cat22:54:042
24699960,0cyclictest0-21swapper/223:50:092
24699960,0cyclictest0-21swapper/223:45:312
24699960,0cyclictest0-21swapper/223:30:382
24699960,0cyclictest0-21swapper/223:05:032
24699960,0cyclictest0-21swapper/222:58:542
24699960,0cyclictest0-21swapper/222:20:302
24699960,0cyclictest0-21swapper/222:08:052
24699960,0cyclictest0-21swapper/221:29:442
24699960,0cyclictest0-21swapper/221:11:112
24679960,0cyclictest547-21diskmemload23:03:120
149960,0migration/06510-21kthreadcore21:29:210
149960,0migration/06503-21sh22:44:320
149960,0migration/015984-21sh22:51:580
339950,0migration/35611-21if_eth1.319:59:213
339950,0migration/33802-21sh22:04:513
339950,0migration/33555-21latency_hist19:34:043
339950,0migration/332553-21latency_hist22:44:043
339950,0migration/332077-21kthreadcore19:54:233
339950,0migration/331418-21timerandwakeup20:39:273
339950,0migration/329153-21munin-node23:39:203
339950,0migration/327753-21kthreadcore22:59:223
339950,0migration/327213-21fschecks_time21:24:183
339950,0migration/325683-21kthreadcore19:49:223
339950,0migration/325298-21fschecks_time20:14:173
339950,0migration/323095-21cpuspeed_turbos22:19:183
339950,0migration/32211-21kthreadcore20:19:213
339950,0migration/321464-21sh00:12:003
339950,0migration/319589-21taskset20:57:443
339950,0migration/317761-21gltestperf19:44:173
339950,0migration/315195-21munin-node22:14:173
339950,0migration/314029-21diskmemload22:13:563
339950,0migration/313799-21kthreadcore21:34:203
339950,0migration/312529-21irqrtprio20:04:223
29725050,0irq/27-eth0-tx-19031-21munin-node20:34:170
29725050,0irq/27-eth0-tx-0-21swapper/021:09:200
29725050,0irq/27-eth0-tx-0-21swapper/020:59:110
29725050,0irq/27-eth0-tx-0-21swapper/020:14:140
29725050,0irq/27-eth0-tx-0-21swapper/019:38:580
29725050,0irq/27-eth0-tx-0-21swapper/019:33:570
29715050,0irq/26-eth0-rx-9808-21ps22:49:213
29715050,0irq/26-eth0-rx-7886-21grep22:29:183
29715050,0irq/26-eth0-rx-7085-21grep23:24:203
29715050,0irq/26-eth0-rx-6559-21runrttasks23:30:053
29715050,0irq/26-eth0-rx-5375-21if_eth123:44:183
29715050,0irq/26-eth0-rx-30327-21perl23:59:193
29715050,0irq/26-eth0-rx-27931-21if_eth1.323:19:213
29715050,0irq/26-eth0-rx-16900-21ssh23:10:593
29715050,0irq/26-eth0-rx-1327-21kthreadcore21:44:233
29715050,0irq/26-eth0-rx-12237-21ssh21:53:013
29715050,0irq/26-eth0-rx-0-21swapper/323:51:503
29715050,0irq/26-eth0-rx-0-21swapper/322:56:573
29715050,0irq/26-eth0-rx-0-21swapper/322:35:313
29715050,0irq/26-eth0-rx-0-21swapper/322:24:063
29715050,0irq/26-eth0-rx-0-21swapper/321:59:413
29715050,0irq/26-eth0-rx-0-21swapper/321:55:153
29715050,0irq/26-eth0-rx-0-21swapper/321:42:203
29715050,0irq/26-eth0-rx-0-21swapper/321:29:213
29715050,0irq/26-eth0-rx-0-21swapper/321:16:423
29715050,0irq/26-eth0-rx-0-21swapper/321:13:263
29715050,0irq/26-eth0-rx-0-21swapper/320:09:243
29715050,0irq/26-eth0-rx-0-21swapper/319:39:203
24699952,0cyclictest9392-21sensors_fan19:34:272
24699952,0cyclictest3556-21lldpd21:54:092
24699950,0cyclictest6099-21munin-node00:04:162
24699950,0cyclictest30327-21perl23:59:192
24699950,0cyclictest26560-21cat21:04:042
24699950,0cyclictest24478-21tr20:59:222
24699950,0cyclictest0-21swapper/223:19:342
24699950,0cyclictest0-21swapper/222:59:582
24699950,0cyclictest0-21swapper/222:45:292
24699950,0cyclictest0-21swapper/222:36:032
24699950,0cyclictest0-21swapper/222:29:272
24699950,0cyclictest0-21swapper/222:25:562
24699950,0cyclictest0-21swapper/222:14:072
24699950,0cyclictest0-21swapper/222:09:212
24699950,0cyclictest0-21swapper/221:34:192
24699950,0cyclictest0-21swapper/200:35:062
24699950,0cyclictest0-21swapper/200:29:212
24699950,0cyclictest0-21swapper/200:19:202
24689955,0cyclictest0-21swapper/123:59:211
24689955,0cyclictest0-21swapper/123:50:031
24689955,0cyclictest0-21swapper/122:33:061
24689955,0cyclictest0-21swapper/100:05:111
24689954,0cyclictest11193-21ps23:09:211
24689950,0cyclictest0-21swapper/123:05:541
24689950,0cyclictest0-21swapper/121:37:081
24689950,0cyclictest0-21swapper/100:34:211
24679955,0cyclictest0-21swapper/023:44:590
24679955,0cyclictest0-21swapper/022:28:310
24679955,0cyclictest0-21swapper/021:59:590
24679950,0cyclictest0-21swapper/022:58:570
24679950,0cyclictest0-21swapper/000:26:080
179950,0migration/132096-21idleruntime22:24:201
179950,0migration/13203-21munin-node22:04:261
179950,0migration/131225-21munin-node22:59:251
179950,0migration/129968-21sendmail_mailqu22:39:251
179950,0migration/120447-21munin-node21:39:161
179950,0migration/118581-21munin-node21:54:241
179950,0migration/117468-21cpuspeed_turbos22:54:171
179950,0migration/111333-21latency22:09:221
179950,0migration/111300-21munin-node20:24:241
179950,0migration/110583-21sh23:27:471
169950,0watchdog/130188-21kthreadcore23:39:221
149950,0migration/09468-21cpuspeed_turbos21:14:170
149950,0migration/03951-21irqrtprio19:09:210
149950,0migration/031319-21kthreadcore23:59:230
149950,0migration/031275-21kthreadcore21:04:210
149950,0migration/028941-21kthreadcore21:24:220
149950,0migration/028523-21proc_pri22:19:280
149950,0migration/027266-21kthreadcore19:24:220
149950,0migration/027112-21sh23:36:590
149950,0migration/024089-21kthreadcore23:54:230
149950,0migration/023154-21fschecks_count00:14:180
149950,0migration/022466-21taskset19:24:000
149950,0migration/020027-21kthreadcore19:44:210
149950,0migration/019765-21kthreadcore00:29:210
149950,0migration/019588-21kthreadcore21:19:200
149950,0migration/017889-21sh21:34:520
149950,0migration/017684-21kthreadcore22:14:200
149950,0migration/015422-21munin-node19:14:280
149950,0migration/015031-21kthreadcore19:39:210
149950,0migration/01434-21diskmemload22:04:220
149950,0migration/013066-21kthreadcore23:29:220
149950,0migration/013064-21turbostat20:54:040
149950,0migration/012642-21sh22:10:420
339940,0migration/327745-21kthreadcore19:24:223
339940,0migration/318361-21kthreadcore19:19:233
29725040,0irq/27-eth0-tx-0-21swapper/020:54:290
29715040,0irq/26-eth0-rx-0-21swapper/323:54:223
29715040,0irq/26-eth0-rx-0-21swapper/323:14:243
29715040,0irq/26-eth0-rx-0-21swapper/323:04:213
29715040,0irq/26-eth0-rx-0-21swapper/322:39:243
29715040,0irq/26-eth0-rx-0-21swapper/300:35:153
29715040,0irq/26-eth0-rx-0-21swapper/300:19:163
29715040,0irq/26-eth0-rx-0-21swapper/300:04:203
24699943,0cyclictest7573-21awk20:49:222
24699943,0cyclictest3556-21lldpd20:54:082
24699943,0cyclictest3556-21lldpd20:24:072
24699943,0cyclictest3556-21lldpd19:42:072
24699943,0cyclictest3514-21snmpd20:40:092
24699943,0cyclictest3514-21snmpd20:29:092
24699943,0cyclictest3514-21snmpd20:09:092
24699943,0cyclictest3514-21snmpd20:07:242
24699943,0cyclictest3514-21snmpd19:27:542
24699943,0cyclictest32161-21awk20:19:192
24699943,0cyclictest25426-21sadc20:39:042
24689943,0cyclictest4532-21awk19:34:211
24689943,0cyclictest24601-21awk19:49:201
24689943,0cyclictest18919-21ssh21:19:191
24679944,0cyclictest0-21swapper/000:20:180
24679943,0cyclictest3556-21lldpd23:53:090
24679943,0cyclictest3556-21lldpd21:53:390
24679943,0cyclictest29381-21awk21:44:200
24679943,0cyclictest25660-21munin-node20:39:270
179940,0migration/19900-21kthreadcore00:24:221
179940,0migration/19135-21sshd19:13:241
179940,0migration/15178-21fschecks_time19:59:191
179940,0migration/14057-21kthreadcore20:44:221
179940,0migration/13329-21kthreadcore20:19:221
179940,0migration/13315-21needreboot19:54:251
179940,0migration/129995-21taskset00:17:171
179940,0migration/128468-21kthreadcore21:24:221
179940,0migration/125833-21timerwakeupswit23:34:271
179940,0migration/125076-21kthreadcore20:59:221
179940,0migration/123518-21idleruntime19:24:191
179940,0migration/122388-21taskset19:23:081
179940,0migration/119630-21sh00:09:221
179940,0migration/118113-21aten_r1power_po20:09:181
179940,0migration/117806-21netstat20:29:231
179940,0migration/114582-21kernelversion20:54:221
179940,0migration/114511-21aten_r1power_po22:14:151
179940,0migration/112162-21gltestperf23:29:191
169940,0watchdog/1798-21grep19:29:231
169940,0watchdog/17427-21ssh21:10:581
169940,0watchdog/131577-21kthreadcore21:44:221
169940,0watchdog/124924-21taskset23:15:061
169940,0watchdog/124737-21switchtime20:34:271
169940,0watchdog/124358-21kthreadcore21:59:221
169940,0watchdog/122553-21ssh00:31:181
169940,0watchdog/110747-21ssh23:44:501
169940,0watchdog/10-21swapper/121:49:221
169940,0watchdog/10-21swapper/121:14:261
169940,0watchdog/10-21swapper/120:04:231
169940,0watchdog/10-21swapper/100:19:221
149940,0migration/08350-21kthreadcore00:04:210
149940,0migration/06407-21sh20:24:170
149940,0migration/05438-21kthreadcore23:24:200
149940,0migration/03425-21kthreadcore20:44:210
149940,0migration/023740-21kthreadcore21:39:200
149940,0migration/021857-21kthreadcore20:09:230
149940,0migration/020514-21diskmemload00:09:470
149940,0migration/016563-21idleruntime22:34:170
149940,0migration/012215-21kthreadcore22:29:220
35130,0ktimersoftd/30-21swapper/319:16:143
29715030,0irq/26-eth0-rx-0-21swapper/320:44:163
29715030,0irq/26-eth0-rx-0-21swapper/320:34:163
29715030,0irq/26-eth0-rx-0-21swapper/320:31:543
29715030,0irq/26-eth0-rx-0-21swapper/319:34:293
24699932,0cyclictest8438-21sensors19:09:262
24699932,0cyclictest3556-21lldpd19:55:372
24699932,0cyclictest3556-21lldpd19:44:072
24699932,0cyclictest3514-21snmpd20:44:092
24699932,0cyclictest3514-21snmpd20:16:392
24699932,0cyclictest30197-21awk19:29:202
24699930,0cyclictest21554-21open_inodes19:19:262
24699930,0cyclictest10438-21iostat_ios19:14:202
24689932,0cyclictest8573-21sensors21:29:251
24689932,0cyclictest6710-21basename20:49:181
24689932,0cyclictest3514-21snmpd22:34:391
24689932,0cyclictest3514-21snmpd19:48:581
24689932,0cyclictest27745-21idleruntime23:19:201
24689932,0cyclictest25494-21awk20:14:181
24689932,0cyclictest22155-21awk23:54:211
24689932,0cyclictest10505-21aten2.4-expect19:39:151
24679932,0cyclictest3514-21snmpd20:29:050
24679932,0cyclictest30628-21expr19:54:190
24679932,0cyclictest29316-21awk23:39:210
24679932,0cyclictest24577-21awk19:49:200
24679932,0cyclictest2038-21diskstats23:04:180
24679932,0cyclictest19533-21awk23:14:220
24679932,0cyclictest16382-21grep20:04:230
24679932,0cyclictest13872-21awk21:54:190
24679932,0cyclictest11019-21awk23:09:200
24679931,0cyclictest0-21swapper/022:39:220
24679930,0cyclictest4631-21munin-node19:59:180
169930,0watchdog/123006-21ssh22:19:181
169930,0watchdog/113079-21kthreadcore22:49:221
169930,0watchdog/10-21swapper/122:44:381
169930,0watchdog/10-21swapper/119:14:271
149930,0migration/030829-21runrttasks00:35:590
149930,0migration/02377-21diskmemload23:23:330
35120,0ktimersoftd/33353-21munin-node19:09:183
339920,0migration/320342-21diskstats20:59:173
29715020,0irq/26-eth0-rx-0-21swapper/321:04:343
29715020,0irq/26-eth0-rx-0-21swapper/320:49:093
29715020,0irq/26-eth0-rx-0-21swapper/320:24:043
24689920,0cyclictest0-21swapper/121:04:041
24689920,0cyclictest0-21swapper/120:39:041
24679920,0cyclictest0-21swapper/020:19:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional