You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-25 - 14:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Sat Apr 25, 2026 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2336992522,2cyclictest13662-21awk22:30:003
232799250,1cyclictest14596-21ssh23:00:591
232699251,1cyclictest62550irq/127-eno100:15:110
232699250,24cyclictest28014-21diskmemload22:27:300
232699250,24cyclictest0-21swapper/021:56:010
232699250,23cyclictest26120-21rm21:38:570
232699250,0cyclictest0-21swapper/023:11:080
233699240,24cyclictest0-21swapper/300:37:083
233699240,23cyclictest28014-21diskmemload22:45:333
233699240,1cyclictest0-21swapper/300:15:023
2331992424,0cyclictest0-21swapper/223:00:102
233199240,24cyclictest10891-21ssh21:24:062
233199240,23cyclictest7816-21apt-get19:20:282
233199240,23cyclictest28014-21diskmemload23:43:162
233199240,23cyclictest28014-21diskmemload23:32:422
233199240,23cyclictest25727-21apt-get22:10:292
233199240,23cyclictest0-21swapper/223:48:562
233199240,23cyclictest0-21swapper/221:32:312
233199240,23cyclictest0-21swapper/221:11:322
232799240,24cyclictest0-21swapper/122:35:481
232799240,23cyclictest30528-21sh23:16:481
232799240,23cyclictest0-21swapper/122:25:001
232799240,23cyclictest0-21swapper/100:18:331
232799240,1cyclictest28590-21sh22:44:231
232799240,1cyclictest0-21swapper/100:34:581
2326992423,1cyclictest13-21ksoftirqd/000:20:280
232699240,23cyclictest28014-21diskmemload22:13:330
232699240,23cyclictest28014-21diskmemload21:46:040
232699240,23cyclictest28014-21diskmemload00:14:220
2336992323,0cyclictest28014-21diskmemload22:18:003
2336992323,0cyclictest28014-21diskmemload21:40:053
2336992323,0cyclictest0-21swapper/322:03:563
2336992323,0cyclictest0-21swapper/300:07:213
2336992322,1cyclictest0-21swapper/300:30:033
2336992321,1cyclictest31689-21users00:20:393
2336992321,1cyclictest0-21swapper/322:55:383
2336992321,1cyclictest0-21swapper/322:05:173
2336992321,1cyclictest0-21swapper/321:33:523
2336992321,1cyclictest0-21swapper/321:11:423
2336992320,2cyclictest3507-21/usr/sbin/munin22:20:383
233699230,2cyclictest8150-21sh23:59:033
233699230,23cyclictest28014-21diskmemload21:21:133
233699230,23cyclictest0-21swapper/323:16:333
233699230,23cyclictest0-21swapper/323:03:193
233699230,22cyclictest28014-21diskmemload22:10:283
233699230,22cyclictest28014-21diskmemload21:55:303
233699230,22cyclictest28014-21diskmemload21:16:273
233699230,22cyclictest19802-21ssh23:05:593
233699230,22cyclictest12846-21sh23:30:403
233699230,22cyclictest12691-21sh00:02:553
233699230,1cyclictest0-21swapper/322:51:073
233699230,1cyclictest0-21swapper/322:35:493
2331992323,0cyclictest28014-21diskmemload21:15:512
2331992322,1cyclictest0-21swapper/222:26:042
2331992322,1cyclictest0-21swapper/221:58:012
2331992321,1cyclictest0-21swapper/222:38:052
2331992320,2cyclictest24389-21sh22:08:482
233199231,21cyclictest15661-21awk00:05:352
233199230,23cyclictest28014-21diskmemload23:38:402
233199230,23cyclictest0-21swapper/200:22:022
233199230,23cyclictest0-21swapper/200:11:442
233199230,22cyclictest8774-21idleruntime-cro20:30:012
233199230,22cyclictest7163-21rm23:25:352
233199230,22cyclictest4054-21sh23:55:172
233199230,22cyclictest28014-21diskmemload21:35:302
233199230,22cyclictest24632-21ssh00:15:232
233199230,22cyclictest12369-21cron19:30:002
233199230,22cyclictest0-21swapper/222:51:072
233199230,1cyclictest31435-21rm23:18:162
233199230,0cyclictest0-21swapper/222:55:182
2327992323,0cyclictest0-21swapper/123:41:001
2327992323,0cyclictest0-21swapper/123:30:131
2327992323,0cyclictest0-21swapper/122:58:301
2327992322,1cyclictest5931-21perf21:20:011
2327992321,2cyclictest8147-21ssh21:20:351
2327992321,1cyclictest0-21swapper/123:45:331
2327992321,1cyclictest0-21swapper/100:06:551
2327992320,2cyclictest31914-21/usr/sbin/munin23:50:321
232799230,23cyclictest28014-21diskmemload22:28:261
232799230,23cyclictest16659-21ssh22:31:381
232799230,23cyclictest0-21swapper/121:45:261
232799230,23cyclictest0-21swapper/121:25:101
232799230,23cyclictest0-21swapper/121:12:211
232799230,22cyclictest5456-21rm00:27:161
232799230,22cyclictest4476-21runrttasks22:51:161
232799230,22cyclictest31735-21ssh00:20:421
232799230,22cyclictest30468-21diskmemload21:42:381
232799230,22cyclictest28001-21cron21:10:001
232799230,22cyclictest23450-21ssh22:07:141
232799230,22cyclictest20090-21latency_hist22:05:011
232799230,22cyclictest15073-21sh00:35:381
232799230,22cyclictest0-21swapper/121:55:031
232799230,0cyclictest0-21swapper/123:09:191
2326992323,0cyclictest28014-21diskmemload22:38:360
2326992323,0cyclictest151rcu_preempt21:30:480
2326992323,0cyclictest0-21swapper/022:30:170
2326992322,1cyclictest0-21swapper/022:03:110
2326992321,1cyclictest15685-21ssh00:36:310
2326992321,1cyclictest0-21swapper/021:50:290
2326992320,2cyclictest32081-21sh21:13:250
2326992320,2cyclictest24049-21sh23:42:110
2326992320,2cyclictest19863-21ssh23:38:480
232699230,2cyclictest9258-21ssh00:00:180
232699230,23cyclictest28014-21diskmemload21:40:390
232699230,23cyclictest0-21swapper/000:27:160
232699230,22cyclictest6347-21cron20:25:010
232699230,22cyclictest606-21dbus-daemon22:20:010
232699230,22cyclictest32127-21sh22:46:470
232699230,22cyclictest26961-21sh22:41:490
232699230,22cyclictest23008-21ssh22:06:350
232699230,22cyclictest20819-21ssh23:07:390
232699230,21cyclictest0-21swapper/022:55:130
232699230,0cyclictest0-21swapper/023:25:120
2336992222,0cyclictest0-21swapper/321:50:143
2336992222,0cyclictest0-21swapper/320:55:283
2336992221,1cyclictest0-21swapper/323:29:463
2336992220,1cyclictest29465-21apt-get20:05:283
2336992220,1cyclictest13951-21sh21:25:463
2336992220,1cyclictest0-21swapper/323:46:523
2336992220,1cyclictest0-21swapper/323:40:473
2336992220,1cyclictest0-21swapper/320:00:003
233699220,2cyclictest8766-21latency_hist20:30:013
233699220,2cyclictest5345-21sh00:27:063
233699220,2cyclictest16037-21/usr/sbin/munin23:35:293
233699220,21cyclictest0-21swapper/323:50:303
233699220,21cyclictest0-21swapper/322:32:523
233699220,1cyclictest12361-21turbostat19:30:003
233699220,1cyclictest0-21swapper/323:23:133
233699220,1cyclictest0-21swapper/322:42:143
233699220,1cyclictest0-21swapper/300:11:513
2331992222,0cyclictest0-21swapper/222:00:432
2331992222,0cyclictest0-21swapper/200:37:082
2331992221,1cyclictest0-21swapper/223:20:402
2331992220,1cyclictest31451-21apt-get23:50:302
2331992220,1cyclictest1191-21ssh22:49:142
2331992219,2cyclictest28014-21diskmemload23:05:302
233199220,21cyclictest4271-21apt-get20:20:292
233199220,21cyclictest32546-21cpuspeed21:45:292
233199220,21cyclictest24923-21df_inode22:40:332
233199220,21cyclictest11854-21ssh21:25:152
233199220,21cyclictest0-21swapper/222:16:002
233199220,21cyclictest0-21swapper/221:50:172
233199220,21cyclictest0-21swapper/220:00:282
233199220,1cyclictest0-21swapper/222:30:092
233199220,1cyclictest0-21swapper/222:21:202
233199220,1cyclictest0-21swapper/200:01:122
233199220,0cyclictest0-21swapper/223:14:072
233199220,0cyclictest0-21swapper/200:30:122
2327992222,0cyclictest0-21swapper/122:10:281
2327992222,0cyclictest0-21swapper/121:52:011
2327992221,1cyclictest28014-21diskmemload23:10:531
2327992220,1cyclictest7867-21ssh23:26:001
2327992220,1cyclictest14783-21munin-run19:35:011
232799220,22cyclictest0-21swapper/100:11:311
232799220,21cyclictest29537-21apt-get22:45:281
232799220,21cyclictest0-21swapper/121:05:001
232799220,21cyclictest0-21swapper/120:55:281
232799220,21cyclictest0-21swapper/100:00:071
232799220,1cyclictest774-21runrttasks21:38:511
232799220,1cyclictest6666-21apt-get20:25:291
232799220,1cyclictest19400-21sh23:38:011
232799220,1cyclictest16964-21sh21:30:191
232799220,1cyclictest0-21swapper/123:56:321
232799220,1cyclictest0-21swapper/120:40:011
2326992222,0cyclictest131ksoftirqd/021:29:360
2326992222,0cyclictest0-21swapper/023:51:280
2326992220,1cyclictest28037-21idleruntime-cro21:10:010
232699220,21cyclictest4145-21sh22:50:380
232699220,21cyclictest3881-21ssh23:55:020
232699220,21cyclictest0-21swapper/021:20:290
232699220,21cyclictest0-21swapper/019:30:270
232699220,21cyclictest0-21swapper/000:30:310
232699220,1cyclictest6320-21sh22:22:080
232699220,1cyclictest10997-21apt-get23:30:280
2336992121,0cyclictest0-21swapper/320:20:293
2336992119,1cyclictest619-21polkitd20:20:003
233699210,1cyclictest0-21swapper/321:35:283
233699210,0cyclictest0-21swapper/323:11:143
2331992120,0cyclictest0-21swapper/221:40:012
2331992120,0cyclictest0-21swapper/220:50:292
233199210,20cyclictest0-21swapper/219:50:282
233199210,1cyclictest26762-21latency_hist20:00:002
232799210,21cyclictest0-21swapper/122:15:071
232799210,20cyclictest31861-21apt-get20:10:291
232799210,20cyclictest0-21swapper/120:45:281
2326992121,0cyclictest0-21swapper/023:15:180
2326992121,0cyclictest0-21swapper/020:45:270
2326992121,0cyclictest0-21swapper/019:35:280
232699210,21cyclictest0-21swapper/023:00:420
232699210,20cyclictest0-21swapper/020:35:280
232699210,20cyclictest0-21swapper/019:25:270
232699210,0cyclictest0-21swapper/021:00:270
232699210,0cyclictest0-21swapper/000:06:100
2336992020,0cyclictest29184-21perf20:05:003
2336992020,0cyclictest22261-21apt-get19:50:283
2336992020,0cyclictest0-21swapper/319:40:283
233699200,0cyclictest25923-21apt-get21:05:273
233699200,0cyclictest18719-21apt-get20:50:293
2331992020,0cyclictest0-21swapper/220:05:282
233199200,0cyclictest0-21swapper/220:55:282
233199200,0cyclictest0-21swapper/220:30:282
2327992018,1cyclictest3990-21munin-run20:20:001
2327992018,1cyclictest0-21swapper/120:25:011
232799200,19cyclictest0-21swapper/119:45:281
2326992020,0cyclictest0-21swapper/020:50:290
2326992020,0cyclictest0-21swapper/020:05:280
2326992020,0cyclictest0-21swapper/019:40:290
2336991919,0cyclictest0-21swapper/320:40:303
2336991919,0cyclictest0-21swapper/319:15:283
2336991917,1cyclictest11228-21dump-pmu-power20:35:013
2331991919,0cyclictest0-21swapper/221:05:282
2331991919,0cyclictest0-21swapper/219:10:282
2331991918,0cyclictest13931-21apt-get20:40:302
2331991917,1cyclictest3960-21cron20:20:002
2326991919,0cyclictest0-21swapper/020:55:270
2326991919,0cyclictest0-21swapper/020:25:280
2326991919,0cyclictest0-21swapper/020:10:290
2336991818,0cyclictest0-21swapper/319:35:283
2336991818,0cyclictest0-21swapper/319:20:283
233699180,17cyclictest606-21dbus-daemon19:15:003
2331991818,0cyclictest0-21swapper/219:40:282
2331991817,1cyclictest16323-21apt-get20:45:272
2331991817,0cyclictest5393-21apt-get19:15:282
2327991818,0cyclictest0-21swapper/119:15:021
2327991816,1cyclictest27071-21apt-get20:00:281
232799180,2cyclictest606-21dbus-daemon19:15:001
2326991818,0cyclictest0-21swapper/019:45:280
2326991818,0cyclictest0-21swapper/019:10:280
2326991817,1cyclictest24383-21perf19:55:010
232799170,7cyclictest313-21systemd-journal20:45:011
2326991717,0cyclictest0-21swapper/020:15:270
2326991717,0cyclictest0-21swapper/019:20:280
2326991716,0cyclictest0-21swapper/020:30:280
2336991614,1cyclictest11512-21apt-get20:35:273
233699160,15cyclictest25011-21sort21:00:363
233699160,15cyclictest0-21swapper/319:45:273
2331991616,0cyclictest0-21swapper/219:35:282
2331991614,1cyclictest12561-21ntp_states20:35:332
2327991616,0cyclictest15059-21apt-get19:35:281
232699168,7cyclictest13-21ksoftirqd/020:05:000
2326991614,2cyclictest13-21ksoftirqd/020:40:300
2326991613,0cyclictest151rcu_preempt19:20:000
2331991513,1cyclictest19866-21apt-get19:45:282
2331991513,1cyclictest0-21swapper/221:00:362
2327991515,0cyclictest0-21swapper/120:05:281
232799150,8cyclictest313-21systemd-journal20:00:001
232799150,1cyclictest0-21swapper/120:30:311
233699140,14cyclictest0-21swapper/320:10:153
233699140,14cyclictest0-21swapper/319:30:503
233199140,14cyclictest0-21swapper/219:30:202
233199140,0cyclictest0-21swapper/220:10:552
2327991414,0cyclictest0-21swapper/119:40:211
232799140,14cyclictest0-21swapper/119:50:131
232799140,0cyclictest0-21swapper/120:50:481
232799140,0cyclictest0-21swapper/119:21:181
2326991414,0cyclictest0-21swapper/019:55:460
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional