You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 09:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue May 12, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
138921330,0sleep30-21swapper/323:34:403
2613229682,9sleep00-21swapper/019:08:390
2601429482,8sleep20-21swapper/219:07:082
2598628669,13sleep10-21swapper/119:06:471
2596628370,9sleep30-21swapper/319:06:333
164422520,0sleep30-21swapper/322:05:323
2632899260,23cyclictest21438-21apt-get21:50:112
2632499260,3cyclictest0-21swapper/123:05:121
2633099251,21cyclictest0-21swapper/322:32:153
2633099251,21cyclictest0-21swapper/322:32:153
26328992524,1cyclictest0-21swapper/200:20:252
2633099240,23cyclictest0-21swapper/323:17:433
26330992322,1cyclictest0-21swapper/323:35:253
2633099230,23cyclictest0-21swapper/300:12:323
26320992313,10cyclictest13-21ksoftirqd/023:19:350
26324992220,2cyclictest0-21swapper/122:10:141
2632499220,20cyclictest0-21swapper/123:30:131
2632499220,1cyclictest18853-21diskmemload23:46:421
2632099220,15cyclictest4782-21apache_accesses00:15:110
2632099220,15cyclictest4782-21apache_accesses00:15:110
26330992119,1cyclictest0-21swapper/322:16:533
2632499210,21cyclictest0-21swapper/121:54:591
2633099201,18cyclictest0-21swapper/322:04:483
26324992020,0cyclictest0-21swapper/100:36:211
2632499200,1cyclictest0-21swapper/100:15:211
2632499200,1cyclictest0-21swapper/100:15:211
26330991917,1cyclictest10721-21apt-get23:40:133
2633099190,1cyclictest0-21swapper/322:10:143
2632099196,6cyclictest8170-21awk19:39:590
2632099193,1cyclictest18853-21diskmemload00:23:260
237602190,0sleep30-21swapper/322:49:523
323032180,1chrt0-21swapper/020:30:000
2633099180,17cyclictest0-21swapper/321:15:323
2632899184,3cyclictest141rcu_preempt23:37:262
2632499187,8cyclictest15222-21/usr/sbin/munin22:25:141
26324991817,1cyclictest0-21swapper/123:50:321
26324991816,1cyclictest0-21swapper/122:40:131
2632499180,18cyclictest0-21swapper/122:32:221
2632499180,18cyclictest0-21swapper/122:32:221
2632099186,6cyclictest27254-21munin-run20:20:010
26330991714,3cyclictest0-21swapper/323:26:363
2632899170,16cyclictest0-21swapper/222:21:172
26324991716,1cyclictest18853-21diskmemload00:30:351
2632499170,12cyclictest0-21swapper/122:16:581
26320991715,1cyclictest0-21swapper/019:10:130
230812170,2sleep20-21swapper/200:25:252
2632899161,1cyclictest141rcu_preempt22:28:532
2632899160,5cyclictest0-21swapper/223:33:342
2632899160,1cyclictest0-21swapper/223:49:442
2632499160,1cyclictest2434-21basename23:35:131
2632499160,14cyclictest0-21swapper/100:23:111
2633099153,9cyclictest12656-21munin-run00:20:003
2633099153,9cyclictest12656-21munin-run00:20:003
2633099150,15cyclictest0-21swapper/323:09:133
2633099150,14cyclictest13123-21apt-get21:45:153
26328991511,3cyclictest0-21swapper/221:47:152
2632499151,1cyclictest141rcu_preempt23:40:051
2632499150,1cyclictest18853-21diskmemload22:24:111
2632499150,14cyclictest0-21swapper/120:50:241
2632099158,5cyclictest20755-21apache223:36:440
26330991413,1cyclictest0-21swapper/323:22:433
26330991411,2cyclictest31605-21wc21:55:253
2633099141,10cyclictest0-21swapper/322:25:253
2632899140,1cyclictest0-21swapper/223:15:422
26324991414,0cyclictest0-21swapper/123:10:101
26324991414,0cyclictest0-21swapper/122:59:311
26324991412,1cyclictest18853-21diskmemload22:53:311
26324991411,2cyclictest30544-21grep19:15:261
2632499140,3cyclictest9913-21tr23:20:131
26320991410,3cyclictest7211-21cron22:40:000
26320991410,3cyclictest12895-21cron19:50:000
26320991410,2cyclictest20548-21wc00:05:150
2632099140,4cyclictest141rcu_preempt22:52:490
2632099140,14cyclictest0-21swapper/023:06:560
2632099140,14cyclictest0-21swapper/022:47:320
2632099140,12cyclictest21403-21grep21:10:290
2633099139,3cyclictest27240-21cron20:20:003
2633099139,3cyclictest21363-21wc00:05:203
26330991310,2cyclictest19642-21grep23:45:163
2633099130,13cyclictest0-21swapper/300:20:383
2633099130,12cyclictest5608-21head21:40:193
2633099130,11cyclictest0-21swapper/323:00:143
2632899139,3cyclictest32286-21cat19:20:182
2632899139,3cyclictest2040-21grep19:25:172
2632899139,3cyclictest19685-21cron00:05:002
26328991310,2cyclictest3894-21sed23:55:162
26328991310,2cyclictest21946-21cat20:05:252
2632899130,12cyclictest23500-21grep22:10:172
2632899130,12cyclictest0-21swapper/221:22:112
2632499139,3cyclictest30139-21grep00:10:211
2632499139,3cyclictest14742-21wc19:50:271
2632499135,7cyclictest31742-21grep21:55:251
26324991310,2cyclictest6287-21grep19:35:141
26324991310,2cyclictest29610-21cut21:35:181
26324991310,2cyclictest29189-21grep20:20:261
2632499130,1cyclictest18853-21diskmemload00:06:471
2632499130,13cyclictest0-21swapper/122:38:421
2632499130,0cyclictest0-21swapper/123:29:121
2632099139,3cyclictest469-21sed22:15:260
2632099132,7cyclictest8686-21/usr/sbin/munin23:00:270
26320991310,2cyclictest6501-21grep00:35:160
26320991310,2cyclictest23299-21grep21:50:250
26320991310,2cyclictest11559-21tail23:20:250
2632099130,0cyclictest0-21swapper/021:30:020
2633099129,2cyclictest999-21cron19:25:003
2633099129,2cyclictest7894-21sort20:45:173
2633099129,2cyclictest26211-21grep23:10:173
2633099129,2cyclictest24725-21ssh22:50:163
2633099129,2cyclictest2332-21grep22:55:263
2633099129,2cyclictest1569-21tr19:25:163
2633099129,2cyclictest14648-21tr21:00:143
2633099129,2cyclictest12508-21sed19:45:273
2633099129,2cyclictest10136-21df20:50:163
2633099122,5cyclictest15570-21aten2_r7power_c19:55:143
2633099121,11cyclictest0-21swapper/322:20:063
2632899129,2cyclictest8154-21ls19:39:592
2632899129,2cyclictest6838-21grep22:00:192
2632899129,2cyclictest4366-21grep19:30:162
2632899129,2cyclictest31216-21grep21:55:212
2632899129,2cyclictest30798-21ssh00:10:282
2632899129,2cyclictest28251-21cron00:10:002
2632899129,2cyclictest27415-21dpkg21:15:012
2632899129,2cyclictest26459-21uname23:10:192
2632899129,2cyclictest24399-21grep22:30:172
2632899129,2cyclictest24399-21grep22:30:172
2632899129,2cyclictest23527-21sh22:49:412
2632899129,2cyclictest14808-21grep19:50:262
2632899129,2cyclictest1467-21grep22:35:292
2632899129,2cyclictest13041-21ssh21:25:182
2632899129,2cyclictest11689-21cron20:55:002
2632899129,2cyclictest11404-21ssh23:40:182
2632899128,3cyclictest30063-21tr19:15:192
2632899124,6cyclictest8225-21munin-run23:00:002
2632899122,9cyclictest10172-21cstates23:20:132
2632899122,9cyclictest0-21swapper/221:31:112
26328991210,1cyclictest31882-21grep00:30:272
2632899120,12cyclictest0-21swapper/221:42:012
2632499129,2cyclictest22374-21head00:25:191
2632499129,2cyclictest17589-21wc22:45:211
2632499129,2cyclictest14636-21tr21:00:141
2632499129,2cyclictest12580-21sh21:45:001
2632499128,3cyclictest4167-21tr21:20:151
2632499125,6cyclictest25730-21sh21:32:561
2632499120,11cyclictest0-21swapper/123:55:291
2632499120,0cyclictest0-21swapper/123:16:371
2632099129,2cyclictest5614-21head21:40:190
2632099129,2cyclictest30668-21sed21:55:160
2632099129,2cyclictest30658-21grep00:10:260
2632099129,2cyclictest28175-21cut21:15:150
2632099129,2cyclictest18893-21sed23:25:180
2632099129,2cyclictest18649-21grep20:00:160
2632099129,2cyclictest16677-21tr22:25:240
2632099129,2cyclictest15546-21uname22:05:190
2632099129,2cyclictest13416-21tr21:45:150
2632099128,3cyclictest5424-21wc21:20:230
2632099128,3cyclictest14402-21grep19:50:230
2632099128,3cyclictest11688-21cron20:55:000
2632099125,5cyclictest7580-21df_inode22:20:140
2632099123,3cyclictest225-21systemd-journal21:10:000
26320991211,1cyclictest0-21swapper/021:26:310
2632099120,6cyclictest4277-21idleruntime-cro21:40:000
2633099118,2cyclictest9288-21readlink22:40:223
2633099118,2cyclictest5077-21sort21:20:213
2633099118,2cyclictest3367-21ssh22:36:593
2633099118,2cyclictest30587-21tail19:15:253
2633099118,2cyclictest28683-21awk19:15:003
2633099118,2cyclictest27384-21apache_accesses23:50:133
2633099118,2cyclictest24066-21ssh00:25:523
2633099118,2cyclictest23331-21sh21:50:243
2633099118,2cyclictest21992-21sed20:05:263
2633099118,2cyclictest20420-21ssh21:30:103
2633099118,2cyclictest15079-21ssh00:01:223
2633099118,2cyclictest14067-21sh21:00:013
2633099118,2cyclictest13681-21gzip21:25:233
2633099118,2cyclictest1357-21sh00:31:543
2633099118,2cyclictest10247-21ssh00:37:223
2633099117,2cyclictest19460-21grep21:10:143
2633099116,2cyclictest5341-21users23:55:273
2633099110,7cyclictest0-21swapper/321:35:423
2632899118,2cyclictest9174-21taskset20:48:142
2632899118,2cyclictest8415-21basename19:40:112
2632899118,2cyclictest8107-21cat22:40:142
2632899118,2cyclictest6612-21ls00:15:272
2632899118,2cyclictest6612-21ls00:15:262
2632899118,2cyclictest32237-21ls20:30:012
2632899118,2cyclictest30824-21ssh22:53:522
2632899118,2cyclictest29384-21sed23:50:262
2632899118,2cyclictest29010-21cut21:35:152
2632899118,2cyclictest2883-21ssh22:17:112
2632899118,2cyclictest22235-21rm23:27:072
2632899118,2cyclictest21820-21ssh23:07:352
2632899118,2cyclictest17468-21grep21:05:172
2632899118,2cyclictest15461-21strings21:00:192
2632899118,2cyclictest14369-21wget22:05:132
2632899118,2cyclictest13863-21rm23:02:542
2632899118,2cyclictest1365-21ssh21:18:262
2632899118,2cyclictest13071-21uname20:55:202
2632899118,2cyclictest12829-21chrt19:49:112
2632899117,3cyclictest28592-21cat20:20:202
2632499119,1cyclictest18445-21grep21:05:261
2632499118,2cyclictest9791-21tr23:00:191
2632499118,2cyclictest9392-21cat20:50:011
2632499118,2cyclictest890-21chrt19:23:381
2632499118,2cyclictest5627-21taskset19:33:221
2632499118,2cyclictest5131-21grep20:40:141
2632499118,2cyclictest5131-21grep20:40:141
2632499118,2cyclictest30446-21expr20:25:151
2632499118,2cyclictest27729-21ssh21:15:021
2632499118,2cyclictest23517-21cut20:10:171
2632499118,2cyclictest2033-21grep19:25:171
2632499118,2cyclictest19809-21ssh21:29:521
2632499118,2cyclictest19273-21ssh21:10:111
2632499118,2cyclictest17540-21seq19:58:391
2632499118,2cyclictest13567-21grep20:55:261
2632499118,2cyclictest12015-21hostname00:00:141
2632499118,2cyclictest10073-21grep19:40:261
2632099119,1cyclictest2487-21cut20:35:140
2632099118,2cyclictest8897-21ssh22:01:080
2632099118,2cyclictest8616-21ssh23:58:020
2632099118,2cyclictest5713-21ssh22:58:040
2632099118,2cyclictest30568-21cut00:30:170
2632099118,2cyclictest27410-21rm23:50:130
2632099118,2cyclictest24648-21sh22:10:270
2632099118,2cyclictest23636-21cut22:30:140
2632099118,2cyclictest23636-21cut22:30:140
2632099118,2cyclictest21616-21cut00:25:150
2632099118,2cyclictest19069-21ssh23:45:140
2632099118,2cyclictest15972-21sed21:00:270
2632099118,2cyclictest12931-21ssh22:42:550
2632099118,2cyclictest12672-21head23:40:270
2632099118,2cyclictest11888-21ssh00:00:130
2632099118,2cyclictest1009-21latency_hist19:25:000
2632099117,3cyclictest29558-21sort19:15:160
2632099117,3cyclictest28319-21sed20:20:170
2632099117,3cyclictest26640-21grep23:30:150
2632099112,8cyclictest28104-21ssh23:10:590
2632099110,10cyclictest21041-21cut20:05:160
2633099108,1cyclictest4331-21chrt20:37:173
2633099108,1cyclictest18612sleep320:30:593
2633099108,1cyclictest185532sleep321:06:253
2633099108,1cyclictest18344-21cat20:00:153
2633099107,2cyclictest4911-21grep20:40:143
2633099107,2cyclictest4911-21grep20:40:143
2633099105,3cyclictest0-21swapper/320:20:133
2633099101,3cyclictest9683-21proc_pri19:40:213
2632899108,1cyclictest6198-21which20:40:222
2632899108,1cyclictest6198-21which20:40:222
2632899108,1cyclictest5821-21rm00:35:132
2632899108,1cyclictest3105-21grep20:35:172
2632899108,1cyclictest26921-21expr19:10:142
2632899108,1cyclictest246442sleep220:12:192
2632899108,1cyclictest2022-21taskset20:33:042
2632899108,1cyclictest19199-21grep20:00:222
2632899107,2cyclictest25482-21expr20:15:162
2632899100,9cyclictest15960-21expr19:55:152
2632499108,1cyclictest14099-21ssh22:05:031
2632499107,2cyclictest7715-21ssh22:00:271
2632499107,2cyclictest3271-21basename20:35:181
2632499107,2cyclictest25148-21cut20:15:161
2632499107,2cyclictest18228-21expr20:00:141
2632499107,2cyclictest12894-21ssh21:45:041
2632499100,9cyclictest20824-21expr20:05:151
2632099108,1cyclictest5523-21grep20:40:170
2632099108,1cyclictest5523-21grep20:40:160
2632099108,1cyclictest3469-21cat19:30:000
2632099108,1cyclictest22954-21grep20:10:150
2632099108,1cyclictest12111-21wc20:55:140
2632099107,2cyclictest9142-21grep19:40:170
2632099107,2cyclictest8968-21chrt20:45:350
2632099107,2cyclictest5311-21tail19:30:250
2632099107,2cyclictest17678-21turbostat20:00:000
263309997,1cyclictest4965-21gzip19:30:213
263309997,1cyclictest23690-21sed20:10:193
263309996,2cyclictest7665-21grep19:35:263
263309996,2cyclictest13896-21grep19:50:173
263249998,1cyclictest507-21expr20:30:141
263249997,1cyclictest10799-21apache_volume19:45:121
263249995,3cyclictest26521-21taskset19:10:101
263209991,6cyclictest1327-21proc_pri20:30:220
263309985,2cyclictest32239-21/usr/sbin/munin20:30:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional