You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 06:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Aug 10, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26731210848,39sleep00-21swapper/019:07:090
2666628774,9sleep20-21swapper/219:06:192
2669428168,9sleep30-21swapper/319:06:413
2677927759,14sleep10-21swapper/119:07:471
125872640,0sleep10-21swapper/123:59:131
256812620,0sleep00-21swapper/000:06:260
27040992927,2cyclictest32-21ksoftirqd/223:05:132
2704099280,28cyclictest32-21ksoftirqd/200:14:572
2704499241,2cyclictest30536-21munin-run00:10:013
2704099243,21cyclictest32-21ksoftirqd/223:39:352
27034992424,0cyclictest0-21swapper/022:26:470
2704499233,1cyclictest0-21swapper/300:17:323
27040992322,1cyclictest0-21swapper/222:10:242
27040992321,2cyclictest0-21swapper/221:41:522
27040992314,9cyclictest0-21swapper/223:16:482
27034992321,2cyclictest13-21ksoftirqd/000:12:290
27040992221,1cyclictest0-21swapper/222:18:442
27040992215,4cyclictest32-21ksoftirqd/221:25:172
27040992213,6cyclictest32-21ksoftirqd/200:00:152
27036992220,2cyclictest94-21kswapd021:38:051
2703699220,1cyclictest0-21swapper/119:15:001
2704499212,5cyclictest0-21swapper/300:10:093
27040992116,4cyclictest0-21swapper/222:08:532
2704099210,21cyclictest0-21swapper/221:16:182
2704099210,20cyclictest0-21swapper/222:55:012
2704099205,12cyclictest0-21swapper/222:45:222
2704099204,16cyclictest0-21swapper/222:27:442
27040992017,2cyclictest20052-21diskmemload23:50:272
27040992012,7cyclictest32-21ksoftirqd/221:55:122
27040992012,7cyclictest32-21ksoftirqd/221:55:122
2703699203,16cyclictest20052-21diskmemload21:48:261
2704099197,7cyclictest20052-21diskmemload00:32:082
2704099197,11cyclictest0-21swapper/221:35:112
2704099194,15cyclictest0-21swapper/221:30:172
27040991917,1cyclictest20052-21diskmemload21:13:412
27036991918,1cyclictest0-21swapper/121:51:081
2703699191,3cyclictest31529-21df20:25:151
2703699190,18cyclictest0-21swapper/122:36:021
2704499184,12cyclictest0-21swapper/321:35:113
27044991816,1cyclictest17433-21kworker/3:021:15:233
2704099189,9cyclictest0-21swapper/221:51:082
2704099189,7cyclictest20052-21diskmemload22:57:512
27040991816,0cyclictest20052-21diskmemload00:15:092
27040991810,6cyclictest311rcuc/222:35:132
2703699186,8cyclictest1142-21runrttasks23:27:461
2703499188,6cyclictest13071-21apache222:20:170
2703499184,7cyclictest741-21munin-run20:30:000
27044991713,2cyclictest0-21swapper/321:56:523
27044991713,2cyclictest0-21swapper/321:56:523
2704099178,9cyclictest0-21swapper/200:35:152
2704099173,14cyclictest0-21swapper/219:50:142
27040991712,2cyclictest225-21systemd-journal00:09:552
27040991711,2cyclictest225-21systemd-journal22:00:332
27040991711,2cyclictest225-21systemd-journal22:00:332
2703699172,0cyclictest0-21swapper/100:19:041
27036991717,0cyclictest0-21swapper/121:18:271
2703699171,13cyclictest0-21swapper/122:10:121
27036991710,2cyclictest141rcu_preempt00:08:561
2703499177,3cyclictest225-21systemd-journal23:20:000
27034991716,1cyclictest0-21swapper/000:21:480
27044991614,1cyclictest17364-21ssh23:42:103
27044991614,1cyclictest17364-21ssh23:42:103
2704499160,16cyclictest0-21swapper/300:28:483
2704499160,14cyclictest0-21swapper/323:55:103
2704099168,8cyclictest0-21swapper/220:40:302
2704099162,8cyclictest0-21swapper/200:20:122
27040991615,1cyclictest311rcuc/223:41:582
27040991615,1cyclictest311rcuc/223:41:582
27040991614,1cyclictest918-21sh23:13:302
2704099160,16cyclictest0-21swapper/222:32:462
2704099160,16cyclictest0-21swapper/221:47:302
2704099160,16cyclictest0-21swapper/219:55:152
2703699160,2cyclictest0-21swapper/100:28:501
2704499150,15cyclictest0-21swapper/323:26:553
2704499150,14cyclictest0-21swapper/322:08:303
2704499150,14cyclictest0-21swapper/321:45:493
2704499150,10cyclictest26441-21ssh22:50:133
2704099158,7cyclictest20052-21diskmemload22:41:002
2704099158,7cyclictest0-21swapper/220:05:232
2704099152,1cyclictest0-21swapper/223:55:362
27040991512,2cyclictest20052-21diskmemload00:29:552
2704099150,1cyclictest0-21swapper/223:23:442
27036991513,1cyclictest20052-21diskmemload22:48:011
2703699150,15cyclictest0-21swapper/123:20:441
2703699150,15cyclictest0-21swapper/122:59:111
2703699150,15cyclictest0-21swapper/100:36:281
27034991512,3cyclictest0-21swapper/022:41:470
2703499150,2cyclictest0-21swapper/023:06:050
2704499146,7cyclictest13955-21apt-get19:50:143
27044991412,2cyclictest39-21ksoftirqd/300:22:273
2704499141,12cyclictest8945-21cat19:40:003
2704099148,6cyclictest32-21ksoftirqd/222:20:112
27036991412,1cyclictest0-21swapper/121:44:251
2703699140,13cyclictest14481-21tr00:00:171
2703499142,7cyclictest30824-21perf21:55:000
27034991411,2cyclictest9035-21grep20:45:170
27034991411,2cyclictest21315-21awk21:30:000
2704499139,3cyclictest39-21ksoftirqd/323:15:023
2704499139,3cyclictest10765-21grep23:00:133
2704499139,2cyclictest17207-21cut19:55:183
2704499134,2cyclictest225-21systemd-journal19:20:013
27044991311,1cyclictest0-21swapper/322:12:263
27044991310,3cyclictest0-21swapper/300:35:543
27044991310,2cyclictest28408-21perf20:20:013
27044991310,2cyclictest27247-21grep22:30:243
2704099139,3cyclictest23147-21ssh23:45:252
27040991311,2cyclictest20052-21diskmemload23:32:072
27040991310,2cyclictest22288-21grep23:25:262
27040991310,2cyclictest18008-21cut21:05:132
2704099130,13cyclictest0-21swapper/221:21:372
2703699138,3cyclictest28971-21ssh23:10:221
2703699138,3cyclictest16610-21tr19:55:161
27036991312,1cyclictest0-21swapper/123:00:221
2703699131,1cyclictest20052-21diskmemload00:30:221
27036991310,2cyclictest9721-21wc22:20:191
27036991310,2cyclictest6260-21ssh21:20:211
27036991310,2cyclictest4187-21awk23:35:001
27036991310,2cyclictest30648-21grep23:50:171
27036991310,2cyclictest25573-21grep20:10:301
27036991310,2cyclictest13315-21cut23:40:141
27036991310,2cyclictest13315-21cut23:40:141
2703699130,1cyclictest0-21swapper/122:07:531
2703699130,12cyclictest31304-21grep19:15:281
2703699130,12cyclictest18614-21grep21:05:161
2703699130,11cyclictest0-21swapper/122:32:391
2703499139,3cyclictest8059-21cron22:20:000
2703499136,6cyclictest11896-21/usr/sbin/munin23:20:170
2703499131,7cyclictest29594-21munin-run21:35:000
2703499131,6cyclictest28219-21munin-run23:30:000
27034991311,2cyclictest13-21ksoftirqd/023:45:270
27034991310,2cyclictest8292-21ls00:15:220
27034991310,2cyclictest7127-21grep21:40:180
27034991310,2cyclictest32333-21grep19:20:160
27034991310,2cyclictest32099-21head21:55:170
27034991310,2cyclictest32099-21head21:55:160
27034991310,2cyclictest31480-21awk00:30:000
27034991310,2cyclictest29367-21cron19:15:000
27034991310,2cyclictest12124-21grep23:00:210
27034991310,2cyclictest1015-21wc00:30:220
2704499129,2cyclictest8385-21sed22:00:193
2704499129,2cyclictest8385-21sed22:00:193
2704499129,2cyclictest8088-21perf22:20:003
2704499129,2cyclictest5017-21rm23:35:143
2704499129,2cyclictest32748-21tr19:20:173
2704499129,2cyclictest28870-21basename23:30:153
2704499129,2cyclictest2826-21expr22:55:163
2704499129,2cyclictest27917-21sort19:10:163
2704499129,2cyclictest19739-21sh22:45:253
2704499129,2cyclictest16210-21grep21:00:173
2704499129,2cyclictest14648-21taskset22:23:333
2704499128,3cyclictest5285-21tr21:20:143
2704499128,3cyclictest24610-21grep20:10:213
2704499128,2cyclictest1266-21tr20:30:143
2704499121,7cyclictest0-21swapper/300:30:233
2704499121,11cyclictest0-21swapper/323:06:353
2704499120,1cyclictest141rcu_preempt23:50:583
2704499120,11cyclictest31648-21grep20:25:163
2704099129,2cyclictest28745-21grep20:20:142
2704099129,2cyclictest27374-21cat20:15:202
2704099129,2cyclictest13311-21cut20:55:172
2704099123,8cyclictest9838-21fschecks_time19:40:192
2703699129,2cyclictest540-21grep00:10:271
2703699129,2cyclictest29659-21wc21:35:001
2703699129,2cyclictest17822-21wc21:05:001
2703699129,2cyclictest14680-21grep21:25:191
2703699129,2cyclictest12248-21grep19:45:171
2703699129,2cyclictest11506-21wc22:40:251
2703699128,3cyclictest718-21grep19:20:201
2703699128,3cyclictest5150-21sed19:30:181
2703699128,3cyclictest14721-21ssh22:04:291
2703699128,3cyclictest14721-21ssh22:04:291
2703699128,2cyclictest131982sleep120:55:181
2703699128,2cyclictest10792-21grep19:40:271
2703699126,2cyclictest225-21systemd-journal20:45:001
2703699125,6cyclictest38150irq/131-snd_hda23:37:391
2703499129,2cyclictest8317-21head22:00:190
2703499129,2cyclictest8317-21head22:00:190
2703499129,2cyclictest7690-21grep23:55:270
2703499129,2cyclictest5261-21cron23:55:000
2703499129,2cyclictest4961-21grep23:35:140
2703499129,2cyclictest3641-21tr20:35:150
2703499129,2cyclictest27817-21grep22:50:210
2703499129,2cyclictest23936-21grep20:10:180
2703499129,2cyclictest20810-21grep20:00:350
2703499129,2cyclictest1596-21grep22:35:140
2703499129,2cyclictest15174-21grep19:50:230
2703499121,3cyclictest0-21swapper/022:05:140
2703499120,9cyclictest0-21swapper/000:00:150
2704499119,1cyclictest18693-21grep21:05:163
2704499118,2cyclictest9108-21grep20:45:173
2704499118,2cyclictest6473-21cron19:35:003
2704499118,2cyclictest6292-21cat20:40:173
2704499118,2cyclictest6072-21ssh21:40:073
2704499118,2cyclictest4108-21latency_hist19:30:003
2704499118,2cyclictest29059-21cat20:20:153
2704499118,2cyclictest23327-21ssh21:11:023
2704499118,2cyclictest23265-21sort23:45:253
2704499118,2cyclictest22710-21ssh21:50:023
2704499118,2cyclictest21625-21ssh21:30:063
2704499118,2cyclictest17280-21grep22:25:153
2704499118,2cyclictest1591-21tr22:35:143
2704499118,2cyclictest15488-21ntpq00:00:223
2704499118,2cyclictest14733-21ntpq21:25:203
2704499118,2cyclictest13924-21head20:55:203
2704499118,2cyclictest11884-21ssh22:40:343
2704499118,2cyclictest11265-21munin-run19:45:003
2704499114,2cyclictest0-21swapper/323:10:163
2704499110,10cyclictest12486-21sh23:20:153
2704099119,2cyclictest32-21ksoftirqd/220:29:462
2704099118,2cyclictest8587-21chrt19:36:402
2704099118,2cyclictest4617-21tr19:30:142
2704099118,2cyclictest4605-21grep20:35:222
2704099118,2cyclictest3251-21cat20:35:002
2704099118,2cyclictest32304-21tr19:20:152
2704099118,2cyclictest25996-21cron20:15:002
2704099115,5cyclictest487-21avahi-daemon23:01:422
2704099111,7cyclictest31339-21unixbench_singl19:15:282
2703699119,1cyclictest6579-21ssh23:16:141
2703699119,1cyclictest17721-21sort22:25:171
2703699118,2cyclictest9685-21grep20:45:221
2703699118,2cyclictest3752-21ls20:35:151
2703699118,2cyclictest32092-21tail21:55:161
2703699118,2cyclictest32092-21tail21:55:161
2703699118,2cyclictest2798-21ssh22:15:531
2703699118,2cyclictest26651-21tr22:50:141
2703699118,2cyclictest26339-21wget20:15:131
2703699118,2cyclictest24202-21ssh21:11:421
2703699118,2cyclictest22318-21sed23:45:161
2703699118,2cyclictest21250-21cron20:05:001
2703699118,2cyclictest19255-21sh00:22:021
2703699118,2cyclictest19242-21grep23:05:141
2703699117,3cyclictest23064-21wc20:05:261
2703499118,2cyclictest8091-21sh20:45:000
2703499118,2cyclictest707-21sh21:36:150
2703499118,2cyclictest6181-21sh22:56:510
2703499118,2cyclictest6066-21sed19:30:260
2703499118,2cyclictest5869-21tail21:20:170
2703499118,2cyclictest3655-21grep19:25:260
2703499118,2cyclictest29720-21tr19:15:150
2703499118,2cyclictest24568-21tr22:10:140
2703499118,2cyclictest24295-21ssh21:11:480
2703499118,2cyclictest22280-21head20:05:180
2703499118,2cyclictest19380-21tr22:45:200
2703499118,2cyclictest17959-21grep21:05:130
2703499118,2cyclictest17334-21taskset21:00:500
2703499118,2cyclictest15227-21chrt20:59:320
2703499118,2cyclictest14585-21sh21:45:120
2703499118,2cyclictest13369-21grep23:40:140
2703499118,2cyclictest13369-21grep23:40:140
2703499118,2cyclictest12755-21chrt20:53:590
2703499118,2cyclictest10015-21ssh00:35:300
2703499116,4cyclictest13-21ksoftirqd/019:55:030
2703499113,4cyclictest225-21systemd-journal19:50:000
2703499110,10cyclictest25295-21sh22:30:090
2703499110,10cyclictest1630-21uniq20:30:160
2704499108,1cyclictest21480-21taskset20:05:033
2704499108,1cyclictest19629-21cat20:00:233
2704499108,1cyclictest12446-21cut19:45:193
2704499108,1cyclictest11570-21cat20:50:163
2704499107,2cyclictest3622-21tr20:35:153
2704099108,1cyclictest28983-21grep19:10:262
2704099107,2cyclictest8559-21tr20:45:152
2704099107,2cyclictest19048-21find20:00:012
2704099107,2cyclictest16650-21grep21:00:202
2704099107,2cyclictest12425-21head20:50:252
2704099103,2cyclictest13675-21cron19:50:002
2704099100,9cyclictest2082-21aten2_r7power_p19:25:132
2703699108,1cyclictest3879-21taskset19:27:111
2703699108,1cyclictest300002sleep120:20:231
2703699108,1cyclictest2551-21sort20:30:261
2703699108,1cyclictest157852sleep119:51:381
2703699107,2cyclictest7061-21cat19:35:181
2703699107,2cyclictest12345-21grep20:50:241
2703499108,1cyclictest8864-21/usr/sbin/munin19:40:000
2703499108,1cyclictest29883-21cut20:20:220
2703499108,1cyclictest29199-21expr23:30:160
2703499108,1cyclictest26226-21chrt20:15:010
2703499107,2cyclictest9902-21head19:40:180
2703499107,2cyclictest28664-21ssh21:15:020
2703499107,2cyclictest28121-21cut23:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional