You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-19 - 09:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Jul 19, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
97129379,9sleep00-21swapper/019:06:500
98329167,18sleep20-21swapper/219:07:002
87928966,18sleep10-21swapper/119:05:381
108728666,16sleep30-21swapper/319:08:173
1321993329,3cyclictest30499-21kworker/3:022:03:033
1321992925,3cyclictest3374-21kworker/3:200:30:533
1321992826,2cyclictest3374-21kworker/3:200:26:033
1321992723,3cyclictest30499-21kworker/3:023:07:233
1321992421,3cyclictest30499-21kworker/3:023:27:033
131799242,22cyclictest0-21swapper/200:18:042
1317992419,4cyclictest27910-21gltestperf21:10:182
131799240,20cyclictest0-21swapper/221:18:132
1321992219,2cyclictest30499-21kworker/3:023:49:533
1321992219,2cyclictest30499-21kworker/3:022:21:133
132199220,21cyclictest30499-21kworker/3:023:37:233
131799221,20cyclictest0-21swapper/222:42:392
131599221,12cyclictest0-21swapper/122:50:141
131599220,0cyclictest0-21swapper/121:55:061
1321992118,3cyclictest30499-21kworker/3:022:49:233
1321992118,2cyclictest3374-21kworker/3:200:35:133
131799210,20cyclictest0-21swapper/200:25:332
132199200,19cyclictest0-21swapper/321:45:163
132199199,9cyclictest8458-21apache200:03:383
1321991917,2cyclictest30499-21kworker/3:023:14:333
1317991918,1cyclictest0-21swapper/223:55:232
131799191,17cyclictest56550irq/124-enp0s3120:00:572
131599194,9cyclictest0-21swapper/121:25:151
131599194,9cyclictest0-21swapper/121:25:151
131599190,0cyclictest0-21swapper/123:29:451
131099196,6cyclictest2248-21munin-run20:20:000
1310991917,1cyclictest0-21swapper/000:20:340
1321991817,1cyclictest0-21swapper/320:10:153
1321991816,2cyclictest30499-21kworker/3:023:56:033
1321991816,2cyclictest30499-21kworker/3:023:15:233
1321991816,2cyclictest30499-21kworker/3:023:15:233
1321991816,1cyclictest0-21swapper/323:51:393
1321991815,3cyclictest30499-21kworker/3:021:59:333
131799183,15cyclictest56550irq/124-enp0s3120:49:552
131799180,18cyclictest29481-21kworker/u8:223:22:002
131799180,0cyclictest0-21swapper/222:36:132
131099185,7cyclictest21604-21/usr/sbin/munin22:05:150
1321991714,3cyclictest30499-21kworker/3:021:11:533
1321991714,2cyclictest30499-21kworker/3:000:07:333
132199170,0cyclictest0-21swapper/322:17:293
1317991716,1cyclictest0-21swapper/223:32:232
1317991716,1cyclictest0-21swapper/200:03:032
131799170,17cyclictest0-21swapper/200:10:022
131799170,15cyclictest0-21swapper/223:48:462
131599171,1cyclictest141rcu_preempt23:20:051
131599170,15cyclictest0-21swapper/123:16:451
131599170,15cyclictest0-21swapper/123:16:451
131099179,6cyclictest8458-21apache223:03:320
131099178,8cyclictest516-21gdbus00:40:000
131099173,7cyclictest161rcuc/000:15:000
183012160,0chrt0-21swapper/122:02:451
132199168,7cyclictest487-21avahi-daemon23:00:033
1321991614,1cyclictest26529-21diskmemload21:37:033
1321991613,3cyclictest30499-21kworker/3:022:43:433
1321991612,4cyclictest30499-21kworker/3:021:43:033
132199160,0cyclictest0-21swapper/300:14:293
131799168,8cyclictest3377-21cupsd00:09:112
1315991615,1cyclictest0-21swapper/123:58:431
1315991615,1cyclictest0-21swapper/122:58:301
131099162,2cyclictest0-21swapper/021:50:140
132199159,4cyclictest10138-21mailstats22:55:243
132199153,8cyclictest8438-21apache221:15:163
1321991513,2cyclictest3374-21kworker/3:200:18:133
1321991512,1cyclictest0-21swapper/321:54:573
131799150,1cyclictest0-21swapper/219:10:132
131799150,15cyclictest0-21swapper/200:30:402
131799150,14cyclictest0-21swapper/222:34:512
131599155,5cyclictest225-21systemd-journal23:05:001
131599150,15cyclictest0-21swapper/100:01:141
131599150,13cyclictest0-21swapper/120:05:131
131099155,8cyclictest13-21ksoftirqd/021:19:400
131099154,10cyclictest0-21swapper/021:57:000
1310991512,1cyclictest0-21swapper/000:02:190
1310991510,3cyclictest23276-21tr19:55:150
131099150,10cyclictest23581-21ssh22:45:000
132199147,6cyclictest8458-21apache222:30:133
132199146,2cyclictest25870-21idleruntime-cro23:24:593
1321991412,2cyclictest30499-21kworker/3:023:34:433
1321991412,2cyclictest30499-21kworker/3:021:20:233
1321991410,2cyclictest15925-21grep19:40:133
132199140,13cyclictest31929-21sed22:10:263
131799149,5cyclictest0-21swapper/222:13:172
1317991413,1cyclictest0-21swapper/221:48:342
1317991411,2cyclictest30822-21cat22:30:002
1317991410,3cyclictest21151-21cron22:05:002
1315991410,3cyclictest18622-21ls23:39:591
1315991410,3cyclictest0-21swapper/122:10:261
131599140,1cyclictest0-21swapper/122:45:471
131099149,3cyclictest2455-21cron21:15:000
131099149,3cyclictest10324-21cut20:35:170
131099145,5cyclictest225-21systemd-journal22:54:590
131099143,6cyclictest11131-21apt-get19:30:130
1310991413,1cyclictest13-21ksoftirqd/023:13:390
1310991411,2cyclictest18071-21wc19:45:000
1310991410,3cyclictest8772-21wc20:30:250
1310991410,3cyclictest31745-21wc22:10:250
1310991410,3cyclictest25618-21tr23:05:160
1310991410,2cyclictest6992-21cron20:30:000
131099140,12cyclictest28254-21sed23:25:260
1321991312,1cyclictest30499-21kworker/3:023:43:033
1321991310,2cyclictest8035-21sed20:30:163
1321991310,2cyclictest29264-21tr21:30:213
1321991310,2cyclictest13303-21wc19:35:003
131799138,4cyclictest11669-21ssh22:56:192
131799135,4cyclictest225-21systemd-journal20:25:002
1317991312,1cyclictest0-21swapper/200:35:222
1317991312,0cyclictest0-21swapper/221:30:022
1317991310,3cyclictest0-21swapper/223:42:332
1317991310,2cyclictest23653-21grep21:00:272
1317991310,2cyclictest23569-21grep22:05:262
1317991310,2cyclictest20177-21basename21:45:002
1317991310,2cyclictest11544-21sed23:15:272
1317991310,2cyclictest11544-21sed23:15:272
131799130,13cyclictest0-21swapper/223:27:172
131799130,12cyclictest25816-21cat22:45:272
131799130,12cyclictest12649-21grep23:35:282
131599139,3cyclictest4565-21sed23:50:191
131599137,6cyclictest25-21ksoftirqd/119:10:131
131599135,4cyclictest225-21systemd-journal00:14:241
1315991310,2cyclictest23045-21grep19:55:141
1315991310,2cyclictest17042-21ssh22:40:211
131599130,9cyclictest0-21swapper/121:48:281
131599130,2cyclictest0-21swapper/100:05:001
131599130,1cyclictest0-21swapper/122:23:201
131099135,6cyclictest0-21swapper/023:15:150
131099135,6cyclictest0-21swapper/023:15:140
131099132,8cyclictest455-21dbus-daemon00:20:000
131099132,6cyclictest21179-21basename22:05:000
131099131,7cyclictest10242-21perf23:35:000
1310991310,2cyclictest29924-21grep20:10:000
1310991310,2cyclictest29238-21head21:30:200
1310991310,2cyclictest20955-21wc20:55:220
1310991310,2cyclictest13-21ksoftirqd/022:35:380
1310991310,2cyclictest11750-21latency_hist23:55:010
131099130,12cyclictest12578-21tr20:40:150
132199129,2cyclictest13769-21tr19:35:143
132199120,11cyclictest15188-21ssh22:40:003
131799129,2cyclictest7225-21ntp_kernel_pll_22:15:182
131799129,2cyclictest25564-21grep21:05:212
131799129,2cyclictest19164-21turbostat20:55:002
1317991210,2cyclictest8473-21apache221:38:492
1317991210,1cyclictest19742-21tr20:55:142
1317991210,1cyclictest13061-21wc21:20:262
131799120,11cyclictest0-21swapper/223:10:402
131599129,2cyclictest8487-21munin-plugin-st19:25:001
131599129,2cyclictest6745-21grep22:15:141
131599129,2cyclictest32717-21ssh22:30:221
131599129,2cyclictest29056-21seq22:09:371
131599129,2cyclictest27238-21grep23:05:271
131599129,2cyclictest23791-21grep22:25:171
131599129,2cyclictest14926-21tr00:35:141
131599129,2cyclictest12803-21grep21:40:131
131599128,4cyclictest38150irq/131-snd_hda21:38:351
131599128,3cyclictest56712sleep119:15:281
131599128,3cyclictest5627-21head20:25:171
131599127,4cyclictest4120-21expr20:20:261
131599127,1cyclictest25-21ksoftirqd/122:35:141
1315991211,1cyclictest0-21swapper/120:39:051
1315991211,1cyclictest0-21swapper/119:45:131
131599121,10cyclictest0-21swapper/100:23:201
1315991210,1cyclictest5424-21ssh23:31:191
131599120,12cyclictest0-21swapper/121:12:231
131099129,2cyclictest29506-21cut00:05:170
131099129,2cyclictest21256-21tail23:40:260
131099129,2cyclictest18900-21gpgv23:40:010
131099129,2cyclictest14122-21sed21:40:230
131099128,2cyclictest22706-21sort21:00:170
131099128,2cyclictest13781-21tr19:35:140
131099121,6cyclictest225-21systemd-journal19:50:000
131099120,11cyclictest32270-21ssh22:30:180
163432110,1sleep00-21swapper/022:59:590
132199118,2cyclictest9045-21tr19:25:173
132199118,2cyclictest5273-21grep19:15:233
132199118,2cyclictest32300-21ssh22:50:123
132199118,2cyclictest27063-21sed20:00:273
132199118,2cyclictest2634-21grep19:10:193
132199118,2cyclictest25229-21ssh22:25:383
132199118,2cyclictest22746-21ssh00:20:193
132199118,2cyclictest22308-21grep19:50:273
132199118,2cyclictest21572-21ssh22:05:083
132199118,2cyclictest19676-21bc21:25:133
132199118,2cyclictest19676-21bc21:25:133
132199118,2cyclictest18263-21apache_accesses19:45:133
131799119,1cyclictest13568-21wc20:40:232
131799118,2cyclictest6261-21cat19:20:002
131799118,2cyclictest5117-21grep21:55:142
131799118,2cyclictest2966-21ssh21:53:522
131799118,2cyclictest23183-21wc00:20:232
131799118,2cyclictest22607-21rm23:03:182
131799118,2cyclictest1166-21cat22:50:212
131599118,2cyclictest7943-21sort20:30:161
131599118,2cyclictest3783-21grep21:15:201
131599118,2cyclictest31930-21cat00:25:261
131599118,2cyclictest19901-21df23:40:161
131599118,2cyclictest17753-21taskset19:41:581
131599118,2cyclictest1668-21rm23:48:501
131599118,2cyclictest15629-21basename19:40:001
131599118,2cyclictest14156-21grep00:15:171
131599118,2cyclictest10863-21dump-pmu-power19:30:001
131599117,3cyclictest7325-21grep00:30:191
131599117,3cyclictest21750-21systemctl21:00:001
131599114,2cyclictest225-21systemd-journal20:44:591
131599111,9cyclictest13187-21timerwakeupswit21:20:261
1315991110,1cyclictest0-21swapper/121:30:561
131599110,10cyclictest495-21expr20:15:131
131599110,10cyclictest1308-21grep23:10:131
131099119,1cyclictest2145-21cut19:10:160
131099118,2cyclictest9686-21ssh22:16:420
131099118,2cyclictest6662-21sed00:30:150
131099118,2cyclictest4437-21ssh00:29:060
131099118,2cyclictest32021-21ssh23:47:140
131099118,2cyclictest30197-21grep20:10:160
131099118,2cyclictest28178-21ssh22:27:570
131099118,2cyclictest20841-21sort21:25:210
131099118,2cyclictest20841-21sort21:25:200
131099118,2cyclictest15747-21uname22:20:210
131099118,2cyclictest14051-21head23:55:270
131099118,2cyclictest10949-21sh21:39:220
131099117,3cyclictest6460-21grep19:20:140
131099117,3cyclictest10270-21wc19:25:290
131099113,6cyclictest0-21swapper/022:47:330
131099111,9cyclictest22753-21munin-run19:55:000
132199109,1cyclictest39-21ksoftirqd/320:27:013
132199108,1cyclictest32626-21cut20:15:133
132199108,1cyclictest29110-21grep20:05:223
132199108,1cyclictest19825-21egrep20:55:143
132199108,1cyclictest15137-21ls20:45:143
132199108,1cyclictest122692sleep320:40:093
132199107,2cyclictest4159-21grep20:20:263
132199107,2cyclictest23341-21gzip21:00:233
132199107,2cyclictest23246-21tr19:55:153
132199107,2cyclictest17237-21grep20:50:143
132199106,2cyclictest25574-21sh21:05:223
132199104,5cyclictest9668-21/usr/sbin/munin20:35:123
132199101,2cyclictest0-21swapper/319:20:123
131799108,2cyclictest20361-21taskset19:50:002
131799108,2cyclictest14396-21ssh22:20:032
131799108,1cyclictest3433-21rm23:50:022
131799108,1cyclictest28080-21cstates20:05:152
131799108,1cyclictest23102-21aten2_r7power_p19:55:142
131799108,1cyclictest17010-21ntp_kernel_err19:40:192
131799108,1cyclictest153362sleep219:36:412
131799108,1cyclictest11431-21cat19:30:152
131799107,2cyclictest7079-21tail19:20:172
131799107,2cyclictest25132-21ssh23:05:062
131799107,2cyclictest21833-21ssh21:25:342
131799107,2cyclictest21833-21ssh21:25:342
131799107,2cyclictest20715-21aten2_r7power_p19:50:152
131799102,7cyclictest7271-21apt-get20:30:132
131799100,9cyclictest0-21swapper/220:10:172
131599108,1cyclictest30135-21cat20:10:131
131599108,1cyclictest26008-21sed21:05:271
131599108,1cyclictest17260-21cat20:50:151
131599107,2cyclictest28928-21ssh21:50:111
131599107,2cyclictest23074-21cat21:00:201
131599107,2cyclictest11266-21tr19:30:141
131599101,5cyclictest225-21systemd-journal20:05:011
131099107,2cyclictest4419-21grep19:15:160
131099107,2cyclictest3215-21cut20:20:170
131099107,2cyclictest26080-21tail21:05:270
131099107,2cyclictest23676-21ssh21:46:140
131099107,2cyclictest18135-21grep23:20:160
131099107,2cyclictest15913-21tr20:45:200
131099107,2cyclictest13-21ksoftirqd/020:00:130
131099106,3cyclictest17333-21tr20:50:150
131099101,6cyclictest12067-21fschecks_count21:20:160
13179998,1cyclictest5100-21cat20:25:152
13179997,2cyclictest9200-21ls19:25:172
13179997,2cyclictest11251-21cat20:35:272
13179997,1cyclictest1230-21ntpq20:15:182
13159996,2cyclictest15056-21cat20:45:141
13159990,2cyclictest0-21swapper/119:50:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional