You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 11:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Mar 13, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2934028961,9sleep30-21swapper/319:06:253
2951428774,9sleep00-21swapper/019:08:390
2943328269,9sleep20-21swapper/219:07:372
2942227659,13sleep10-21swapper/119:07:281
144322570,0sleep30-21swapper/322:02:593
29708992523,1cyclictest14925-21kworker/3:000:13:013
2970499250,1cyclictest0-21swapper/222:45:152
29708992421,3cyclictest0-21swapper/323:58:183
2970499240,22cyclictest0-21swapper/200:33:002
29704992323,0cyclictest0-21swapper/223:41:572
29708992120,1cyclictest0-21swapper/322:06:483
29708992119,2cyclictest13351-21kworker/3:222:19:123
2970499210,21cyclictest0-21swapper/222:41:312
2970499210,21cyclictest0-21swapper/221:30:362
2969499210,2cyclictest0-21swapper/023:29:270
29708992018,2cyclictest13351-21kworker/3:221:34:423
29700992020,0cyclictest0-21swapper/123:53:521
2969499200,2cyclictest0-21swapper/023:34:160
29704991919,0cyclictest0-21swapper/222:03:422
2970099191,16cyclictest487-21avahi-daemon22:25:241
2970099190,1cyclictest0-21swapper/123:30:521
29694991917,1cyclictest0-21swapper/019:45:120
29708991816,2cyclictest13351-21kworker/3:222:28:323
29704991814,3cyclictest32-21ksoftirqd/221:35:112
2970099187,6cyclictest537-21snmpd22:02:561
29700991817,1cyclictest0-21swapper/123:12:521
29700991817,1cyclictest0-21swapper/100:34:031
2970099180,1cyclictest0-21swapper/123:23:491
2970099180,1cyclictest0-21swapper/120:30:141
2970099180,18cyclictest0-21swapper/121:57:301
2969499184,7cyclictest612-21perf20:25:000
29708991715,2cyclictest14925-21kworker/3:023:45:123
29708991715,2cyclictest14925-21kworker/3:000:23:223
29708991715,2cyclictest13351-21kworker/3:222:54:213
29708991715,2cyclictest13351-21kworker/3:222:22:413
2970899170,16cyclictest0-21swapper/321:25:143
2970499176,8cyclictest30646-21turbostat.cron20:20:002
2970499175,12cyclictest32-21ksoftirqd/223:19:122
2970499172,0cyclictest141rcu_preempt00:11:162
2970499170,15cyclictest0-21swapper/222:05:332
2970099171,1cyclictest22065-21diskmemload22:36:291
2970099170,2cyclictest0-21swapper/100:26:211
2970099170,13cyclictest0-21swapper/100:18:511
2969499175,8cyclictest17010-21/usr/sbin/munin00:20:180
29694991715,2cyclictest0-21swapper/021:27:040
29708991613,2cyclictest13351-21kworker/3:222:55:523
29700991616,0cyclictest0-21swapper/122:49:301
2970099161,14cyclictest0-21swapper/123:56:561
2970099160,3cyclictest0-21swapper/123:48:571
2970099160,15cyclictest0-21swapper/123:19:451
2970099160,15cyclictest0-21swapper/100:00:431
2970899156,1cyclictest22065-21diskmemload00:25:393
2970899155,3cyclictest1897-21sshd21:16:213
29708991514,1cyclictest13351-21kworker/3:222:12:413
29708991511,4cyclictest14925-21kworker/3:023:43:423
2970899150,14cyclictest0-21swapper/323:38:583
29704991513,2cyclictest29689-21cyclictest21:20:132
2970499150,1cyclictest0-21swapper/200:15:522
2970499150,13cyclictest0-21swapper/221:56:312
2970499150,0cyclictest141rcu_preempt23:51:562
2970099157,6cyclictest38150irq/131-snd_hda21:30:441
2970099152,12cyclictest28480-21ssh23:09:591
2969499158,7cyclictest29689-21cyclictest22:20:210
2969499151,2cyclictest0-21swapper/023:55:470
29694991512,1cyclictest7739-21perf20:39:590
29694991510,4cyclictest0-21swapper/022:30:140
2969499150,6cyclictest0-21swapper/023:44:150
2969499150,1cyclictest0-21swapper/022:52:230
2970899148,2cyclictest455-21dbus-daemon21:00:003
29708991413,1cyclictest0-21swapper/321:41:523
29708991412,2cyclictest13351-21kworker/3:221:22:423
2970899141,0cyclictest0-21swapper/323:02:263
29708991410,2cyclictest18748-21cat19:55:003
29708991410,2cyclictest16569-21grep00:00:153
2970499149,4cyclictest26263-21grep00:25:202
29704991413,1cyclictest55650irq/127-enp2s0-00:08:112
29704991413,1cyclictest0-21swapper/223:25:142
29704991411,2cyclictest1411-21df20:25:162
29704991410,3cyclictest16857-21df21:45:152
2970499140,12cyclictest4736-21cut19:25:142
29700991414,0cyclictest0-21swapper/121:07:171
29700991412,1cyclictest22065-21diskmemload21:11:491
29700991412,1cyclictest22065-21diskmemload21:11:491
2970099141,11cyclictest17049-21sh00:20:111
2970099140,13cyclictest0-21swapper/123:04:461
29694991413,1cyclictest0-21swapper/022:29:070
29694991410,4cyclictest13-21ksoftirqd/000:17:500
29694991410,3cyclictest32175-21tail23:50:160
2970899139,3cyclictest4911-21grep19:25:153
2970899139,3cyclictest32185-21grep23:30:263
2970899139,3cyclictest23913-21grep23:25:273
2970899136,2cyclictest32708-21cron21:55:003
2970899131,2cyclictest141rcu_preempt00:38:123
29708991310,2cyclictest4506-21cat19:25:003
29708991310,2cyclictest21434-21apt23:25:003
29708991310,2cyclictest20587-21grep22:45:183
29708991310,2cyclictest19329-21ssh22:45:003
29708991310,2cyclictest17636-21ntpq21:45:203
2970899130,13cyclictest0-21swapper/322:35:413
2970499139,3cyclictest3984-21grep19:20:232
2970499139,3cyclictest30972-21cut23:30:152
2970499134,8cyclictest14016-21wc20:50:232
2970499131,7cyclictest28958-21if_err_enp2s022:50:172
29704991310,2cyclictest5728-21ssh22:55:262
29704991310,2cyclictest4216-21sed22:35:212
29704991310,2cyclictest3520-21grep20:30:152
29704991310,2cyclictest11567-21sed22:20:202
2970499130,1cyclictest0-21swapper/223:55:472
2970099139,2cyclictest651-21grep19:15:171
2970099135,4cyclictest225-21systemd-journal19:50:001
2970099132,1cyclictest0-21swapper/121:20:161
29700991310,2cyclictest9808-21cat22:20:001
29700991310,2cyclictest3547-21wc19:20:191
29700991310,2cyclictest15966-21grep20:55:171
29700991310,2cyclictest15944-21tr23:40:201
29700991310,2cyclictest15902-21cat21:45:011
2970099130,13cyclictest0-21swapper/122:42:381
2970099130,12cyclictest0-21swapper/122:56:321
2970099130,10cyclictest0-21swapper/122:05:041
2969499138,3cyclictest14835-21ssh21:25:010
2969499136,3cyclictest225-21systemd-journal23:50:000
2969499135,7cyclictest25422-21/usr/sbin/munin00:25:230
29694991310,2cyclictest8291-21tr20:40:130
29694991310,2cyclictest4253-21grep22:35:210
29694991310,2cyclictest32196-21cat19:15:000
29694991310,2cyclictest3017-21ssh21:36:320
29694991310,2cyclictest24173-21munin-plugin-st21:50:000
2969499130,1cyclictest0-21swapper/022:13:560
2969499130,1cyclictest0-21swapper/021:34:590
2970899129,2cyclictest9989-21grep00:15:203
2970899129,2cyclictest5041-21tr23:15:133
2970899129,2cyclictest4527-21wc23:15:013
2970899129,2cyclictest3953-21head20:30:173
2970899129,2cyclictest25744-21systemd-cgroups00:05:233
2970899129,2cyclictest21477-21tail23:05:173
2970899129,2cyclictest16361-21munin-plugin-st19:50:003
2970899129,2cyclictest1514-21grep21:35:253
2970899129,2cyclictest11108-21grep20:45:173
2970899128,3cyclictest18291-21grep21:00:173
2970899125,4cyclictest23761-21apt-get20:05:133
2970899123,6cyclictest22059-21cron21:10:003
2970899122,5cyclictest0-21swapper/322:34:173
2970499129,2cyclictest6679-21wc21:20:012
2970499129,2cyclictest32-21ksoftirqd/223:35:262
2970499129,2cyclictest30691-21tail19:10:152
2970499129,2cyclictest23237-21sleep21:30:002
2970499129,2cyclictest21657-21grep21:05:262
2970499129,2cyclictest14207-21sed23:00:272
2970499129,2cyclictest11313-21uname20:45:202
2970499123,4cyclictest225-21systemd-journal23:25:002
2970499122,1cyclictest225-21systemd-journal19:51:202
2970499120,9cyclictest0-21swapper/200:01:192
2970499120,6cyclictest0-21swapper/221:53:402
2970099129,2cyclictest7423-21cron21:40:001
2970099129,2cyclictest38150irq/131-snd_hda21:15:251
2970099129,2cyclictest24437-21tr00:05:151
2970099129,2cyclictest19269-21grep19:55:161
2970099129,2cyclictest1388-21sort00:10:181
2970099129,2cyclictest11148-21cut20:45:161
2970099129,2cyclictest11028-21sed19:35:241
2970099122,2cyclictest225-21systemd-journal21:04:561
2970099121,11cyclictest0-21swapper/122:11:461
29700991210,1cyclictest22065-21diskmemload23:26:471
2970099120,2cyclictest0-21swapper/120:25:191
2970099120,11cyclictest24031-21grep20:05:151
2969499129,2cyclictest9101-21turbostat00:35:000
2969499129,2cyclictest4900-21tr19:25:140
2969499129,2cyclictest2808-21sed21:55:280
2969499129,2cyclictest26905-21grep20:10:140
2969499129,2cyclictest25409-21munin-run22:09:590
2969499129,2cyclictest24564-21grep00:05:160
2969499129,2cyclictest22510-21tr20:00:190
2969499129,2cyclictest20697-21grep19:55:270
2969499129,2cyclictest19531-21ssh22:45:020
2969499129,2cyclictest16833-21sed20:55:260
2969499129,2cyclictest11027-21sort20:45:160
2969499124,6cyclictest22065-21diskmemload21:43:500
2969499120,6cyclictest22065-21diskmemload22:05:000
2969499120,2cyclictest0-21swapper/022:56:460
2970899118,2cyclictest7961-21taskset20:40:013
2970899118,2cyclictest7752-21head19:30:163
2970899118,2cyclictest6873-21wc20:35:253
2970899118,2cyclictest681-21sort23:50:253
2970899118,2cyclictest32558-21expr19:15:153
2970899118,2cyclictest31731-21taskset19:11:083
2970899118,2cyclictest28215-21ssh21:13:073
2970899118,2cyclictest28215-21ssh21:13:063
2970899118,2cyclictest2638-21grep00:30:213
2970899118,2cyclictest2230-21run-parts20:25:243
2970899118,2cyclictest14099-21cat19:45:013
2970899118,2cyclictest1058-21egrep21:55:163
2970899110,1cyclictest0-21swapper/320:50:143
2970899110,10cyclictest28895-21cat20:15:153
2970899110,10cyclictest28079-21chrt20:13:033
2970899110,10cyclictest19843-21if_enp0s31f619:55:183
2970499118,3cyclictest0-21swapper/222:17:092
2970499118,2cyclictest9898-21cut00:35:152
2970499118,2cyclictest8678-21tail19:30:272
2970499118,2cyclictest30069-21wc23:10:182
2970499118,2cyclictest26687-21grep22:10:152
2970499118,2cyclictest24430-21wc23:45:212
2970499118,2cyclictest23266-21sshd20:02:152
2970499118,2cyclictest21488-21ssh23:05:162
2970499118,2cyclictest20807-21ssh22:25:372
2970499118,2cyclictest20219-21grep19:55:222
2970499118,2cyclictest16859-21ssh00:20:042
2970499118,2cyclictest16118-21uname20:55:182
2970499118,2cyclictest14248-21tr19:45:122
2970499118,2cyclictest13426-21sed19:40:272
2970499118,2cyclictest13425-21ssh21:43:112
2970499118,2cyclictest11550-21tr19:40:002
2970499117,3cyclictest24205-21tr21:10:232
2970499117,3cyclictest24205-21tr21:10:222
29704991110,1cyclictest0-21swapper/222:34:052
2970099119,1cyclictest13530-21head20:50:171
2970099119,1cyclictest0-21swapper/123:39:521
2970099118,2cyclictest8136-21cut20:40:131
2970099118,2cyclictest6367-21sed20:35:191
2970099118,2cyclictest32176-21wc20:20:231
2970099118,2cyclictest30657-21sendmail20:20:001
2970099118,2cyclictest28649-21wc22:30:271
2970099118,2cyclictest26396-21tr20:10:121
2970099118,2cyclictest16376-21aten2_r7power_e21:45:131
2970099118,2cyclictest15863-21cat21:25:151
2970099118,2cyclictest15720-21ssh00:38:401
2970099118,2cyclictest10112-21ssh22:20:101
2970099117,3cyclictest31705-21chrt19:10:471
2970099117,1cyclictest27529-21ssh21:50:551
2970099115,5cyclictest22065-21diskmemload22:54:331
2969499118,2cyclictest9944-21grep00:35:160
2969499118,2cyclictest7557-21ssh23:35:210
2969499118,2cyclictest7182-21egrep19:30:150
2969499118,2cyclictest32635-21grep21:15:210
2969499118,2cyclictest3080-21ssh22:15:180
2969499118,2cyclictest29287-21grep20:15:170
2969499118,2cyclictest28052-21ssh21:51:210
2969499118,2cyclictest23750-21apache_accesses20:05:120
2969499118,2cyclictest23735-21ssh23:06:160
2969499118,2cyclictest23409-21df21:10:170
2969499118,2cyclictest23409-21df21:10:160
2969499118,2cyclictest18477-21chrt19:52:020
2969499118,2cyclictest16189-21ssh23:20:540
2969499118,2cyclictest16022-21aten2_r7power_p00:00:140
2969499118,2cyclictest13280-21if_err_enp2s023:00:190
2969499118,2cyclictest13193-21rm22:40:260
2969499118,2cyclictest1217-21expr19:15:230
2969499118,2cyclictest10327-21ssh23:18:050
2969499110,10cyclictest21738-21taskset21:05:590
2970899108,1cyclictest21657-21expr20:00:143
2970899108,1cyclictest10525-21sshd19:35:193
2970899107,2cyclictest31173-21cut20:20:153
2970499108,1cyclictest8388-21expr20:40:132
2970499108,1cyclictest32579-21grep20:20:292
2970499108,1cyclictest279582sleep220:11:302
2970499107,2cyclictest6284-21cut20:35:192
2970499107,2cyclictest32322-21chrt19:15:142
2970499107,2cyclictest23755-21apache_processe20:05:132
2970499107,2cyclictest19195-21grep21:00:282
2970099108,1cyclictest23268-21chrt20:02:161
2970099108,1cyclictest17397-21expr19:50:161
2970099100,5cyclictest5415-21gltestperf19:25:171
2969499108,1cyclictest3741-21sed20:30:170
2969499108,1cyclictest28928-21ssh23:10:110
2969499108,1cyclictest2079-21ssh00:10:250
2969499108,1cyclictest11447-21chrt19:39:150
2969499107,2cyclictest18674-21cat21:00:220
2969499107,2cyclictest1466-21sort20:25:160
2969499100,9cyclictest2606-21cat19:20:130
2969499100,9cyclictest14507-21chrt20:50:360
297009998,1cyclictest8214-21open_files19:30:221
297009997,1cyclictest11944-21grep19:40:141
296949997,1cyclictest11859-21aten2_r7power_e19:40:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional