You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 06:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Jan 10, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
323729380,9sleep00-21swapper/019:06:230
341328864,19sleep10-21swapper/119:08:421
138628642,39sleep30-21swapper/319:05:113
332728269,9sleep20-21swapper/219:07:352
72192630,1sleep17220-21sh23:23:131
276182540,0sleep30-21swapper/319:59:553
361299230,20cyclictest0-21swapper/320:35:133
3612992117,3cyclictest21334-31frcode00:08:103
3604992118,2cyclictest25-21ksoftirqd/100:10:231
360499211,2cyclictest24956-21kworker/1:123:29:331
3604992010,9cyclictest1081-21ssh22:25:131
360499200,2cyclictest0-21swapper/100:04:551
360899180,4cyclictest0-21swapper/223:40:122
360499180,1cyclictest1271-21kworker/1:100:19:431
360899173,8cyclictest9519-21awk00:20:002
3608991716,1cyclictest0-21swapper/223:36:512
360499171,13cyclictest0-21swapper/120:15:131
360499170,16cyclictest0-21swapper/120:00:141
361299160,0cyclictest0-21swapper/321:16:323
3608991615,1cyclictest0-21swapper/200:10:372
3604991612,3cyclictest38150irq/131-snd_hda22:45:161
3608991512,2cyclictest32-21ksoftirqd/220:40:162
360499150,2cyclictest0-21swapper/100:24:401
360299154,6cyclictest16125-21apt-get23:10:120
3612991410,3cyclictest21174-21sed20:50:253
361299140,1cyclictest0-21swapper/320:15:133
360899143,8cyclictest28839-21/usr/sbin/munin22:40:192
360899143,10cyclictest0-21swapper/221:36:372
3608991412,1cyclictest0-21swapper/200:03:412
360899141,1cyclictest0-21swapper/221:00:132
3608991411,2cyclictest27108-21cron00:30:002
360499149,3cyclictest4122-21ssh23:40:111
360499145,3cyclictest0-21swapper/121:35:001
360499142,11cyclictest0-21swapper/121:26:051
3604991411,2cyclictest14711-21wc22:50:231
360299144,9cyclictest11197-21apache_accesses19:25:130
3602991411,2cyclictest9403-21sh22:29:590
255342140,12sleep10-21swapper/119:55:141
3612991311,1cyclictest227942sleep300:26:523
3612991310,2cyclictest2855-21cat22:25:253
3612991310,2cyclictest24637-21sort19:50:253
3612991310,2cyclictest18286-21cron00:24:593
3612991310,2cyclictest17423-21tr20:45:153
3612991310,2cyclictest16496-21grep19:35:133
3612991310,2cyclictest16163-21cut23:10:143
3612991310,2cyclictest13326-21awk19:30:003
3612991310,2cyclictest10220-21grep22:30:143
360899132,5cyclictest0-21swapper/223:23:292
3608991312,1cyclictest55650irq/127-enp2s0-22:53:322
3608991311,1cyclictest0-21swapper/222:30:262
3608991310,2cyclictest32069-21ls23:00:202
3608991310,2cyclictest20711-21grep20:50:212
3608991310,2cyclictest12367-21wc00:40:012
3608991310,2cyclictest12367-21wc00:40:012
360899130,12cyclictest22945-21ls22:00:202
360499139,3cyclictest3607-21dump-pmu-power23:40:001
360499139,3cyclictest20844-21ssh22:00:011
360499139,3cyclictest17204-21basename20:45:151
360499139,2cyclictest15806-21ssh22:15:131
360499137,4cyclictest516-21gdbus20:55:221
360499134,5cyclictest225-21systemd-journal22:59:591
360499131,6cyclictest30466-21ssh22:05:121
3604991310,2cyclictest32338-21wc23:00:221
3604991310,2cyclictest15473-21uniq21:20:141
3604991310,2cyclictest12869-21grep19:25:261
3604991310,2cyclictest11916-21sort21:55:011
360499130,12cyclictest14258-21cron21:20:001
360499130,0cyclictest0-21swapper/122:20:181
360299138,5cyclictest0-21swapper/023:19:570
360299138,1cyclictest13-21ksoftirqd/021:55:270
360299134,5cyclictest13569-21apache_accesses19:30:120
360299130,7cyclictest18067-21munin-run19:40:000
360299130,6cyclictest0-21swapper/000:39:140
360299130,6cyclictest0-21swapper/000:39:130
108462132,1sleep210985-21ssh00:20:172
361299129,2cyclictest7381-21basename20:25:013
361299129,2cyclictest730-21perf00:15:003
361299129,2cyclictest3623-21apt22:45:003
361299129,2cyclictest29421-21chrt21:45:453
361299129,2cyclictest29092-21ssh22:22:103
361299129,2cyclictest2398-21wc20:15:013
361299129,2cyclictest23538-21chrt22:37:233
361299129,2cyclictest2057-21head00:15:173
361299129,2cyclictest15421-21wc22:15:013
361299129,2cyclictest14471-21tr22:50:213
361299129,2cyclictest11723-21basename21:55:003
361299129,2cyclictest1087-21grep21:30:183
3612991210,1cyclictest21590-21apt-get23:50:123
360899129,2cyclictest6006-21head20:20:162
360899129,2cyclictest3493-21aten2_r7power_v21:50:142
360899129,2cyclictest31258-21ssh00:31:372
360899129,2cyclictest28394-21rm22:21:412
360899129,2cyclictest28346-21sh23:16:252
360899129,2cyclictest27843-21ntpq21:05:192
360899129,2cyclictest24272-21rm22:55:342
360899129,2cyclictest18885-21cut21:40:162
360899129,2cyclictest18762-21ssh23:48:172
360899129,2cyclictest17767-21grep00:05:262
360899129,2cyclictest17177-21uname23:10:192
360899129,2cyclictest13581-21ssh23:26:272
360899129,2cyclictest12684-21wc20:35:142
360899121,2cyclictest141rcu_preempt19:45:002
3608991211,1cyclictest0-21swapper/221:27:482
360899121,10cyclictest0-21swapper/221:11:222
360899120,11cyclictest22652-21grep20:55:162
360499129,2cyclictest8728-21head20:25:191
360499129,2cyclictest6002-21sed00:35:251
360499129,2cyclictest6002-21sed00:35:251
360499129,2cyclictest5502-21grep19:10:251
360499129,2cyclictest4365-21chrt21:14:191
360499129,2cyclictest3615-21munin-plugin-st22:45:001
360499129,2cyclictest2830-21cron21:50:001
360499129,2cyclictest27496-21sed21:05:161
360499129,2cyclictest21959-21cut23:50:131
360499129,2cyclictest21003-21cut19:45:131
360499129,2cyclictest21003-21cut19:45:131
360499129,2cyclictest19793-21grep20:50:141
360499129,2cyclictest19739-21chrt19:40:231
360499129,2cyclictest15224-21sh22:15:001
360499127,4cyclictest31246-21cut20:05:191
360499126,5cyclictest682-21taskset00:32:581
360499122,10cyclictest28801-21kworker/1:122:04:451
360499120,4cyclictest225-21systemd-journal19:55:001
360299129,2cyclictest9444-21chrt19:20:140
360299129,2cyclictest30800-21ssh21:46:460
360299129,2cyclictest3021-21grep00:15:250
360299129,2cyclictest25667-21ssh21:44:290
360299129,2cyclictest2442-21ssh21:30:340
360299129,2cyclictest21698-21munin-run20:55:000
360299129,2cyclictest2060-21ssh22:06:530
360299129,2cyclictest14733-21chrt20:40:080
360299129,2cyclictest14504-21perf20:40:000
360299129,2cyclictest13617-21cut23:45:160
360299129,2cyclictest11886-21sed22:30:260
360299129,2cyclictest11751-21ssh21:18:130
360299128,3cyclictest9836-21ssh21:35:150
360299128,3cyclictest3610-21sh22:49:550
360299123,3cyclictest18316-21perf00:25:000
360299122,6cyclictest8453-21latency_hist19:19:590
360299122,4cyclictest0-21swapper/020:45:140
3602991211,1cyclictest21642-21apache223:41:490
360299120,8cyclictest0-21swapper/021:05:120
360299120,4cyclictest0-21swapper/000:07:030
360299120,11cyclictest31384-21cut23:55:150
360299120,11cyclictest29604-21grep22:40:250
360299120,11cyclictest29403-21basename21:10:120
360299120,11cyclictest21917-21ssh22:55:080
361299119,1cyclictest19320-21wc23:30:143
361299118,2cyclictest9320-21sed20:25:253
361299118,2cyclictest7438-21wget23:05:143
361299118,2cyclictest6764-21ssh23:40:513
361299118,2cyclictest5590-21ssh22:09:263
361299118,2cyclictest5395-21sh22:45:193
361299118,2cyclictest4847-21ssh23:21:293
361299118,2cyclictest3498-21ssh23:57:493
361299118,2cyclictest30800-21ssh23:00:103
361299118,2cyclictest30670-21ssh21:10:203
361299118,2cyclictest27637-21ssh00:30:123
361299118,2cyclictest27489-21sed21:05:153
361299118,2cyclictest26802-21wc23:15:273
361299118,2cyclictest2654-21ssh23:39:193
361299118,2cyclictest25091-21grep21:00:173
361299118,2cyclictest25075-21ssh22:56:093
361299118,2cyclictest25030-21ntp_kernel_pll_21:25:193
361299118,2cyclictest22104-21df22:00:133
361299118,2cyclictest20053-21ssh21:40:263
361299118,2cyclictest18044-21sh22:15:383
361299118,2cyclictest16204-21ssh21:56:413
361299118,2cyclictest15058-21grep21:20:133
361299118,2cyclictest13863-21ssh21:37:243
361299118,2cyclictest13574-21forks23:45:163
361299118,2cyclictest11721-21ssh00:39:373
361299118,2cyclictest11721-21ssh00:39:373
361299118,2cyclictest10971-21head23:25:173
361299118,2cyclictest10006-21ssh00:01:103
361299110,10cyclictest15650-21cat20:40:163
360899119,2cyclictest31388-21cut20:05:192
360899119,2cyclictest29812-21sh21:46:022
360899119,2cyclictest29805-21seq20:02:142
360899119,2cyclictest25369-21ssh23:51:402
360899119,2cyclictest22246-21sh23:31:192
360899119,1cyclictest96322sleep220:28:522
360899119,1cyclictest885-21ssh22:25:052
360899119,1cyclictest6728-21grep21:15:162
360899119,1cyclictest6251-21ssh23:59:512
360899119,1cyclictest30398-21cat22:05:112
360899119,1cyclictest26624-21grep19:55:192
360899119,1cyclictest23223-21tr19:50:142
360899119,1cyclictest212622sleep222:35:462
360899119,1cyclictest20940-21grep19:45:132
360899119,1cyclictest20940-21grep19:45:132
360899119,1cyclictest16697-21tail19:35:162
360899119,1cyclictest15800-21hostname22:15:132
360899119,1cyclictest15514-21ssh21:20:142
360899119,1cyclictest13566-21sh21:55:182
360899118,2cyclictest1312-21sort20:10:192
360899118,2cyclictest12228-21cat19:25:212
360899116,2cyclictest475-21in:imuxsock21:31:322
360899113,7cyclictest3623-21apt-get22:45:002
360899110,1cyclictest0-21swapper/222:10:402
360899110,10cyclictest8646-21dpkg19:20:012
360899110,10cyclictest7155-21ssh23:05:082
360499118,2cyclictest32003-21grep23:55:191
360499118,2cyclictest27146-21ssh23:15:331
360499118,2cyclictest25328-21ssh21:44:141
360499118,2cyclictest19224-21tr23:30:151
360499118,2cyclictest18874-21wget00:25:121
360499118,2cyclictest18629-21ssh22:35:071
360499118,2cyclictest18078-21turbostat.cron19:40:001
360499118,2cyclictest16551-21ssh21:39:201
360499118,2cyclictest15922-21ssh23:10:041
360499118,2cyclictest15704-21ssh00:05:101
360499118,2cyclictest12795-21ssh23:45:071
360499118,2cyclictest11667-21ssh23:07:091
360499118,2cyclictest10332-21cat22:30:141
360499118,2cyclictest10207-21taskset19:20:211
360499116,3cyclictest14899-21sendmail_mailst19:30:221
360299119,2cyclictest11679-21ssh23:25:220
360299119,1cyclictest7975-21expr20:25:140
360299119,1cyclictest7809-21grep00:00:170
360299119,1cyclictest4525-21grep19:10:160
360299119,1cyclictest32340-21grep23:00:220
360299119,1cyclictest31702-21ssh23:36:490
360299119,1cyclictest299552sleep020:04:100
360299119,1cyclictest25039-21sh23:51:250
360299119,1cyclictest24428-21sh22:20:040
360299119,1cyclictest21766-21ssh23:30:580
360299119,1cyclictest19382-21ssh22:35:150
360299119,1cyclictest15866-21ssh22:15:130
360299119,1cyclictest14015-21sh23:08:520
360299119,1cyclictest10672-21cut20:30:180
360299118,2cyclictest6912-21cut22:10:120
360299118,2cyclictest5272-21grep21:50:250
360299118,2cyclictest3107-21wc23:20:250
360299118,2cyclictest30308-21basename20:05:130
360299118,2cyclictest22536-21chrt19:45:390
360299118,2cyclictest22536-21chrt19:45:380
360299118,2cyclictest19820-21wc19:40:240
360299118,2cyclictest18632-21sh00:25:050
360299118,2cyclictest16617-21cat21:20:240
360299118,2cyclictest13330-21ssh22:50:140
360299117,3cyclictest29046-21head00:30:210
360299111,1cyclictest141rcu_preempt00:10:150
360299110,10cyclictest25896-21ssh22:02:050
360299110,10cyclictest24110-21grep21:00:000
360299110,10cyclictest23608-21ssh21:25:070
361299108,1cyclictest32229-21taskset20:07:533
361299108,1cyclictest27966-21aten2_r7power_e20:00:143
361299108,1cyclictest10044-21aten2_r7power_e20:30:143
361299107,2cyclictest8168-21chrt19:18:093
361299107,2cyclictest13943-21expr19:30:153
360899109,1cyclictest3772-21kernelversion20:15:182
360899109,1cyclictest18596-21cat20:45:262
360899108,1cyclictest5532-21grep19:10:252
360899108,1cyclictest14736-21ntpq19:30:212
360899107,2cyclictest8320-21sh19:20:002
360499108,1cyclictest6619-21tr20:20:201
360499108,1cyclictest22182sleep120:13:161
360499108,1cyclictest12597-21expr20:35:141
360499107,2cyclictest6353-21grep19:15:121
360499100,9cyclictest16640-21seq20:42:041
360299108,1cyclictest5377-21cut20:20:130
360299108,1cyclictest27181-21tail19:55:260
360299107,2cyclictest25826-21wc21:00:250
360299107,2cyclictest23525-21expr19:50:140
360299100,10cyclictest1277-21wc20:10:180
36129997,1cyclictest8667-21cat19:20:013
36129997,1cyclictest5645-21chrt19:11:203
36129997,1cyclictest23274-21expr20:55:223
36129997,1cyclictest18300-21chrt19:40:013
36129996,2cyclictest21002-21tr19:45:133
36129996,2cyclictest21002-21tr19:45:123
36089998,1cyclictest10130-21basename20:30:152
36049997,1cyclictest10306-21cat20:30:161
36049994,4cyclictest24624-21cut21:00:141
36029998,1cyclictest3947-21sed20:15:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional