You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 01:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Aug 22, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
422129669,16sleep20-21swapper/207:05:562
421429575,16sleep00-21swapper/007:05:500
448528067,9sleep30-21swapper/307:09:233
426227758,15sleep10-21swapper/107:06:281
462799260,25cyclictest0-21swapper/210:56:262
462399250,0cyclictest0-21swapper/112:23:451
462399250,0cyclictest0-21swapper/112:23:451
4623992422,2cyclictest0-21swapper/111:25:351
462399231,1cyclictest25222-21ssh09:25:391
462399231,1cyclictest25222-21ssh09:25:381
462399210,19cyclictest0-21swapper/110:00:121
462199211,5cyclictest141rcu_preempt11:47:460
463199201,18cyclictest28067-21ssh10:25:283
4627992020,0cyclictest0-21swapper/209:32:522
462399200,2cyclictest0-21swapper/110:31:031
462199205,1cyclictest141rcu_preempt11:28:160
462199200,16cyclictest0-21swapper/011:36:050
463199191,11cyclictest25536-21sensors09:45:233
462799192,16cyclictest0-21swapper/211:35:252
4623991917,1cyclictest0-21swapper/111:54:091
462399191,18cyclictest32429-21if_enp0s31f612:05:181
462199193,15cyclictest0-21swapper/010:10:130
462399188,4cyclictest0-21swapper/107:40:011
462399180,1cyclictest0-21swapper/109:34:591
462199184,8cyclictest3398-21perf08:14:590
4621991818,0cyclictest0-21swapper/010:18:400
462199180,17cyclictest0-21swapper/009:31:030
462799175,6cyclictest537-21snmpd10:19:572
462399170,2cyclictest0-21swapper/109:42:391
462199171,8cyclictest6976-21munin-run07:15:010
462199170,17cyclictest0-21swapper/010:56:490
462199170,15cyclictest0-21swapper/010:44:080
463199161,1cyclictest141rcu_preempt10:05:223
463199160,1cyclictest0-21swapper/311:23:343
4627991615,1cyclictest0-21swapper/212:26:052
462799160,2cyclictest0-21swapper/210:36:182
462799160,2cyclictest0-21swapper/210:36:172
462799160,15cyclictest0-21swapper/211:47:302
4623991611,4cyclictest25-21ksoftirqd/110:12:591
462399160,16cyclictest0-21swapper/111:14:211
462399160,14cyclictest0-21swapper/109:20:311
4627991514,1cyclictest0-21swapper/207:55:132
462799150,4cyclictest0-21swapper/209:19:562
462799150,15cyclictest0-21swapper/210:54:582
462799150,14cyclictest0-21swapper/209:26:532
462799150,14cyclictest0-21swapper/209:26:522
4623991515,0cyclictest0-21swapper/110:50:031
462399150,1cyclictest0-21swapper/112:10:151
462199153,11cyclictest4956-21munin-node11:01:440
4621991511,3cyclictest13-21ksoftirqd/012:30:170
462199150,9cyclictest6872-21awk12:10:000
462199150,1cyclictest0-21swapper/010:21:480
462199150,15cyclictest0-21swapper/011:24:380
462199150,12cyclictest0-21swapper/007:45:120
463199149,3cyclictest15567-21cat09:40:143
4631991413,1cyclictest0-21swapper/309:10:203
4631991411,2cyclictest1816-21cut10:10:163
4631991410,3cyclictest32486-21grep09:50:143
463199140,6cyclictest29937-21apache212:07:343
462799146,4cyclictest0-21swapper/209:15:012
462799142,11cyclictest0-21swapper/212:24:052
462799142,11cyclictest0-21swapper/212:24:042
4627991413,1cyclictest0-21swapper/210:43:152
4627991413,1cyclictest0-21swapper/209:43:202
4627991412,2cyclictest32-21ksoftirqd/211:15:252
4627991412,1cyclictest0-21swapper/210:48:382
462799141,11cyclictest21478-21/usr/sbin/munin11:20:162
4627991410,3cyclictest8459-21cat12:10:202
4623991414,0cyclictest0-21swapper/111:56:351
4623991413,1cyclictest0-21swapper/110:25:451
462399141,1cyclictest0-21swapper/111:44:521
4623991411,2cyclictest22524-21awk11:20:251
4623991410,3cyclictest20253-21latency_hist08:49:591
462199144,4cyclictest225-21systemd-journal12:38:380
4621991414,0cyclictest0-21swapper/009:10:130
4621991411,2cyclictest0-21swapper/011:30:230
75712130,0sleep30-21swapper/312:29:513
463199139,3cyclictest4600-21cron11:29:593
463199139,3cyclictest22924-21sort11:40:163
4631991313,0cyclictest0-21swapper/310:31:103
463199131,2cyclictest141rcu_preempt12:14:443
4631991312,1cyclictest39-21ksoftirqd/309:29:073
4631991312,1cyclictest39-21ksoftirqd/309:29:063
4631991311,1cyclictest39-21ksoftirqd/311:16:193
4631991310,2cyclictest9110-21grep09:35:243
4631991310,2cyclictest5757-21cron08:19:593
4631991310,2cyclictest26492-21grep07:55:143
4631991310,2cyclictest16561-21grep09:20:263
463199130,12cyclictest17855-21grep07:35:223
462799139,3cyclictest6755-21head11:30:222
462799139,2cyclictest31016-21cat08:05:012
462799139,2cyclictest29291-21grep11:25:162
462799139,2cyclictest15491-21cut09:20:172
462799133,8cyclictest32-21ksoftirqd/209:55:012
4627991310,2cyclictest9883-21cut07:20:162
462799130,12cyclictest2860-21sed10:10:252
462799130,12cyclictest0-21swapper/209:45:272
462399131,8cyclictest12010-21apt-get07:25:121
4623991310,2cyclictest5765-21sh08:20:001
4623991310,2cyclictest28625-21wc09:05:191
4623991310,2cyclictest25774-21head10:05:171
462399130,2cyclictest0-21swapper/112:19:571
462399130,1cyclictest0-21swapper/112:27:141
462399130,1cyclictest0-21swapper/112:03:081
462399130,12cyclictest0-21swapper/109:12:361
462399130,0cyclictest0-21swapper/109:18:311
462199139,2cyclictest18283-21grep08:45:120
462199135,7cyclictest16104-21munin-run10:00:010
462199135,6cyclictest25258-21ssh09:45:220
4621991312,1cyclictest0-21swapper/009:42:570
4621991310,2cyclictest31598-21chrt12:24:470
4621991310,2cyclictest31598-21chrt12:24:460
4621991310,2cyclictest2593-21chrt09:50:580
4621991310,2cyclictest25151-21wc07:50:220
4621991310,2cyclictest13998-21rm09:19:560
463199129,2cyclictest9748-21gzip09:55:223
463199129,2cyclictest9669-21grep07:20:143
463199129,2cyclictest7668-21grep08:20:253
463199129,2cyclictest28518-21grep11:05:153
463199129,2cyclictest27915-21grep09:05:153
463199129,2cyclictest25024-21awk08:59:593
463199129,2cyclictest24128-21tr07:50:153
463199129,2cyclictest23530-21sleep12:20:003
463199129,2cyclictest23158-21tail07:45:263
463199129,2cyclictest21722-21cut08:50:223
463199129,2cyclictest19492-21tr10:40:173
463199129,2cyclictest14668-21grep07:30:153
463199129,2cyclictest14668-21grep07:30:143
463199129,2cyclictest12551-21df07:25:153
463199128,3cyclictest2030-21sed08:10:173
463199128,2cyclictest16755-21basename10:00:143
463199125,6cyclictest31918-21dump-pmu-power12:25:013
463199125,6cyclictest31918-21dump-pmu-power12:25:003
463199121,3cyclictest0-21swapper/312:30:133
4631991211,1cyclictest0-21swapper/312:36:033
463199120,1cyclictest0-21swapper/310:21:383
462799129,2cyclictest6541-21grep08:20:142
462799129,2cyclictest5118-21grep07:10:132
462799129,2cyclictest27403-21sleep09:05:002
462799129,2cyclictest22629-21turbostat.cron08:55:002
462799129,2cyclictest22545-21sh12:00:012
462799129,2cyclictest18501-21grep10:00:262
462799129,2cyclictest15183-21sh09:40:002
462799129,2cyclictest10178-21grep12:30:272
462799128,3cyclictest15141-21grep07:30:182
462799128,3cyclictest15141-21grep07:30:172
462799127,4cyclictest25961-21sh10:25:102
4627991210,1cyclictest985-21grep12:05:272
462399129,2cyclictest9746-21gzip09:55:231
462399129,2cyclictest675-21grep09:50:151
462399129,2cyclictest4596-21ssh11:48:581
462399129,2cyclictest21144-21grep08:50:171
462399129,2cyclictest20644-21ssh12:37:011
462399129,2cyclictest20168-21grep11:00:151
462399129,2cyclictest18002-21grep10:20:141
462399129,2cyclictest17186-21grep08:40:271
462399129,2cyclictest14472-21grep11:35:181
462399129,2cyclictest12889-21cron08:34:591
462399129,2cyclictest10481-21ssh10:35:151
462399129,2cyclictest10481-21ssh10:35:141
462399128,3cyclictest25685-21taskset07:51:041
4623991211,0cyclictest0-21swapper/110:41:171
4623991210,2cyclictest25-21ksoftirqd/111:30:111
462199129,2cyclictest6038-21sed11:10:240
462199129,2cyclictest31372-21tr08:05:170
462199129,2cyclictest28602-21cat11:05:150
462199129,2cyclictest27866-21cat10:25:260
462199129,2cyclictest24420-21idleruntime-cro10:04:590
462199129,2cyclictest20840-21sed07:40:280
462199129,2cyclictest1985-21sed12:25:280
462199129,2cyclictest16053-21sed07:30:270
462199129,2cyclictest16053-21sed07:30:270
462199129,2cyclictest13432-21grep08:35:160
462199128,3cyclictest4244-21sort08:15:160
462199128,2cyclictest25218-21grep10:05:150
462199127,5cyclictest29937-21apache209:23:160
4621991211,1cyclictest13-21ksoftirqd/011:17:290
463199119,1cyclictest14486-21fgrep08:35:243
463199118,2cyclictest5889-21sh10:51:003
463199118,2cyclictest5211-21wc11:10:183
463199118,2cyclictest30822-21cron08:05:013
463199118,2cyclictest30378-21sh11:45:063
463199118,2cyclictest2498-21ssh10:50:013
463199118,2cyclictest20406-21wc07:40:223
463199118,2cyclictest20218-21expr11:00:153
463199118,2cyclictest19285-21rm10:59:593
463199118,2cyclictest12898-21ssh11:53:583
463199118,2cyclictest11316-21head10:15:273
463199118,2cyclictest10904-21sh11:33:143
463199118,2cyclictest10033-21sed08:25:283
463199117,3cyclictest17662-21cron08:44:593
463199116,4cyclictest39-21ksoftirqd/311:55:153
463199113,4cyclictest225-21systemd-journal11:40:003
463199113,4cyclictest225-21systemd-journal10:40:013
463199113,4cyclictest225-21systemd-journal10:40:003
463199110,10cyclictest6297-21taskset09:15:133
463199110,0cyclictest141rcu_preempt12:02:353
462799118,2cyclictest7296-21grep11:50:172
462799118,2cyclictest700-21ssh11:07:422
462799118,2cyclictest4673-21ssh11:10:142
462799118,2cyclictest27662-21ssh10:06:002
462799118,2cyclictest24561-21grep07:50:172
462799118,2cyclictest24282-21wc12:00:192
462799118,2cyclictest23262-21ssh11:40:182
462799118,2cyclictest22636-21nfs4_client07:45:192
462799118,2cyclictest20160-21taskset08:48:572
462799118,2cyclictest20147-21cut11:00:152
462799118,2cyclictest20072-21ssh12:36:332
462799118,2cyclictest19757-21ssh10:20:282
462799118,2cyclictest1850-21ssh10:30:072
462799118,2cyclictest18491-21ssh12:16:042
462799118,2cyclictest17208-21sed08:40:262
462799111,3cyclictest0-21swapper/207:35:152
462799110,10cyclictest9320-21cut09:55:202
462399119,1cyclictest27809-21grep10:45:171
462399118,2cyclictest9910-21grep07:20:161
462399118,2cyclictest9410-21cat08:25:211
462399118,2cyclictest9343-21ssh10:15:141
462399118,2cyclictest6183-21mailstats07:10:201
462399118,2cyclictest28395-21awk08:00:001
462399118,2cyclictest2749-21cat08:10:251
462399118,2cyclictest24686-21grep09:45:171
462399118,2cyclictest14000-21ssh12:33:211
4623991110,0cyclictest0-21swapper/109:36:351
462399110,10cyclictest551-21chrt08:07:221
462399110,10cyclictest2204-21ssh11:08:521
462199118,2cyclictest5825-21ssh10:50:580
462199118,2cyclictest30377-21sed08:00:260
462199118,2cyclictest29690-21chrt09:09:060
462199118,2cyclictest28654-21ssh10:45:260
462199118,2cyclictest26056-21ssh09:26:200
462199118,2cyclictest26056-21ssh09:26:190
462199118,2cyclictest26025-21tail09:00:170
462199118,2cyclictest22345-21ssh11:40:130
462199118,2cyclictest22181-21grep08:50:270
462199118,2cyclictest18684-21ssh11:56:570
462199118,2cyclictest17528-21head07:35:180
462199118,2cyclictest16179-21cut08:40:170
462199118,2cyclictest15944-21ssh12:15:140
462199118,2cyclictest12966-21ssh10:35:550
462199118,2cyclictest12966-21ssh10:35:540
462199118,2cyclictest12251-21ssh12:12:420
462199118,2cyclictest11560-21ssh11:52:550
462199117,2cyclictest23248-21sh12:00:150
462199116,2cyclictest225-21systemd-journal08:30:000
462199114,5cyclictest2021-21apache_accesses10:30:130
462199112,1cyclictest7846-21df_inode09:35:140
463199108,1cyclictest31134-21sh09:30:053
463199108,1cyclictest26828-21grep09:00:253
463199108,1cyclictest11508-21grep08:30:173
463199107,2cyclictest8210-21sed07:15:193
463199107,2cyclictest6490-21grep07:10:243
463199100,9cyclictest651-21taskset08:08:343
463199100,9cyclictest19998-21taskset08:46:543
462799108,1cyclictest31623-21sed08:05:172
462799108,1cyclictest27909-21grep09:05:152
462799108,1cyclictest20508-21cat07:40:232
462799108,1cyclictest1506-21expr08:10:152
462799107,2cyclictest9294-21cat08:25:202
462799107,2cyclictest3678-21grep08:15:142
462799107,2cyclictest23784-21cat08:55:192
462799107,2cyclictest13837-21taskset07:26:372
462799100,9cyclictest11043-21tr08:30:142
462399108,1cyclictest19944-21which07:40:191
462399108,1cyclictest14828-21cat07:30:161
462399108,1cyclictest14828-21cat07:30:151
462399107,2cyclictest7250-21hostname07:15:131
462399107,2cyclictest7034-21cat08:20:181
462399107,2cyclictest15179-21ssh10:56:451
462399102,7cyclictest12586-21tr11:15:161
462199108,1cyclictest6469-21expr08:20:140
462199108,1cyclictest22894-21apache_volume08:55:130
462199108,1cyclictest115212sleep007:22:210
462199107,2cyclictest27831-21timerwakeupswit07:55:250
462199107,2cyclictest11205-21expr08:30:140
462199100,9cyclictest7429-21wc07:15:140
46279997,1cyclictest9263-21chrt07:19:072
46279996,2cyclictest13183-21cut08:35:162
46239997,1cyclictest28776-21basename08:00:151
46239997,1cyclictest25643-21cat09:00:141
46239997,1cyclictest235922sleep108:55:171
46239997,1cyclictest21757-21expr07:45:141
46239996,2cyclictest14001-21basename08:35:201
46219997,1cyclictest12626-21fschecks_count07:25:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional