You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 03:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Apr 13, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1085229671,9sleep30-21swapper/319:05:563
1114329266,8sleep20-21swapper/219:09:412
1084929279,9sleep00-21swapper/019:05:530
1098227767,6sleep10-21swapper/119:07:351
89652580,0sleep00-21swapper/023:07:480
1126099352,3cyclictest55650irq/127-enp2s0-22:30:132
1125799252,1cyclictest4261-21diskmemload23:04:091
1126199230,1cyclictest0-21swapper/323:59:243
11261992221,1cyclictest0-21swapper/300:39:403
11261992216,5cyclictest39-21ksoftirqd/322:30:083
1126199220,3cyclictest0-21swapper/323:50:293
11257992218,4cyclictest25-21ksoftirqd/122:23:031
11261992112,8cyclictest32388-21munin-node23:12:423
11257992119,1cyclictest0-21swapper/123:30:571
1125799210,19cyclictest0-21swapper/122:05:351
1126099205,13cyclictest0-21swapper/223:15:162
1126099205,13cyclictest0-21swapper/223:15:162
1126099200,18cyclictest0-21swapper/221:30:292
1126199192,14cyclictest30219-21ssh23:40:143
11260991917,1cyclictest4261-21diskmemload22:54:242
11257991916,2cyclictest4261-21diskmemload21:32:391
1126199180,1cyclictest0-21swapper/322:21:373
1126199180,1cyclictest0-21swapper/321:33:563
11260991814,2cyclictest0-21swapper/200:35:112
11257991815,2cyclictest20053-21df_abs22:55:151
11257991814,3cyclictest38150irq/131-snd_hda23:50:181
1125599186,6cyclictest30972-21munin-run21:45:000
11255991816,1cyclictest15932-21fschecks_time20:25:170
11261991716,1cyclictest0-21swapper/321:46:553
1126099170,1cyclictest0-21swapper/200:26:442
1125799171,14cyclictest4261-21diskmemload22:37:361
1125799170,17cyclictest0-21swapper/120:35:001
1125599170,2cyclictest0-21swapper/022:39:070
1125599170,1cyclictest0-21swapper/000:15:200
11261991615,1cyclictest0-21swapper/322:44:193
11260991611,3cyclictest21644-21ls20:35:262
1125599160,2cyclictest0-21swapper/021:10:090
11261991512,2cyclictest39-21ksoftirqd/321:50:183
1126199150,14cyclictest0-21swapper/300:23:303
11260991513,1cyclictest0-21swapper/223:31:342
1126099150,2cyclictest0-21swapper/223:13:442
1126099150,15cyclictest0-21swapper/200:31:132
1125799153,11cyclictest4261-21diskmemload23:26:131
11257991511,3cyclictest25-21ksoftirqd/120:55:211
1125799150,4cyclictest0-21swapper/122:01:341
1125799150,11cyclictest38150irq/131-snd_hda19:40:121
1125599150,14cyclictest0-21swapper/022:52:310
11261991411,2cyclictest94-21kswapd021:40:003
11261991410,3cyclictest24327-21sed20:40:263
1126199140,1cyclictest0-21swapper/323:45:133
1126199140,14cyclictest0-21swapper/321:55:113
1126199140,13cyclictest0-21swapper/321:11:593
1126199140,0cyclictest0-21swapper/322:19:333
1126099149,4cyclictest55650irq/127-enp2s0-20:42:182
11260991411,2cyclictest6907-21wc23:25:262
11260991411,2cyclictest32134-21tail23:40:272
11260991411,2cyclictest11873-21cat23:10:002
11260991410,3cyclictest31835-21cron21:00:002
11260991410,2cyclictest19886-21expr22:55:142
1125799145,6cyclictest30849-21cron00:00:001
1125799144,8cyclictest7464-21/usr/sbin/munin21:50:271
1125799144,8cyclictest0-21swapper/120:30:141
11257991413,1cyclictest0-21swapper/121:28:081
11257991412,1cyclictest12402-21ssh21:14:401
11257991411,2cyclictest6737-21ssh23:45:151
11257991411,2cyclictest16107-21sed21:35:221
11257991410,3cyclictest25021-21grep21:40:261
1125799140,13cyclictest0-21swapper/123:43:071
1125599141,8cyclictest0-21swapper/021:05:140
11255991411,2cyclictest19944-21chmod20:35:000
11255991411,2cyclictest13-21ksoftirqd/022:30:040
11255991410,2cyclictest9805-21tail00:25:180
11255991410,2cyclictest1340-21grep19:55:160
1125599140,1cyclictest0-21swapper/021:18:370
1126199139,3cyclictest21179-21cron21:19:593
11261991310,2cyclictest26168-21ssh23:37:173
11261991310,2cyclictest20917-21grep20:35:173
1126099139,3cyclictest9425-21grep21:50:282
1126099139,3cyclictest19730-21wc22:35:222
1126099139,2cyclictest16130-21chrt00:10:092
1126099133,9cyclictest22565-21gltestperf21:20:182
1126099133,4cyclictest0-21swapper/222:29:082
1126099132,5cyclictest225-21systemd-journal19:44:552
1126099131,11cyclictest15678-21sh23:50:192
11260991310,2cyclictest9639-21sed22:10:172
11260991310,2cyclictest7454-21grep20:05:252
11260991310,2cyclictest6952-21grep23:45:162
11260991310,2cyclictest25921-21tr22:20:162
11260991310,2cyclictest25728-21awk22:00:202
11260991310,2cyclictest22101-21grep23:35:152
11260991310,2cyclictest16142-21cat19:20:002
1126099130,1cyclictest0-21swapper/221:35:142
1126099130,1cyclictest0-21swapper/200:20:342
1125799131,2cyclictest0-21swapper/123:05:431
1125799131,11cyclictest4261-21diskmemload23:24:231
11257991310,2cyclictest3968-21uname22:45:191
11257991310,2cyclictest25010-21sh00:15:141
11257991310,2cyclictest17012-21tail00:10:171
11257991310,2cyclictest13934-21cut19:15:141
1125799130,12cyclictest0-21swapper/100:37:211
1125599139,3cyclictest5563-21cron21:30:000
1125599139,3cyclictest22137-21grep23:35:150
1125599139,3cyclictest20826-21ssh00:12:270
1125599135,4cyclictest225-21systemd-journal00:36:100
1125599134,6cyclictest31857-21grep21:00:000
11255991310,2cyclictest8849-21head20:10:170
11255991310,2cyclictest32362-21grep19:50:280
11255991310,2cyclictest27607-21cut22:40:180
11255991310,2cyclictest16749-21grep21:55:170
1125599130,2cyclictest0-21swapper/023:40:310
1125599130,0cyclictest0-21swapper/022:17:150
1126199129,2cyclictest9249-21grep00:25:153
1126199129,2cyclictest31336-21cut21:25:203
1126199129,2cyclictest30480-21munin-plugin-st19:50:003
1126199129,2cyclictest26517-21grep00:15:253
1126199129,2cyclictest25048-21sed21:40:263
1126199129,2cyclictest24656-21chrt23:17:003
1126199129,2cyclictest24656-21chrt23:17:003
1126199129,2cyclictest22648-21sed19:30:253
1126199129,2cyclictest18555-21cat19:25:003
1126199129,2cyclictest17436-21grep00:30:153
1126199129,2cyclictest12170-21cut22:50:203
1126199128,3cyclictest55650irq/127-enp2s0-23:30:163
1126199121,3cyclictest225-21systemd-journal00:06:063
1126199120,1cyclictest0-21swapper/322:59:073
1126199120,11cyclictest29361-21chrt22:02:433
1126099129,2cyclictest32646-21head00:00:202
1126099129,2cyclictest32456-21ssh00:20:002
1126099129,2cyclictest29092-21cut23:00:202
1126099129,2cyclictest28032-21cut20:50:182
1126099129,2cyclictest24603-21sed19:35:222
1126099129,2cyclictest24386-21tr23:55:212
1126099129,2cyclictest12644-21basename20:20:012
1126099128,3cyclictest30476-21sleep19:50:002
1126099128,3cyclictest16331-21tr19:20:132
1126099128,3cyclictest15140-21cat20:25:002
1126099128,1cyclictest10533-21ssh21:13:142
1126099125,3cyclictest0-21swapper/222:05:152
1126099122,8cyclictest1534-21gltestperf19:55:172
1125799129,2cyclictest9107-21tr22:10:141
1125799129,2cyclictest32258-21sed00:00:181
1125799129,2cyclictest23190-21sed23:35:221
1125799129,2cyclictest2255-21ssh22:25:161
1125799129,2cyclictest22105-21ssh21:20:161
1125799129,2cyclictest21112-21grep19:30:131
1125799129,2cyclictest12641-21sleep20:20:011
1125799129,2cyclictest10232-21grep22:30:151
1125799120,10cyclictest0-21swapper/121:05:141
1125599129,2cyclictest9338-21df22:10:150
1125599129,2cyclictest7805-21grep21:50:160
1125599129,2cyclictest6519-21grep20:05:160
1125599129,2cyclictest3484-21cut22:45:160
1125599129,2cyclictest29594-21wc21:25:000
1125599129,2cyclictest28515-21wc19:45:160
1125599129,2cyclictest26748-21head19:40:160
1125599129,2cyclictest2376-21grep22:25:160
1125599129,2cyclictest15069-21rm23:50:160
1125599129,2cyclictest1246-21head22:05:170
1125599126,4cyclictest25536-21/usr/sbin/munin22:20:170
1125599126,1cyclictest225-21systemd-journal23:03:560
1125599121,9cyclictest0-21swapper/023:17:130
1125599121,9cyclictest0-21swapper/023:17:130
11255991211,1cyclictest0-21swapper/021:31:280
1125599120,6cyclictest18394-21perf19:25:000
1125599120,2cyclictest0-21swapper/023:33:490
1125599120,1cyclictest0-21swapper/023:56:120
1125599120,1cyclictest0-21swapper/000:05:180
1126199119,1cyclictest28137-21grep23:00:153
1126199119,1cyclictest12627-21basename20:20:013
1126199118,2cyclictest8758-21sort20:10:163
1126199118,2cyclictest7877-21ssh23:06:573
1126199118,2cyclictest7489-21rm23:25:453
1126199118,2cyclictest6385-21grep20:05:163
1126199118,2cyclictest542-21ssh00:00:243
1126199118,2cyclictest4847-21sort20:00:263
1126199118,2cyclictest3621-21sh22:25:283
1126199118,2cyclictest3560-21sh22:06:183
1126199118,2cyclictest31473-21ssh23:20:463
1126199118,2cyclictest29838-21cut20:55:143
1126199118,2cyclictest25714-21tail20:45:173
1126199118,2cyclictest24459-21ssh22:38:393
1126199118,2cyclictest2418-21tr21:05:153
1126199118,2cyclictest23788-21ssh21:20:293
1126199118,2cyclictest20541-21taskset19:27:293
1126199118,2cyclictest17899-21grep20:30:163
1126199118,2cyclictest16389-21egrep00:10:143
1126199118,2cyclictest10036-21cut22:10:193
1126199115,4cyclictest14969-21strings19:15:193
11261991111,0cyclictest0-21swapper/322:45:423
1126199110,1cyclictest0-21swapper/321:00:133
1126199110,10cyclictest171732sleep320:27:383
1126099118,2cyclictest853-21ssh21:26:222
1126099118,2cyclictest4006-21fschecks_time20:00:172
1126099118,2cyclictest3557-21ssh22:45:172
1126099118,2cyclictest30258-21ssh23:20:192
1126099118,2cyclictest27111-21ssh21:42:002
1126099118,2cyclictest26489-21taskset22:40:032
1126099118,2cyclictest25754-21expr20:45:172
1126099118,2cyclictest22141-21tr19:30:182
1126099118,2cyclictest18605-21wc22:15:212
1126099118,2cyclictest14020-21ssh21:15:172
1126099118,2cyclictest13271-21chrt19:11:002
1126099113,4cyclictest225-21systemd-journal00:05:052
1126099111,9cyclictest32621-21df_abs21:00:152
1126099110,2cyclictest0-21swapper/221:05:142
1126099110,10cyclictest3322-21ssh21:47:092
1126099110,10cyclictest23826-21ssh21:59:562
1125799118,2cyclictest5414-21ssh00:22:401
1125799118,2cyclictest32254-21grep21:45:171
1125799118,2cyclictest31677-21uname19:50:201
1125799118,2cyclictest29470-21grep19:45:211
1125799118,2cyclictest26796-21grep22:40:151
1125799118,2cyclictest24026-21cut19:35:151
1125799118,2cyclictest17088-21ssh21:16:481
1125799118,2cyclictest16507-21ssh23:12:101
1125799118,2cyclictest16492-21cat21:55:151
1125799118,2cyclictest15055-21cat20:25:001
1125799118,2cyclictest11908-21grep22:50:181
1125799118,2cyclictest11064-21ssh00:06:321
1125799115,5cyclictest4261-21diskmemload22:15:241
1125799114,5cyclictest13602-21turbostat19:15:001
1125799110,10cyclictest22635-21sh23:15:291
1125799110,10cyclictest22635-21sh23:15:291
1125599118,2cyclictest8836-21ssh23:45:550
1125599118,2cyclictest5844-21ssh00:23:000
1125599118,2cyclictest31145-21ssh00:00:030
1125599118,2cyclictest29756-21sort23:20:160
1125599118,2cyclictest27796-21ssh22:01:300
1125599118,2cyclictest24770-21sensors_fan19:35:230
1125599118,2cyclictest22970-21ssh22:56:370
1125599118,2cyclictest22682-21grep20:40:130
1125599118,2cyclictest19758-21ntpq19:25:190
1125599118,2cyclictest17437-21cut00:30:150
1125599118,2cyclictest16912-21ssh21:35:350
1125599118,2cyclictest12083-21sh23:10:030
1125599118,2cyclictest11317-21basename20:15:170
1125599114,4cyclictest32153-21fschecks_count21:45:160
1125599110,10cyclictest8263-21ssh23:26:220
1125599110,10cyclictest23099-21taskset19:34:540
1125599110,10cyclictest11522-21tr19:10:140
1126199108,1cyclictest25885-21dpkg19:40:013
1126199108,1cyclictest24424-21cat19:35:193
1126199108,1cyclictest13172-21sed19:10:263
1126199108,1cyclictest12965-21aten2_r7power_p20:20:143
1126199107,2cyclictest671-21bc19:55:113
1126199107,2cyclictest31064-21cat19:50:163
1126199107,2cyclictest27829-21expr20:50:173
1126099108,1cyclictest8149-21grep20:10:142
1126099108,1cyclictest19485-21cat19:25:172
1126099107,2cyclictest30807-21wget19:50:152
1126099107,2cyclictest18560-21uname20:30:202
1126099107,2cyclictest15401-21grep20:25:142
1125799108,1cyclictest8433-21expr20:10:151
1125799108,1cyclictest28008-21cut20:50:181
1125799108,1cyclictest17661-21swap19:20:241
1125799107,2cyclictest904-21cut21:00:211
1125799107,2cyclictest3317-21apt-get20:00:031
1125799107,2cyclictest25372-21cat20:45:151
1125799107,2cyclictest19360-21grep19:25:161
1125799107,2cyclictest18797-21ssh00:30:231
1125799107,2cyclictest17155-21taskset20:27:231
1125799107,2cyclictest11621-21ssh00:25:551
11257991010,0cyclictest0-21swapper/120:05:171
1125599108,1cyclictest4081-21tail20:00:170
1125599108,1cyclictest20220-21apache_processe20:35:130
1125599108,1cyclictest15912-21chrt19:19:200
1125599107,2cyclictest26586-21expr20:45:250
1125599107,2cyclictest14581-21sed20:20:270
112579997,1cyclictest649-21taskset19:55:101
112579997,1cyclictest23169-21sed20:40:151
112559997,1cyclictest32547-21expr21:00:150
112559997,1cyclictest27359-21apache_accesses20:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional