You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Sep 05, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
89229178,9sleep00-21swapper/019:05:570
115529167,8sleep20-21swapper/219:08:452
110428966,6sleep10-21swapper/119:08:081
93228771,9sleep30-21swapper/319:06:293
135099290,28cyclictest0-21swapper/300:39:173
134899280,28cyclictest0-21swapper/222:22:372
134899260,24cyclictest0-21swapper/200:04:122
134899240,23cyclictest0-21swapper/222:44:482
135099233,14cyclictest8107-21df_inode22:35:133
1344992319,3cyclictest0-21swapper/121:25:161
1344992319,3cyclictest0-21swapper/121:25:151
1344992318,2cyclictest225-21systemd-journal19:30:001
134499230,22cyclictest0-21swapper/121:53:411
12722230,0sleep30-21swapper/300:26:053
134899221,1cyclictest4028-21apt-get21:35:122
134899220,1cyclictest0-21swapper/222:34:222
134899220,19cyclictest0-21swapper/221:46:202
134499224,18cyclictest26347-21diskmemload23:25:161
1344992219,2cyclictest25-21ksoftirqd/119:45:131
134899210,21cyclictest0-21swapper/221:55:402
134899210,19cyclictest0-21swapper/221:28:112
134899210,19cyclictest0-21swapper/221:28:102
1344992115,5cyclictest12137-21sh23:55:021
1350992018,1cyclictest263512sleep322:26:173
1348992017,2cyclictest32-21ksoftirqd/221:21:512
134899200,20cyclictest0-21swapper/221:31:452
134899200,19cyclictest0-21swapper/223:40:132
134499203,17cyclictest0-21swapper/122:05:141
1344992019,1cyclictest26347-21diskmemload21:22:591
1344992015,4cyclictest26347-21diskmemload23:17:081
1344992011,8cyclictest0-21swapper/123:00:001
134099200,19cyclictest0-21swapper/023:40:130
1348991919,0cyclictest0-21swapper/221:15:072
1344991917,1cyclictest14813-21kworker/1:022:53:111
1344991917,1cyclictest0-21swapper/100:05:041
1344991914,5cyclictest31464-21ssh23:46:321
1340991918,1cyclictest0-21swapper/000:13:080
135099181,5cyclictest225-21systemd-journal23:26:433
1350991812,3cyclictest13942-21cpuspeed_turbos22:00:153
135099180,1cyclictest141rcu_preempt23:24:553
135099180,13cyclictest0-21swapper/323:01:483
1348991815,2cyclictest32-21ksoftirqd/200:15:182
134899181,1cyclictest141rcu_preempt23:04:162
134899180,18cyclictest0-21swapper/223:51:262
134899180,17cyclictest0-21swapper/221:14:492
134499189,6cyclictest0-21swapper/122:30:271
134499188,6cyclictest23837-21munin-run22:45:001
1344991816,1cyclictest26347-21diskmemload22:37:161
1344991814,4cyclictest25-21ksoftirqd/121:05:011
1344991814,3cyclictest23104-21sendmail00:40:001
1344991813,5cyclictest25-21ksoftirqd/100:25:031
1344991813,1cyclictest0-21swapper/122:19:381
1344991810,7cyclictest0-21swapper/121:40:251
134099187,6cyclictest32211-21munin-run22:50:000
134899174,1cyclictest141rcu_preempt00:35:482
134899173,0cyclictest141rcu_preempt22:54:392
134899171,5cyclictest225-21systemd-journal19:19:592
1348991715,1cyclictest32-21ksoftirqd/223:58:502
134899170,17cyclictest0-21swapper/223:14:162
134899170,17cyclictest0-21swapper/223:14:152
134499174,13cyclictest0-21swapper/121:12:061
134499173,1cyclictest241rcuc/122:13:341
1344991711,6cyclictest0-21swapper/100:12:101
134499170,2cyclictest0-21swapper/120:20:131
1348991616,0cyclictest0-21swapper/200:11:552
134899161,3cyclictest141rcu_preempt00:34:122
134499169,7cyclictest0-21swapper/123:51:591
1344991616,0cyclictest0-21swapper/120:40:141
134499161,2cyclictest25-21ksoftirqd/100:03:091
1344991612,4cyclictest25-21ksoftirqd/100:34:041
1344991612,3cyclictest26347-21diskmemload00:24:511
134099165,6cyclictest25376-21munin-run20:00:000
134099160,1cyclictest0-21swapper/021:59:210
134099160,0cyclictest0-21swapper/021:54:590
134899154,2cyclictest18740-21tail20:50:262
1348991511,3cyclictest16891-21wc19:40:182
134899150,15cyclictest0-21swapper/223:09:272
134899150,15cyclictest0-21swapper/222:10:332
134899150,13cyclictest0-21swapper/200:08:202
134499157,7cyclictest0-21swapper/123:20:111
134499156,9cyclictest0-21swapper/122:03:171
134499156,8cyclictest0-21swapper/123:37:161
1344991514,1cyclictest25-21ksoftirqd/121:32:371
1344991514,1cyclictest0-21swapper/120:30:111
1344991513,1cyclictest6069-21cron19:19:591
134099156,4cyclictest455-21dbus-daemon00:08:290
1340991514,1cyclictest0-21swapper/021:19:330
1340991512,2cyclictest0-21swapper/021:26:170
1340991512,2cyclictest0-21swapper/021:26:160
134099150,2cyclictest141rcu_preempt22:55:310
1350991410,3cyclictest30688-21tr20:10:163
1350991410,3cyclictest19844-21wc19:45:323
135099140,14cyclictest0-21swapper/322:47:203
135099140,13cyclictest0-21swapper/319:35:133
135099140,13cyclictest0-21swapper/319:35:133
1348991411,0cyclictest141rcu_preempt23:19:402
1348991410,3cyclictest8951-21grep19:25:132
1348991410,2cyclictest25781-21wc21:05:322
1348991410,2cyclictest19720-21wc19:45:322
1348991410,1cyclictest32-21ksoftirqd/222:16:042
134899140,14cyclictest0-21swapper/222:37:012
134899140,13cyclictest0-21swapper/222:58:162
134899140,12cyclictest0-21swapper/223:30:302
134499149,5cyclictest0-21swapper/121:46:451
134499148,6cyclictest0-21swapper/122:20:151
134499147,7cyclictest0-21swapper/123:30:141
134499143,10cyclictest25-21ksoftirqd/120:25:131
1344991412,2cyclictest0-21swapper/123:05:441
1344991412,1cyclictest26347-21diskmemload22:25:191
1344991411,3cyclictest0-21swapper/123:10:131
1344991411,3cyclictest0-21swapper/123:10:121
1344991411,3cyclictest0-21swapper/122:47:401
1344991411,3cyclictest0-21swapper/121:16:281
134099147,6cyclictest18592-21apache223:50:020
1340991412,1cyclictest18565-21apache223:55:000
135099139,3cyclictest25874-21tr20:00:143
135099139,3cyclictest12282-21ssh21:20:183
135099137,3cyclictest18875-21sh00:17:433
135099132,5cyclictest19191-21apt20:54:593
1350991310,2cyclictest8452-21cron19:25:003
1350991310,2cyclictest5966-21grep21:55:163
1350991310,2cyclictest29966-21grep21:30:263
135099130,12cyclictest32329-21wc22:10:243
135099130,12cyclictest24948-21grep19:55:263
134899139,2cyclictest29420-21grep23:45:212
1348991311,1cyclictest0-21swapper/221:00:202
1348991310,2cyclictest31106-21grep21:50:252
1348991310,2cyclictest24109-21cut22:25:162
1348991310,2cyclictest20149-21tune2fs20:55:162
1348991310,2cyclictest12709-21grep19:30:242
1348991310,2cyclictest12709-21grep19:30:242
134899130,12cyclictest0-21swapper/221:43:322
134499135,2cyclictest0-21swapper/100:16:001
1344991312,1cyclictest0-21swapper/121:35:121
1344991311,1cyclictest19695-21sh23:40:141
1344991310,2cyclictest32123-21grep20:10:281
1344991310,2cyclictest24821-21timerwakeupswit19:55:251
1344991310,2cyclictest18781-21cat23:00:241
1340991310,2cyclictest9382-21grep22:35:240
1340991310,2cyclictest4735-21head19:15:180
1340991310,2cyclictest31974-21cat20:10:260
1340991310,2cyclictest27854-21ssh21:10:180
1340991310,2cyclictest27494-21head23:25:180
1340991310,2cyclictest15190-21grep22:20:160
134099130,1cyclictest0-21swapper/022:06:100
134099130,12cyclictest0-21swapper/021:21:570
135099129,2cyclictest8995-21ssh22:55:143
135099129,2cyclictest3687-21wc23:30:173
135099129,2cyclictest30822-21wc00:05:203
135099129,2cyclictest29680-21tail20:05:243
135099129,2cyclictest25990-21df23:05:153
135099129,2cyclictest24449-21sort21:05:133
135099129,2cyclictest22952-21wc22:24:593
135099129,2cyclictest19951-21chrt21:44:443
135099129,2cyclictest11910-21sendmail-msp21:40:003
135099129,2cyclictest10060-21cut23:15:133
135099128,3cyclictest7746-21tr20:30:133
135099128,3cyclictest39-21ksoftirqd/321:45:123
135099128,3cyclictest21888-21tr19:50:213
135099121,10cyclictest0-21swapper/319:40:123
1350991210,2cyclictest39-21ksoftirqd/321:51:193
1350991210,2cyclictest3022-21kworker/3:100:33:123
135099120,2cyclictest0-21swapper/322:05:283
135099120,1cyclictest0-21swapper/323:38:573
135099120,1cyclictest0-21swapper/322:40:263
134899129,2cyclictest8223-21grep20:30:152
134899129,2cyclictest32304-21cat22:50:002
134899129,2cyclictest26848-21sed20:00:212
134899129,2cyclictest23145-21grep00:20:192
134899128,3cyclictest14418-21ssh22:00:162
134899125,3cyclictest225-21systemd-journal20:44:312
134899125,2cyclictest225-21systemd-journal22:06:332
134899124,6cyclictest11957-21grep23:35:162
134899120,11cyclictest31816-21cat00:25:182
134899120,11cyclictest11585-21grep20:35:262
134499129,2cyclictest1396-21wc20:15:191
134499129,2cyclictest11521-21grep20:35:261
134499128,3cyclictest0-21swapper/119:35:171
134499128,3cyclictest0-21swapper/119:35:161
134499121,9cyclictest0-21swapper/121:58:401
134499121,5cyclictest225-21systemd-journal20:51:321
134499120,1cyclictest0-21swapper/119:30:121
134499120,1cyclictest0-21swapper/119:30:121
134099129,2cyclictest3697-21latency_hist19:15:000
134099129,2cyclictest25459-21grep21:05:280
134099129,2cyclictest2487-21idleruntime-cro20:20:000
134099129,2cyclictest18011-21cat20:50:190
134099129,2cyclictest17963-21cron19:45:010
134099129,2cyclictest16018-21grep00:35:150
134099129,2cyclictest15526-21cat22:00:250
134099129,2cyclictest13947-21grep20:40:260
134099129,2cyclictest10472-21apt23:35:010
134099128,3cyclictest26288-21cut23:05:160
134099121,6cyclictest20311-21munin-run21:45:010
1340991210,1cyclictest3365-21cut20:20:150
134099120,11cyclictest14849-21tail00:15:180
135099118,2cyclictest7458-21cat00:10:273
135099118,2cyclictest4694-21cut23:50:153
135099118,2cyclictest4222-21ssh22:51:563
135099118,2cyclictest3418-21taskset19:11:433
135099118,2cyclictest31649-21ssh22:30:163
135099118,2cyclictest28860-21tail23:45:163
135099118,2cyclictest27722-21sed21:10:173
135099118,2cyclictest2745-21ssh21:15:093
135099118,2cyclictest22123-21ssh00:20:043
135099118,2cyclictest20916-21head21:25:203
135099118,2cyclictest20916-21head21:25:203
135099118,2cyclictest19948-21grep20:55:153
135099118,2cyclictest19657-21ssh23:40:123
135099118,2cyclictest16473-21taskset20:46:273
135099118,2cyclictest14490-21wc22:20:003
135099118,2cyclictest13530-21cat23:55:183
135099117,3cyclictest3318-21tail23:10:263
135099117,3cyclictest3318-21tail23:10:253
135099117,3cyclictest11455-21wc20:35:263
135099114,6cyclictest487-21avahi-daemon00:00:093
134899118,2cyclictest4088-21wc20:20:222
134899118,2cyclictest3631-21taskset19:14:302
134899118,2cyclictest26575-21ssh23:25:122
134899118,2cyclictest23358-21cut19:55:132
134899118,2cyclictest19618-21head23:20:202
134499118,2cyclictest7458-21cut19:20:201
134499117,3cyclictest24854-21grep21:05:151
134099119,1cyclictest10551-21grep23:15:160
134099118,2cyclictest8185-21chrt19:21:360
134099118,2cyclictest8126-21ntpq00:30:190
134099118,2cyclictest7848-21grep22:15:200
134099118,2cyclictest7534-21basename20:30:110
134099118,2cyclictest6650-21ssh22:53:500
134099118,2cyclictest5916-21tail21:35:260
134099118,2cyclictest31891-21ssh22:10:200
134099118,2cyclictest31732-21grep00:25:180
134099118,2cyclictest31668-21ssh21:31:410
134099118,2cyclictest27919-21ssh23:45:070
134099118,2cyclictest25888-21ssh00:02:510
134099118,2cyclictest24728-21ssh22:25:200
134099118,2cyclictest22894-21ssh00:20:160
134099118,2cyclictest22782-21dump-pmu-power19:55:010
134099118,2cyclictest22180-21sh21:45:200
134099118,2cyclictest21886-21cut21:00:120
134099118,2cyclictest20161-21tr20:55:160
134099118,2cyclictest18485-21expr19:45:130
134099118,2cyclictest18173-21ssh23:20:120
134099118,2cyclictest17308-21tr23:00:140
134099118,2cyclictest13384-21date19:35:010
134099118,2cyclictest13384-21date19:35:000
134099113,3cyclictest1429-21apt-get23:10:130
134099113,3cyclictest1429-21apt-get23:10:120
134099110,10cyclictest31339-21sh22:30:030
135099108,1cyclictest57422sleep319:16:033
135099108,1cyclictest47772sleep320:23:543
135099108,1cyclictest11980-21cat19:30:173
135099108,1cyclictest11980-21cat19:30:173
135099108,1cyclictest1000-21grep20:15:153
135099107,2cyclictest5766-21sort20:25:163
135099107,2cyclictest23487-21sed21:00:253
135099107,2cyclictest13507-21grep20:40:203
135099103,3cyclictest9777-21missed_timers19:25:183
134899108,1cyclictest27998-21chrt20:05:092
134899108,1cyclictest2285-21taskset20:17:372
134899108,1cyclictest16436-21taskset20:45:592
134899107,2cyclictest7671-21find19:20:232
134899107,2cyclictest20896-21tr19:50:142
134899107,2cyclictest13752-21tr19:35:142
134899107,2cyclictest13752-21tr19:35:132
134899105,3cyclictest30543-21grep20:10:162
134499107,2cyclictest28266-21tr20:05:141
134499107,2cyclictest21402-21expr19:50:171
134499106,4cyclictest0-21swapper/119:40:131
134499106,3cyclictest0-21swapper/120:55:101
134499106,2cyclictest25-21ksoftirqd/120:45:131
134499101,6cyclictest25-21ksoftirqd/120:00:141
134499100,9cyclictest1691-21cut19:10:131
134099108,1cyclictest5504-21expr20:25:150
134099108,1cyclictest15800-21ntpq20:45:200
134099108,1cyclictest109812sleep023:35:040
134099107,2cyclictest9887-21apache_volume20:35:120
134099107,2cyclictest10117-21python19:25:240
134099106,3cyclictest17101-21grep22:40:180
13489996,2cyclictest7072-21taskset20:27:542
13409997,1cyclictest14169-21gltestperf19:35:170
13409997,1cyclictest14169-21gltestperf19:35:160
13409996,2cyclictest26230-21seq20:00:150
13409985,2cyclictest29678-21grep20:05:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional