You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 13:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Jun 28, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1478029281,6sleep20-21swapper/219:05:232
1566528977,8sleep00-21swapper/019:09:040
1557928878,6sleep10-21swapper/119:07:551
1564828471,9sleep30-21swapper/319:08:503
169132550,2sleep116902-21apt-get23:30:141
1583199280,27cyclictest0-21swapper/222:00:352
1583699242,21cyclictest1142-21runrttasks23:03:053
1583199240,23cyclictest0-21swapper/222:50:362
1583199232,5cyclictest2470-21fschecks_count21:25:182
15831992221,1cyclictest0-21swapper/200:05:142
1583699200,0cyclictest0-21swapper/323:42:243
1582899201,18cyclictest9978-21expr23:45:161
1583199190,19cyclictest0-21swapper/223:32:462
15828991914,3cyclictest15820-21cyclictest00:03:451
15826991915,2cyclictest13-21ksoftirqd/023:00:010
15836991815,2cyclictest39-21ksoftirqd/300:02:243
1583699180,14cyclictest0-21swapper/300:37:433
1583699180,14cyclictest0-21swapper/300:37:423
1582899182,16cyclictest2305-21awk19:45:261
1582899181,1cyclictest4148-21ssh22:05:141
15826991817,1cyclictest0-21swapper/023:52:560
15836991715,1cyclictest0-21swapper/323:55:163
1583199170,2cyclictest0-21swapper/221:55:332
1582899179,1cyclictest8622-21diskmemload22:23:521
1582899170,9cyclictest0-21swapper/123:10:131
1582899170,0cyclictest0-21swapper/100:19:471
1582699175,6cyclictest10023-21awk00:05:000
1582699173,4cyclictest225-21systemd-journal21:29:230
15826991716,1cyclictest0-21swapper/022:25:050
1583699164,1cyclictest24323-21apt-get23:15:133
1583199162,1cyclictest141rcu_preempt00:23:222
15831991615,1cyclictest0-21swapper/200:33:032
15831991610,5cyclictest5093-21wc21:00:192
1583199160,4cyclictest0-21swapper/221:45:132
1583199160,15cyclictest19591-21/usr/sbin/munin20:25:142
1583199160,0cyclictest0-21swapper/222:36:262
1582899163,7cyclictest28264-21aten2_r7power_p19:35:141
15826991615,1cyclictest0-21swapper/022:09:310
15826991613,2cyclictest13-21ksoftirqd/000:11:230
1583699159,5cyclictest4945-21tr22:05:183
15836991511,2cyclictest26140-21ssh21:20:153
15836991511,2cyclictest19949-21wc19:15:263
1583699150,15cyclictest0-21swapper/321:45:253
1583199154,7cyclictest30317-21/usr/sbin/munin22:40:162
1583199151,1cyclictest141rcu_preempt23:40:202
1583199150,1cyclictest0-21swapper/200:18:072
15828991511,3cyclictest15736-21kworker/1:221:40:151
1583699141,13cyclictest0-21swapper/320:00:113
15836991411,2cyclictest5434-21tr22:25:153
15836991411,2cyclictest28819-21cut19:35:163
15836991411,2cyclictest17674-21grep19:10:283
15836991411,2cyclictest15970-21wc22:50:203
1583699140,7cyclictest0-21swapper/319:21:213
1583699140,7cyclictest0-21swapper/319:21:213
1583699140,1cyclictest0-21swapper/321:17:073
1583699140,1cyclictest0-21swapper/321:17:073
1583699140,13cyclictest0-21swapper/322:55:013
1583199149,4cyclictest0-21swapper/221:40:002
15831991413,1cyclictest11171-21sensors23:45:232
1583199141,2cyclictest12370-21/usr/sbin/munin22:10:172
1583199141,1cyclictest141rcu_preempt23:58:222
15831991410,2cyclictest0-21swapper/221:32:392
1583199140,14cyclictest0-21swapper/223:21:312
15828991413,1cyclictest0-21swapper/121:31:031
15828991411,3cyclictest30372-21kworker/1:221:25:331
15828991411,2cyclictest0-21swapper/100:28:531
15828991410,3cyclictest5617-21tr19:55:151
15828991410,3cyclictest28092-21grep19:35:011
1582899140,2cyclictest8622-21diskmemload00:05:151
1582899140,14cyclictest0-21swapper/123:35:141
1582899140,14cyclictest0-21swapper/100:38:131
1582899140,14cyclictest0-21swapper/100:38:131
1582899140,12cyclictest0-21swapper/121:45:341
1582699149,3cyclictest23332-21tr20:30:220
1582699146,4cyclictest225-21systemd-journal20:45:010
1582699142,6cyclictest32686-21munin-run19:45:000
1582699141,6cyclictest23632-21apt-get23:15:000
15826991411,2cyclictest19426-21wc19:15:200
15826991410,3cyclictest31811-21grep23:00:150
1582699140,8cyclictest18636-21idleruntime-cro00:10:000
1582699140,13cyclictest0-21swapper/020:15:130
1583699139,3cyclictest21297-21cron22:35:003
1583699139,3cyclictest10369-21head21:10:183
1583699138,3cyclictest8210-21grep21:05:273
1583699136,3cyclictest225-21systemd-journal23:29:593
15836991310,2cyclictest20010-21tail00:10:193
15836991310,2cyclictest17849-21cut20:20:173
15836991310,2cyclictest10762-21cut21:30:163
1583699130,13cyclictest0-21swapper/323:51:233
15831991312,1cyclictest0-21swapper/222:59:102
15831991312,1cyclictest0-21swapper/221:21:412
15831991311,1cyclictest610-21ssh23:40:002
15831991310,2cyclictest31529-21cat20:50:002
15831991310,2cyclictest23786-21wc23:15:012
15831991310,2cyclictest17219-21ssh22:31:492
1583199130,13cyclictest0-21swapper/200:02:242
1582899139,3cyclictest31665-21grep19:40:201
1582899139,3cyclictest19758-21grep00:10:171
1582899139,2cyclictest22006-21grep22:15:181
15828991310,2cyclictest32368-21wc23:20:011
15828991310,2cyclictest31313-21ssh22:40:181
15828991310,2cyclictest26121-21grep21:20:151
15828991310,2cyclictest22925-21head20:30:171
15828991310,2cyclictest19839-21grep23:50:261
1582899130,12cyclictest5676-21cat21:00:261
1582699139,3cyclictest9056-21sh23:45:000
1582699139,3cyclictest5222-21cut21:00:200
1582699139,3cyclictest18554-21ssh21:35:140
1582699139,2cyclictest4831-21sed00:20:230
1582699139,2cyclictest10717-21cut20:05:160
1582699137,4cyclictest2006-21apache221:15:290
1582699137,4cyclictest2006-21apache221:15:280
15826991312,1cyclictest0-21swapper/023:47:550
15826991310,2cyclictest7039-21grep19:55:270
15826991310,2cyclictest42822sleep023:21:580
15826991310,2cyclictest27217-21ssh21:20:220
15826991310,2cyclictest26199-21grep20:35:250
15826991310,2cyclictest14150-21cut22:30:140
1583699129,2cyclictest6994-21tail19:55:263
1583699129,2cyclictest4618-21cron22:25:003
1583699129,2cyclictest4564-21grep19:50:253
1583699129,2cyclictest39-21ksoftirqd/323:49:473
1583699129,2cyclictest3386-21tail20:55:263
1583699129,2cyclictest30825-21tr19:40:163
1583699129,2cyclictest28891-21grep00:15:233
1583699129,2cyclictest23693-21tr19:25:153
1583699129,2cyclictest20317-21cut20:25:173
1583699129,2cyclictest18322-21wc23:30:223
1583699129,2cyclictest16972-21awk20:20:013
1583699129,2cyclictest12198-21cron20:09:593
1583699128,3cyclictest14462-21taskset20:13:373
1583699121,5cyclictest0-21swapper/321:00:213
1583699120,7cyclictest0-21swapper/321:27:043
1583199129,2cyclictest5282-21wget22:25:152
1583199129,2cyclictest3841-21cron21:00:002
1583199129,2cyclictest32201-21tune2fs23:00:172
1583199129,2cyclictest30854-21grep19:40:162
1583199129,2cyclictest25711-21grep23:15:212
1583199129,2cyclictest25345-21grep20:35:162
1583199129,2cyclictest10881-21grep20:05:182
1583199128,4cyclictest21656-21hwlatdetect22:15:162
1583199128,3cyclictest7442-21dump-pmu-power19:59:592
1583199124,6cyclictest9907-21sh22:08:252
1582899129,2cyclictest7844-21wc22:45:231
1582899129,2cyclictest7733-21head21:05:231
1582899129,2cyclictest4556-21grep19:50:251
1582899129,2cyclictest32441-21grep20:50:161
1582899129,2cyclictest28195-21ls20:40:221
1582899129,2cyclictest20966-21grep00:30:181
1582899129,2cyclictest14976-21ssh22:30:201
1582899128,3cyclictest124312sleep120:10:051
1582899127,5cyclictest25-21ksoftirqd/123:00:441
1582899127,4cyclictest1866-21grep20:55:141
15828991211,1cyclictest0-21swapper/121:59:431
15828991210,1cyclictest0-21swapper/122:57:311
1582899120,2cyclictest0-21swapper/123:59:301
1582899120,11cyclictest12381-21seq22:29:331
1582699129,2cyclictest7770-21wc21:05:230
1582699129,2cyclictest7740-21tr23:05:140
1582699129,2cyclictest3218-21tr19:50:130
1582699129,2cyclictest30100-21sed20:45:170
1582699129,2cyclictest27375-21cron21:59:590
1582699129,2cyclictest22327-21grep22:35:140
1582699129,2cyclictest20247-21cut20:25:160
1582699129,2cyclictest1739-21head19:45:210
1582699129,2cyclictest12614-21ssh00:25:190
1582699128,3cyclictest29273-21grep19:35:200
1582699128,2cyclictest17999-21wc23:30:200
1582699123,6cyclictest11199-21apt-get21:50:140
1583699118,2cyclictest8256-21ssh23:05:173
1583699118,2cyclictest772-21tr19:45:143
1583699118,2cyclictest31988-21grep20:50:133
1583699118,2cyclictest30221-21chrt22:40:063
1583699118,2cyclictest29243-21ssh21:40:443
1583699118,2cyclictest27022-21tr19:30:193
1583699118,2cyclictest25613-21ssh22:36:413
1583699118,2cyclictest24622-21taskset20:35:133
1583699118,2cyclictest24167-21ssh00:32:023
1583699118,2cyclictest22318-21ssh21:36:533
1583699118,2cyclictest22008-21grep22:15:183
1583699118,2cyclictest2138-21sed23:20:253
1583699118,2cyclictest20836-21ssh21:55:213
1583699118,2cyclictest17439-21ssh21:53:443
1583699118,2cyclictest16659-21ssh23:10:173
1583699118,2cyclictest15723-21ssh00:26:553
1583699118,2cyclictest14109-21wc22:10:253
1583699118,2cyclictest13061-21ssh22:49:053
1583699118,2cyclictest10271-21ssh00:05:033
1583699117,3cyclictest28680-21ssh23:36:343
1583699110,11cyclictest0-21swapper/300:21:493
1583699110,10cyclictest2633-21ssh22:04:253
1583199119,1cyclictest27238-21grep20:40:152
1583199118,2cyclictest771-21grep19:45:142
1583199118,2cyclictest28392-21grep19:35:142
1583199118,2cyclictest28100-21strings21:40:202
1583199118,2cyclictest17414-21ssh21:15:132
1583199118,2cyclictest17414-21ssh21:15:132
1583199118,2cyclictest15943-21ssh00:27:052
1583199118,2cyclictest15674-21ssh21:52:222
1583199117,3cyclictest30841-21sh22:20:252
1583199115,5cyclictest20567-21grep00:10:242
1582899118,2cyclictest8770-21cut23:25:141
1582899118,2cyclictest7222-21ssh23:43:341
1582899118,2cyclictest5142-21cat00:20:281
1582899118,2cyclictest22931-21basename22:35:171
1582899118,2cyclictest21940-21cut19:20:211
1582899118,2cyclictest21940-21cut19:20:201
1582899118,2cyclictest21164-21ssh22:53:541
1582899118,2cyclictest19756-21ssh22:14:351
1582899118,2cyclictest18803-21tr21:35:151
1582899118,2cyclictest15907-21cat20:15:191
1582899118,2cyclictest12019-21grep21:50:171
1582899118,2cyclictest11260-21ssh21:10:311
1582899117,3cyclictest29624-21cut20:45:141
1582899113,7cyclictest25-21ksoftirqd/119:25:121
15828991110,1cyclictest0-21swapper/123:06:061
1582899110,11cyclictest0-21swapper/123:20:091
1582899110,10cyclictest29161-21needreboot22:00:211
1582699119,1cyclictest26050-21tr19:30:120
1582699118,2cyclictest7365-21ssh21:47:260
1582699118,2cyclictest3365-21grep20:55:260
1582699118,2cyclictest32514-21sh22:21:300
1582699118,2cyclictest31906-21sh22:02:000
1582699118,2cyclictest31090-21ssh00:35:550
1582699118,2cyclictest31090-21ssh00:35:550
1582699118,2cyclictest30508-21grep22:40:140
1582699118,2cyclictest27655-21cut00:15:150
1582699118,2cyclictest26870-21ssh21:40:130
1582699118,2cyclictest26089-21ssh23:55:090
1582699118,2cyclictest25043-21sh23:35:040
1582699118,2cyclictest21559-21grep19:20:170
1582699118,2cyclictest21559-21grep19:20:170
1582699118,2cyclictest14781-21ssh22:10:420
1582699118,2cyclictest12348-21ssh22:48:320
1582699118,2cyclictest12200-21ssh23:26:390
1582699117,2cyclictest7857-21grep20:00:120
1582699117,2cyclictest13246-21cut20:10:160
1582699116,3cyclictest17944-21cut20:20:170
1582699115,3cyclictest16405-21python22:50:260
1582699112,6cyclictest8622-21diskmemload23:15:130
1582699111,9cyclictest0-21swapper/021:30:520
1582699110,1cyclictest141rcu_preempt22:17:580
1582699110,10cyclictest18192-21basename19:15:000
1583699107,2cyclictest29820-21expr20:45:153
1583699107,2cyclictest22463-21tr20:30:153
1583199108,2cyclictest8944-21grep20:00:202
1583199108,2cyclictest22986-21date19:25:002
1583199108,2cyclictest22986-21date19:25:002
1583199108,2cyclictest22410-21cat20:30:152
1583199108,1cyclictest8451-21ssh23:25:042
1583199108,1cyclictest26189-21cat19:30:122
1583199108,1cyclictest205132sleep219:19:422
1583199108,1cyclictest17684-21ssh23:50:042
1583199108,1cyclictest14843-21apache_processe20:15:122
1583199108,1cyclictest13697-21grep20:10:202
1583199108,1cyclictest12854-21sh23:08:072
1583199107,2cyclictest7471-21cat21:05:202
1583199107,2cyclictest6241-21ssh22:45:062
1583199107,2cyclictest32425-21grep20:50:162
1583199107,2cyclictest31530-21ssh00:36:162
1583199107,2cyclictest31530-21ssh00:36:162
1583199107,2cyclictest16708-21grep19:10:192
1583199107,2cyclictest10917-21grep21:10:262
1582899108,1cyclictest24878-21cat20:35:141
1582899108,1cyclictest17051-21cat19:10:211
1582899107,2cyclictest20754-21ntpq20:25:191
1582899107,2cyclictest19403-21grep21:15:271
1582899107,2cyclictest19403-21grep21:15:271
1582899107,2cyclictest18540-21grep19:15:151
1582899107,2cyclictest17246-21taskset20:20:121
1582899100,9cyclictest11854-21chrt20:05:371
1582699108,1cyclictest25250-21chrt19:26:290
1582699107,2cyclictest21881-21chrt00:30:270
1582699107,2cyclictest10274-21ssh21:10:180
1582699106,2cyclictest31861-21grep20:50:120
158319998,1cyclictest4279-21mailstats19:50:212
158319998,1cyclictest192752sleep220:24:052
158319997,1cyclictest23298-21apt-key19:25:012
158289998,1cyclictest8080-21expr20:00:131
158369986,1cyclictest27773-21sh20:40:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional