You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Mar 24, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
967628976,9sleep00-21swapper/019:06:450
989628661,9sleep20-21swapper/219:09:372
962228371,8sleep30-21swapper/319:06:053
967228059,6sleep10-21swapper/119:06:421
1001799299,1cyclictest0-21swapper/322:04:333
1001799270,1cyclictest0-21swapper/321:55:213
1001399260,25cyclictest56550irq/124-enp0s3120:44:042
1001099260,24cyclictest0-21swapper/122:50:161
10017992524,1cyclictest0-21swapper/323:23:513
1001399250,25cyclictest0-21swapper/200:20:372
1001099250,1cyclictest0-21swapper/122:20:141
1001099240,24cyclictest0-21swapper/123:41:061
1001799230,1cyclictest141rcu_preempt00:34:513
10013992321,2cyclictest0-21swapper/222:57:132
1001399230,22cyclictest0-21swapper/223:10:132
1001099230,22cyclictest0-21swapper/100:25:191
1000399230,1cyclictest0-21swapper/021:55:480
1001399223,18cyclictest32-21ksoftirqd/220:10:142
10003992221,1cyclictest0-21swapper/000:20:370
10017992119,2cyclictest0-21swapper/321:49:513
1001399212,14cyclictest3330-21sensors23:25:242
10010992120,1cyclictest0-21swapper/121:53:401
10010992119,1cyclictest10293-21aten2_r7power_e19:10:111
1001099210,0cyclictest0-21swapper/122:26:131
1001799190,1cyclictest2825-21diskmemload22:40:383
10013991917,2cyclictest0-21swapper/222:30:022
10013991916,3cyclictest0-21swapper/223:00:132
10013991916,2cyclictest0-21swapper/200:33:582
1001399190,7cyclictest0-21swapper/220:45:162
10010991919,0cyclictest0-21swapper/121:35:581
1001099190,2cyclictest0-21swapper/123:49:221
10003991915,2cyclictest17745-21seq21:37:560
1000399190,1cyclictest0-21swapper/000:07:310
179292180,1sleep232-21ksoftirqd/221:57:282
1001799189,2cyclictest2411-21tail20:00:173
1001799182,9cyclictest7967-21apache_processe22:50:123
1001799180,16cyclictest0-21swapper/323:11:173
10013991816,2cyclictest0-21swapper/200:07:132
1001399181,17cyclictest56550irq/124-enp0s3120:26:242
1001399180,5cyclictest0-21swapper/222:05:112
1001399180,1cyclictest0-21swapper/222:23:552
10010991815,2cyclictest38150irq/131-snd_hda23:30:011
10010991814,2cyclictest22633-21df_abs19:35:151
1000399187,1cyclictest141rcu_preempt22:08:450
1000399181,16cyclictest0-21swapper/020:50:160
1001799179,2cyclictest1878-21egrep23:25:143
1001799178,3cyclictest18879-21wc19:25:253
10017991710,2cyclictest24171-21cat23:00:003
1001799170,4cyclictest0-21swapper/300:35:173
1001399178,8cyclictest0-21swapper/223:15:472
10013991716,1cyclictest0-21swapper/222:42:272
1001399170,16cyclictest0-21swapper/222:50:322
1001399170,16cyclictest0-21swapper/222:39:322
10010991717,0cyclictest0-21swapper/100:07:111
1000399174,9cyclictest2825-21diskmemload23:35:170
1001799169,2cyclictest13011-21ssh21:15:213
1001799168,2cyclictest24293-21sleep19:40:003
1001799168,2cyclictest17511-21wc20:30:243
1001799168,2cyclictest14529-21grep20:25:173
1001799164,4cyclictest20933-21apt-get21:40:133
1001799162,4cyclictest452-21systemd-logind23:30:233
10017991610,2cyclictest12835-21cut21:35:143
1001399160,16cyclictest0-21swapper/223:41:312
1001099162,13cyclictest537-21snmpd23:34:441
10010991615,1cyclictest0-21swapper/100:20:161
1000399160,1cyclictest0-21swapper/022:10:140
1001799158,2cyclictest9759-21grep20:15:183
1001799158,2cyclictest5822-21sed00:05:283
1001799158,2cyclictest4941-21ssh22:09:333
1001799158,2cyclictest3532-21ssh00:24:173
1001799158,2cyclictest2025-21wc21:05:233
1001799158,2cyclictest18908-21ssh00:13:593
1001799158,2cyclictest1393-21chrt23:44:203
1001799157,2cyclictest19659-21grep20:35:213
1001799152,12cyclictest22232-21grep22:20:003
1001799150,2cyclictest0-21swapper/323:07:093
1001799150,15cyclictest0-21swapper/323:39:053
1001799150,14cyclictest0-21swapper/323:17:053
10013991513,2cyclictest0-21swapper/200:15:232
1001399150,15cyclictest0-21swapper/221:36:512
1001399150,14cyclictest31127-21ssh00:01:272
10010991514,1cyclictest0-21swapper/122:10:231
1001099150,0cyclictest0-21swapper/100:02:221
1000399152,5cyclictest24748-21irqstats22:40:190
10003991511,4cyclictest0-21swapper/000:37:470
1000399150,1cyclictest0-21swapper/022:23:050
1000399150,14cyclictest0-21swapper/021:19:440
1001799147,2cyclictest5246-21grep21:10:293
10017991414,0cyclictest0-21swapper/319:40:133
10017991411,2cyclictest28060-21needreboot19:45:203
1001399149,5cyclictest32-21ksoftirqd/219:40:122
1001399148,3cyclictest32-21ksoftirqd/221:54:282
1001399140,11cyclictest0-21swapper/223:50:262
10010991412,2cyclictest142062sleep123:12:541
10010991410,3cyclictest2825-21diskmemload22:17:351
1001099140,2cyclictest0-21swapper/121:45:101
1000399145,7cyclictest24665-21/usr/sbin/munin23:00:180
1000399144,2cyclictest225-21systemd-journal20:50:000
1001799139,3cyclictest3838-21wc23:45:203
1001799139,3cyclictest22174-21sed00:15:243
10017991312,1cyclictest0-21swapper/323:00:513
10017991310,2cyclictest7521-21df22:30:123
10017991310,2cyclictest29749-21grep00:00:263
10017991310,2cyclictest22394-21cat20:40:223
1001799130,1cyclictest0-21swapper/321:25:343
1001799130,1cyclictest0-21swapper/320:10:143
1001399133,8cyclictest0-21swapper/219:21:032
1001399133,1cyclictest29275-21apt-get21:45:142
1001399132,1cyclictest0-21swapper/221:40:132
10013991311,2cyclictest7795-21ssh21:31:392
10013991311,2cyclictest0-21swapper/222:10:432
10013991311,1cyclictest0-21swapper/221:10:152
10013991310,2cyclictest796-21wc22:45:212
1001399130,7cyclictest0-21swapper/200:14:312
1001399130,13cyclictest0-21swapper/200:26:552
1001099135,7cyclictest6004-21timerandwakeup21:30:261
10010991310,2cyclictest30536-21ls19:50:201
10010991310,2cyclictest25300-21df23:00:161
10010991310,2cyclictest24358-21cat20:45:221
10010991310,2cyclictest23961-21grep22:40:151
10010991310,2cyclictest21139-21grep00:35:001
10010991310,2cyclictest13476-21grep21:55:121
1001099130,1cyclictest0-21swapper/123:24:361
1001099130,13cyclictest0-21swapper/123:37:351
1001099130,12cyclictest0-21swapper/122:02:521
1001099130,12cyclictest0-21swapper/100:10:371
1000399138,3cyclictest24634-21apache223:22:260
1000399133,10cyclictest0-21swapper/022:16:570
1000399131,7cyclictest0-21swapper/019:20:130
1000399131,4cyclictest0-21swapper/021:52:550
10003991310,2cyclictest6872-21grep00:25:280
10003991310,2cyclictest3512-21ssh23:25:260
10003991310,2cyclictest29074-21cron19:50:010
10003991310,2cyclictest17055-21grep22:55:140
1000399130,9cyclictest0-21swapper/021:22:250
1000399130,5cyclictest0-21swapper/022:25:140
1000399130,0cyclictest0-21swapper/023:15:140
1001799129,2cyclictest4558-21tr21:30:153
1001799129,2cyclictest30796-21tr21:00:163
1001799129,2cyclictest27317-21cron00:00:003
1001799129,2cyclictest24671-21grep22:20:283
1001799129,2cyclictest17284-21grep22:35:273
1001799126,2cyclictest225-21systemd-journal19:20:003
1001799123,2cyclictest29349-21apt-get19:50:133
1001799120,12cyclictest0-21swapper/322:13:413
1001399129,2cyclictest3397-21sh23:05:502
1001399129,2cyclictest30871-21cat19:50:242
1001399129,2cyclictest26722-21wc20:50:202
1001399129,2cyclictest21332-21tail23:55:262
1001399129,2cyclictest19510-21ls19:30:002
1001399128,3cyclictest8897-21ssh23:48:462
1001399128,3cyclictest20412-21grep19:30:162
1001399123,1cyclictest56550irq/124-enp0s3119:45:242
1001399122,9cyclictest56550irq/124-enp0s3121:05:002
1001399122,3cyclictest21943-21/usr/sbin/munin22:00:152
10013991211,1cyclictest0-21swapper/220:55:172
1001399120,9cyclictest0-21swapper/222:28:342
1001399120,1cyclictest0-21swapper/221:20:172
1001399120,11cyclictest19089-21ssh22:17:352
1001399120,11cyclictest13656-21wc20:25:012
1001099129,2cyclictest7596-21grep22:30:131
1001099129,2cyclictest26709-21ls19:45:011
1001099129,2cyclictest21213-21tr21:40:141
1001099129,2cyclictest1666-21head22:45:291
1001099129,2cyclictest12188-21sed20:20:191
1001099128,2cyclictest19535-21ssh23:55:141
1001099122,9cyclictest0-21swapper/120:32:151
10010991211,1cyclictest0-21swapper/123:08:301
1001099121,0cyclictest2825-21diskmemload23:53:251
1000399129,2cyclictest30989-21grep19:50:250
1000399129,2cyclictest30488-21sed21:45:190
1000399129,2cyclictest2412-21head20:00:170
1000399129,2cyclictest21578-21grep00:15:190
1000399129,2cyclictest21356-21sed23:55:260
1000399129,2cyclictest18633-21cut20:35:160
1000399129,2cyclictest12524-21cut00:10:150
1000399127,4cyclictest7474-21perf22:49:590
1000399126,5cyclictest8060-21swap20:10:260
1000399122,7cyclictest28155-21/usr/sbin/munin21:25:130
1000399122,4cyclictest18082-21sshd20:32:240
10003991210,1cyclictest21516-21sh21:40:160
10003991210,1cyclictest16082-21cut22:35:160
10003991210,1cyclictest12210-21chrt23:30:310
10003991210,1cyclictest11019-21grep23:10:240
10003991210,1cyclictest105442sleep021:14:310
1000399120,4cyclictest0-21swapper/000:30:160
1000399120,11cyclictest5075-21head21:30:170
1000399120,11cyclictest12661-21tr19:15:140
1000399120,11cyclictest11930-21grep20:20:170
1001799118,2cyclictest914-21sh22:25:413
1001799118,2cyclictest7075-21ssh00:25:323
1001799118,2cyclictest5455-21tr21:50:143
1001799118,2cyclictest26884-21grep20:50:213
1001799118,2cyclictest24640-21wc20:45:243
1001799118,2cyclictest23704-21ssh21:21:503
1001799118,2cyclictest2135-21ssh22:45:503
1001799118,2cyclictest13142-21sh23:50:313
1001799110,10cyclictest1049-21taskset19:55:383
1001399118,2cyclictest9603-21sed20:15:152
1001399118,2cyclictest6323-21cat20:10:012
1001399118,2cyclictest3647-21chrt20:03:042
1001399118,2cyclictest31669-21chrt21:26:362
1001399118,2cyclictest25866-21cut23:20:142
1001399118,2cyclictest21714-21cut00:35:152
1001399118,2cyclictest18962-21expr20:35:172
1001399118,2cyclictest18419-21cut23:35:132
1001399118,2cyclictest11930-21sh21:15:142
1001399118,2cyclictest1071-21taskset19:55:562
1001399118,2cyclictest10389-21tr23:30:142
1001399111,6cyclictest0-21swapper/219:15:152
1001099119,1cyclictest0-21swapper/122:35:141
1001099118,2cyclictest8299-21tail20:10:271
1001099118,2cyclictest5719-21tail20:05:271
1001099118,2cyclictest4669-21rm22:09:201
1001099118,2cyclictest4642-21tr21:10:231
1001099118,2cyclictest32710-21ls19:55:191
1001099118,2cyclictest28897-21grep20:55:171
1001099118,2cyclictest26814-21ssh00:18:431
1001099118,2cyclictest25428-21ssh21:23:091
1001099118,2cyclictest22602-21sed20:40:231
1001099118,2cyclictest18929-21ssh23:15:251
1001099118,2cyclictest17638-21pmu-power22:55:181
1001099118,2cyclictest13719-21cat19:15:201
1001099118,2cyclictest10727-21sed20:15:261
1001099117,3cyclictest28815-21ssh00:39:281
1001099115,2cyclictest225-21systemd-journal21:25:261
1001099111,1cyclictest141rcu_preempt19:25:191
1000399119,1cyclictest4512-21ssh23:06:450
1000399119,1cyclictest31129-21grep21:00:180
1000399119,1cyclictest301142sleep020:58:280
1000399119,1cyclictest2548-21ssh23:45:040
1000399119,1cyclictest11845-21tail23:50:180
1000399119,1cyclictest10242-21ssh22:50:390
1000399118,2cyclictest29791-21ssh23:41:130
1000399118,2cyclictest27537-21ssh22:03:250
1000399118,2cyclictest22090-21cat19:35:010
1000399118,2cyclictest21353-21cat20:40:120
1000399118,2cyclictest1448-21grep21:05:170
1000399114,4cyclictest0-21swapper/000:02:490
1000399110,10cyclictest6688-21ssh22:30:000
1001799108,1cyclictest29216-21sed20:55:203
1001799108,1cyclictest11677-21expr20:20:163
1001799108,1cyclictest11301-21sort19:10:163
1001799107,2cyclictest21418-21tail19:30:253
1001799105,2cyclictest225-21systemd-journal20:10:013
1001399107,2cyclictest30819-21tr21:00:162
1001399107,2cyclictest22947-21sed19:35:172
1001399107,2cyclictest17983-21chrt20:31:102
1001399101,8cyclictest56550irq/124-enp0s3119:10:062
1001099108,2cyclictest19466-21cat20:35:211
1001099108,1cyclictest34912sleep120:01:071
1001099108,1cyclictest2555-21seq21:06:531
1001099108,1cyclictest15633-21cut19:20:161
1001099107,2cyclictest27017-21wc20:50:231
1001099107,2cyclictest21143-21basename19:30:221
1001099107,2cyclictest13369-21cat21:15:261
1000399108,2cyclictest716-21cat19:55:250
1000399108,2cyclictest24127-21taskset19:38:320
1000399108,1cyclictest6387-21cut20:10:010
1000399108,1cyclictest24676-21cut19:40:120
1000399108,1cyclictest17527-21cut19:25:130
1000399107,2cyclictest10199-21taskset19:10:010
100179997,1cyclictest16348-21which19:20:223
100109997,1cyclictest30719-21grep21:00:161
100109997,1cyclictest27290-21seq19:45:161
100109997,1cyclictest14189-21expr20:25:151
100039998,1cyclictest10435-21python20:15:240
100039997,2cyclictest15402-21cat20:25:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional