You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-02 - 15:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 02, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2719729077,9sleep00-21swapper/007:07:230
2713228659,8sleep30-21swapper/307:06:333
2731628561,8sleep20-21swapper/207:08:572
2705327958,6sleep10-21swapper/107:05:311
214722790,1sleep221476-21latency_hist09:30:002
27493993131,0cyclictest0-21swapper/308:56:333
2748899291,2cyclictest13991-21grep08:55:161
2749099280,27cyclictest0-21swapper/212:19:522
2749099240,23cyclictest0-21swapper/210:40:162
2749099240,23cyclictest0-21swapper/210:40:152
2748399240,23cyclictest0-21swapper/011:00:260
27490992322,0cyclictest0-21swapper/211:10:262
2749099230,23cyclictest0-21swapper/211:30:492
2748899220,1cyclictest20292-21diskmemload10:24:441
2749099212,10cyclictest225-21systemd-journal08:00:002
2749099200,20cyclictest0-21swapper/211:22:052
2748899200,19cyclictest0-21swapper/107:15:131
2749399192,2cyclictest141rcu_preempt11:22:213
2749099190,0cyclictest0-21swapper/212:26:502
27488991914,4cyclictest25-21ksoftirqd/111:50:001
2748899190,4cyclictest23634-21cron08:10:001
2748399197,6cyclictest21647-21ssh11:45:000
2749399186,7cyclictest0-21swapper/310:40:143
2749399186,7cyclictest0-21swapper/310:40:133
27490991814,3cyclictest0-21swapper/209:05:122
2749099181,15cyclictest0-21swapper/209:37:092
2749099180,4cyclictest0-21swapper/210:55:142
27488991817,1cyclictest0-21swapper/109:05:581
2748899180,1cyclictest0-21swapper/109:24:581
2748899180,15cyclictest0-21swapper/110:32:251
2748399185,7cyclictest20291-21munin-run09:10:000
27493991716,1cyclictest0-21swapper/308:00:143
27490991715,1cyclictest6994-21apache211:35:372
2749099170,1cyclictest0-21swapper/209:40:332
2748899171,1cyclictest20292-21diskmemload10:45:161
2748899171,1cyclictest141rcu_preempt11:40:241
2748899171,15cyclictest0-21swapper/108:12:051
2748399175,6cyclictest9368-21munin-run07:40:010
2748399174,7cyclictest2134-21munin-run10:55:000
24942170,1sleep02491-21sh09:56:270
2749399160,2cyclictest0-21swapper/308:35:143
2749099162,3cyclictest0-21swapper/212:39:122
2749099160,16cyclictest0-21swapper/210:31:392
2749099160,15cyclictest0-21swapper/209:30:512
2749099160,14cyclictest0-21swapper/211:01:582
2748899169,5cyclictest38150irq/131-snd_hda10:00:001
27488991616,0cyclictest0-21swapper/112:33:531
2748899160,1cyclictest20292-21diskmemload11:35:461
2748399167,8cyclictest530-21ssh12:30:130
2748399166,5cyclictest32478-21turbostat12:30:010
2749099157,7cyclictest6994-21apache211:47:182
27490991515,0cyclictest0-21swapper/212:13:532
27490991515,0cyclictest0-21swapper/210:28:432
27490991513,0cyclictest0-21swapper/211:28:362
2749099150,1cyclictest0-21swapper/210:47:362
2749099150,14cyclictest0-21swapper/209:47:502
27488991515,0cyclictest0-21swapper/111:54:431
27488991512,2cyclictest10189-21ssh09:41:471
2748399151,12cyclictest6434-21ntp_states09:20:200
2748399150,8cyclictest4634-21perf09:20:010
2748399150,7cyclictest0-21swapper/009:29:560
2748399150,1cyclictest141rcu_preempt10:00:160
2749399149,3cyclictest31382-21tail11:50:173
2749399145,1cyclictest28846-21latency_hist09:15:013
2749399145,1cyclictest28846-21latency_hist09:15:013
2749399143,9cyclictest28043-21/usr/sbin/munin11:10:143
27493991412,2cyclictest11096-21kworker/u8:310:30:483
27493991412,1cyclictest55650irq/127-enp2s0-12:25:133
27493991412,1cyclictest0-21swapper/312:01:253
27493991411,2cyclictest7434-21ssh12:15:013
27493991411,2cyclictest1814-21ssh09:55:563
27493991410,4cyclictest7020-21apache210:57:173
27490991411,2cyclictest16323-21tr10:05:132
27490991410,2cyclictest6043-21tr07:30:212
2749099140,14cyclictest0-21swapper/212:30:092
2749099140,10cyclictest0-21swapper/211:50:132
2748899143,10cyclictest8036-21ssh10:00:131
2748899143,10cyclictest1691-21ssh10:15:201
2748899142,6cyclictest12774-21timerwakeupswit11:00:231
2748899142,11cyclictest20292-21diskmemload10:10:371
27488991413,1cyclictest0-21swapper/112:19:281
2748899141,1cyclictest12027-21dpkg07:45:121
27488991411,2cyclictest19138-21grep10:25:271
27488991411,2cyclictest17007-21ssh12:40:001
2748899140,1cyclictest0-21swapper/112:10:451
2748399141,13cyclictest0-21swapper/012:00:500
2749399139,3cyclictest32075-21wc09:35:263
2749399139,3cyclictest28593-21ls11:30:003
2749399139,3cyclictest2353-21grep10:35:173
2749399139,3cyclictest19767-21ssh09:47:443
27493991311,2cyclictest0-21swapper/310:23:413
27493991310,2cyclictest9873-21sed12:35:183
27493991310,2cyclictest8353-21cron10:19:593
27493991310,2cyclictest7751-21cut12:15:113
27493991310,2cyclictest32322-21ls10:15:003
27493991310,2cyclictest29960-21grep09:15:173
27493991310,2cyclictest27544-21sort08:15:203
27493991310,2cyclictest25550-21sh09:50:423
27493991310,2cyclictest23885-21cron10:10:003
27493991310,2cyclictest18872-21cron11:05:003
27493991310,2cyclictest17387-21ssh12:20:183
27493991310,2cyclictest16560-21sed09:00:183
2749099139,3cyclictest1189-21perf08:30:002
2749099131,1cyclictest0-21swapper/210:10:432
27490991310,2cyclictest4958-21ntpq08:35:212
27490991310,2cyclictest30279-21wc09:15:182
27490991310,2cyclictest2175-21cron07:25:002
2749099130,9cyclictest0-21swapper/208:05:122
2749099130,12cyclictest0-21swapper/211:41:582
2749099130,0cyclictest0-21swapper/210:04:472
2748899139,3cyclictest2671-21tr07:25:141
2748899137,5cyclictest488-21ssh11:12:541
2748899136,6cyclictest26716-21taskset12:06:321
2748899133,4cyclictest225-21systemd-journal08:17:571
27488991310,2cyclictest30704-21tr09:35:151
27488991310,2cyclictest30377-21wc08:20:221
27488991310,2cyclictest22719-21tail09:10:271
27488991310,2cyclictest22719-21tail09:10:261
27488991310,2cyclictest14262-21grep09:25:181
2748899130,6cyclictest0-21swapper/110:36:021
2748899130,13cyclictest0-21swapper/111:32:151
2748399137,3cyclictest9146-21aten2_r7power_p12:35:140
27483991310,2cyclictest3642-21ls07:25:210
27483991310,2cyclictest26064-21wc10:10:250
27483991310,2cyclictest25127-21ls08:10:190
27483991310,2cyclictest19842-21grep08:00:180
27483991310,2cyclictest15543-21sendmail-msp09:00:000
27483991310,2cyclictest12322-21ssh11:20:140
27483991310,1cyclictest0-21swapper/009:33:300
2748399130,13cyclictest0-21swapper/009:44:130
2748399130,12cyclictest16388-21grep09:00:160
2749399129,2cyclictest772-21head07:20:163
2749399129,2cyclictest6491-21grep07:30:263
2749399129,2cyclictest3593-21chrt08:35:003
2749399129,2cyclictest24599-21ntpq12:05:203
2749399129,2cyclictest20173-21grep11:05:163
2749399129,2cyclictest17314-21tr10:25:143
2749399123,8cyclictest14619-21munin-node10:52:113
2749399123,5cyclictest13463-21apache_processe09:25:123
27493991210,1cyclictest27958-21tr07:10:133
2749399120,11cyclictest9009-21wc10:00:203
2749399120,11cyclictest4586-21tail11:15:163
2749099129,2cyclictest6907-21chrt10:38:092
2749099129,2cyclictest6746-21tr11:55:152
2749099129,2cyclictest6318-21tr08:40:142
2749099129,2cyclictest5583-21sed11:15:242
2749099129,2cyclictest2705-21wc08:30:212
2749099129,2cyclictest19681-21ssh12:02:352
2749099129,2cyclictest15070-21cut07:50:162
2749099129,2cyclictest14716-21wc08:55:222
2749099129,2cyclictest11254-21sed07:40:262
2749099122,10cyclictest0-21swapper/210:20:442
2749099121,10cyclictest0-21swapper/208:45:112
2749099120,12cyclictest0-21swapper/212:20:122
2749099120,12cyclictest0-21swapper/209:52:452
2749099120,12cyclictest0-21swapper/209:22:182
2749099120,11cyclictest0-21swapper/212:06:592
2748899129,2cyclictest26822-21tr10:50:141
2748899129,2cyclictest24176-21ssh12:25:011
2748899129,2cyclictest23287-21cut09:50:131
2748899129,2cyclictest23224-21tr09:30:221
2748899129,2cyclictest15513-21grep12:00:151
2748899128,3cyclictest7481-21tr07:35:131
2748899128,3cyclictest23127-21ssh11:25:451
2748899128,3cyclictest21284-21sed11:05:251
2748899128,3cyclictest14394-21sh12:00:001
2748899127,4cyclictest7020-21apache212:25:461
2748899123,7cyclictest4635-21/usr/sbin/munin11:15:161
2748899123,4cyclictest225-21systemd-journal08:44:551
2748899121,4cyclictest0-21swapper/110:40:141
2748899121,4cyclictest0-21swapper/110:40:141
2748899120,11cyclictest0-21swapper/109:15:521
2748399129,2cyclictest5514-21sed08:35:280
2748399129,2cyclictest29702-21grep11:10:270
2748399129,2cyclictest27703-21ls08:15:220
2748399129,2cyclictest20215-21grep11:05:160
2748399129,2cyclictest19663-21grep10:45:220
2748399129,2cyclictest19091-21tail10:25:270
2748399129,2cyclictest18105-21grep12:20:260
2748399129,2cyclictest10396-21grep07:40:170
2748399128,2cyclictest30676-21basename11:50:140
2748399122,5cyclictest14426-21perf12:00:000
2748399121,3cyclictest225-21systemd-journal10:36:430
27483991210,1cyclictest4946-21aten2_r7power_v07:30:130
2748399120,1cyclictest0-21swapper/009:11:410
2748399120,1cyclictest0-21swapper/009:11:400
2748399120,11cyclictest1084-21grep10:15:150
2749399118,2cyclictest9822-21ssh11:56:253
2749399118,2cyclictest9531-21ssh11:37:193
2749399118,2cyclictest8459-21taskset09:40:283
2749399118,2cyclictest30673-21df07:15:153
2749399118,2cyclictest29172-21ssh11:30:143
2749399118,2cyclictest24846-21ssh09:31:143
2749399118,2cyclictest22231-21grep11:45:133
2749399118,2cyclictest22026-21grep08:05:143
2749399118,2cyclictest20605-21sh10:45:413
2749399118,2cyclictest16221-21taskset07:51:333
2749399118,2cyclictest14372-21df11:40:153
2749399118,2cyclictest12344-21expr07:45:143
2749399118,2cyclictest10486-21chrt08:46:423
2749399117,3cyclictest10385-21grep07:40:173
2749399112,6cyclictest7255-21apt-get07:35:113
2749399111,9cyclictest537-21snmpd12:30:133
2749399111,3cyclictest0-21swapper/309:20:123
2749099118,2cyclictest7495-21expr07:35:122
2749099118,2cyclictest31318-21ssh09:55:012
2749099118,2cyclictest30165-21sed08:20:192
2749099118,2cyclictest28555-21if_enp0s31f607:10:162
2749099118,2cyclictest28335-21ssh09:14:462
2749099118,2cyclictest28335-21ssh09:14:462
2749099118,2cyclictest27499-21wc08:15:192
2749099118,2cyclictest26030-21ls08:14:592
2749099118,2cyclictest19893-21tail08:00:172
2749099118,2cyclictest16036-21tr09:00:152
2749099118,2cyclictest12268-21tr07:45:142
2749099118,2cyclictest1086-21sort10:15:152
2749099117,3cyclictest6994-21apache210:51:182
2748899119,1cyclictest3090-21grep08:30:261
2748899119,1cyclictest16330-21tr10:05:131
2748899118,2cyclictest4886-21ssh10:55:381
2748899118,2cyclictest3891-21tr08:35:141
2748899118,2cyclictest1757-21grep07:20:271
2748899118,2cyclictest17443-21grep07:55:161
2748899118,2cyclictest17241-21ssh09:45:451
2748899118,2cyclictest16037-21tr09:00:151
2748899118,2cyclictest14530-21ssh11:20:401
2748899118,2cyclictest13050-21chrt08:54:091
2748899118,2cyclictest10349-21grep08:45:251
2748899114,6cyclictest29842-21idleruntime-cro07:15:001
2748399119,1cyclictest31409-21interrupts09:35:190
2748399118,2cyclictest9978-21sed10:20:180
2748399118,2cyclictest9808-21ssh10:40:110
2748399118,2cyclictest9808-21ssh10:40:110
2748399118,2cyclictest9666-21cut08:45:190
2748399118,2cyclictest9085-21grep12:15:190
2748399118,2cyclictest6354-21ssh10:56:510
2748399118,2cyclictest5087-21ls11:15:200
2748399118,2cyclictest3898-21ssh12:12:270
2748399118,2cyclictest31982-21taskset07:17:160
2748399118,2cyclictest31636-21cpu08:25:140
2748399118,2cyclictest28453-21sh10:31:070
2748399118,2cyclictest27066-21ssh12:06:480
2748399118,2cyclictest26633-21ssh09:51:330
2748399118,2cyclictest23384-21ssh11:45:200
2748399118,2cyclictest1991-21df08:30:150
2748399118,2cyclictest1642-21cat07:20:260
2748399118,2cyclictest14703-21rm09:45:080
2748399118,2cyclictest12303-21grep07:45:140
2748399118,2cyclictest12044-21chrt08:50:200
2748399114,6cyclictest14619-21munin-node11:36:100
2748399110,10cyclictest0-21swapper/011:28:350
2749399108,1cyclictest30202-21needreboot08:20:193
2749399108,1cyclictest25379-21cat08:10:223
2749399108,1cyclictest17074-21cat07:55:143
2749399107,2cyclictest32697-21cut08:25:223
2749399107,2cyclictest2422-21cat07:25:123
2749399106,2cyclictest6238-21basename08:40:143
2749399102,5cyclictest18209-21aten2_r7power_e09:05:123
2749399100,9cyclictest11313-21cstates08:50:153
2749099108,1cyclictest31879-21taskset07:15:582
2749099108,1cyclictest10976-21taskset08:50:022
2749099107,2cyclictest4391-21chrt07:28:042
2749099107,2cyclictest19151-21ssh11:05:032
2748899108,1cyclictest19237-21tr08:00:141
2748899108,1cyclictest1273-21cat08:29:591
2748899108,1cyclictest116002sleep107:43:091
2748899107,2cyclictest6150-21mailstats07:30:221
2748899107,2cyclictest16141-21chrt07:50:301
2748399108,1cyclictest22444-21cat08:05:170
2748399108,1cyclictest17019-21ssh10:05:190
2748399107,2cyclictest6558-21expr08:40:150
2748399107,2cyclictest29208-21aten2_r7power_p08:20:140
2748399107,2cyclictest28989-21sh11:30:090
2748399107,2cyclictest28031-21expr07:10:140
2748399107,2cyclictest16155-21taskset07:50:410
2748399106,2cyclictest17444-21cut07:55:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional