You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 03:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Aug 02, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20819210765,37sleep00-21swapper/019:09:020
2058929351,38sleep30-21swapper/319:06:043
2060028976,9sleep20-21swapper/219:06:112
2061428467,13sleep10-21swapper/119:06:221
122952600,1sleep339-21ksoftirqd/323:24:293
275152520,0sleep20-21swapper/200:11:292
2098399282,25cyclictest0-21swapper/122:35:181
2098699270,26cyclictest0-21swapper/222:47:472
20986992524,1cyclictest0-21swapper/221:46:452
92502230,1sleep00-21swapper/000:40:000
2098399230,1cyclictest0-21swapper/120:45:151
20983992221,1cyclictest13564-21diskmemload00:13:261
2098899191,17cyclictest5579-21ssh00:18:083
20986991918,1cyclictest20975-21cyclictest21:20:232
2098699190,19cyclictest0-21swapper/223:46:442
2098699190,19cyclictest0-21swapper/223:46:442
2098699190,19cyclictest0-21swapper/200:24:072
2098699180,1cyclictest0-21swapper/200:01:412
20983991814,2cyclictest13564-21diskmemload22:15:381
20981991816,2cyclictest0-21swapper/022:29:360
20986991715,1cyclictest0-21swapper/200:39:162
2098699170,16cyclictest0-21swapper/223:04:572
2098699170,0cyclictest0-21swapper/221:41:422
20988991615,1cyclictest0-21swapper/300:12:113
20988991614,2cyclictest39-21ksoftirqd/321:43:543
2098899160,15cyclictest0-21swapper/322:29:123
2098699166,4cyclictest452-21systemd-logind00:08:322
2098699160,5cyclictest0-21swapper/219:45:132
2098699160,2cyclictest0-21swapper/223:58:082
2098699160,16cyclictest0-21swapper/223:08:062
2098699160,16cyclictest0-21swapper/222:21:352
2098399160,1cyclictest0-21swapper/121:50:211
2098199165,9cyclictest19023-21munin-run22:50:000
2098199160,2cyclictest0-21swapper/022:32:290
2098899159,5cyclictest8023-21sh22:42:583
20988991511,2cyclictest13045-21grep22:45:263
2098699159,2cyclictest0-21swapper/222:30:162
20986991510,3cyclictest25557-21cat22:15:002
2098699150,15cyclictest0-21swapper/223:52:162
2098699150,15cyclictest0-21swapper/223:52:162
2098699150,15cyclictest0-21swapper/223:37:432
20983991515,0cyclictest0-21swapper/123:38:471
2098399150,15cyclictest0-21swapper/122:40:591
2098399150,14cyclictest0-21swapper/121:25:211
2098199152,6cyclictest273-21systemd-udevd22:10:040
2098199151,12cyclictest32052-21grep20:40:160
153462150,0sleep00-21swapper/000:24:130
2098899149,3cyclictest2621-21grep19:35:263
20988991411,2cyclictest14072-21grep23:05:273
2098899140,9cyclictest16882-21ssh00:25:133
20986991413,1cyclictest55650irq/127-enp2s0-22:50:052
20986991411,2cyclictest4541-21cat22:40:262
20986991410,2cyclictest25845-21dpkg19:20:002
2098699140,14cyclictest0-21swapper/221:19:062
2098699140,13cyclictest0-21swapper/222:26:362
2098399149,3cyclictest5720-21grep23:00:261
2098399144,9cyclictest26004-21munin-node21:35:041
2098399141,6cyclictest20487-21/usr/sbin/munin23:10:201
2098399141,12cyclictest31952-21ssh22:57:021
2098399141,12cyclictest25698-21cron19:20:001
20983991411,2cyclictest11556-21cut21:05:141
2098399140,14cyclictest0-21swapper/122:09:411
2098199144,9cyclictest26004-21munin-node22:41:040
20981991413,1cyclictest0-21swapper/022:55:560
2098199141,10cyclictest8145-21aten2_r7power_c19:50:140
20981991410,2cyclictest11106-21grep19:55:160
2098199140,9cyclictest710-21idleruntime-cro19:35:000
2098199140,3cyclictest0-21swapper/021:35:340
134912140,2chrt13480-21apt-get23:25:122
2098899139,3cyclictest27940-21tail20:30:183
2098899139,3cyclictest27140-21grep22:35:163
2098899131,2cyclictest0-21swapper/323:32:353
20988991310,2cyclictest31191-21grep20:35:243
20988991310,2cyclictest30440-21tr23:35:153
20988991310,2cyclictest28765-21grep22:55:173
20988991310,2cyclictest27229-21sed00:30:263
2098899130,12cyclictest0-21swapper/322:06:073
2098699139,3cyclictest14999-21munin-run20:05:012
2098699139,2cyclictest3089-21perf19:40:002
2098699137,2cyclictest225-21systemd-journal20:10:002
2098699131,9cyclictest0-21swapper/222:55:162
20986991312,1cyclictest55850irq/129-enp2s0-23:40:422
20986991310,2cyclictest651-21grep20:40:222
20986991310,2cyclictest11449-21chrt21:27:232
2098699130,0cyclictest0-21swapper/221:30:132
2098399139,3cyclictest14077-21sh23:25:141
2098399139,3cyclictest12961-21cron23:25:001
2098399139,3cyclictest0-21swapper/119:35:201
2098399139,2cyclictest31995-21cat19:30:171
20983991312,1cyclictest13564-21diskmemload00:26:291
20983991310,2cyclictest8839-21head19:50:181
20983991310,2cyclictest6968-21tr19:45:211
20983991310,2cyclictest32076-21ssh21:59:221
20983991310,2cyclictest26423-21cat22:35:011
20983991310,2cyclictest16088-21dpkg21:50:001
20983991310,2cyclictest1555-21grep21:40:251
20983991310,2cyclictest10462-21grep00:20:261
2098399130,1cyclictest0-21swapper/121:11:321
2098399130,12cyclictest17097-21ssh22:48:311
2098399130,0cyclictest0-21swapper/100:39:481
2098199139,3cyclictest11684-21cat23:05:000
2098199139,2cyclictest25711-21turbostat.cron19:20:000
2098199138,3cyclictest7277-21cstates21:25:140
2098199136,6cyclictest7454-21munin-run21:45:000
2098199134,8cyclictest13955-21perf23:45:000
2098199134,5cyclictest23029-21apt-get23:50:130
2098199134,5cyclictest23029-21apt-get23:50:130
2098199132,10cyclictest14031-21/usr/sbin/munin21:10:260
20981991310,2cyclictest7437-21sed20:55:160
20981991310,2cyclictest7392-21tail19:45:250
20981991310,2cyclictest20875-21if_enp2s020:15:200
2098199130,13cyclictest0-21swapper/023:10:510
2098899129,2cyclictest9858-21cut21:00:163
2098899129,2cyclictest3088-21grep22:20:193
2098899129,2cyclictest2797-21sh21:22:193
2098899129,2cyclictest25424-21grep20:25:173
2098899129,2cyclictest23773-21wget21:35:153
2098899129,2cyclictest20826-21tr23:10:153
2098899129,2cyclictest18532-21ssh22:30:143
2098899129,2cyclictest17897-21grep21:50:193
2098899129,2cyclictest16340-21ls00:24:593
2098899128,3cyclictest8147-21sed00:00:193
2098899128,3cyclictest6262-21tr23:40:153
2098899128,3cyclictest32617-21cut23:55:213
2098899128,3cyclictest19530-21sed22:10:263
2098899128,3cyclictest13548-21cron21:10:003
2098899128,2cyclictest22681-21grep20:20:153
2098899121,1cyclictest0-21swapper/322:15:523
2098899121,10cyclictest0-21swapper/322:52:463
20988991210,1cyclictest28389-21sleep19:25:003
2098899120,11cyclictest0-21swapper/321:32:563
2098699129,2cyclictest9363-21tr21:00:132
2098699129,2cyclictest8043-21cat00:20:012
2098699129,2cyclictest7401-21grep20:55:162
2098699129,2cyclictest29401-21grep19:25:162
2098699129,2cyclictest26533-21rm21:55:242
2098699129,2cyclictest22357-21wc23:10:262
2098699129,2cyclictest21877-21cron21:15:002
2098699129,2cyclictest17310-21cat22:10:002
2098699129,2cyclictest16943-21grep21:50:132
2098699128,3cyclictest55850irq/129-enp2s0-23:23:082
2098699128,3cyclictest5051-21grep20:50:172
2098699128,3cyclictest10621-21grep19:55:142
2098699120,12cyclictest0-21swapper/221:36:062
2098399129,2cyclictest27550-21cat22:55:011
2098399128,3cyclictest19651-21ssh00:07:051
2098399128,3cyclictest1162-21grep20:40:281
2098399121,10cyclictest0-21swapper/100:18:281
2098399120,3cyclictest225-21systemd-journal23:10:001
2098399120,1cyclictest0-21swapper/123:15:151
2098199129,2cyclictest9679-21basename22:25:000
2098199129,2cyclictest6092-21tail23:20:200
2098199129,2cyclictest27969-21sed20:30:180
2098199129,2cyclictest26475-21grep19:20:140
2098199129,2cyclictest24009-21sed20:20:250
2098199129,2cyclictest16968-21cat21:30:230
2098199129,2cyclictest16915-21tail20:05:260
2098199129,2cyclictest14249-21grep23:25:160
2098199129,2cyclictest10738-21head22:05:220
2098199129,2cyclictest10068-21tail21:45:280
2098199128,3cyclictest26863-21cut22:35:150
2098199128,2cyclictest17760-21basename20:10:180
2098199128,2cyclictest17760-21basename20:10:170
2098199127,4cyclictest30481-21tr23:35:150
2098199127,3cyclictest30894-21wc19:30:000
2098899118,2cyclictest9347-21ssh21:25:443
2098899118,2cyclictest8779-21grep21:45:163
2098899118,2cyclictest5108-21ssh00:36:533
2098899118,2cyclictest3743-21ssh23:00:103
2098899118,2cyclictest31526-21ssh23:15:413
2098899118,2cyclictest27556-21ssh21:17:523
2098899118,2cyclictest26018-21ssh21:55:193
2098899118,2cyclictest25432-21chrt19:16:423
2098899118,2cyclictest24043-21ssh23:50:183
2098899118,2cyclictest24043-21ssh23:50:183
2098899118,2cyclictest22701-21ls19:10:283
2098899118,2cyclictest21469-21ssh23:49:223
2098899118,2cyclictest21469-21ssh23:49:223
2098899118,2cyclictest20259-21cut20:15:173
2098899118,2cyclictest19267-21tail20:10:293
2098899118,2cyclictest19267-21tail20:10:293
2098899118,2cyclictest18734-21ssh00:06:223
2098899118,2cyclictest17041-21ssh21:11:123
2098899118,2cyclictest16354-21grep20:05:203
2098899118,2cyclictest15725-21ssh23:25:373
2098899118,2cyclictest11471-21wc19:55:183
2098899118,1cyclictest39-21ksoftirqd/322:00:153
2098899117,3cyclictest6405-21cron20:54:593
2098899113,4cyclictest225-21systemd-journal19:45:053
2098699118,2cyclictest4782-21wc19:40:272
2098699118,2cyclictest30285-21ssh23:15:212
2098699118,2cyclictest28125-21grep22:35:232
2098699118,2cyclictest27188-21basename20:30:122
2098699118,2cyclictest27047-21grep19:20:192
2098699118,2cyclictest26871-21tr00:30:222
2098699118,2cyclictest26773-21taskset20:28:412
2098699118,2cyclictest26188-21tr22:15:152
2098699118,2cyclictest25497-21rm23:31:482
2098699118,2cyclictest23129-21grep20:20:172
2098699118,2cyclictest22900-21sed19:10:282
2098699118,2cyclictest1025-21ssh22:00:132
2098699113,4cyclictest0-21swapper/200:25:142
2098699112,5cyclictest225-21systemd-journal20:20:002
2098699111,6cyclictest225-21systemd-journal20:49:592
2098399119,1cyclictest0-21swapper/120:05:001
2098399118,2cyclictest8617-21awk00:00:231
2098399118,2cyclictest8288-21ssh23:40:351
2098399118,2cyclictest6099-21ssh22:22:111
2098399118,2cyclictest5325-21chrt19:43:161
2098399118,2cyclictest28995-21taskset20:30:591
2098399118,2cyclictest27029-21tail19:20:181
2098399118,2cyclictest24457-21grep21:15:291
2098399118,2cyclictest23339-21systemctl23:30:231
2098399118,2cyclictest21986-21sed19:10:191
2098399118,2cyclictest19339-21grep20:10:291
2098399118,2cyclictest19339-21grep20:10:291
2098399118,2cyclictest19024-21head22:10:221
2098399118,2cyclictest1853-21ssh22:00:171
2098399118,2cyclictest17061-21cat21:30:251
2098399118,2cyclictest14318-21ssh23:45:081
2098399118,2cyclictest14318-21ssh23:45:081
2098399118,2cyclictest10758-21grep21:00:261
2098399118,2cyclictest10536-21cat22:25:151
2098399117,3cyclictest28673-21tr19:25:131
2098199118,2cyclictest9470-21ssh00:00:350
2098199118,2cyclictest4898-21cat19:40:270
2098199118,2cyclictest4598-21tr20:50:150
2098199118,2cyclictest4214-21ssh23:58:020
2098199118,2cyclictest32646-21basename00:15:140
2098199118,2cyclictest3079-21dump-pmu-power19:40:000
2098199118,2cyclictest28156-21rm23:33:520
2098199118,2cyclictest2641-21ssh21:22:110
2098199118,2cyclictest25752-21chrt22:15:060
2098199118,2cyclictest25143-21cut21:55:150
2098199118,2cyclictest25108-21sh00:30:040
2098199118,2cyclictest24768-21sh21:15:420
2098199118,2cyclictest24585-21expr00:10:140
2098199118,2cyclictest24166-21ssh21:54:440
2098199118,2cyclictest18713-21ssh00:06:210
2098199118,2cyclictest15381-21ssh23:06:220
2098199118,2cyclictest15308-21head23:45:180
2098199118,2cyclictest15308-21head23:45:180
2098199117,3cyclictest17697-21grep00:25:160
2098199117,2cyclictest954-21cat22:00:110
2098199117,2cyclictest4625-21cron23:20:000
2098199117,2cyclictest24918-21grep20:25:140
2098199116,4cyclictest20334-21head22:50:170
2098899108,1cyclictest6655-21cat20:55:113
2098899108,1cyclictest32018-21grep20:40:163
2098899108,1cyclictest14901-21taskset20:04:053
2098899107,2cyclictest4093-21grep19:40:183
2098899107,2cyclictest28758-21cut19:25:133
2098699108,1cyclictest8070-21sshd19:50:052
2098699108,1cyclictest32067-21grep19:30:172
2098699108,1cyclictest18302-21cut20:10:192
2098699108,1cyclictest18302-21cut20:10:192
2098699108,1cyclictest11417-21taskset21:05:092
2098699107,2cyclictest31099-21cat20:35:252
2098399108,1cyclictest817-21expr23:55:271
2098399108,1cyclictest4286-21acpi20:50:131
2098399108,1cyclictest2799-21ssh21:22:191
2098399108,1cyclictest27118-21cat00:30:261
2098399108,1cyclictest22961-21ssh23:50:111
2098399108,1cyclictest22961-21ssh23:50:111
2098399108,1cyclictest22597-21ls20:20:141
2098399108,1cyclictest15396-21egrep20:05:141
2098399108,1cyclictest11044-21sed19:55:151
2098399107,2cyclictest29781-21wc20:35:131
2098399107,2cyclictest25346-21sed20:25:151
2098399107,2cyclictest21209-21cut20:15:241
2098199107,2cyclictest30908-21find20:35:200
2098199107,2cyclictest23218-21chrt19:13:510
2098199107,2cyclictest11770-21expr21:05:160
2098199106,2cyclictest13474-21grep20:00:160
2098199100,6cyclictest3815-21sshd20:47:140
209889997,1cyclictest31827-21basename19:30:163
209889997,1cyclictest2466-21sed20:45:173
209889996,2cyclictest8559-21cat19:50:153
209839997,1cyclictest7765-21needreboot20:55:191
209819997,1cyclictest11073-21taskset21:03:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional