You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-15 - 01:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Aug 14, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
98782238162,25sleep00-21swapper/007:05:560
100202237164,24sleep30-21swapper/307:07:503
99912204167,24sleep10-21swapper/107:07:261
101932202165,25sleep20-21swapper/207:10:012
150302560,2sleep10-21swapper/112:10:251
106362560,3sleep00-21swapper/008:20:210
83662490,2sleep011-21rcuc/011:55:230
1029199441,30cyclictest21541-40dnf11:14:353
1027399421,32cyclictest10817-21/usr/sbin/munin10:50:150
1027399411,10cyclictest135632proc_pri09:40:220
1027399396,23cyclictest16877-21cat09:50:120
1027399361,4cyclictest22652-21grep10:00:240
10273993611,12cyclictest9972-21munin-run09:35:010
19822340,2sleep20-21swapper/210:30:012
19822340,2sleep20-21swapper/210:30:012
1027399347,12cyclictest30734-21perf07:55:010
1027399341,6cyclictest20580-21users09:55:270
1027399341,3cyclictest11225-21netstat09:35:200
1029199331,19cyclictest0-21swapper/307:13:273
1029199331,11cyclictest0-21swapper/312:09:393
1027399331,5cyclictest14982-21latency_hist11:00:020
10273993312,18cyclictest17877-21tr07:25:160
1027399330,26cyclictest0-21swapper/007:40:190
45652320,4sleep341-21ksoftirqd/309:20:193
1029199321,23cyclictest0-21swapper/308:38:033
1027399329,20cyclictest15898-21hwlatdetect11:00:180
1027399328,21cyclictest22430-21/usr/sbin/munin12:30:140
1027399325,3cyclictest4839-21/usr/sbin/munin11:50:260
1027399325,3cyclictest24974-21/usr/sbin/munin08:55:220
1027399324,3cyclictest87350irq/126-enp1s0-11:25:200
1027399323,3cyclictest87550irq/128-enp1s0-09:20:210
1027399322,5cyclictest29668-21smart_sda11:30:230
1027399321,5cyclictest14596-21/usr/sbin/munin09:45:270
1027399321,21cyclictest18035-21/usr/sbin/munin12:20:190
10273993212,17cyclictest21816-21/usr/sbin/munin11:15:120
1027399317,21cyclictest24010-21/usr/sbin/munin11:20:240
1027399317,20cyclictest6722-21/usr/sbin/munin09:25:220
1027399313,24cyclictest31571-21/usr/sbin/munin09:10:180
1027399311,6cyclictest22768-21/usr/sbin/munin08:50:240
1027399311,5cyclictest31958-21latency_hist10:25:010
1027399311,3cyclictest28881-21/usr/sbin/munin10:15:210
1029199300,23cyclictest0-21swapper/309:13:153
10285993022,3cyclictest0-21swapper/212:01:252
1027399309,18cyclictest11783-21/usr/sbin/munin08:25:190
1027399308,19cyclictest16518-21sort07:20:190
1027399304,12cyclictest15628-21perf12:15:010
1027399302,25cyclictest13411-21munin-run12:10:010
1027399302,25cyclictest11229-21awk12:05:010
1027399301,5cyclictest25595-21/usr/sbin/munin10:10:220
1027399301,4cyclictest25888-21smart_sda12:35:250
1027399301,4cyclictest14780-21/usr/sbin/munin08:30:190
1027399301,3cyclictest909-21sssd_nss08:45:230
1027399301,3cyclictest862-21cpuspeed_turbos08:00:120
1027399301,3cyclictest862-21cpuspeed_turbos08:00:120
1027399301,3cyclictest30030-21sed09:05:140
1027399301,3cyclictest20611-21gltestperf07:30:180
1027399301,3cyclictest16521-21/usr/sbin/munin08:35:160
10273993010,17cyclictest3377-21dump-pmu-power09:20:010
10273993010,17cyclictest30945-21/usr/sbin/munin07:55:270
10273993010,17cyclictest12504-21date07:15:010
1029199290,21cyclictest0-21swapper/307:51:363
1029199290,11cyclictest0-21swapper/307:33:593
1028599295,7cyclictest21541-40dnf11:14:382
1027399299,17cyclictest8301-21missed_timers08:15:200
1027399299,17cyclictest13372-21/usr/sbin/munin07:15:240
1027399298,18cyclictest18858-21fschecks_time08:40:160
1027399296,3cyclictest7301-21cat10:40:180
1027399293,3cyclictest17505-21/usr/sbin/munin11:05:110
1027399292,4cyclictest28527-21munin-run07:50:010
1027399292,3cyclictest3319-21hddtemp_smartct11:45:160
1027399291,5cyclictest2026-21/usr/sbin/munin10:30:130
1027399291,4cyclictest27510-21df_inode09:00:140
1027399291,4cyclictest21334-21/usr/sbin/munin12:25:220
1027399291,3cyclictest6786-21/usr/sbin/munin08:10:270
1027399291,3cyclictest5033-21/usr/sbin/munin10:35:180
1027399291,3cyclictest30668-21/usr/sbin/munin11:35:120
1027399291,3cyclictest23484-21/usr/sbin/munin10:05:120
1029199281,7cyclictest0-21swapper/307:55:153
1029199281,18cyclictest0-21swapper/307:49:003
1029199281,12cyclictest904-21gdbus10:14:233
1027399289,16cyclictest19827-21expr11:10:120
1027399288,18cyclictest22186-21cpu07:35:130
1027399281,4cyclictest1517-21/usr/sbin/munin11:40:210
1027399281,3cyclictest32253-21/usr/sbin/munin10:25:120
1027399281,3cyclictest32253-21/usr/sbin/munin10:25:110
1027399281,3cyclictest16021-21cstates12:15:130
1027399280,13cyclictest10605-21idleruntime-cro10:50:010
1029199271,11cyclictest0-21swapper/307:36:513
1029199270,12cyclictest0-21swapper/311:08:353
1029199270,11cyclictest0-21swapper/310:07:393
1028599271,11cyclictest1760-21diskstats09:15:132
1028199271,24cyclictest0-21swapper/112:05:201
1027399271,4cyclictest3007-21/usr/sbin/munin08:05:130
75092260,1sleep30-21swapper/310:40:223
1029199267,2cyclictest0-21swapper/308:17:513
1029199261,8cyclictest495-21jbd2/dm-0-809:35:393
10285992623,2cyclictest0-21swapper/210:21:372
1028199262,15cyclictest87550irq/128-enp1s0-10:00:221
10281992613,12cyclictest1758-21kworker/u8:111:00:211
1029199258,3cyclictest0-21swapper/312:00:433
1029199256,9cyclictest2837-21irqstats10:30:173
1029199255,3cyclictest0-21swapper/308:44:033
10291992510,3cyclictest0-21swapper/311:20:233
1029199250,8cyclictest0-21swapper/308:59:313
1029199250,18cyclictest0-21swapper/308:04:523
1029199250,18cyclictest0-21swapper/308:04:513
1029199250,11cyclictest0-21swapper/310:25:193
1029199250,11cyclictest0-21swapper/310:25:193
1029199250,10cyclictest0-21swapper/311:57:363
1029199250,10cyclictest0-21swapper/311:39:073
1029199250,10cyclictest0-21swapper/310:56:193
1029199241,9cyclictest0-21swapper/307:22:593
1029199241,7cyclictest0-21swapper/312:36:553
10291992414,3cyclictest7014-40dnf08:13:233
1029199240,9cyclictest0-21swapper/309:40:193
1029199240,9cyclictest0-21swapper/308:47:393
1029199240,8cyclictest0-21swapper/312:32:553
1029199240,17cyclictest0-21swapper/308:23:473
1029199240,10cyclictest0-21swapper/308:32:513
1028599241,10cyclictest12455-21perf07:15:012
1029199239,4cyclictest31340-21diskmemload09:33:473
1029199236,3cyclictest0-21swapper/309:02:113
1029199231,12cyclictest0-21swapper/309:46:353
1029199230,9cyclictest0-21swapper/308:51:393
1029199230,8cyclictest0-21swapper/312:10:313
1029199230,12cyclictest0-21swapper/307:25:353
1029199230,11cyclictest0-21swapper/311:02:113
1029199230,10cyclictest0-21swapper/309:15:073
10285992311,3cyclictest20977-21awk10:00:002
10281992314,2cyclictest3509-21latency11:45:181
1028199231,19cyclictest8300-21memory08:15:201
1028199231,16cyclictest0-21swapper/112:33:071
1028199231,15cyclictest0-21swapper/110:18:391
1029199228,3cyclictest0-21swapper/311:30:073
1029199227,3cyclictest0-21swapper/310:52:233
1029199226,3cyclictest0-21swapper/309:27:593
1029199226,2cyclictest0-21swapper/310:01:523
1029199225,3cyclictest0-21swapper/312:18:553
1029199221,9cyclictest0-21swapper/308:26:393
1029199221,7cyclictest0-21swapper/311:50:523
10291992214,3cyclictest0-21swapper/309:50:243
10291992211,2cyclictest0-21swapper/310:38:233
1029199221,10cyclictest0-21swapper/307:41:453
1029199220,9cyclictest0-21swapper/310:24:473
1028599221,5cyclictest0-21swapper/210:41:062
1028199229,11cyclictest1758-21kworker/u8:110:49:051
1028199221,8cyclictest0-21swapper/111:55:111
10281992217,3cyclictest7097-21smartctl10:40:161
1028199221,4cyclictest28753-21diskstats11:30:141
1028199221,3cyclictest7162-21perf08:15:011
1028199221,18cyclictest4788-21seq08:08:011
1028199221,18cyclictest1025-21awk10:25:221
1028199221,18cyclictest1025-21awk10:25:221
1028199221,11cyclictest0-21swapper/111:41:571
1028199220,5cyclictest0-21swapper/109:49:071
1029199217,3cyclictest0-21swapper/310:19:313
1029199215,3cyclictest0-21swapper/312:24:553
1029199214,3cyclictest0-21swapper/309:09:453
1029199213,2cyclictest0-21swapper/312:28:073
1029199211,8cyclictest1233-21/usr/sbin/munin11:40:173
1028599219,5cyclictest0-21swapper/210:00:242
1028599219,4cyclictest12795-21awk10:55:012
1028199219,4cyclictest0-21swapper/112:35:351
10281992116,3cyclictest24255-21cstates11:20:121
1028199211,14cyclictest0-21swapper/112:15:031
1028199211,14cyclictest0-21swapper/111:15:101
1028199211,11cyclictest21793-21cat11:15:011
1028199211,11cyclictest0-21swapper/111:40:011
1028199211,11cyclictest0-21swapper/110:56:561
1028199211,10cyclictest0-21swapper/111:51:051
1028199211,10cyclictest0-21swapper/109:32:421
1028199210,17cyclictest23-21ksoftirqd/110:13:281
6082200,1sleep20-21swapper/211:40:122
1029199206,3cyclictest0-21swapper/311:25:473
1029199206,2cyclictest0-21swapper/309:58:443
1029199205,4cyclictest14138-21cat07:15:173
1029199205,3cyclictest0-21swapper/311:19:593
1029199202,10cyclictest10772-21latency_hist10:50:023
1028599201,4cyclictest12417-21grep09:40:112
10285992014,3cyclictest6458-21sed09:25:192
1028199208,3cyclictest13984-21users09:40:251
1028199206,3cyclictest0-21swapper/110:20:421
1028199204,9cyclictest21-21rcuc/107:50:111
10281992016,2cyclictest21-21rcuc/109:00:011
10281992015,3cyclictest17646-21grep08:35:271
1028199201,17cyclictest0-21swapper/109:55:181
1028199201,17cyclictest0-21swapper/109:20:201
1028199201,14cyclictest0-21swapper/111:25:431
1028199201,14cyclictest0-21swapper/111:06:471
1028199201,14cyclictest0-21swapper/110:34:541
1028199201,13cyclictest0-21swapper/110:51:451
1028199201,11cyclictest27876-21cat07:45:251
1028199201,11cyclictest1853-21ntp_states08:00:221
1028199201,11cyclictest1853-21ntp_states08:00:221
1028199201,11cyclictest0-21swapper/110:35:101
1028199201,10cyclictest0-21swapper/112:00:301
1028199201,10cyclictest0-21swapper/109:18:331
1028199200,6cyclictest0-21swapper/109:28:521
1028199200,5cyclictest0-21swapper/112:21:421
99402191,2sleep20-21swapper/209:34:482
1029199193,3cyclictest0-21swapper/308:08:313
1028599196,3cyclictest32530-21cat09:10:202
1028599193,3cyclictest24462-21ntp_states10:05:212
1028599193,3cyclictest0-21swapper/208:09:192
1028599193,10cyclictest7020-21chrt08:13:252
10285991913,3cyclictest5152-21grep09:20:272
1028199191,8cyclictest31831-21expr09:10:131
1028199191,17cyclictest0-21swapper/110:08:051
1028199191,17cyclictest0-21swapper/109:06:081
1028199191,17cyclictest0-21swapper/109:01:301
1028199191,16cyclictest21-21rcuc/109:35:121
1028199191,16cyclictest0-21swapper/108:20:171
1028199191,11cyclictest0-21swapper/112:28:231
1028199191,11cyclictest0-21swapper/107:20:151
1028199190,3cyclictest16526-21kworker/u8:009:53:051
180202180,4sleep123-21ksoftirqd/107:25:161
147452181,2sleep20-21swapper/210:57:072
1029199182,3cyclictest0-21swapper/311:45:033
1028599185,4cyclictest23423-21cut12:30:192
1028599185,4cyclictest12050irq/123-ahci[0008:45:012
1028599185,3cyclictest25967-21cut08:55:242
1028599181,7cyclictest15334-21crond07:20:012
1028599181,6cyclictest3454-21irqstats11:45:182
1028599181,6cyclictest0-21swapper/210:10:422
1028599181,6cyclictest0-21swapper/209:45:392
1028599181,4cyclictest0-21swapper/212:09:532
1028599181,3cyclictest0-21swapper/211:08:272
1028199187,3cyclictest25943-21users07:40:261
10281991814,3cyclictest0-21swapper/108:46:031
10281991814,3cyclictest0-21swapper/107:10:161
1028199181,3cyclictest31280-21df07:55:141
1028199181,16cyclictest0-21swapper/108:26:471
1028199181,16cyclictest0-21swapper/107:39:141
1028199181,16cyclictest0-21swapper/107:16:251
1028199181,15cyclictest0-21swapper/108:42:151
1028199180,17cyclictest0-21swapper/108:30:561
1028199180,16cyclictest0-21swapper/107:32:141
1028599174,4cyclictest21895-21cpuspeed_turbos11:15:122
1028599173,2cyclictest87550irq/128-enp1s0-08:35:172
1028599172,6cyclictest31275-21hddtemp_smartct11:35:162
1028599172,3cyclictest87350irq/126-enp1s0-12:25:262
1028599171,8cyclictest15602-21crond12:15:012
1028599171,7cyclictest27456-21sed07:45:192
1028599171,7cyclictest23671-21sed07:35:272
1028599171,7cyclictest15996-21cat08:35:012
1028599171,7cyclictest0-21swapper/207:30:182
1028599171,6cyclictest31352-21perf09:10:012
1028599171,6cyclictest18382-21diskstats12:20:152
1028599171,6cyclictest0-21swapper/211:00:252
1028599171,5cyclictest0-21swapper/209:35:222
1028599171,5cyclictest0-21swapper/208:20:152
1028599171,3cyclictest0-21swapper/211:57:502
1028599171,3cyclictest0-21swapper/211:28:362
1028599171,1cyclictest0-21swapper/208:15:172
1028599171,15cyclictest0-21swapper/210:19:332
1028599171,13cyclictest24934-21wc11:20:192
10285991710,4cyclictest8949-21df10:45:132
10285991710,4cyclictest30323-21unixbench_singl07:50:242
1028199174,11cyclictest23930-21proc_pri08:50:221
1028599169,5cyclictest13288-21sed08:25:252
1028599169,4cyclictest28508-21tr11:30:122
1028599169,4cyclictest24688-21df07:40:122
1028599169,4cyclictest16477-21grep12:15:162
1028599166,3cyclictest87350irq/126-enp1s0-10:30:232
1028599161,7cyclictest30945-21/usr/sbin/munin07:55:222
10285991612,3cyclictest0-21swapper/209:54:162
10285991612,3cyclictest0-21swapper/209:02:122
10285991612,3cyclictest0-21swapper/208:51:252
1028599161,14cyclictest0-21swapper/207:27:082
1028599161,14cyclictest0-21swapper/207:23:342
10285991611,3cyclictest4908-21sed10:35:162
10285991611,3cyclictest2862-21wc08:05:012
10285991611,3cyclictest2862-21wc08:05:012
1028599158,4cyclictest5946-21pmu-power11:50:222
1028599151,8cyclictest0-21swapper/212:35:142
10285991510,3cyclictest22159-21users08:45:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional