You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 04:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Aug 13, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287362238163,24sleep20-21swapper/219:05:472
288022235165,57sleep30-21swapper/319:06:413
290112228160,57sleep00-21swapper/019:09:250
289092219191,19sleep10-21swapper/119:08:041
29154999585,8cyclictest17848-21diskmemload23:42:492
178522770,7sleep230-21rcuc/221:10:012
29154996764,2cyclictest0-21swapper/221:18:572
155312660,4sleep339-21rcuc/323:30:153
29154996562,2cyclictest0-21swapper/221:57:222
4892610,3sleep32916099cyclictest19:15:173
302782560,3sleep330280-21missed_timers19:10:183
84412550,2sleep121-21rcuc/120:45:241
308962540,3sleep341-21ksoftirqd/300:05:203
82542510,1sleep30-21swapper/320:45:213
12592500,4sleep32916099cyclictest22:57:253
2914599329,11cyclictest8745-21awk00:30:000
2914599325,24cyclictest18075-21/usr/sbin/munin21:10:200
262552320,4sleep339-21rcuc/320:15:113
2914599314,9cyclictest5680-21cstates21:55:130
29145993117,9cyclictest2970-40dnf23:00:250
2915499301,19cyclictest0-21swapper/223:27:172
2914599304,14cyclictest3035-21sed00:15:160
2914599301,14cyclictest12975-21sort19:45:010
2915499291,12cyclictest0-21swapper/222:10:502
97632280,3sleep011-21rcuc/022:05:000
2915499281,13cyclictest0-21swapper/222:55:172
2915499281,12cyclictest0-21swapper/221:45:582
2914599286,10cyclictest10304-21awk23:20:010
2914599285,19cyclictest18470-21munin-run22:25:010
2914599285,13cyclictest22505-21cpu21:20:120
2914599285,13cyclictest22505-21cpu21:20:120
2914599284,9cyclictest25064-21perf22:40:020
2914599284,16cyclictest87350irq/126-enp1s0-22:25:250
2914599284,16cyclictest87350irq/126-enp1s0-22:25:250
2914599272,17cyclictest0-21swapper/023:10:250
2914599271,13cyclictest5904-21munin-run23:10:020
2915499261,16cyclictest0-21swapper/221:50:182
29150992612,12cyclictest29122-21kworker/u8:223:10:071
2914599262,9cyclictest15953-21cat21:05:100
29145992613,4cyclictest17108-21/usr/sbin/munin23:35:110
2914599261,15cyclictest7079-21/usr/sbin/munin20:45:150
2915499259,10cyclictest0-21swapper/200:04:382
2915499251,16cyclictest0-21swapper/223:20:252
2915099251,5cyclictest0-21swapper/123:50:251
2915099251,16cyclictest13769-21ntpq23:25:211
2914599257,15cyclictest0-21swapper/022:55:020
2914599256,12cyclictest3199-21latency_hist21:50:010
2914599256,10cyclictest0-21swapper/020:40:220
29145992515,6cyclictest694-21smart_sda22:55:230
2914599250,16cyclictest22568-21ntp_states23:45:190
2915499248,8cyclictest0-21swapper/219:43:172
2915499241,9cyclictest0-21swapper/219:27:172
2915499241,13cyclictest29468-21perf22:50:012
2915099242,17cyclictest0-21swapper/121:37:041
2915099241,6cyclictest0-21swapper/123:31:251
2915099241,4cyclictest17229-21munin-run19:55:001
2915099241,16cyclictest27360-21sort22:45:011
2915099240,18cyclictest0-21swapper/100:17:421
2914599246,8cyclictest5478-21memory00:20:190
2914599244,4cyclictest10199-21expr00:30:250
2915499239,8cyclictest0-21swapper/221:22:582
2915499239,8cyclictest0-21swapper/221:22:582
2915499236,3cyclictest17848-21diskmemload22:00:222
2915499231,5cyclictest0-21swapper/223:03:462
2915499231,11cyclictest0-21swapper/222:23:592
2915499230,14cyclictest0-21swapper/220:31:172
2915499230,12cyclictest0-21swapper/219:59:172
2915099232,8cyclictest27779-21memory21:30:191
2915099231,19cyclictest27490-21users23:55:281
2915099231,16cyclictest0-21swapper/122:59:011
2914599236,4cyclictest29700-21users00:00:260
2914599236,3cyclictest21862-21/usr/sbin/munin20:05:130
2914599235,2cyclictest29271-21cstates21:35:130
2914599234,8cyclictest20371-21ntp_states23:40:210
2914599232,9cyclictest0-21swapper/022:05:250
29145992315,5cyclictest19359-40dnf19:59:230
2914599231,15cyclictest0-21swapper/023:27:140
2914599230,11cyclictest28851-21perf21:35:010
2916099229,2cyclictest8564-1kworker/u9:121:41:083
29160992215,4cyclictest22006-21grep20:05:113
2915499227,10cyclictest0-21swapper/200:08:102
29154992210,3cyclictest0-21swapper/200:28:502
2915099228,12cyclictest19991-21kworker/u8:021:49:591
2915099221,18cyclictest26613-21forks20:15:151
2915099221,18cyclictest23433-21diskstats22:35:121
2915099221,18cyclictest11473-21/usr/sbin/munin20:55:181
2915099221,16cyclictest0-21swapper/122:48:121
2915099221,11cyclictest2071-21/usr/sbin/munin23:00:131
2914599224,8cyclictest30654-21/usr/sbin/munin20:25:170
2914599221,8cyclictest13531-21cut19:45:170
29145992211,4cyclictest87350irq/126-enp1s0-20:00:160
2914599221,12cyclictest25903-21/usr/sbin/munin23:55:140
2914599221,10cyclictest0-21swapper/022:47:290
2914599220,10cyclictest32315-21perf00:10:010
2916099218,3cyclictest0-21swapper/320:36:593
2916099211,7cyclictest0-21swapper/322:33:573
2916099211,11cyclictest29061-21/usr/sbin/munin21:35:153
2915499218,4cyclictest0-21swapper/223:35:132
2915499214,8cyclictest0-21swapper/220:12:252
2915499211,8cyclictest0-21swapper/221:31:182
2915499211,4cyclictest0-21swapper/200:37:592
2915499210,13cyclictest0-21swapper/223:45:182
29150992118,2cyclictest0-21swapper/121:50:131
29150992116,3cyclictest24042-21grep21:20:271
29150992116,3cyclictest24042-21grep21:20:271
29150992113,3cyclictest0-21swapper/123:48:221
2915099211,18cyclictest12462-21/usr/sbin/munin22:10:131
2915099211,15cyclictest0-21swapper/122:53:031
2915099211,11cyclictest18781-21cpuspeed_turbos22:25:121
2915099211,11cyclictest18781-21cpuspeed_turbos22:25:121
2915099211,10cyclictest26167-21tail21:25:271
2915099210,5cyclictest0-21swapper/121:18:381
2914599215,4cyclictest15261-21diskstats23:30:150
2914599214,9cyclictest2075-21/usr/sbin/munin19:20:210
2914599214,6cyclictest0-21swapper/000:12:220
2914599214,10cyclictest20893-21/usr/sbin/munin22:30:230
29145992112,6cyclictest13417-40dnf20:59:530
29145992112,2cyclictest87550irq/128-enp1s0-20:23:220
2914599211,11cyclictest31261-21/usr/sbin/munin21:40:130
2914599210,9cyclictest25282-21/usr/sbin/munin22:40:230
2914599210,7cyclictest31356-21munin-node19:15:020
2914599210,18cyclictest4209-21kworker/u8:000:35:130
2914599210,14cyclictest0-21swapper/022:11:140
2914599210,11cyclictest0-21swapper/023:23:450
8182201,2sleep2837-21turbostat21:45:022
2916099208,3cyclictest87650irq/129-enp1s0-21:20:203
2916099208,3cyclictest87650irq/129-enp1s0-21:20:203
2916099203,3cyclictest0-21swapper/322:40:023
2916099201,6cyclictest0-21swapper/323:44:473
2916099201,11cyclictest839-21lldpd00:35:253
2916099201,11cyclictest7265-21taskset21:57:353
2916099201,11cyclictest19654-21/usr/sbin/munin20:00:273
2916099201,10cyclictest0-21swapper/300:11:183
2915499205,3cyclictest87650irq/129-enp1s0-22:50:152
2915499205,3cyclictest0-21swapper/222:40:092
2915499204,2cyclictest0-21swapper/222:30:012
2915499204,2cyclictest0-21swapper/222:30:012
2915499201,6cyclictest0-21swapper/222:38:462
2915499201,12cyclictest0-21swapper/223:05:132
2915099205,13cyclictest1282-21interrupts20:30:211
2915099204,9cyclictest0-21swapper/121:55:111
2915099202,9cyclictest31356-21munin-node19:15:021
2915099201,6cyclictest0-21swapper/122:15:351
2915099201,17cyclictest0-21swapper/120:01:301
2915099201,17cyclictest0-21swapper/119:20:221
2915099201,16cyclictest32624-21munin-run20:30:021
2915099201,14cyclictest0-21swapper/123:37:141
2915099201,14cyclictest0-21swapper/123:19:481
2915099201,11cyclictest19580-21sed21:10:261
2915099201,11cyclictest18616-21sed22:25:011
2915099201,11cyclictest1757-21users00:10:261
2915099201,10cyclictest7406-21sed00:25:161
2915099201,10cyclictest579-21systemd-journal20:15:011
2914599206,5cyclictest0-21swapper/021:18:570
2914599205,2cyclictest0-21swapper/023:50:110
2914599204,6cyclictest11826-21kworker/u8:122:15:420
2914599204,12cyclictest13667-21/usr/sbin/munin21:00:230
2914599202,15cyclictest4094-21cat20:35:260
2916099197,5cyclictest401ktimersoftd/321:25:143
2916099196,3cyclictest14694-21sleep23:30:003
2916099195,2cyclictest12050irq/123-ahci[0021:55:023
2916099194,6cyclictest755-21rngd23:24:503
2916099192,4cyclictest909-21sssd_nss22:54:293
2916099191,8cyclictest30654-21/usr/sbin/munin20:25:253
2916099191,8cyclictest27446-21unixbench_singl23:55:263
2916099191,12cyclictest0-21swapper/323:07:113
2916099191,10cyclictest27481-21/usr/sbin/munin22:45:203
2916099190,2cyclictest89250irq/135-enp2s0-21:10:153
2915499196,2cyclictest17848-21diskmemload00:31:022
2915499194,3cyclictest0-21swapper/223:34:052
2915499191,3cyclictest0-21swapper/221:27:552
2915499191,11cyclictest495-21jbd2/dm-0-820:57:212
2915499191,10cyclictest0-21swapper/220:36:372
2915499191,10cyclictest0-21swapper/220:00:172
2915499190,9cyclictest0-21swapper/220:24:592
2915499190,3cyclictest0-21swapper/219:54:492
2915099199,7cyclictest4418-21kworker/u8:220:50:191
2915099199,2cyclictest9682-21if_enp1s000:30:181
2915099197,3cyclictest7279-21irqstats19:30:171
2915099195,5cyclictest0-21swapper/123:20:361
2915099192,15cyclictest23-21ksoftirqd/120:37:011
29150991915,3cyclictest0-21swapper/122:31:451
29150991915,3cyclictest0-21swapper/122:03:501
29150991915,3cyclictest0-21swapper/121:40:211
29150991915,3cyclictest0-21swapper/119:36:471
2915099191,16cyclictest4418-21kworker/u8:221:00:131
2915099191,16cyclictest13118-21chrt00:40:001
2915099191,16cyclictest0-21swapper/120:06:561
2915099191,16cyclictest0-21swapper/119:25:151
2915099191,10cyclictest5190-21smart_sda23:05:241
2915099191,10cyclictest28940-21fschecks_time20:20:151
2915099191,10cyclictest19423-21latency_hist20:00:011
2915099191,10cyclictest12826-21latency_hist19:45:001
2915099191,10cyclictest1122-21sh19:15:241
2915099190,14cyclictest0-21swapper/123:43:151
2914599196,6cyclictest0-21swapper/021:26:400
2914599193,13cyclictest5127-21grep19:25:190
2914599192,5cyclictest612-21expr20:30:160
29145991910,6cyclictest24403-21diskstats20:10:130
85062180,3sleep28509-21latency_hist19:35:012
2916099188,2cyclictest17441-21sshd19:55:033
2916099187,3cyclictest808-21snmpd23:39:353
2916099185,5cyclictest87350irq/126-enp1s0-22:00:253
2916099181,8cyclictest9237-21memory23:15:183
2916099181,7cyclictest22260-21taskset21:20:003
2916099181,6cyclictest0-21swapper/322:23:373
2916099181,5cyclictest0-21swapper/323:00:123
29160991812,4cyclictest28722-21grep00:00:153
2916099181,15cyclictest23058-21grep23:45:243
2916099181,15cyclictest0-21swapper/321:32:213
2916099181,11cyclictest0-21swapper/322:43:273
2916099181,10cyclictest808-21snmpd22:14:203
2916099181,10cyclictest0-21swapper/300:30:463
2916099181,10cyclictest0-21swapper/300:15:193
2915499187,3cyclictest0-21swapper/200:17:222
2915499186,4cyclictest0-21swapper/219:37:572
2915499186,2cyclictest0-21swapper/220:53:262
2915499186,2cyclictest0-21swapper/220:47:172
2915499186,2cyclictest0-21swapper/220:25:292
2915499185,4cyclictest2412-21df19:20:122
2915499184,3cyclictest87650irq/129-enp1s0-20:18:212
2915499183,3cyclictest87650irq/129-enp1s0-23:53:592
2915499183,3cyclictest87650irq/129-enp1s0-22:05:162
2915499183,3cyclictest0-21swapper/219:10:142
2915499181,7cyclictest6125-21/usr/sbin/munin23:10:242
2915499181,7cyclictest17848-21diskmemload22:30:122
2915499181,7cyclictest0-21swapper/221:14:102
2915499181,4cyclictest592-21irqstats19:15:182
2915499181,4cyclictest0-21swapper/223:16:462
2915499181,4cyclictest0-21swapper/221:36:102
29154991811,4cyclictest15524-21fgrep22:15:202
2915499181,10cyclictest0-21swapper/200:22:362
2915099188,2cyclictest0-21swapper/100:05:241
2915099187,3cyclictest10866-21sed22:05:191
29150991815,2cyclictest0-21swapper/121:07:421
2915099181,16cyclictest0-21swapper/120:42:141
2915099181,16cyclictest0-21swapper/100:23:091
2915099180,17cyclictest0-21swapper/100:00:231
2914599189,5cyclictest26602-21diskstats20:15:140
2914599185,4cyclictest0-21swapper/021:51:390
2914599181,5cyclictest87350irq/126-enp1s0-20:50:110
60972170,2sleep30-21swapper/319:28:223
2916099177,3cyclictest12050irq/123-ahci[0020:25:013
2916099171,8cyclictest808-21snmpd00:21:143
2916099171,7cyclictest909-21sssd_nss20:10:183
2916099171,7cyclictest0-21swapper/319:50:133
2916099171,6cyclictest7572-21irqstats00:25:193
29160991712,3cyclictest2688-21users21:45:263
2916099170,4cyclictest87350irq/126-enp1s0-22:15:233
2916099170,15cyclictest0-21swapper/322:06:043
2916099170,10cyclictest0-21swapper/320:33:253
2915499171,3cyclictest0-21swapper/219:49:452
29154991712,3cyclictest13667-21/usr/sbin/munin21:00:192
29150991712,3cyclictest13870-21irqstats19:45:191
2914599173,2cyclictest89250irq/135-enp2s0-19:35:120
29145991710,5cyclictest15254-21/usr/sbin/munin19:50:260
147112171,2sleep30-21swapper/319:46:083
2916099164,3cyclictest17848-21diskmemload22:25:083
2916099164,3cyclictest17848-21diskmemload22:25:083
2916099164,3cyclictest0-21swapper/321:05:123
2916099161,6cyclictest808-21snmpd19:38:493
2916099161,6cyclictest0-21swapper/320:52:113
2916099161,4cyclictest23702-21/usr/sbin/munin23:50:113
29160991612,3cyclictest0-21swapper/320:56:513
29160991612,2cyclictest11206-21df_inode19:40:143
2916099161,14cyclictest0-21swapper/321:00:253
29160991611,3cyclictest7731-21smart_sda19:30:243
29160991611,3cyclictest6125-21/usr/sbin/munin23:10:163
29160991610,3cyclictest5829-21missed_timers20:40:173
29160991610,3cyclictest2202-21grep19:20:113
2915499163,4cyclictest87350irq/126-enp1s0-20:07:072
29154991611,3cyclictest4883-21/usr/sbin/munin20:40:232
29154991611,3cyclictest32527-21/usr/sbin/munin00:10:172
29154991611,3cyclictest25903-21/usr/sbin/munin23:55:252
2914599169,4cyclictest6815-21diskstats19:30:140
2914599160,7cyclictest31885-21users19:15:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional