You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 05:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon May 18, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
588224470,3sleep011-21rcuc/019:05:260
63912241166,24sleep10-21swapper/119:07:401
63922203167,24sleep20-21swapper/219:07:412
63482203166,25sleep30-21swapper/319:07:063
6670991070,2cyclictest0-21swapper/221:25:002
6670997875,2cyclictest0-21swapper/200:34:472
45882750,2sleep00-21swapper/023:55:210
127222520,4sleep0665799cyclictest00:15:170
194452480,2sleep20-21swapper/219:35:182
666399441,31cyclictest14996-21tune2fs00:20:161
666399421,26cyclictest2733-21grep23:50:261
666399401,30cyclictest16045-21hddtemp_smartct21:55:151
666399391,21cyclictest0-21swapper/122:36:021
665799389,13cyclictest15791-21awk23:10:010
666399372,4cyclictest7615-21munin-run00:05:011
666399371,3cyclictest23324-21munin-run21:00:011
666399364,7cyclictest16091-40dnf19:29:561
666399361,3cyclictest8870-21munin-node19:15:021
666399361,3cyclictest8870-21munin-node19:15:011
666399352,5cyclictest0-21swapper/121:40:151
666399351,4cyclictest12337-21munin-run20:35:021
666399351,22cyclictest29014-21ntp_states21:10:221
6663993512,20cyclictest10418-21head19:15:181
666399351,21cyclictest0-21swapper/122:50:131
666399351,21cyclictest0-21swapper/122:50:131
666399350,27cyclictest27948-21kworker/u8:021:18:131
666399345,3cyclictest87550irq/128-enp1s0-23:05:161
666399341,4cyclictest22955-21cpu19:45:131
6663993410,21cyclictest5728-21latency_hist20:20:011
666399340,8cyclictest23-21ksoftirqd/119:21:591
666399339,11cyclictest25887-21expr21:05:131
666399331,7cyclictest10982-21/usr/sbin/munin21:45:261
666399331,4cyclictest12362-21cpuspeed_turbos00:15:131
666399331,3cyclictest22592-21/usr/sbin/munin23:25:181
666399331,29cyclictest26463-40dnf23:31:231
6663993311,19cyclictest18494-21/usr/sbin/munin20:45:261
6663993310,19cyclictest25550-21ls21:05:011
666399328,20cyclictest7753-21ps21:35:241
666399328,20cyclictest7753-21ps21:35:241
666399326,3cyclictest87550irq/128-enp1s0-20:00:141
666399325,2cyclictest0-21swapper/123:40:171
666399321,7cyclictest25129-21/usr/sbin/munin19:50:171
666399321,5cyclictest3090-21memory21:25:171
666399321,5cyclictest18539-21/usr/sbin/munin19:35:241
6663993211,18cyclictest3531-21turbostat.cron20:15:011
6663993210,19cyclictest5051-21smartctl21:30:161
6663993210,19cyclictest13180-21/usr/sbin/munin21:50:181
667499311,20cyclictest0-21swapper/323:36:263
666399319,19cyclictest31383-21/usr/sbin/munin23:45:201
666399315,3cyclictest87550irq/128-enp1s0-22:05:141
666399311,5cyclictest9235-21ntp_states20:25:221
666399311,5cyclictest9235-21ntp_states20:25:221
666399311,5cyclictest11602-21/usr/sbin/munin23:00:171
666399311,5cyclictest0-21swapper/123:10:191
666399311,3cyclictest0-21swapper/100:25:261
6663993110,18cyclictest577-21wc20:05:261
6663993110,18cyclictest16797-21cut19:30:141
666399310,3cyclictest22408-21fschecks_count22:10:151
666399309,17cyclictest21215-21cstates00:35:131
666399307,19cyclictest4368-21grep22:40:261
666399307,19cyclictest18597-21sa122:00:201
666399302,3cyclictest7098-21/usr/sbin/munin20:20:251
666399301,5cyclictest28314-21taskset22:24:511
666399301,5cyclictest20056-21chrt23:18:421
666399301,4cyclictest8727-21latency00:05:181
666399301,4cyclictest8727-21latency00:05:181
666399301,4cyclictest10471-21ntp_states22:55:211
666399301,3cyclictest19322-21expr20:50:121
666399301,26cyclictest27725-21diskmemload22:48:501
666399301,26cyclictest27511-21fschecks_time23:35:151
666399293,4cyclictest14054-21/usr/sbin/munin20:35:271
666399291,4cyclictest4947-21/usr/sbin/munin23:55:241
666399291,4cyclictest21903-21/usr/sbin/munin19:40:231
666399291,4cyclictest21310-21memory23:20:191
666399291,25cyclictest28352-21nfs4_client19:55:201
666399290,26cyclictest20812-21awk00:35:021
665799291,22cyclictest11980-21kworker/0:000:29:220
666399287,18cyclictest24231-21cpu22:15:131
666399281,4cyclictest32699-21/usr/sbin/munin21:20:151
666399281,3cyclictest32400-40dnf22:30:561
666399281,3cyclictest15661-21/usr/sbin/munin20:40:191
6663992810,15cyclictest11698-21seq00:11:021
6657992818,2cyclictest30460-21kworker/0:122:36:030
666399278,16cyclictest28818-21expr22:25:131
665799274,11cyclictest18327-21perf19:35:020
6657992716,4cyclictest28640-21kworker/0:123:41:230
667499260,12cyclictest0-21swapper/323:05:173
665799261,9cyclictest17935-21kworker/0:323:15:550
665799261,9cyclictest12100-21kworker/0:220:35:290
665799260,11cyclictest7836-21kworker/0:100:14:580
667499251,11cyclictest0-21swapper/319:20:133
667499251,11cyclictest0-21swapper/300:38:333
667499250,11cyclictest0-21swapper/322:43:133
667099258,6cyclictest29463-21cat23:40:122
665799257,15cyclictest13596-21awk23:05:010
6657992516,2cyclictest11980-21kworker/0:000:24:540
665799251,19cyclictest0-21swapper/023:53:000
665799251,12cyclictest18929-21munin-run20:50:010
665799251,11cyclictest23014-21awk00:40:010
665799250,9cyclictest6831-21kworker/0:222:52:470
665799250,9cyclictest6831-21kworker/0:222:52:470
667499240,1cyclictest0-21swapper/319:35:253
665799241,12cyclictest28952-21idleruntime-cro23:40:000
665799241,11cyclictest25910-21kworker/0:222:22:020
665799240,17cyclictest0-21swapper/021:21:190
667499237,3cyclictest0-21swapper/323:49:303
667499231,9cyclictest0-21swapper/321:15:183
667499230,10cyclictest0-21swapper/323:25:003
667099231,5cyclictest0-21swapper/222:50:082
667099231,5cyclictest0-21swapper/222:50:082
665799239,3cyclictest3965-21proc_pri22:40:220
665799236,14cyclictest3549-21awk20:15:000
665799232,9cyclictest26566-21kworker/0:221:10:550
665799232,8cyclictest25916-21kworker/0:322:25:360
665799232,7cyclictest30853-21kworker/0:000:00:550
665799230,9cyclictest12712-21kworker/0:121:48:110
665799230,12cyclictest32750-21perf22:35:010
667499228,3cyclictest0-21swapper/300:15:433
667499221,14cyclictest87650irq/129-enp1s0-22:50:133
667499221,14cyclictest87650irq/129-enp1s0-22:50:133
667499221,13cyclictest0-21swapper/300:10:533
667499221,11cyclictest29636-21ntp_states22:25:213
667499220,10cyclictest0-21swapper/300:34:473
667099221,8cyclictest0-21swapper/221:59:022
6670992212,5cyclictest11404-21ntp_kernel_pll_20:30:212
667099221,10cyclictest0-21swapper/222:43:382
665799228,5cyclictest25764-21/usr/sbin/munin23:30:180
665799226,2cyclictest6004-21kworker/0:321:35:070
665799226,2cyclictest6004-21kworker/0:321:35:070
665799222,9cyclictest31092-21kworker/0:223:49:030
665799222,9cyclictest14862-21kworker/0:321:56:520
665799222,8cyclictest29655-21kworker/0:021:41:270
665799222,8cyclictest22274-21kworker/0:123:25:310
665799222,7cyclictest21499-21kworker/0:022:10:520
665799222,14cyclictest87650irq/129-enp1s0-21:00:210
665799222,10cyclictest6831-21kworker/0:222:56:070
665799221,7cyclictest7932-21kworker/0:219:11:350
665799221,7cyclictest7932-21kworker/0:219:11:350
667099211,6cyclictest0-21swapper/222:31:152
667099211,6cyclictest0-21swapper/222:02:112
6670992114,4cyclictest12380-21tr00:15:132
667099210,4cyclictest9008-21kworker/u8:023:34:072
665799215,7cyclictest26566-21kworker/0:221:15:470
665799212,8cyclictest17208-21kworker/0:122:06:200
665799212,8cyclictest13747-21kworker/0:319:29:060
665799211,9cyclictest13620-21kworker/0:219:43:040
665799211,8cyclictest14442-21kworker/0:200:31:170
665799211,7cyclictest6004-21kworker/0:321:33:270
665799211,7cyclictest11304-21kworker/0:023:14:110
665799211,13cyclictest0-21swapper/019:15:190
665799211,12cyclictest8165-21/usr/sbin/munin20:25:120
665799211,12cyclictest8165-21/usr/sbin/munin20:25:120
6657992111,2cyclictest13747-21kworker/0:319:35:260
667499209,3cyclictest20736-21/usr/sbin/munin19:40:193
667499209,2cyclictest0-21swapper/321:42:223
667499206,3cyclictest0-21swapper/319:28:573
667499202,5cyclictest773-21NetworkManager20:17:043
6674992015,3cyclictest30178-21sed20:00:153
667499201,11cyclictest0-21swapper/321:25:183
667499201,10cyclictest19778-21/usr/sbin/munin22:05:153
667099204,4cyclictest0-21swapper/221:10:192
667099201,6cyclictest0-21swapper/200:00:502
667099201,5cyclictest0-21swapper/223:36:282
6670992014,3cyclictest8978-21irqstats20:25:172
6670992014,3cyclictest8978-21irqstats20:25:172
665799209,3cyclictest7839-21/usr/sbin/munin00:05:260
665799209,3cyclictest7839-21/usr/sbin/munin00:05:260
665799204,12cyclictest87650irq/129-enp1s0-22:15:220
665799201,12cyclictest9008-21kworker/u8:023:20:210
665799201,11cyclictest22935-21/usr/sbin/munin19:45:190
665799200,8cyclictest29655-21kworker/0:021:27:150
667499196,5cyclictest774-21avahi-daemon00:00:023
667499196,2cyclictest10982-21/usr/sbin/munin21:45:133
667499192,2cyclictest87650irq/129-enp1s0-00:29:053
667499191,8cyclictest0-21swapper/322:46:533
6674991914,3cyclictest8341-21grep19:10:263
6674991914,3cyclictest8341-21grep19:10:253
6674991913,4cyclictest919-21sh22:35:113
667499191,10cyclictest808-21snmpd23:28:593
667499191,10cyclictest25752-21/usr/sbin/munin21:05:183
667499191,10cyclictest23294-21sshd20:59:543
667499191,10cyclictest15374-21/usr/sbin/munin21:55:133
667499191,10cyclictest0-21swapper/321:33:213
667099191,7cyclictest0-21swapper/223:26:562
667099191,7cyclictest0-21swapper/221:33:062
667099191,6cyclictest10288-21sort22:55:192
667099191,6cyclictest0-21swapper/200:27:192
667099191,3cyclictest0-21swapper/200:08:442
667099191,3cyclictest0-21swapper/200:08:442
667099190,12cyclictest0-21swapper/223:48:232
665799199,3cyclictest10362-21/usr/sbin/munin20:30:200
665799198,3cyclictest4996-21/usr/sbin/munin22:45:200
665799192,6cyclictest14862-21kworker/0:321:54:480
665799191,16cyclictest16732-21munin-run20:45:010
310122191,3sleep30-21swapper/323:43:223
128662190,4sleep012870-21missed_timers19:20:200
667499182,5cyclictest0-21swapper/322:30:553
667499181,7cyclictest21123-21munin-run20:55:013
667499181,7cyclictest0-21swapper/323:13:573
667499181,6cyclictest1979-21if_enp2s023:50:193
667499181,15cyclictest0-21swapper/320:39:223
667499181,10cyclictest0-21swapper/323:18:563
667499181,10cyclictest0-21swapper/300:00:223
667499180,7cyclictest0-21swapper/321:52:223
667099189,2cyclictest32-21ksoftirqd/223:00:202
667099185,5cyclictest755-21rngd23:55:332
667099181,8cyclictest0-21swapper/223:11:412
667099181,7cyclictest0-21swapper/221:16:512
667099181,6cyclictest0-21swapper/200:22:092
667099181,5cyclictest843-21expr22:35:102
667099181,5cyclictest0-21swapper/223:06:402
667099181,4cyclictest5297-21unixbench_singl20:15:262
6670991813,3cyclictest14522-21crond20:40:022
6670991813,3cyclictest10145-21timerandwakeup21:40:242
667099181,15cyclictest0-21swapper/222:15:012
667099181,15cyclictest0-21swapper/222:07:112
667099181,15cyclictest0-21swapper/221:26:402
667099181,15cyclictest0-21swapper/219:42:232
6670991811,4cyclictest21118-21crond20:55:022
665799181,9cyclictest27327-21/usr/sbin/munin19:55:250
665799180,2cyclictest16538-21kworker/0:121:05:170
667499178,3cyclictest0-21swapper/322:00:183
667499175,3cyclictest87350irq/126-enp1s0-20:20:143
667499172,4cyclictest8827-21needreboot00:05:203
667499172,4cyclictest8827-21needreboot00:05:193
667499171,9cyclictest0-21swapper/322:10:403
667499171,7cyclictest6580-21/usr/sbin/munin21:35:193
667499171,7cyclictest6580-21/usr/sbin/munin21:35:193
667499171,7cyclictest26145-21munin-run22:20:013
667499171,6cyclictest24086-21/usr/sbin/munin19:45:233
667499171,6cyclictest11390-40dnf20:30:203
667499171,6cyclictest0-21swapper/322:24:013
667499171,5cyclictest0-21swapper/321:20:353
6674991713,3cyclictest0-21swapper/323:04:533
6674991712,3cyclictest23542-21/usr/sbin/munin21:00:233
6674991712,3cyclictest14973-21grep00:20:153
6674991711,4cyclictest754-21grep20:05:263
6674991711,4cyclictest16956-21/usr/sbin/munin20:45:193
6674991711,3cyclictest827-21runrttasks20:27:403
6674991711,3cyclictest827-21runrttasks20:27:403
667499171,11cyclictest87350irq/126-enp1s0-20:40:463
667499171,10cyclictest0-21swapper/319:50:143
667099177,7cyclictest24181-21gltestperf21:00:172
667099175,3cyclictest87650irq/129-enp1s0-21:48:382
667099172,8cyclictest0-21swapper/222:20:542
667099171,9cyclictest0-21swapper/200:35:522
667099171,7cyclictest24891-21basename22:15:172
667099171,7cyclictest23323-21perf21:00:022
667099171,7cyclictest0-21swapper/223:15:312
667099171,7cyclictest0-21swapper/200:12:002
667099171,6cyclictest5098-21grep22:45:132
667099171,5cyclictest0-21swapper/222:26:232
667099171,4cyclictest17796-21cut20:45:192
6670991712,3cyclictest27327-21/usr/sbin/munin19:55:182
667099171,15cyclictest0-21swapper/219:12:052
667099171,15cyclictest0-21swapper/219:12:052
667099171,13cyclictest13913-21latency_hist19:25:012
667099171,10cyclictest0-21swapper/223:50:282
6670991710,4cyclictest29983-21tune2fs20:00:142
665799177,2cyclictest0-21swapper/020:57:540
665799175,3cyclictest13620-21kworker/0:220:00:280
665799174,2cyclictest13620-21kworker/0:220:21:570
665799173,8cyclictest16588-21kworker/0:320:50:070
665799172,4cyclictest25129-21/usr/sbin/munin19:50:250
665799171,13cyclictest31729-21/usr/sbin/munin20:05:240
665799171,10cyclictest17208-21kworker/0:122:01:280
247822171,2sleep30-21swapper/323:30:103
667499161,8cyclictest87350irq/126-enp1s0-21:10:193
667499161,4cyclictest808-21snmpd19:34:353
667499161,12cyclictest1548-21/usr/sbin/munin20:10:143
6674991610,3cyclictest9595-21grep19:15:053
667099169,4cyclictest13326-21cut21:50:132
667099161,9cyclictest0-21swapper/223:21:212
667099161,9cyclictest0-21swapper/219:30:532
667099161,8cyclictest0-21swapper/219:50:072
667099161,5cyclictest15828-21ntp_states20:40:232
667099161,4cyclictest0-21swapper/221:07:562
6670991612,2cyclictest311ktimersoftd/220:08:052
6670991611,3cyclictest9648-21processes19:15:042
6670991611,3cyclictest6345-21latency_hist21:35:022
6670991611,3cyclictest6345-21latency_hist21:35:012
6670991611,2cyclictest7045-21ntpq20:20:242
6657991610,3cyclictest5857-21date20:20:010
113412161,2sleep30-21swapper/322:59:473
6674991510,3cyclictest28819-21unixbench_multi19:55:253
667099159,3cyclictest14424-21ls19:25:122
6670991510,3cyclictest25031-21cat19:50:012
667099141,5cyclictest0-21swapper/220:10:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional