You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-14 - 05:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 14, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
200862236162,24sleep00-21swapper/019:07:370
200302203166,25sleep10-21swapper/119:06:541
199752202166,24sleep30-21swapper/319:06:123
199292202166,24sleep20-21swapper/219:05:352
2046521720,4sleep22036299cyclictest21:35:122
2827521440,3sleep02034799cyclictest00:20:120
2827521440,3sleep02034799cyclictest00:20:110
852521360,3sleep22036299cyclictest21:05:252
65002660,2sleep10-21swapper/123:30:131
20355996562,2cyclictest0-21swapper/122:09:421
213692570,1sleep30-21swapper/321:35:203
10522430,2sleep20-21swapper/222:05:022
10522430,2sleep20-21swapper/222:05:022
229412370,4sleep322945-21fschecks_count21:40:143
20347993611,12cyclictest9067-21munin-run21:10:010
20347993511,11cyclictest32355-21idleruntime-cro00:30:010
2036299331,19cyclictest0-21swapper/200:09:192
2034799331,23cyclictest22191-21kworker/0:019:30:300
2036299321,10cyclictest0-21swapper/220:21:582
2036299310,11cyclictest0-21swapper/219:20:192
2036299301,19cyclictest0-21swapper/221:11:342
2036299301,13cyclictest0-21swapper/223:16:342
2036299301,10cyclictest0-21swapper/223:57:592
2036299301,10cyclictest0-21swapper/222:49:192
2036299300,13cyclictest0-21swapper/200:10:232
2036299291,18cyclictest0-21swapper/200:27:032
2036299291,11cyclictest0-21swapper/220:43:302
2034799292,17cyclictest22191-21kworker/0:019:25:020
2034799291,13cyclictest14078-21munin-run22:35:010
120802290,3sleep310862-21/usr/sbin/munin20:00:223
2034799288,16cyclictest22258-21munin-run21:40:010
2034799285,11cyclictest1861-21basename19:40:000
2034799284,12cyclictest27615-21munin-run19:25:010
102452280,4sleep230-21rcuc/222:25:122
2036299271,3cyclictest0-21swapper/220:05:272
2036299271,10cyclictest0-21swapper/221:34:272
2036799261,13cyclictest0-21swapper/321:55:093
2036299261,12cyclictest0-21swapper/223:28:262
2036299261,11cyclictest0-21swapper/223:09:382
2036299261,11cyclictest0-21swapper/221:15:192
2036299261,10cyclictest0-21swapper/200:23:472
2036299261,10cyclictest0-21swapper/200:23:472
2036299260,10cyclictest0-21swapper/222:43:072
2035599261,17cyclictest28473-21df_inode00:20:121
2035599261,17cyclictest28473-21df_inode00:20:121
2036299251,9cyclictest0-21swapper/221:57:112
2036299251,10cyclictest0-21swapper/200:40:002
2036299250,10cyclictest0-21swapper/223:22:022
2035599251,16cyclictest29734-21cpu23:10:131
2035599251,16cyclictest29734-21cpu23:10:121
2035599251,16cyclictest1283-40dnf23:17:211
2034799255,17cyclictest23516-21munin-run00:10:000
2034799251,8cyclictest6211-21kworker/0:019:54:020
2034799251,10cyclictest32315-21kworker/0:020:51:270
2034799251,10cyclictest32315-21kworker/0:020:51:270
2036799241,11cyclictest800-21snmpd21:00:213
2036299247,9cyclictest0-21swapper/200:17:432
2036299247,3cyclictest24401-21df20:30:142
2036299241,9cyclictest0-21swapper/223:43:272
2036299241,10cyclictest0-21swapper/223:51:232
2036299241,10cyclictest0-21swapper/223:35:272
2036299240,10cyclictest0-21swapper/219:10:202
2034799242,9cyclictest764-21kworker/0:322:07:420
2034799242,9cyclictest27240-21kworker/0:123:06:590
2034799241,9cyclictest13216-21kworker/0:021:21:110
2034799241,8cyclictest2213-21kworker/0:321:01:170
202952240,5sleep02034799cyclictest22:45:240
2036799231,5cyclictest821-21lldpd23:35:353
2036299231,11cyclictest0-21swapper/221:04:542
2036299230,9cyclictest0-21swapper/222:54:432
2036299230,9cyclictest0-21swapper/221:53:512
2036299230,10cyclictest0-21swapper/223:12:222
2036299230,10cyclictest0-21swapper/223:12:222
2035599231,3cyclictest1754-21smart_sda20:50:241
2035599231,3cyclictest1754-21smart_sda20:50:241
2035599231,16cyclictest0-21swapper/123:38:211
2035599231,14cyclictest12730-21bc23:45:121
2034799232,9cyclictest23695-21kworker/0:220:40:350
2034799231,9cyclictest4432-21kworker/0:120:58:510
2034799231,9cyclictest14916-21kworker/0:320:21:270
2036799228,3cyclictest0-21swapper/320:35:273
2036799224,11cyclictest0-21swapper/300:05:173
2036799221,9cyclictest0-21swapper/319:43:383
2036799221,9cyclictest0-21swapper/319:43:383
2036799221,10cyclictest0-21swapper/322:56:413
2036299221,13cyclictest0-21swapper/219:54:192
2036299221,11cyclictest0-21swapper/222:56:342
2036299220,9cyclictest0-21swapper/220:39:342
2035599221,16cyclictest0-21swapper/122:44:431
2035599221,16cyclictest0-21swapper/100:26:021
20355992211,3cyclictest23831-21latency_hist20:30:001
2035599221,11cyclictest831-21runrttasks19:15:011
2035599221,11cyclictest10643-21cpuspeed_turbos23:40:121
2034799229,2cyclictest27240-21kworker/0:123:12:030
2034799229,2cyclictest27240-21kworker/0:123:12:030
2034799228,2cyclictest1421-21kworker/0:023:24:350
2034799222,8cyclictest8926-21kworker/0:121:17:170
2034799222,8cyclictest446-21kworker/0:222:15:560
2034799222,8cyclictest30615-21kworker/0:022:00:080
2034799222,8cyclictest30615-21kworker/0:022:00:080
2034799222,8cyclictest27240-21kworker/0:123:19:170
2034799222,8cyclictest21145-21kworker/0:100:14:120
2034799222,8cyclictest1577-21kworker/0:219:48:040
2034799222,8cyclictest14692-21kworker/0:220:12:410
2034799222,8cyclictest1421-21kworker/0:023:28:130
2034799222,8cyclictest13216-21kworker/0:021:29:130
2034799221,9cyclictest23500-21kworker/0:020:26:150
2034799221,8cyclictest446-21kworker/0:222:25:280
2034799221,8cyclictest24412-21kworker/0:121:59:280
2034799221,8cyclictest21145-21kworker/0:100:15:080
2034799221,8cyclictest16571-21kworker/0:000:03:440
306362210,7sleep3401ktimersoftd/300:25:133
2036799211,9cyclictest4852-21missed_timers23:25:193
2036799211,14cyclictest0-21swapper/319:48:333
2036799211,13cyclictest0-21swapper/319:50:193
2036799211,11cyclictest3322-21memory00:35:203
2036799211,11cyclictest29693-21/usr/sbin/munin23:10:173
2036799211,11cyclictest29693-21/usr/sbin/munin23:10:173
2036799211,11cyclictest26556-21memory19:20:163
2036799210,13cyclictest0-21swapper/322:30:533
2036299217,2cyclictest0-21swapper/219:35:592
2036299211,9cyclictest0-21swapper/219:16:342
2036299211,14cyclictest0-21swapper/222:12:272
2036299211,11cyclictest0-21swapper/222:05:392
2036299211,10cyclictest0-21swapper/222:37:312
2036299210,10cyclictest0-21swapper/221:45:312
2035599211,6cyclictest3077-21fschecks_count20:55:141
2035599211,3cyclictest21094-21irqstats21:35:171
20355992113,3cyclictest0-21swapper/122:56:381
20355992113,3cyclictest0-21swapper/122:02:161
20355992113,3cyclictest0-21swapper/122:02:161
20355992113,3cyclictest0-21swapper/100:14:201
20355992112,2cyclictest9062-21diskmemload00:07:011
2035599211,15cyclictest0-21swapper/121:57:501
2035599211,15cyclictest0-21swapper/121:42:421
2035599211,15cyclictest0-21swapper/121:05:341
2035599211,15cyclictest0-21swapper/100:02:171
2035599211,14cyclictest0-21swapper/123:06:151
2035599211,14cyclictest0-21swapper/122:15:051
2035599211,14cyclictest0-21swapper/121:47:091
2035599211,14cyclictest0-21swapper/121:21:421
2035599211,14cyclictest0-21swapper/121:01:121
2035599211,11cyclictest27623-21sed20:35:231
2035599211,11cyclictest23444-21/usr/sbin/munin19:15:231
2035599210,3cyclictest24232-21cstates20:30:131
2034799219,2cyclictest23695-21kworker/0:220:31:190
2034799215,3cyclictest11888-21kworker/0:020:01:580
2034799212,9cyclictest5671-21kworker/0:223:54:340
2034799212,9cyclictest1889-21kworker/0:200:39:450
2034799212,8cyclictest446-21kworker/0:222:11:440
2034799212,8cyclictest31965-21kworker/0:100:30:330
2034799212,7cyclictest22191-21kworker/0:019:17:530
2034799212,7cyclictest22070-21kworker/0:023:04:370
2034799211,8cyclictest24412-21kworker/0:121:47:570
2034799211,8cyclictest19759-21kworker/0:221:41:130
2034799211,8cyclictest16029-21kworker/0:022:41:320
2034799211,8cyclictest14916-21kworker/0:320:15:570
2036799207,3cyclictest0-21swapper/323:56:333
2036799204,13cyclictest11490-21/usr/sbin/munin21:15:193
2036799203,14cyclictest21939-21cpuspeed_turbos20:25:103
2036799203,10cyclictest31933-21sshd19:34:133
20367992015,3cyclictest0-21swapper/322:25:253
2036799201,10cyclictest32323-21ntpq22:00:193
2036799201,10cyclictest32323-21ntpq22:00:193
2036799201,10cyclictest21360-21irqstats19:10:143
2036799201,10cyclictest1385-21ps00:30:243
2036799201,10cyclictest0-21swapper/320:05:403
2036299200,9cyclictest0-21swapper/220:15:502
2036299200,8cyclictest0-21swapper/222:30:152
2035599209,3cyclictest3908-21cat23:25:021
2035599203,15cyclictest0-21swapper/120:41:051
20355992016,2cyclictest2338-21cstates19:40:121
20355992016,2cyclictest2338-21cstates19:40:121
20355992013,2cyclictest0-21swapper/121:18:441
2035599201,2cyclictest21588-21runrttasks22:50:141
20355992012,3cyclictest0-21swapper/100:36:561
2035599201,17cyclictest27819-21grep21:50:191
2035599201,17cyclictest0-21swapper/100:15:061
2035599201,15cyclictest16542-21grep21:25:161
2035599201,14cyclictest0-21swapper/123:51:501
2035599201,14cyclictest0-21swapper/123:26:071
2035599201,14cyclictest0-21swapper/123:04:491
2035599201,14cyclictest0-21swapper/122:39:591
2035599201,14cyclictest0-21swapper/122:28:401
2035599201,14cyclictest0-21swapper/122:11:131
2035599201,10cyclictest8048-21unixbench_singl19:50:261
2034799209,3cyclictest26870-21/usr/sbin/munin21:50:170
2034799209,2cyclictest14916-21kworker/0:320:09:390
2034799207,2cyclictest18315-21kworker/0:122:51:250
2034799203,2cyclictest1421-21kworker/0:023:31:030
2034799202,7cyclictest8926-21kworker/0:121:14:450
2034799201,8cyclictest22070-21kworker/0:022:56:110
2034799201,11cyclictest14299-21/usr/sbin/munin22:35:150
2034799201,11cyclictest12714-21/usr/sbin/munin23:45:160
2034799200,8cyclictest17521-21kworker/0:121:30:050
2036799195,5cyclictest377-21/usr/sbin/munin20:50:163
2036799195,5cyclictest377-21/usr/sbin/munin20:50:163
2036799193,14cyclictest39-21rcuc/320:45:173
2036799192,14cyclictest821-21lldpd19:40:023
2036799192,11cyclictest831-21runrttasks00:17:593
2036799191,9cyclictest10050-21ethtool21:10:153
2036799191,7cyclictest25595-21latency_hist19:20:003
2036799191,4cyclictest20892-21/usr/sbin/munin22:50:133
2036799191,13cyclictest0-21swapper/300:12:363
2036799191,10cyclictest9721-21ntpq19:55:203
2036799191,10cyclictest11592-21sshd23:40:203
2036799191,10cyclictest1119-21/usr/sbin/munin22:05:213
2036799191,10cyclictest0-21swapper/322:45:223
2036799191,10cyclictest0-21swapper/320:55:173
2035599198,2cyclictest29815-21munin-run19:30:011
2035599197,3cyclictest0-21swapper/122:47:191
20355991913,2cyclictest0-21swapper/121:10:031
20355991912,2cyclictest0-21swapper/100:32:001
2035599191,16cyclictest0-21swapper/119:46:401
2035599191,16cyclictest0-21swapper/119:36:281
2035599191,15cyclictest9062-21diskmemload21:30:431
2035599191,10cyclictest17276-21expr23:55:131
2035599191,10cyclictest13326-21gunzip22:30:231
2035599190,17cyclictest0-21swapper/120:45:181
2034799198,2cyclictest23695-21kworker/0:220:36:450
2034799191,9cyclictest30100-21kworker/0:120:48:110
20347991913,3cyclictest799-21sshd23:40:200
2034799191,10cyclictest17109-21/usr/sbin/munin23:55:170
2034799190,7cyclictest4976-21kworker/0:119:10:070
84282181,3sleep00-21swapper/019:55:010
2036799184,5cyclictest770-21rngd21:27:433
2036799181,9cyclictest0-21swapper/323:22:383
2036799181,7cyclictest25456-21tr23:00:123
2036799181,5cyclictest4380-21ntp_states22:10:193
20367991812,3cyclictest418-21needreboot23:15:163
20367991812,3cyclictest14753-21ntp_states21:20:203
2036799181,15cyclictest0-21swapper/322:21:003
2036799181,15cyclictest0-21swapper/320:21:383
2036799181,15cyclictest0-21swapper/320:11:093
2036799181,14cyclictest14306-21vmstat23:45:253
2036799181,10cyclictest0-21swapper/322:40:213
2036299187,3cyclictest12050irq/123-ahci[0021:20:242
2036299181,6cyclictest6515-21grep23:30:132
2036299180,9cyclictest0-21swapper/222:20:192
2036299180,9cyclictest0-21swapper/219:25:152
2035599185,3cyclictest87350irq/128-enp1s0-19:55:111
2035599181,9cyclictest19752-21cpuspeed_turbos20:20:111
2035599181,5cyclictest30954-21memory19:30:191
2035599181,5cyclictest11707-21grep20:00:161
20355991813,3cyclictest7820-21cut22:20:131
2035599181,16cyclictest0-21swapper/119:24:281
2035599180,17cyclictest0-21swapper/120:05:091
2035599180,16cyclictest0-21swapper/120:15:221
2034799187,2cyclictest7697-21kworker/0:122:20:420
2034799185,2cyclictest1577-21kworker/0:219:41:500
2034799185,2cyclictest1577-21kworker/0:219:41:500
2034799181,7cyclictest8320-21/usr/sbin/munin23:35:120
263902171,2sleep30-21swapper/321:46:533
2036799178,3cyclictest0-21swapper/323:09:403
2036799178,3cyclictest0-21swapper/321:06:373
2036799171,7cyclictest28447-21/usr/sbin/munin20:40:183
2036799171,15cyclictest0-21swapper/322:16:333
2036799171,15cyclictest0-21swapper/300:01:343
20367991711,3cyclictest14646-21diskstats22:35:143
2036799171,10cyclictest0-21swapper/319:26:363
20367991710,4cyclictest27480-21head21:50:153
2036799170,9cyclictest0-21swapper/323:50:233
2036299176,3cyclictest87150irq/126-enp1s0-20:25:092
2036299173,3cyclictest0-21swapper/200:02:132
20362991712,3cyclictest3145-21ntpq19:40:212
20362991712,3cyclictest3145-21ntpq19:40:212
20362991712,3cyclictest10862-21/usr/sbin/munin20:00:182
2036299171,15cyclictest0-21swapper/223:45:392
2036299171,15cyclictest0-21swapper/219:49:452
2035599171,2cyclictest0-21swapper/120:10:411
2036799162,12cyclictest39-21rcuc/320:30:183
2036799162,12cyclictest18922-21grep21:30:153
2036799161,4cyclictest28132-21/usr/sbin/munin00:20:143
2036799161,4cyclictest28132-21/usr/sbin/munin00:20:143
20367991612,3cyclictest0-21swapper/320:16:373
2036799161,13cyclictest8088-21crond23:35:013
2036299161,6cyclictest30236-21expr19:30:122
2036299161,6cyclictest25528-21cstates23:00:112
2036299161,5cyclictest31913-21smart_sda20:45:252
2036299161,5cyclictest15395-21tr20:10:102
2036299161,2cyclictest0-21swapper/219:57:312
20362991612,3cyclictest0-21swapper/222:16:462
20362991612,3cyclictest0-21swapper/221:44:402
20362991612,3cyclictest0-21swapper/220:55:332
20362991611,3cyclictest2117-21sshd00:34:212
20362991611,3cyclictest16798-21seq21:25:192
2036299144,3cyclictest87350irq/128-enp1s0-20:50:182
2036299144,3cyclictest87350irq/128-enp1s0-20:50:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional