You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-24 - 08:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 24, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
168882238162,25sleep30-21swapper/319:05:563
170322235163,24sleep10-21swapper/119:07:471
170282231163,55sleep20-21swapper/219:07:442
171342229161,55sleep00-21swapper/019:09:070
2327721300,2sleep31729799cyclictest23:00:213
2327721300,2sleep31729799cyclictest23:00:213
633621060,1sleep10-21swapper/123:35:221
2020421000,5sleep21729399cyclictest20:25:252
17293999288,3cyclictest0-21swapper/219:47:452
17293996455,3cyclictest0-21swapper/220:15:022
235712620,5sleep2311ktimersoftd/200:15:172
134682580,4sleep230-21rcuc/221:25:172
17293995249,2cyclictest0-21swapper/200:32:162
1729799341,21cyclictest0-21swapper/319:57:133
1729799340,20cyclictest0-21swapper/322:01:213
1727799334,13cyclictest18257-21awk00:05:010
1729799321,19cyclictest0-21swapper/320:00:253
1729799320,15cyclictest0-21swapper/300:23:213
1729799320,12cyclictest0-21swapper/321:11:573
1729799311,20cyclictest0-21swapper/319:44:173
1729799310,14cyclictest0-21swapper/321:50:213
1729799310,14cyclictest0-21swapper/321:50:213
1729799301,23cyclictest0-21swapper/321:19:453
1729799301,12cyclictest0-21swapper/323:09:333
1729799301,12cyclictest0-21swapper/323:09:333
17297992916,3cyclictest0-21swapper/323:26:053
1729799291,2cyclictest0-21swapper/322:54:353
1729799291,12cyclictest0-21swapper/321:45:333
1729799290,12cyclictest0-21swapper/322:27:173
1727799292,14cyclictest11980-21munin-run20:10:010
1729799281,12cyclictest0-21swapper/323:32:053
1727799289,16cyclictest31761-21munin-run20:55:010
1729799271,8cyclictest0-21swapper/319:31:233
1729799271,8cyclictest0-21swapper/319:31:233
1729799271,24cyclictest0-21swapper/323:44:573
1729799271,12cyclictest13771-21df22:40:133
1729799271,12cyclictest0-21swapper/322:56:293
1729799270,13cyclictest0-21swapper/321:29:333
1729799270,12cyclictest0-21swapper/321:59:333
1729799270,11cyclictest0-21swapper/323:49:373
1729799266,10cyclictest0-21swapper/300:05:153
1729799261,12cyclictest0-21swapper/320:50:033
1729799260,13cyclictest0-21swapper/300:35:493
1729799260,11cyclictest0-21swapper/321:31:253
1728599260,20cyclictest0-21swapper/122:35:111
1727799261,13cyclictest26767-21idleruntime-cro19:30:010
1729799258,6cyclictest180142sleep300:02:133
1729799251,9cyclictest0-21swapper/321:08:253
17297992518,2cyclictest0-21swapper/321:24:013
1729799251,13cyclictest0-21swapper/319:24:193
1729799251,13cyclictest0-21swapper/300:11:413
1729799251,11cyclictest0-21swapper/322:19:333
1729799250,11cyclictest0-21swapper/320:37:313
1729799248,3cyclictest0-21swapper/319:19:273
1729799246,3cyclictest0-21swapper/320:49:353
1729799241,12cyclictest0-21swapper/322:47:533
1729799241,12cyclictest0-21swapper/320:25:593
1729799241,12cyclictest0-21swapper/320:17:153
1729799240,17cyclictest0-21swapper/322:39:273
1729799240,12cyclictest0-21swapper/323:13:373
1729799240,10cyclictest0-21swapper/320:24:173
1729399241,3cyclictest0-21swapper/223:39:492
1727799245,16cyclictest14794-21munin-run21:30:010
17277992411,3cyclictest8589-21expr21:15:130
1729799238,3cyclictest0-21swapper/320:07:333
1729799237,3cyclictest0-21swapper/300:16:073
1729799237,2cyclictest0-21swapper/321:38:193
1729799231,13cyclictest0-21swapper/323:36:533
1729799231,13cyclictest0-21swapper/320:55:213
1729799231,10cyclictest0-21swapper/320:43:353
1729799231,10cyclictest0-21swapper/320:43:353
17285992313,5cyclictest0-21swapper/119:22:021
1727799233,11cyclictest7457-21/usr/sbin/munin23:40:110
1727799232,12cyclictest26345-21df21:55:130
1727799231,6cyclictest827-21runrttasks22:49:520
186112220,3sleep20-21swapper/220:25:012
1729799228,3cyclictest0-21swapper/323:56:353
1729799221,9cyclictest0-21swapper/320:30:393
1729799221,10cyclictest0-21swapper/319:35:473
1729799220,9cyclictest0-21swapper/319:53:233
1729799220,21cyclictest0-21swapper/323:19:213
1729399221,6cyclictest8202-21munin-plugin-st21:15:012
1729399221,3cyclictest0-21swapper/222:38:352
1728599223,12cyclictest20765-21kworker/u8:121:53:351
1728599223,12cyclictest20765-21kworker/u8:121:53:351
1728599222,7cyclictest26365-40dnf21:55:181
1728599221,18cyclictest23211-21df00:15:131
1728599221,18cyclictest19187-21chrt19:11:361
1728599221,18cyclictest124982sleep121:23:551
1728599221,12cyclictest30421-21/usr/sbin/munin22:05:241
1728599221,11cyclictest28848-21unixbench_singl00:25:281
17285992210,4cyclictest3525-21ntp_states22:15:211
1727799220,11cyclictest0-21swapper/023:22:280
1729799218,3cyclictest0-21swapper/300:27:213
1729799217,3cyclictest0-21swapper/322:14:513
1729799215,3cyclictest0-21swapper/319:27:553
1729799211,12cyclictest0-21swapper/323:51:573
1729799211,12cyclictest0-21swapper/322:31:013
1729799211,10cyclictest0-21swapper/321:40:513
1729399211,8cyclictest0-21swapper/221:16:362
1729399211,12cyclictest0-21swapper/222:11:292
1728599216,12cyclictest5560-21awk21:05:251
17285992117,3cyclictest0-21swapper/122:13:561
17285992117,2cyclictest8763-21df21:15:141
1728599211,2cyclictest9043-21seq22:30:081
1728599211,18cyclictest0-21swapper/120:31:541
1728599211,18cyclictest0-21swapper/120:21:391
1728599211,17cyclictest20973-21sed22:55:211
1728599211,14cyclictest4134-21ntp_states23:30:231
1728599211,14cyclictest0-21swapper/123:02:421
1728599211,14cyclictest0-21swapper/123:02:421
1728599211,11cyclictest32473-21irqstats00:35:191
1728599211,11cyclictest18628-21cut00:05:141
1728599211,11cyclictest10166-21cpuspeed_turbos20:05:131
17285992110,3cyclictest7568-21latency_hist20:00:021
1727799211,12cyclictest15021-21/usr/sbin/munin21:30:130
1727799211,11cyclictest0-21swapper/023:06:160
1727799211,11cyclictest0-21swapper/023:06:160
1729799207,3cyclictest0-21swapper/319:12:473
1729799205,3cyclictest0-21swapper/322:05:253
1729799205,2cyclictest3803-21latency_hist21:05:013
1729799201,9cyclictest0-21swapper/322:20:393
1729799201,2cyclictest0-21swapper/300:33:213
1729799201,12cyclictest0-21swapper/320:13:533
1729399201,8cyclictest11632-21munin-run23:50:022
1729399201,7cyclictest1-21systemd00:00:012
17293992011,5cyclictest26889-21sed21:55:192
1728599206,3cyclictest0-21swapper/123:09:591
1728599206,3cyclictest0-21swapper/123:09:591
17285992016,3cyclictest0-21swapper/123:40:171
17285992016,3cyclictest0-21swapper/100:12:341
1728599201,3cyclictest0-21swapper/122:25:421
1728599201,18cyclictest0-21swapper/121:11:541
1728599201,11cyclictest953-21latency_hist19:45:011
1728599201,11cyclictest7859-21cpu20:00:111
1728599201,11cyclictest541-21latency_hist23:25:011
1728599201,11cyclictest28951-21awk19:35:011
1728599201,11cyclictest28951-21awk19:35:011
1728599201,11cyclictest2279-21ntp_states19:45:221
1728599201,11cyclictest22439-21latency_hist19:20:011
1728599201,11cyclictest0-21swapper/120:50:201
1728599201,10cyclictest26745-21dump-pmu-power19:30:011
1728599200,5cyclictest23813-21kworker/u8:222:03:491
1728599200,18cyclictest0-21swapper/123:15:121
1728599200,18cyclictest0-21swapper/121:35:441
1728599200,18cyclictest0-21swapper/121:25:271
1727799209,3cyclictest17218-21/usr/sbin/munin21:35:170
1727799206,6cyclictest0-21swapper/023:50:060
1727799206,6cyclictest0-21swapper/023:04:090
1727799206,6cyclictest0-21swapper/023:04:090
1727799206,4cyclictest0-21swapper/022:34:160
1727799202,11cyclictest22875-21/usr/sbin/munin00:15:260
17277992014,3cyclictest29550-21crond20:50:010
17277992014,3cyclictest23122-21grep21:45:260
1727799201,13cyclictest808-21snmpd23:13:320
1727799201,11cyclictest10625-21/usr/sbin/munin21:20:270
1729799194,3cyclictest0-21swapper/319:45:413
1729399195,4cyclictest6000-21diskmemload23:20:212
1729399195,3cyclictest87350irq/126-enp1s0-22:30:132
1729399191,6cyclictest0-21swapper/200:09:322
17293991914,3cyclictest21985-21grep21:45:152
1729399191,10cyclictest21809-21irqstats20:30:172
1728599199,2cyclictest16192-21perl22:45:161
1728599195,4cyclictest0-21swapper/100:00:361
1728599193,10cyclictest8564-1kworker/u9:123:25:031
1728599191,5cyclictest18426-21grep22:50:131
17285991915,3cyclictest0-21swapper/123:47:361
17285991915,3cyclictest0-21swapper/123:12:471
17285991915,3cyclictest0-21swapper/121:48:051
17285991915,3cyclictest0-21swapper/120:14:241
17285991913,4cyclictest14584-21grep22:40:221
1728599191,17cyclictest0-21swapper/100:34:491
1728599191,16cyclictest6000-21diskmemload21:44:381
1728599191,16cyclictest0-21swapper/120:36:181
1728599191,13cyclictest0-21swapper/123:57:451
1728599191,10cyclictest5725-21ntp_states22:20:221
1728599191,10cyclictest29849-21which19:35:161
1728599191,10cyclictest26653-21uptime00:20:271
1728599191,10cyclictest15738-21threads20:15:261
1727799198,3cyclictest16595-21/usr/sbin/munin20:20:240
1727799197,4cyclictest19416-21/usr/sbin/munin21:40:160
1727799197,3cyclictest0-21swapper/023:19:130
1727799197,11cyclictest0-21swapper/000:25:010
17277991912,4cyclictest19044-21grep19:10:240
1727799191,13cyclictest0-21swapper/021:52:450
1727799191,13cyclictest0-21swapper/021:52:450
1727799191,11cyclictest0-21swapper/022:52:170
1727799191,10cyclictest31663-21/usr/sbin/munin00:35:210
1727799191,10cyclictest20041-21/usr/sbin/munin22:55:240
1727799191,10cyclictest11242-21/usr/sbin/munin22:35:150
1727799191,10cyclictest0-21swapper/023:36:420
1727799191,10cyclictest0-21swapper/000:34:200
1727799190,8cyclictest0-21swapper/023:55:500
317492181,13sleep00-21swapper/020:55:020
1729399186,4cyclictest0-21swapper/221:41:452
1729399181,7cyclictest8717-21memory20:00:182
1729399181,7cyclictest5232-21cat23:35:022
1729399181,7cyclictest27947-21hddtemp_smartct00:25:192
1729399181,7cyclictest0-21swapper/222:57:132
1729399181,6cyclictest0-21swapper/221:39:422
1729399181,5cyclictest0-21swapper/222:45:182
1729399181,5cyclictest0-21swapper/222:07:492
1729399181,5cyclictest0-21swapper/222:05:012
1729399181,4cyclictest0-21swapper/223:14:122
17293991813,3cyclictest8020-21ps22:25:212
17293991813,3cyclictest0-21swapper/220:46:022
17293991812,3cyclictest25415-21sh00:20:142
17293991811,4cyclictest8722-21smart_sda23:40:222
17293991811,4cyclictest25358-21memory23:05:192
17293991811,4cyclictest25358-21memory23:05:192
1729399181,10cyclictest0-21swapper/200:37:032
17293991810,3cyclictest0-21swapper/222:52:532
17285991816,1cyclictest0-21swapper/120:49:131
17285991815,2cyclictest0-21swapper/120:40:551
17285991815,2cyclictest0-21swapper/120:40:551
1728599181,16cyclictest0-21swapper/121:30:071
1728599181,16cyclictest0-21swapper/120:55:361
1728599181,16cyclictest0-21swapper/120:27:591
1728599181,10cyclictest0-21swapper/119:50:181
1728599180,16cyclictest0-21swapper/123:50:161
1727799188,3cyclictest20382-21/usr/sbin/munin19:15:170
1727799187,3cyclictest4650-21/usr/sbin/munin22:20:150
1727799187,3cyclictest14117-21hddtemp_smartct22:40:160
1727799186,4cyclictest0-21swapper/022:05:110
1727799183,6cyclictest0-21swapper/022:26:220
1727799182,8cyclictest3068-21/usr/sbin/munin23:30:150
1727799181,9cyclictest819-21lldpd20:18:400
1727799181,8cyclictest4031-21/usr/sbin/munin21:05:240
1727799181,5cyclictest0-21swapper/022:15:570
1727799181,4cyclictest6230-21/usr/sbin/munin21:10:130
17277991813,3cyclictest32446-21ntpq19:40:220
17277991813,3cyclictest26864-21grep20:40:250
17277991813,3cyclictest26864-21grep20:40:250
17277991813,3cyclictest19142-21cut00:05:170
1727799181,15cyclictest0-21swapper/022:10:400
1727799180,9cyclictest26843-21kworker/0:300:25:570
1727799180,5cyclictest0-21swapper/023:46:170
139102181,2sleep00-21swapper/020:11:470
35122171,2sleep00-21swapper/021:01:240
266752171,2sleep20-21swapper/219:29:162
223362171,2sleep00-21swapper/000:11:100
17297991712,3cyclictest32104-21ntp_states23:20:223
1729399179,2cyclictest87350irq/126-enp1s0-21:50:112
1729399179,2cyclictest87350irq/126-enp1s0-21:50:112
1729399176,3cyclictest87650irq/129-enp1s0-21:30:142
1729399176,3cyclictest87650irq/129-enp1s0-19:36:152
1729399174,4cyclictest22933-21hwlatdetect23:00:172
1729399174,4cyclictest22933-21hwlatdetect23:00:172
1729399171,9cyclictest0-21swapper/221:20:572
1729399171,7cyclictest21558-21ps19:15:222
1729399171,7cyclictest0-21swapper/223:16:412
1729399171,6cyclictest0-21swapper/223:25:132
1729399171,6cyclictest0-21swapper/219:20:172
1729399171,6cyclictest0-21swapper/200:11:232
1729399171,5cyclictest30-21rcuc/221:10:012
1729399171,4cyclictest31380-21users20:50:282
17293991713,3cyclictest0-21swapper/222:43:102
17293991713,3cyclictest0-21swapper/222:22:552
17293991713,3cyclictest0-21swapper/220:41:452
17293991713,3cyclictest0-21swapper/220:41:452
17293991713,2cyclictest6000-21diskmemload22:16:332
1729399171,14cyclictest0-21swapper/219:12:432
17293991711,3cyclictest32648-21gltestperf20:55:162
17293991711,3cyclictest12295-21sort20:10:112
17293991711,3cyclictest10819-21awk20:05:182
1729399171,10cyclictest0-21swapper/219:30:362
1729399171,10cyclictest0-21swapper/219:30:362
1728599171,7cyclictest1825-21/usr/sbin/munin21:00:181
1727799171,6cyclictest0-21swapper/020:30:120
1727799171,4cyclictest808-21snmpd20:04:440
17277991712,3cyclictest3405-21/usr/sbin/munin19:50:250
1727799171,13cyclictest26978-21/usr/sbin/munin19:30:240
1727799171,13cyclictest26978-21/usr/sbin/munin19:30:240
1727799171,11cyclictest0-21swapper/019:49:560
1727799170,5cyclictest0-21swapper/023:27:230
1729399161,6cyclictest3406-21acpi19:50:122
1729399161,6cyclictest0-21swapper/223:50:042
1729399161,5cyclictest0-21swapper/200:00:042
1727799167,8cyclictest87350irq/126-enp1s0-19:20:130
1727799165,3cyclictest28212-21/usr/sbin/munin22:00:150
1727799161,4cyclictest24857-21sshd20:36:150
17277991611,3cyclictest7130-21grep19:55:270
17277991611,3cyclictest29370-21expr19:35:130
1729399159,3cyclictest6552-21missed_timers19:55:212
1729399158,4cyclictest31509-21cut19:40:132
1729399154,3cyclictest87550irq/128-enp1s0-20:35:232
1729399151,5cyclictest2528-21basename21:00:162
17277991510,3cyclictest19747-21missed_timers20:25:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional