You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 09:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat May 02, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
187032214163,38sleep20-21swapper/219:05:592
188592206169,24sleep10-21swapper/119:07:591
187642201162,26sleep00-21swapper/019:06:440
170422201165,24sleep30-21swapper/319:05:033
19108998480,3cyclictest0-21swapper/220:23:192
309152590,7sleep341-21ksoftirqd/323:15:133
98312470,4sleep011-21rcuc/022:25:220
276872440,1sleep20-21swapper/220:40:142
276872440,1sleep20-21swapper/220:40:132
19114994415,21cyclictest0-21swapper/300:00:163
19114993814,14cyclictest0-21swapper/322:05:163
19114993813,18cyclictest0-21swapper/321:10:123
19108993722,4cyclictest0-21swapper/223:44:572
1910899371,13cyclictest0-21swapper/220:34:592
1911499368,5cyclictest909-21sssd_nss21:05:133
19114993511,3cyclictest1470-21hddtemp_smartct19:40:153
19114993511,2cyclictest0-21swapper/319:55:163
19114993510,4cyclictest0-21swapper/300:20:263
19114993510,4cyclictest0-21swapper/300:20:263
1910899351,7cyclictest14011-21/usr/sbin/munin20:10:152
1911499343,3cyclictest0-21swapper/319:37:573
19114993411,13cyclictest8366-21switchtime23:35:223
1910899346,3cyclictest87650irq/129-enp1s0-22:55:262
1910899342,5cyclictest7071-21/usr/sbin/munin23:35:272
1910899341,13cyclictest0-21swapper/223:54:072
1910899341,13cyclictest0-21swapper/220:00:392
1911499339,3cyclictest8650-21/usr/sbin/munin22:25:253
1910899339,2cyclictest32620-21awk23:20:012
1910899331,6cyclictest19022-21/usr/sbin/munin21:35:142
1910899331,4cyclictest0-21swapper/221:25:402
19108993313,3cyclictest1-21systemd19:50:212
1910899331,12cyclictest0-21swapper/220:17:472
1910899331,11cyclictest0-21swapper/220:46:192
19114993211,3cyclictest10376-21cut21:15:113
1911499321,10cyclictest23564-21sshd22:56:593
19114993210,3cyclictest19639-21/usr/sbin/munin22:50:243
1910899327,3cyclictest0-21swapper/223:13:502
1910899321,15cyclictest0-21swapper/222:32:512
1910899321,11cyclictest12382-21memory23:45:192
19108993210,3cyclictest0-21swapper/223:56:232
1911499319,3cyclictest827-21runrttasks21:22:033
1911499318,4cyclictest909-21sssd_nss22:40:593
1911499315,7cyclictest30426-40dnf19:30:593
1911499315,7cyclictest30426-40dnf19:30:593
19114993112,16cyclictest29075-21/usr/sbin/munin00:25:193
1911499311,12cyclictest25620-21/usr/sbin/munin21:50:133
19114993110,3cyclictest27590-21dump-pmu-power21:59:563
19114993110,3cyclictest17998-21cat00:00:013
19114993110,2cyclictest23740-21which00:10:233
1910899311,6cyclictest0-21swapper/221:57:402
1910899311,15cyclictest0-21swapper/221:43:272
1910899311,14cyclictest0-21swapper/219:21:352
1910899311,12cyclictest0-21swapper/221:33:032
1910899311,12cyclictest0-21swapper/219:29:352
1910899311,11cyclictest7710-21smart_sda22:20:232
1911499309,4cyclictest26368-21timerandwakeup20:35:233
1911499309,3cyclictest2059-21/usr/sbin/munin22:10:213
1911499309,2cyclictest31956-21sed00:30:163
1911499305,3cyclictest0-21swapper/320:07:573
1911499301,14cyclictest19302-21cstates21:35:103
1911499301,12cyclictest16822-21/usr/sbin/munin21:30:173
1911499301,12cyclictest0-21swapper/322:33:353
1911499301,10cyclictest26355-21latency_hist19:25:013
19114993010,4cyclictest0-21swapper/323:25:203
19114993010,3cyclictest4876-21/usr/sbin/munin23:30:143
19114993010,3cyclictest1053-21gltestperf23:20:163
1910899309,4cyclictest0-21swapper/222:51:472
1910899309,3cyclictest21353-21ntp_states00:05:272
1910899304,3cyclictest0-21swapper/221:47:342
19108993014,3cyclictest31613-21gltestperf19:35:152
1910899301,15cyclictest0-21swapper/200:00:312
1910899301,14cyclictest0-21swapper/223:20:352
1910899301,14cyclictest0-21swapper/220:25:112
1910899301,12cyclictest0-21swapper/222:17:592
1909599304,13cyclictest854-21awk00:35:010
1911499299,3cyclictest9846-21expr20:00:133
1911499299,3cyclictest8293-21seq22:23:293
1911499299,3cyclictest31855-21expr20:50:113
1911499299,2cyclictest0-21swapper/322:45:163
1911499298,3cyclictest6222-21latency_hist22:20:023
1911499298,3cyclictest3109-21grep19:45:113
1911499295,6cyclictest0-21swapper/300:15:163
1911499291,4cyclictest808-21snmpd21:40:013
19114992911,3cyclictest27268-21netstat23:05:183
19114992911,3cyclictest105712sleep323:40:253
19114992910,3cyclictest827-21runrttasks19:15:263
1911499290,15cyclictest0-21swapper/300:36:163
1910899297,3cyclictest0-21swapper/221:11:432
1910899291,14cyclictest0-21swapper/219:15:272
1910899291,13cyclictest0-21swapper/220:54:542
1910899291,13cyclictest0-21swapper/219:30:272
1910899291,13cyclictest0-21swapper/219:30:272
1910899291,13cyclictest0-21swapper/200:34:032
1910899291,13cyclictest0-21swapper/200:25:352
1910899291,12cyclictest0-21swapper/221:23:152
1910899291,12cyclictest0-21swapper/221:03:552
1910899291,12cyclictest0-21swapper/220:07:032
1910899291,12cyclictest0-21swapper/219:41:432
1909599291,13cyclictest7827-21taskset21:10:010
1911499289,3cyclictest26584-21/usr/sbin/munin19:25:253
1911499289,3cyclictest14010-21diskstats23:50:133
1911499289,3cyclictest0-21swapper/319:14:023
1911499288,3cyclictest29371-21cat20:45:023
1911499288,3cyclictest29371-21cat20:45:023
1911499287,3cyclictest0-21swapper/320:55:073
1911499285,3cyclictest0-21swapper/320:47:513
1911499281,15cyclictest0-21swapper/321:49:473
19114992811,3cyclictest24081-21switchtime20:30:233
19114992810,3cyclictest3629-21/usr/sbin/munin21:00:253
1910899281,13cyclictest0-21swapper/223:08:262
1910899281,13cyclictest0-21swapper/222:26:072
1910899281,13cyclictest0-21swapper/221:18:112
1910899281,13cyclictest0-21swapper/219:55:232
1910899281,13cyclictest0-21swapper/219:13:592
1910899281,13cyclictest0-21swapper/200:22:592
1910899281,13cyclictest0-21swapper/200:22:592
1910899281,12cyclictest0-21swapper/222:42:152
1910899281,12cyclictest0-21swapper/222:37:352
1910899281,12cyclictest0-21swapper/222:00:032
1910899281,12cyclictest0-21swapper/220:55:072
1910899281,12cyclictest0-21swapper/219:47:232
1910899281,12cyclictest0-21swapper/200:38:152
1910899281,11cyclictest0-21swapper/223:00:342
1910899281,11cyclictest0-21swapper/222:46:032
1910899281,11cyclictest0-21swapper/200:14:552
1910299284,15cyclictest13192-21sshd23:47:021
1909599281,14cyclictest4035-21munin-run22:15:010
1911499279,3cyclictest14853-21expr21:25:113
19114992711,3cyclictest21199-21missed_timers00:05:243
1910899271,14cyclictest0-21swapper/223:32:552
1910899271,12cyclictest0-21swapper/223:25:232
1910899271,12cyclictest0-21swapper/222:07:552
1910899271,12cyclictest0-21swapper/220:39:392
1910899271,11cyclictest0-21swapper/221:05:072
1910899271,11cyclictest0-21swapper/200:16:152
19108992710,4cyclictest26203-21gltestperf21:50:152
1910899270,12cyclictest0-21swapper/222:10:072
1911499269,3cyclictest18405-21/usr/sbin/munin20:20:173
1911499267,3cyclictest0-21swapper/322:37:353
1911499267,2cyclictest7325-21cat19:55:023
1911499264,3cyclictest87650irq/129-enp1s0-23:10:203
1911499264,3cyclictest0-21swapper/323:05:003
1911499264,3cyclictest0-21swapper/322:00:073
1911499261,11cyclictest13431-21latency_hist23:50:023
19114992610,3cyclictest0-21swapper/320:25:113
1911499257,3cyclictest0-21swapper/320:17:113
1910299251,16cyclictest30131-21ntpq00:25:201
1909599251,13cyclictest14418-21perf21:25:020
41932240,4sleep11910299cyclictest21:00:151
1911499247,3cyclictest0-21swapper/320:13:113
1910299245,7cyclictest0-21swapper/121:45:251
1909599248,14cyclictest0-21swapper/019:35:000
1909599248,14cyclictest0-21swapper/019:35:000
1909599241,11cyclictest32010-21perf22:05:000
1910299231,3cyclictest26090-21taskset20:35:201
1910299231,3cyclictest19667-21which20:20:221
1910299231,17cyclictest0-21swapper/121:14:261
1909599236,15cyclictest15018-21munin-run22:40:020
1910299229,3cyclictest3406-21munin-run21:00:011
1910299222,3cyclictest0-21swapper/120:07:381
1910299222,15cyclictest0-21swapper/119:46:371
1910299221,8cyclictest0-21swapper/122:35:501
19102992215,4cyclictest25304-21grep00:15:171
19102992215,2cyclictest0-21swapper/123:03:191
1910299221,3cyclictest17846-40updatedb00:00:021
1910299221,16cyclictest0-21swapper/120:17:231
1910299221,14cyclictest0-21swapper/123:50:281
1910299221,13cyclictest0-21swapper/121:38:021
1910299221,11cyclictest32002-21perf22:05:011
1910299221,11cyclictest28858-21perf00:25:011
1910299221,11cyclictest28858-21perf00:25:011
1910299220,6cyclictest0-21swapper/121:44:521
1909599221,8cyclictest11188-21kworker/0:223:49:290
1909599221,6cyclictest0-21swapper/000:02:000
1909599221,5cyclictest0-21swapper/000:29:570
1909599221,11cyclictest13359-21kworker/u8:123:55:120
1909599220,4cyclictest0-21swapper/022:42:050
1910299216,5cyclictest7508-21cpuspeed_turbos19:55:111
1910299212,15cyclictest0-21swapper/122:49:021
1910299211,9cyclictest0-21swapper/123:07:411
19102992113,3cyclictest0-21swapper/123:18:231
19102992112,2cyclictest29824-21fschecks_count20:45:151
1910299211,17cyclictest28399-21perl21:55:151
1910299211,17cyclictest19819-21grep19:10:151
1910299211,15cyclictest0-21swapper/122:42:411
1910299211,15cyclictest0-21swapper/120:32:471
1910299211,15cyclictest0-21swapper/119:52:051
1910299211,15cyclictest0-21swapper/119:22:101
1910299211,15cyclictest0-21swapper/119:20:001
1910299211,15cyclictest0-21swapper/100:38:271
1910299211,14cyclictest0-21swapper/123:25:141
1910299211,11cyclictest7215-21tr23:35:121
1910299211,11cyclictest2167-21chrt23:21:331
1910299211,11cyclictest15048-21fschecks_count21:25:151
1910299211,10cyclictest22578-21grep00:10:121
1909599213,15cyclictest24792-21perf20:35:010
1909599212,8cyclictest1483-21kworker/0:222:23:010
1909599212,4cyclictest0-21swapper/021:35:010
1909599212,3cyclictest4602-21diskstats22:15:130
1909599212,11cyclictest14011-21/usr/sbin/munin20:10:210
1909599211,8cyclictest0-21swapper/021:36:230
1909599211,7cyclictest32546-21kworker/0:223:29:530
1909599211,6cyclictest0-21swapper/022:07:350
1909599211,5cyclictest0-21swapper/022:46:400
19095992114,4cyclictest20882-21head00:05:200
1909599211,11cyclictest1646-21ps23:20:230
19095992110,3cyclictest13664-21/usr/sbin/munin23:50:160
119282210,5sleep11910299cyclictest22:30:201
1910299208,3cyclictest15979-21crond20:15:011
1910299201,8cyclictest0-21swapper/100:31:511
1910299201,6cyclictest26324-21idleruntime21:50:161
1910299201,6cyclictest0-21swapper/100:07:291
1910299201,5cyclictest0-21swapper/121:16:491
19102992013,2cyclictest0-21swapper/123:10:321
1910299201,17cyclictest0-21swapper/122:20:231
1910299201,14cyclictest0-21swapper/123:32:591
1910299201,14cyclictest0-21swapper/122:16:191
1910299201,14cyclictest0-21swapper/122:10:501
1910299201,14cyclictest0-21swapper/122:09:321
1910299201,14cyclictest0-21swapper/121:06:551
1910299201,12cyclictest0-21swapper/123:41:191
1910299201,11cyclictest30426-40dnf19:30:581
1910299201,11cyclictest30426-40dnf19:30:581
1910299201,10cyclictest31753-21grep20:50:101
1910299200,4cyclictest11934-21kworker/u8:021:21:391
1910299200,2cyclictest87650irq/129-enp1s0-22:57:591
1909599209,3cyclictest30642-21/usr/sbin/munin23:15:260
1909599209,3cyclictest29398-21/usr/sbin/munin20:45:130
1909599204,7cyclictest0-21swapper/023:32:250
1909599202,6cyclictest8911-21kworker/0:123:39:210
1909599201,7cyclictest0-21swapper/019:40:200
1909599201,6cyclictest23480-21cut00:10:180
1909599201,5cyclictest0-21swapper/021:12:380
1909599201,4cyclictest0-21swapper/021:51:190
1909599201,11cyclictest0-21swapper/023:42:560
1909599201,11cyclictest0-21swapper/022:55:090
1909599201,11cyclictest0-21swapper/021:58:160
1909599201,11cyclictest0-21swapper/021:25:490
1909599201,10cyclictest0-21swapper/023:00:030
1910299194,6cyclictest21-21rcuc/122:25:211
1910299192,16cyclictest0-21swapper/120:00:501
1910299191,3cyclictest0-21swapper/120:44:331
1910299191,3cyclictest0-21swapper/120:44:331
19102991913,2cyclictest0-21swapper/121:31:081
1910299191,2cyclictest0-21swapper/122:50:281
19102991911,5cyclictest17412-21grep23:55:271
1910299191,10cyclictest20667-21cpu20:25:111
1910299190,14cyclictest0-21swapper/119:35:481
1910299190,14cyclictest0-21swapper/119:26:591
1909599198,3cyclictest26244-21/usr/sbin/munin23:05:130
1909599193,7cyclictest0-21swapper/000:36:230
1909599191,5cyclictest0-21swapper/021:15:380
1909599191,11cyclictest0-21swapper/023:14:200
1909599191,10cyclictest30981-21/usr/sbin/munin19:35:200
1909599191,10cyclictest23102-21sshd21:43:570
1909599191,10cyclictest0-21swapper/022:50:130
1909599191,10cyclictest0-21swapper/022:30:140
1909599191,10cyclictest0-21swapper/021:49:520
1909599190,7cyclictest18106-21kworker/0:320:23:530
1909599190,6cyclictest0-21swapper/000:23:510
1909599190,6cyclictest0-21swapper/000:23:510
1910299180,10cyclictest1142-21diskstats19:40:131
1909599187,3cyclictest20610-21/usr/sbin/munin20:25:240
1909599185,3cyclictest0-21swapper/000:18:260
1909599181,6cyclictest0-21swapper/020:35:210
19095991812,3cyclictest27847-21smartctl19:25:220
1909599181,10cyclictest3386-21sshd20:59:570
1909599177,6cyclictest0-21swapper/019:14:560
19095991712,3cyclictest808-21snmpd20:09:550
1909599161,9cyclictest0-21swapper/021:04:440
1909599161,9cyclictest0-21swapper/019:55:130
1909599161,6cyclictest5215-21/usr/sbin/munin19:50:220
1909599161,6cyclictest0-21swapper/020:40:170
1909599161,6cyclictest0-21swapper/020:40:170
1909599161,6cyclictest0-21swapper/019:21:360
19095991612,3cyclictest0-21swapper/019:45:130
1909599161,14cyclictest0-21swapper/020:02:580
19095991611,3cyclictest31591-21/usr/sbin/munin20:50:100
1909599161,12cyclictest16208-21/usr/sbin/munin20:15:160
1909599151,5cyclictest21854-21hddtemp_smartct19:15:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional