You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-07 - 13:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Nov 07, 2025 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19351632940,7sleep126-21ksoftirqd/121:10:201
189111328974,9sleep20-21swapper/219:06:202
19733122880,8sleep1251rcuc/122:55:281
162399860,9rtkit-daemon869-21irqbalance21:36:152
162399860,11rtkit-daemon1926404-40lsb_release20:45:252
162399860,11rtkit-daemon1926404-40lsb_release20:45:252
188992028266,9sleep00-21swapper/019:05:070
162399800,7rtkit-daemon1900617-21dump-pmu-power19:34:552
162399790,6rtkit-daemon2197014-21PK-Backend22:46:112
162399790,6rtkit-daemon2197014-21PK-Backend22:46:112
162399770,5rtkit-daemon0-21swapper/221:01:452
162399740,6rtkit-daemon2197014-21PK-Backend20:07:152
162399740,6rtkit-daemon2197014-21PK-Backend20:07:152
162399740,6rtkit-daemon2197014-21PK-Backend00:11:232
162399740,5rtkit-daemon0-21swapper/219:55:092
162399730,6rtkit-daemon0-21swapper/223:50:182
162399730,5rtkit-daemon0-21swapper/221:06:072
19493432720,7sleep02197014-21PK-Backend21:50:150
162399720,6rtkit-daemon2197014-21PK-Backend20:17:272
162399720,5rtkit-daemon0-21swapper/222:52:252
162399720,5rtkit-daemon0-21swapper/222:52:242
189116627140,25sleep30-21swapper/319:07:033
162399710,5rtkit-daemon0-21swapper/221:52:002
162399710,5rtkit-daemon0-21swapper/220:53:322
162399700,5rtkit-daemon0-21swapper/219:53:042
1891538996929,35cyclictest1931438-21expr21:00:150
189082526543,14sleep10-21swapper/119:05:281
19236222610,7sleep22197014-21PK-Backend20:40:012
1891538996019,15cyclictest855-21dbus-daemon00:00:050
1891541995916,11cyclictest438-20systemd-journal23:15:011
189154399588,12cyclictest884-21gdbus21:20:352
1891541995826,6cyclictest1925229-21/usr/sbin/munin20:45:001
1891541995821,8cyclictest1991165-21wc23:45:311
1891541995816,12cyclictest438-20systemd-journal20:36:341
1891538995816,12cyclictest1952601-21cpuspeed_turbos22:00:150
1891548995721,8cyclictest17531gnome-shell19:30:003
189154399572,10cyclictest34-21ksoftirqd/221:40:132
189154399570,39cyclictest1753-21gnome-shell22:55:312
1891541995717,13cyclictest1986958-21missed_timers23:35:251
1891538995719,8cyclictest1978718-21wc23:10:310
189153899571,11cyclictest2197014-21PK-Backend20:30:260
1891541995616,13cyclictest1976685-21ntpq23:05:251
1891541995616,13cyclictest1976685-21ntpq23:05:251
189153899561,27cyclictest1942313-21cstates21:30:140
1891538995611,15cyclictest2179-21gvfsd-metadata23:20:010
1891541995520,11cyclictest1949462-21basename21:50:201
1891538995523,8cyclictest1753-21gnome-shell23:45:180
1891538995516,12cyclictest3242254-21gmain21:45:010
1891543995429,20cyclictest1753-21gnome-shell20:40:232
1891541995421,8cyclictest1912465-21processes20:05:321
1891541995421,8cyclictest1912465-21processes20:05:321
1891541995416,14cyclictest1960379-21df_inode22:20:171
1891541995415,12cyclictest1918633-21cat20:25:151
1891538995422,6cyclictest1069-21runrttasks20:43:260
1891538995421,9cyclictest13-21ksoftirqd/019:50:000
1891538995418,11cyclictest1933178-21grep21:05:130
1891538995417,12cyclictest1923355-21sshd20:38:330
1891538995417,10cyclictest1959742-21cut22:20:020
1891538995417,10cyclictest1959742-21cut22:20:020
19532582530,7sleep20-21swapper/222:00:282
19445692530,8sleep10-21swapper/121:35:231
1891543995329,19cyclictest1753-21gnome-shell20:25:322
1891543995329,19cyclictest1753-21gnome-shell00:35:172
1891541995323,5cyclictest0-21swapper/123:15:171
1891541995315,12cyclictest2962348-20rs:main1
189153899531,26cyclictest1993644-21turbostat23:55:000
1891548995240,6cyclictest42-21ksoftirqd/300:00:043
1891548995240,6cyclictest42-21ksoftirqd/300:00:033
1891543995229,19cyclictest1753-21gnome-shell22:40:312
189154399520,26cyclictest1937547-21ntp_states21:15:302
1891541995217,8cyclictest1932221-21wc21:00:301
1891541995216,11cyclictest1910372-21kernelversion20:00:211
189154199521,13cyclictest2004851-21ls00:25:211
1891538995221,7cyclictest1069-21runrttasks20:13:140
162399520,8rtkit-daemon1622-21rtkit-daemon22:35:230
189154399519,16cyclictest1994996-21/usr/sbin/munin23:55:322
189154399519,16cyclictest1994996-21/usr/sbin/munin23:55:322
189154399517,25cyclictest1753-21gnome-shell21:45:182
189154399516,23cyclictest1753-21gnome-shell19:20:162
189154399513,31cyclictest1753-21gnome-shell19:19:352
1891543995127,20cyclictest1753-21gnome-shell22:23:112
1891543995111,21cyclictest1753-21gnome-shell20:03:022
189154199519,6cyclictest19754452chrt23:02:011
1891541995122,9cyclictest1941821-21cron21:30:001
1891541995120,6cyclictest1943781-21latency_hist21:35:021
189154199511,11cyclictest1961797-21wc22:25:131
1891538995126,5cyclictest0-21swapper/023:24:050
1891538995122,8cyclictest1753-21gnome-shell19:44:150
1891538995118,11cyclictest13-21ksoftirqd/020:50:180
189154399509,19cyclictest1959668-21cut22:20:022
189154399509,19cyclictest1959668-21cut22:20:022
189154399507,24cyclictest1753-21gnome-shell22:38:002
189154399507,24cyclictest1753-21gnome-shell00:34:112
189154399506,25cyclictest1753-21gnome-shell20:10:442
189154399506,24cyclictest1753-21gnome-shell00:18:152
189154399506,24cyclictest1753-21gnome-shell00:18:142
189154399504,27cyclictest1753-21gnome-shell21:56:252
1891543995028,8cyclictest34-21ksoftirqd/200:20:132
189154199503,24cyclictest1972215-21cron22:55:011
189154199503,24cyclictest1972215-21cron22:55:001
189154199502,8cyclictest1901650-21ntp_states19:35:251
1891541995022,7cyclictest1953981-21sh22:05:001
1891541995020,7cyclictest1069-21runrttasks19:28:541
1891541995017,11cyclictest1957971-21dpkg22:15:111
1891541995017,11cyclictest1957971-21dpkg22:15:111
189154199501,22cyclictest1904134-21systemd-run19:45:011
1891538995016,5cyclictest13-21ksoftirqd/022:55:120
1891538995014,7cyclictest1941819-21cron21:30:010
189154399496,24cyclictest1753-21gnome-shell22:29:472
189154399496,24cyclictest1753-21gnome-shell00:06:572
189154399495,26cyclictest1753-21gnome-shell23:20:462
189154399495,26cyclictest1753-21gnome-shell19:45:412
189154399495,26cyclictest1753-21gnome-shell19:36:112
189154399494,26cyclictest1753-21gnome-shell00:27:532
1891543994926,19cyclictest1753-21gnome-shell21:30:252
1891543994910,20cyclictest1753-21gnome-shell19:14:552
1891541994915,12cyclictest1963678-21cut22:30:141
1891541994915,11cyclictest2002631-21sed00:20:031
1891541994915,11cyclictest2002631-21sed00:20:031
1891541994915,10cyclictest1981583-21idleruntime23:20:221
19999462480,11sleep11999950-21ntp_kernel_pll_00:10:271
189154399488,16cyclictest1995979-21cstates00:00:192
189154399488,12cyclictest1753-21gnome-shell21:10:252
189154399487,16cyclictest1974487-21expr23:00:142
189154399486,23cyclictest1753-21gnome-shell23:49:322
189154399486,23cyclictest1753-21gnome-shell23:15:422
189154399485,24cyclictest1753-21gnome-shell23:41:172
189154399481,28cyclictest1753-21gnome-shell20:22:282
1891543994811,12cyclictest1753-21gnome-shell23:05:182
1891543994811,12cyclictest1753-21gnome-shell23:05:172
1891543994810,13cyclictest1753-21gnome-shell23:10:152
1891541994825,19cyclictest1914211-21/usr/sbin/munin20:10:291
1891541994815,12cyclictest1939188-21needreboot21:20:241
189154399479,19cyclictest1753-21gnome-shell23:27:402
1891548994620,21cyclictest42-21ksoftirqd/300:10:303
189154399465,16cyclictest1753-21gnome-shell23:35:202
189154399465,15cyclictest1929958-21/usr/sbin/munin20:55:212
1891543994623,19cyclictest1753-21gnome-shell22:30:312
189154399461,27cyclictest1753-21gnome-shell21:29:372
1891543994610,12cyclictest1753-21gnome-shell22:10:122
1891541994619,22cyclictest2962348-21rs:main1
1891541994617,9cyclictest26-21ksoftirqd/120:25:011
1891541994614,7cyclictest0-21swapper/119:15:021
1891538994525,5cyclictest2197014-21PK-Backend22:27:420
162399450,32rtkit-daemon1892111-21df_inode19:10:173
189154899441,6cyclictest141rcu_preempt20:10:013
189154899441,6cyclictest141rcu_preempt20:10:003
1891548994415,6cyclictest42-21ksoftirqd/321:15:183
189154399444,22cyclictest1753-21gnome-shell19:26:402
1891543994410,17cyclictest1753-21gnome-shell22:07:332
1891543994410,10cyclictest1904111-21munin-run19:45:012
1891538994422,17cyclictest1827-21JS14:31:440
162399440,32rtkit-daemon2197014-21PK-Backend22:50:273
162399440,32rtkit-daemon2197014-21PK-Backend22:50:273
162399440,31rtkit-daemon2197014-21PK-Backend00:20:333
19154702430,17chrt26-21ksoftirqd/120:15:211
1891548994324,6cyclictest42-21ksoftirqd/323:30:173
189154899431,4cyclictest141rcu_preempt19:44:313
189154399434,16cyclictest1753-21gnome-shell20:31:292
189153899431,19cyclictest1069-21runrttasks23:05:420
189153899431,19cyclictest1069-21runrttasks23:05:420
1891538994311,6cyclictest141rcu_preempt23:40:010
162399430,29rtkit-daemon1893524-21switchtime19:15:033
189154899427,17cyclictest1753-21gnome-shell19:52:113
1891548994223,6cyclictest42-21ksoftirqd/322:21:193
1891548994222,15cyclictest884-21gdbus20:40:013
1891548994220,17cyclictest42-21ksoftirqd/323:30:003
189154399426,11cyclictest1985279-21/usr/sbin/munin23:30:222
1891541994210,7cyclictest0-21swapper/120:30:341
1891538994212,7cyclictest19522922sed22:00:010
162399420,29rtkit-daemon2197014-21PK-Backend23:01:563
19705072410,6chrt0-21swapper/022:49:590
19705072410,6chrt0-21swapper/022:49:590
1891548994117,20cyclictest1753-21gnome-shell19:37:093
1891548994116,6cyclictest854-21avahi-daemon20:27:263
189154199412,9cyclictest1753-21gnome-shell19:50:011
1891538994123,6cyclictest13-21ksoftirqd/021:49:480
162399410,29rtkit-daemon2197014-21PK-Backend23:51:233
162399410,29rtkit-daemon2197014-21PK-Backend21:40:263
162399410,28rtkit-daemon2197014-21PK-Backend23:47:333
162399410,28rtkit-daemon2197014-21PK-Backend23:36:233
162399410,27rtkit-daemon2197014-21PK-Backend20:50:233
162399410,26rtkit-daemon2197014-21PK-Backend20:15:153
189154899406,5cyclictest141rcu_preempt23:20:023
1891548994026,10cyclictest42-21ksoftirqd/319:20:313
1891548994023,6cyclictest42-21ksoftirqd/319:34:033
1891548994021,6cyclictest42-21ksoftirqd/321:50:153
189153899407,11cyclictest1069-21runrttasks19:53:270
189153899405,10cyclictest1069-21runrttasks20:15:530
1891538994029,7cyclictest1753-21gnome-shell20:21:150
1891538994025,11cyclictest1999812-21memory00:10:240
1891538994024,9cyclictest1827-21JS14:31:440
1891538994024,9cyclictest1827-21JS14:31:440
1891538994022,13cyclictest13-21ksoftirqd/023:25:090
189153899401,35cyclictest2005902-21turbostat00:30:000
162399400,28rtkit-daemon2197014-21PK-Backend21:07:023
162399400,28rtkit-daemon2197014-21PK-Backend19:55:173
162399400,27rtkit-daemon2197014-21PK-Backend20:00:143
189154899399,5cyclictest141rcu_preempt21:20:353
189154899396,14cyclictest1753-21gnome-shell22:46:153
189154899396,14cyclictest1753-21gnome-shell22:46:153
1891548993922,6cyclictest42-21ksoftirqd/321:55:033
1891548993922,13cyclictest42-21ksoftirqd/322:35:033
1891548993920,6cyclictest141rcu_preempt22:35:253
1891548993919,17cyclictest1753-21gnome-shell00:32:333
1891548993916,13cyclictest42-21ksoftirqd/323:40:223
189154199399,7cyclictest0-21swapper/100:35:251
1891541993922,13cyclictest1957448-21apt-get22:15:011
1891541993921,13cyclictest1993288-40apt-check23:51:191
189153899399,5cyclictest141rcu_preempt22:11:330
1891538993923,12cyclictest1961465-21sshd22:24:320
1891538993912,7cyclictest17531gnome-shell23:40:100
162399390,30rtkit-daemon2197014-21PK-Backend00:25:053
162399390,28rtkit-daemon2197014-21PK-Backend23:10:273
162399390,27rtkit-daemon2197014-21PK-Backend22:44:063
162399390,26rtkit-daemon0-21swapper/322:15:193
162399390,26rtkit-daemon0-21swapper/322:15:193
1891548993821,5cyclictest42-21ksoftirqd/319:50:013
1891548993817,6cyclictest42-21ksoftirqd/320:56:283
189154899381,32cyclictest1936423-21pool-org.gnome.21:15:023
189154899381,22cyclictest141rcu_preempt00:05:273
1891541993822,12cyclictest1900506-21sshd19:33:331
1891541993821,14cyclictest1997306-21cat00:05:011
1891541993821,12cyclictest438-20systemd-journal20:50:011
1891541993821,12cyclictest438-20systemd-journal20:50:011
1891541993821,12cyclictest1995234-21sshd23:58:241
1891541993821,12cyclictest1995234-21sshd23:58:241
1891541993819,14cyclictest3242253-21nautilus23:40:031
1891538993827,7cyclictest1974905-21strings23:00:220
189153899382,32cyclictest1753-21gnome-shell19:59:160
1891538993823,11cyclictest1895302-21sshd19:18:330
1891538993821,13cyclictest1900644-21/usr/sbin/munin19:35:010
162399380,30rtkit-daemon2197014-21PK-Backend21:29:153
162399380,30rtkit-daemon1753-21gnome-shell20:49:293
162399380,30rtkit-daemon1753-21gnome-shell20:49:283
189154899375,14cyclictest1753-21gnome-shell21:48:083
1891548993726,7cyclictest1753-21gnome-shell20:23:563
189154899372,32cyclictest0-21swapper/300:38:473
189154899372,13cyclictest1753-21gnome-shell22:02:073
1891548993720,13cyclictest42-21ksoftirqd/320:30:263
1891541993726,7cyclictest1997634-21cpuspeed_turbos00:05:131
1891541993723,9cyclictest1930397-21python320:55:311
1891541993722,11cyclictest1955409-21seq22:06:291
1891541993721,12cyclictest2007041-21python300:30:341
1891541993714,18cyclictest1946654-21users21:40:321
189153899376,5cyclictest141rcu_preempt20:01:180
1891538993721,11cyclictest1939648-21systemd-udevd21:20:350
189153899371,5cyclictest141rcu_preempt22:07:320
162399370,26rtkit-daemon2197014-21PK-Backend00:15:133
162399370,26rtkit-daemon2197014-21PK-Backend00:15:123
1891548993626,6cyclictest2197014-21PK-Backend22:25:513
1891548993625,7cyclictest1753-21gnome-shell23:09:583
1891548993625,7cyclictest1753-21gnome-shell23:09:573
1891548993622,10cyclictest42-21ksoftirqd/320:11:313
1891548993620,12cyclictest1973833-21sshd22:58:333
189154899361,12cyclictest1753-21gnome-shell20:40:303
1891541993625,7cyclictest64850irq/126-i91519:19:591
1891541993623,8cyclictest1952399-21ls22:00:011
1891541993621,11cyclictest1937255-21/usr/sbin/munin21:15:241
1891541993612,20cyclictest2003644-21cat00:20:331
1891538993624,8cyclictest1897575-21cpuspeed_turbos19:25:170
1891538993622,10cyclictest2001149-21tr00:15:190
1891538993622,10cyclictest2001149-21tr00:15:180
189153899361,31cyclictest0-21swapper/000:05:030
189153899361,12cyclictest13-21ksoftirqd/022:42:120
189153899361,11cyclictest0-21swapper/020:06:320
189153899361,11cyclictest0-21swapper/020:06:320
162399360,28rtkit-daemon2197014-21PK-Backend21:37:203
162399360,27rtkit-daemon2197014-21PK-Backend21:00:153
162399360,23rtkit-daemon2197014-21PK-Backend22:08:493
1891548993524,7cyclictest1753-21gnome-shell23:23:463
1891541993524,7cyclictest1933004-21/usr/sbin/munin21:05:221
1891541993519,12cyclictest1986198-21date23:35:021
1891541993519,12cyclictest1928767-21seq20:52:231
1891541993519,12cyclictest1909115-21sort19:55:331
189154199351,30cyclictest1969272-21cstates22:45:171
189154199351,30cyclictest1969272-21cstates22:45:171
1891538993524,7cyclictest1069-21runrttasks21:15:060
1891538993520,11cyclictest1930200-21head20:55:250
1891538993520,11cyclictest1926322-21grep20:45:220
1891538993520,11cyclictest1926322-21grep20:45:220
1891538993519,12cyclictest1896801-21sshd19:20:330
1891538993517,13cyclictest438-20systemd-journal23:35:010
1891538993516,14cyclictest907-21udisksd00:21:560
189153899351,30cyclictest2006571-21smartctl00:30:230
189153899351,18cyclictest1069-21runrttasks21:14:120
189153899351,10cyclictest13-21ksoftirqd/000:36:210
162399350,27rtkit-daemon2197014-21PK-Backend22:14:583
189154899342,13cyclictest2197014-21PK-Backend21:30:263
1891548993410,8cyclictest141rcu_preempt00:02:113
1891541993422,8cyclictest1984357-21sh23:30:001
189154199341,9cyclictest0-21swapper/121:45:351
1891538993420,10cyclictest13-21ksoftirqd/022:30:160
1891538993419,11cyclictest1918555-21dpkg20:25:120
1891538993418,12cyclictest1893192-21cron19:15:000
189153899341,29cyclictest2197014-21PK-Backend19:37:230
1891538993321,8cyclictest1069-21runrttasks23:58:270
1891538993321,8cyclictest1069-21runrttasks23:58:270
1891541993216,11cyclictest1896218-21latency19:20:221
1891541993112,14cyclictest1966697-40lsb_release22:36:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional