You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 06:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Nov 22, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
206993222070,5sleep12069870-21cstates19:05:121
207143821010,5sleep0161rcuc/019:10:010
20956612840,7sleep00-21swapper/020:15:140
207080228145,9sleep20-21swapper/219:06:032
207088227958,14sleep30-21swapper/319:07:043
207135599600,55cyclictest1622-21rtkit-daemon23:29:400
162399590,9rtkit-daemon1622-21rtkit-daemon23:43:151
207136199559,42cyclictest1622-21rtkit-daemon22:42:351
21113112540,8chrt0-21swapper/221:00:112
21192022530,6sleep10-21swapper/121:20:311
207135599530,49cyclictest1622-21rtkit-daemon23:14:570
21342722510,6sleep00-21swapper/022:05:010
20873822510,13sleep1251rcuc/119:50:311
207137599481,15cyclictest2197014-21PK-Backend21:40:043
2071366994821,22cyclictest438-20systemd-journal23:09:552
2071375994720,23cyclictest2180036-40apt-check00:12:233
207136199473,24cyclictest2107456-21seq20:48:341
207136199473,24cyclictest2107456-21seq20:48:341
20767332460,5chrt0-21swapper/019:20:330
20767332460,5chrt0-21swapper/019:20:320
2071375994616,11cyclictest2105090-21awk20:40:213
207136199468,19cyclictest1753-21gnome-shell20:00:201
162399460,9rtkit-daemon1753-21gnome-shell00:29:362
162399460,4rtkit-daemon2197014-21PK-Backend21:08:573
162399460,4rtkit-daemon2197014-21PK-Backend21:08:573
207136199455,14cyclictest2124864-21taskset21:35:341
207136199452,18cyclictest1753-21gnome-shell19:10:271
207137599449,20cyclictest1753-21gnome-shell20:55:173
2071366994432,8cyclictest2107598-21munin-run20:50:002
2071366994432,8cyclictest2107598-21munin-run20:50:002
207136199447,14cyclictest2146490-21sort22:35:311
2071355994434,7cyclictest1069-21runrttasks20:59:010
162399440,23rtkit-daemon2197014-21PK-Backend22:52:252
2071375994325,14cyclictest3242255-21gdbus21:15:003
207137599432,11cyclictest2181653-31apt-check00:15:303
2071366994315,23cyclictest2093065-40lsb_release20:06:212
207136199439,29cyclictest2174415-21sshd23:55:331
207136199436,13cyclictest2114545-21pool-org.gnome.21:10:021
207136199436,13cyclictest2114545-21pool-org.gnome.21:10:011
207136199435,14cyclictest2121136-21runrttasks21:26:131
207136199434,34cyclictest2081048-21tr19:35:131
207136199434,13cyclictest2158804-21taskset23:10:371
2071361994335,6cyclictest251rcuc/122:34:241
2071361994313,26cyclictest2078945-21pool-org.gnome.19:30:001
207135599433,35cyclictest2147038-21kworker/0:222:40:140
21309962420,8chrt2197014-21PK-Backend21:55:132
207137599425,20cyclictest1753-21gnome-shell00:00:193
2071375994222,8cyclictest2166103-21munin-plugin-st23:35:013
207137599421,36cyclictest1753-21gnome-shell00:05:143
207137599421,18cyclictest21172722sh21:15:293
2071366994228,10cyclictest2182344-21cron00:20:012
2071366994222,14cyclictest855-21dbus-daemon22:25:042
2071366994222,14cyclictest855-21dbus-daemon22:25:042
2071366994219,18cyclictest2073264-21munin-node19:15:012
2071366994218,20cyclictest2180036-40apt-check00:12:172
207136199425,11cyclictest2183942-21seq00:23:191
2071361994229,9cyclictest2136010-21kworker/1:122:55:121
2071361994229,8cyclictest2099698-21grep20:25:271
2071361994229,8cyclictest2099698-21grep20:25:271
2071361994225,13cyclictest855-21dbus-daemon00:35:011
2071355994227,11cyclictest13-21ksoftirqd/022:05:400
2071355994227,11cyclictest13-21ksoftirqd/022:05:400
2071355994226,12cyclictest2151987-21sshd22:52:060
162399420,5rtkit-daemon0-21swapper/321:24:513
162399420,35rtkit-daemon2197014-21PK-Backend00:12:161
207137599417,16cyclictest1753-21gnome-shell22:37:573
2071375994115,23cyclictest2122375-40lsb_release21:30:213
207136199417,11cyclictest1753-21gnome-shell23:25:281
207136199416,18cyclictest1753-21gnome-shell22:46:261
207136199416,18cyclictest1753-21gnome-shell22:46:261
207136199415,15cyclictest2182122-21taskset00:17:141
2071361994125,11cyclictest438-20systemd-journal00:08:401
207135599412,21cyclictest1753-21gnome-shell21:25:160
2071355994122,15cyclictest2082631-21/usr/sbin/munin19:40:340
21349612400,7chrt2197014-21PK-Backend22:05:202
21349612400,7chrt2197014-21PK-Backend22:05:202
20978702400,6chrt2097120-21/usr/sbin/munin20:20:282
207137599401,35cyclictest1753-21gnome-shell19:39:353
2071366994023,12cyclictest2133979-21systemd-udevd22:02:262
207136699401,9cyclictest2197014-21PK-Backend00:24:282
2071366994013,13cyclictest438-20systemd-journal20:05:002
207136699401,20cyclictest3242253-21nautilus19:50:042
207136199407,10cyclictest1753-21gnome-shell22:50:491
207136199405,13cyclictest2117822-21taskset21:18:071
207136199405,10cyclictest2095372-21/usr/sbin/munin20:15:171
207136199404,14cyclictest2083836-21taskset19:41:361
207136199404,10cyclictest2100891-21/usr/sbin/munin20:30:311
2071361994022,14cyclictest2185099-21ntpq00:25:231
2071361994022,14cyclictest2135264-21ntpq22:05:261
2071361994022,14cyclictest2135264-21ntpq22:05:261
2071355994029,7cyclictest1069-21runrttasks22:14:510
2071355994024,11cyclictest438-20systemd-journal21:50:330
2071355994023,13cyclictest2117720-21sshd21:16:510
2071355994022,13cyclictest1-21systemd00:15:010
207137599397,28cyclictest1753-21gnome-shell22:01:113
207137599392,8cyclictest1753-21gnome-shell23:17:583
2071375993928,7cyclictest1753-21gnome-shell23:35:473
2071375993928,7cyclictest1069-21runrttasks21:39:263
2071375993925,10cyclictest3242253-21nautilus22:20:003
207137599391,10cyclictest2197014-21PK-Backend22:26:043
207137599391,10cyclictest2197014-21PK-Backend22:26:043
207136699398,11cyclictest1753-21gnome-shell00:35:022
207136699397,11cyclictest1753-21gnome-shell21:30:192
2071366993923,12cyclictest2085945-21gpgv19:50:022
207136699391,9cyclictest2197014-21PK-Backend23:46:152
2071366993917,5cyclictest141rcu_preempt23:50:312
207136199398,15cyclictest26-21ksoftirqd/119:30:151
207136199397,9cyclictest2156278-21expr23:05:141
207136199397,11cyclictest26-21ksoftirqd/121:48:361
207136199396,11cyclictest2176640-21cat00:05:021
207136199394,14cyclictest2132702-21seq22:00:061
207136199394,12cyclictest2106013-21sort20:45:021
2071361993931,6cyclictest251rcuc/121:01:281
2071361993930,6cyclictest251rcuc/121:51:311
2071361993924,11cyclictest2155120-21switchtime23:00:311
2071361993923,12cyclictest2075842-21grep19:20:151
2071361993923,12cyclictest2075842-21grep19:20:151
207136199392,15cyclictest2179-21gvfsd-metadata20:40:011
207136199392,14cyclictest2171104-21runrttasks23:47:361
2071361993919,7cyclictest1069-21runrttasks23:19:201
2071355993923,12cyclictest2109110-21sshd20:52:340
2071355993921,14cyclictest2185578-21apt-get00:26:430
2071355993920,14cyclictest2141123-40apt-check22:21:180
2071355993916,19cyclictest2072947-21apt-check19:12:180
207135599391,35cyclictest0-21swapper/020:01:050
21840412387,5sleep31753-21gnome-shell00:24:323
207137599388,25cyclictest2101089-21grep20:30:133
2071375993826,8cyclictest1753-21gnome-shell22:48:153
2071375993826,8cyclictest1753-21gnome-shell22:48:153
2071375993824,11cyclictest2100667-21awk20:30:003
2071375993824,11cyclictest2100667-21awk20:30:003
207137599382,32cyclictest1753-21gnome-shell20:46:363
207137599382,32cyclictest1753-21gnome-shell20:46:363
207137599382,32cyclictest1753-21gnome-shell19:40:563
2071375993820,13cyclictest855-21dbus-daemon19:10:023
207137599381,9cyclictest2197014-21PK-Backend23:07:503
2071375993810,14cyclictest42-21ksoftirqd/320:35:293
207136699387,13cyclictest1753-21gnome-shell21:20:302
2071366993826,8cyclictest2089656-21unin-run20:00:022
2071366993826,8cyclictest1753-21gnome-shell21:10:152
2071366993822,12cyclictest2168110-21apt23:40:142
2071366993821,12cyclictest438-20systemd-journal21:36:332
207136699381,9cyclictest2197014-21PK-Backend23:17:272
2071366993819,14cyclictest438-20systemd-journal20:11:332
207136199387,12cyclictest26-21ksoftirqd/121:14:401
207136199385,11cyclictest2098523-21seq20:24:071
207136199384,10cyclictest1753-21gnome-shell00:37:191
207136199383,11cyclictest2126925-21runrttasks21:41:261
207136199382,8cyclictest1753-21gnome-shell19:16:191
2071361993820,14cyclictest2089188-21timerwakeupswit19:55:291
2071361993815,19cyclictest2164648-21cpu23:30:131
207135599382,8cyclictest2100422-40lsb_release20:27:270
207135599382,8cyclictest2100422-40lsb_release20:27:270
2071355993826,8cyclictest1753-21gnome-shell19:28:390
207135599382,32cyclictest1753-21gnome-shell22:59:440
2071355993823,11cyclictest1358-1kworker/u9:123:02:330
2071355993822,11cyclictest2115437-21grep21:10:230
2071355993818,15cyclictest1827-21JS14:31:440
2071355993818,15cyclictest1827-21JS14:31:440
207135599381,12cyclictest1069-21runrttasks23:20:020
162399380,5rtkit-daemon0-21swapper/222:10:282
21215652370,7sleep22121566-21latency_hist21:30:022
207137599373,16cyclictest1753-21gnome-shell23:55:283
2071375993725,8cyclictest2077632-21tr19:25:163
207137599372,31cyclictest1753-21gnome-shell20:10:393
2071375993722,11cyclictest42-21ksoftirqd/300:25:123
2071375993721,12cyclictest2147478-21grep22:40:153
2071375993721,12cyclictest2085843-21apt-get19:50:043
2071375993721,11cyclictest2121616-21apt-key21:30:023
2071375993720,13cyclictest2164078-21sshd23:26:333
2071375993717,8cyclictest141rcu_preempt20:51:583
2071375993713,20cyclictest0-21swapper/319:20:403
2071375993713,20cyclictest0-21swapper/319:20:393
2071375993712,22cyclictest0-21swapper/322:23:173
207136699377,26cyclictest1069-21runrttasks20:43:522
207136699376,9cyclictest141rcu_preempt22:38:052
2071366993725,8cyclictest1753-21gnome-shell22:45:002
2071366993721,11cyclictest2128986-21cron21:50:002
2071366993719,13cyclictest2177547-21ntpq00:05:252
2071361993719,13cyclictest2122347-21expr21:30:181
2071355993727,6cyclictest2093395-21latency_hist20:10:000
207135599372,6cyclictest0-21swapper/000:22:340
2071355993726,7cyclictest2166101-21munin-plugin-st23:35:010
2071355993726,7cyclictest1753-21gnome-shell00:32:180
2071355993725,8cyclictest1753-21gnome-shell21:55:520
207135599372,31cyclictest2098230-21users20:20:340
2071355993723,10cyclictest2087762-21sshd19:53:330
2071355993722,10cyclictest1753-21gnome-shell22:46:590
2071355993722,10cyclictest1753-21gnome-shell22:46:590
2071355993719,13cyclictest855-21dbus-daemon00:00:030
2071355993719,13cyclictest2181272-21sh00:15:220
2071355993710,7cyclictest141rcu_preempt23:10:000
162399370,26rtkit-daemon0-21swapper/219:37:102
2071375993623,9cyclictest1753-21gnome-shell22:54:033
2071375993619,13cyclictest1753-21gnome-shell21:52:423
2071375993619,13cyclictest1753-21gnome-shell20:09:113
2071375993619,12cyclictest2189555-21sshd00:38:443
2071375993618,13cyclictest2084754-21df_inode19:45:163
2071375993612,20cyclictest391irq_work/323:46:203
207136699364,10cyclictest2140708-21cat22:20:232
207136699361,8cyclictest2197014-21PK-Backend19:15:552
2071366993618,13cyclictest3242255-21gdbus23:05:012
2071366993618,13cyclictest2150495-21apt22:50:012
2071366993618,13cyclictest2150495-21apt22:50:012
2071366993610,22cyclictest34-21ksoftirqd/220:27:132
2071366993610,22cyclictest34-21ksoftirqd/220:27:132
207136199366,26cyclictest2166350-21/usr/sbin/munin23:35:191
207136199366,14cyclictest26-21ksoftirqd/120:05:411
207136199364,10cyclictest26-21ksoftirqd/123:21:131
207136199364,10cyclictest2140554-21memory22:20:211
2071361993628,6cyclictest251rcuc/122:16:461
2071361993623,9cyclictest1753-21gnome-shell23:50:161
2071361993621,11cyclictest2136423-21grep22:10:131
2071361993620,12cyclictest2110146-21cat20:55:211
2071361993619,13cyclictest2093758-21wc20:10:121
2071361993613,8cyclictest26-21ksoftirqd/120:54:141
2071355993625,7cyclictest2175950-21smartctl00:00:280
207135599362,30cyclictest1753-21gnome-shell19:16:540
2071355993622,10cyclictest2112476-21sshd21:01:180
2071355993621,11cyclictest2178286-21dpkg00:10:000
2071355993621,11cyclictest2161311-21grep23:20:130
2071355993619,13cyclictest855-21dbus-daemon00:40:020
2071355993619,12cyclictest438-20systemd-journal23:45:000
2071355993619,12cyclictest438-20systemd-journal20:35:010
2071355993619,12cyclictest2102957-21expr20:35:170
2071355993616,15cyclictest2088528-21df_inode19:55:150
162399360,29rtkit-daemon0-21swapper/023:52:480
207137599353,28cyclictest2157830-21cpuspeed_turbos23:10:183
2071375993523,8cyclictest1753-21gnome-shell20:17:063
2071375993523,8cyclictest1753-21gnome-shell19:17:443
2071375993522,9cyclictest1753-21gnome-shell21:58:003
2071375993520,11cyclictest2098340-21sshd20:21:473
2071375993519,12cyclictest2155700-21munin-plugin-st23:05:013
2071375993519,12cyclictest2127754-21expr21:45:133
2071375993519,11cyclictest2112967-21date21:05:023
2071366993522,9cyclictest2138625-21needreboot22:15:232
2071366993520,11cyclictest855-21dbus-daemon23:21:562
207136699351,9cyclictest2197014-21PK-Backend20:33:082
207136699351,8cyclictest0-21swapper/219:30:252
207136699351,30cyclictest2197014-21PK-Backend20:55:142
207136699351,29cyclictest2197014-21PK-Backend19:20:292
207136699351,29cyclictest2197014-21PK-Backend19:20:292
207136699351,16cyclictest141rcu_preempt00:00:022
2071366993510,5cyclictest141rcu_preempt23:11:492
2071366993510,21cyclictest2153963-21sh23:00:002
207136199356,11cyclictest26-21ksoftirqd/121:55:401
2071361993520,11cyclictest2084727-21expr19:45:141
2071361993513,17cyclictest1632-21dbus-daemon22:25:031
2071361993513,17cyclictest1632-21dbus-daemon22:25:031
207135599358,23cyclictest2962348-21rs:main0
207135599352,30cyclictest0-21swapper/023:47:160
2071355993519,11cyclictest2084806-21basename19:45:190
2071355993518,12cyclictest2081181-21cat19:35:130
207135599351,7cyclictest2080651-21turbostat19:35:000
2071355993516,14cyclictest2121982-40apt-check21:30:180
207135599351,5cyclictest0-21swapper/022:15:570
2071355993515,15cyclictest2126991-40apt-check21:42:190
207137599348,23cyclictest1753-21gnome-shell23:21:123
207137599344,5cyclictest141rcu_preempt23:50:303
2071375993422,7cyclictest854-21avahi-daemon22:31:033
2071375993420,9cyclictest2137415-21sshd22:11:323
2071366993421,9cyclictest1753-21gnome-shell21:44:592
207136699342,15cyclictest141rcu_preempt19:25:222
207136699341,9cyclictest2197014-21PK-Backend20:52:442
2071366993419,11cyclictest2163441-21sed23:25:232
2071366993418,12cyclictest2188915-21awk00:35:252
2071366993417,13cyclictest1753-21gnome-shell22:33:072
207136699341,30cyclictest2197014-21PK-Backend23:35:582
207136699341,30cyclictest2197014-21PK-Backend19:44:532
207136699341,29cyclictest2197014-21PK-Backend21:15:152
207136699341,10cyclictest2197014-21PK-Backend20:35:292
207135599343,14cyclictest141rcu_preempt22:34:030
207135599342,29cyclictest0-21swapper/022:37:130
2071355993420,10cyclictest2105435-21cat20:40:280
2071355993415,14cyclictest13-21ksoftirqd/020:45:200
2071355993415,14cyclictest13-21ksoftirqd/020:45:200
207135599341,28cyclictest2113179-21cpuspeed_turbos21:05:130
207135599341,28cyclictest2113179-21cpuspeed_turbos21:05:130
2071375993321,8cyclictest1753-21gnome-shell22:06:493
2071375993321,8cyclictest1753-21gnome-shell22:06:493
2071375993318,11cyclictest2168481-21df_inode23:40:183
207137599331,28cyclictest2079533-21expr19:30:163
2071355993318,11cyclictest2124501-21cat21:35:270
2071355993315,13cyclictest438-20systemd-journal21:50:000
162399330,5rtkit-daemon0-21swapper/221:09:232
162399330,5rtkit-daemon0-21swapper/221:09:232
2071375993221,7cyclictest2187128-21cut00:30:243
2071375993216,11cyclictest42-21ksoftirqd/320:00:313
207137599321,28cyclictest0-21swapper/322:55:083
2071375993210,18cyclictest1753-21gnome-shell19:55:563
2071366993221,7cyclictest1069-21runrttasks20:15:582
207136699321,11cyclictest2197014-21PK-Backend00:03:312
2071355993220,9cyclictest1753-21gnome-shell23:36:300
2071355993216,12cyclictest2094733-21cat20:10:290
2071366993123,5cyclictest2197014-21PK-Backend23:30:402
2071366993120,7cyclictest1753-21gnome-shell21:50:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional