You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-13 - 07:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Jul 13, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
225473822390,13sleep0161rcuc/019:05:300
225434221190,10sleep1261ksoftirqd/119:05:241
235851321130,4sleep20-21swapper/200:00:152
22553579910812,10cyclictest331rcuc/219:49:242
1623991020,9rtkit-daemon1622-21rtkit-daemon22:53:511
2255359991012,94cyclictest2282549-21dpkg20:25:023
225535999992,92cyclictest2274048-21/usr/sbin/munin20:00:173
225535799983,90cyclictest2282979-21diskstats20:25:192
225535999972,91cyclictest2281998-21runrttasks20:24:193
162399970,8rtkit-daemon1622-21rtkit-daemon23:57:051
225535799962,90cyclictest2260800-21seq19:22:532
2255353999518,62cyclictest26-21ksoftirqd/120:50:271
162399950,8rtkit-daemon1622-21rtkit-daemon21:08:301
2255357999485,5cyclictest2197014-21PK-Backend19:33:422
225535799942,88cyclictest0-21swapper/219:29:532
225534899942,88cyclictest2263408-21cat19:30:200
22694412930,6sleep10-21swapper/119:45:311
2255359999385,5cyclictest0-21swapper/319:29:283
225535999932,87cyclictest2265429-21awk19:35:233
2255357999384,5cyclictest0-21swapper/219:40:022
225535799932,87cyclictest0-21swapper/220:12:192
225535799932,6cyclictest2197014-21PK-Backend20:54:062
225535799932,6cyclictest2197014-21PK-Backend19:50:582
225535799932,6cyclictest0-21swapper/220:46:232
225535799932,6cyclictest0-21swapper/220:41:102
2255353999319,18cyclictest26-21ksoftirqd/119:44:451
2255359999284,5cyclictest0-21swapper/320:54:033
2255357999285,4cyclictest0-21swapper/220:20:002
2255357999285,4cyclictest0-21swapper/220:20:002
225535799922,7cyclictest0-21swapper/220:22:532
162399920,9rtkit-daemon1622-21rtkit-daemon22:46:191
2255357999183,5cyclictest0-21swapper/220:36:252
2255357999183,5cyclictest0-21swapper/220:30:222
2255357999183,5cyclictest0-21swapper/220:05:442
2255357999183,5cyclictest0-21swapper/219:18:392
2255357999182,5cyclictest2197014-21PK-Backend19:35:252
225535799912,7cyclictest0-21swapper/220:00:162
225535799912,6cyclictest0-21swapper/219:55:142
162399910,6rtkit-daemon2197014-21PK-Backend21:57:412
2255359999082,5cyclictest0-21swapper/319:58:493
2255359999082,5cyclictest0-21swapper/319:53:203
2255359999081,6cyclictest0-21swapper/320:55:073
2255359999081,6cyclictest0-21swapper/320:55:073
162399900,9rtkit-daemon1622-21rtkit-daemon00:10:541
162399900,9rtkit-daemon1622-21rtkit-daemon00:10:541
162399900,8rtkit-daemon1622-21rtkit-daemon21:51:261
162399900,8rtkit-daemon1622-21rtkit-daemon00:02:131
2255359998981,5cyclictest0-21swapper/320:47:133
162399890,8rtkit-daemon1622-21rtkit-daemon20:57:471
162399890,8rtkit-daemon1622-21rtkit-daemon20:57:471
2255353998620,7cyclictest26-21ksoftirqd/120:20:561
162399850,6rtkit-daemon1622-21rtkit-daemon21:41:021
162399850,6rtkit-daemon1622-21rtkit-daemon21:41:021
23005812840,5sleep02197014-21PK-Backend21:15:230
2255348998434,45cyclictest1753-21gnome-shell19:43:250
2255359998375,5cyclictest0-21swapper/320:10:063
2255359998274,5cyclictest0-21swapper/319:49:243
2255359998173,5cyclictest2264453-21perf19:35:003
2255353998173,5cyclictest0-21swapper/119:30:211
2255348998131,46cyclictest1753-21gnome-shell19:56:320
2255348998069,7cyclictest1753-21gnome-shell20:28:320
2255348998032,44cyclictest1753-21gnome-shell19:20:430
162399800,6rtkit-daemon2197014-21PK-Backend23:36:572
225535799792,19cyclictest1753-21gnome-shell20:55:152
225535799792,19cyclictest1753-21gnome-shell20:55:152
2255353997928,9cyclictest1753-21gnome-shell19:51:091
2255348997930,45cyclictest1753-21gnome-shell20:13:260
2255348997928,47cyclictest1753-21gnome-shell20:05:560
225534899791,9cyclictest1753-21gnome-shell20:48:040
2255348997829,45cyclictest1753-21gnome-shell19:39:330
2255348997828,45cyclictest1753-21gnome-shell20:34:120
2255348997827,46cyclictest1753-21gnome-shell19:54:400
2255359997769,5cyclictest0-21swapper/320:44:353
2255348997765,7cyclictest2268846-21df19:45:170
225487427739,10sleep20-21swapper/219:05:502
162399770,5rtkit-daemon0-21swapper/219:13:192
2255353997611,9cyclictest26-21ksoftirqd/120:40:381
2255348997624,48cyclictest1753-21gnome-shell19:28:080
225535399759,10cyclictest26-21ksoftirqd/119:21:501
162399750,6rtkit-daemon2197014-21PK-Backend23:31:282
2255359997466,5cyclictest0-21swapper/319:18:303
2255348997422,41cyclictest1753-21gnome-shell19:15:080
2255359997365,5cyclictest2197014-21PK-Backend20:15:203
2255359997365,5cyclictest2197014-21PK-Backend20:15:203
225535399736,10cyclictest26-21ksoftirqd/119:56:401
2255359997265,4cyclictest0-21swapper/320:37:323
225535399729,42cyclictest26-21ksoftirqd/120:10:151
225535399728,8cyclictest26-21ksoftirqd/119:15:511
2255353997211,58cyclictest0-21swapper/120:37:441
2255348997221,9cyclictest1753-21gnome-shell19:13:130
225534899711,14cyclictest2197014-21PK-Backend20:41:490
225535399704,10cyclictest26-21ksoftirqd/119:29:231
2255348997062,4cyclictest2197014-21PK-Backend20:00:090
225535999691,12cyclictest2197014-21PK-Backend19:21:033
2255357996919,28cyclictest1069-21runrttasks21:07:392
2255353996935,13cyclictest141rcu_preempt20:30:451
225534899692,15cyclictest2197014-21PK-Backend20:20:510
225535999682,16cyclictest2197014-21PK-Backend20:35:023
225535399684,8cyclictest26-21ksoftirqd/120:01:201
2255359996757,7cyclictest1069-21runrttasks19:13:113
2255353996738,14cyclictest141rcu_preempt19:35:171
225518826747,14sleep30-21swapper/319:09:523
2255359996659,5cyclictest0-21swapper/320:09:583
2255359996659,5cyclictest0-21swapper/319:43:203
2255353996639,7cyclictest141rcu_preempt20:16:531
2255353996639,7cyclictest141rcu_preempt20:16:521
162399660,8rtkit-daemon1622-21rtkit-daemon21:34:061
2255348996556,6cyclictest13-21ksoftirqd/020:38:120
2255348996530,12cyclictest0-21swapper/020:54:360
2255353996436,13cyclictest141rcu_preempt20:46:561
225535399643,41cyclictest2257057-21munin-run19:15:021
225535399632,19cyclictest1753-21gnome-shell20:27:161
225535399632,19cyclictest1753-21gnome-shell20:09:591
2255348996343,14cyclictest2278962-21cat20:15:160
2255348996343,14cyclictest2278962-21cat20:15:150
225535799627,18cyclictest1753-21gnome-shell22:30:312
162399620,10rtkit-daemon1622-21rtkit-daemon22:47:520
162399590,7rtkit-daemon1622-21rtkit-daemon23:38:060
162399580,8rtkit-daemon1622-21rtkit-daemon21:00:461
225534899563,49cyclictest1753-21gnome-shell20:55:280
225534899563,49cyclictest1753-21gnome-shell20:55:280
2255357995530,21cyclictest1753-21gnome-shell00:30:012
2255357995521,29cyclictest855-21dbus-daemon21:30:012
2255357995518,31cyclictest1753-21gnome-shell23:40:312
2255357995414,35cyclictest2360015-21grep00:05:182
2255348995443,8cyclictest13-21ksoftirqd/022:15:540
2255357995323,25cyclictest2308366-21idleruntime-cro21:40:012
2255353995325,23cyclictest2336990-31apt-check23:00:121
2255359995222,7cyclictest42-21ksoftirqd/322:50:113
2255357995228,20cyclictest1753-21gnome-shell23:25:012
2255357995213,34cyclictest2302681-21grep21:20:322
2255357995120,26cyclictest2332031-21kernelversion22:45:222
2255357995113,33cyclictest2323289-21awk22:20:272
162399510,8rtkit-daemon1622-21rtkit-daemon22:53:200
162399510,8rtkit-daemon1622-21rtkit-daemon21:42:210
162399510,8rtkit-daemon1622-21rtkit-daemon21:42:210
23262492500,14sleep342-21ksoftirqd/322:30:053
2255357995014,31cyclictest2318780-21cron22:10:002
2255357995011,12cyclictest438-20systemd-journal23:00:022
225535999494,20cyclictest2341948-21cron23:14:593
2255357994924,20cyclictest1753-21gnome-shell23:55:022
2255357994915,29cyclictest855-21dbus-daemon00:00:032
225535999483,10cyclictest2197014-21PK-Backend00:40:013
225535799488,17cyclictest1753-21gnome-shell22:52:132
225535799488,17cyclictest1753-21gnome-shell00:15:552
225535799481,23cyclictest1753-21gnome-shell23:05:172
225535799481,23cyclictest1753-21gnome-shell23:05:172
225535999475,13cyclictest2323118-21sed22:20:243
225535999472,11cyclictest42-21ksoftirqd/321:40:013
225535999471,12cyclictest42-21ksoftirqd/323:35:003
2255357994717,25cyclictest2347451-21cron23:30:012
2255357994717,25cyclictest2347451-21cron23:30:012
2255357994711,12cyclictest2319634-21latency22:10:212
2255357994710,19cyclictest1753-21gnome-shell22:57:422
2255357994710,17cyclictest1753-21gnome-shell23:16:152
225535999461,15cyclictest2319769-21/usr/sbin/munin22:10:243
225535999461,11cyclictest42-21ksoftirqd/322:45:243
225535799466,20cyclictest1753-21gnome-shell22:45:012
225535799462,7cyclictest1753-21gnome-shell21:30:272
2255357994618,25cyclictest23008282sleep221:15:292
2255353994622,20cyclictest2326273-40apt-check22:30:181
2255353994618,23cyclictest2360625-21ntpq00:05:301
225535999454,17cyclictest2364923-21cut00:20:013
225535799459,17cyclictest1753-21gnome-shell23:45:442
225535799459,17cyclictest1753-21gnome-shell22:37:562
225535799459,17cyclictest1753-21gnome-shell21:45:182
2255357994513,27cyclictest2365060-21/usr/sbin/munin00:20:152
2255357994512,29cyclictest1753-21gnome-shell00:40:022
2255357994512,28cyclictest2362105-21kernelversion00:10:232
2255357994512,28cyclictest2362105-21kernelversion00:10:222
2255353994515,14cyclictest2338765-21apt23:05:141
2255353994515,14cyclictest2338765-21apt23:05:131
225534899454,11cyclictest0-21swapper/021:57:250
22986012440,9sleep21753-21gnome-shell21:10:182
225535999446,16cyclictest2302918-21taskset21:22:153
225535999444,23cyclictest1069-21runrttasks21:07:183
2255357994415,25cyclictest34-21ksoftirqd/222:25:252
2255357994415,24cyclictest2340705-21cstates23:10:152
225535799441,14cyclictest438-20systemd-journal21:05:002
2255357994411,29cyclictest875-21polkitd22:15:562
2255357994410,15cyclictest1753-21gnome-shell21:52:492
23504022430,7sleep32197014-21PK-Backend23:35:333
225535999435,21cyclictest2342202-21/usr/sbin/munin23:15:153
225535999431,11cyclictest2351198-21apt23:40:143
225535799432,17cyclictest1753-21gnome-shell22:00:112
225535799431,13cyclictest2308628-21grep21:40:172
225535799431,13cyclictest2308628-21grep21:40:162
2255348994332,8cyclictest13-21ksoftirqd/000:00:000
2255348994325,13cyclictest2336990-40apt-check23:00:170
225535999421,9cyclictest2197014-21PK-Backend23:00:303
225535999421,11cyclictest42-21ksoftirqd/323:29:563
225535999421,11cyclictest42-21ksoftirqd/323:29:553
225535999421,11cyclictest1753-21gnome-shell21:40:163
225535999421,11cyclictest1753-21gnome-shell21:40:153
225535999421,10cyclictest2318800-21munin-run22:10:013
2255357994213,25cyclictest2368636-21cpuspeed00:30:162
2255357994213,25cyclictest2368636-21cpuspeed00:30:162
2255353994226,12cyclictest2336393-21sshd22:59:241
2255353994222,15cyclictest3242253-21nautilus23:40:021
2255353994215,23cyclictest2343231-21sed23:15:321
2255353994213,11cyclictest26-21ksoftirqd/122:10:161
2255353994212,11cyclictest26-21ksoftirqd/123:21:181
2255348994210,7cyclictest141rcu_preempt00:27:270
23079422410,7sleep1251rcuc/121:35:331
225535999417,18cyclictest1753-21gnome-shell23:53:443
225535999417,18cyclictest1753-21gnome-shell00:21:023
225535999417,17cyclictest2311224-21python321:45:303
225535999414,22cyclictest2325749-21cron22:30:003
225535999414,10cyclictest2361506-21wc00:10:033
225535999413,19cyclictest2314742-21grep21:55:313
225535999413,10cyclictest854-21avahi-daemon00:00:003
225535999411,9cyclictest2197014-21PK-Backend22:56:353
225535999411,16cyclictest2306648-21munin-run21:35:013
225535999411,10cyclictest42-21ksoftirqd/321:50:183
2255353994119,18cyclictest2348286-21cat23:30:261
2255353994115,11cyclictest2316679-21uptime22:00:341
225535399411,36cyclictest1622-21rtkit-daemon21:19:131
2255353994113,11cyclictest26-21ksoftirqd/123:13:571
225535399411,28cyclictest1753-21gnome-shell23:27:121
225535399411,28cyclictest1753-21gnome-shell23:27:111
2255348994128,8cyclictest1753-21gnome-shell22:26:460
2255348994126,10cyclictest1753-21gnome-shell23:44:000
2255348994123,13cyclictest438-20systemd-journal21:04:240
2255348994112,6cyclictest141rcu_preempt22:35:240
2255348994111,7cyclictest141rcu_preempt22:05:580
2255348994110,6cyclictest141rcu_preempt00:19:510
225535999405,20cyclictest2197014-21PK-Backend00:32:083
225535999405,20cyclictest2197014-21PK-Backend00:32:073
225535999403,8cyclictest2328050-21/usr/sbin/munin22:35:333
225535999403,15cyclictest42-21ksoftirqd/322:00:213
225535999403,11cyclictest42-21ksoftirqd/300:10:153
225535999403,11cyclictest42-21ksoftirqd/300:10:153
225535999403,10cyclictest42-21ksoftirqd/321:00:253
225535999401,10cyclictest2197014-21PK-Backend22:40:233
2255359994010,16cyclictest42-21ksoftirqd/321:12:513
225535399407,19cyclictest26-21ksoftirqd/123:45:181
225535399401,7cyclictest2321668-21python322:15:311
2255353994012,12cyclictest26-21ksoftirqd/121:24:051
225534899402,7cyclictest1753-21gnome-shell00:34:230
225534899402,7cyclictest1753-21gnome-shell00:34:220
225534899401,35cyclictest2353334-21sshd23:45:240
225534899401,11cyclictest2326807-21irqstats22:30:250
225535999391,17cyclictest42-21ksoftirqd/321:27:353
225535999391,10cyclictest2197014-21PK-Backend23:07:013
225535999391,10cyclictest2197014-21PK-Backend23:07:003
225535999391,10cyclictest2197014-21PK-Backend21:15:143
225535399399,17cyclictest26-21ksoftirqd/121:55:281
225535399398,19cyclictest141rcu_preempt00:30:321
225535399398,19cyclictest141rcu_preempt00:30:321
225535399398,18cyclictest26-21ksoftirqd/122:05:191
225535399398,18cyclictest26-21ksoftirqd/100:25:211
2255353993921,14cyclictest2370499-21cat00:35:161
225535399391,21cyclictest2364633-21taskset00:17:551
2255348993913,11cyclictest141rcu_preempt23:05:140
2255348993913,11cyclictest141rcu_preempt23:05:130
225535999385,12cyclictest42-21ksoftirqd/323:48:283
225535999382,12cyclictest42-21ksoftirqd/300:00:173
225535999381,10cyclictest2197014-21PK-Backend22:15:553
225535999381,10cyclictest2197014-21PK-Backend00:30:023
225535399387,18cyclictest26-21ksoftirqd/100:20:241
225535399382,7cyclictest2324894-21memory22:25:281
2255353993812,15cyclictest26-21ksoftirqd/121:45:131
225534899382,8cyclictest1753-21gnome-shell00:01:530
225534899382,31cyclictest1753-21gnome-shell21:54:090
2255348993820,13cyclictest2298207-21find21:10:110
225534899381,13cyclictest0-21swapper/023:54:590
225535399372,17cyclictest2330198-21hddtemp_smartct22:40:181
2255353993720,13cyclictest2354940-21expr23:50:161
225535399371,12cyclictest2304789-21chrt21:29:001
225534899372,31cyclictest1753-21gnome-shell23:31:470
2255348993722,11cyclictest2364799-21cron00:24:550
2255348993716,16cyclictest3242253-21nautilus23:00:030
225535999367,12cyclictest2197014-21PK-Backend23:23:483
225535399366,14cyclictest26-21ksoftirqd/122:20:251
2255353993620,12cyclictest2298416-21expr21:10:151
2255353993619,13cyclictest1753-21gnome-shell23:40:231
225535399361,13cyclictest1753-21gnome-shell22:37:481
225534899362,8cyclictest1753-21gnome-shell00:10:290
225534899362,8cyclictest1753-21gnome-shell00:10:280
2255348993625,7cyclictest1069-21runrttasks23:25:120
2255348993625,7cyclictest1069-21runrttasks23:25:110
2255348993619,12cyclictest2344810-21sshd23:20:230
2255348993618,13cyclictest438-20systemd-journal22:25:000
2255348993618,12cyclictest2303378-21sed21:25:030
2255348993613,5cyclictest141rcu_preempt22:44:080
225534899352,7cyclictest2303967-21/usr/sbin/munin21:25:230
225534899352,28cyclictest1753-21gnome-shell00:08:530
2255348993520,11cyclictest13-21ksoftirqd/023:16:550
2255348993519,11cyclictest2370258-21find00:35:120
2255348993518,12cyclictest2296500-21cpu21:05:150
2255348993517,14cyclictest2305685-21cat21:30:220
2255348993516,14cyclictest3242253-21nautilus21:50:020
2255348993516,14cyclictest2307259-21df_inode21:35:170
2255348993511,6cyclictest141rcu_preempt22:01:340
225534899351,12cyclictest0-21swapper/022:10:230
2255348993318,11cyclictest2340995-21sed23:10:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional