You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 11:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Jul 12, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17497152980,7sleep30-21swapper/322:15:313
162399840,5rtkit-daemon0-21swapper/100:05:461
17523302830,8sleep12197014-21PK-Backend22:25:031
17523302830,8sleep12197014-21PK-Backend22:25:031
162399820,9rtkit-daemon1791627-21ntp_states00:15:261
162399820,14rtkit-daemon1938-21gsd-housekeepin00:21:071
162399820,14rtkit-daemon1938-21gsd-housekeepin00:21:071
168325199789,18cyclictest1753-21gnome-shell00:30:222
162399770,6rtkit-daemon2197014-21PK-Backend00:13:521
162399770,5rtkit-daemon0-21swapper/123:40:451
162399770,5rtkit-daemon0-21swapper/123:35:121
168288427646,22sleep30-21swapper/319:07:153
168279727139,9sleep10-21swapper/119:06:091
168300326848,14sleep00-21swapper/019:08:450
1683251996514,19cyclictest1753-21gnome-shell21:30:132
168162326542,16sleep20-21swapper/219:05:102
1683251996215,18cyclictest1753-21gnome-shell21:35:272
1683251996215,18cyclictest1753-21gnome-shell21:35:262
162399620,9rtkit-daemon1622-21rtkit-daemon20:27:011
1683251996115,18cyclictest1753-21gnome-shell00:25:132
1683251996115,18cyclictest1753-21gnome-shell00:25:122
162399610,8rtkit-daemon1622-21rtkit-daemon22:38:541
162399610,8rtkit-daemon1622-21rtkit-daemon21:24:141
162399600,9rtkit-daemon1622-21rtkit-daemon21:50:511
168325199596,24cyclictest1753-21gnome-shell20:35:262
168325199596,24cyclictest1753-21gnome-shell20:35:262
1683251995914,17cyclictest1753-21gnome-shell20:50:252
1683251995912,30cyclictest1753-21gnome-shell19:24:422
162399590,9rtkit-daemon1622-21rtkit-daemon21:12:391
162399590,9rtkit-daemon1622-21rtkit-daemon20:09:331
162399590,9rtkit-daemon1622-21rtkit-daemon20:09:321
162399590,40rtkit-daemon0-21swapper/120:42:501
1683251995815,23cyclictest1753-21gnome-shell20:24:212
1683251995813,27cyclictest1753-21gnome-shell22:55:212
1683251995813,18cyclictest1753-21gnome-shell22:45:292
1683251995810,18cyclictest1753-21gnome-shell22:30:162
1683251995810,17cyclictest1753-21gnome-shell23:45:012
162399580,9rtkit-daemon1622-21rtkit-daemon22:18:371
162399580,9rtkit-daemon1622-21rtkit-daemon20:14:300
168325199577,26cyclictest1753-21gnome-shell22:39:202
1683251995715,24cyclictest1753-21gnome-shell22:43:392
1683251995714,24cyclictest1753-21gnome-shell23:06:052
1683251995713,26cyclictest1753-21gnome-shell21:14:402
1683251995713,26cyclictest1753-21gnome-shell20:05:242
1683251995713,26cyclictest1753-21gnome-shell20:05:232
1683251995713,17cyclictest1753-21gnome-shell19:45:162
1683251995712,24cyclictest1753-21gnome-shell21:47:292
1683251995712,24cyclictest1753-21gnome-shell20:04:252
168324199579,43cyclictest1769802-21perf23:15:000
168325199567,30cyclictest1753-21gnome-shell20:26:492
1683251995615,24cyclictest1753-21gnome-shell23:53:092
1683251995615,23cyclictest1753-21gnome-shell19:32:202
1683251995613,17cyclictest1753-21gnome-shell23:45:162
1683251995612,26cyclictest1753-21gnome-shell19:12:292
1683251995612,26cyclictest1753-21gnome-shell00:39:332
1683251995612,23cyclictest1753-21gnome-shell22:07:272
1683251995612,18cyclictest1753-21gnome-shell21:15:272
1683251995612,18cyclictest1753-21gnome-shell21:15:262
1683251995611,23cyclictest1753-21gnome-shell23:32:382
162399560,9rtkit-daemon1622-21rtkit-daemon20:32:511
1683251995514,24cyclictest1753-21gnome-shell21:29:542
1683251995514,23cyclictest1753-21gnome-shell19:25:282
1683251995513,24cyclictest1753-21gnome-shell23:37:342
1683251995513,17cyclictest1753-21gnome-shell20:10:202
1683251995513,13cyclictest1753-21gnome-shell20:55:152
1683251995512,23cyclictest1753-21gnome-shell22:50:372
1683251995512,15cyclictest1693242-21/usr/sbin/munin19:35:242
1683251995511,24cyclictest1753-21gnome-shell21:59:562
1683251995511,23cyclictest1753-21gnome-shell00:07:422
162399550,8rtkit-daemon1622-21rtkit-daemon21:18:511
162399550,8rtkit-daemon1622-21rtkit-daemon21:18:501
168325599542,32cyclictest854-21avahi-daemon19:42:513
168325199549,19cyclictest1753-21gnome-shell22:15:012
168325199549,19cyclictest1753-21gnome-shell00:22:062
168325199549,19cyclictest1753-21gnome-shell00:22:052
168325199548,21cyclictest1713126-21sensors_temp20:30:272
168325199548,19cyclictest1753-21gnome-shell20:44:322
168325199545,12cyclictest1700231-21hddtemp_smartct19:55:232
1683251995414,23cyclictest1753-21gnome-shell21:53:142
1683251995413,16cyclictest1753-21gnome-shell23:15:262
1683251995410,24cyclictest1753-21gnome-shell19:19:432
168325199540,50cyclictest1622-21rtkit-daemon00:18:262
168324699540,50cyclictest1622-21rtkit-daemon20:03:491
1683241995417,11cyclictest438-20systemd-journal21:05:220
1683251995313,21cyclictest1753-21gnome-shell00:11:402
1683251995312,26cyclictest1753-21gnome-shell21:40:292
1683251995312,17cyclictest1698756-21needreboot19:50:252
168324699530,49cyclictest1622-21rtkit-daemon20:57:511
1683241995320,28cyclictest1727740-21cut21:15:020
168325199527,20cyclictest1765216-21needreboot23:00:232
168325199527,16cyclictest1753-21gnome-shell23:27:252
1683251995213,23cyclictest1753-21gnome-shell23:56:402
1683241995210,13cyclictest2181-21gdbus23:05:010
162399520,8rtkit-daemon1622-21rtkit-daemon21:29:440
16978972510,7sleep02197014-21PK-Backend19:50:020
1683251995113,20cyclictest1753-21gnome-shell21:03:272
1683251995112,23cyclictest1753-21gnome-shell00:03:482
1683251995111,23cyclictest1753-21gnome-shell20:17:412
1683241995117,11cyclictest1747556-21cat22:10:160
1683251995013,14cyclictest1769776-21chrt23:14:532
1683251995011,15cyclictest1772671-21sensors23:20:292
1683241995016,10cyclictest1702056-21cut20:00:230
1683241995016,10cyclictest1691164-21expr19:30:140
168324199501,28cyclictest13-21ksoftirqd/020:40:170
162399500,43rtkit-daemon881143-21gmain20:11:201
17424872490,8sleep02197014-21PK-Backend21:55:310
168325199499,16cyclictest1753-21gnome-shell22:15:232
168325199491,14cyclictest1752423-21/usr/sbin/munin22:25:162
1683241994920,24cyclictest1069-21runrttasks23:54:320
1683241994918,8cyclictest1753-21gnome-shell00:35:280
1683241994917,12cyclictest1779134-21which23:40:190
1683241994916,11cyclictest13-21ksoftirqd/019:40:200
1683241994915,11cyclictest1758571-21cat22:40:330
1683241994914,13cyclictest1797487-21awk00:30:340
1683241994913,10cyclictest1785001-21uptime23:55:310
1683241994912,13cyclictest1632-21dbus-daemon21:25:010
1683255994836,9cyclictest42-21ksoftirqd/322:25:023
1683255994836,9cyclictest42-21ksoftirqd/322:25:013
168325199486,15cyclictest1753-21gnome-shell21:22:302
168325199485,16cyclictest1724378-21apt21:05:132
168324199489,16cyclictest1753-21gnome-shell22:16:450
1683241994818,11cyclictest1703846-21cat20:05:240
1683241994818,11cyclictest1703846-21cat20:05:240
1683241994813,9cyclictest1787352-21latency_hist00:05:020
1683241994813,8cyclictest0-21swapper/021:45:250
168324199481,23cyclictest1753-21gnome-shell19:15:550
1683241994812,12cyclictest1771135-21cat23:15:320
1683241994811,11cyclictest1781134-21strings23:45:230
1683241994811,11cyclictest1753541-21vmstat22:25:310
168325199478,15cyclictest1694967-21diskstats19:40:202
168324199477,11cyclictest1709435-21cat20:20:260
1683241994726,17cyclictest2197014-21PK-Backend20:30:220
1683241994716,7cyclictest1698907-21proc_pri19:50:290
1683241994716,10cyclictest1774983-21cron23:34:550
1683241994715,10cyclictest1778254-21seq23:37:340
1683241994712,11cyclictest1707821-21awk20:15:260
17473852460,7sleep32197014-21PK-Backend22:10:153
1683255994616,16cyclictest42-21ksoftirqd/319:40:003
168325599461,14cyclictest54-21kcompactd020:10:223
168324199468,18cyclictest1753-21gnome-shell19:24:360
168324199467,15cyclictest1753-21gnome-shell21:31:520
1683241994618,24cyclictest2197014-21PK-Backend20:47:010
1683241994617,9cyclictest1753-21gnome-shell21:50:290
1683241994616,10cyclictest1722012-21vmstat20:55:330
168324199461,25cyclictest1753-21gnome-shell23:05:420
1683241994610,16cyclictest1753-21gnome-shell22:46:430
162399460,32rtkit-daemon1753-21gnome-shell22:55:550
16996112450,6sleep126-21ksoftirqd/119:55:011
1683255994514,17cyclictest42-21ksoftirqd/319:55:473
1683255994512,17cyclictest42-21ksoftirqd/322:03:473
168325599451,15cyclictest1795851-21taskset00:27:193
168325599451,15cyclictest1795851-21taskset00:27:193
168324199459,16cyclictest1753-21gnome-shell21:39:100
168324199459,16cyclictest1753-21gnome-shell21:39:100
168324199459,16cyclictest1753-21gnome-shell21:15:290
168324199459,16cyclictest1753-21gnome-shell21:15:290
168324199459,16cyclictest1753-21gnome-shell20:25:090
1683241994512,8cyclictest0-21swapper/022:04:120
162399450,33rtkit-daemon0-21swapper/100:30:481
17197692440,5sleep00-21swapper/020:50:230
168325599447,19cyclictest42-21ksoftirqd/323:05:013
1683255994417,12cyclictest141rcu_preempt20:35:153
1683255994417,12cyclictest141rcu_preempt20:35:153
1683255994413,16cyclictest42-21ksoftirqd/321:00:473
1683255994412,6cyclictest141rcu_preempt19:30:143
1683255994410,7cyclictest42-21ksoftirqd/300:21:533
1683255994410,7cyclictest42-21ksoftirqd/300:21:533
168325199447,14cyclictest1744183-21ntpq22:00:282
1683246994418,22cyclictest1725024-21needreboot21:05:231
168324199449,19cyclictest1753-21gnome-shell19:25:060
168324199448,17cyclictest1753-21gnome-shell20:36:430
168324199448,17cyclictest1753-21gnome-shell20:36:420
168324199446,16cyclictest1753-21gnome-shell21:40:160
1683241994417,8cyclictest2197014-21PK-Backend23:28:290
1683241994415,8cyclictest1753-21gnome-shell22:22:150
1683241994415,8cyclictest1753-21gnome-shell22:22:140
1683241994414,7cyclictest1683532-21cpuspeed_turbos19:10:130
168324199441,38cyclictest2197014-21PK-Backend00:06:080
1683241994412,8cyclictest0-21swapper/021:00:180
168324199441,23cyclictest1753-21gnome-shell22:09:080
168324199441,23cyclictest1753-21gnome-shell19:39:330
168325599438,16cyclictest141rcu_preempt21:45:013
1683255994325,14cyclictest42-21ksoftirqd/323:13:223
1683255994318,5cyclictest16893762cat19:25:163
1683246994325,6cyclictest26-21ksoftirqd/119:40:171
1683246994322,13cyclictest26-21ksoftirqd/123:00:431
168324199435,18cyclictest1753-21gnome-shell19:55:160
168325599426,20cyclictest42-21ksoftirqd/323:55:003
1683255994231,8cyclictest42-21ksoftirqd/323:00:013
1683255994222,15cyclictest1701924-21df20:00:163
1683246994221,11cyclictest26-21ksoftirqd/119:19:521
1683246994217,20cyclictest1767755-40lsb_release23:06:251
1683241994215,7cyclictest1753-21gnome-shell00:23:400
1683241994215,7cyclictest1753-21gnome-shell00:23:390
162399420,36rtkit-daemon921-21thermald20:46:351
168325599419,9cyclictest42-21ksoftirqd/319:15:253
1683255994124,12cyclictest1699546-21cron19:55:013
1683255994122,15cyclictest1-21systemd00:00:003
1683255994120,8cyclictest42-21ksoftirqd/300:15:023
1683255994114,12cyclictest141rcu_preempt23:20:273
168325599411,17cyclictest42-21ksoftirqd/300:33:023
168325599411,14cyclictest2197014-21PK-Backend00:16:563
168324699419,28cyclictest1739688-21cron21:50:001
1683246994124,6cyclictest26-21ksoftirqd/119:50:001
1683246994122,7cyclictest26-21ksoftirqd/120:35:211
1683246994122,7cyclictest26-21ksoftirqd/120:35:201
1683246994121,5cyclictest26-21ksoftirqd/123:15:151
168324199415,12cyclictest2179-21gvfsd-metadata00:15:010
1683241994115,7cyclictest1753-21gnome-shell22:30:030
1683241994113,9cyclictest1753-21gnome-shell22:52:290
162399410,33rtkit-daemon869-21irqbalance20:18:351
162399410,33rtkit-daemon869-21irqbalance19:58:351
162399410,28rtkit-daemon2197014-21PK-Backend23:33:091
17618572400,7chrt1753-21gnome-shell22:50:283
1683255994018,8cyclictest42-21ksoftirqd/321:58:023
168325599401,15cyclictest2197014-21PK-Backend23:15:323
168325599401,14cyclictest2197014-21PK-Backend00:05:153
168325199407,11cyclictest1753-21gnome-shell22:24:052
168325199407,11cyclictest1753-21gnome-shell22:24:042
1683251994012,20cyclictest1753-21gnome-shell20:48:152
1683246994028,8cyclictest26-21ksoftirqd/100:40:011
1683246994024,6cyclictest26-21ksoftirqd/123:50:261
1683246994022,6cyclictest26-21ksoftirqd/100:25:321
1683246994022,6cyclictest26-21ksoftirqd/100:25:311
1683246994018,14cyclictest26-21ksoftirqd/122:52:081
1683241994014,8cyclictest1069-21runrttasks00:17:520
1683241994013,7cyclictest2197014-21PK-Backend23:22:050
1683241994010,8cyclictest1632-21dbus-daemon00:25:060
1683241994010,8cyclictest1632-21dbus-daemon00:25:060
17158172390,7chrt0-21swapper/320:40:153
168325599396,13cyclictest17531gnome-shell21:18:133
168325599396,13cyclictest17531gnome-shell21:18:133
168325599392,33cyclictest1753-21gnome-shell23:27:473
1683255993923,12cyclictest855-21dbus-daemon22:35:013
1683255993923,12cyclictest1697605-21sshd19:47:233
168325599391,5cyclictest141rcu_preempt20:22:013
168325599391,34cyclictest1753-21gnome-shell20:50:223
168325599391,21cyclictest1776137-21/usr/sbin/munin23:30:283
1683255993912,15cyclictest42-21ksoftirqd/321:45:213
168325599391,14cyclictest2197014-21PK-Backend22:43:593
168325599391,14cyclictest2197014-21PK-Backend20:05:193
168325599391,14cyclictest2197014-21PK-Backend20:05:193
1683246993927,8cyclictest1753-21gnome-shell22:40:281
168324699392,33cyclictest1753-21gnome-shell23:13:461
1683246993921,13cyclictest26-21ksoftirqd/123:45:281
1683246993920,6cyclictest26-21ksoftirqd/121:00:101
1683246993919,16cyclictest26-21ksoftirqd/120:20:311
168325599384,5cyclictest141rcu_preempt23:07:553
168325599382,6cyclictest141rcu_preempt00:35:173
1683255993826,8cyclictest42-21ksoftirqd/321:10:003
1683255993822,12cyclictest1732175-21sensors_temp21:25:293
1683255993816,5cyclictest141rcu_preempt22:35:323
1683255993815,8cyclictest42-21ksoftirqd/322:05:213
1683255993813,21cyclictest1722365-21cron21:00:013
168325599381,21cyclictest1760565-21chrt22:46:423
168325599381,21cyclictest1753549-21chrt22:25:333
168325599381,14cyclictest2197014-21PK-Backend21:14:123
168325599381,12cyclictest2197014-21PK-Backend21:40:013
168325599381,12cyclictest2197014-21PK-Backend21:40:013
1683246993828,7cyclictest1753-21gnome-shell23:23:441
1683246993827,7cyclictest1753-21gnome-shell22:06:381
1683246993826,7cyclictest26-21ksoftirqd/122:30:151
1683246993826,7cyclictest26-21ksoftirqd/121:35:201
1683246993826,7cyclictest26-21ksoftirqd/121:35:191
1683246993821,5cyclictest26-21ksoftirqd/122:29:301
1683246993821,13cyclictest1731894-21memory21:25:241
1683246993820,6cyclictest26-21ksoftirqd/122:10:141
1683246993815,19cyclictest26-21ksoftirqd/119:10:131
162399380,30rtkit-daemon26-21ksoftirqd/121:31:321
168325599374,10cyclictest42-21ksoftirqd/323:35:313
1683255993724,9cyclictest1687577-21tr19:20:123
1683255993723,10cyclictest1708223-21sshd20:16:223
1683255993721,11cyclictest438-20systemd-journal23:46:243
1683246993726,7cyclictest1753-21gnome-shell00:01:131
1683246993725,7cyclictest26-21ksoftirqd/121:40:151
1683246993720,12cyclictest26-21ksoftirqd/122:45:141
1683246993719,14cyclictest26-21ksoftirqd/119:39:271
1683246993717,16cyclictest1719968-21/usr/sbin/munin20:50:281
168325599363,20cyclictest141rcu_preempt00:01:253
1683255993623,9cyclictest1753-21gnome-shell20:47:583
1683255993620,12cyclictest1729728-21cpuspeed_turbos21:20:193
1683255993618,13cyclictest855-21dbus-daemon20:31:063
1683255993615,6cyclictest141rcu_preempt21:51:593
168325599361,15cyclictest2197014-21PK-Backend19:10:213
168325599361,14cyclictest2197014-21PK-Backend23:40:183
168325599361,14cyclictest2197014-21PK-Backend20:25:143
1683246993625,6cyclictest26-21ksoftirqd/123:55:331
1683246993624,7cyclictest26-21ksoftirqd/119:25:001
168325599351,15cyclictest2197014-21PK-Backend21:32:013
1683246993524,8cyclictest1692324-21perf19:35:011
1683246993524,6cyclictest26-21ksoftirqd/122:00:281
1683246993524,6cyclictest26-21ksoftirqd/119:25:211
1683246993517,14cyclictest1742073-21df_inode21:55:201
1683246993418,12cyclictest1762900-21cat22:55:131
1683246993324,5cyclictest26-21ksoftirqd/123:25:341
168324199331,13cyclictest2197014-21PK-Backend22:36:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional