You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-19 - 03:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Sep 19, 2025 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991000,24rtkit-daemon1785-21gmain22:50:521
162399990,31rtkit-daemon2197014-21PK-Backend19:19:213
331817129878,13sleep00-21swapper/019:06:350
331817129878,13sleep00-21swapper/019:06:350
162399980,76rtkit-daemon0-21swapper/219:06:402
162399980,76rtkit-daemon0-21swapper/219:06:402
34163732970,4chrt3416255-21/usr/sbin/munin23:50:131
162399960,25rtkit-daemon2197014-21PK-Backend19:34:253
33541392950,6sleep00-21swapper/020:50:020
162399940,30rtkit-daemon854-21avahi-daemon20:56:453
162399940,30rtkit-daemon2197014-21PK-Backend21:28:513
162399930,28rtkit-daemon2197014-21PK-Backend21:55:473
162399930,28rtkit-daemon2197014-21PK-Backend21:55:463
162399930,28rtkit-daemon0-21swapper/320:00:133
162399920,7rtkit-daemon2197014-21PK-Backend20:28:223
162399920,26rtkit-daemon2197014-21PK-Backend19:21:473
162399920,13rtkit-daemon42-21ksoftirqd/320:51:523
162399920,13rtkit-daemon42-21ksoftirqd/320:51:513
34046882910,5sleep30-21swapper/323:15:233
162399910,27rtkit-daemon2197014-21PK-Backend22:23:543
162399910,26rtkit-daemon0-21swapper/323:26:503
162399910,26rtkit-daemon0-21swapper/323:26:503
162399910,26rtkit-daemon0-21swapper/320:06:403
162399900,27rtkit-daemon2197014-21PK-Backend19:47:183
162399900,26rtkit-daemon2197014-21PK-Backend21:31:313
162399900,23rtkit-daemon0-21swapper/320:14:143
331836728869,13sleep30-21swapper/319:09:033
331836728869,13sleep30-21swapper/319:09:033
162399870,6rtkit-daemon2197014-21PK-Backend20:32:143
162399870,6rtkit-daemon2197014-21PK-Backend20:32:133
162399870,23rtkit-daemon0-21swapper/322:11:263
162399850,27rtkit-daemon2197014-21PK-Backend23:31:413
162399850,26rtkit-daemon2197014-21PK-Backend19:51:453
162399840,28rtkit-daemon2197014-21PK-Backend22:35:323
162399840,28rtkit-daemon2197014-21PK-Backend19:12:363
162399840,27rtkit-daemon2197014-21PK-Backend20:15:073
162399840,26rtkit-daemon2197014-21PK-Backend21:51:313
162399840,26rtkit-daemon2197014-21PK-Backend21:51:313
162399830,5rtkit-daemon0-21swapper/322:17:453
162399820,8rtkit-daemon1622-21rtkit-daemon22:46:440
33709092810,6sleep01753-21gnome-shell21:40:170
33633622770,14sleep1261ksoftirqd/121:20:011
33633622770,14sleep1261ksoftirqd/121:20:011
33613192760,4chrt3361318-21sshd21:11:452
162399750,8rtkit-daemon1753-21gnome-shell23:44:101
162399740,8rtkit-daemon1506-21systemd23:49:351
162399740,5rtkit-daemon0-21swapper/320:37:273
162399730,9rtkit-daemon1506-21systemd23:12:051
162399730,9rtkit-daemon1506-21systemd21:42:571
162399730,8rtkit-daemon1506-21systemd21:48:031
162399730,8rtkit-daemon1506-21systemd21:11:081
162399710,9rtkit-daemon1506-21systemd23:09:591
162399700,9rtkit-daemon1622-21rtkit-daemon23:09:460
162399700,9rtkit-daemon1622-21rtkit-daemon19:36:360
331826426844,15sleep10-21swapper/119:07:451
331826426844,15sleep10-21swapper/119:07:451
3318593996610,17cyclictest1753-21gnome-shell23:20:002
3318593996210,31cyclictest1753-21gnome-shell21:56:552
3318593996210,31cyclictest1753-21gnome-shell21:56:552
331859399619,31cyclictest1753-21gnome-shell19:17:242
3318593996111,28cyclictest1753-21gnome-shell22:25:202
331859399605,32cyclictest1753-21gnome-shell23:25:112
331859399605,32cyclictest1753-21gnome-shell23:25:112
3318593996011,27cyclictest1753-21gnome-shell21:15:212
3318593996011,27cyclictest1753-21gnome-shell21:15:212
3318593996010,28cyclictest1753-21gnome-shell19:45:472
34167602590,6sleep01753-21gnome-shell23:50:220
331859399599,28cyclictest1753-21gnome-shell21:30:382
331859399597,31cyclictest1753-21gnome-shell22:16:302
331859399597,31cyclictest1753-21gnome-shell22:14:052
3318593995911,26cyclictest1753-21gnome-shell23:45:242
331859399586,32cyclictest1753-21gnome-shell20:56:032
3318593995811,26cyclictest1753-21gnome-shell22:35:512
3318593995811,25cyclictest1753-21gnome-shell21:40:302
3318593995811,25cyclictest1753-21gnome-shell20:25:332
3318593995811,25cyclictest1753-21gnome-shell20:15:302
3318593995810,26cyclictest1753-21gnome-shell22:55:162
3318593995810,24cyclictest1753-21gnome-shell00:10:112
331858299586,22cyclictest3341582-21latency_hist20:15:020
331859399579,27cyclictest1753-21gnome-shell19:23:452
331859399579,26cyclictest1753-21gnome-shell19:10:042
331859399576,28cyclictest1753-21gnome-shell23:30:382
3318593995712,24cyclictest1753-21gnome-shell21:20:052
3318593995711,24cyclictest1753-21gnome-shell21:48:382
3318593995710,26cyclictest1753-21gnome-shell23:35:292
3318593995710,25cyclictest1753-21gnome-shell21:25:222
3318593995710,25cyclictest1753-21gnome-shell20:20:222
3318593995710,25cyclictest1753-21gnome-shell20:10:082
3318593995710,25cyclictest1753-21gnome-shell00:28:512
3318593995710,25cyclictest1753-21gnome-shell00:00:222
3318593995710,24cyclictest1753-21gnome-shell20:33:112
3318593995710,24cyclictest1753-21gnome-shell20:33:112
3318593995611,26cyclictest1753-21gnome-shell19:36:352
3318593995611,24cyclictest1753-21gnome-shell00:31:162
3318593995611,23cyclictest1753-21gnome-shell22:52:022
3318593995611,23cyclictest1753-21gnome-shell22:07:242
3318593995610,25cyclictest1753-21gnome-shell22:22:312
3318593995610,25cyclictest1753-21gnome-shell20:05:232
3318593995610,24cyclictest1753-21gnome-shell22:42:462
3318593995610,24cyclictest1753-21gnome-shell22:30:402
3318593995610,24cyclictest1753-21gnome-shell19:44:022
331858299566,27cyclictest3427183-21cut00:20:170
331859399554,31cyclictest3352562-21apt20:45:122
3318593995513,17cyclictest3427031-21/usr/sbin/munin00:20:152
3318593995512,18cyclictest1753-21gnome-shell23:55:052
3318593995511,24cyclictest1753-21gnome-shell22:48:152
331859399548,24cyclictest1753-21gnome-shell19:33:312
3318593995411,22cyclictest3336270-21taskset19:58:452
3318593995410,28cyclictest1753-21gnome-shell23:24:542
3318593995410,25cyclictest1753-21gnome-shell21:01:232
331858299541,11cyclictest869-21irqbalance23:04:350
162399540,41rtkit-daemon2197014-21PK-Backend22:07:283
33977332530,7chrt2197014-21PK-Backend22:55:233
331859399536,26cyclictest1753-21gnome-shell21:51:252
331859399536,26cyclictest1753-21gnome-shell21:51:252
331859399534,31cyclictest3416643-21df23:50:162
331859399534,25cyclictest1753-21gnome-shell22:01:402
331859399533,29cyclictest3369550-21ntp_states21:35:282
331859399533,29cyclictest3369550-21ntp_states21:35:282
331859399532,28cyclictest854-21avahi-daemon23:02:062
3318593995311,16cyclictest1753-21gnome-shell00:16:302
3318593995310,24cyclictest3337030-21diskstats20:00:192
3318593995310,16cyclictest1753-21gnome-shell20:40:342
3318585995335,13cyclictest26-21ksoftirqd/122:30:001
331858299531,12cyclictest1753-21gnome-shell22:35:020
331859399529,18cyclictest1753-21gnome-shell23:10:192
3318585995223,6cyclictest26-21ksoftirqd/122:55:121
331858299521,11cyclictest3343956-21df20:20:190
331859399516,20cyclictest1753-21gnome-shell00:05:172
3318593995111,21cyclictest1753-21gnome-shell20:35:402
3318593995110,23cyclictest3325226-21ntp_states19:25:272
3318585995124,23cyclictest26-21ksoftirqd/120:25:491
3318585995123,5cyclictest26-21ksoftirqd/119:35:151
3318585995110,17cyclictest141rcu_preempt19:15:011
331858299512,46cyclictest0-21swapper/021:21:200
331859399506,20cyclictest3334057-21python319:50:292
3318593995011,16cyclictest1753-21gnome-shell23:40:292
3318593995010,17cyclictest1753-21gnome-shell20:54:402
3318593995010,17cyclictest1753-21gnome-shell20:54:402
3318585995023,6cyclictest26-21ksoftirqd/122:37:501
3318585995022,24cyclictest26-21ksoftirqd/119:21:421
162399500,36rtkit-daemon2197014-21PK-Backend22:49:403
3318593994910,15cyclictest3358510-21ls21:05:152
3318585994912,17cyclictest141rcu_preempt19:45:001
331858299491,29cyclictest1753-21gnome-shell20:01:410
331858299491,18cyclictest3318756-21/usr/sbin/munin19:10:150
162399490,41rtkit-daemon438-20systemd-journal21:47:500
3318585994821,23cyclictest26-21ksoftirqd/123:35:191
3318585994821,14cyclictest26-21ksoftirqd/100:05:301
3318585994820,17cyclictest26-21ksoftirqd/120:35:301
331858299486,21cyclictest13-21ksoftirqd/019:52:410
331858299486,21cyclictest13-21ksoftirqd/000:08:410
331858299480,16cyclictest1753-21gnome-shell23:20:260
331859399475,26cyclictest3432970-21ntp_states00:35:242
331859399473,16cyclictest0-21swapper/223:05:052
331858599477,6cyclictest141rcu_preempt20:25:011
3318585994735,8cyclictest1753-21gnome-shell21:05:301
3318585994721,14cyclictest26-21ksoftirqd/122:40:151
3318585994718,24cyclictest26-21ksoftirqd/120:10:161
331858299477,20cyclictest13-21ksoftirqd/022:00:090
331858299476,22cyclictest1753-21gnome-shell20:30:560
331858299476,22cyclictest1753-21gnome-shell20:30:550
331858299474,18cyclictest13-21ksoftirqd/023:39:270
331858299471,8cyclictest1753-21gnome-shell21:55:320
331858299471,8cyclictest1753-21gnome-shell21:55:320
3318582994713,13cyclictest13-21ksoftirqd/000:00:040
331858299471,27cyclictest1753-21gnome-shell20:41:390
331858299471,25cyclictest3330465-21ntp_states19:40:280
3318582994711,15cyclictest13-21ksoftirqd/000:25:210
331858299470,28cyclictest1753-21gnome-shell23:46:560
331858299470,28cyclictest1753-21gnome-shell20:53:500
331858299470,28cyclictest1753-21gnome-shell20:53:490
162399470,33rtkit-daemon1753-21gnome-shell19:38:093
3318585994621,6cyclictest26-21ksoftirqd/119:45:181
331858299466,14cyclictest13-21ksoftirqd/021:00:570
331858299466,13cyclictest3374272-21dpkg21:50:130
331858299466,13cyclictest3374272-21dpkg21:50:120
331858299466,12cyclictest1753-21gnome-shell23:42:450
331858299465,28cyclictest13-21ksoftirqd/021:28:390
331858299462,26cyclictest1753-21gnome-shell20:36:050
3318582994620,10cyclictest3326575-21df19:30:170
3318582994614,13cyclictest13-21ksoftirqd/019:55:180
331858299460,28cyclictest1753-21gnome-shell21:33:240
331858299460,27cyclictest1753-21gnome-shell23:14:210
331858299460,27cyclictest1753-21gnome-shell19:28:380
331858299460,15cyclictest3345907-21memory20:25:240
162399460,32rtkit-daemon2197014-21PK-Backend23:21:283
3318585994526,6cyclictest26-21ksoftirqd/121:57:241
3318585994526,6cyclictest26-21ksoftirqd/121:57:241
3318585994525,7cyclictest26-21ksoftirqd/123:55:041
3318585994523,7cyclictest26-21ksoftirqd/100:25:331
331858299457,19cyclictest1753-21gnome-shell23:30:270
331858299457,19cyclictest1753-21gnome-shell22:16:000
331858299457,11cyclictest1753-21gnome-shell23:55:140
331858299455,23cyclictest1753-21gnome-shell22:53:050
331858299451,26cyclictest1753-21gnome-shell20:08:490
331858299451,25cyclictest1753-21gnome-shell20:55:120
3318582994511,18cyclictest13-21ksoftirqd/019:20:200
331858599443,14cyclictest26-21ksoftirqd/121:34:311
3318585994426,7cyclictest1753-21gnome-shell23:23:191
3318585994426,6cyclictest26-21ksoftirqd/122:15:231
3318585994424,7cyclictest26-21ksoftirqd/100:34:481
3318585994419,13cyclictest26-21ksoftirqd/121:50:311
3318585994419,13cyclictest26-21ksoftirqd/121:50:301
3318585994415,5cyclictest26-21ksoftirqd/120:50:391
3318585994415,5cyclictest26-21ksoftirqd/120:50:391
3318585994412,29cyclictest251rcuc/121:27:441
331858599441,14cyclictest34250212idleruntime-cro00:15:011
331858299447,16cyclictest13-21ksoftirqd/022:40:120
331858299447,11cyclictest3391752-21latency_hist22:40:020
3318582994436,4cyclictest161rcuc/021:07:540
3318582994433,8cyclictest161rcuc/022:08:090
331858299443,24cyclictest1753-21gnome-shell00:36:400
331858299443,21cyclictest13-21ksoftirqd/019:15:130
3318582994429,11cyclictest3426591-21sshd00:18:140
162399440,32rtkit-daemon2197014-21PK-Backend19:44:303
3318601994317,16cyclictest42-21ksoftirqd/323:45:123
3318585994325,6cyclictest26-21ksoftirqd/121:35:151
3318585994325,6cyclictest26-21ksoftirqd/121:35:151
3318585994325,6cyclictest26-21ksoftirqd/121:23:491
3318585994325,6cyclictest26-21ksoftirqd/120:18:251
3318585994325,6cyclictest26-21ksoftirqd/119:33:561
3318585994325,6cyclictest26-21ksoftirqd/100:39:281
3318585994325,6cyclictest26-21ksoftirqd/100:21:091
3318585994324,7cyclictest26-21ksoftirqd/100:00:301
3318585994324,6cyclictest26-21ksoftirqd/120:55:341
3318585994324,6cyclictest26-21ksoftirqd/120:30:041
3318585994324,6cyclictest26-21ksoftirqd/120:30:041
3318585994324,6cyclictest26-21ksoftirqd/120:01:281
3318585994321,12cyclictest26-21ksoftirqd/120:44:251
3318585994319,6cyclictest26-21ksoftirqd/121:00:181
3318585994311,28cyclictest251rcuc/123:20:021
331858299439,18cyclictest13-21ksoftirqd/019:47:230
331858299439,11cyclictest13-21ksoftirqd/021:39:310
331858299439,11cyclictest13-21ksoftirqd/021:39:300
331858299437,17cyclictest1753-21gnome-shell22:10:560
331858299436,11cyclictest1753-21gnome-shell23:15:490
331858299432,16cyclictest1753-21gnome-shell22:58:070
331858299431,16cyclictest1753-21gnome-shell22:24:550
331858299430,16cyclictest1753-21gnome-shell20:18:390
3318601994224,6cyclictest42-21ksoftirqd/319:58:443
3318601994218,19cyclictest42-21ksoftirqd/323:10:203
3318585994225,6cyclictest26-21ksoftirqd/122:08:561
3318585994225,6cyclictest26-21ksoftirqd/122:00:131
3318585994224,6cyclictest26-21ksoftirqd/123:00:181
3318585994224,6cyclictest26-21ksoftirqd/122:45:331
3318585994224,6cyclictest26-21ksoftirqd/122:30:061
3318585994224,6cyclictest26-21ksoftirqd/119:56:391
3318585994224,6cyclictest26-21ksoftirqd/119:50:581
3318585994223,7cyclictest26-21ksoftirqd/120:05:181
3318585994223,6cyclictest26-21ksoftirqd/120:45:251
3318585994223,6cyclictest26-21ksoftirqd/119:25:141
3318585994223,6cyclictest26-21ksoftirqd/119:15:021
3318585994223,6cyclictest26-21ksoftirqd/100:15:201
3318582994217,11cyclictest1753-21gnome-shell21:13:410
3318582994212,16cyclictest13-21ksoftirqd/021:20:020
3318582994212,16cyclictest13-21ksoftirqd/021:20:010
3318601994122,14cyclictest42-21ksoftirqd/300:00:033
3318601994112,12cyclictest141rcu_preempt00:17:313
3318601994111,12cyclictest141rcu_preempt00:27:053
3318585994123,6cyclictest26-21ksoftirqd/122:20:121
3318585994123,6cyclictest26-21ksoftirqd/122:10:221
162399410,28rtkit-daemon2197014-21PK-Backend20:20:213
331860199405,18cyclictest2197014-21PK-Backend00:05:193
3318601994026,11cyclictest42-21ksoftirqd/300:30:253
3318601994025,11cyclictest42-21ksoftirqd/322:50:243
3318601994025,11cyclictest3357777-21sshd21:00:453
3318601994023,6cyclictest42-21ksoftirqd/320:45:013
3318601994023,13cyclictest42-21ksoftirqd/321:50:013
3318601994022,13cyclictest42-21ksoftirqd/323:00:313
331860199401,35cyclictest3379497-21seq22:04:363
331860199401,11cyclictest3362708-21processes21:15:303
331860199401,11cyclictest3362708-21processes21:15:303
3318585994023,5cyclictest26-21ksoftirqd/123:30:331
331858299401,15cyclictest1753-21gnome-shell23:25:200
331858299401,15cyclictest1753-21gnome-shell23:25:190
3318601993922,13cyclictest42-21ksoftirqd/323:57:283
3318601993922,13cyclictest3363970-21df_inode21:20:173
331858299398,16cyclictest13-21ksoftirqd/000:32:350
331858299391,13cyclictest13-21ksoftirqd/000:10:130
331858299391,11cyclictest3387497-21ntp_states22:25:260
162399390,28rtkit-daemon2197014-21PK-Backend22:25:583
3318601993824,10cyclictest3401894-21sshd23:07:453
3318601993822,6cyclictest42-21ksoftirqd/321:05:183
3318601993814,13cyclictest855-21dbus-daemon20:50:013
3318601993813,16cyclictest42-21ksoftirqd/321:14:553
3318585993821,6cyclictest26-21ksoftirqd/123:25:221
3318585993821,6cyclictest26-21ksoftirqd/123:25:211
162399380,27rtkit-daemon2197014-21PK-Backend19:25:463
3318601993726,6cyclictest42-21ksoftirqd/323:50:313
3318601993726,6cyclictest42-21ksoftirqd/322:30:063
162399370,5rtkit-daemon0-21swapper/322:40:123
3318601993619,13cyclictest42-21ksoftirqd/300:10:323
162399360,24rtkit-daemon2197014-21PK-Backend21:37:313
162399360,24rtkit-daemon2197014-21PK-Backend21:37:303
331860199352,11cyclictest141rcu_preempt00:36:173
331860199351,4cyclictest141rcu_preempt23:40:423
3318601993424,6cyclictest42-21ksoftirqd/321:45:003
3318601993314,14cyclictest42-21ksoftirqd/300:20:393
331860199322,4cyclictest141rcu_preempt23:35:353
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional