You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 09:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Nov 25, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
379170021020,19sleep20-21swapper/219:05:152
162399990,5rtkit-daemon0-21swapper/319:30:183
38347512960,6sleep32197014-21PK-Backend21:05:143
38528972910,6sleep00-21swapper/021:55:260
379269628569,9sleep00-21swapper/019:07:510
162399800,6rtkit-daemon2197014-21PK-Backend22:10:562
38896952760,5sleep00-21swapper/023:40:270
379252227641,9sleep10-21swapper/119:05:401
39072672710,7chrt2197014-21PK-Backend00:30:253
379254427147,16sleep30-21swapper/319:05:573
379302299637,35cyclictest3891807-21chrt23:46:212
3793015996339,11cyclictest13-21ksoftirqd/023:20:270
38875642620,5sleep02197014-21PK-Backend23:35:200
38875642620,5sleep02197014-21PK-Backend23:35:200
379302299621,12cyclictest1753-21gnome-shell21:05:222
379301599626,10cyclictest3895620-40mandb00:00:030
379302299610,16cyclictest3877884-21chrt23:05:392
379302299607,29cyclictest1753-21gnome-shell20:15:152
3793020996040,7cyclictest1753-21gnome-shell00:32:161
3793020996030,17cyclictest26-21ksoftirqd/123:45:101
3793020996012,30cyclictest26-21ksoftirqd/122:45:161
3793025995926,9cyclictest3833929-21munin-plugin-st21:05:003
379301599599,31cyclictest13-21ksoftirqd/019:15:230
379301599598,6cyclictest141rcu_preempt23:32:060
379301599581,14cyclictest3846410-21ls21:40:020
3793015995810,20cyclictest13-21ksoftirqd/022:15:000
3793015995714,6cyclictest141rcu_preempt00:17:460
379302299568,21cyclictest1753-21gnome-shell23:53:142
379302299567,24cyclictest3805903-21munin-run19:45:012
379302299559,16cyclictest3887292-21cat23:35:152
379302299559,16cyclictest3887292-21cat23:35:152
379302299550,12cyclictest3824706-21chrt20:37:082
379301599559,5cyclictest141rcu_preempt20:39:470
3793015995520,15cyclictest13-21ksoftirqd/019:44:300
3793015995513,5cyclictest141rcu_preempt19:26:300
38793052540,5sleep2331rcuc/223:10:262
38364992540,5sleep20-21swapper/221:10:162
379302299548,15cyclictest3821481-21latency_hist20:30:022
379302299541,23cyclictest0-21swapper/219:50:022
3793022995410,22cyclictest1753-21gnome-shell19:14:162
3793015995431,8cyclictest13-21ksoftirqd/023:25:210
3793015995413,19cyclictest13-21ksoftirqd/021:50:010
3793015995413,19cyclictest13-21ksoftirqd/021:50:010
379302299538,23cyclictest1753-21gnome-shell00:17:062
379302299538,11cyclictest3802435-21systemd-run19:35:002
379302299536,23cyclictest1753-21gnome-shell19:29:172
379302299536,23cyclictest1753-21gnome-shell00:25:352
379302299531,11cyclictest3870379-21chrt22:45:232
3793020995345,6cyclictest251rcuc/120:13:511
3793020995314,22cyclictest141rcu_preempt21:10:161
379301599538,19cyclictest13-21ksoftirqd/022:20:200
379302299528,22cyclictest1753-21gnome-shell22:33:552
379302299528,20cyclictest1753-21gnome-shell23:44:282
379302299527,23cyclictest1753-21gnome-shell20:55:382
379302299526,18cyclictest3847768-21taskset21:40:592
3793022995217,17cyclictest1069-21runrttasks20:34:322
379302299520,30cyclictest1753-21gnome-shell21:57:502
379302299518,21cyclictest1753-21gnome-shell21:34:392
379302299518,14cyclictest3906781-21taskset00:30:172
379302299517,22cyclictest1753-21gnome-shell22:23:092
379302299517,22cyclictest1753-21gnome-shell22:00:042
379302299517,22cyclictest1753-21gnome-shell20:05:202
379302299517,22cyclictest1753-21gnome-shell00:14:182
379302299516,22cyclictest1753-21gnome-shell22:28:042
379302299515,23cyclictest1753-21gnome-shell20:05:012
379302299515,23cyclictest1753-21gnome-shell20:05:012
3793022995114,31cyclictest3859850-40lsb_release22:15:232
379302299511,28cyclictest1753-21gnome-shell21:27:232
379302299511,28cyclictest1753-21gnome-shell21:27:232
3793022995110,19cyclictest1753-21gnome-shell21:47:542
3793022995110,19cyclictest1753-21gnome-shell21:47:532
379302299510,30cyclictest3830566-21latency_hist20:55:012
3793020995125,8cyclictest26-21ksoftirqd/100:30:011
379302099512,18cyclictest26-21ksoftirqd/123:14:181
3793020995120,9cyclictest26-21ksoftirqd/119:15:251
3793020995115,19cyclictest26-21ksoftirqd/121:05:001
379301599519,18cyclictest13-21ksoftirqd/022:50:270
162399510,24rtkit-daemon1753-21gnome-shell21:26:383
162399510,24rtkit-daemon1753-21gnome-shell21:26:383
379302299509,21cyclictest1753-21gnome-shell23:31:222
379302299508,21cyclictest3840096-21hddtemp_smartct21:20:192
379302299507,22cyclictest1753-21gnome-shell19:53:202
379302299506,22cyclictest1753-21gnome-shell21:35:032
379302299506,22cyclictest1753-21gnome-shell00:23:452
379302299506,22cyclictest1753-21gnome-shell00:23:452
3793022995010,19cyclictest1753-21gnome-shell00:02:522
3793020995025,10cyclictest26-21ksoftirqd/121:27:181
3793020995025,10cyclictest26-21ksoftirqd/121:27:171
3793020995013,16cyclictest26-21ksoftirqd/119:20:091
379302099501,27cyclictest438-20systemd-journal23:06:211
3793020995011,19cyclictest26-21ksoftirqd/122:04:311
379302099500,47cyclictest1622-21rtkit-daemon21:08:421
379301599508,19cyclictest13-21ksoftirqd/020:15:330
3793015995021,11cyclictest13-21ksoftirqd/023:54:420
3793015995020,9cyclictest13-21ksoftirqd/021:06:520
379302299498,20cyclictest1753-21gnome-shell23:27:182
379302299497,21cyclictest1753-21gnome-shell23:03:482
379302299497,14cyclictest3832721-21expr21:00:142
379302299496,21cyclictest1753-21gnome-shell23:23:282
379302299496,21cyclictest1753-21gnome-shell20:10:102
379302299491,24cyclictest3828212-21threads20:45:332
3793022994911,20cyclictest34-21ksoftirqd/220:40:182
3793020994922,10cyclictest26-21ksoftirqd/120:50:221
3793020994919,10cyclictest26-21ksoftirqd/119:35:021
3793020994911,21cyclictest26-21ksoftirqd/120:20:321
3793020994911,21cyclictest26-21ksoftirqd/120:20:321
379301599497,17cyclictest3870154-21df22:45:170
379301599495,8cyclictest13-21ksoftirqd/020:28:110
3793015994918,8cyclictest13-21ksoftirqd/021:14:260
379301599491,10cyclictest2197014-21PK-Backend19:50:010
162399490,5rtkit-daemon0-21swapper/321:34:343
38256522480,15chrt26-21ksoftirqd/120:40:211
379302299482,26cyclictest1753-21gnome-shell19:36:112
379302299481,17cyclictest3796359-40lsb_release19:15:232
379302299481,10cyclictest3809604-21apt19:55:112
3793020994819,6cyclictest3866008-21taskset22:34:351
3793020994817,8cyclictest26-21ksoftirqd/121:43:151
379301599489,17cyclictest13-21ksoftirqd/020:30:310
379301599487,17cyclictest13-21ksoftirqd/000:00:160
379301599485,21cyclictest13-21ksoftirqd/023:16:490
379301599485,19cyclictest13-21ksoftirqd/022:35:020
3793015994826,7cyclictest13-21ksoftirqd/020:40:200
3793015994825,8cyclictest13-21ksoftirqd/021:18:180
3793015994819,10cyclictest13-21ksoftirqd/022:40:270
3793015994818,11cyclictest141rcu_preempt20:53:100
3793015994817,12cyclictest141rcu_preempt00:29:180
379301599481,29cyclictest3851517-40apt-check21:51:160
3793015994812,8cyclictest13-21ksoftirqd/020:55:290
379301599480,44cyclictest1622-21rtkit-daemon00:34:120
3793025994726,17cyclictest3825653-21/usr/sbin/munin20:40:203
379302299479,17cyclictest1753-21gnome-shell20:21:582
379302299479,17cyclictest1753-21gnome-shell20:21:582
379302299477,18cyclictest1753-21gnome-shell21:50:162
379302299477,17cyclictest3869468-21taskset22:44:292
379302299475,20cyclictest1753-21gnome-shell22:35:312
379302299470,15cyclictest1753-21gnome-shell22:57:022
379302099479,21cyclictest26-21ksoftirqd/119:40:181
379302099478,27cyclictest26-21ksoftirqd/119:10:181
3793020994717,17cyclictest26-21ksoftirqd/119:39:251
379301599477,18cyclictest1753-21gnome-shell23:05:000
379301599477,17cyclictest13-21ksoftirqd/022:05:010
379301599476,20cyclictest13-21ksoftirqd/020:22:350
379301599476,20cyclictest13-21ksoftirqd/020:22:350
379301599475,23cyclictest13-21ksoftirqd/021:30:280
379301599472,20cyclictest1753-21gnome-shell19:55:300
3793015994716,10cyclictest13-21ksoftirqd/021:25:200
3793015994716,10cyclictest13-21ksoftirqd/021:25:200
3793015994712,10cyclictest13-21ksoftirqd/020:00:220
3793015994712,10cyclictest13-21ksoftirqd/020:00:210
162399470,33rtkit-daemon42-21ksoftirqd/319:23:363
37951252460,5sleep03795037-21hddtemp_smartct19:15:030
379302299468,14cyclictest3898078-21cat00:05:152
379302299468,14cyclictest1753-21gnome-shell23:15:392
3793020994616,17cyclictest26-21ksoftirqd/121:16:531
3793020994615,14cyclictest26-21ksoftirqd/100:20:311
3793020994615,14cyclictest26-21ksoftirqd/100:20:301
3793020994614,18cyclictest26-21ksoftirqd/121:49:511
3793020994614,18cyclictest26-21ksoftirqd/121:49:501
3793020994614,12cyclictest141rcu_preempt23:20:221
3793020994613,10cyclictest26-21ksoftirqd/122:20:001
3793020994611,17cyclictest26-21ksoftirqd/119:45:191
379301599468,18cyclictest13-21ksoftirqd/000:10:190
379301599467,5cyclictest141rcu_preempt23:06:050
379301599466,20cyclictest13-21ksoftirqd/020:07:100
379301599466,20cyclictest13-21ksoftirqd/000:38:190
379301599466,17cyclictest3815190-21diskstats20:10:170
3793015994622,8cyclictest13-21ksoftirqd/020:47:150
379301599461,10cyclictest2197014-21PK-Backend23:50:000
162399460,33rtkit-daemon42-21ksoftirqd/320:29:183
379302299452,23cyclictest1753-21gnome-shell21:19:372
379302099457,22cyclictest26-21ksoftirqd/122:35:241
379302099457,20cyclictest26-21ksoftirqd/120:30:241
379302099455,21cyclictest26-21ksoftirqd/123:35:151
379302099455,21cyclictest26-21ksoftirqd/123:35:151
379302099454,16cyclictest26-21ksoftirqd/123:35:031
3793020994520,11cyclictest26-21ksoftirqd/123:00:011
3793020994516,13cyclictest26-21ksoftirqd/121:53:411
379302099451,28cyclictest3799318-21cpuspeed_turbos19:25:141
379302099451,12cyclictest1753-21gnome-shell00:10:041
379302099451,10cyclictest3863942-21users22:25:321
379301599457,20cyclictest13-21ksoftirqd/023:10:290
379301599456,24cyclictest13-21ksoftirqd/022:18:550
379301599455,21cyclictest13-21ksoftirqd/019:30:470
379301599455,20cyclictest13-21ksoftirqd/022:29:190
379301599455,20cyclictest13-21ksoftirqd/022:06:200
379301599451,40cyclictest3867803-21turbostat22:40:000
379301599451,22cyclictest3798223-21chrt19:20:260
38434682440,6sleep12197014-21PK-Backend21:30:151
3793025994424,16cyclictest3895614-40ogrotate00:00:013
3793025994414,25cyclictest438-20systemd-journal00:20:003
379302299443,17cyclictest3242253-21nautilus00:00:032
379302299441,11cyclictest3908318-21apt00:35:122
379302099447,10cyclictest26-21ksoftirqd/120:15:261
379302099446,15cyclictest3789211-21kworker/1:120:55:161
3793020994423,10cyclictest26-21ksoftirqd/122:11:391
3793020994417,8cyclictest26-21ksoftirqd/120:47:261
3793020994416,12cyclictest141rcu_preempt23:01:111
3793020994413,15cyclictest26-21ksoftirqd/121:20:501
379301599449,16cyclictest1753-21gnome-shell21:23:300
379301599446,5cyclictest141rcu_preempt23:00:000
379301599446,23cyclictest141rcu_preempt21:01:120
3793015994410,11cyclictest13-21ksoftirqd/000:06:370
3793025994329,10cyclictest42-21ksoftirqd/320:49:563
3793025994323,16cyclictest1753-21gnome-shell22:09:453
379302299436,18cyclictest2197014-21PK-Backend22:50:072
379302099437,32cyclictest26-21ksoftirqd/120:37:471
379302099435,16cyclictest1753-21gnome-shell00:07:301
3793020994323,10cyclictest26-21ksoftirqd/121:39:591
3793020994317,10cyclictest26-21ksoftirqd/123:20:011
3793020994315,12cyclictest141rcu_preempt20:06:421
3793020994314,15cyclictest26-21ksoftirqd/120:26:301
3793020994314,11cyclictest141rcu_preempt22:40:411
379301599431,18cyclictest3903123-21apt00:20:150
379301599431,18cyclictest3903123-21apt00:20:150
3793025994223,14cyclictest3848733-21df21:45:163
3793025994223,14cyclictest3848733-21df21:45:163
379302599421,37cyclictest1753-21gnome-shell19:54:553
379302299427,13cyclictest3798937-21latency_hist19:25:012
379302099422,19cyclictest1753-21gnome-shell00:00:481
3793020994212,16cyclictest26-21ksoftirqd/119:56:451
3793020994212,15cyclictest26-21ksoftirqd/123:25:041
3793020994212,15cyclictest26-21ksoftirqd/122:05:501
379301599427,5cyclictest141rcu_preempt21:40:250
379301599426,16cyclictest3804143-21munin-run19:40:000
39001002410,16sleep342-21ksoftirqd/300:10:203
3793025994127,10cyclictest3862362-21sshd22:22:223
3793025994124,13cyclictest855-21dbus-daemon21:45:003
3793025994115,7cyclictest141rcu_preempt21:10:273
379302599411,36cyclictest3867023-21netstat22:35:243
3793025994112,6cyclictest141rcu_preempt20:34:103
379302299412,19cyclictest3857198-21sort22:10:032
379302099417,18cyclictest26-21ksoftirqd/100:15:121
379302099415,10cyclictest26-21ksoftirqd/121:55:141
379302099411,35cyclictest438-20systemd-journal19:50:211
3793020994113,15cyclictest26-21ksoftirqd/123:50:301
3793020994112,8cyclictest26-21ksoftirqd/123:55:151
379301599415,17cyclictest3808461-21memory19:50:240
162399410,29rtkit-daemon2197014-21PK-Backend00:25:363
379302599409,13cyclictest141rcu_preempt23:23:323
3793025994031,5cyclictest391irq_work/323:10:213
3793025994021,15cyclictest1753-21gnome-shell23:43:133
3793025994021,14cyclictest1-21systemd00:10:013
379302599401,4cyclictest141rcu_preempt19:11:113
3793025994014,6cyclictest141rcu_preempt20:55:353
3793025994013,7cyclictest141rcu_preempt22:35:003
379302599401,34cyclictest438-20systemd-journal23:37:083
379302599401,34cyclictest438-20systemd-journal23:37:083
3793025994010,12cyclictest0-21swapper/321:35:323
379302099405,18cyclictest26-21ksoftirqd/120:00:041
379302099405,18cyclictest26-21ksoftirqd/120:00:041
3793020994012,10cyclictest26-21ksoftirqd/123:40:541
3793020994011,15cyclictest26-21ksoftirqd/100:37:101
162399400,29rtkit-daemon2197014-21PK-Backend20:52:453
379302599395,30cyclictest3875016-21/usr/sbin/munin23:00:173
379302599392,33cyclictest1753-21gnome-shell19:41:143
3793025993921,14cyclictest3795696-40apt-check19:15:203
379302599391,23cyclictest42-21ksoftirqd/322:17:253
379302599391,17cyclictest1753-21gnome-shell19:35:253
379302599391,10cyclictest2197014-21PK-Backend23:54:153
3793020993923,5cyclictest26-21ksoftirqd/122:21:481
3793020993913,9cyclictest26-21ksoftirqd/122:52:301
162399390,27rtkit-daemon2197014-21PK-Backend23:16:133
379302599382,13cyclictest1753-21gnome-shell20:15:303
3793025993821,12cyclictest438-20systemd-journal20:06:283
3793025993815,19cyclictest1753-21gnome-shell22:40:203
162399380,26rtkit-daemon0-21swapper/321:58:233
3793025993728,6cyclictest1753-21gnome-shell20:02:563
3793025993728,6cyclictest1753-21gnome-shell20:02:553
3793025993724,9cyclictest1753-21gnome-shell20:35:133
3793025993723,10cyclictest3819440-21sshd20:21:203
3793025993723,10cyclictest3819440-21sshd20:21:203
3793025993721,12cyclictest3891040-21forks23:45:183
3793025993713,20cyclictest42-21ksoftirqd/321:50:423
3793025993712,22cyclictest0-21swapper/300:23:063
3793025993712,22cyclictest0-21swapper/300:23:063
3793025993710,7cyclictest141rcu_preempt19:25:153
162399370,26rtkit-daemon2197014-21PK-Backend00:01:483
379302599369,6cyclictest141rcu_preempt22:15:013
3793025993625,8cyclictest1753-21gnome-shell19:45:123
379302599361,24cyclictest1753-21gnome-shell23:08:563
379302599361,16cyclictest3872335-21taskset22:50:263
3793025993611,6cyclictest141rcu_preempt23:33:233
162399360,25rtkit-daemon2197014-21PK-Backend21:22:343
162399360,25rtkit-daemon2197014-21PK-Backend19:56:173
162399360,24rtkit-daemon2197014-21PK-Backend00:39:003
379302599354,5cyclictest141rcu_preempt22:00:163
3793025993523,8cyclictest1753-21gnome-shell22:57:343
3793025993523,8cyclictest1753-21gnome-shell22:46:053
3793025993512,5cyclictest141rcu_preempt21:15:293
3793025993411,6cyclictest141rcu_preempt20:10:143
379302599322,9cyclictest1753-21gnome-shell22:26:183
3793025993216,11cyclictest994-21lldpd23:29:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional