You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 22:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Tue Dec 02, 2025 12:44:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
386054522720,14sleep23860549-21latency_hist07:10:012
3860085210683,14sleep10-21swapper/107:05:471
1623991010,9rtkit-daemon1622-21rtkit-daemon09:45:031
162399910,78rtkit-daemon0-21swapper/007:05:450
162399900,9rtkit-daemon1622-21rtkit-daemon10:43:191
162399860,8rtkit-daemon1622-21rtkit-daemon10:30:091
162399860,6rtkit-daemon1622-21rtkit-daemon10:46:021
162399810,8rtkit-daemon2287-21gmain11:12:373
39357072760,6sleep22197014-21PK-Backend10:40:142
3860681997564,8cyclictest13-21ksoftirqd/007:30:030
3860681997564,8cyclictest13-21ksoftirqd/007:30:030
386016426840,22sleep30-21swapper/307:06:503
38906032660,6chrt0-21swapper/108:30:221
162399630,10rtkit-daemon1622-21rtkit-daemon11:21:482
3860690996022,12cyclictest438-20systemd-journal12:33:222
3860690996022,12cyclictest438-20systemd-journal12:33:222
3860681996037,8cyclictest13-21ksoftirqd/009:30:170
3860695995927,11cyclictest3944205-21perf11:05:003
162399590,8rtkit-daemon1622-21rtkit-daemon10:51:291
162399590,8rtkit-daemon1622-21rtkit-daemon10:51:291
3860695995815,25cyclictest1753-21gnome-shell11:57:293
3860695995815,25cyclictest1753-21gnome-shell11:57:283
3860695995815,25cyclictest1753-21gnome-shell11:07:133
3860690995820,13cyclictest3938650-21seq10:49:352
3860690995819,12cyclictest3924098-21awk10:05:262
162399580,9rtkit-daemon1622-21rtkit-daemon07:49:172
386069599578,20cyclictest3886695-21expr08:20:133
3860695995613,20cyclictest860997-21snmpd08:02:243
3860695995613,20cyclictest860997-21snmpd08:02:243
3860690995621,12cyclictest3977351-21sshd12:37:222
386068199561,10cyclictest3960765-21chrt11:50:160
162399560,8rtkit-daemon1622-21rtkit-daemon11:19:482
38818332550,16sleep342-21ksoftirqd/308:05:243
3860695995516,22cyclictest1753-21gnome-shell07:14:243
3860695995515,17cyclictest1753-21gnome-shell09:16:443
3860695995514,25cyclictest1753-21gnome-shell07:39:583
3860695995514,25cyclictest1753-21gnome-shell07:39:583
3860695995514,24cyclictest1753-21gnome-shell12:02:003
3860695995514,23cyclictest1753-21gnome-shell08:47:013
3860695995514,23cyclictest1753-21gnome-shell08:47:013
3860690995523,12cyclictest3928284-21cron10:19:592
3860690995517,12cyclictest3885806-21cat08:15:292
3860687995513,18cyclictest26-21ksoftirqd/107:15:021
3860681995521,18cyclictest13-21ksoftirqd/009:57:480
386069599548,28cyclictest1753-21gnome-shell10:05:363
3860695995414,16cyclictest1753-21gnome-shell10:48:243
3860695995413,23cyclictest1753-21gnome-shell10:32:463
3860695995413,19cyclictest3900712-21cstates09:00:153
3860681995427,9cyclictest13-21ksoftirqd/007:35:030
3860681995427,9cyclictest13-21ksoftirqd/007:35:030
3860695995316,15cyclictest1753-21gnome-shell09:26:423
3860695995313,23cyclictest1753-21gnome-shell11:41:143
3860695995312,23cyclictest1753-21gnome-shell10:02:203
3860695995311,18cyclictest3976517-21smartctl12:35:223
3860695995310,26cyclictest1753-21gnome-shell12:33:323
3860695995310,26cyclictest1753-21gnome-shell12:33:323
3860690995316,11cyclictest3879917-21cat08:00:142
3860690995316,11cyclictest3879917-21cat08:00:142
386069599529,19cyclictest3916129-21cron09:49:553
386069599528,20cyclictest3967763-21expr12:10:173
3860695995213,34cyclictest3959163-21if_enp2s011:45:213
3860690995225,7cyclictest1753-21gnome-shell09:51:402
3860690995218,11cyclictest3967387-21cut12:10:012
3860690995217,13cyclictest3969650-21irqstats12:15:212
3860690995214,11cyclictest3864276-21grep07:15:302
3860690995214,11cyclictest3864276-21grep07:15:302
386068799528,5cyclictest141rcu_preempt09:33:301
3860695995115,11cyclictest1753-21gnome-shell12:09:103
3860695995113,15cyclictest1753-21gnome-shell11:39:193
3860695995111,24cyclictest1753-21gnome-shell08:26:143
3860695995111,17cyclictest1753-21gnome-shell09:44:453
3860690995119,9cyclictest1069-21runrttasks08:58:592
3860690995118,13cyclictest3931025-21ntpq10:25:272
386069599509,18cyclictest3864044-21ntpq07:15:263
386069599509,18cyclictest3864044-21ntpq07:15:263
3860695995014,15cyclictest1753-21gnome-shell07:23:263
3860695995010,17cyclictest1753-21gnome-shell08:12:043
3860690995018,11cyclictest3862610-21munin-node07:15:022
3860690995014,12cyclictest3946207-21awk11:10:142
3860690995014,12cyclictest2962348-21rs:main2
3860690995010,15cyclictest855-21dbus-daemon12:30:002
3860687995022,24cyclictest26-21ksoftirqd/108:25:311
386068199501,24cyclictest141rcu_preempt12:00:020
386068199501,24cyclictest141rcu_preempt12:00:010
3860681995012,19cyclictest13-21ksoftirqd/011:40:280
386069599498,19cyclictest3902333-21cpuspeed_turbos09:05:143
3860695994910,18cyclictest3886172-21taskset08:18:573
3860695994910,17cyclictest1753-21gnome-shell10:50:153
3860695994910,17cyclictest1753-21gnome-shell10:50:153
3860695994910,16cyclictest1753-21gnome-shell09:14:453
386069099492,32cyclictest1753-21gnome-shell07:52:312
3860687994916,17cyclictest26-21ksoftirqd/111:50:151
3860681994927,6cyclictest141rcu_preempt12:35:180
386069599488,16cyclictest854-21avahi-daemon09:59:313
3860690994816,11cyclictest34-21ksoftirqd/210:02:342
3860690994810,18cyclictest1753-21gnome-shell10:50:222
3860690994810,18cyclictest1753-21gnome-shell10:50:222
386068199480,44cyclictest1622-21rtkit-daemon10:55:290
386069599479,15cyclictest1753-21gnome-shell07:55:033
386069599479,11cyclictest1753-21gnome-shell10:19:043
386069599478,18cyclictest3907881-21latency09:20:213
3860695994716,27cyclictest3919664-21sed09:55:013
3860695994712,15cyclictest1753-21gnome-shell10:41:283
3860695994712,15cyclictest1753-21gnome-shell08:33:573
386069099479,18cyclictest1753-21gnome-shell10:56:522
3860690994717,10cyclictest3968928-21systemctl12:15:012
386069099471,6cyclictest3963705-21perf12:00:012
386069099471,6cyclictest3963705-21perf12:00:012
3860690994715,10cyclictest1617-10pipewire-media-07:35:152
3860690994715,10cyclictest1617-10pipewire-media-07:35:152
386069099471,24cyclictest1753-21gnome-shell09:30:452
3860681994710,15cyclictest13-21ksoftirqd/010:51:500
3860681994710,15cyclictest13-21ksoftirqd/010:51:490
162399470,5rtkit-daemon2197014-21PK-Backend07:14:050
3860695994618,23cyclictest1753-21gnome-shell07:53:483
3860695994611,14cyclictest1753-21gnome-shell08:59:263
3860690994618,8cyclictest1753-21gnome-shell08:12:472
386069099461,23cyclictest1753-21gnome-shell08:30:152
386068799464,8cyclictest1753-21gnome-shell08:40:191
3860681994622,7cyclictest3888050-21latency_hist08:25:020
386068199461,7cyclictest3884544-21perf08:19:550
162399460,5rtkit-daemon2197014-21PK-Backend09:13:560
386069599456,13cyclictest3914407-21latency_hist09:40:023
3860690994520,7cyclictest1753-21gnome-shell10:13:082
3860690994520,7cyclictest1753-21gnome-shell10:13:082
386068799459,19cyclictest1753-21gnome-shell11:42:281
386068799453,16cyclictest26-21ksoftirqd/108:00:181
386068799453,16cyclictest26-21ksoftirqd/108:00:181
386068199455,15cyclictest0-21swapper/011:10:330
3860681994532,9cyclictest13-21ksoftirqd/008:35:010
3860681994516,12cyclictest13-21ksoftirqd/007:55:010
3860681994515,7cyclictest13-21ksoftirqd/012:29:480
386069599443,15cyclictest0-21swapper/311:24:213
3860695994418,22cyclictest1753-21gnome-shell08:36:103
3860695994416,24cyclictest1753-21gnome-shell10:23:343
3860695994416,11cyclictest1753-21gnome-shell07:29:163
386069599441,38cyclictest1753-21gnome-shell10:56:423
3860690994433,6cyclictest3870660-21cat07:35:032
386069099441,24cyclictest1753-21gnome-shell10:34:162
386069099441,24cyclictest1753-21gnome-shell07:58:022
386069099441,23cyclictest1753-21gnome-shell09:02:102
386068799442,25cyclictest3875284-21python307:45:301
3860687994413,9cyclictest26-21ksoftirqd/111:40:011
386068199447,20cyclictest13-21ksoftirqd/008:55:290
386068199445,6cyclictest141rcu_preempt10:25:010
386068199443,5cyclictest141rcu_preempt10:30:010
3860681994434,6cyclictest13-21ksoftirqd/012:13:050
3860681994424,6cyclictest141rcu_preempt12:34:160
3860681994424,6cyclictest141rcu_preempt12:34:160
386068199442,17cyclictest1753-21gnome-shell11:16:340
3860681994419,9cyclictest13-21ksoftirqd/008:14:200
386069599437,14cyclictest1753-21gnome-shell11:15:223
3860695994316,7cyclictest1753-21gnome-shell10:36:053
3860695994315,24cyclictest3954066-21wc11:30:303
3860695994310,12cyclictest1753-21gnome-shell09:32:573
3860690994323,15cyclictest994-21lldpd09:26:162
3860690994321,7cyclictest3957155-21cstates11:40:182
386069099431,23cyclictest1753-21gnome-shell11:52:442
386069099431,23cyclictest1753-21gnome-shell09:49:472
386069099431,23cyclictest1753-21gnome-shell09:39:482
386069099431,23cyclictest1753-21gnome-shell09:16:012
386069099431,23cyclictest1753-21gnome-shell07:26:172
3860687994323,16cyclictest26-21ksoftirqd/108:08:041
3860687994310,18cyclictest26-21ksoftirqd/107:56:221
386068199437,16cyclictest13-21ksoftirqd/010:32:440
386068199435,15cyclictest0-21swapper/007:43:120
3860681994330,9cyclictest13-21ksoftirqd/008:00:020
386068199431,14cyclictest2197014-21PK-Backend08:10:000
162399430,5rtkit-daemon0-21swapper/012:06:220
386069599427,14cyclictest0-21swapper/308:53:313
386069599425,15cyclictest0-21swapper/312:25:123
3860695994215,23cyclictest3961202-21ntp_states11:50:273
3860695994215,23cyclictest1753-21gnome-shell12:21:053
3860695994215,23cyclictest1753-21gnome-shell11:25:123
3860695994215,23cyclictest1753-21gnome-shell10:28:473
3860695994214,12cyclictest1753-21gnome-shell12:17:193
3860695994213,24cyclictest3874565-21tr07:45:143
3860690994216,21cyclictest3869252-40lsb_release07:30:232
386068799428,20cyclictest1753-21gnome-shell09:08:131
3860687994224,13cyclictest26-21ksoftirqd/109:15:011
3860687994213,16cyclictest26-21ksoftirqd/109:25:021
3860681994225,13cyclictest13-21ksoftirqd/012:15:120
3860681994225,13cyclictest13-21ksoftirqd/010:46:520
3860681994217,6cyclictest141rcu_preempt09:05:140
3860681994216,6cyclictest141rcu_preempt07:16:270
3860681994216,6cyclictest141rcu_preempt07:16:270
386068199421,21cyclictest3965259-21chrt12:03:000
386068199421,19cyclictest3893177-21chrt08:39:430
386068199421,18cyclictest3972053-21chrt12:21:110
386068199421,17cyclictest3914104-21chrt09:36:260
386068199421,15cyclictest3953891-21taskset11:30:250
39137742410,8sleep11753-21gnome-shell09:35:301
3860687994130,8cyclictest0-21swapper/111:55:271
3860687994130,8cyclictest0-21swapper/111:55:271
3860687994124,13cyclictest26-21ksoftirqd/107:51:041
3860687994121,6cyclictest26-21ksoftirqd/111:45:051
3860687994121,16cyclictest26-21ksoftirqd/107:29:081
3860687994112,16cyclictest26-21ksoftirqd/111:15:291
3860681994130,8cyclictest13-21ksoftirqd/008:55:000
386068199412,8cyclictest1753-21gnome-shell08:43:090
3860681994118,8cyclictest13-21ksoftirqd/011:23:000
3860681994117,6cyclictest141rcu_preempt08:47:030
3860681994117,6cyclictest141rcu_preempt08:47:030
386068199411,5cyclictest2197014-21PK-Backend11:40:000
386068199411,22cyclictest3889420-21taskset08:26:000
386068199411,17cyclictest3934912-21chrt10:36:020
39434962400,5sleep2331rcuc/211:00:282
3860695994020,15cyclictest3869403-21/usr/sbin/munin07:30:253
3860695994020,15cyclictest3869403-21/usr/sbin/munin07:30:253
3860687994025,5cyclictest141rcu_preempt08:22:481
3860687994022,13cyclictest26-21ksoftirqd/110:05:181
386068799402,11cyclictest1753-21gnome-shell07:21:121
3860687994021,15cyclictest3973505-21sensors_temp12:25:291
3860687994018,9cyclictest26-21ksoftirqd/108:46:091
3860687994018,9cyclictest26-21ksoftirqd/108:46:091
3860687994016,6cyclictest141rcu_preempt12:37:471
3860687994014,14cyclictest26-21ksoftirqd/111:30:301
3860687994012,24cyclictest3928547-21/usr/sbin/munin10:20:121
3860687994012,16cyclictest141rcu_preempt12:35:011
3860687994012,16cyclictest141rcu_preempt12:35:001
386068199407,17cyclictest13-21ksoftirqd/011:45:320
386068199402,7cyclictest1753-21gnome-shell11:05:260
3860681994022,5cyclictest2197014-21PK-Backend11:25:300
3860681994013,9cyclictest13-21ksoftirqd/010:19:250
3860681994013,8cyclictest13-21ksoftirqd/009:54:500
386068199401,21cyclictest3915973-21taskset09:43:010
386068199401,18cyclictest3880887-21chrt08:02:520
386068199401,18cyclictest3880887-21chrt08:02:520
386068199401,18cyclictest13-21ksoftirqd/009:00:260
386068199401,15cyclictest3906860-21taskset09:17:190
162399400,33rtkit-daemon3862610-21munin-node08:06:382
3860695993912,22cyclictest854-21avahi-daemon07:40:053
386068799392,5cyclictest0-21swapper/110:01:091
3860687993922,5cyclictest26-21ksoftirqd/110:25:151
3860687993922,13cyclictest26-21ksoftirqd/110:56:361
3860687993922,13cyclictest26-21ksoftirqd/109:40:381
3860687993921,6cyclictest26-21ksoftirqd/107:15:191
3860687993921,6cyclictest26-21ksoftirqd/107:15:191
3860687993917,9cyclictest26-21ksoftirqd/109:25:511
3860687993916,5cyclictest141rcu_preempt08:36:401
386068199399,13cyclictest13-21ksoftirqd/010:03:080
3860681993928,6cyclictest13-21ksoftirqd/010:42:300
386068199392,17cyclictest1753-21gnome-shell11:01:280
386068199392,17cyclictest13-21ksoftirqd/009:20:300
386068199391,9cyclictest2197014-21PK-Backend09:25:260
3860681993911,11cyclictest13-21ksoftirqd/007:45:120
3860695993815,19cyclictest1753-21gnome-shell10:12:003
3860695993815,19cyclictest1753-21gnome-shell10:12:003
3860690993822,12cyclictest3955260-21diskstats11:35:192
3860690993813,13cyclictest2179-21gvfsd-metadata10:40:012
386068799382,32cyclictest1753-21gnome-shell09:56:371
3860687993822,13cyclictest26-21ksoftirqd/110:38:051
3860687993822,12cyclictest26-21ksoftirqd/107:31:561
3860687993822,12cyclictest26-21ksoftirqd/107:31:561
3860687993814,11cyclictest141rcu_preempt10:10:191
3860687993814,11cyclictest141rcu_preempt10:10:191
3860687993813,21cyclictest3965704-21dpkg12:05:141
386068799381,23cyclictest3944211-21grep11:05:001
386068799381,21cyclictest2197014-21PK-Backend11:27:001
3860681993827,7cyclictest13-21ksoftirqd/009:50:000
3860681993819,6cyclictest13-21ksoftirqd/007:28:370
3860690993726,7cyclictest3952865-21cron11:30:002
386069099372,31cyclictest1753-21gnome-shell10:21:222
3860690993721,12cyclictest3960217-21pluginstate11:50:002
3860690993718,14cyclictest3242255-21gdbus09:15:002
386068799379,8cyclictest3931807-21kworker/1:211:09:591
386068799379,13cyclictest26-21ksoftirqd/109:17:001
386068799377,17cyclictest26-21ksoftirqd/112:16:001
3860687993722,11cyclictest3927559-21grep10:15:251
3860687993722,11cyclictest3897501-21wc08:50:231
3860687993721,5cyclictest26-21ksoftirqd/107:40:211
3860687993721,12cyclictest3946938-21ntp_kernel_pll_11:10:271
3860687993721,12cyclictest3899470-21pluginstate08:55:301
3860687993720,13cyclictest26-21ksoftirqd/109:00:351
3860687993717,7cyclictest141rcu_preempt12:20:141
3860687993717,16cyclictest1069-21runrttasks08:19:031
3860687993713,21cyclictest3968588-21sshd12:11:111
3860687993713,11cyclictest141rcu_preempt07:35:141
3860687993713,11cyclictest141rcu_preempt07:35:141
3860687993711,14cyclictest26-21ksoftirqd/109:54:041
3860681993721,6cyclictest13-21ksoftirqd/010:12:580
3860681993721,6cyclictest13-21ksoftirqd/010:12:580
3860681993721,12cyclictest3865035-21/usr/sbin/munin07:20:210
3860681993721,12cyclictest13-21ksoftirqd/010:06:330
3860690993618,13cyclictest3953890-21ntp_states11:30:262
386068799366,17cyclictest26-21ksoftirqd/111:22:401
3860687993619,13cyclictest3964682-21ntp_kernel_pll_12:00:231
386069599353,7cyclictest0-21swapper/308:40:403
386069099353,27cyclictest3945494-21timerwakeupswit11:05:322
3860690993523,9cyclictest3866162-21awk07:20:332
3860690993520,11cyclictest3887245-21awk08:20:232
3860690993415,15cyclictest3902283-21cpuspeed_turbos09:05:142
3860690993415,14cyclictest3893849-21df08:40:172
386069099341,29cyclictest3965232-21seq12:02:402
386069099341,28cyclictest994-21lldpd08:35:312
386068799345,10cyclictest26-21ksoftirqd/108:10:181
162399340,27rtkit-daemon2197014-21PK-Backend07:41:232
3860690993320,9cyclictest1753-21gnome-shell09:40:492
3860690993318,11cyclictest3889269-21sed08:25:302
3860690993218,10cyclictest3908807-21cron09:25:002
3860690993217,11cyclictest3897147-21cstates08:50:152
3860690993110,17cyclictest1753-21gnome-shell08:47:072
3860690993110,17cyclictest1753-21gnome-shell08:47:072
386069099299,16cyclictest1753-21gnome-shell12:20:282
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional