You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 09:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Mon Nov 24, 2025 00:44:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
322063922800,13sleep23220641-21cpuspeed_turbos19:05:142
32411262830,6sleep20-21swapper/220:00:152
162399800,5rtkit-daemon2197014-21PK-Backend00:35:073
162399770,5rtkit-daemon2197014-21PK-Backend23:59:203
322188927458,10sleep30-21swapper/319:08:413
322176727445,20sleep00-21swapper/019:07:050
322169127239,9sleep10-21swapper/119:06:091
32947092650,7chrt2197014-21PK-Backend22:30:293
33016412640,13sleep2341ksoftirqd/222:50:282
3222222996321,19cyclictest42-21ksoftirqd/319:44:103
3222207996149,9cyclictest13-21ksoftirqd/019:15:010
162399610,10rtkit-daemon1622-21rtkit-daemon19:52:051
3222222996020,19cyclictest1753-21gnome-shell21:24:153
3222222996020,19cyclictest1753-21gnome-shell21:24:153
162399600,9rtkit-daemon1622-21rtkit-daemon21:30:043
162399600,9rtkit-daemon1622-21rtkit-daemon21:00:503
162399600,9rtkit-daemon1622-21rtkit-daemon21:00:493
162399600,8rtkit-daemon1622-21rtkit-daemon19:29:440
3222212995914,10cyclictest3243879-21seq20:08:221
162399590,9rtkit-daemon1622-21rtkit-daemon00:21:162
162399590,8rtkit-daemon1622-21rtkit-daemon20:40:102
162399590,8rtkit-daemon1622-21rtkit-daemon20:40:102
162399590,8rtkit-daemon1622-21rtkit-daemon19:53:060
162399590,8rtkit-daemon1622-21rtkit-daemon00:39:172
3222222995823,17cyclictest1753-21gnome-shell21:12:323
3222222995820,19cyclictest2197014-21PK-Backend23:29:363
3222222995820,19cyclictest2197014-21PK-Backend23:29:363
3222212995825,28cyclictest3309472-21cron23:14:591
3222212995816,13cyclictest438-20systemd-journal22:25:031
162399580,9rtkit-daemon1622-21rtkit-daemon23:09:162
162399580,8rtkit-daemon1622-21rtkit-daemon23:05:583
162399580,8rtkit-daemon1622-21rtkit-daemon19:18:000
3222222995717,19cyclictest860997-21snmpd20:52:393
3222222995717,19cyclictest860997-21snmpd20:52:393
3222222995717,19cyclictest0-21swapper/300:06:363
3222212995726,27cyclictest1753-21gnome-shell00:12:321
162399570,9rtkit-daemon1622-21rtkit-daemon22:08:313
162399570,8rtkit-daemon1622-21rtkit-daemon21:24:392
162399570,8rtkit-daemon1622-21rtkit-daemon21:24:382
162399570,8rtkit-daemon1622-21rtkit-daemon21:01:462
162399570,8rtkit-daemon1622-21rtkit-daemon21:01:462
162399570,8rtkit-daemon1622-21rtkit-daemon19:56:080
162399570,8rtkit-daemon1622-21rtkit-daemon19:32:510
162399570,8rtkit-daemon1622-21rtkit-daemon19:32:510
162399570,24rtkit-daemon0-21swapper/320:07:033
33076202560,7sleep02197014-21PK-Backend23:10:020
3222222995619,16cyclictest1753-21gnome-shell20:03:423
3222212995619,32cyclictest3338705-21smartctl00:35:291
162399560,8rtkit-daemon1622-21rtkit-daemon23:58:112
162399560,8rtkit-daemon1622-21rtkit-daemon00:08:552
3222222995519,17cyclictest1753-21gnome-shell20:36:203
3222212995518,13cyclictest438-20systemd-journal00:19:561
3222207995514,18cyclictest13-21ksoftirqd/021:05:000
3222207995514,18cyclictest13-21ksoftirqd/021:05:000
162399550,9rtkit-daemon1622-21rtkit-daemon00:32:553
162399550,9rtkit-daemon1622-21rtkit-daemon00:32:553
3222217995412,7cyclictest32539742timerandwakeup20:35:312
3222212995420,29cyclictest1069-21runrttasks23:00:101
3222212995419,11cyclictest438-20systemd-journal19:13:151
3222212995417,10cyclictest3319950-21seq23:42:021
3222207995420,20cyclictest13-21ksoftirqd/020:40:240
3222207995420,20cyclictest13-21ksoftirqd/020:40:240
3222217995310,10cyclictest3266690-21systemd-udevd21:12:212
3222212995317,31cyclictest1753-21gnome-shell00:00:191
3222212995317,13cyclictest3315973-21ntp_kernel_pll_23:30:261
3222212995316,11cyclictest3269543-21cut21:20:221
3222212995316,11cyclictest3269543-21cut21:20:221
3222212995316,10cyclictest3255296-21ls20:40:231
3222212995316,10cyclictest3255296-21ls20:40:231
162399530,8rtkit-daemon1622-21rtkit-daemon19:36:450
322222299528,17cyclictest1753-21gnome-shell21:15:123
3222212995220,27cyclictest3233261-21latency_hist19:40:031
3222212995217,11cyclictest1-21systemd22:20:011
3222212995213,11cyclictest3273486-21cat21:30:311
3222207995240,8cyclictest161rcuc/022:30:190
162399520,8rtkit-daemon1622-21rtkit-daemon22:49:322
162399520,8rtkit-daemon1622-21rtkit-daemon21:52:082
322222299514,27cyclictest1753-21gnome-shell20:29:133
3222222995113,16cyclictest3305798-21latency_hist23:05:033
3222212995120,27cyclictest3280462-21vmstat21:50:321
3222212995120,27cyclictest3227558-21timerandwakeup19:20:311
3222212995117,10cyclictest3265314-21cat21:10:011
3222212995115,12cyclictest3258524-21cat20:50:161
3222212995115,12cyclictest3258524-21cat20:50:151
3222212995113,9cyclictest3297773-21if_err_enp2s022:40:191
3222207995117,19cyclictest13-21ksoftirqd/021:00:010
162399510,8rtkit-daemon1622-21rtkit-daemon19:18:481
162399510,8rtkit-daemon1622-21rtkit-daemon00:33:082
162399510,8rtkit-daemon1622-21rtkit-daemon00:33:082
322222299503,28cyclictest1753-21gnome-shell23:43:213
3222222995026,20cyclictest1753-21gnome-shell21:55:123
3222212995017,10cyclictest3235635-21cat19:45:141
3222212995016,12cyclictest3290522-21sh22:20:281
3222212995015,13cyclictest1632-21dbus-daemon23:59:591
3222212995013,10cyclictest3268285-21cat21:15:301
3222207995030,8cyclictest13-21ksoftirqd/000:39:530
32997552490,6chrt2197014-21PK-Backend22:45:230
32485032490,14sleep126-21ksoftirqd/120:20:261
3222222994924,20cyclictest1753-21gnome-shell23:20:303
322221299492,23cyclictest3242253-21nautilus22:30:041
3222212994916,13cyclictest3313814-21grep23:25:231
3222212994916,13cyclictest3313814-21grep23:25:231
3222212994913,11cyclictest3285983-21seq22:09:371
3222212994912,14cyclictest855-21dbus-daemon23:20:011
3222212994912,10cyclictest3260099-21dpkg20:55:031
3222207994932,7cyclictest13-21ksoftirqd/020:05:230
3222207994918,12cyclictest13-21ksoftirqd/021:25:230
33342032480,22sleep126-21ksoftirqd/100:25:021
3222222994825,19cyclictest1753-21gnome-shell23:45:123
3222222994824,20cyclictest1753-21gnome-shell20:55:303
322222299482,27cyclictest1753-21gnome-shell22:02:063
3222222994812,31cyclictest3275159-21ls21:35:323
3222222994810,19cyclictest1753-21gnome-shell20:34:353
3222212994818,8cyclictest1069-21runrttasks20:17:291
3222212994818,8cyclictest1069-21runrttasks20:17:281
3222212994817,7cyclictest2197014-21PK-Backend21:38:261
3222212994816,11cyclictest3321961-21systemctl23:50:011
3222212994814,9cyclictest3313036-21ls23:25:021
3222212994814,11cyclictest855-21dbus-daemon20:05:011
3222212994812,9cyclictest1946-21gsd-media-keys19:58:331
3222212994812,11cyclictest3301295-21latency22:50:231
3222212994812,10cyclictest3230044-21acpi19:30:121
3222212994812,10cyclictest3230044-21acpi19:30:121
3222212994810,12cyclictest2962348-21rs:main1
3222207994829,5cyclictest141rcu_preempt23:02:420
322222299477,20cyclictest1753-21gnome-shell23:33:183
3222222994717,25cyclictest855-21dbus-daemon00:00:043
3222222994712,31cyclictest1753-21gnome-shell23:52:123
322222299471,10cyclictest3279101-21perf21:50:003
3222212994710,10cyclictest3334334-21seq00:25:051
3222207994711,14cyclictest13-21ksoftirqd/023:50:130
322222299462,17cyclictest3271010-21expr21:25:163
3222212994618,7cyclictest1069-21runrttasks22:48:001
3222212994615,7cyclictest3257401-21switchtime20:45:301
3222212994613,8cyclictest3262123-21grep21:00:161
3222212994613,8cyclictest3262123-21grep21:00:161
3222212994613,11cyclictest3328543-21seq00:05:411
3222212994613,10cyclictest3287799-21latency_hist22:15:011
3222212994612,10cyclictest3322411-21cstates23:50:161
3222212994611,12cyclictest3234225-21basename19:40:221
3222212994610,11cyclictest3265641-21cstates21:10:141
3222207994623,7cyclictest13-21ksoftirqd/022:22:110
322222299457,16cyclictest3248354-21wc20:20:253
322222299454,16cyclictest1753-21gnome-shell19:23:223
322222299454,15cyclictest1753-21gnome-shell21:50:453
322222299453,19cyclictest1753-21gnome-shell20:47:013
322222299453,16cyclictest1753-21gnome-shell19:46:313
322222299453,15cyclictest3310681-21sed23:15:283
3222212994517,6cyclictest3279095-21latency_hist21:50:011
3222212994514,8cyclictest3276012-21cpuspeed_turbos21:40:131
3222212994513,11cyclictest3254152-21sshd20:36:411
3222212994513,10cyclictest3284251-21seq22:04:381
3222212994510,11cyclictest3252026-21which20:30:261
3222212994510,11cyclictest1069-21runrttasks23:40:021
3222207994522,19cyclictest3273927-21unin-run21:35:010
162399450,38rtkit-daemon1775-21gvfs-afc-volume20:10:410
322222299449,18cyclictest1753-21gnome-shell22:45:473
322222299445,11cyclictest1358-1kworker/u9:122:18:533
322222299444,17cyclictest1753-21gnome-shell20:44:233
322222299444,17cyclictest1753-21gnome-shell20:44:223
322222299444,16cyclictest1753-21gnome-shell19:50:193
322222299444,16cyclictest1753-21gnome-shell00:10:323
322222299443,19cyclictest3264025-21expr21:05:153
3222212994418,6cyclictest3336245-21tr00:30:141
3222212994418,6cyclictest3336245-21tr00:30:141
3222212994413,8cyclictest0-21swapper/123:05:261
3222212994411,8cyclictest3250345-21uname20:25:271
322221299441,11cyclictest3271925-21runrttasks21:27:131
322220799448,17cyclictest13-21ksoftirqd/020:00:140
3222207994410,16cyclictest13-21ksoftirqd/023:19:260
322222299437,14cyclictest3247253-21runrttasks20:17:073
322222299437,14cyclictest3247253-21runrttasks20:17:073
322222299436,19cyclictest1753-21gnome-shell19:36:093
322222299432,17cyclictest3334300-21latency_hist00:25:023
322222299432,17cyclictest3276642-21ntp_states21:40:243
322222299432,15cyclictest3229144-21/usr/sbin/munin19:25:293
322222299431,19cyclictest3301642-21gunzip22:50:283
3222217994323,16cyclictest3257967-21unin-run20:50:002
3222217994323,16cyclictest3242253-21nautilus22:45:022
322221299432,15cyclictest3296016-21if_enp2s022:35:211
322221299432,14cyclictest1069-21runrttasks22:55:291
3222207994321,14cyclictest1632-21dbus-daemon00:00:000
3222207994318,9cyclictest13-21ksoftirqd/019:42:180
3222207994318,6cyclictest141rcu_preempt20:15:270
3222207994318,6cyclictest141rcu_preempt20:15:270
3222207994317,7cyclictest141rcu_preempt19:45:030
3222207994311,9cyclictest13-21ksoftirqd/019:25:020
322222299429,9cyclictest3297680-21df_inode22:40:153
322222299429,17cyclictest1753-21gnome-shell23:13:233
322222299423,16cyclictest3296238-21cut22:35:243
322222299423,16cyclictest3222467-21gpg-connect-age19:10:023
322222299422,17cyclictest1753-21gnome-shell19:31:103
322222299422,17cyclictest1753-21gnome-shell19:31:103
322221299421,7cyclictest3281497-21/usr/sbin/munin21:55:191
3222212994210,10cyclictest3244481-21tr20:10:151
322220799429,14cyclictest13-21ksoftirqd/000:19:140
322220799427,16cyclictest3280111-21/usr/sbin/munin21:50:250
322220799424,19cyclictest13-21ksoftirqd/023:25:020
3222207994230,9cyclictest13-21ksoftirqd/022:55:310
322220799422,13cyclictest0-21swapper/000:28:520
3222207994215,15cyclictest13-21ksoftirqd/021:40:140
3222207994215,15cyclictest13-21ksoftirqd/021:20:280
3222207994215,15cyclictest13-21ksoftirqd/021:20:280
3222207994212,18cyclictest13-21ksoftirqd/023:13:220
32333742410,6sleep22197014-21PK-Backend19:40:022
322222299415,19cyclictest1753-21gnome-shell19:16:373
322222299414,14cyclictest1753-21gnome-shell23:35:263
322222299413,15cyclictest1753-21gnome-shell00:25:173
3222222994110,10cyclictest3303696-21taskset22:56:373
322221799415,17cyclictest0-21swapper/221:15:432
3222217994130,7cyclictest1069-21runrttasks22:02:482
3222217994110,15cyclictest3229416-21uptime19:25:332
322220799417,18cyclictest13-21ksoftirqd/000:05:270
322220799417,13cyclictest13-21ksoftirqd/021:15:290
322220799416,11cyclictest1069-21runrttasks00:03:560
322220799415,19cyclictest13-21ksoftirqd/022:40:000
322220799414,20cyclictest13-21ksoftirqd/022:15:170
3222207994129,9cyclictest13-21ksoftirqd/023:35:170
3222207994125,12cyclictest3265606-21grep21:10:130
3222207994123,7cyclictest13-21ksoftirqd/023:40:390
3222207994121,12cyclictest13-21ksoftirqd/021:58:370
3222207994120,7cyclictest13-21ksoftirqd/022:27:360
3222207994115,9cyclictest13-21ksoftirqd/021:45:310
322220799411,23cyclictest13-21ksoftirqd/021:35:160
3222207994112,10cyclictest13-21ksoftirqd/020:25:160
322220799411,10cyclictest3259838-21latency_hist20:55:020
322220799411,10cyclictest3259838-21latency_hist20:55:020
322222299407,17cyclictest1753-21gnome-shell00:18:583
322222299407,11cyclictest3242253-21nautilus22:25:033
3222217994022,14cyclictest855-21dbus-daemon20:10:002
3222217994021,14cyclictest3310628-21swap23:15:282
322220799409,18cyclictest13-21ksoftirqd/020:31:560
322220799408,17cyclictest1753-21gnome-shell22:50:210
322220799404,9cyclictest13-21ksoftirqd/000:35:020
322220799404,9cyclictest13-21ksoftirqd/000:35:020
3222207994027,9cyclictest13-21ksoftirqd/020:37:210
3222207994017,6cyclictest141rcu_preempt22:03:190
3222207994017,6cyclictest141rcu_preempt21:05:120
3222207994016,7cyclictest13-21ksoftirqd/023:30:310
3222207994016,6cyclictest141rcu_preempt00:14:360
322222299395,30cyclictest1753-21gnome-shell22:20:253
322222299394,16cyclictest1753-21gnome-shell22:10:473
322222299394,10cyclictest0-21swapper/320:10:293
3222217993928,7cyclictest3259851-21apt-config20:55:022
3222217993928,7cyclictest3259851-21apt-config20:55:022
322221799392,7cyclictest1753-21gnome-shell21:48:032
3222217993924,12cyclictest3277375-21awk21:45:002
322221799392,33cyclictest1753-21gnome-shell00:01:532
3222217993923,12cyclictest3236723-21sshd19:46:152
322220799398,17cyclictest13-21ksoftirqd/023:26:340
322220799398,17cyclictest13-21ksoftirqd/023:26:340
322220799395,13cyclictest13-21ksoftirqd/022:10:020
3222207993915,6cyclictest141rcu_preempt22:12:210
3222207993911,14cyclictest13-21ksoftirqd/020:50:000
3222222993820,13cyclictest3239035-21basename19:55:123
322221799382,7cyclictest1753-21gnome-shell23:11:132
322221799382,7cyclictest1753-21gnome-shell22:32:262
3222217993820,13cyclictest3242254-21gmain20:55:032
322220799385,13cyclictest13-21ksoftirqd/000:20:140
3222207993815,8cyclictest13-21ksoftirqd/023:45:440
3222207993811,6cyclictest141rcu_preempt20:20:050
322221799377,5cyclictest141rcu_preempt19:22:162
3222217993726,7cyclictest1753-21gnome-shell22:14:372
3222217993725,8cyclictest1753-21gnome-shell22:39:002
3222217993723,10cyclictest3242253-21nautilus23:30:032
3222217993720,13cyclictest3330127-21timerandwakeup00:10:352
3222217993720,12cyclictest3235289-21cron19:45:012
3222217993713,6cyclictest141rcu_preempt21:35:022
3222217993710,15cyclictest3242253-21nautilus22:25:032
162399370,30rtkit-daemon0-21swapper/223:22:092
32519812360,8sleep22197014-21PK-Backend20:30:252
322221799362,7cyclictest1753-21gnome-shell21:06:272
3222217993626,7cyclictest1753-21gnome-shell20:21:252
3222217993623,10cyclictest3314706-21perf23:30:002
3222217993623,10cyclictest3314706-21perf23:30:002
322221799362,30cyclictest3302502-21sort22:55:122
3222217993620,12cyclictest3285725-21sshd22:06:202
3222217993619,12cyclictest3274812-21cat21:35:242
3222217993615,17cyclictest1753-21gnome-shell20:25:212
322221799361,16cyclictest2197014-21PK-Backend20:14:592
322220799361,14cyclictest1069-21runrttasks22:40:070
3222217993517,14cyclictest3238807-21systemd19:55:012
3222217993517,13cyclictest3320840-21df23:45:152
322221799343,27cyclictest3270787-21cpuspeed_turbos21:25:152
3222217993423,7cyclictest3318915-21tr23:40:142
3222217993422,8cyclictest1753-21gnome-shell22:16:582
3222217993422,8cyclictest1753-21gnome-shell00:19:232
3222217993419,11cyclictest3240244-21seq19:56:502
3222217993417,12cyclictest438-20systemd-journal22:00:002
3222217993416,13cyclictest875-21polkitd19:12:272
3222217993416,13cyclictest438-20systemd-journal23:05:002
162399340,28rtkit-daemon0-21swapper/200:29:482
3222217993321,8cyclictest3289743-21/usr/sbin/munin22:20:292
3222217993318,11cyclictest3230490-21sort19:30:202
3222217993318,11cyclictest3230490-21sort19:30:202
3222217993316,12cyclictest994-21lldpd23:53:122
322221799322,13cyclictest1753-21gnome-shell19:15:052
3222217993218,10cyclictest3317341-21expr23:35:142
3222217993119,8cyclictest3246752-21grep20:15:262
3222217993119,8cyclictest3246752-21grep20:15:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional