You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 15:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 04, 2025 12:44:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991090,88rtkit-daemon0-21swapper/307:07:033
81225221010,7sleep31617-10pipewire-media-07:20:003
80676821000,5sleep1790182-21kworker/1:107:05:141
8162242920,6sleep30-21swapper/307:30:143
8162242920,6sleep30-21swapper/307:30:143
8885832770,6sleep20-21swapper/210:55:232
8885832770,6sleep20-21swapper/210:55:232
80790527560,9sleep20-21swapper/207:06:032
80837199669,27cyclictest1753-21gnome-shell07:39:132
8791282640,5sleep02197014-21PK-Backend10:30:140
80787326347,9sleep00-21swapper/007:05:370
80837199629,27cyclictest1753-21gnome-shell09:37:562
162399620,10rtkit-daemon1622-21rtkit-daemon09:03:190
80837199618,28cyclictest1753-21gnome-shell12:30:252
80837199617,27cyclictest901784-21unin-run11:35:012
80837199617,26cyclictest872208-21df_inode10:10:162
808371996115,23cyclictest1753-21gnome-shell10:30:052
808371996114,26cyclictest1753-21gnome-shell12:07:542
808371996112,21cyclictest902384-21/usr/sbin/munin11:35:162
80837199609,26cyclictest1753-21gnome-shell11:08:212
80837199608,27cyclictest1753-21gnome-shell09:28:122
80837199608,26cyclictest1753-21gnome-shell12:03:062
80837199607,28cyclictest1753-21gnome-shell09:07:182
80837199607,27cyclictest831568-21date08:15:022
80837199607,27cyclictest1753-21gnome-shell09:15:512
80837199607,27cyclictest1753-21gnome-shell09:15:502
80837199607,27cyclictest1753-21gnome-shell08:59:142
80837199607,27cyclictest1753-21gnome-shell08:05:182
80837199606,28cyclictest809429-21python307:10:282
80837199606,28cyclictest809429-21python307:10:282
80837199606,28cyclictest1753-21gnome-shell11:23:242
808371996016,23cyclictest1753-21gnome-shell07:55:192
808371996016,23cyclictest1753-21gnome-shell07:55:192
808371996013,26cyclictest815955-21/usr/sbin/munin07:30:112
808371996013,26cyclictest815955-21/usr/sbin/munin07:30:112
808371996013,24cyclictest1753-21gnome-shell10:23:212
162399600,9rtkit-daemon1622-21rtkit-daemon10:25:191
80837199598,25cyclictest883689-21users10:40:312
808371995916,16cyclictest1753-21gnome-shell10:17:232
808371995913,27cyclictest1753-21gnome-shell12:27:522
808371995913,26cyclictest1753-21gnome-shell10:08:592
808371995913,25cyclictest1753-21gnome-shell12:10:042
808371995913,25cyclictest1753-21gnome-shell12:10:032
808371995913,25cyclictest1753-21gnome-shell08:15:532
808371995912,23cyclictest1753-21gnome-shell11:11:082
808371995911,24cyclictest1753-21gnome-shell07:25:022
808371995910,19cyclictest3242253-21nautilus08:25:022
80837199586,28cyclictest1753-21gnome-shell08:25:202
808371995816,20cyclictest1753-21gnome-shell09:10:132
808371995814,25cyclictest1753-21gnome-shell07:18:002
808371995813,25cyclictest1753-21gnome-shell11:42:012
808371995813,25cyclictest1753-21gnome-shell09:21:132
808371995813,25cyclictest1753-21gnome-shell09:21:132
808371995812,26cyclictest1753-21gnome-shell10:39:472
808371995812,25cyclictest1753-21gnome-shell09:00:422
162399580,8rtkit-daemon1622-21rtkit-daemon10:09:591
162399580,8rtkit-daemon1622-21rtkit-daemon07:31:170
162399580,8rtkit-daemon1622-21rtkit-daemon07:31:160
808373995721,19cyclictest42-21ksoftirqd/312:40:013
808373995717,11cyclictest42-21ksoftirqd/309:05:013
80837199578,28cyclictest1753-21gnome-shell09:34:322
808371995714,22cyclictest1753-21gnome-shell12:17:522
808371995713,25cyclictest1753-21gnome-shell11:01:262
808371995713,25cyclictest1753-21gnome-shell08:37:252
808371995712,23cyclictest867977-21taskset09:58:362
162399570,8rtkit-daemon1622-21rtkit-daemon11:50:141
162399570,14rtkit-daemon1622-21rtkit-daemon12:03:371
808373995636,15cyclictest438-20systemd-journal10:31:503
80837199569,28cyclictest1753-21gnome-shell10:51:132
8428502550,6chrt0-21swapper/008:45:210
808373995520,19cyclictest42-21ksoftirqd/309:00:013
80837199559,26cyclictest1753-21gnome-shell10:00:172
80837199559,26cyclictest1753-21gnome-shell10:00:172
80837199555,16cyclictest827447-21sed08:00:292
808371995513,17cyclictest1753-21gnome-shell11:55:292
808371995511,24cyclictest1753-21gnome-shell11:52:592
162399550,9rtkit-daemon1622-21rtkit-daemon11:35:450
162399550,8rtkit-daemon1622-21rtkit-daemon11:45:551
162399550,38rtkit-daemon1753-21gnome-shell10:16:560
808373995420,18cyclictest42-21ksoftirqd/307:40:013
80837199547,28cyclictest1753-21gnome-shell07:20:582
808371995417,17cyclictest895799-21ntp_states11:15:292
808371995415,17cyclictest843036-21needreboot08:45:232
808371995414,16cyclictest820716-21taskset07:41:512
808371995413,11cyclictest1753-21gnome-shell09:51:202
162399540,9rtkit-daemon1622-21rtkit-daemon10:11:131
80837399537,14cyclictest42-21ksoftirqd/310:25:023
808371995312,23cyclictest864348-21taskset09:46:302
162399530,9rtkit-daemon1622-21rtkit-daemon12:13:441
162399530,9rtkit-daemon1622-21rtkit-daemon12:13:431
162399530,9rtkit-daemon1622-21rtkit-daemon08:43:241
162399530,9rtkit-daemon1622-21rtkit-daemon08:24:511
162399530,9rtkit-daemon1622-21rtkit-daemon08:16:170
162399530,9rtkit-daemon1622-21rtkit-daemon07:59:450
162399530,9rtkit-daemon1622-21rtkit-daemon07:59:450
162399530,8rtkit-daemon1622-21rtkit-daemon08:34:111
808373995219,9cyclictest42-21ksoftirqd/310:35:333
808373995218,9cyclictest42-21ksoftirqd/308:40:023
808371995213,15cyclictest862019-21taskset09:40:212
808371995213,15cyclictest862019-21taskset09:40:212
808371995212,19cyclictest1753-21gnome-shell08:51:032
808366995218,29cyclictest822238-21grep07:45:301
162399520,9rtkit-daemon1622-21rtkit-daemon12:17:001
162399520,8rtkit-daemon1622-21rtkit-daemon11:10:361
162399520,8rtkit-daemon1622-21rtkit-daemon09:16:351
162399520,8rtkit-daemon1622-21rtkit-daemon09:16:351
808371995114,18cyclictest1753-21gnome-shell08:40:192
808371995112,16cyclictest1753-21gnome-shell07:53:592
808371995111,22cyclictest1753-21gnome-shell10:29:122
808366995118,29cyclictest861882-21awk09:40:191
808366995118,29cyclictest861882-21awk09:40:191
162399510,8rtkit-daemon1622-21rtkit-daemon11:01:531
162399510,8rtkit-daemon1622-21rtkit-daemon10:19:311
162399510,8rtkit-daemon1622-21rtkit-daemon09:13:091
162399510,8rtkit-daemon1622-21rtkit-daemon08:14:210
162399510,8rtkit-daemon1622-21rtkit-daemon07:38:160
808373995011,14cyclictest42-21ksoftirqd/312:30:243
808371995013,17cyclictest1753-21gnome-shell12:23:542
808366995017,28cyclictest885798-21apt-get10:50:011
808373994918,9cyclictest42-21ksoftirqd/310:50:063
80837199498,16cyclictest1753-21gnome-shell11:45:292
80837199496,26cyclictest899384-21python311:25:292
80837199496,20cyclictest923578-21cpuspeed_turbos12:35:152
808366994919,25cyclictest817713-21apt07:35:111
808366994916,28cyclictest848635-21expr09:01:061
8125072480,8sleep01753-21gnome-shell07:20:130
80837399487,20cyclictest42-21ksoftirqd/308:20:233
80837399485,20cyclictest42-21ksoftirqd/310:08:593
808373994815,17cyclictest42-21ksoftirqd/307:25:003
808373994815,17cyclictest1753-21gnome-shell07:44:483
808373994814,19cyclictest42-21ksoftirqd/309:20:003
808373994814,19cyclictest42-21ksoftirqd/309:20:003
808371994812,15cyclictest1753-21gnome-shell07:47:042
80836699482,11cyclictest1753-21gnome-shell08:45:231
80836699481,8cyclictest924593-40lsb_release12:36:261
162399480,40rtkit-daemon1775-21gvfs-afc-volume08:56:161
808373994713,18cyclictest42-21ksoftirqd/310:55:263
808373994713,18cyclictest42-21ksoftirqd/310:55:263
808373994713,10cyclictest42-21ksoftirqd/307:45:193
808373994711,20cyclictest42-21ksoftirqd/309:25:013
808373994711,20cyclictest42-21ksoftirqd/309:25:013
80837199471,20cyclictest837736-21load08:30:222
80836699472,41cyclictest895422-21/usr/sbin/munin11:15:191
80836699472,41cyclictest3242253-21nautilus09:25:001
80836699472,41cyclictest3242253-21nautilus09:25:001
808366994717,25cyclictest1753-21gnome-shell10:40:171
80836699471,6cyclictest859426-21dump-pmu-power09:35:001
808366994715,27cyclictest909579-21cat11:55:231
808366994715,27cyclictest823791-21processes07:50:291
808366994713,30cyclictest857694-21dump-pmu-power09:30:001
80836699471,24cyclictest1753-21gnome-shell12:30:421
808366994712,30cyclictest902255-21expr11:35:151
808366994712,30cyclictest808714-21tr07:10:131
808366994712,30cyclictest808714-21tr07:10:131
80837399469,22cyclictest42-21ksoftirqd/311:45:093
80837399467,22cyclictest42-21ksoftirqd/310:25:193
808373994617,9cyclictest42-21ksoftirqd/309:46:573
808373994617,9cyclictest42-21ksoftirqd/308:19:033
808373994617,8cyclictest42-21ksoftirqd/311:40:283
808373994611,17cyclictest2197014-21PK-Backend09:55:263
808373994610,20cyclictest42-21ksoftirqd/309:35:013
808366994619,23cyclictest813834-21sshd07:23:221
80836699461,7cyclictest835962-21/usr/sbin/munin08:25:211
808366994615,26cyclictest3242253-21nautilus10:50:031
808366994612,29cyclictest920110-21ntp_states12:25:281
80836699461,10cyclictest891946-21chrt11:05:151
808362994622,20cyclictest861163-21cron09:40:000
8152492450,5sleep30-21swapper/307:25:313
80837399454,17cyclictest42-21ksoftirqd/312:04:593
808373994529,12cyclictest893432-21grep11:10:143
808373994521,20cyclictest1753-21gnome-shell11:27:253
808373994519,10cyclictest42-21ksoftirqd/311:55:143
808373994515,15cyclictest42-21ksoftirqd/310:40:083
80837399451,13cyclictest42-21ksoftirqd/309:35:223
808373994510,19cyclictest42-21ksoftirqd/312:25:163
80836699452,9cyclictest1753-21gnome-shell08:03:311
80836699451,9cyclictest1753-21gnome-shell11:26:521
808366994519,22cyclictest912405-21acpi12:05:121
80836699451,14cyclictest867355-21ntp_kernel_pll_09:55:251
80836699451,11cyclictest888332-21expr10:55:171
80836699451,11cyclictest888332-21expr10:55:171
80836699451,11cyclictest832733-21chrt08:15:321
162399450,37rtkit-daemon2197014-21PK-Backend11:04:000
80837399446,18cyclictest42-21ksoftirqd/309:55:023
80837399445,16cyclictest42-21ksoftirqd/308:05:143
808373994414,8cyclictest0-21swapper/308:10:143
808373994412,15cyclictest42-21ksoftirqd/307:10:213
808373994412,15cyclictest42-21ksoftirqd/307:10:203
80837399441,17cyclictest838561-21taskset08:34:233
808373994411,19cyclictest42-21ksoftirqd/310:02:533
808373994411,19cyclictest42-21ksoftirqd/310:02:533
808373994411,18cyclictest42-21ksoftirqd/311:15:083
808373994411,17cyclictest1753-21gnome-shell10:10:313
80837199447,11cyclictest1753-21gnome-shell10:46:262
808366994412,29cyclictest845552-21taskset08:55:001
80836699441,11cyclictest811109-21expr07:15:161
808362994427,12cyclictest13-21ksoftirqd/008:45:030
80836299441,6cyclictest141rcu_preempt11:27:260
162399440,5rtkit-daemon810060-21munin-node09:09:431
162399440,37rtkit-daemon2197014-21PK-Backend09:31:330
162399440,36rtkit-daemon808353-21cyclictest10:11:370
8431952430,5sleep32197014-21PK-Backend08:45:323
80837399439,17cyclictest42-21ksoftirqd/312:10:153
80837399439,17cyclictest42-21ksoftirqd/312:10:153
808373994312,14cyclictest42-21ksoftirqd/309:08:113
80836699432,8cyclictest1753-21gnome-shell07:55:251
80836699432,8cyclictest1753-21gnome-shell07:55:251
808366994326,13cyclictest817433-21cron07:35:001
808366994326,13cyclictest817433-21cron07:35:001
80836699432,17cyclictest1753-21gnome-shell08:06:451
80836699431,8cyclictest1753-21gnome-shell10:24:341
80836699431,7cyclictest1753-21gnome-shell10:01:011
80836699431,7cyclictest1753-21gnome-shell10:01:011
808366994316,23cyclictest865114-21cstates09:50:151
808366994315,24cyclictest882551-21latency_hist10:40:021
80836699431,11cyclictest904999-21chrt11:42:231
80836699431,11cyclictest1753-21gnome-shell12:22:451
808362994325,6cyclictest13-21ksoftirqd/008:25:190
80837399428,19cyclictest42-21ksoftirqd/308:00:133
80837399428,18cyclictest42-21ksoftirqd/309:12:263
80837399426,18cyclictest42-21ksoftirqd/309:25:313
808373994215,9cyclictest42-21ksoftirqd/308:41:203
808373994213,6cyclictest141rcu_preempt07:50:313
80836699428,16cyclictest1753-21gnome-shell11:21:101
80836699422,16cyclictest1753-21gnome-shell09:46:011
808366994220,17cyclictest831439-21sed08:15:011
80836699421,8cyclictest1753-21gnome-shell10:30:421
80836699421,7cyclictest1753-21gnome-shell07:40:161
80836699421,16cyclictest1753-21gnome-shell11:30:231
80836299424,15cyclictest13-21ksoftirqd/010:40:020
80836299422,7cyclictest896782-21apt11:20:150
808362994224,13cyclictest438-20systemd-journal12:36:250
162399420,35rtkit-daemon2197014-21PK-Backend10:08:320
8905972412,7sleep31753-21gnome-shell11:00:283
80837399418,19cyclictest42-21ksoftirqd/311:05:593
80837399418,13cyclictest42-21ksoftirqd/310:45:203
80837399417,17cyclictest42-21ksoftirqd/311:52:053
80837399413,22cyclictest42-21ksoftirqd/308:25:283
808373994115,22cyclictest1753-21gnome-shell07:55:103
808373994115,22cyclictest1753-21gnome-shell07:55:103
808373994115,10cyclictest42-21ksoftirqd/312:05:133
80836699418,8cyclictest0-21swapper/109:35:251
80836699411,9cyclictest1753-21gnome-shell08:35:131
80836699411,36cyclictest8156072sleep107:28:521
80836299413,19cyclictest1753-21gnome-shell12:15:120
808362994123,14cyclictest13-21ksoftirqd/012:30:310
808362994122,6cyclictest13-21ksoftirqd/007:40:250
808362994114,7cyclictest141rcu_preempt10:41:000
80837399408,17cyclictest2197014-21PK-Backend10:15:253
808373994023,13cyclictest42-21ksoftirqd/311:23:123
808373994011,13cyclictest42-21ksoftirqd/312:20:243
80836299402,9cyclictest1753-21gnome-shell11:06:350
808362994025,11cyclictest911867-21sshd12:01:320
808362994024,6cyclictest13-21ksoftirqd/011:40:260
808362994022,6cyclictest13-21ksoftirqd/010:50:310
808362994021,15cyclictest13-21ksoftirqd/009:25:490
808362994013,6cyclictest884499-21tr10:45:140
162399400,32rtkit-daemon13-21ksoftirqd/008:37:350
80837399396,17cyclictest42-21ksoftirqd/311:35:393
808373993930,6cyclictest411rcuc/311:30:353
808362993922,13cyclictest13-21ksoftirqd/011:13:060
808362993922,12cyclictest854892-21fw_packets09:20:180
808362993922,12cyclictest854892-21fw_packets09:20:180
808362993921,6cyclictest13-21ksoftirqd/011:45:300
808362993921,14cyclictest812273-21pool-org.gnome.07:20:000
808362993921,13cyclictest13-21ksoftirqd/010:20:150
808362993911,6cyclictest141rcu_preempt08:01:140
9143732380,6sleep02197014-21PK-Backend12:10:150
9143732380,6sleep02197014-21PK-Backend12:10:150
80837399381,21cyclictest917337-21taskset12:19:583
80836299389,9cyclictest141rcu_preempt11:55:420
808362993821,6cyclictest13-21ksoftirqd/007:13:190
808362993821,6cyclictest13-21ksoftirqd/007:13:190
808362993821,12cyclictest849298-21apt09:05:150
80836299381,33cyclictest1753-21gnome-shell09:42:040
80836299381,33cyclictest1753-21gnome-shell09:42:040
162399380,30rtkit-daemon13-21ksoftirqd/008:05:400
808373993711,11cyclictest1753-21gnome-shell09:40:533
808373993711,11cyclictest1753-21gnome-shell09:40:533
808362993726,7cyclictest1753-21gnome-shell09:19:480
808362993726,7cyclictest1753-21gnome-shell09:19:480
808362993720,11cyclictest13-21ksoftirqd/009:55:130
808362993718,7cyclictest2287-21gmain10:58:010
808362993718,7cyclictest2287-21gmain10:58:010
808362993718,11cyclictest13-21ksoftirqd/011:15:280
162399370,30rtkit-daemon2197014-21PK-Backend10:04:250
162399370,30rtkit-daemon2197014-21PK-Backend10:04:250
80836299362,7cyclictest1753-21gnome-shell12:05:190
808362993625,6cyclictest13-21ksoftirqd/010:27:000
80836299362,30cyclictest838632-21turbostat08:35:000
808362993621,11cyclictest920913-21systemd12:30:020
808362993620,6cyclictest13-21ksoftirqd/007:27:520
808362993611,6cyclictest141rcu_preempt11:53:010
808362993611,5cyclictest141rcu_preempt09:45:230
808362993610,6cyclictest141rcu_preempt09:55:010
80837399351,16cyclictest1753-21gnome-shell08:50:543
808362993524,6cyclictest13-21ksoftirqd/007:50:000
808362993519,5cyclictest13-21ksoftirqd/009:11:570
808362993515,6cyclictest141rcu_preempt11:33:040
808362993423,6cyclictest13-21ksoftirqd/012:20:280
808362993423,6cyclictest13-21ksoftirqd/008:55:150
808362993310,6cyclictest141rcu_preempt07:54:260
808362993310,5cyclictest141rcu_preempt08:20:060
8449422310,7sleep02197014-21PK-Backend08:50:280
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional