You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 22:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Nov 25, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
407868221280,8sleep24078671-21perf07:10:012
419364921060,7sleep22197014-21PK-Backend12:35:012
1623991040,5rtkit-daemon0-21swapper/007:45:211
1623991030,5rtkit-daemon0-21swapper/010:43:351
40905782980,8sleep32197014-21PK-Backend07:40:253
162399870,72rtkit-daemon0-21swapper/007:08:540
162399790,8rtkit-daemon906-21gmain11:12:183
407857427758,13sleep30-21swapper/307:09:093
407846827651,16sleep10-21swapper/107:07:481
162399750,5rtkit-daemon2197014-21PK-Backend10:21:433
162399660,10rtkit-daemon1622-21rtkit-daemon10:18:142
41055532650,4sleep10-21swapper/108:25:021
4078787996225,32cyclictest1753-21gnome-shell08:10:142
162399610,10rtkit-daemon1622-21rtkit-daemon11:41:082
4078787996018,38cyclictest1753-21gnome-shell11:00:022
4078787996018,38cyclictest1753-21gnome-shell11:00:022
162399570,9rtkit-daemon1622-21rtkit-daemon11:20:433
162399570,9rtkit-daemon1622-21rtkit-daemon11:20:433
407878799557,42cyclictest4105719-21sort08:25:142
4078787995518,32cyclictest4109221-21cut08:35:022
4078787995515,35cyclictest1753-21gnome-shell08:50:272
41184762540,6chrt0-21swapper/109:00:151
162399540,9rtkit-daemon1622-21rtkit-daemon10:16:433
4078789995313,17cyclictest42-21ksoftirqd/307:14:343
407878799532,32cyclictest1753-21gnome-shell08:16:322
4078787995321,28cyclictest4165149-21wc11:10:312
4078787995321,28cyclictest1753-21gnome-shell08:00:472
4078787995313,36cyclictest1753-21gnome-shell11:55:012
4078787995216,13cyclictest4133605-21cat09:40:312
41625952510,7sleep34162600-21cpuspeed_turbos11:05:163
4078789995111,20cyclictest42-21ksoftirqd/310:55:033
162399510,8rtkit-daemon1622-21rtkit-daemon12:21:370
4078789995025,8cyclictest42-21ksoftirqd/308:15:013
407878799508,16cyclictest1753-21gnome-shell09:39:432
4078787995027,19cyclictest1753-21gnome-shell08:20:132
4078787995021,25cyclictest34-21ksoftirqd/208:44:492
4078787995013,11cyclictest1-21systemd07:50:012
40969112490,5chrt0-21swapper/108:00:151
407878799499,16cyclictest1753-21gnome-shell09:31:362
407878799499,14cyclictest2181-21gdbus11:50:002
4078787994920,25cyclictest4157992-21ntpq10:50:252
4078787994919,7cyclictest4155689-21cstates10:45:212
4078787994919,26cyclictest4137552-21latency_hist09:55:022
4078787994916,14cyclictest4168746-21users11:20:322
4078787994916,14cyclictest4168746-21users11:20:312
4078787994915,12cyclictest4099774-21uptime08:05:322
4078787994914,13cyclictest4142048-21ntpq10:05:272
4078787994913,11cyclictest4109946-21cat08:35:232
41890302480,8chrt0-21swapper/212:20:252
41032452480,16sleep013-21ksoftirqd/008:15:310
407878999488,20cyclictest42-21ksoftirqd/312:40:013
4078789994812,19cyclictest42-21ksoftirqd/308:20:003
407878799488,16cyclictest1753-21gnome-shell12:04:322
407878799487,16cyclictest1753-21gnome-shell09:10:192
4078787994821,23cyclictest4114390-21dump-pmu-power08:50:012
4078787994817,7cyclictest4186822-21cpuspeed12:15:142
4078787994816,7cyclictest4173563-21netstat11:35:232
4078787994816,6cyclictest4141015-21latency_hist10:05:012
4078787994816,11cyclictest34-21ksoftirqd/209:17:492
4078787994815,7cyclictest4160726-21cpu11:00:142
4078787994815,7cyclictest4160726-21cpu11:00:132
4078787994815,7cyclictest1069-21runrttasks07:52:372
4078787994812,8cyclictest4190633-21sh12:25:212
4078787994812,10cyclictest4167558-21cat11:20:022
407878999479,14cyclictest2197014-21PK-Backend07:55:473
407878799479,17cyclictest1753-21gnome-shell10:33:192
407878799478,14cyclictest1753-21gnome-shell07:15:012
407878799476,16cyclictest1753-21gnome-shell11:58:342
407878799475,16cyclictest1753-21gnome-shell12:08:342
4078787994717,6cyclictest4155149-21sort10:45:022
4078787994716,7cyclictest4090940-21timerandwakeup07:40:322
4078787994716,6cyclictest4135815-21latency_hist09:50:012
4078787994715,10cyclictest34-21ksoftirqd/211:27:112
4078787994714,7cyclictest4137915-21cpuspeed_turbos09:55:132
407878799471,25cyclictest1753-21gnome-shell07:30:312
4078787994712,10cyclictest4172036-21switchtime11:30:292
4078783994718,23cyclictest4165905-21find11:15:121
162399470,6rtkit-daemon1622-21rtkit-daemon07:18:542
41306102460,7sleep02197014-21PK-Backend09:35:010
407878999463,28cyclictest0-21swapper/308:27:003
407878799464,12cyclictest1825-21JS14:31:442
407878799463,16cyclictest1753-21gnome-shell09:23:102
407878799462,17cyclictest1753-21gnome-shell07:35:232
407878799462,16cyclictest1753-21gnome-shell07:20:282
4078787994619,7cyclictest1753-21gnome-shell10:20:282
4078787994615,6cyclictest311irq_work/208:58:062
4078787994613,9cyclictest4119047-21sort09:00:262
407878799461,22cyclictest1753-21gnome-shell07:57:442
4078787994610,16cyclictest1753-21gnome-shell11:05:392
4078780994633,8cyclictest4117850-21apt-get09:00:110
407878099462,41cyclictest0-21swapper/007:57:490
407878099461,12cyclictest4167247-21sshd11:18:280
162399460,5rtkit-daemon0-21swapper/010:27:490
162399460,34rtkit-daemon2197014-21PK-Backend08:46:463
4078789994511,17cyclictest42-21ksoftirqd/312:30:003
407878799453,15cyclictest4128079-21needreboot09:25:242
407878799452,24cyclictest1753-21gnome-shell10:25:072
4078787994516,7cyclictest1753-21gnome-shell10:10:212
4078787994516,6cyclictest311irq_work/212:10:342
407878399451,20cyclictest40965942sshd07:59:441
4078780994533,9cyclictest13-21ksoftirqd/012:40:010
4078789994433,8cyclictest411rcuc/308:40:333
4078789994423,18cyclictest0-21swapper/307:26:563
4078789994423,18cyclictest0-21swapper/307:26:553
4078789994422,16cyclictest1632-21dbus-daemon11:20:013
407878799447,16cyclictest1753-21gnome-shell07:28:212
407878799447,16cyclictest1753-21gnome-shell07:28:212
407878799446,16cyclictest1753-21gnome-shell12:36:442
4078787994417,8cyclictest2197014-21PK-Backend10:36:252
4078783994419,10cyclictest4129322-21expr09:30:131
4078783994417,22cyclictest2104-21gdbus08:29:221
407878099442,7cyclictest4154951-21munin-run10:45:020
4078780994426,13cyclictest13-21ksoftirqd/011:00:010
4078780994426,13cyclictest13-21ksoftirqd/011:00:000
4078780994425,15cyclictest13-21ksoftirqd/009:10:120
4078780994425,14cyclictest13-21ksoftirqd/011:45:120
407878999439,15cyclictest42-21ksoftirqd/307:35:263
407878999439,14cyclictest2197014-21PK-Backend10:00:213
407878999439,13cyclictest42-21ksoftirqd/310:49:253
407878999438,16cyclictest1753-21gnome-shell11:36:373
407878999436,10cyclictest42-21ksoftirqd/310:10:293
4078783994320,19cyclictest4175321-21/usr/sbin/munin11:40:241
4078780994323,15cyclictest13-21ksoftirqd/007:30:130
407878999429,14cyclictest42-21ksoftirqd/309:30:103
4078789994212,8cyclictest42-21ksoftirqd/310:57:483
4078789994212,8cyclictest42-21ksoftirqd/310:57:483
4078789994211,13cyclictest2197014-21PK-Backend08:32:463
4078789994210,18cyclictest42-21ksoftirqd/311:00:223
4078789994210,18cyclictest42-21ksoftirqd/311:00:223
4078783994230,8cyclictest4169371-21kworker/1:111:40:011
4078783994225,12cyclictest4082703-21/usr/sbin/munin07:20:011
4078783994210,5cyclictest141rcu_preempt08:40:151
4078780994225,6cyclictest13-21ksoftirqd/009:20:300
4078780994224,6cyclictest13-21ksoftirqd/010:45:110
4078780994224,13cyclictest855-21dbus-daemon08:55:020
4078780994223,7cyclictest13-21ksoftirqd/007:39:260
4078780994223,6cyclictest13-21ksoftirqd/012:00:160
4078780994220,17cyclictest2962348-21rs:main0
4078780994219,9cyclictest141rcu_preempt08:33:570
407878099421,12cyclictest4134081-21turbostat09:45:000
4078780994211,10cyclictest855-21dbus-daemon08:29:250
162399420,33rtkit-daemon4078773-21cyclictest07:22:593
41415232410,9sleep02197014-21PK-Backend10:05:140
41109752410,14sleep3411rcuc/308:40:013
407878999419,13cyclictest42-21ksoftirqd/312:00:023
407878999418,13cyclictest42-21ksoftirqd/309:59:373
407878999416,13cyclictest2197014-21PK-Backend08:50:393
407878999415,13cyclictest2197014-21PK-Backend12:30:293
407878999414,17cyclictest42-21ksoftirqd/307:50:093
4078789994116,9cyclictest42-21ksoftirqd/309:30:023
407878999411,36cyclictest1753-21gnome-shell09:18:013
407878999411,22cyclictest42-21ksoftirqd/312:20:003
4078783994124,13cyclictest4156257-21ntpq10:45:331
407878399411,11cyclictest0-21swapper/108:36:141
4078780994128,9cyclictest13-21ksoftirqd/012:30:280
4078780994123,14cyclictest3426826-21update-manager08:03:520
162399410,29rtkit-daemon0-21swapper/007:45:360
41899832400,11sleep34189984-21perf-exec12:25:003
41551242400,5sleep32197014-21PK-Backend10:45:033
407878999408,13cyclictest2197014-21PK-Backend08:04:323
407878999406,20cyclictest42-21ksoftirqd/312:12:593
407878999405,13cyclictest2197014-21PK-Backend11:55:063
407878999405,13cyclictest2197014-21PK-Backend07:30:023
407878999402,16cyclictest2197014-21PK-Backend09:45:463
407878999401,10cyclictest2197014-21PK-Backend09:20:283
4078789994010,13cyclictest42-21ksoftirqd/308:08:443
407878399405,10cyclictest4116102-21sshd08:54:551
4078783994028,8cyclictest4089715-21pool-org.gnome.07:40:011
4078783994018,7cyclictest4142036-21ntp_states10:05:271
4078783994015,21cyclictest1-21systemd10:35:011
407878099408,6cyclictest141rcu_preempt08:05:270
4078780994024,12cyclictest4135820-21/usr/sbin/munin09:50:010
4078780994023,6cyclictest13-21ksoftirqd/009:05:280
4078780994023,6cyclictest13-21ksoftirqd/008:50:020
4078780994023,13cyclictest13-21ksoftirqd/010:50:310
4078780994022,14cyclictest13-21ksoftirqd/007:20:280
4078780994021,6cyclictest13-21ksoftirqd/011:00:230
4078780994021,6cyclictest13-21ksoftirqd/011:00:230
4078780994012,6cyclictest141rcu_preempt09:38:500
407878999397,13cyclictest42-21ksoftirqd/310:39:363
407878999396,12cyclictest2197014-21PK-Backend09:40:203
407878999396,10cyclictest42-21ksoftirqd/308:58:563
407878999395,13cyclictest2197014-21PK-Backend11:26:183
407878999395,13cyclictest2197014-21PK-Backend08:20:203
407878999394,17cyclictest42-21ksoftirqd/309:10:303
407878999392,7cyclictest42-21ksoftirqd/309:00:313
4078789993920,15cyclictest4141747-21/usr/sbin/munin10:05:203
407878999391,34cyclictest1753-21gnome-shell09:37:273
407878999391,16cyclictest2197014-21PK-Backend11:50:503
407878999391,15cyclictest2197014-21PK-Backend12:09:113
407878999391,15cyclictest2197014-21PK-Backend10:34:363
407878999391,14cyclictest2197014-21PK-Backend09:05:243
4078783993927,8cyclictest1069-21runrttasks09:51:071
4078783993921,13cyclictest855-21dbus-daemon12:36:271
4078783993913,22cyclictest4145344-21memory10:15:241
4078783993911,12cyclictest4180063-21cat11:55:161
4078783993911,11cyclictest26-21ksoftirqd/112:13:171
4078783993911,11cyclictest141rcu_preempt11:33:551
4078780993926,10cyclictest1753-21gnome-shell09:00:020
407878099392,5cyclictest0-21swapper/010:22:440
4078780993922,13cyclictest13-21ksoftirqd/012:15:220
4078780993922,13cyclictest13-21ksoftirqd/009:25:240
4078780993922,12cyclictest13-21ksoftirqd/010:15:010
4078780993921,6cyclictest13-21ksoftirqd/012:25:140
4078780993921,13cyclictest13-21ksoftirqd/007:25:250
4078780993921,13cyclictest13-21ksoftirqd/007:25:250
407878099391,34cyclictest4082688-21dump-pmu-power07:20:000
162399390,25rtkit-daemon0-21swapper/009:50:190
407878999387,13cyclictest2197014-21PK-Backend10:25:053
407878999385,13cyclictest2197014-21PK-Backend11:33:253
407878999383,19cyclictest42-21ksoftirqd/311:41:563
407878999381,16cyclictest2197014-21PK-Backend11:46:473
407878999381,13cyclictest42-21ksoftirqd/307:47:093
407878399384,10cyclictest0-21swapper/112:30:261
407878399383,6cyclictest141rcu_preempt09:09:441
407878399383,11cyclictest141rcu_preempt09:15:491
407878399382,32cyclictest1753-21gnome-shell08:15:491
4078783993823,11cyclictest4191484-21sshd12:27:291
4078783993822,11cyclictest438-20systemd-journal11:49:271
4078783993820,14cyclictest4090717-21awk07:40:271
407878099389,10cyclictest141rcu_preempt11:35:260
407878099385,15cyclictest4139694-21grep10:00:130
4078780993826,7cyclictest13-21ksoftirqd/008:35:140
4078780993822,12cyclictest4079381-21fw_packets07:10:180
4078780993821,5cyclictest13-21ksoftirqd/011:30:010
4078780993821,12cyclictest4169960-21grep11:25:220
4078780993819,15cyclictest13-21ksoftirqd/011:20:290
4078780993819,15cyclictest13-21ksoftirqd/011:20:290
4078780993819,14cyclictest4185773-21ntp_states12:10:270
407878099381,7cyclictest4181434-21cat12:00:010
407878999377,6cyclictest2197014-21PK-Backend09:50:183
407878999373,10cyclictest42-21ksoftirqd/307:15:293
407878399379,15cyclictest26-21ksoftirqd/109:37:311
407878399375,5cyclictest141rcu_preempt08:05:291
4078783993728,6cyclictest1753-21gnome-shell08:30:301
407878399372,7cyclictest4157629-21diskstats10:50:171
407878399372,7cyclictest4128508-21users09:25:321
4078783993710,12cyclictest26-21ksoftirqd/109:56:201
4078780993726,7cyclictest4150107-21cpuspeed_turbos10:30:140
4078780993726,7cyclictest1753-21gnome-shell07:40:250
4078780993726,6cyclictest13-21ksoftirqd/011:40:230
4078780993725,7cyclictest4105347-21cron08:24:590
4078780993725,7cyclictest13-21ksoftirqd/012:05:280
4078780993724,8cyclictest1753-21gnome-shell11:13:340
4078780993724,5cyclictest13-21ksoftirqd/010:35:320
162399370,23rtkit-daemon0-21swapper/008:11:330
407878399365,5cyclictest141rcu_preempt12:15:211
4078783993626,6cyclictest4094909-21latency_hist07:55:011
4078783993620,12cyclictest4151742-21awk10:35:121
4078783993620,11cyclictest438-20systemd-journal11:29:591
4078783993619,13cyclictest4122502-21expr09:10:151
4078783993619,12cyclictest4146494-21sshd10:20:111
4078783993619,12cyclictest4079570-21memory07:10:231
4078783993618,13cyclictest4148468-21sshd10:25:131
407878399361,5cyclictest141rcu_preempt08:57:541
4078783993614,9cyclictest26-21ksoftirqd/111:00:511
4078783993614,9cyclictest26-21ksoftirqd/111:00:501
4078780993625,7cyclictest1753-21gnome-shell11:05:150
4078780993625,7cyclictest1753-21gnome-shell09:15:300
4078780993624,7cyclictest13-21ksoftirqd/008:42:490
407878099362,30cyclictest0-21swapper/011:51:250
4078780993610,6cyclictest141rcu_preempt09:56:150
407878399359,6cyclictest26-21ksoftirqd/111:20:151
407878399359,6cyclictest26-21ksoftirqd/111:20:141
407878399356,12cyclictest26-21ksoftirqd/109:20:141
407878399353,5cyclictest141rcu_preempt12:00:141
4078783993520,11cyclictest4134341-21basename09:45:121
4078783993520,11cyclictest4084195-21seq07:22:171
4078783993519,12cyclictest4101673-21sshd08:12:261
4078783993517,14cyclictest4178793-21ntpq11:50:231
4078783993513,9cyclictest141rcu_preempt11:05:161
4078783993513,9cyclictest141rcu_preempt07:30:201
4078780993524,6cyclictest13-21ksoftirqd/007:50:120
407878399347,10cyclictest141rcu_preempt12:06:451
407878399343,18cyclictest26-21ksoftirqd/108:45:141
407878399342,11cyclictest141rcu_preempt10:01:041
4078783993419,11cyclictest4143118-21cpuspeed_turbos10:10:141
4078783993419,11cyclictest26-21ksoftirqd/109:43:461
4078783993418,12cyclictest4159472-21awk10:55:221
4078783993418,12cyclictest4159472-21awk10:55:221
4078783993412,9cyclictest141rcu_preempt12:20:301
4078783993321,8cyclictest1069-21runrttasks07:25:261
4078783993321,8cyclictest1069-21runrttasks07:25:261
4078783993317,11cyclictest4165207-21timerwakeupswit11:10:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional