You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 00:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sat Nov 29, 2025 12:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
214451823000,11sleep10-21swapper/107:05:221
222127021110,6sleep3411rcuc/310:40:293
218205321080,4sleep1251rcuc/108:50:231
222135821010,8sleep01753-21gnome-shell10:40:300
214516028166,9sleep00-21swapper/007:09:130
214489728057,15sleep30-21swapper/307:05:503
22525712790,8sleep2331rcuc/212:10:162
22525712790,8sleep2331rcuc/212:10:162
21469932770,5sleep00-21swapper/007:14:430
21469932770,5sleep00-21swapper/007:14:430
162399750,5rtkit-daemon0-21swapper/211:11:072
214415427458,9sleep20-21swapper/207:05:172
214538499732,43cyclictest1753-21gnome-shell07:11:382
214538499732,43cyclictest1753-21gnome-shell07:11:382
162399730,5rtkit-daemon0-21swapper/207:56:352
214538499711,44cyclictest2161984-21taskset07:53:222
214538499711,44cyclictest2161984-21taskset07:53:222
214538499702,39cyclictest1753-21gnome-shell07:17:242
214538499702,39cyclictest1753-21gnome-shell07:17:242
214538499691,23cyclictest2259823-21irqstats12:30:222
214538499684,46cyclictest2171876-21/usr/sbin/munin08:20:272
214538499678,27cyclictest2158838-21cstates07:45:142
214538499671,40cyclictest2209783-21cstates10:10:162
214538499671,40cyclictest2209783-21cstates10:10:162
162399670,32rtkit-daemon0-21swapper/109:50:441
214538499662,42cyclictest1753-21gnome-shell11:50:012
214538499662,34cyclictest2199467-21/usr/sbin/munin09:40:232
214538499662,34cyclictest2199467-21/usr/sbin/munin09:40:232
2145384996618,23cyclictest2224122-21cpuspeed_turbos10:50:142
214538499652,40cyclictest1753-21gnome-shell10:05:132
214538499652,37cyclictest1753-21gnome-shell07:42:482
214538499642,42cyclictest1753-21gnome-shell10:43:162
214538499641,43cyclictest2181803-21cstates08:50:152
214538499632,40cyclictest1753-21gnome-shell10:02:242
214538499632,40cyclictest1753-21gnome-shell10:02:242
214538499631,42cyclictest2246893-21latency_hist11:55:032
214538499631,29cyclictest2216527-21systemd-run10:30:012
214538499622,34cyclictest1753-21gnome-shell11:55:472
214538499622,33cyclictest1753-21gnome-shell12:35:292
214538499621,41cyclictest2236833-21sshd11:25:202
214538499621,33cyclictest2167571-21/usr/sbin/munin08:10:342
214538499621,33cyclictest2167571-21/usr/sbin/munin08:10:342
214538499621,30cyclictest2238675-21/usr/sbin/munin11:30:222
162399620,9rtkit-daemon1622-21rtkit-daemon12:07:261
162399620,9rtkit-daemon1622-21rtkit-daemon12:07:251
214538499612,40cyclictest1753-21gnome-shell12:19:022
214538499612,39cyclictest2228340-40lsb_release11:00:242
214538499612,32cyclictest1753-21gnome-shell09:57:202
214538499611,36cyclictest2180820-21users08:45:332
214538499611,35cyclictest2259288-21latency_hist12:30:032
214538499611,35cyclictest2235118-21taskset11:20:202
214538499611,32cyclictest860997-21snmpd07:38:232
162399610,8rtkit-daemon1622-21rtkit-daemon08:46:001
214538499603,31cyclictest1753-21gnome-shell10:30:152
214538499602,32cyclictest2226107-21df10:55:162
214538499602,31cyclictest1753-21gnome-shell10:40:022
214538499602,29cyclictest1753-21gnome-shell09:02:552
214538499601,36cyclictest2230922-21taskset11:09:272
214538499601,35cyclictest860997-21snmpd10:15:132
162399600,9rtkit-daemon1622-21rtkit-daemon09:34:432
214538499592,34cyclictest1753-21gnome-shell09:23:312
214538499592,33cyclictest1753-21gnome-shell08:37:362
214538499592,32cyclictest1753-21gnome-shell09:47:052
214538499591,34cyclictest2194611-21cat09:25:312
214538499591,33cyclictest1753-21gnome-shell11:35:282
214538499591,32cyclictest438-20systemd-journal10:25:012
214538499591,30cyclictest2153227-21latency_hist07:30:022
214538499583,30cyclictest1753-21gnome-shell11:18:342
214538499582,33cyclictest1753-21gnome-shell08:44:282
214538499582,32cyclictest1753-21gnome-shell12:00:132
214538499582,30cyclictest2170782-21cron08:20:002
214538499581,30cyclictest2165220-21/usr/sbin/munin08:00:322
162399580,8rtkit-daemon1622-21rtkit-daemon09:05:211
21943702570,7sleep30-21swapper/309:25:273
214538499572,33cyclictest1753-21gnome-shell08:58:202
214538499572,32cyclictest1753-21gnome-shell09:50:082
214538499572,30cyclictest1753-21gnome-shell09:14:252
214538499572,27cyclictest2151375-21munin-run07:25:002
214538499571,30cyclictest2251030-21sh12:05:212
214538499571,30cyclictest2251030-21sh12:05:202
22229262560,7sleep02197014-21PK-Backend10:45:250
22229262560,7sleep02197014-21PK-Backend10:45:240
214538699561,13cyclictest1753-21gnome-shell10:40:023
214538499562,37cyclictest0-21swapper/212:20:322
214538499562,30cyclictest1753-21gnome-shell10:48:072
214538499562,30cyclictest1753-21gnome-shell10:48:062
214538499562,29cyclictest2155066-21cat07:35:032
214538499561,31cyclictest2190532-21sh09:15:202
214538499561,29cyclictest2187546-21python309:05:322
2145376995619,19cyclictest26-21ksoftirqd/112:22:391
162399560,9rtkit-daemon1622-21rtkit-daemon12:15:521
162399560,9rtkit-daemon1622-21rtkit-daemon12:11:381
162399560,9rtkit-daemon1622-21rtkit-daemon12:11:381
162399560,9rtkit-daemon1622-21rtkit-daemon07:25:351
214538499552,29cyclictest1753-21gnome-shell08:33:242
214538499551,31cyclictest2173176-21df08:25:182
162399550,9rtkit-daemon1622-21rtkit-daemon08:38:481
162399550,9rtkit-daemon1622-21rtkit-daemon08:32:421
162399550,8rtkit-daemon1622-21rtkit-daemon12:00:371
162399550,8rtkit-daemon1622-21rtkit-daemon08:14:401
162399550,8rtkit-daemon1622-21rtkit-daemon08:14:391
162399550,8rtkit-daemon1622-21rtkit-daemon07:18:371
162399550,8rtkit-daemon1622-21rtkit-daemon07:18:371
2145386995416,19cyclictest42-21ksoftirqd/310:20:023
2145386995415,17cyclictest42-21ksoftirqd/312:15:023
2145386995415,17cyclictest42-21ksoftirqd/312:15:013
214538499542,30cyclictest2242245-21memory11:40:252
214538499542,29cyclictest1753-21gnome-shell08:06:032
214538499542,28cyclictest1753-21gnome-shell09:37:512
214538699525,19cyclictest42-21ksoftirqd/310:05:293
2145386995213,35cyclictest1753-21gnome-shell08:02:283
214538699521,14cyclictest2239658-21munin-plugin-st11:35:003
214538699512,29cyclictest42-21ksoftirqd/311:00:313
2145386995113,11cyclictest42-21ksoftirqd/309:25:013
2145386995018,10cyclictest42-21ksoftirqd/311:43:273
2145386995016,8cyclictest42-21ksoftirqd/309:00:323
214538699501,10cyclictest2197014-21PK-Backend12:22:393
2145376995019,14cyclictest2161774-40apt-check07:51:191
2145376995019,14cyclictest2161774-40apt-check07:51:191
214538699498,19cyclictest42-21ksoftirqd/307:25:003
214538699498,18cyclictest42-21ksoftirqd/308:25:293
2145386994918,8cyclictest42-21ksoftirqd/311:55:273
2145386994911,19cyclictest42-21ksoftirqd/311:45:043
214537699491,17cyclictest26-21ksoftirqd/110:35:211
2145386994836,8cyclictest1069-21runrttasks08:44:343
214538699483,18cyclictest42-21ksoftirqd/310:12:343
214538699483,18cyclictest42-21ksoftirqd/310:12:343
2145386994819,8cyclictest42-21ksoftirqd/312:30:203
2145376994838,6cyclictest26-21ksoftirqd/109:55:251
2145376994822,22cyclictest26-21ksoftirqd/112:25:311
21990882470,6chrt0-21swapper/109:40:141
21990882470,6chrt0-21swapper/109:40:141
214538699478,21cyclictest42-21ksoftirqd/307:45:133
2145386994716,9cyclictest42-21ksoftirqd/309:57:413
2145386994716,10cyclictest42-21ksoftirqd/311:25:243
2145386994715,11cyclictest42-21ksoftirqd/312:05:003
2145386994711,17cyclictest2197014-21PK-Backend07:42:403
2145376994729,7cyclictest1753-21gnome-shell11:11:151
214537299477,5cyclictest141rcu_preempt12:22:410
214538699468,19cyclictest42-21ksoftirqd/308:49:553
214538699463,8cyclictest42-21ksoftirqd/307:30:023
2145386994610,10cyclictest42-21ksoftirqd/308:11:033
2145386994610,10cyclictest42-21ksoftirqd/308:11:033
214537699462,6cyclictest141rcu_preempt10:59:591
2145376994613,17cyclictest1069-21runrttasks11:05:231
214537699461,24cyclictest2211641-21cpuspeed_turbos10:15:161
214538699459,19cyclictest42-21ksoftirqd/310:00:303
214538699459,19cyclictest42-21ksoftirqd/310:00:293
214538699459,19cyclictest42-21ksoftirqd/308:50:153
214538699455,21cyclictest42-21ksoftirqd/311:52:233
2145386994526,14cyclictest438-20systemd-journal08:35:143
2145386994518,8cyclictest42-21ksoftirqd/309:36:023
2145386994514,6cyclictest141rcu_preempt11:36:383
2145386994511,6cyclictest141rcu_preempt09:08:353
2145376994535,6cyclictest26-21ksoftirqd/107:10:291
2145376994535,6cyclictest26-21ksoftirqd/107:10:291
2145376994520,7cyclictest26-21ksoftirqd/111:23:291
22546062440,4sleep30-21swapper/312:15:243
214538699448,16cyclictest42-21ksoftirqd/307:55:143
214538699445,19cyclictest42-21ksoftirqd/311:20:003
214538699442,17cyclictest1753-21gnome-shell08:59:293
214538699441,9cyclictest2197014-21PK-Backend11:00:023
2145386994415,14cyclictest42-21ksoftirqd/310:30:013
2145386994412,15cyclictest42-21ksoftirqd/309:48:593
2145386994410,18cyclictest42-21ksoftirqd/310:45:293
2145386994410,18cyclictest42-21ksoftirqd/310:45:283
214537699446,14cyclictest141rcu_preempt08:25:001
214537699443,21cyclictest26-21ksoftirqd/111:45:181
214537699441,20cyclictest1753-21gnome-shell07:35:351
2145376994411,8cyclictest26-21ksoftirqd/109:35:131
162399440,31rtkit-daemon1753-21gnome-shell10:06:340
22237702430,6sleep1251rcuc/110:50:011
22237702430,6sleep1251rcuc/110:50:011
21485922430,7sleep02197014-21PK-Backend07:15:280
21485922430,7sleep02197014-21PK-Backend07:15:280
2145386994316,10cyclictest42-21ksoftirqd/311:10:003
2145386994311,8cyclictest42-21ksoftirqd/309:15:003
2145386994310,18cyclictest42-21ksoftirqd/307:15:293
2145386994310,18cyclictest42-21ksoftirqd/307:15:293
2145386994310,15cyclictest2197014-21PK-Backend11:10:463
214537699434,18cyclictest26-21ksoftirqd/111:33:351
214537699434,18cyclictest26-21ksoftirqd/109:19:001
2145376994315,5cyclictest141rcu_preempt09:27:351
214538699427,15cyclictest2197014-21PK-Backend07:10:143
214538699427,15cyclictest2197014-21PK-Backend07:10:133
214538699425,17cyclictest2197014-21PK-Backend09:30:253
214538699422,11cyclictest42-21ksoftirqd/307:50:043
214538699422,11cyclictest42-21ksoftirqd/307:50:043
2145386994210,15cyclictest42-21ksoftirqd/308:20:313
214537699429,23cyclictest141rcu_preempt10:05:311
214537699429,14cyclictest141rcu_preempt10:25:301
214537699426,18cyclictest141rcu_preempt08:55:551
2145376994218,21cyclictest26-21ksoftirqd/108:00:161
2145376994217,5cyclictest141rcu_preempt07:58:411
214537699421,20cyclictest26-21ksoftirqd/111:50:211
2145376994210,15cyclictest26-21ksoftirqd/112:30:241
2145372994223,14cyclictest2197832-21ntpq09:35:260
2145372994223,14cyclictest2151244-40apt-check07:24:170
214537299421,11cyclictest2214769-21munin-run10:25:000
214538699415,17cyclictest42-21ksoftirqd/308:19:593
214538699414,16cyclictest42-21ksoftirqd/310:33:233
2145386994120,8cyclictest42-21ksoftirqd/310:21:523
2145386994115,9cyclictest42-21ksoftirqd/309:15:193
214538699411,15cyclictest2197014-21PK-Backend08:05:183
214537699416,18cyclictest26-21ksoftirqd/108:44:111
2145376994122,5cyclictest141rcu_preempt11:59:561
2145376994119,16cyclictest2227954-21cstates11:00:161
2145372994122,14cyclictest2211495-40apt-check10:15:170
2145372994117,20cyclictest2227743-40apt-check11:00:180
2145372994113,7cyclictest141rcu_preempt10:25:170
22012622400,7chrt2197014-21PK-Backend09:45:240
21659622400,5chrt0-21swapper/108:05:161
214538699404,18cyclictest42-21ksoftirqd/312:10:013
214538699404,18cyclictest42-21ksoftirqd/312:10:013
214538699404,18cyclictest42-21ksoftirqd/307:36:223
2145386994023,6cyclictest2225470-21turbostat10:55:003
214537699408,14cyclictest141rcu_preempt10:40:031
2145376994016,10cyclictest26-21ksoftirqd/107:50:011
2145376994015,8cyclictest26-21ksoftirqd/111:39:071
2145372994030,7cyclictest1069-21runrttasks11:08:470
214537299401,6cyclictest2157081-21tr07:40:150
2145372994014,6cyclictest141rcu_preempt12:25:220
2145372994013,6cyclictest141rcu_preempt11:11:560
2145372994013,6cyclictest141rcu_preempt09:05:170
214538699395,9cyclictest2197014-21PK-Backend12:25:173
214537699398,15cyclictest2172570-21kworker/1:110:13:391
214537699398,15cyclictest2172570-21kworker/1:110:13:381
214537699394,19cyclictest26-21ksoftirqd/111:40:261
214537699392,15cyclictest26-21ksoftirqd/107:40:061
214537699392,12cyclictest1753-21gnome-shell07:30:141
2145376993921,13cyclictest1632-21dbus-daemon09:35:001
2145376993919,16cyclictest26-21ksoftirqd/110:00:311
2145376993919,16cyclictest26-21ksoftirqd/110:00:311
2145372993927,8cyclictest1753-21gnome-shell09:00:340
2145372993915,6cyclictest141rcu_preempt12:37:190
2145372993913,11cyclictest141rcu_preempt09:30:220
2145372993912,6cyclictest141rcu_preempt10:15:030
2145372993912,6cyclictest141rcu_preempt10:15:030
2145372993911,7cyclictest141rcu_preempt09:40:170
2145372993911,7cyclictest141rcu_preempt09:40:160
2145372993910,7cyclictest141rcu_preempt08:30:010
162399390,28rtkit-daemon2197014-21PK-Backend10:52:510
214538699389,16cyclictest2197014-21PK-Backend09:51:353
214538699384,18cyclictest42-21ksoftirqd/311:20:283
214537699383,10cyclictest0-21swapper/110:33:581
2145376993824,10cyclictest2186404-21cron09:05:001
214537699382,32cyclictest1753-21gnome-shell09:21:191
2145376993818,6cyclictest26-21ksoftirqd/111:19:361
214537699381,18cyclictest2236431-21/usr/sbin/munin11:25:171
214537299389,7cyclictest141rcu_preempt08:24:440
2145372993821,12cyclictest438-20systemd-journal07:52:410
2145372993821,12cyclictest438-20systemd-journal07:52:400
2145372993811,6cyclictest141rcu_preempt09:51:560
2145386993719,7cyclictest42-21ksoftirqd/312:35:463
2145386993719,14cyclictest1632-21dbus-daemon07:35:013
214537699373,9cyclictest0-21swapper/108:27:401
2145376993726,8cyclictest64850irq/126-i91507:22:201
2145376993721,11cyclictest2261954-21swap12:35:321
2145376993716,5cyclictest141rcu_preempt09:10:121
214537699371,18cyclictest2170781-21cron08:20:001
214537699371,10cyclictest0-21swapper/110:24:411
2145376993710,8cyclictest26-21ksoftirqd/110:52:261
2145372993728,5cyclictest2197014-21PK-Backend08:00:160
2145372993721,12cyclictest2174822-21expr08:30:140
2145372993720,13cyclictest2246902-21ls11:55:000
2145372993712,6cyclictest141rcu_preempt12:30:300
2145372993712,6cyclictest141rcu_preempt11:43:390
214537299371,14cyclictest2197014-21PK-Backend08:47:310
214538699362,14cyclictest2197014-21PK-Backend09:41:563
214538699362,14cyclictest2197014-21PK-Backend09:41:553
2145386993611,10cyclictest42-21ksoftirqd/308:33:083
214537299369,6cyclictest141rcu_preempt08:05:130
214537299362,7cyclictest1753-21gnome-shell09:27:180
2145372993622,10cyclictest1753-21gnome-shell10:59:090
214537699359,5cyclictest141rcu_preempt09:45:121
214537299359,6cyclictest141rcu_preempt11:20:000
214537299359,6cyclictest141rcu_preempt09:10:270
214537299358,6cyclictest141rcu_preempt08:15:010
214537299358,6cyclictest141rcu_preempt08:15:010
2145372993526,5cyclictest2197014-21PK-Backend07:30:140
214537299351,9cyclictest0-21swapper/007:55:340
2145372993519,11cyclictest2160097-21/usr/sbin/munin07:50:020
2145372993518,13cyclictest2243627-40apt-check11:45:130
2145372993514,6cyclictest141rcu_preempt07:29:130
214537299351,30cyclictest2197014-21PK-Backend09:57:240
2145372993512,6cyclictest141rcu_preempt12:10:150
2145372993512,6cyclictest141rcu_preempt12:10:140
2145372993512,6cyclictest141rcu_preempt10:35:220
2145372993510,12cyclictest141rcu_preempt07:36:170
2145372993510,11cyclictest141rcu_preempt11:35:000
214537299347,6cyclictest141rcu_preempt08:50:350
214537299346,12cyclictest141rcu_preempt12:17:100
2145372993426,5cyclictest2197014-21PK-Backend11:25:450
214537299342,5cyclictest0-21swapper/008:38:030
214537299341,8cyclictest2197014-21PK-Backend08:42:100
214537299341,4cyclictest2197014-21PK-Backend08:16:210
214537299341,14cyclictest2250369-21cut12:05:000
214537299339,6cyclictest141rcu_preempt11:20:200
214537299331,29cyclictest2197014-21PK-Backend12:05:230
214537299331,29cyclictest2197014-21PK-Backend12:05:230
2145372993310,7cyclictest141rcu_preempt08:56:470
214537299321,4cyclictest2197014-21PK-Backend10:30:190
214537299321,27cyclictest2197014-21PK-Backend09:20:010
2145372993210,6cyclictest141rcu_preempt11:40:000
214537299311,4cyclictest2197014-21PK-Backend11:55:180
214537299311,27cyclictest2197014-21PK-Backend09:21:120
162399310,24rtkit-daemon2197014-21PK-Backend10:05:010
162399310,24rtkit-daemon2197014-21PK-Backend10:05:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional