You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 21:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Sat Nov 29, 2025 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21804392800,6sleep20-21swapper/209:11:192
2054920996150,8cyclictest282-21systemd-journal07:56:251
2054920996150,5cyclictest2209096-21kworker/u8:1+flush-8:011:01:211
2054920996051,7cyclictest2209096-21kworker/u8:1+events_unbound11:11:211
2054920995951,6cyclictest2368205-21kworker/u8:3+flush-8:012:26:271
2054920995850,6cyclictest2096702-21kworker/u8:0+flush-8:009:06:241
2054916995747,8cyclictest2314822-21ntpq11:26:240
2054920995550,4cyclictest2186539-21kworker/u8:0+flush-8:009:21:261
2054920995448,4cyclictest2236241-21kworker/u8:0+events_unbound10:16:291
2054920995446,5cyclictest2040055-21kworker/u8:4+flush-8:007:51:211
2054925995347,4cyclictest2040055-21kworker/u8:4+events_unbound08:21:232
2054920995346,5cyclictest2209096-21kworker/u8:1+flush-8:011:36:221
2054920995245,5cyclictest2328470-21kworker/u8:2+events_unbound11:46:281
2054920995245,5cyclictest2186548-21kworker/u8:4+flush-8:009:51:221
2054920995245,5cyclictest2186548-21kworker/u8:4+flush-8:009:51:221
2054920995146,3cyclictest2176122-21kworker/u8:2+flush-8:009:11:251
2054920995145,3cyclictest2040055-21kworker/u8:4+events_unbound08:01:281
2054920995144,5cyclictest2273548-21kworker/u8:3+flush-8:011:26:231
2054920995144,5cyclictest2096702-21kworker/u8:0+flush-8:008:46:201
2054920995144,4cyclictest2186548-21kworker/u8:4+events_unbound10:01:111
2054920995143,5cyclictest2236241-21kworker/u8:0+events_unbound10:26:131
2054916995146,3cyclictest1998356-21kworker/u8:2+flush-8:007:20:000
2054920995046,3cyclictest2081025-21kworker/u8:2+events_unbound07:36:291
2054920995043,5cyclictest2096702-21kworker/u8:0+events_unbound08:56:231
2054920995043,4cyclictest2171180-21kworker/u8:3+flush-8:009:16:201
2054920994943,4cyclictest2096702-21kworker/u8:0+events_unbound08:36:201
2054920994942,5cyclictest2186539-21kworker/u8:0+flush-8:009:26:221
2054920994942,5cyclictest1998356-21kworker/u8:2+flush-8:007:16:271
2054925994841,5cyclictest2240978-21gpgv10:15:002
2054920994844,3cyclictest2096702-21kworker/u8:0+events_unbound08:11:121
2054920994842,4cyclictest2186539-21kworker/u8:0+flush-8:009:31:241
2054920994841,5cyclictest2209096-21kworker/u8:1+events_unbound10:46:121
2054916994839,6cyclictest2071916-21latency_hist07:25:010
2054920994739,5cyclictest2368205-21kworker/u8:3+flush-8:012:21:221
2054916994737,7cyclictest282-21systemd-journal07:51:240
2054925994640,4cyclictest2072143-21kworker/u8:0+flush-8:007:31:212
2054920994640,4cyclictest2328470-21kworker/u8:2+flush-8:011:56:131
2054920994639,5cyclictest2236241-21kworker/u8:0+flush-8:010:51:271
2054920994540,4cyclictest2209096-21kworker/u8:1+events_unbound09:56:241
2054920994540,4cyclictest2186539-21kworker/u8:0+events_unbound09:36:151
2054920994439,4cyclictest2186548-21kworker/u8:4+events_unbound10:31:231
2054920994439,3cyclictest2059886-21kworker/u8:1+events_unbound07:26:231
2054920994438,4cyclictest2368205-21kworker/u8:3+events_unbound12:31:211
2054920994438,4cyclictest2209096-21kworker/u8:1+flush-8:009:46:181
2054920994437,5cyclictest2345617-21kworker/u8:0+flush-8:012:16:281
2054920994437,5cyclictest2345617-21kworker/u8:0+flush-8:012:16:281
2054920994437,5cyclictest2328470-21kworker/u8:2+flush-8:011:51:121
2054925994337,4cyclictest2280526-21kworker/u8:2+events_unbound10:56:092
2054925994336,5cyclictest2106862-21kworker/u8:1+events_unbound08:31:262
2054920994340,2cyclictest2328470-21kworker/u8:2+events_unbound12:01:271
2054920994339,3cyclictest2209096-21kworker/u8:1+flush-8:009:41:251
2054920994338,4cyclictest2236241-21kworker/u8:0+flush-8:010:21:121
2054920994338,4cyclictest2209096-21kworker/u8:1+flush-8:011:21:431
2054920994335,6cyclictest2209096-21kworker/u8:1+events_unbound11:16:231
2054916994333,7cyclictest2374458-21ntpq12:26:220
2054925994237,4cyclictest2123095-21kworker/u8:2+events_unbound08:16:192
2054920994238,3cyclictest2280526-21kworker/u8:2+events_unbound10:56:271
2054920994237,4cyclictest2260683-21kworker/u8:2+events_unbound10:41:231
2054920994237,3cyclictest2328470-21kworker/u8:2+events_unbound12:06:241
2054920994237,3cyclictest2106862-21kworker/u8:1+events_unbound09:01:231
2054916994238,3cyclictest2209096-21kworker/u8:1+flush-8:011:11:250
2054916994236,4cyclictest2328470-21kworker/u8:2+events_unbound11:56:270
2054916994231,8cyclictest2356073-21diskstats12:11:120
2054916994231,8cyclictest2356073-21diskstats12:11:120
2054925994137,3cyclictest2273548-21kworker/u8:3+flush-8:010:46:232
2054920994135,5cyclictest2096702-21kworker/u8:0+events_unbound08:51:121
2054916994135,4cyclictest2040055-21kworker/u8:4+flush-8:007:41:250
2054916994135,4cyclictest2040055-21kworker/u8:4+flush-8:007:41:240
2054920994035,4cyclictest2273548-21kworker/u8:3+events_unbound11:31:221
2054920994035,4cyclictest2273548-21kworker/u8:3+events_unbound11:31:221
2054920994035,4cyclictest2209096-21kworker/u8:1+flush-8:011:41:151
2054920994035,4cyclictest2040055-21kworker/u8:4+flush-8:008:21:271
2054920994034,5cyclictest2096702-21kworker/u8:0+flush-8:008:41:221
2054920994033,5cyclictest2209096-21kworker/u8:1+events_unbound10:11:151
2054920994031,6cyclictest282-21systemd-journal08:30:001
2054920994030,7cyclictest282-21systemd-journal07:22:091
2054916994034,4cyclictest2186539-21kworker/u8:0+events_unbound09:36:260
2054916994028,9cyclictest2186612-21/usr/sbin/munin09:21:220
2054925993935,3cyclictest2081025-21kworker/u8:2+events_unbound07:36:182
2054925993934,4cyclictest2340352-21kworker/u8:3+flush-8:012:01:222
2054920993935,3cyclictest2260683-21kworker/u8:2+flush-8:010:36:271
2054916993932,4cyclictest2076852-21latency_hist07:30:010
2054916993931,6cyclictest2186548-21kworker/u8:4+flush-8:010:31:150
2054916993929,7cyclictest2266251-21/usr/sbin/munin10:41:200
2054920993828,8cyclictest2291289-21apt-get11:06:071
2054916993832,4cyclictest2186548-21kworker/u8:4+flush-8:009:41:110
2054925993732,4cyclictest2096702-21kworker/u8:0+events_unbound08:51:222
2054925993730,5cyclictest2260683-21kworker/u8:2+flush-8:010:36:292
2054920993733,3cyclictest2106862-21kworker/u8:1+events_unbound08:31:181
2054920993631,4cyclictest2059886-21kworker/u8:1+flush-8:007:46:271
2054916993628,5cyclictest2225924-21latency_hist10:00:000
2054920993531,3cyclictest2358497-21kworker/u8:1+flush-8:012:11:271
2054920993531,3cyclictest2358497-21kworker/u8:1+flush-8:012:11:271
2054920993531,3cyclictest2096702-21kworker/u8:0+flush-8:008:06:421
2054916993528,4cyclictest2081025-21kworker/u8:2+flush-8:007:31:260
2054925993428,5cyclictest282-21systemd-journal09:45:002
2054920993430,3cyclictest2040055-21kworker/u8:4+flush-8:007:41:291
2054920993430,3cyclictest2040055-21kworker/u8:4+flush-8:007:41:291
2054920993425,7cyclictest2077186-21apt-get07:31:061
2054916993427,5cyclictest282-21systemd-journal10:36:360
2054916993424,7cyclictest282-21systemd-journal09:51:150
2054916993424,7cyclictest282-21systemd-journal09:51:140
2054920993326,5cyclictest2059417-21grep07:11:271
2054916993325,5cyclictest2131465-21grep08:21:270
2054925993227,4cyclictest2059886-21kworker/u8:1+flush-8:007:46:202
2054920993227,4cyclictest2201385-21kworker/u8:2+events_unbound10:05:011
2054916993227,3cyclictest2106862-21kworker/u8:1+flush-8:008:56:170
2054916993224,6cyclictest2209096-21kworker/u8:1+flush-8:011:21:230
2054916993224,6cyclictest2096702-21kworker/u8:0+flush-8:008:36:210
2054916993224,5cyclictest2106807-21apt-get08:00:010
2054925993126,4cyclictest2375316-21kworker/u8:2+flush-8:012:36:282
2054925993126,3cyclictest2341305-21fschecks_time11:56:132
2054925993121,8cyclictest2236021-21awk10:10:002
2054920993127,3cyclictest2375316-21kworker/u8:2+flush-8:012:36:261
2054916993127,2cyclictest2251094-21turbostat.cron10:25:000
2054916993125,4cyclictest2273548-21kworker/u8:3+flush-8:011:15:460
2054916993125,2cyclictest2285630-21unixbench_multi10:56:270
2054916993123,6cyclictest2085912-21ntpq07:36:250
2054916993121,7cyclictest2147297-21diskstats08:41:120
2054916993121,7cyclictest2054742-21latency_hist07:10:010
2054916993121,7cyclictest2054742-21latency_hist07:10:010
2054925993026,3cyclictest2209096-21kworker/u8:1+events_unbound09:56:212
2054916993024,4cyclictest2186539-21kworker/u8:0+events_unbound09:26:240
2054925992925,2cyclictest2241733-21cpuspeed_turbos10:16:122
2054925992923,4cyclictest2066565-21unixbench_singl07:16:272
2054925992922,5cyclictest282-21systemd-journal08:15:012
2054916992924,3cyclictest2345617-21kworker/u8:0+events_unbound12:16:160
2054916992924,3cyclictest2345617-21kworker/u8:0+events_unbound12:16:150
2054916992924,3cyclictest2295859-21cron11:10:000
2054916992923,4cyclictest2230475-21unixbench_multi10:01:270
2054916992922,5cyclictest2209096-21kworker/u8:1+flush-8:010:11:230
2054925992823,4cyclictest2186548-21kworker/u8:4+flush-8:010:31:272
2054925992823,3cyclictest2191970-21fschecks_time09:26:122
2054925992820,6cyclictest2055366-21cpuspeed_turbos07:11:092
2054916992823,4cyclictest2176122-21kworker/u8:2+flush-8:009:11:220
2054916992822,5cyclictest2209096-21kworker/u8:1+flush-8:009:46:220
2054925992719,6cyclictest2367910-21kthreadcore12:21:172
2054920992722,4cyclictest2106862-21kworker/u8:1+flush-8:008:19:181
2054916992722,4cyclictest2358497-21kworker/u8:1+flush-8:012:38:450
2054916992722,3cyclictest2245202-21needreboot10:16:210
2054916992720,5cyclictest0-21swapper/011:46:140
205491699271,24cyclictest2091787-21kworker/0:1+events09:06:260
2054925992621,3cyclictest2320244-21unixbench_singl11:31:272
2054925992621,3cyclictest2320244-21unixbench_singl11:31:272
2054925992620,4cyclictest2196607-21aten2_r5power_c09:31:092
2054925992620,4cyclictest2101244-21unixbench_multi07:51:282
2054925992618,6cyclictest2172009-21cpuspeed_turbos09:06:112
2054916992622,3cyclictest2171180-21kworker/u8:3+events_unbound09:16:250
2054916992622,3cyclictest2117818-21sort08:11:170
205491699262,23cyclictest2091787-21kworker/0:1+events10:51:170
205491699262,22cyclictest2091787-21kworker/0:1+events12:21:100
2054916992616,7cyclictest2275814-21latency_hist10:50:010
2054925992519,4cyclictest2107620-21cpuspeed_turbos08:01:112
2054925992517,6cyclictest282-21systemd-journal11:06:152
2054925992517,6cyclictest2156224-21unixbench_multi08:46:272
2054916992521,2cyclictest0-21swapper/007:15:010
2054916992520,3cyclictest0-21swapper/009:35:000
2054916992517,6cyclictest2116073-21ntpq08:06:200
205491699251,19cyclictest2091787-21kworker/0:1+events11:50:170
2054925992420,3cyclictest2320603-21kworker/u8:0+flush-8:011:41:392
2054925992420,2cyclictest2336283-21fschecks_count11:51:112
2054925992419,3cyclictest2096702-21kworker/u8:0+events_unbound08:41:292
2054925992419,3cyclictest2087352-21cpuspeed_turbos07:41:102
2054925992419,3cyclictest2087352-21cpuspeed_turbos07:41:092
2054925992419,3cyclictest0-21swapper/210:55:002
2054925992418,2cyclictest161rcu_preempt11:21:182
2054925992415,7cyclictest2136397-21unixbench_multi08:26:292
205492599240,8cyclictest22475051kthreadcore10:21:162
2054916992420,3cyclictest831-21mta-sts-daemon10:26:170
2054925992320,2cyclictest2216627-21fschecks_count09:51:132
2054925992320,2cyclictest2216627-21fschecks_count09:51:122
2054925992319,3cyclictest2328470-21kworker/u8:2+flush-8:011:46:302
2054925992319,3cyclictest0-21swapper/209:46:282
2054925992319,3cyclictest0-21swapper/209:00:002
2054925992319,2cyclictest2102410-21cpuspeed_turbos07:56:102
2054925992317,4cyclictest2325593-21gpgv11:40:002
2054916992319,2cyclictest0-21swapper/008:51:190
2054916992317,4cyclictest2328470-21kworker/u8:2+events_unbound12:05:000
2054916992316,5cyclictest2328470-21kworker/u8:2+events_unbound11:41:280
205491699231,21cyclictest2091787-21kworker/0:1+events12:31:090
205491699231,21cyclictest2091787-21kworker/0:1+events12:06:300
205491699230,22cyclictest2231882-21sshd10:06:140
20878102220,3chrt451irq_work/307:41:153
20878102220,3chrt451irq_work/307:41:153
205491699223,18cyclictest2091787-21kworker/0:1+events08:32:020
205491699223,18cyclictest2091787-21kworker/0:1+events08:17:580
2054916992217,3cyclictest2156298-21unixbench_singl08:46:270
205492599212,8cyclictest0-21swapper/208:36:122
205492599212,13cyclictest2201491-21apt-get09:36:102
2054925992118,2cyclictest0-21swapper/212:26:182
2054925992117,3cyclictest2230508-21unixbench_singl10:01:282
2054925992117,3cyclictest2171077-21unixbench_multi09:01:282
2054925992116,4cyclictest2357994-21kthreadcore12:11:172
2054925992116,4cyclictest2357994-21kthreadcore12:11:162
2054925992115,4cyclictest2364997-21unixbench_multi12:16:292
2054925992115,4cyclictest2364997-21unixbench_multi12:16:292
205491699210,20cyclictest2091787-21kworker/0:1+events08:02:340
2054916992016,3cyclictest2324934-21mailstats11:36:270
2054916992015,4cyclictest2318136-21kthreadcore11:31:170
2054916992015,4cyclictest2318136-21kthreadcore11:31:170
2054916992015,3cyclictest0-21swapper/007:46:240
20501802204,9sleep20-21swapper/207:06:072
20501802204,9sleep20-21swapper/207:06:072
20501042204,10sleep10-21swapper/107:05:101
20501042204,10sleep10-21swapper/107:05:101
205492599193,1cyclictest0-21swapper/209:21:232
205492599192,13cyclictest2251429-21apt-get10:26:112
2054925991915,3cyclictest1998356-21kworker/u8:2+events_unbound07:21:432
2054925991911,6cyclictest2297043-21sed11:11:162
2054925991911,6cyclictest2183371-21tr09:16:162
2054925991910,7cyclictest2270890-21cron10:45:002
2054916991915,3cyclictest2287062-21tr11:01:160
2054916991915,3cyclictest2132251-21cpu08:26:100
2054925991815,2cyclictest2286392-21aten2_r5power_e11:01:102
2054925991814,2cyclictest0-21swapper/212:06:292
2054925991813,3cyclictest0-21swapper/212:31:162
2054925991811,5cyclictest0-21swapper/211:20:002
2054925991810,6cyclictest282-21systemd-journal11:25:142
20545872183,8sleep30-21swapper/307:08:093
20545872183,8sleep30-21swapper/307:08:093
2054925991713,2cyclictest0-21swapper/207:26:122
2054916991711,4cyclictest2106862-21kworker/u8:1+flush-8:009:01:160
2054925991612,3cyclictest0-21swapper/208:06:082
23459062152,8sleep30-21swapper/312:01:073
21591652156,5sleep30-21swapper/308:51:173
23742512132,7sleep30-21swapper/312:26:203
23397612132,7sleep30-21swapper/311:51:213
23266392133,6sleep30-21swapper/311:41:183
23244692132,7sleep30-21swapper/311:36:203
23060632132,7sleep30-21swapper/311:21:073
23026292134,5sleep30-21swapper/311:16:173
22550842132,7sleep30-21swapper/310:26:213
22469622132,7sleep30-21swapper/310:21:163
22259502132,6sleep30-21swapper/310:00:013
22161392132,6sleep30-21swapper/309:50:023
22161392132,6sleep30-21swapper/309:50:023
22154152131,7sleep30-21swapper/309:46:283
22064392132,7sleep30-21swapper/309:41:073
22053442133,7sleep30-21swapper/309:36:253
21665072132,6sleep30-21swapper/309:00:013
21516312132,7sleep30-21swapper/308:45:003
21420502132,7sleep30-21swapper/308:36:083
21416972132,6sleep30-21swapper/308:35:013
20917672132,6sleep30-21swapper/307:45:013
20677772132,6sleep30-21swapper/307:21:173
20659792133,6sleep30-21swapper/307:16:193
20549792132,6sleep30-21swapper/307:10:023
23852242122,6sleep30-21swapper/312:40:013
23657432123,6sleep30-21swapper/312:21:113
23643752122,6sleep30-21swapper/312:16:213
23643752122,6sleep30-21swapper/312:16:213
22934922122,6sleep30-21swapper/311:06:183
22903692122,6sleep30-21swapper/311:01:293
22757972122,6sleep30-21swapper/310:50:013
22713192122,6sleep30-21swapper/310:46:123
22708262122,6sleep30-21swapper/310:44:153
22603732122,6sleep30-21swapper/310:31:273
22318172122,6sleep30-21swapper/310:06:153
22212992122,6sleep30-21swapper/309:56:123
22011422122,5sleep30-21swapper/309:35:013
21879612123,6sleep30-21swapper/309:21:163
21770952122,6sleep30-21swapper/309:11:133
21522902122,6sleep30-21swapper/308:46:133
21077752122,6sleep30-21swapper/308:01:133
21037762122,6sleep30-21swapper/307:56:173
21008002122,6sleep30-21swapper/307:51:233
20757782122,6sleep30-21swapper/307:26:193
23760032112,5sleep30-21swapper/312:31:143
23557562112,6sleep30-21swapper/312:10:353
23557562112,6sleep30-21swapper/312:10:353
23505902112,5sleep30-21swapper/312:05:013
23413782112,5sleep30-21swapper/311:56:153
23310262112,6sleep30-21swapper/311:45:593
23164592112,5sleep30-21swapper/311:31:143
23164592112,5sleep30-21swapper/311:31:143
23141362112,6sleep30-21swapper/311:26:213
22961732112,5sleep30-21swapper/311:11:083
22813292112,6sleep30-21swapper/310:56:073
22653332112,5sleep30-21swapper/310:36:293
22424342112,5sleep30-21swapper/310:16:183
22363202112,5sleep30-21swapper/310:11:073
21913112112,5sleep30-21swapper/309:25:003
21816522112,5sleep30-21swapper/309:16:063
21716792112,5sleep30-21swapper/309:05:073
21627582112,5sleep30-21swapper/308:56:163
21324802112,5sleep30-21swapper/308:26:143
21275742112,6sleep30-21swapper/308:21:133
21223852112,5sleep30-21swapper/308:16:113
21172572112,5sleep30-21swapper/308:11:093
21159292112,5sleep30-21swapper/308:06:203
20826062112,5sleep30-21swapper/307:36:183
20775662112,5sleep30-21swapper/307:31:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional