You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-16 - 15:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Thu Apr 16, 2026 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2341620995550,4cyclictest2309430-21kworker/u8:0+flush-259:007:35:263
2341620995249,2cyclictest2527886-21kworker/u8:4+flush-259:010:50:163
2341612995247,4cyclictest2632316-21kworker/u8:0+flush-259:012:15:251
2341620995148,2cyclictest2596160-21kworker/u8:1+events_unbound11:45:233
2341620994642,3cyclictest2527886-21kworker/u8:4+flush-259:010:45:163
2341620994642,3cyclictest2434877-21kworker/u8:2+events_unbound08:40:143
2341620994441,2cyclictest2372918-21kworker/u8:3+events_unbound09:25:223
2341615994441,2cyclictest2326297-21kworker/u8:1+flush-259:007:20:202
2341620994340,2cyclictest2632316-21kworker/u8:0+events_unbound11:50:223
2341620994340,2cyclictest2419076-21kworker/u8:1+flush-259:010:25:213
2341612994337,4cyclictest2341802-21/usr/sbin/munin07:10:131
2341620994238,3cyclictest2434877-21kworker/u8:2+flush-259:009:00:113
2341615994238,3cyclictest2491375-21kworker/u8:2+events_unbound09:50:142
2341620994138,2cyclictest2527886-21kworker/u8:4+events_unbound11:40:233
2341620994138,2cyclictest2434877-21kworker/u8:2+flush-259:008:45:153
2341620994138,2cyclictest2434877-21kworker/u8:2+events_unbound08:55:193
2341615994034,4cyclictest252-21systemd-journal11:55:222
2341612994035,4cyclictest252-21systemd-journal08:59:541
2341612994033,5cyclictest252-21systemd-journal10:55:171
2341620993936,2cyclictest2564163-21kworker/u8:3+events_unbound12:00:183
2341620993935,2cyclictest2564163-21kworker/u8:3+events_unbound11:10:173
2341615993936,2cyclictest2527886-21kworker/u8:4+flush-259:011:35:162
2341615993935,3cyclictest2632316-21kworker/u8:0+flush-259:012:20:252
2341615993935,3cyclictest2507273-21kworker/u8:3+flush-259:010:10:212
2341620993834,3cyclictest2309430-21kworker/u8:0+events_unbound08:15:003
2341620993834,2cyclictest2424413-21kworker/u8:0+flush-259:008:35:223
2341615993835,2cyclictest2326297-21kworker/u8:1+flush-259:007:45:162
2341615993835,2cyclictest2326297-21kworker/u8:1+flush-259:007:45:152
2341612993834,3cyclictest2419076-21kworker/u8:1+flush-259:008:40:241
2341620993734,2cyclictest2527886-21kworker/u8:4+flush-259:011:15:263
2341620993734,2cyclictest2326297-21kworker/u8:1+events_unbound07:20:153
2341620993733,3cyclictest2657595-21kworker/u8:2+events_unbound12:25:223
2341615993731,4cyclictest252-21systemd-journal11:50:002
2341612993731,4cyclictest2595447-21apt-get11:15:111
2341620993633,2cyclictest2434877-21kworker/u8:2+flush-259:009:10:233
2341620993633,2cyclictest2372918-21kworker/u8:3+flush-259:008:15:243
2341620993633,2cyclictest2309430-21kworker/u8:0+flush-259:008:20:213
2341620993632,3cyclictest2372918-21kworker/u8:3+flush-259:008:25:153
2341615993632,3cyclictest2596160-21kworker/u8:1+flush-259:011:16:222
2341615993632,3cyclictest2434877-21kworker/u8:2+flush-259:008:50:222
2341615993632,2cyclictest2564163-21kworker/u8:3+flush-259:010:55:272
2341620993532,2cyclictest2491375-21kworker/u8:2+events_unbound09:35:143
2341620993532,2cyclictest2372918-21kworker/u8:3+events_unbound09:16:223
2341620993531,3cyclictest2372918-21kworker/u8:3+events_unbound08:00:193
2341615993532,2cyclictest2372918-21kworker/u8:3+flush-259:008:00:232
2341615993531,3cyclictest2522764-21kworker/u8:0+flush-259:010:05:192
2341615993531,3cyclictest2419076-21kworker/u8:1+flush-259:008:25:202
2341615993531,3cyclictest2326297-21kworker/u8:1+flush-259:007:25:202
2341612993528,5cyclictest2383193-21latency_hist07:50:011
2341612993528,5cyclictest2383193-21latency_hist07:50:001
2341610993529,4cyclictest2683673-21latency_hist12:40:010
2341610993528,5cyclictest2595183-21latency_hist11:15:010
2341620993431,2cyclictest2632316-21kworker/u8:0+events_unbound12:05:203
2341620993431,2cyclictest2491375-21kworker/u8:2+events_unbound09:45:223
2341620993431,2cyclictest2419076-21kworker/u8:1+flush-259:010:35:163
2341620993431,2cyclictest2419076-21kworker/u8:1+flush-259:010:35:163
2341615993431,2cyclictest2596160-21kworker/u8:1+events_unbound11:50:152
2341612993430,3cyclictest2326297-21kworker/u8:1+flush-259:007:20:011
2341620993330,2cyclictest2372918-21kworker/u8:3+events_unbound09:30:133
2341615993329,3cyclictest2491375-21kworker/u8:2+flush-259:009:40:232
2341615993326,5cyclictest2366937-21ntpq07:30:212
2341610993327,4cyclictest2428599-21ntpq08:30:210
2341620993229,2cyclictest2663076-21kworker/u8:1+events_unbound12:30:193
2341620993229,2cyclictest2632316-21kworker/u8:0+events_unbound11:55:243
2341620993229,2cyclictest2574540-21kworker/u8:0+flush-259:011:05:213
2341620993229,2cyclictest2507273-21kworker/u8:3+events_unbound10:10:223
2341615993228,3cyclictest2419076-21kworker/u8:1+flush-259:009:20:012
2341615993228,3cyclictest2326297-21kworker/u8:1+flush-259:007:45:002
2341615993228,3cyclictest2326297-21kworker/u8:1+events_unbound07:15:222
2341615993226,4cyclictest2496916-21latency_hist09:40:002
2341620993128,2cyclictest2527886-21kworker/u8:4+events_unbound11:00:243
2341620993127,3cyclictest2372918-21kworker/u8:3+events_unbound07:50:223
2341620993127,3cyclictest2372918-21kworker/u8:3+events_unbound07:50:213
2341615993128,2cyclictest2424413-21kworker/u8:0+flush-259:008:30:262
2341615993127,3cyclictest2434877-21kworker/u8:2+events_unbound09:05:202
2341612993129,1cyclictest2475521-21fwupd09:16:211
2341612993125,4cyclictest2558985-21kworker/u8:0+flush-259:010:40:221
2341612993123,6cyclictest2615352-21ntpq11:30:201
2341610993126,3cyclictest2528208-21apt-get10:10:110
2341620993027,2cyclictest2632316-21kworker/u8:0+events_unbound12:20:223
2341620993027,2cyclictest2507273-21kworker/u8:3+events_unbound10:00:163
2341620993027,2cyclictest2507273-21kworker/u8:3+events_unbound09:55:233
2341620993027,2cyclictest2309430-21kworker/u8:0+flush-259:007:40:183
2341620993025,4cyclictest2429508-21latency_hist08:35:003
2341615993026,3cyclictest2527886-21kworker/u8:4+events_unbound10:35:262
2341615993026,3cyclictest2527886-21kworker/u8:4+events_unbound10:35:252
2341615993026,3cyclictest2419076-21kworker/u8:1+flush-259:008:45:252
2341615993026,3cyclictest2372918-21kworker/u8:3+flush-259:009:25:002
2341612993026,3cyclictest2632316-21kworker/u8:0+flush-259:012:25:001
2341612993024,4cyclictest2553805-21latency_hist10:35:011
2341612993023,5cyclictest2367783-21latency_hist07:35:011
2341610993022,5cyclictest252-21systemd-journal11:20:200
2341620992926,2cyclictest2632316-21kworker/u8:0+events_unbound12:10:253
2341620992926,2cyclictest2491375-21kworker/u8:2+flush-259:009:40:163
2341620992921,2cyclictest2434877-21kworker/u8:2+events_unbound08:50:233
2341615992926,2cyclictest2507273-21kworker/u8:3+events_unbound10:00:242
2341615992925,3cyclictest2419076-21kworker/u8:1+events_unbound08:45:002
2341612992925,3cyclictest2564163-21kworker/u8:3+flush-259:012:00:201
2341612992922,5cyclictest2626547-21latency_hist11:45:011
2341620992825,2cyclictest2632316-21kworker/u8:0+events_unbound12:15:173
2341620992825,2cyclictest2507273-21kworker/u8:3+events_unbound10:15:133
2341620992825,2cyclictest2491375-21kworker/u8:2+flush-259:009:50:203
2341620992825,2cyclictest2372918-21kworker/u8:3+events_unbound07:55:193
2341615992825,2cyclictest2663076-21kworker/u8:1+events_unbound12:30:142
2341615992825,2cyclictest2527886-21kworker/u8:4+flush-259:010:50:182
2341615992824,3cyclictest2657595-21kworker/u8:2+events_unbound12:25:142
2341610992822,4cyclictest2636843-21latency_hist11:55:010
2341610992821,5cyclictest2511608-21ntpq09:50:200
2341620992724,2cyclictest2326297-21kworker/u8:1+events_unbound07:45:123
2341620992724,2cyclictest2326297-21kworker/u8:1+events_unbound07:45:123
2341620992724,2cyclictest2309430-21kworker/u8:0+events_unbound07:30:153
2341620992724,2cyclictest2288767-21kworker/u8:2+flush-259:007:10:233
2341615992724,2cyclictest2491375-21kworker/u8:2+flush-259:009:45:202
2341612992722,3cyclictest2527886-21kworker/u8:4+flush-259:011:35:261
2341610992721,4cyclictest2647988-21diskstats12:05:140
2341610992714,11cyclictest2409201-21apt-get08:15:110
2341620992622,3cyclictest2309430-21kworker/u8:0+flush-259:007:29:403
2341615992623,2cyclictest2527886-21kworker/u8:4+flush-259:011:40:202
2341615992622,3cyclictest2596160-21kworker/u8:1+flush-259:011:30:012
2341610992620,4cyclictest2657893-21latency_hist12:14:590
2341610992620,4cyclictest2574514-21latency_hist10:55:010
2341610992620,4cyclictest2465748-21/usr/sbin/munin09:10:150
2341610992620,4cyclictest2393923-21diskstats08:00:130
2341615992522,2cyclictest2491375-21kworker/u8:2+events_unbound09:59:252
2341615992522,2cyclictest2309430-21kworker/u8:0+events_unbound08:20:152
2341615992519,4cyclictest252-21systemd-journal07:40:002
2341612992520,4cyclictest2445190-21apt-get08:50:121
2341610992521,3cyclictest2419076-21kworker/u8:1+flush-259:010:35:290
2341610992521,3cyclictest2419076-21kworker/u8:1+flush-259:010:35:280
2341610992510,13cyclictest2367787-21turbostat07:35:000
2341615992421,2cyclictest2558985-21kworker/u8:0+events_unbound10:40:212
2341615992421,2cyclictest2507273-21kworker/u8:3+events_unbound10:15:212
2341615992420,3cyclictest2419076-21kworker/u8:1+events_unbound08:40:002
2341612992421,2cyclictest2309430-21kworker/u8:0+flush-259:008:05:251
2341612992420,3cyclictest2596160-21kworker/u8:1+flush-259:011:20:261
2341612992420,3cyclictest2522764-21kworker/u8:0+flush-259:010:05:291
2341612992420,3cyclictest2372918-21kworker/u8:3+flush-259:007:50:251
2341612992420,3cyclictest2372918-21kworker/u8:3+flush-259:007:50:251
234161099248,14cyclictest2527842-21apt-get10:09:250
2341620992321,1cyclictest2372918-21kworker/u8:3+flush-259:008:10:013
2341620992319,3cyclictest2419076-21kworker/u8:1+events_unbound10:20:163
2341615992320,2cyclictest2632316-21kworker/u8:0+flush-259:012:15:142
2341615992320,2cyclictest2372918-21kworker/u8:3+flush-259:008:00:002
2341615992320,2cyclictest2288767-21kworker/u8:2+events_unbound07:10:202
2341615992319,3cyclictest2632316-21kworker/u8:0+events_unbound12:10:232
2341615992317,4cyclictest252-21systemd-journal11:02:252
2341612992320,2cyclictest2424413-21kworker/u8:0+events_unbound08:30:241
2341612992319,3cyclictest2491375-21kworker/u8:2+flush-259:009:35:221
2341612992319,3cyclictest2372918-21kworker/u8:3+events_unbound09:25:211
2341612992318,4cyclictest2491375-21kworker/u8:2+events_unbound09:45:171
2341610992312,10cyclictest60350irq/132-enp0s29f1:rx-009:25:200
234161099231,19cyclictest2362189-21latency_hist07:30:000
2341615992219,2cyclictest2403824-21kworker/u8:2+flush-259:008:10:152
2341615992218,3cyclictest2647653-21base6412:05:012
2341615992218,3cyclictest2596160-21kworker/u8:1+flush-259:011:35:012
2341615992218,3cyclictest2372918-21kworker/u8:3+flush-259:008:17:262
2341612992218,3cyclictest2632316-21kworker/u8:0+flush-259:011:55:171
2341612992215,5cyclictest2532195-21ntpq10:10:191
2341610992216,4cyclictest252-21systemd-journal10:21:240
2341610992215,5cyclictest2288767-21kworker/u8:2+flush-259:007:10:150
2341610992214,6cyclictest2480999-21latency_hist09:25:020
234161599210,20cyclictest1-21systemd10:30:002
2341612992117,3cyclictest2309430-21kworker/u8:0+events_unbound08:15:191
2341612992114,5cyclictest2537438-21ntpq10:15:201
234161099213,4cyclictest252-21systemd-journal12:35:000
2341610992114,5cyclictest252-21systemd-journal10:19:240
2341620992018,2cyclictest2564163-21kworker/u8:3+flush-259:011:30:263
2341620992017,2cyclictest2596160-21kworker/u8:1+flush-259:011:20:163
2341620992017,2cyclictest2596160-21kworker/u8:1+events_unbound11:39:253
2341620992017,2cyclictest2527886-21kworker/u8:4+flush-259:010:55:173
2341620992017,2cyclictest2419076-21kworker/u8:1+events_unbound09:24:253
2341615992017,2cyclictest2564163-21kworker/u8:3+events_unbound12:00:152
2341615992017,2cyclictest2527886-21kworker/u8:4+events_unbound10:34:242
2341615992017,2cyclictest2434877-21kworker/u8:2+events_unbound09:00:192
2341615992016,3cyclictest2564163-21kworker/u8:3+events_unbound10:50:012
2341612992018,1cyclictest2309430-21kworker/u8:0+events_unbound07:55:251
2341612992017,2cyclictest2429771-21apt-get08:35:111
2341612992016,3cyclictest2403824-21kworker/u8:2+flush-259:008:10:251
2341612992015,4cyclictest2574540-21kworker/u8:0+flush-259:011:05:241
2341612992015,4cyclictest2491375-21kworker/u8:2+flush-259:009:55:001
2341612992015,4cyclictest2434877-21kworker/u8:2+flush-259:009:00:221
2341612992015,3cyclictest2657595-21kworker/u8:2+events_unbound12:35:001
234161099208,11cyclictest2470981-21cpuspeed_turbos09:15:120
234161099201,5cyclictest2616779-21/usr/sbin/munin11:35:160
234161099201,4cyclictest2455212-21cron09:00:010
2341610992014,4cyclictest252-21systemd-journal10:29:590
2341610992014,4cyclictest2418355-21ntpq08:20:210
2341610992012,5cyclictest2611095-21latency_hist11:30:010
234161099200,8cyclictest2351912-21latency_hist07:20:000
2341620991916,2cyclictest2326297-21kworker/u8:1+events_unbound07:18:463
2341615991916,2cyclictest2372918-21kworker/u8:3+events_unbound09:10:222
2341612991916,2cyclictest2419076-21kworker/u8:1+flush-259:009:45:001
2341612991915,3cyclictest2309430-21kworker/u8:0+flush-259:007:40:151
2341612991914,3cyclictest2372918-21kworker/u8:3+flush-259:008:00:211
2341612991913,5cyclictest2372918-21kworker/u8:3+flush-259:009:20:161
2341612991912,5cyclictest2517580-21latency_hist10:00:001
234161099194,3cyclictest513-21gdbus08:00:000
234161099194,2cyclictest17250irq/128-nvme0q111:20:000
234161099192,6cyclictest2402940-21ntpq08:05:220
234161099192,3cyclictest2522344-21users10:00:250
234161099191,5cyclictest2615363-21ntpq11:30:210
2341610991914,4cyclictest2372918-21kworker/u8:3+flush-259:008:25:170
2341610991913,4cyclictest2491375-21kworker/u8:2+flush-259:009:55:150
234161099190,5cyclictest2388366-21idleruntime-cro07:55:010
234161099190,5cyclictest2388366-21idleruntime-cro07:55:000
234161099190,4cyclictest2502508-21grep09:45:120
2341620991815,2cyclictest2434877-21kworker/u8:2+events_unbound09:10:003
2341620991814,3cyclictest2663076-21kworker/u8:1+events_unbound12:38:243
2341615991815,2cyclictest2527886-21kworker/u8:4+events_unbound11:11:272
2341615991813,2cyclictest2434877-21kworker/u8:2+events_unbound08:55:252
2341612991814,3cyclictest2632316-21kworker/u8:0+flush-259:012:10:181
234161099186,1cyclictest0-21swapper/007:20:160
234161099182,3cyclictest2647388-21date12:05:010
2341610991815,2cyclictest2553807-21idleruntime-cro10:35:000
234161099181,4cyclictest2373357-21grep07:40:130
2341610991814,3cyclictest60350irq/132-enp0s29f1:rx-010:40:120
2341610991811,5cyclictest252-21systemd-journal07:50:000
2341610991811,5cyclictest252-21systemd-journal07:50:000
2341610991810,7cyclictest60350irq/132-enp0s29f1:rx-011:40:120
234161099180,6cyclictest2673395-21latency_hist12:29:590
234161099180,5cyclictest2631703-21latency_hist11:50:010
234161099180,4cyclictest2584875-21apt11:04:590
234161099180,4cyclictest2433934-21/usr/sbin/munin08:35:230
2341615991712,4cyclictest2538370-21kworker/u8:2+events_unbound10:23:202
2341612991713,3cyclictest2596160-21kworker/u8:1+flush-259:011:45:201
2341612991711,4cyclictest2368018-21/usr/sbin/munin07:35:151
2341612991710,5cyclictest2594278-21ntpq11:10:221
234161099171,4cyclictest2460388-21idleruntime-cro09:05:000
2341610991712,3cyclictest2434885-21/usr/sbin/munin08:40:200
234161099170,8cyclictest2372036-21ntpq07:35:220
234161099170,4cyclictest2575340-21irqcore10:55:180
2341620991612,3cyclictest2527886-21kworker/u8:4+events_unbound11:28:193
2341620991612,3cyclictest2491375-21kworker/u8:2+events_unbound10:09:023
2341615991613,2cyclictest2372918-21kworker/u8:3+events_unbound07:51:262
2341615991613,2cyclictest2372918-21kworker/u8:3+events_unbound07:51:252
2341615991612,3cyclictest2603699-21sshd11:20:192
2341615991610,2cyclictest2481045-21kworker/u8:0+events_unbound09:30:142
2341612991612,3cyclictest2527886-21kworker/u8:4+events_unbound11:00:261
234161099169,5cyclictest252-21systemd-journal12:15:170
234161099163,3cyclictest2502223-21cat09:45:000
2341610991614,1cyclictest60350irq/132-enp0s29f1:rx-008:10:240
2341610991612,4cyclictest60350irq/132-enp0s29f1:rx-012:24:160
2341610991612,3cyclictest2574540-21kworker/u8:0+flush-259:011:05:170
234161099160,4cyclictest2444120-21smartctl08:45:220
2341615991511,3cyclictest2663076-21kworker/u8:1+events_unbound12:35:272
2341615991511,3cyclictest2309430-21kworker/u8:0+flush-259:008:05:282
2341612991512,2cyclictest2672569-21ntp_offset12:25:201
2341612991510,4cyclictest2663076-21kworker/u8:1+flush-259:012:35:251
2341610991512,2cyclictest60350irq/132-enp0s29f1:rx-008:50:210
2341610991511,3cyclictest60350irq/132-enp0s29f1:rx-009:35:130
2341612991411,2cyclictest0-21swapper/108:50:011
2341612991410,3cyclictest2596160-21kworker/u8:1+flush-259:011:50:161
2341612991410,3cyclictest2569783-21fschecks_count10:50:151
2341612991410,3cyclictest2538370-21kworker/u8:2+flush-259:010:25:001
2341612991410,3cyclictest2460654-21apt-get09:05:111
2341612991410,3cyclictest2414397-21cpuspeed_turbos08:20:131
234161099141,4cyclictest2460956-21hddtemp_smartct09:05:150
2341610991412,1cyclictest60350irq/132-enp0s29f1:rx-009:30:130
234161299139,3cyclictest2481045-21kworker/u8:0+flush-259:009:30:201
2341612991310,2cyclictest0-21swapper/109:15:011
234161299129,1cyclictest0-21swapper/111:30:011
234161299121,4cyclictest2558959-21awk10:40:001
234161299121,4cyclictest2558959-21awk10:39:591
234161299117,3cyclictest2419076-21kworker/u8:1+flush-259:008:25:201
234161299116,3cyclictest2564382-21/usr/sbin/munin10:45:211
234161299116,3cyclictest2326297-21kworker/u8:1+flush-259:007:25:521
234161299108,1cyclictest0-21swapper/112:05:251
234161299100,1cyclictest490-21dbus-daemon10:29:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional