You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 22:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Sat Jun 27, 2026 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2182942996259,2cyclictest2348738-21kworker/u8:0+events_unbound09:55:132
2182934996153,5cyclictest2467604-21latency_hist11:45:000
2182942995044,4cyclictest2452094-21latency_hist11:30:012
2182937995046,3cyclictest2499162-21kworker/u8:3+flush-259:012:15:441
2182937994845,2cyclictest2431695-21kworker/u8:4+events_unbound12:10:211
2182937994845,2cyclictest2281545-21kworker/u8:4+events_unbound10:55:011
2182942994744,2cyclictest2177785-21kworker/u8:1+flush-259:007:35:012
2182942994642,3cyclictest2472944-21kworker/u8:0+flush-259:012:00:352
2182937994643,2cyclictest2322723-21kworker/u8:0+events_unbound09:35:221
2182937994643,2cyclictest2213942-21kworker/u8:3+flush-259:007:45:011
2182937994642,3cyclictest2426369-21kworker/u8:3+events_unbound11:15:211
2182942994542,2cyclictest2462668-21kworker/u8:1+events_unbound12:20:152
2182942994441,2cyclictest2462668-21kworker/u8:1+flush-259:011:50:452
2182942994340,2cyclictest2281545-21kworker/u8:4+flush-259:010:40:252
2182942994339,3cyclictest2281545-21kworker/u8:4+events_unbound09:40:202
2182942994339,3cyclictest1939814-21kworker/u8:3+events_unbound07:20:222
2182934994237,3cyclictest2208629-21kworker/u8:0+flush-259:008:10:000
2182942994138,2cyclictest2281545-21kworker/u8:4+flush-259:009:50:152
2182942994138,2cyclictest2276370-21kworker/u8:3+flush-259:009:05:362
2182942994138,2cyclictest2260508-21kworker/u8:2+events_unbound08:40:022
2182937994138,2cyclictest2379917-21kworker/u8:2+events_unbound10:35:221
2182937994138,2cyclictest2281545-21kworker/u8:4+flush-259:009:00:151
2182937994138,2cyclictest2239896-21kworker/u8:1+flush-259:008:15:231
2182942994037,2cyclictest2462668-21kworker/u8:1+events_unbound11:55:422
2182937994037,2cyclictest2379917-21kworker/u8:2+flush-259:010:30:341
2182937994037,2cyclictest2281545-21kworker/u8:4+events_unbound09:40:141
2182937994037,2cyclictest2213942-21kworker/u8:3+flush-259:008:30:001
2182937994036,3cyclictest2404790-21kworker/u8:1+flush-259:010:48:351
2182942993936,2cyclictest2472944-21kworker/u8:0+events_unbound12:05:242
2182942993936,2cyclictest2431695-21kworker/u8:4+flush-259:011:45:392
2182942993936,2cyclictest2431695-21kworker/u8:4+events_unbound12:10:252
2182942993936,2cyclictest2281545-21kworker/u8:4+events_unbound10:50:242
2182942993936,2cyclictest2177785-21kworker/u8:1+flush-259:007:40:352
2182942993935,3cyclictest1939814-21kworker/u8:3+flush-259:007:19:492
2182937993935,3cyclictest335557-21systemd-journal11:34:341
2182937993935,3cyclictest2348738-21kworker/u8:0+flush-259:010:10:221
2182937993931,6cyclictest2369293-21cp10:10:011
2182942993836,2cyclictest2260508-21kworker/u8:2+events_unbound08:33:352
2182942993835,2cyclictest2208629-21kworker/u8:0+flush-259:007:35:242
2182942993834,3cyclictest2462668-21kworker/u8:1+flush-259:012:25:452
2182942993834,3cyclictest2270819-21kworker/u8:1+flush-259:008:35:252
2182937993835,2cyclictest2177785-21kworker/u8:1+events_unbound07:30:211
2182937993834,3cyclictest2431695-21kworker/u8:4+flush-259:011:20:411
2182944993730,5cyclictest509-21rs:main3
2182942993734,2cyclictest2348738-21kworker/u8:0+flush-259:010:10:202
2182942993734,2cyclictest2213942-21kworker/u8:3+events_unbound07:50:202
2182942993733,3cyclictest2208629-21kworker/u8:0+events_unbound08:19:362
2182937993734,2cyclictest2472944-21kworker/u8:0+flush-259:012:05:151
2182937993734,2cyclictest2276370-21kworker/u8:3+events_unbound09:25:191
2182937993734,2cyclictest2260508-21kworker/u8:2+events_unbound08:30:231
2182934993732,4cyclictest2426369-21kworker/u8:3+events_unbound11:05:340
2182934993732,4cyclictest2426369-21kworker/u8:3+events_unbound11:05:330
2182942993633,2cyclictest2276370-21kworker/u8:3+flush-259:009:20:152
2182942993633,2cyclictest2276370-21kworker/u8:3+events_unbound09:25:232
2182942993633,2cyclictest2260508-21kworker/u8:2+events_unbound08:24:342
2182942993632,3cyclictest2276370-21kworker/u8:3+events_unbound08:57:342
2182942993632,2cyclictest2426369-21kworker/u8:3+events_unbound11:19:352
2182937993632,3cyclictest2198321-21kworker/u8:2+events_unbound07:36:361
2182937993632,3cyclictest1939814-21kworker/u8:3+events_unbound07:25:221
2182937993632,2cyclictest2281545-21kworker/u8:4+flush-259:009:20:201
2182937993631,3cyclictest2457298-21kworker/u8:0+flush-259:011:35:211
2182934993632,3cyclictest2281545-21kworker/u8:4+flush-259:010:19:590
2182934993632,3cyclictest2177785-21kworker/u8:1+events_unbound07:24:420
2182934993631,4cyclictest2431695-21kworker/u8:4+events_unbound12:00:480
2182944993528,4cyclictest2520236-21/usr/sbin/munin12:35:163
2182944993527,5cyclictest2400560-21latency_hist10:40:003
2182942993532,2cyclictest2426369-21kworker/u8:3+flush-259:011:30:412
2182942993532,2cyclictest2281545-21kworker/u8:4+flush-259:009:00:192
2182942993532,2cyclictest1939814-21kworker/u8:3+flush-259:007:25:192
2182942993531,3cyclictest2410908-21kworker/u8:2+events_unbound10:57:362
2182942993531,3cyclictest2379917-21kworker/u8:2+events_unbound10:28:362
2182942993531,3cyclictest2276370-21kworker/u8:3+events_unbound09:10:082
2182942993531,3cyclictest2213942-21kworker/u8:3+events_unbound08:13:352
2182942993530,4cyclictest335557-21systemd-journal12:20:002
2182937993532,2cyclictest2208629-21kworker/u8:0+flush-259:008:04:541
2182937993531,3cyclictest2472944-21kworker/u8:0+events_unbound11:58:361
2182937993531,3cyclictest2451808-21kworker/u8:1+events_unbound11:29:351
2182937993531,3cyclictest2281545-21kworker/u8:4+events_unbound10:00:221
2182937993531,3cyclictest1939814-21kworker/u8:3+events_unbound07:24:361
2182944993425,6cyclictest2431463-21latency_hist11:10:013
2182944993425,6cyclictest2431463-21latency_hist11:10:003
2182942993431,2cyclictest2457298-21kworker/u8:0+events_unbound11:35:232
2182937993431,2cyclictest2514625-21kworker/u8:2+events_unbound12:30:261
2182937993431,2cyclictest2281545-21kworker/u8:4+events_unbound08:52:341
2182937993430,3cyclictest2462668-21kworker/u8:1+events_unbound12:00:011
2182937993430,3cyclictest2281545-21kworker/u8:4+events_unbound09:15:151
2182937993430,3cyclictest2276370-21kworker/u8:3+events_unbound09:14:341
2182937993429,4cyclictest335557-21systemd-journal08:45:001
2182934993429,4cyclictest2410908-21kworker/u8:2+events_unbound11:12:350
2182934993429,4cyclictest2410908-21kworker/u8:2+events_unbound11:12:340
2182934993428,4cyclictest509-21rs:main0
2182944993327,4cyclictest2276370-21kworker/u8:3+flush-259:009:10:183
2182942993331,1cyclictest2322723-21kworker/u8:0+flush-259:009:35:252
2182942993329,3cyclictest2426369-21kworker/u8:3+events_unbound11:10:362
2182942993329,3cyclictest2426369-21kworker/u8:3+events_unbound11:10:352
2182937993329,3cyclictest2276370-21kworker/u8:3+events_unbound08:55:211
2182937993329,3cyclictest2270819-21kworker/u8:1+events_unbound08:36:341
2182937993329,3cyclictest2208629-21kworker/u8:0+flush-259:007:59:541
2182937993328,3cyclictest2281545-21kworker/u8:4+flush-259:009:55:201
2182944993228,3cyclictest1939814-21kworker/u8:3+flush-259:007:15:003
2182942993229,2cyclictest2410908-21kworker/u8:2+events_unbound11:04:342
2182942993229,2cyclictest2307294-21kworker/u8:1+flush-259:009:45:272
2182942993229,2cyclictest2281545-21kworker/u8:4+events_unbound10:45:132
2182942993229,2cyclictest2281545-21kworker/u8:4+events_unbound08:53:352
2182934993227,4cyclictest2431695-21kworker/u8:4+events_unbound12:14:350
2182944993126,4cyclictest335557-21systemd-journal11:13:333
2182944993126,4cyclictest335557-21systemd-journal11:13:333
2182942993128,2cyclictest2379917-21kworker/u8:2+events_unbound10:36:352
2182942993128,2cyclictest2213942-21kworker/u8:3+flush-259:008:00:202
2182942993128,2cyclictest2208629-21kworker/u8:0+flush-259:007:55:212
2182942993127,3cyclictest2281545-21kworker/u8:4+flush-259:009:15:162
2182937993128,2cyclictest2281545-21kworker/u8:4+events_unbound08:45:141
2182937993128,2cyclictest2213942-21kworker/u8:3+flush-259:007:45:211
2182934993127,3cyclictest2379917-21kworker/u8:2+events_unbound10:30:220
2182934993127,3cyclictest2281545-21kworker/u8:4+events_unbound10:45:230
2182934993123,6cyclictest335557-21systemd-journal09:50:000
2182944993025,3cyclictest2213942-21kworker/u8:3+flush-259:007:50:163
2182942993027,2cyclictest2431695-21kworker/u8:4+flush-259:011:20:252
2182942993025,4cyclictest2348738-21kworker/u8:0+events_unbound10:05:122
2182937993027,3cyclictest2213942-21kworker/u8:3+events_unbound08:10:161
2182937993027,2cyclictest2281545-21kworker/u8:4+flush-259:009:45:161
2182937993027,2cyclictest2255995-21apt-get08:20:121
2182937993026,3cyclictest2426369-21kworker/u8:3+events_unbound11:10:141
2182937993026,3cyclictest2426369-21kworker/u8:3+events_unbound11:10:131
2182937993026,3cyclictest1939814-21kworker/u8:3+events_unbound07:19:401
2182934993027,2cyclictest2239896-21kworker/u8:1+events_unbound08:14:540
2182934993026,3cyclictest2281545-21kworker/u8:4+events_unbound08:52:360
2182934993026,3cyclictest2177785-21kworker/u8:1+flush-259:007:15:350
2182944992924,4cyclictest335557-21systemd-journal10:47:343
2182942992926,2cyclictest2348738-21kworker/u8:0+events_unbound10:20:192
2182942992926,2cyclictest2177785-21kworker/u8:1+flush-259:007:10:262
2182942992924,4cyclictest2519690-21gpgv12:35:002
2182942992924,2cyclictest2281545-21kworker/u8:4+events_unbound09:30:202
2182934992925,3cyclictest2260508-21kworker/u8:2+events_unbound08:31:360
2182944992824,3cyclictest2276370-21kworker/u8:3+flush-259:009:20:003
2182937992825,2cyclictest2499162-21kworker/u8:3+flush-259:012:25:191
2182937992825,2cyclictest2431695-21kworker/u8:4+flush-259:011:45:141
2182937992824,3cyclictest2410908-21kworker/u8:2+events_unbound10:58:351
2182937992824,3cyclictest2348738-21kworker/u8:0+events_unbound10:15:151
2182934992824,3cyclictest2431695-21kworker/u8:4+flush-259:012:05:470
2182934992824,3cyclictest2276370-21kworker/u8:3+events_unbound09:05:240
2182934992824,3cyclictest2239896-21kworker/u8:1+events_unbound08:20:020
2182942992725,2cyclictest2431695-21kworker/u8:4+events_unbound11:40:142
2182942992725,2cyclictest2213942-21kworker/u8:3+events_unbound07:45:142
2182937992723,3cyclictest2213942-21kworker/u8:3+events_unbound07:50:121
2182937992723,2cyclictest2281545-21kworker/u8:4+events_unbound09:30:231
2182934992724,2cyclictest2499162-21kworker/u8:3+events_unbound12:25:250
2182934992723,3cyclictest2348738-21kworker/u8:0+events_unbound10:13:350
2182934992723,3cyclictest2281545-21kworker/u8:4+flush-259:009:55:210
2182944992618,5cyclictest2333045-21latency_hist09:35:013
2182942992623,2cyclictest2281545-21kworker/u8:4+flush-259:010:00:202
2182942992622,3cyclictest2208629-21kworker/u8:0+flush-259:008:05:142
2182942992621,4cyclictest335557-21systemd-journal12:40:002
2182937992623,2cyclictest2462668-21kworker/u8:1+flush-259:012:20:241
2182937992623,2cyclictest2416256-21kworker/u8:0+flush-259:011:00:201
2182937992623,2cyclictest2379917-21kworker/u8:2+flush-259:010:25:151
2182934992622,3cyclictest2213942-21kworker/u8:3+events_unbound08:29:590
2182944992519,4cyclictest2281545-21kworker/u8:4+flush-259:010:50:173
2182944992518,5cyclictest335557-21systemd-journal11:04:593
2182934992522,2cyclictest2514625-21kworker/u8:2+flush-259:012:35:250
2182934992522,2cyclictest2431695-21kworker/u8:4+events_unbound11:45:250
2182934992521,3cyclictest2431695-21kworker/u8:4+events_unbound11:20:250
2182934992519,5cyclictest2503282-21ntpq12:15:230
2182944992416,5cyclictest2260865-21latency_hist08:25:003
2182942992421,2cyclictest2281545-21kworker/u8:4+events_unbound08:45:242
2182942992420,2cyclictest2260508-21kworker/u8:2+events_unbound08:25:222
2182942992418,3cyclictest2426369-21kworker/u8:3+events_unbound11:05:262
2182942992418,3cyclictest2426369-21kworker/u8:3+events_unbound11:05:252
2182937992421,2cyclictest2431695-21kworker/u8:4+flush-259:011:40:201
2182937992421,2cyclictest2177785-21kworker/u8:1+events_unbound07:10:171
2182934992420,3cyclictest2281545-21kworker/u8:4+events_unbound08:47:350
2182944992319,3cyclictest2369179-21idleruntime-cro10:10:003
2182944992318,3cyclictest1-21systemd08:55:003
2182937992320,2cyclictest2276370-21kworker/u8:3+events_unbound09:05:221
2182934992320,2cyclictest2281545-21kworker/u8:4+events_unbound09:50:230
2182934992320,2cyclictest2213942-21kworker/u8:3+events_unbound08:03:350
2182934992319,3cyclictest2426369-21kworker/u8:3+events_unbound11:25:160
2182934992319,3cyclictest2348738-21kworker/u8:0+events_unbound10:05:180
2182934992319,3cyclictest2177785-21kworker/u8:1+events_unbound07:28:350
2182934992318,4cyclictest2281545-21kworker/u8:4+events_unbound10:01:340
2182944992218,3cyclictest2276370-21kworker/u8:3+flush-259:008:55:163
2182944992218,3cyclictest2276370-21kworker/u8:3+flush-259:008:50:003
2182944992216,4cyclictest2380134-21/usr/sbin/munin10:20:123
2182944992214,5cyclictest2514391-21latency_hist12:30:013
2182937992217,2cyclictest2514625-21kworker/u8:2+events_unbound12:35:261
2182934992219,2cyclictest2177785-21kworker/u8:1+events_unbound07:49:360
2182934992218,3cyclictest2431695-21kworker/u8:4+events_unbound11:35:020
2182934992218,3cyclictest2426369-21kworker/u8:3+events_unbound11:15:380
2182934992218,3cyclictest2307294-21kworker/u8:1+events_unbound09:37:350
2182934992218,3cyclictest2281545-21kworker/u8:4+flush-259:008:55:340
2182934992218,3cyclictest2281545-21kworker/u8:4+events_unbound10:42:360
2182944992115,4cyclictest2213227-21ntpq07:35:223
2182944992114,5cyclictest2245033-21latency_hist08:10:003
2182937992116,4cyclictest2281545-21kworker/u8:4+flush-259:009:50:241
2182934992118,2cyclictest2379917-21kworker/u8:2+events_unbound10:35:250
2182934992118,2cyclictest2281545-21kworker/u8:4+events_unbound09:00:220
2182934992117,3cyclictest2426369-21kworker/u8:3+events_unbound11:30:230
2182934992117,3cyclictest2281545-21kworker/u8:4+flush-259:009:30:100
2182934992117,3cyclictest2281545-21kworker/u8:4+events_unbound10:50:350
2182944992015,4cyclictest2431695-21kworker/u8:4+flush-259:011:45:253
218294499201,17cyclictest1-21systemd07:30:353
2182937992017,2cyclictest2348738-21kworker/u8:0+events_unbound10:21:361
2182934992016,3cyclictest2270819-21kworker/u8:1+events_unbound08:40:180
2182934992014,4cyclictest509-21rs:main0
2182934992014,4cyclictest2219287-21latency_hist07:45:010
2182944991913,4cyclictest2385298-21/usr/sbin/munin10:25:213
2182942991916,2cyclictest2348738-21kworker/u8:0+events_unbound10:15:192
2182934991916,2cyclictest2276370-21kworker/u8:3+events_unbound09:25:090
2182934991916,2cyclictest2208629-21kworker/u8:0+events_unbound08:18:360
2182934991916,2cyclictest2177785-21kworker/u8:1+flush-259:007:30:200
2182934991915,3cyclictest2416256-21kworker/u8:0+events_unbound10:55:160
2182934991915,3cyclictest2281545-21kworker/u8:4+flush-259:009:15:240
2182934991914,4cyclictest2462668-21kworker/u8:1+events_unbound12:21:360
2182944991815,2cyclictest0-21swapper/311:35:003
2182944991814,2cyclictest0-21swapper/310:40:163
2182944991813,4cyclictest2307294-21kworker/u8:1+flush-259:009:25:013
2182944991810,5cyclictest2375157-21diskstats10:15:153
2182937991816,2cyclictest2426369-21kworker/u8:3+events_unbound11:05:281
2182937991816,2cyclictest2426369-21kworker/u8:3+events_unbound11:05:271
2182937991814,3cyclictest2208629-21kworker/u8:0+events_unbound08:06:221
2182937991813,4cyclictest2400827-21apt-get10:40:121
2182934991815,2cyclictest2208629-21kworker/u8:0+events_unbound07:35:170
2182934991814,3cyclictest2472944-21kworker/u8:0+events_unbound11:50:190
2182944991712,3cyclictest2426369-21kworker/u8:3+flush-259:011:15:253
2182944991711,4cyclictest2229813-21/usr/sbin/munin07:55:233
2182944991711,4cyclictest1939814-21kworker/u8:3+flush-259:007:15:173
2182942991715,2cyclictest2379917-21kworker/u8:2+events_unbound10:30:202
2182934991713,3cyclictest2208629-21kworker/u8:0+flush-259:007:55:190
2182944991613,2cyclictest2343373-21systemd-run09:45:003
2182944991613,2cyclictest0-21swapper/311:40:003
2182944991613,2cyclictest0-21swapper/308:40:003
2182944991612,3cyclictest2343617-21apt-get09:45:123
2182937991613,2cyclictest2476861-21needreboot11:50:191
2182934991614,1cyclictest60350irq/132-enp0s29f1:rx-009:10:280
2182944991512,2cyclictest0-21swapper/311:00:013
2182944991512,2cyclictest0-21swapper/307:30:013
2182944991511,3cyclictest335557-21systemd-journal09:05:003
2182944991510,4cyclictest2472944-21kworker/u8:0+flush-259:012:00:253
2182934991514,1cyclictest60350irq/132-enp0s29f1:rx-007:10:020
2182934991513,1cyclictest60350irq/132-enp0s29f1:rx-009:20:210
2182934991513,1cyclictest60350irq/132-enp0s29f1:rx-007:52:440
2182934991512,2cyclictest60350irq/132-enp0s29f1:rx-008:36:490
2182934991512,2cyclictest2410908-21kworker/u8:2+events_unbound11:00:220
2182934991511,3cyclictest2514625-21kworker/u8:2+events_unbound12:30:200
2182934991511,3cyclictest2348738-21kworker/u8:0+events_unbound10:20:170
218294499149,4cyclictest2498497-21apt-get12:15:013
218294499148,4cyclictest335557-21systemd-journal09:40:003
2182944991412,1cyclictest2364012-21munin-plugin-st10:05:013
2182944991411,2cyclictest0-21swapper/309:55:003
2182944991411,2cyclictest0-21swapper/308:35:003
2182944991410,3cyclictest2369941-21cpuspeed_turbos10:10:123
2182944991410,2cyclictest0-21swapper/308:25:173
218294499136,5cyclictest335557-21systemd-journal10:34:353
218294499136,4cyclictest2255726-21latency_hist08:20:003
218294499136,4cyclictest2213942-21kworker/u8:3+flush-259:008:10:173
2182944991310,2cyclictest2353977-21cpuspeed_turbos09:55:123
218294499128,2cyclictest0-21swapper/312:35:003
218294499125,4cyclictest335557-21systemd-journal07:41:343
218294499120,10cyclictest2224411-21unin-run07:50:003
218294499118,2cyclictest2478559-21sed11:55:183
218294499115,4cyclictest2499189-21apt-get12:15:123
218294499115,4cyclictest2302362-21/usr/sbin/munin09:05:153
218294499111,2cyclictest2446958-21latency_hist11:25:003
218294499108,1cyclictest3622550-21kworker/3:3+events_power_efficient11:40:193
21829449994,3cyclictest2504502-21grep12:20:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional