You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 02:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot4s.osadl.org (updated Mon Jul 14, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2496116994542,2cyclictest2604984-21kworker/u8:1+events_unbound09:00:182
2496110994133,4cyclictest2734392-21kworker/u8:2+flush-259:011:50:180
2496110994133,4cyclictest2734392-21kworker/u8:2+flush-259:011:50:180
2496116994037,2cyclictest2522232-21kworker/u8:2+flush-259:008:20:162
2496116994036,3cyclictest2661611-21kworker/u8:1+flush-259:010:25:012
2496112993936,2cyclictest2734392-21kworker/u8:2+events_unbound11:10:301
2496110993932,5cyclictest3754036-21systemd-journal11:00:010
2496116993835,2cyclictest2645298-21kworker/u8:1+events_unbound09:30:282
2496116993835,2cyclictest2496078-21kworker/u8:3+flush-259:007:25:182
2496110993834,3cyclictest2496078-21kworker/u8:3+flush-259:007:30:150
2496116993734,2cyclictest2812543-21kworker/u8:0+events_unbound12:20:162
2496116993734,2cyclictest2661611-21kworker/u8:1+events_unbound10:30:212
2496116993734,2cyclictest2532511-21kworker/u8:0+events_unbound07:45:192
2496116993733,2cyclictest2496078-21kworker/u8:3+events_unbound07:30:162
2496112993631,3cyclictest2807648-21/usr/sbin/munin12:10:181
2496116993532,2cyclictest2501474-21kworker/u8:1+flush-259:007:35:172
2496120993431,2cyclictest2661611-21kworker/u8:1+flush-259:010:50:013
2496116993431,2cyclictest2766129-21kworker/u8:0+events_unbound11:30:142
2496116993431,2cyclictest2532511-21kworker/u8:0+events_unbound08:55:172
2496116993430,2cyclictest2522232-21kworker/u8:2+events_unbound08:25:162
2496116993428,2cyclictest2522232-21kworker/u8:2+events_unbound07:50:162
2496112993430,3cyclictest2645298-21kworker/u8:1+events_unbound09:41:361
2496110993427,5cyclictest2609201-21ntpq08:55:220
2496116993331,2cyclictest2522232-21kworker/u8:2+events_unbound08:15:162
2496116993330,2cyclictest2501474-21kworker/u8:1+flush-259:007:40:262
2496116993329,3cyclictest2604984-21kworker/u8:1+flush-259:009:05:152
2496116993229,2cyclictest2812543-21kworker/u8:0+flush-259:012:35:252
2496116993229,2cyclictest2734392-21kworker/u8:2+events_unbound11:40:142
2496116993229,2cyclictest2734392-21kworker/u8:2+events_unbound11:00:252
2496116993229,2cyclictest2708242-21kworker/u8:4+flush-259:010:50:252
2496110993227,3cyclictest2532511-21kworker/u8:0+flush-259:008:45:190
2496116993128,2cyclictest2781616-21kworker/u8:0+flush-259:011:45:262
2496116993128,2cyclictest2771524-21kworker/u8:4+events_unbound11:35:232
2496116993127,3cyclictest2573660-21kworker/u8:1+flush-259:008:30:272
2496116993127,3cyclictest2532511-21kworker/u8:0+flush-259:008:45:282
2496116993125,3cyclictest2802473-21kworker/u8:3+events_unbound12:10:222
2496116993123,2cyclictest2734392-21kworker/u8:2+events_unbound11:05:182
2496112993129,2cyclictest2812543-21kworker/u8:0+events_unbound12:19:381
2496112993128,2cyclictest2812543-21kworker/u8:0+events_unbound12:20:291
2496110993125,4cyclictest3754036-21systemd-journal11:15:000
2496116993027,2cyclictest2661611-21kworker/u8:1+events_unbound09:50:162
2496112993027,2cyclictest2496078-21kworker/u8:3+events_unbound07:21:371
2496110993023,5cyclictest2703083-21latency_hist10:30:010
2496120992922,5cyclictest2600098-21apt-get08:50:143
2496116992926,2cyclictest2661611-21kworker/u8:1+events_unbound10:00:142
2496116992926,2cyclictest2522232-21kworker/u8:2+flush-259:008:05:212
2496116992925,3cyclictest2656176-21kworker/u8:3+flush-259:009:55:242
2496116992925,3cyclictest2501474-21kworker/u8:1+flush-259:007:20:272
2496116992924,2cyclictest2661611-21kworker/u8:1+events_unbound10:45:262
2496112992927,2cyclictest2812543-21kworker/u8:0+events_unbound12:31:061
2496112992926,2cyclictest2734392-21kworker/u8:2+events_unbound12:04:371
2496112992926,2cyclictest2687739-21kworker/u8:0+events_unbound10:20:371
2496112992926,2cyclictest2604984-21kworker/u8:1+events_unbound09:00:281
2496110992922,5cyclictest2776456-21latency_hist11:40:010
2496116992824,3cyclictest2802473-21kworker/u8:3+flush-259:012:05:172
2496116992824,3cyclictest2734392-21kworker/u8:2+events_unbound11:20:162
2496116992824,3cyclictest2645298-21kworker/u8:1+events_unbound09:40:142
2496112992826,2cyclictest2687739-21kworker/u8:0+events_unbound10:32:381
2496112992825,2cyclictest2661611-21kworker/u8:1+events_unbound11:05:531
2496110992824,3cyclictest2568793-21apt-get08:20:110
2496116992724,2cyclictest2771524-21kworker/u8:4+flush-259:012:00:212
2496116992724,2cyclictest2734392-21kworker/u8:2+events_unbound11:15:202
2496116992724,2cyclictest2656176-21kworker/u8:3+flush-259:010:10:162
2496116992724,2cyclictest2532511-21kworker/u8:0+flush-259:009:45:112
2496112992725,2cyclictest2501474-21kworker/u8:1+events_unbound07:40:191
2496110992720,5cyclictest2796180-21ntpq11:55:210
2496116992622,3cyclictest2708242-21kworker/u8:4+events_unbound11:25:192
2496112992623,2cyclictest2687739-21kworker/u8:0+events_unbound10:40:371
2496112992623,2cyclictest2661611-21kworker/u8:1+events_unbound09:55:271
2496112992623,2cyclictest2656176-21kworker/u8:3+events_unbound10:05:291
2496112992621,2cyclictest2532511-21kworker/u8:0+events_unbound08:42:371
2496116992522,2cyclictest2661611-21kworker/u8:1+flush-259:010:55:162
2496116992522,2cyclictest2522232-21kworker/u8:2+events_unbound07:55:142
2496112992523,2cyclictest2812543-21kworker/u8:0+events_unbound12:35:281
2496112992523,2cyclictest2573660-21kworker/u8:1+events_unbound08:27:371
2496112992522,2cyclictest2645298-21kworker/u8:1+events_unbound09:39:571
2496112992522,2cyclictest2645298-21kworker/u8:1+events_unbound09:39:571
2496110992520,3cyclictest2522232-21kworker/u8:2+flush-259:008:00:160
2496120992419,3cyclictest2548167-21apt-get08:00:123
2496120992418,5cyclictest2620364-21latency_hist09:10:013
2496116992421,2cyclictest2645298-21kworker/u8:1+events_unbound09:35:232
2496116992421,2cyclictest2645298-21kworker/u8:1+events_unbound09:35:222
2496116992420,3cyclictest2620591-21kworker/u8:2+events_unbound09:15:172
2496112992421,2cyclictest2802473-21kworker/u8:3+events_unbound12:25:371
2496110992421,2cyclictest2501474-21kworker/u8:1+flush-259:008:10:000
2496120992319,2cyclictest2516642-21apt-get07:30:023
2496120992317,4cyclictest2765203-21ntpq11:25:233
2496116992320,2cyclictest2462436-21kworker/u8:2+flush-259:007:10:132
2496116992318,2cyclictest2522232-21kworker/u8:2+events_unbound08:40:292
2496112992321,2cyclictest2532511-21kworker/u8:0+events_unbound08:55:211
2496112992321,1cyclictest2604984-21kworker/u8:1+events_unbound09:10:211
2496112992320,2cyclictest2802473-21kworker/u8:3+events_unbound12:07:361
2496110992319,3cyclictest2532511-21kworker/u8:0+events_unbound09:20:250
2496120992219,2cyclictest2604984-21kworker/u8:1+flush-259:009:14:593
2496120992216,4cyclictest2496312-21/usr/sbin/munin07:10:183
2496116992219,2cyclictest2522232-21kworker/u8:2+flush-259:008:00:262
2496116992219,2cyclictest2522232-21kworker/u8:2+events_unbound08:50:142
2496110992216,4cyclictest2610534-21diskstats09:00:150
2496120992118,2cyclictest3002-21cron08:20:003
2496116992118,2cyclictest3754036-21systemd-journal09:20:132
2496116992117,3cyclictest2791943-21latency_hist11:55:012
2496116992117,3cyclictest2791943-21latency_hist11:55:012
2496116992117,3cyclictest2661611-21kworker/u8:1+events_unbound10:25:152
2496112992119,2cyclictest2522232-21kworker/u8:2+events_unbound07:56:371
2496112992118,2cyclictest2734392-21kworker/u8:2+events_unbound11:20:301
2496112992118,2cyclictest2687739-21kworker/u8:0+events_unbound10:25:401
2496112992118,2cyclictest2661611-21kworker/u8:1+events_unbound09:50:381
2496112992118,2cyclictest2656176-21kworker/u8:3+events_unbound10:02:371
2496112992118,2cyclictest2496078-21kworker/u8:3+events_unbound07:29:381
2496112992117,3cyclictest2501481-21/usr/sbin/munin07:15:261
2496112992116,4cyclictest2766117-21latency_hist11:30:011
2496112992116,4cyclictest2527333-21latency_hist07:40:011
2496120992016,3cyclictest2771524-21kworker/u8:4+flush-259:011:59:593
2496120992016,3cyclictest2496078-21kworker/u8:3+flush-259:007:25:213
2496120992015,3cyclictest2532511-21kworker/u8:0+flush-259:008:10:203
2496120992014,4cyclictest2822874-21latency_hist12:25:013
2496120992013,5cyclictest2511482-21latency_hist07:25:003
2496116992016,3cyclictest2708242-21kworker/u8:4+events_unbound10:42:382
2496112992017,2cyclictest2708242-21kworker/u8:4+events_unbound10:38:271
2496112992017,2cyclictest2563354-21kworker/u8:3+events_unbound08:22:371
2496112992017,2cyclictest2532511-21kworker/u8:0+events_unbound09:48:111
2496112992017,2cyclictest2532511-21kworker/u8:0+events_unbound08:45:371
2496112992017,2cyclictest2522232-21kworker/u8:2+events_unbound07:54:571
2496112992017,2cyclictest2501474-21kworker/u8:1+events_unbound08:10:291
2496112992017,2cyclictest2501474-21kworker/u8:1+events_unbound07:32:371
2496112992013,5cyclictest2686803-21ntpq10:10:211
2496110992016,3cyclictest2532511-21kworker/u8:0+flush-259:009:35:170
2496110992016,3cyclictest2532511-21kworker/u8:0+flush-259:009:35:170
2496110992015,4cyclictest2812543-21kworker/u8:0+flush-259:012:15:210
2496110992015,3cyclictest2682560-21cron10:10:000
2496110992013,6cyclictest2707359-21ntpq10:30:270
2496120991915,3cyclictest3000061-21in:imuxsock07:20:003
2496120991914,4cyclictest2573660-21kworker/u8:1+flush-259:008:35:003
2496120991914,4cyclictest2501474-21kworker/u8:1+flush-259:007:35:313
2496116991916,2cyclictest2812543-21kworker/u8:0+flush-259:012:30:262
2496116991916,2cyclictest2802473-21kworker/u8:3+events_unbound12:25:232
2496116991916,2cyclictest2532511-21kworker/u8:0+flush-259:008:10:232
2496112991916,2cyclictest2719163-21kworker/u8:2+events_unbound10:48:431
2496112991916,2cyclictest2661611-21kworker/u8:1+events_unbound11:19:551
2496112991916,2cyclictest2568519-21idleruntime-cro08:20:011
2496112991912,5cyclictest2641397-21diskstats09:30:171
2496110991914,4cyclictest2734392-21kworker/u8:2+events_unbound12:00:170
2496110991913,4cyclictest2713604-21/usr/sbin/munin10:40:250
2496120991814,3cyclictest2594292-21latency_hist08:45:013
2496120991814,2cyclictest2683267-21sed10:10:183
2496120991814,2cyclictest0-21swapper/309:05:003
2496120991812,4cyclictest2734344-21apt-get11:00:023
2496120991812,4cyclictest2584239-21apt-get08:35:163
249612099181,15cyclictest0-21swapper/311:15:003
2496116991815,2cyclictest2734392-21kworker/u8:2+events_unbound11:55:132
2496112991816,2cyclictest2771524-21kworker/u8:4+events_unbound11:47:371
2496112991816,2cyclictest2620591-21kworker/u8:2+events_unbound09:25:341
2496112991815,3cyclictest2734392-21kworker/u8:2+events_unbound11:50:121
2496112991815,3cyclictest2734392-21kworker/u8:2+events_unbound11:50:111
2496112991815,2cyclictest2708242-21kworker/u8:4+events_unbound10:51:361
2496112991815,2cyclictest2522232-21kworker/u8:2+events_unbound08:50:281
2496112991815,2cyclictest2340821-21kworker/u8:0+events_unbound07:10:251
2496112991814,3cyclictest2656176-21kworker/u8:3+events_unbound10:18:431
2496110991814,3cyclictest2573660-21kworker/u8:1+flush-259:008:35:270
2496110991814,3cyclictest2462436-21kworker/u8:2+flush-259:007:10:150
2496110991813,4cyclictest2802473-21kworker/u8:3+flush-259:012:30:190
2496110991813,3cyclictest2687739-21kworker/u8:0+flush-259:010:40:000
2496120991714,2cyclictest2547890-21cron08:00:013
2496120991711,4cyclictest2677865-21diskstats10:05:163
2496120991710,5cyclictest2828459-21diskstats12:30:173
2496120991710,5cyclictest2568953-21diskstats08:20:143
2496120991710,5cyclictest2557276-21ntpq08:05:223
2496112991715,2cyclictest2771524-21kworker/u8:4+events_unbound11:35:581
2496110991714,2cyclictest835-21runrttasks11:10:000
2496110991714,2cyclictest60650irq/133-enp0s29f1:rx-012:36:460
2496110991714,2cyclictest60650irq/133-enp0s29f1:rx-011:34:440
2496110991714,2cyclictest60650irq/133-enp0s29f1:rx-008:40:290
2496110991711,4cyclictest3754036-21systemd-journal08:10:010
2496110991711,4cyclictest2577891-21ntpq08:25:210
2496110991710,5cyclictest2619475-21ntpq09:05:230
249612099169,5cyclictest2667117-21base6409:55:023
2496120991612,3cyclictest2755775-21gpgv11:20:013
2496120991610,4cyclictest3754036-21systemd-journal10:40:253
2496116991614,2cyclictest2522232-21kworker/u8:2+events_unbound08:39:562
2496116991613,2cyclictest2661611-21kworker/u8:1+flush-259:010:05:212
2496112991612,3cyclictest2766130-21kworker/u8:3+events_unbound11:33:191
2496112991611,1cyclictest191rcu_preempt11:40:161
249611099169,5cyclictest2506622-21/usr/sbin/munin07:20:260
2496110991614,1cyclictest60650irq/133-enp0s29f1:rx-011:45:170
2496110991614,1cyclictest60650irq/133-enp0s29f1:rx-007:55:250
2496110991613,2cyclictest60650irq/133-enp0s29f1:rx-012:08:380
2496110991613,2cyclictest60650irq/133-enp0s29f1:rx-011:18:320
2496110991613,2cyclictest60650irq/133-enp0s29f1:rx-009:54:200
2496110991613,2cyclictest60650irq/133-enp0s29f1:rx-009:41:560
2496110991613,2cyclictest60650irq/133-enp0s29f1:rx-008:54:220
2496110991610,4cyclictest2656712-21apt-get09:45:120
2496120991511,3cyclictest2661611-21kworker/u8:1+flush-259:011:35:373
2496120991511,3cyclictest1-21systemd10:25:013
2496116991513,2cyclictest2687739-21kworker/u8:0+events_unbound10:39:572
2496116991512,2cyclictest2501474-21kworker/u8:1+events_unbound07:17:002
2496116991511,3cyclictest2661611-21kworker/u8:1+events_unbound11:10:582
249611299159,1cyclictest191rcu_preempt11:05:011
249611299157,1cyclictest0-21swapper/109:05:151
249611299152,3cyclictest32-21ksoftirqd/108:05:261
2496112991513,1cyclictest2620591-21kworker/u8:2-events_unbound09:15:221
2496112991513,1cyclictest0-21swapper/111:00:011
2496112991512,2cyclictest2573660-21kworker/u8:1+events_unbound08:36:241
2496112991512,2cyclictest2573660-21kworker/u8:1+events_unbound08:32:001
2496112991512,2cyclictest2535040-21kthreadcore07:45:211
2496112991512,2cyclictest2532511-21kworker/u8:0+events_unbound09:20:281
249611099159,4cyclictest2667402-21/usr/sbin/munin09:55:200
2496110991514,1cyclictest60650irq/133-enp0s29f1:rx-011:28:110
2496110991513,1cyclictest60650irq/133-enp0s29f1:rx-011:40:150
2496110991513,1cyclictest60650irq/133-enp0s29f1:rx-011:20:240
2496110991513,1cyclictest60650irq/133-enp0s29f1:rx-007:44:290
2496110991513,1cyclictest0-21swapper/010:25:000
2496110991512,2cyclictest60650irq/133-enp0s29f1:rx-012:20:090
2496110991512,2cyclictest60650irq/133-enp0s29f1:rx-011:00:310
2496110991512,2cyclictest60650irq/133-enp0s29f1:rx-010:19:460
2496110991512,2cyclictest60650irq/133-enp0s29f1:rx-009:16:450
2496110991512,2cyclictest60650irq/133-enp0s29f1:rx-008:15:530
2496110991512,2cyclictest60650irq/133-enp0s29f1:rx-007:28:350
2496110991511,3cyclictest2496078-21kworker/u8:3+events_unbound07:15:190
2496110991510,3cyclictest2802473-21kworker/u8:3+flush-259:012:25:190
249612099147,5cyclictest3754036-21systemd-journal11:09:593
2496120991410,3cyclictest2802227-21cron12:05:003
249611299147,1cyclictest191rcu_preempt11:55:121
249611299143,3cyclictest32-21ksoftirqd/107:10:001
249611099149,3cyclictest2532738-21apt-get07:45:150
2496110991412,2cyclictest60650irq/133-enp0s29f1:rx-007:51:470
2496110991411,2cyclictest60650irq/133-enp0s29f1:rx-009:31:120
2496110991411,2cyclictest60650irq/133-enp0s29f1:rx-008:33:370
2496110991411,2cyclictest2636519-21grep09:25:200
2496110991410,3cyclictest60650irq/133-enp0s29f1:rx-010:45:120
249612099139,3cyclictest0-21swapper/311:00:003
2496120991311,1cyclictest0-21swapper/308:59:593
2496120991310,2cyclictest2802473-21kworker/u8:3+flush-259:012:15:003
2496120991310,2cyclictest2712938-21unixbench_singl10:35:293
2496116991310,2cyclictest2661611-21kworker/u8:1+flush-259:010:15:412
2496116991310,2cyclictest2636054-21kworker/u8:3+events_unbound09:28:172
2496116991310,2cyclictest2604984-21kworker/u8:1+flush-259:009:10:092
249611099138,3cyclictest2729215-21awk10:55:010
2496110991311,2cyclictest60650irq/133-enp0s29f1:rx-010:14:030
2496110991311,1cyclictest60650irq/133-enp0s29f1:rx-010:00:150
2496110991311,1cyclictest60650irq/133-enp0s29f1:rx-007:35:230
249612099129,2cyclictest2537624-21apt07:50:003
249612099127,3cyclictest2676544-21ntpq10:00:213
249612099125,5cyclictest3754036-21systemd-journal11:55:003
249612099125,5cyclictest3754036-21systemd-journal11:55:003
2496120991210,1cyclictest0-21swapper/312:35:183
249611699128,3cyclictest2812543-21kworker/u8:0+events_unbound12:18:462
249611099129,2cyclictest60650irq/133-enp0s29f1:rx-012:11:530
249611099129,2cyclictest2621050-21sed09:10:170
249612099118,2cyclictest2782120-21fschecks_time11:45:153
249612099118,2cyclictest0-21swapper/308:49:593
249612099118,1cyclictest0-21swapper/309:24:593
249612099117,3cyclictest1-21systemd07:45:013
249612099116,4cyclictest2802473-21kworker/u8:3+flush-259:012:29:073
249612099106,3cyclictest2656176-21kworker/u8:3+flush-259:010:15:233
24961209997,1cyclictest0-21swapper/311:30:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional