You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 07:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Fri Mar 13, 2026 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2971421994532,11cyclictest2980811-21ntpq19:15:200
2971421993927,10cyclictest3147408-21apt-get22:00:110
2971429993834,3cyclictest3147193-21kworker/u8:0+flush-259:023:00:002
2971421993732,3cyclictest3168395-21apt-get22:20:110
2971429993529,4cyclictest252-21systemd-journal23:15:002
2971421993531,3cyclictest3220398-21apt-get23:10:100
2971429993430,3cyclictest3033022-21kworker/u8:1+flush-259:020:35:192
2971436993329,3cyclictest3183857-21/usr/sbin/munin22:35:213
2971429993226,4cyclictest3059142-21latency_hist20:35:012
2971421993228,3cyclictest2981981-21apt-get19:20:100
2971423993124,5cyclictest252-21systemd-journal00:10:001
2971421993125,4cyclictest3292734-21latency_hist00:20:010
2971423993014,3cyclictest3064296-21idleruntime-cro20:40:001
2971436992923,4cyclictest3303086-21latency_hist00:30:013
2971423992822,5cyclictest3001502-21ntpq19:35:201
2971423992816,4cyclictest3048847-21latency_hist20:25:011
2971436992720,5cyclictest252-21systemd-journal20:54:593
2971423992715,2cyclictest0-21swapper/122:25:011
2971429992622,3cyclictest2997437-21kworker/u8:4+flush-259:020:20:182
2971429992620,4cyclictest3033326-21latency_hist20:10:002
2971423992620,4cyclictest3199769-21/usr/sbin/munin22:50:141
2971421992622,3cyclictest3173340-21kworker/u8:1+flush-259:022:53:210
2971423992519,4cyclictest3126945-21diskstats21:40:141
2971421992518,5cyclictest3187921-21ntpq22:35:200
2971436992420,3cyclictest3246168-21kworker/u8:3+events_unbound23:35:163
2971429992421,2cyclictest3246168-21kworker/u8:3+flush-259:000:20:122
2971423992421,2cyclictest2986928-21kworker/u8:2+flush-259:020:02:261
297143699230,21cyclictest500-21systemd-logind21:00:003
2971429992320,2cyclictest3230186-21kworker/u8:1+events_unbound23:30:172
2971423992318,4cyclictest252-21systemd-journal22:10:211
2971423992318,3cyclictest3215236-21apt-get23:05:111
2971421992311,11cyclictest3126512-21idleruntime-cro21:40:000
2971429992218,3cyclictest3251515-21kworker/u8:0+flush-259:023:55:182
2971429992218,3cyclictest3043916-21kworker/u8:0+flush-259:020:29:592
2971421992218,3cyclictest3178718-21/usr/sbin/munin22:30:140
2971429992118,2cyclictest3251515-21kworker/u8:0+flush-259:000:05:262
2971429992115,4cyclictest3126563-21kworker/u8:1+events_unbound21:50:172
2971423992117,3cyclictest3136436-21kworker/u8:2+flush-259:021:55:001
2971423992116,3cyclictest3298139-21/usr/sbin/munin00:25:181
2971423992115,4cyclictest3292993-21apt-get00:20:101
2971421992117,3cyclictest2997437-21kworker/u8:4+flush-259:021:10:170
297142199210,19cyclictest3090430-21turbostat21:05:000
2971436992017,2cyclictest3246168-21kworker/u8:3+flush-259:000:25:003
2971436992014,4cyclictest3193969-21latency_hist22:45:003
2971429992016,3cyclictest3075158-21kworker/u8:0+flush-259:021:00:122
2971429992016,2cyclictest2986928-21kworker/u8:2+events_unbound20:10:162
2971429992015,3cyclictest509-21rs:main2
2971423992016,3cyclictest3116455-21/usr/sbin/munin21:30:141
2971423992016,3cyclictest2997437-21kworker/u8:4+flush-259:021:35:151
2971423992015,4cyclictest3084867-21latency_hist21:00:011
297142199209,10cyclictest0-21swapper/021:55:180
297142199205,14cyclictest252-21systemd-journal22:30:000
2971421992014,4cyclictest3167245-21ntpq22:15:190
297142199201,2cyclictest3063453-21ntpq20:35:230
2971429991916,2cyclictest3173340-21kworker/u8:1+flush-259:023:00:252
2971429991915,2cyclictest3084860-21apt-get21:00:002
2971429991914,4cyclictest3012978-21apt-get19:50:112
2971423991916,2cyclictest3267614-21irqcore23:55:161
2971423991916,2cyclictest3189066-21apt-get22:40:111
2971423991916,2cyclictest1-21systemd00:05:001
2971423991913,4cyclictest507-21in:imuxsock19:50:001
297142199199,9cyclictest0-21swapper/023:20:150
297142199195,13cyclictest3297032-21ntpq00:20:200
297142199194,13cyclictest3065008-21sed20:40:150
297142199194,13cyclictest3065008-21sed20:40:150
297142199193,3cyclictest490-21dbus-daemon19:15:010
2971421991914,4cyclictest60350irq/132-enp0s29f1:rx-021:48:250
2971421991914,3cyclictest3090688-21apt-get21:05:120
297142199191,2cyclictest3245419-21ntpq23:30:210
297142199191,1cyclictest0-21swapper/020:20:180
297142199190,5cyclictest3018422-21tune2fs19:55:150
297142199190,2cyclictest3028205-21latency_hist20:05:000
297143699186,7cyclictest3106483-21hddtemp_smartct21:20:163
2971436991813,4cyclictest3199794-21apt-get22:50:113
2971429991815,2cyclictest2997437-21kworker/u8:4+flush-259:022:05:222
2971429991815,2cyclictest2997437-21kworker/u8:4+flush-259:021:40:262
2971429991814,3cyclictest2997437-21kworker/u8:4+events_unbound20:40:122
2971429991814,3cyclictest2997437-21kworker/u8:4+events_unbound20:40:112
2971429991814,3cyclictest2992274-21kworker/u8:1+flush-259:019:30:242
2971429991814,3cyclictest2986928-21kworker/u8:2+flush-259:020:15:252
2971429991813,3cyclictest3147193-21kworker/u8:0+flush-259:022:50:182
2971423991815,2cyclictest3051398-21kthreadcore20:25:181
2971423991814,3cyclictest3272101-21kworker/u8:4+flush-259:000:20:001
2971423991814,2cyclictest2971-21cron22:05:001
2971423991814,2cyclictest0-21swapper/122:20:001
297142199184,12cyclictest3076902-21tr20:50:180
297142199182,1cyclictest3156551-21ntpq22:05:200
297142199181,3cyclictest3084888-21gpgv21:00:000
297142199181,2cyclictest3250628-21ntpq23:35:200
297142199181,2cyclictest3241166-21apt23:30:000
297142199181,2cyclictest3229620-21ntpq23:15:230
297142199181,2cyclictest3073720-21ntpq20:45:220
297142199181,2cyclictest3016967-21ntpq19:50:200
297142199181,2cyclictest3001493-21ntpq19:35:200
297142199181,2cyclictest252-21systemd-journal00:15:010
2971421991812,5cyclictest3260922-21ntpq23:45:200
2971421991812,4cyclictest3161651-21ntpq22:10:200
297142199181,1cyclictest3208983-21ntpq22:55:220
297142199181,1cyclictest3141162-21ntpq21:50:220
297142199180,4cyclictest3214064-21ntpq23:00:220
297142199180,3cyclictest3312383-21memory00:35:200
297142199180,3cyclictest3032333-21needreboot20:05:200
297142199180,1cyclictest3272505-21/usr/sbin/munin00:00:130
297142199180,17cyclictest3278725-21kthreadcore00:05:190
297142199180,17cyclictest3268676-21cut23:55:180
297143699173,3cyclictest3099839-21ntpq21:10:223
2971436991712,3cyclictest3085522-21/usr/sbin/munin21:00:203
2971429991714,2cyclictest3246168-21kworker/u8:3+events_unbound00:30:232
2971429991714,2cyclictest3224628-21kworker/u8:2+events_unbound23:25:242
2971429991714,2cyclictest3173340-21kworker/u8:1+flush-259:022:30:002
2971429991714,2cyclictest3111775-21irqcore21:25:172
2971429991713,3cyclictest2997437-21kworker/u8:4+events_unbound21:55:202
2971429991711,4cyclictest3163223-21/usr/sbin/munin22:15:122
297142399174,7cyclictest3188813-21munin-run22:40:001
2971423991713,3cyclictest3017919-21kworker/u8:1+flush-259:019:55:211
2971423991713,3cyclictest2912477-21kworker/u8:3+flush-259:019:25:011
2971423991713,3cyclictest252-21systemd-journal22:32:131
2971423991711,4cyclictest3110219-21ntpq21:20:221
2971423991711,4cyclictest3002630-21/usr/sbin/munin19:40:131
297142199176,10cyclictest0-21swapper/023:50:130
297142199175,11cyclictest0-21swapper/021:20:260
297142199173,2cyclictest3049338-21fschecks_time20:25:150
297142199172,1cyclictest2992723-21irqcore19:30:170
297142199171,3cyclictest2988187-21kthreadcore19:25:170
297142199171,2cyclictest3130772-21ntpq21:40:210
297142199171,2cyclictest3111365-21apt-get21:25:130
2971421991712,4cyclictest3198605-21ntpq22:45:210
297142199171,1cyclictest3193141-21ntpq22:40:210
297142199171,1cyclictest3120482-21ntpq21:30:210
297142199171,1cyclictest3037638-21ntpq20:10:190
297142199170,4cyclictest3302585-21grep00:25:290
297142199170,4cyclictest3255923-21/usr/sbin/munin23:40:240
297142199170,4cyclictest3039121-21idleruntime20:15:160
297142199170,4cyclictest3008239-21irqcore19:45:150
297142199170,4cyclictest3002897-21tune2fs19:40:130
297142199170,3cyclictest3308259-21sh00:34:590
297142199170,2cyclictest3215386-21/usr/sbin/munin23:05:130
2971436991612,3cyclictest3162769-21store22:15:003
2971436991612,3cyclictest3049108-21apt-get20:25:123
2971436991611,4cyclictest3231298-21diskstats23:20:143
2971429991613,2cyclictest3261801-21kworker/u8:1+flush-259:000:00:132
2971429991613,2cyclictest3105902-21cron21:20:002
2971429991613,2cyclictest3033022-21kworker/u8:1+events_unbound21:20:192
2971429991612,2cyclictest0-21swapper/220:55:002
2971429991611,3cyclictest3246168-21kworker/u8:3+events_unbound23:35:172
2971429991610,4cyclictest3261778-21latency_hist23:50:012
297142399169,5cyclictest3236272-21/usr/sbin/munin23:25:151
2971423991613,2cyclictest3038535-21idleruntime-cro20:14:591
2971423991613,2cyclictest0-21swapper/123:20:001
2971423991612,3cyclictest3131919-21/usr/sbin/munin21:45:191
2971423991612,3cyclictest2997437-21kworker/u8:4+flush-259:020:05:171
2971423991611,3cyclictest3246546-21/usr/sbin/munin23:35:131
2971423991610,3cyclictest3017868-21latency_hist19:55:001
297142199160,4cyclictest3101110-21tr21:15:120
2971436991512,2cyclictest3044369-21irqcore20:20:183
2971436991511,3cyclictest2997437-21kworker/u8:4+flush-259:021:30:163
2971436991510,4cyclictest252-21systemd-journal22:04:213
2971436991510,1cyclictest191rcu_preempt23:43:433
297143699150,5cyclictest3025250-21cat20:00:163
2971429991511,3cyclictest3147193-21kworker/u8:0+events_unbound22:40:172
2971429991511,3cyclictest2986928-21kworker/u8:2+events_unbound19:45:132
2971429991511,3cyclictest2912477-21kworker/u8:3+flush-259:019:15:242
2971429991510,4cyclictest3126563-21kworker/u8:1+flush-259:021:45:212
2971423991512,2cyclictest3173340-21kworker/u8:1+flush-259:023:10:171
2971423991511,3cyclictest3136436-21kworker/u8:2+flush-259:021:57:371
2971423991511,2cyclictest0-21swapper/123:55:001
297143699149,4cyclictest252-21systemd-journal00:05:003
297143699147,3cyclictest0-21swapper/323:14:593
297143699147,3cyclictest0-21swapper/322:09:203
297143699145,1cyclictest191rcu_preempt20:17:293
297143699142,3cyclictest50-21ksoftirqd/322:30:213
297143699142,3cyclictest3059716-21hddtemp_smartct20:35:173
297143699142,3cyclictest2980860-21ntpq19:15:223
297143699141,5cyclictest2988674-21cut19:25:183
297143699141,3cyclictest3068525-21ntpq20:40:193
297143699141,3cyclictest3068525-21ntpq20:40:193
2971436991411,2cyclictest3241153-21cron23:30:003
2971436991410,3cyclictest0-21swapper/321:10:003
297143699140,4cyclictest3006719-21gcc19:40:223
297142999149,3cyclictest3246168-21kworker/u8:3+flush-259:023:40:172
297142999144,8cyclictest3188792-21idleruntime-cro22:40:002
2971429991411,2cyclictest3167872-21kworker/u8:3+events_unbound22:20:182
2971429991411,2cyclictest3027345-21ntp_kernel_pll_20:00:222
2971429991410,3cyclictest3033022-21kworker/u8:1+flush-259:021:10:222
297142199140,2cyclictest2971471-21grep19:10:010
297143699139,3cyclictest3126783-21apt-get21:40:123
297143699138,3cyclictest3178487-21latency_hist22:30:003
297143699137,4cyclictest3246311-21latency_hist23:35:003
297143699136,1cyclictest191rcu_preempt19:56:173
297143699132,7cyclictest50-21ksoftirqd/323:00:223
297143699132,6cyclictest3167207-21ntpq22:15:183
297143699132,3cyclictest50-21ksoftirqd/323:55:203
297143699132,3cyclictest50-21ksoftirqd/300:30:233
297143699132,2cyclictest50-21ksoftirqd/320:05:123
297143699132,2cyclictest3219276-21ntpq23:05:213
297143699132,2cyclictest3038222-21modprobe20:11:203
297143699131,8cyclictest50-21ksoftirqd/319:30:213
297143699131,8cyclictest3146290-21ntpq21:55:233
297143699131,7cyclictest3115324-21ntpq21:25:223
297143699131,4cyclictest50-21ksoftirqd/320:30:233
2971436991310,2cyclictest2982239-21fschecks_time19:20:123
2971436991310,2cyclictest0-21swapper/323:20:003
297143699130,8cyclictest3007973-21df_inode19:45:123
297143699130,3cyclictest2971595-21/usr/sbin/munin19:10:223
297142999139,3cyclictest3246168-21kworker/u8:3+flush-259:000:10:252
297142999139,3cyclictest3246168-21kworker/u8:3+events_unbound00:15:152
297142999139,3cyclictest3147193-21kworker/u8:0+flush-259:022:47:202
297142999139,3cyclictest2960662-21kworker/u8:1+flush-259:019:12:182
297142999138,3cyclictest3266928-21latency_hist23:55:012
2971429991310,2cyclictest3147193-21kworker/u8:0+events_unbound23:15:242
2971429991310,2cyclictest2912442-21kworker/u8:0+events_unbound19:21:212
297142399139,2cyclictest0-21swapper/119:30:011
297142399136,5cyclictest3157425-21latency_hist22:10:011
297142399133,2cyclictest29927292sed19:30:161
2971423991311,1cyclictest0-21swapper/121:05:231
297143699129,2cyclictest3261768-21idleruntime-cro23:50:003
297143699129,2cyclictest3168464-21cpuspeed_turbos22:20:133
297143699129,2cyclictest3101101-21cpuspeed_turbos21:15:123
297143699124,1cyclictest191rcu_preempt00:15:173
297143699122,3cyclictest50-21ksoftirqd/320:45:213
297143699121,2cyclictest50-21ksoftirqd/322:45:223
297143699120,3cyclictest50-21ksoftirqd/300:10:233
297143699120,3cyclictest3205120-21diskstats22:55:143
297143699120,2cyclictest3281557-21ntpq00:05:233
297143699120,2cyclictest3125619-21ntpq21:35:203
297143699120,2cyclictest3016950-21ntpq19:50:203
297142999129,2cyclictest2912442-21kworker/u8:0+events_unbound19:30:012
297142999128,3cyclictest3017919-21kworker/u8:1+events_unbound19:55:172
297142999128,3cyclictest2912442-21kworker/u8:0+flush-259:019:40:252
297142999127,3cyclictest3095613-21latency_hist21:10:012
297142999126,4cyclictest3126533-21awk21:40:002
2971429991210,1cyclictest3246168-21kworker/u8:3+events_unbound00:25:192
297142399129,2cyclictest3173999-21irqcore22:25:171
297142399128,2cyclictest0-21swapper/121:20:001
297142399127,4cyclictest3198556-21grep22:45:201
297142399126,4cyclictest252-21systemd-journal00:35:221
297142399125,5cyclictest513-21gdbus00:15:001
2971423991210,1cyclictest0-21swapper/119:15:011
297143699118,2cyclictest491ktimers/321:45:113
297143699112,2cyclictest50-21ksoftirqd/323:50:223
297143699111,2cyclictest50-21ksoftirqd/300:35:123
297143699111,2cyclictest3137084-21apt-get21:50:123
297143699110,3cyclictest50-21ksoftirqd/319:35:203
297142999118,2cyclictest2997437-21kworker/u8:4+events_unbound20:45:142
297142999118,2cyclictest0-21swapper/222:30:142
297142999118,2cyclictest0-21swapper/222:05:012
297142399118,2cyclictest3205026-21cpuspeed_turbos22:55:131
297142399117,3cyclictest3064568-21apt-get20:40:101
297142399117,3cyclictest3064568-21apt-get20:40:101
297142399116,3cyclictest3246168-21kworker/u8:3+flush-259:000:30:171
297142999106,3cyclictest3246168-21kworker/u8:3+flush-259:000:35:272
297142399108,1cyclictest0-21swapper/123:30:151
297142399102,2cyclictest0-21swapper/120:31:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional