You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-25 - 23:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 25, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
314415222190,8sleep04497-21PK-Backend07:05:170
314415222190,8sleep04497-21PK-Backend07:05:170
197399780,5rtkit-daemon0-21swapper/308:59:463
197399780,11rtkit-daemon3236102-21gunzip11:25:240
197399760,63rtkit-daemon0-21swapper/207:06:332
197399760,63rtkit-daemon0-21swapper/207:06:322
197399730,5rtkit-daemon0-21swapper/207:20:192
197399690,5rtkit-daemon0-21swapper/210:02:102
314510026545,14sleep30-21swapper/307:08:553
314510026545,14sleep30-21swapper/307:08:553
31935702630,5sleep3411rcuc/309:25:203
31935702630,5sleep3411rcuc/309:25:193
314500526243,13sleep10-21swapper/107:07:401
314500526243,13sleep10-21swapper/107:07:401
32394632560,4sleep10-21swapper/111:35:251
32304292560,5sleep3411rcuc/311:10:153
314531199550,50cyclictest1972-21rtkit-daemon11:44:540
197399520,37rtkit-daemon3200063-21munin-run09:45:003
32444812510,7chrt4497-21PK-Backend11:50:211
197399510,37rtkit-daemon3182463-21munin-run08:55:013
314532399504,20cyclictest3218611-21head10:35:203
314532399495,19cyclictest3205344-21cat10:00:023
197399490,6rtkit-daemon0-21swapper/208:45:152
32367772480,5sleep1251rcuc/111:30:011
32315222480,8sleep13231517-21apt11:15:011
32098552480,14sleep013-21ksoftirqd/010:10:280
32098552480,14sleep013-21ksoftirqd/010:10:280
3145319994822,23cyclictest3219435-21unin-run10:40:012
197399480,6rtkit-daemon1972-21rtkit-daemon11:15:071
197399480,34rtkit-daemon0-21swapper/309:35:143
197399480,33rtkit-daemon0-21swapper/310:20:023
32052382450,5sleep2331rcuc/209:59:592
31759262450,14sleep126-21ksoftirqd/108:35:121
31759262450,14sleep126-21ksoftirqd/108:35:121
314532399453,17cyclictest3146611-21users07:10:293
197399440,31rtkit-daemon0-21swapper/307:40:103
197399440,31rtkit-daemon0-21swapper/208:20:172
314532399435,11cyclictest0-21swapper/307:20:023
3145323994310,30cyclictest3164861-21turbostat08:05:013
3145311994328,11cyclictest1972-21rtkit-daemon07:47:180
32104922420,4sleep20-21swapper/210:15:012
32104922420,4sleep20-21swapper/210:15:002
3145323994220,18cyclictest3192870-21unin-run09:25:003
32378532410,7sleep04497-21PK-Backend11:30:280
31666072410,6chrt0-21swapper/208:10:022
314532399415,14cyclictest3260839-21timerwakeupswit12:35:303
3145323994115,21cyclictest3226902-21cstates11:00:163
3145323994111,25cyclictest3212705-21apt10:20:143
3145319994126,11cyclictest1-21systemd07:15:012
3145315994122,15cyclictest3172101-21unin-run08:25:001
32584952400,7chrt4497-21PK-Backend12:30:200
31878332400,6chrt0-21swapper/209:10:022
314532399404,31cyclictest438-20systemd-journal08:09:163
314532399404,14cyclictest3244300-21/usr/sbin/munin11:50:193
314532399404,14cyclictest3177294-21munin-run08:40:003
314532399404,14cyclictest3177294-21munin-run08:40:003
3145323994010,25cyclictest3207490-21cstates10:05:163
3145319994022,14cyclictest3255376-21users12:20:312
3145311994022,13cyclictest3222111-21ntp_kernel_err10:45:230
31700922390,9sleep33034-21update-notifier08:17:073
314532399399,26cyclictest3178652-21seq08:40:463
314532399396,9cyclictest0-21swapper/311:55:133
314532399394,14cyclictest3220887-21taskset10:42:003
314532399393,15cyclictest3161859-21cstates07:55:123
3145323993911,9cyclictest1-21systemd12:20:013
3145323993911,6cyclictest0-21swapper/307:30:153
3145323993911,24cyclictest3214962-21irqstats10:25:193
3145323993911,22cyclictest3239665-21/usr/sbin/munin11:35:283
3145323993910,26cyclictest3175532-21perf08:35:013
3145323993910,24cyclictest3203727-21sed09:55:013
3145319993923,12cyclictest3250491-21sshd12:09:172
3145311993926,9cyclictest13-21ksoftirqd/007:25:000
3145311993922,13cyclictest3167203-21df08:10:150
3145311993921,13cyclictest438-20systemd-journal10:45:010
314531199391,32cyclictest3169570-21ntpq08:15:240
314532399387,5cyclictest0-21swapper/312:26:023
314532399385,11cyclictest3189814-21cpuspeed_turbos09:15:143
3145323993825,10cyclictest3229781-21perf11:09:593
314532399382,31cyclictest3232370-21memory11:15:223
3145323993823,11cyclictest3180420-21sshd08:46:183
3145323993820,13cyclictest3167593-21ntp_states08:10:243
3145323993812,21cyclictest3149496-21grep07:20:153
3145319993822,12cyclictest3212171-21apt-get10:20:002
3145315993822,11cyclictest438-20systemd-journal09:45:011
314531199382,8cyclictest3191745-21df_inode09:20:150
197399380,25rtkit-daemon0-21swapper/208:36:022
197399380,25rtkit-daemon0-21swapper/208:36:012
197399380,25rtkit-daemon0-21swapper/208:33:192
314532399378,10cyclictest3255035-21ntpq12:20:253
3145323993723,10cyclictest3234966-21cron11:25:013
3145323993723,10cyclictest3234966-21cron11:25:013
3145323993712,21cyclictest3251574-21ntp_states12:10:243
3145323993710,22cyclictest3258644-21memory12:30:223
3145319993726,7cyclictest3252750-21tr12:15:132
314531999372,31cyclictest3222152-21ntp_states10:45:242
3145319993722,11cyclictest3258438-21sshd12:30:182
3145319993720,13cyclictest3252323-21pluginstate12:15:002
3145319993720,13cyclictest3147639-21ntp_states07:15:062
3145319993720,12cyclictest438-20systemd-journal09:54:592
3145315993726,7cyclictest3233575-21ls11:20:121
3145315993726,7cyclictest3233575-21ls11:20:121
3145315993724,5cyclictest231irq_work/110:46:581
197399370,25rtkit-daemon0-21swapper/307:30:003
197399370,24rtkit-daemon0-21swapper/210:25:352
31636922360,4chrt0-21swapper/208:00:162
314532399369,23cyclictest3188499-21memory09:10:223
314532399369,22cyclictest3209354-21df_inode10:10:163
314532399369,22cyclictest3209354-21df_inode10:10:163
3145323993614,19cyclictest42-21ksoftirqd/307:35:383
3145319993620,12cyclictest3231061-21sed11:10:292
3145319993618,13cyclictest3234600-21users11:20:302
3145319993618,13cyclictest3234600-21users11:20:292
3145319993614,6cyclictest141rcu_preempt09:50:012
3145315993624,8cyclictest3170347-21turbostat08:20:001
3145315993621,11cyclictest923-21dbus-daemon07:15:011
3145315993618,6cyclictest26-21ksoftirqd/112:25:001
314531599361,10cyclictest0-21swapper/111:00:341
3145311993622,10cyclictest13-21ksoftirqd/010:55:260
3145311993620,12cyclictest3174938-21python308:30:250
314531199361,13cyclictest3231509-21idleruntime-cro11:15:000
314532399359,10cyclictest3216202-21sort10:30:123
314532399354,27cyclictest3206524-21timerandwakeup10:00:303
3145323993517,13cyclictest438-20systemd-journal11:34:543
3145323993517,13cyclictest3172314-21find08:25:103
3145323993511,20cyclictest3236719-21munin-plugin-st11:29:593
3145323993510,21cyclictest3223508-21diskstats10:50:163
3145319993521,10cyclictest3173786-21cron08:30:002
3145319993521,10cyclictest3159771-21cat07:50:022
3145319993521,10cyclictest3154394-21cron07:35:002
3145319993518,12cyclictest3248246-21python312:00:292
314531999351,22cyclictest3178977-21taskset08:44:582
3145315993524,7cyclictest3242365-21tr11:45:141
3145315993524,7cyclictest3242365-21tr11:45:141
3145315993521,10cyclictest3250052-21awk12:05:281
314531599351,9cyclictest4497-21PK-Backend11:30:411
3145315993517,14cyclictest3151574-21df07:25:151
3145311993525,6cyclictest3246986-21taskset11:58:410
3145311993519,12cyclictest3177920-21forks08:40:180
314531199351,30cyclictest4497-21PK-Backend08:08:190
197399350,24rtkit-daemon0-21swapper/210:50:362
197399350,24rtkit-daemon0-21swapper/209:23:442
314532399347,23cyclictest3161017-21vmstat07:50:303
3145323993422,8cyclictest3225346-21irqstats10:55:223
3145323993417,13cyclictest3186934-21ntp_states09:05:253
3145323993413,17cyclictest3243663-21unin-run11:50:013
3145323993413,17cyclictest3243663-21unin-run11:50:013
3145323993410,20cyclictest3240435-21/usr/sbin/munin11:40:093
3145319993422,9cyclictest34-21ksoftirqd/210:35:022
3145319993419,11cyclictest3167623-21pluginstate08:10:262
3145319993418,12cyclictest3197441-21smartctl09:35:282
3145319993418,12cyclictest3194554-21cron09:30:002
3145319993418,12cyclictest3194554-21cron09:29:592
3145319993418,12cyclictest3181290-21expr08:50:152
3145319993418,11cyclictest3151348-21grep07:25:132
3145319993417,12cyclictest3229040-21ntp_kernel_pll_11:05:232
3145319993416,13cyclictest3257043-21ntpq12:25:272
314531999341,29cyclictest4497-21PK-Backend09:40:032
314531999341,27cyclictest3242544-21df11:45:172
314531999341,27cyclictest3242544-21df11:45:162
314531599347,23cyclictest3180713-21cron08:49:591
3145315993424,6cyclictest26-21ksoftirqd/108:57:381
3145315993422,6cyclictest26-21ksoftirqd/110:35:261
3145315993420,10cyclictest3196402-21sed09:35:011
314531599341,9cyclictest4497-21PK-Backend10:40:591
314531599341,9cyclictest4497-21PK-Backend10:28:351
314531599341,9cyclictest4497-21PK-Backend10:03:561
314531599341,8cyclictest0-21swapper/110:23:191
314531599341,30cyclictest4497-21PK-Backend10:54:091
314531599341,29cyclictest3163761-21sshd08:00:171
314531599341,13cyclictest32041162ls09:55:141
3145315993410,5cyclictest26-21ksoftirqd/107:15:131
314531199342,9cyclictest13-21ksoftirqd/009:15:000
3145311993425,5cyclictest4497-21PK-Backend10:20:040
3145311993424,6cyclictest3180726-21turbostat08:50:000
3145311993423,7cyclictest3198017-21cron09:40:000
3145311993420,10cyclictest3260837-21wc12:35:300
3145311993417,13cyclictest3231989-21cat11:15:150
3145311993417,13cyclictest3207653-21which10:05:190
3145311993417,12cyclictest438-20systemd-journal08:30:000
3145311993415,14cyclictest13-21ksoftirqd/009:50:240
314531199341,12cyclictest3159917-21acpi07:50:110
32499552330,7sleep04497-21PK-Backend12:05:260
31522522330,8chrt4497-21PK-Backend07:25:280
314532399332,27cyclictest3200566-21cstates09:45:133
314532399332,12cyclictest3249779-21seq12:05:213
3145323993319,11cyclictest42-21ksoftirqd/309:30:323
3145323993314,14cyclictest438-20systemd-journal07:50:013
314532399331,29cyclictest0-21swapper/309:02:293
314532399331,29cyclictest0-21swapper/309:02:283
314532399331,11cyclictest3223076-21sed10:50:023
3145319993325,5cyclictest0-21swapper/211:39:122
3145319993322,8cyclictest3241602-21sshd11:41:182
3145319993320,8cyclictest3190360-21ntp_kernel_pll_09:15:232
3145319993318,11cyclictest3195291-21cat09:30:202
3145319993318,11cyclictest3188010-21wc09:10:142
3145319993318,11cyclictest3184985-21sed09:00:202
3145319993318,11cyclictest3184985-21sed09:00:192
3145319993317,12cyclictest3220642-21sed10:40:282
3145319993317,12cyclictest3156117-21sshd07:39:162
3145319993316,13cyclictest3157544-21users07:40:272
3145319993316,12cyclictest438-20systemd-journal11:00:012
314531999331,27cyclictest3232265-21awk11:15:202
3145315993326,4cyclictest4497-21PK-Backend11:41:421
3145315993322,6cyclictest26-21ksoftirqd/109:00:141
3145315993322,6cyclictest26-21ksoftirqd/109:00:141
3145315993321,8cyclictest3197069-21sed09:35:201
3145315993319,10cyclictest3186805-21grep09:05:211
3145315993319,10cyclictest3167842-21sed08:10:291
3145315993318,11cyclictest3165709-21cat08:05:201
3145315993317,12cyclictest3247091-21cron12:00:011
3145315993314,6cyclictest3181943-21xargs08:50:281
314531599331,29cyclictest4497-21PK-Backend11:06:401
314531599331,29cyclictest0-21swapper/110:58:501
314531599331,29cyclictest0-21swapper/110:19:271
314531599331,29cyclictest0-21swapper/107:31:311
314531599331,14cyclictest3202777-21awk09:50:231
3145311993326,4cyclictest4497-21PK-Backend10:02:070
3145311993325,4cyclictest4497-21PK-Backend07:31:070
3145311993324,6cyclictest3243644-21grep11:50:000
3145311993324,6cyclictest3243644-21grep11:50:000
3145311993323,6cyclictest3205252-21sh10:00:000
3145311993318,11cyclictest3233770-21ls11:20:140
3145311993318,11cyclictest3233770-21ls11:20:140
3145311993317,12cyclictest1-21systemd10:35:010
314531199331,29cyclictest4497-21PK-Backend12:17:340
314531199331,29cyclictest4497-21PK-Backend10:18:260
314531199331,29cyclictest4497-21PK-Backend09:44:010
314531199331,29cyclictest0-21swapper/010:29:400
314531199331,29cyclictest0-21swapper/009:16:110
314531199331,28cyclictest4497-21PK-Backend07:35:490
314531199331,11cyclictest4497-21PK-Backend08:57:160
32004392320,7sleep04497-21PK-Backend09:45:120
3145323993221,7cyclictest3170583-21/usr/sbin/munin08:20:223
3145323993215,13cyclictest3248455-21users12:00:303
314531999325,10cyclictest141rcu_preempt11:25:172
3145319993217,10cyclictest3159643-21cron07:50:022
3145319993216,12cyclictest3246067-21basename11:55:192
3145319993216,12cyclictest3238132-21seq11:31:132
3145319993216,12cyclictest3207624-21df10:05:172
3145319993215,12cyclictest438-20systemd-journal07:55:162
3145315993225,4cyclictest4497-21PK-Backend12:37:031
3145315993225,4cyclictest0-21swapper/110:09:261
3145315993223,6cyclictest3257783-21latency_hist12:30:011
3145315993222,5cyclictest26-21ksoftirqd/109:15:231
3145315993220,7cyclictest3149610-21tr07:20:151
3145315993219,9cyclictest3252757-21ls12:15:131
3145315993219,9cyclictest26-21ksoftirqd/109:20:271
3145315993218,10cyclictest3161484-21sed07:55:011
3145315993217,11cyclictest3209359-21diskstats10:10:181
3145315993217,11cyclictest3209359-21diskstats10:10:181
3145315993216,13cyclictest3194156-21timerwakeupswit09:25:301
3145315993216,13cyclictest3194156-21timerwakeupswit09:25:301
314531599321,4cyclictest4497-21PK-Backend12:32:031
314531599321,4cyclictest4497-21PK-Backend08:31:151
314531599321,4cyclictest0-21swapper/112:10:281
314531599321,4cyclictest0-21swapper/110:33:321
3145315993211,6cyclictest141rcu_preempt12:00:121
3145311993225,4cyclictest4497-21PK-Backend12:04:060
3145311993225,4cyclictest4497-21PK-Backend09:31:310
3145311993220,8cyclictest3172073-21idleruntime-cro08:25:000
3145311993219,9cyclictest3244447-21sed11:50:210
3145311993218,10cyclictest3194556-21cron09:30:000
3145311993218,10cyclictest3194556-21cron09:30:000
3145311993217,11cyclictest3692989-21snapd07:41:100
3145311993216,12cyclictest3226601-21cpu11:00:130
3145311993215,13cyclictest3229074-21ntp_states11:05:250
314531199321,29cyclictest0-21swapper/007:58:380
314531199321,28cyclictest0-21swapper/012:24:070
314531199321,27cyclictest3252304-21turbostat12:15:000
32231862310,7sleep04497-21PK-Backend10:50:130
3145319993117,10cyclictest3212830-21grep10:20:152
314531999311,4cyclictest4497-21PK-Backend09:00:022
314531999311,4cyclictest0-21swapper/211:52:392
3145319993112,15cyclictest2345-21gdbus08:19:562
314531999311,13cyclictest4497-21PK-Backend11:00:122
3145315993124,4cyclictest4497-21PK-Backend08:29:591
3145315993116,11cyclictest3158712-21latency07:45:211
3145315993115,11cyclictest438-20systemd-journal09:50:011
314531599311,4cyclictest4497-21PK-Backend07:43:231
3145315993113,14cyclictest3155045-21df_inode07:35:161
3145311993120,7cyclictest3218151-21cstates10:35:130
3145311993117,11cyclictest3176434-21sed08:35:200
3145311993117,11cyclictest3176434-21sed08:35:190
3145311993117,10cyclictest3146150-21grep07:10:210
3145311993116,11cyclictest3164256-21python308:00:280
3145311993115,12cyclictest923-21dbus-daemon07:20:010
314531199311,4cyclictest4497-21PK-Backend12:27:030
314531199311,4cyclictest4497-21PK-Backend09:05:350
314531199311,10cyclictest0-21swapper/008:50:140
3145319993014,12cyclictest3259790-21apt12:35:122
3145315993016,10cyclictest3161647-21apt07:55:111
3145311993015,11cyclictest3239228-21latency11:35:190
3145311992917,9cyclictest3184453-21apt09:00:110
3145311992917,9cyclictest3184453-21apt09:00:100
314531599282,4cyclictest141rcu_preempt08:40:201
3145315992812,14cyclictest2212-21gvfs-afc-volume09:12:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional