You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 13:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot5s.osadl.org (updated Sat Nov 22, 2025 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
325516599542,24cyclictest3255375-21/usr/sbin/munin19:10:130
3255169995019,3cyclictest0-21swapper/121:25:171
3255174994935,11cyclictest311ktimers/223:30:122
3255165994739,4cyclictest3370387-21sh00:40:000
3255165994739,4cyclictest3370387-21sh00:40:000
325516599471,37cyclictest0-21swapper/022:10:120
3255174994636,7cyclictest3276812-21grep20:05:262
3255174994636,7cyclictest3276812-21grep20:05:262
3255165994628,15cyclictest3347809-21idleruntime-cro23:35:000
325516599461,40cyclictest3353488-21/usr/sbin/munin23:50:160
3255169994436,4cyclictest3370398-21munin-plugin-st00:40:011
3255169994436,4cyclictest3370398-21munin-plugin-st00:40:001
325516599442,25cyclictest3361616-21date00:15:000
325516599441,40cyclictest3364632-21/usr/sbin/munin00:20:260
325516599441,38cyclictest3287921-21/usr/sbin/munin20:40:090
325516599440,38cyclictest3335897-21apt-get23:00:140
3255174994334,6cyclictest3315186-21cat22:00:002
3255174994332,7cyclictest3316426-21unixbench_multi22:00:262
3255169994318,5cyclictest3320740-21fschecks_time22:15:151
325516599431,37cyclictest3310305-21munin-node21:45:100
325516599431,37cyclictest3305287-21/usr/sbin/munin21:30:150
325516599431,37cyclictest3298684-21/usr/sbin/munin21:10:210
3255178994228,10cyclictest3363377-21sendmail00:20:003
325516599428,30cyclictest3350770-21grep23:40:270
325516599421,38cyclictest3333041-21/usr/sbin/munin22:50:230
3255178994126,11cyclictest391ktimers/320:15:123
3255174994134,4cyclictest3257491-21fschecks_count19:15:032
3255174994131,6cyclictest3343949-21grep23:20:242
325516599416,31cyclictest3355157-21/usr/sbin/munin23:55:160
325516599412,36cyclictest3328023-21/usr/sbin/munin22:35:230
325516599412,33cyclictest3351458-21apt-get23:45:130
325516599411,37cyclictest3357185-21/usr/sbin/munin00:00:220
325516599411,37cyclictest3348716-21/usr/sbin/munin23:35:230
325516599411,37cyclictest3290922-21cron20:48:000
325516599411,37cyclictest3272038-21/usr/sbin/munin19:55:140
325516599411,37cyclictest3265645-21/usr/sbin/munin19:35:200
325516599411,35cyclictest3312459-21/usr/sbin/munin21:50:200
325516599411,35cyclictest3283202-21/usr/sbin/munin20:25:170
325516599411,34cyclictest3330779-21apt-get22:45:160
325516599411,34cyclictest3270221-21apt-get19:50:140
3255174994028,8cyclictest3336896-21grep23:00:262
3255169994032,5cyclictest3282564-21cron20:25:011
325516599401,37cyclictest3286726-21/usr/sbin/munin20:35:220
325516599401,36cyclictest3365520-21apt-get00:25:110
325516599401,36cyclictest3314274-21/usr/sbin/munin21:55:240
325516599401,36cyclictest3302868-21users21:20:270
325516599401,36cyclictest3267376-21/usr/sbin/munin19:40:250
325516599401,36cyclictest3259966-21/usr/sbin/munin19:20:240
325516599401,34cyclictest3296734-21/usr/sbin/munin21:05:170
325516599401,34cyclictest3284620-21/usr/sbin/munin20:30:160
325516599401,33cyclictest3334176-21apt-get22:55:110
325516599401,23cyclictest3263538-21sshd19:30:200
325516599401,23cyclictest3263538-21sshd19:30:200
3255165994012,23cyclictest3304484-21grep21:25:270
325516599400,34cyclictest0-21swapper/023:05:110
325516599395,30cyclictest3258623-21grep19:15:270
325516599393,33cyclictest3291864-21/usr/sbin/munin20:50:190
3255165993931,4cyclictest0-21swapper/021:20:010
325516599391,35cyclictest3362512-21proc_pri00:15:240
325516599391,35cyclictest3341727-21/usr/sbin/munin23:15:190
325516599391,35cyclictest3340086-21/usr/sbin/munin23:10:270
325516599391,35cyclictest3322883-21/usr/sbin/munin22:20:220
325516599391,35cyclictest3317760-21/usr/sbin/munin22:05:220
325516599391,35cyclictest3317760-21/usr/sbin/munin22:05:220
325516599391,35cyclictest3280017-21/usr/sbin/munin20:15:310
325516599391,35cyclictest3276137-21/usr/sbin/munin20:05:200
325516599391,35cyclictest3276137-21/usr/sbin/munin20:05:190
325516599391,33cyclictest3308840-21/usr/sbin/munin21:40:150
325516599391,33cyclictest3295194-21/usr/sbin/munin21:00:190
325516599391,33cyclictest3268518-21apt-get19:45:120
325516599390,33cyclictest3277463-21apt-get20:10:100
325516599384,31cyclictest3325761-21/usr/sbin/munin22:30:140
325516599383,30cyclictest3320585-21/usr/sbin/munin22:15:140
3255165993829,5cyclictest0-21swapper/020:00:280
325516599382,33cyclictest3262189-21/usr/sbin/munin19:25:280
325516599381,34cyclictest3344842-21/usr/sbin/munin23:25:190
325516599381,34cyclictest3344842-21/usr/sbin/munin23:25:180
325516599381,34cyclictest3329987-21/usr/sbin/munin22:40:250
325516599381,34cyclictest3307225-21/usr/sbin/munin21:35:200
325516599381,34cyclictest3281702-21/usr/sbin/munin20:20:250
3255178993725,7cyclictest3290632-21grep20:45:263
3255174993730,4cyclictest3370401-21sendmail00:40:012
3255174993730,4cyclictest3370401-21sendmail00:40:002
3255174993729,4cyclictest3280683-21latency_hist20:20:012
3255174993728,6cyclictest3357590-21unixbench_multi00:00:252
3255169993728,6cyclictest3262770-21latency_hist19:30:011
325516599372,32cyclictest3367328-21/usr/sbin/munin00:30:140
325516599371,33cyclictest3358801-21/usr/sbin/munin00:05:200
325516599371,33cyclictest3343613-21/usr/sbin/munin23:20:220
325516599371,33cyclictest3316597-21sshd22:01:310
325516599371,33cyclictest3293558-21/usr/sbin/munin20:55:210
3255174993630,4cyclictest3289144-21kworker/u8:2+events_unbound20:56:402
3255169993623,9cyclictest0-21swapper/120:05:001
3255174993527,4cyclictest0-21swapper/222:06:492
3255174993527,4cyclictest0-21swapper/222:06:482
3255174993525,7cyclictest3271699-21munin-plugin-st19:55:002
3255174993525,5cyclictest0-21swapper/220:31:392
3255178993424,7cyclictest3342122-21cat23:15:243
3255174993426,5cyclictest311ktimers/219:10:112
3255169993426,4cyclictest0-21swapper/122:35:261
3255178993322,8cyclictest3335652-21idleruntime-cro23:00:003
3255174993325,4cyclictest0-21swapper/219:41:332
3255174993325,4cyclictest0-21swapper/200:05:172
3255174993322,7cyclictest0-21swapper/223:10:202
3255169993323,7cyclictest3296211-21sh21:05:001
3255169993322,7cyclictest0-21swapper/123:30:241
3255178993225,4cyclictest3330542-21auditctl22:45:003
3255178993224,4cyclictest0-21swapper/322:00:273
3255178993223,6cyclictest3282171-21unixbench_singl20:20:283
3255178993223,5cyclictest0-21swapper/321:55:273
3255178993223,4cyclictest0-21swapper/321:45:143
3255174993226,4cyclictest3326812-21kworker/u8:2+events_unbound22:33:442
3255174993224,4cyclictest0-21swapper/223:55:222
3255174993223,4cyclictest0-21swapper/220:35:182
3255174993222,7cyclictest3289787-21fschecks_count20:45:172
325517499321,5cyclictest131rcu_preempt22:20:112
3255169993225,4cyclictest0-21swapper/122:05:241
3255169993225,4cyclictest0-21swapper/122:05:241
3255169993225,3cyclictest0-21swapper/119:40:201
3255169993224,5cyclictest3335642-21cron23:00:001
3255169993223,6cyclictest231ktimers/120:40:101
3255178993124,4cyclictest0-21swapper/321:10:003
3255178993124,4cyclictest0-21swapper/300:15:003
3255178993124,3cyclictest0-21swapper/322:15:243
3255178993121,6cyclictest0-21swapper/323:50:233
3255174993120,7cyclictest3297443-21grep21:05:252
325517499310,2cyclictest0-21swapper/221:50:122
3255169993123,4cyclictest0-21swapper/121:35:281
3255169993121,6cyclictest0-21swapper/122:10:251
3255169993118,9cyclictest0-21swapper/119:40:011
3255178993024,3cyclictest0-21swapper/321:40:003
3255178993022,4cyclictest0-21swapper/321:15:273
3255178993021,5cyclictest0-21swapper/319:10:243
3255178993020,7cyclictest3323290-21grep22:20:253
3255178993020,6cyclictest0-21swapper/319:20:273
3255174993022,4cyclictest0-21swapper/219:35:242
3255174993021,6cyclictest3335651-21munin-plugin-st23:00:012
3255174993018,8cyclictest0-21swapper/219:25:192
325517499301,6cyclictest0-21swapper/223:45:132
325517499301,5cyclictest131rcu_preempt20:25:112
3255169993023,4cyclictest0-21swapper/123:40:271
3255169993022,4cyclictest0-21swapper/121:50:271
3255169993022,4cyclictest0-21swapper/119:10:141
3255169993022,4cyclictest0-21swapper/100:00:251
3255169993020,6cyclictest0-21swapper/120:10:231
3255178992923,3cyclictest0-21swapper/319:35:013
3255178992923,3cyclictest0-21swapper/319:35:013
3255174992922,5cyclictest3356044-21kworker/u8:1+events_unbound00:13:492
3255174992921,4cyclictest0-21swapper/221:10:262
3255174992919,7cyclictest3320291-21cron22:15:002
3255174992918,6cyclictest0-21swapper/221:35:002
325517499291,5cyclictest131rcu_preempt21:15:142
325517499291,15cyclictest0-21swapper/200:25:102
3255169992922,4cyclictest0-21swapper/100:10:261
3255169992921,4cyclictest3287681-21munin-run20:40:011
3255178992819,6cyclictest0-21swapper/320:45:003
3255178992819,5cyclictest0-21swapper/323:30:253
3255174992822,4cyclictest3289144-21kworker/u8:2+events_unbound20:54:182
3255174992821,5cyclictest3361178-21kworker/u8:0+events_unbound00:23:552
3255174992821,5cyclictest3299538-21kworker/u8:0+events_unbound21:22:422
3255174992821,5cyclictest3273171-21kworker/u8:0+events_unbound20:00:432
3255174992819,4cyclictest0-21swapper/222:25:252
325517499281,4cyclictest131rcu_preempt23:15:122
3255174992814,10cyclictest1-21systemd22:46:492
3255169992820,4cyclictest0-21swapper/119:55:261
3255178992719,4cyclictest0-21swapper/320:25:243
3255178992717,6cyclictest0-21swapper/323:20:243
3255174992720,5cyclictest3267958-21kworker/u8:1+events_unbound19:57:462
3255174992720,4cyclictest0-21swapper/223:35:262
3255174992718,7cyclictest3256473-21kworker/u8:2+events_unbound19:32:392
3255174992718,7cyclictest3256473-21kworker/u8:2+events_unbound19:32:392
3255174992716,8cyclictest215-21dbus-daemon22:41:492
3255169992721,3cyclictest0-21swapper/122:45:011
3255169992720,4cyclictest0-21swapper/123:00:181
3255169992719,4cyclictest0-21swapper/100:05:251
3255169992718,6cyclictest3343116-21fschecks_time23:20:151
3255169992716,7cyclictest0-21swapper/120:45:271
3255178992618,4cyclictest0-21swapper/321:00:003
3255178992618,4cyclictest0-21swapper/320:00:183
3255178992616,6cyclictest0-21swapper/320:00:003
3255178992613,10cyclictest391ktimers/320:50:113
3255174992620,4cyclictest3339196-21latency_hist23:10:002
3255174992619,5cyclictest3308243-21kworker/u8:0+events_unbound21:41:332
3255174992618,5cyclictest3273171-21kworker/u8:0+events_unbound20:20:062
3255174992618,4cyclictest0-21swapper/223:50:232
3255169992619,3cyclictest0-21swapper/121:15:001
3255169992619,3cyclictest0-21swapper/119:20:011
3255169992618,5cyclictest3342676-21sh23:20:001
3255169992617,6cyclictest0-21swapper/123:10:301
3255178992518,4cyclictest0-21swapper/322:50:283
3255178992518,4cyclictest0-21swapper/320:05:203
3255178992518,4cyclictest0-21swapper/320:05:203
3255178992518,4cyclictest0-21swapper/319:55:013
3255178992514,7cyclictest0-21swapper/319:35:203
325517499258,4cyclictest131rcu_preempt21:35:142
3255174992519,4cyclictest3347568-21kworker/u8:0+events_unbound23:43:042
3255174992519,4cyclictest3233912-21kworker/u8:0+events_unbound19:20:522
3255174992518,5cyclictest3330783-21kworker/u8:2+events_unbound22:50:242
325517499250,4cyclictest131rcu_preempt19:45:122
3255169992518,4cyclictest0-21swapper/123:10:001
3255169992517,4cyclictest0-21swapper/122:25:161
3255169992515,6cyclictest0-21swapper/121:05:201
3255169992515,6cyclictest0-21swapper/100:35:001
3255169992514,7cyclictest3301071-21grep21:15:271
3255178992418,3cyclictest0-21swapper/323:50:013
3255178992418,3cyclictest0-21swapper/319:15:153
3255178992417,4cyclictest0-21swapper/323:40:273
3255178992416,4cyclictest0-21swapper/322:11:483
3255178992416,4cyclictest0-21swapper/322:05:243
3255178992416,4cyclictest0-21swapper/322:05:243
3255178992416,4cyclictest0-21swapper/321:40:153
3255178992416,4cyclictest0-21swapper/321:00:263
3255178992414,6cyclictest0-21swapper/323:25:253
3255178992414,6cyclictest0-21swapper/323:25:253
325517499247,10cyclictest0-21swapper/200:30:192
3255174992417,5cyclictest3310001-21kworker/u8:1+events_unbound22:15:112
3255174992417,4cyclictest3326812-21kworker/u8:2+events_unbound22:35:222
3255169992418,3cyclictest0-21swapper/122:30:221
3255169992418,3cyclictest0-21swapper/120:05:241
3255169992418,3cyclictest0-21swapper/120:05:241
3255169992417,3cyclictest0-21swapper/123:55:121
3255169992416,5cyclictest0-21swapper/122:00:231
3255169992416,5cyclictest0-21swapper/119:35:011
3255169992416,5cyclictest0-21swapper/119:35:001
3255169992415,6cyclictest3303247-21cron21:25:021
3255169992415,5cyclictest0-21swapper/119:45:241
3255178992317,3cyclictest0-21swapper/320:35:293
3255178992316,4cyclictest0-21swapper/319:45:243
3255178992316,3cyclictest0-21swapper/321:25:243
3255178992315,5cyclictest33390542sleep323:10:003
3255178992315,5cyclictest0-21swapper/323:10:213
3255178992315,5cyclictest0-21swapper/300:00:013
3255178992315,4cyclictest0-21swapper/321:50:253
3255178992313,5cyclictest440-21cron21:35:013
3255178992312,8cyclictest3349523-21tr23:39:593
3255178992312,7cyclictest3336918-21grep23:00:263
325517499234,14cyclictest311ktimers/200:15:132
3255169992317,3cyclictest0-21swapper/123:50:231
3255169992316,5cyclictest231ktimers/123:35:161
3255169992316,5cyclictest231ktimers/120:25:111
3255169992315,5cyclictest0-21swapper/120:55:151
3255178992216,3cyclictest0-21swapper/320:10:143
3255178992215,4cyclictest3370388-21sh00:40:003
3255178992215,4cyclictest3370388-21sh00:40:003
3255178992215,4cyclictest0-21swapper/321:10:163
3255178992215,4cyclictest0-21swapper/300:25:193
3255178992215,3cyclictest0-21swapper/322:45:263
3255178992215,3cyclictest0-21swapper/322:35:203
3255178992214,5cyclictest0-21swapper/300:25:003
3255178992212,7cyclictest3358046-21apt00:05:003
3255174992216,3cyclictest0-21swapper/223:25:272
3255174992216,3cyclictest0-21swapper/223:25:272
3255174992216,3cyclictest0-21swapper/220:10:232
3255169992215,4cyclictest3366582-21unixbench_singl00:25:231
3255169992213,6cyclictest3270786-21ntp_states19:50:221
3255169992213,6cyclictest0-21swapper/121:30:271
3255169992212,7cyclictest3322789-21ntpq22:20:191
325517499218,5cyclictest3310001-21kworker/u8:1+events_unbound21:47:192
3255174992114,5cyclictest3294214-21kworker/u8:3+events_unbound21:00:122
3255169992115,3cyclictest0-21swapper/121:45:151
3255169992113,5cyclictest0-21swapper/121:45:011
3255169992113,5cyclictest0-21swapper/119:20:221
3255169992112,6cyclictest3364093-21cat00:20:191
325516999210,12cyclictest0-21swapper/123:26:281
325516999210,12cyclictest0-21swapper/123:26:271
3255178992014,3cyclictest0-21swapper/322:25:173
3255178992013,4cyclictest3267844-21tail19:40:293
3255178992010,7cyclictest3325809-21tune2fs22:30:153
3255174992014,3cyclictest3304949-21cron21:30:002
3255169992015,2cyclictest0-21swapper/120:50:151
3255169992012,5cyclictest231ktimers/100:15:051
3255169992010,7cyclictest3284590-21awk20:30:141
3255178991913,3cyclictest0-21swapper/300:30:163
3255178991912,4cyclictest0-21swapper/300:10:003
325516999199,7cyclictest3351844-21cat23:45:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional