You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 09:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Wed Nov 19, 2025 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3604856995144,5cyclictest3648087-21kworker/u8:1+events_unbound21:27:113
3604853995037,10cyclictest311ktimers/220:05:142
3604844995034,10cyclictest121ktimers/021:45:160
3604844994636,6cyclictest3655077-21grep21:30:290
3604844994532,10cyclictest121ktimers/000:15:150
3604848994436,5cyclictest299-21systemd-logind20:01:561
3604856994321,13cyclictest131rcu_preempt21:37:233
3604844994310,27cyclictest3606788-21latency_hist19:15:030
3604856994236,4cyclictest3662267-21kworker/u8:1+events_unbound22:12:193
3604856994235,5cyclictest3620704-21kworker/u8:0+events_unbound20:01:583
3604856994233,5cyclictest3644832-21cron21:05:003
3604856994226,5cyclictest131rcu_preempt00:35:193
360484499422,8cyclictest3649805-21/usr/sbin/munin21:15:280
360484499421,9cyclictest3712541-21/usr/sbin/munin00:20:150
360484499421,3cyclictest3697124-21cat23:35:190
360484499421,10cyclictest3703958-21/usr/sbin/munin23:55:160
360484499421,10cyclictest3643362-21/usr/sbin/munin21:00:120
3604856994135,4cyclictest3620704-21kworker/u8:0+events_unbound20:11:573
3604856994135,4cyclictest3620704-21kworker/u8:0+events_unbound20:11:573
360484499418,30cyclictest3692812-21unixbench_singl23:20:270
360484499413,8cyclictest3674542-21/usr/sbin/munin22:30:160
360484499413,5cyclictest3658559-21cron21:41:000
3604844994129,8cyclictest3615131-21grep19:35:280
360484499411,9cyclictest3687393-21sensors_temp23:05:250
360484499411,9cyclictest3651460-21/usr/sbin/munin21:20:260
360484499411,8cyclictest3679919-21/usr/sbin/munin22:45:210
360484499411,8cyclictest3671092-21/usr/sbin/munin22:20:150
360484499411,8cyclictest3664826-21/usr/sbin/munin22:00:250
360484499411,8cyclictest3620266-21grep19:50:270
360484499411,5cyclictest3632286-21users20:25:260
3604856994034,4cyclictest3684400-21kworker/u8:0+events_unbound23:07:313
3604856994032,4cyclictest3682786-21munin-plugin-st22:55:003
3604848994017,7cyclictest131rcu_preempt22:45:151
360484499403,10cyclictest3646483-21cron21:07:000
360484499402,5cyclictest3624487-21df20:05:160
360484499402,4cyclictest3702990-21grep23:50:250
360484499401,9cyclictest3690389-21/usr/sbin/munin23:15:210
360484499401,9cyclictest3660998-21/usr/sbin/munin21:50:180
360484499401,8cyclictest3656457-21grep21:35:240
360484499401,8cyclictest3628386-21/usr/sbin/munin20:15:280
360484499401,8cyclictest3628386-21/usr/sbin/munin20:15:280
360484499401,7cyclictest3696463-21cron23:33:000
360484499401,7cyclictest3683009-21/usr/sbin/munin22:55:130
360484499401,4cyclictest3626349-21/usr/sbin/munin20:10:210
360484499401,4cyclictest3626349-21/usr/sbin/munin20:10:210
360484499401,3cyclictest36378502chrt20:45:010
360484499401,12cyclictest3607187-21df_inode19:15:040
360484499401,10cyclictest3673696-21/usr/sbin/munin22:25:250
3604844994010,26cyclictest3612797-21kernelversion19:30:220
3604856993933,4cyclictest3677597-21kworker/u8:3+events_unbound22:42:273
3604856993928,9cyclictest391ktimers/321:05:153
3604856993928,8cyclictest3619036-21munin-plugin-st19:50:003
3604856993928,7cyclictest0-21swapper/321:00:013
3604848993932,4cyclictest3709242-21cut00:10:191
360484499395,8cyclictest3710038-21grep00:10:280
360484499393,4cyclictest0-21swapper/019:48:350
360484499392,6cyclictest3685058-21df_inode23:00:170
360484499391,9cyclictest3648459-21munin-run21:15:010
360484499391,9cyclictest3610735-21munin-node19:25:130
360484499391,8cyclictest3676636-21meminfo22:35:210
360484499391,8cyclictest3662877-21irqstats21:55:180
360484499391,8cyclictest3662877-21irqstats21:55:180
360484499391,5cyclictest3694498-21/usr/sbin/munin23:25:270
360484499391,5cyclictest3635186-21ntp_states20:35:250
360484499391,11cyclictest3714725-21ntp_states00:25:240
3604844993910,26cyclictest3666686-21/usr/sbin/munin22:05:260
3604856993832,4cyclictest3672396-21kworker/u8:0+events_unbound22:32:223
3604856993829,6cyclictest3707102-21grep00:05:003
3604853993830,4cyclictest0-21swapper/221:05:012
360484499382,9cyclictest3668233-21ntp_states22:10:250
3604844993827,8cyclictest121ktimers/019:55:130
360484499381,9cyclictest3623716-21grep20:00:310
360484499381,8cyclictest3700820-21/usr/sbin/munin23:45:210
360484499381,8cyclictest3700820-21/usr/sbin/munin23:45:210
360484499381,8cyclictest3616043-21fschecks_time19:40:160
360484499381,6cyclictest3682340-21grep22:50:260
360484499381,6cyclictest3652446-21/usr/sbin/munin21:25:180
360484499381,6cyclictest3609607-21ntp_states19:20:240
360484499381,5cyclictest3716424-21ntp_states00:30:240
360484499381,11cyclictest3718128-21ntp_states00:35:240
360484499381,11cyclictest3669927-21ntp_states22:15:230
3604856993731,4cyclictest3698016-21kworker/u8:0+events_unbound00:12:433
3604856993721,4cyclictest131rcu_preempt20:29:203
3604848993726,7cyclictest3685954-21grep23:00:291
3604844993729,4cyclictest0-21swapper/020:20:280
360484499372,7cyclictest3640947-21sshd20:52:030
360484499371,9cyclictest3707431-21df00:05:160
360484499371,9cyclictest3688690-21irqstats23:10:270
360484499371,5cyclictest3678356-21meminfo22:40:220
3604856993625,8cyclictest391ktimers/321:55:123
3604856993625,8cyclictest391ktimers/321:55:123
3604856993620,4cyclictest131rcu_preempt20:32:233
3604856993619,5cyclictest131rcu_preempt20:40:293
3604853993627,4cyclictest0-21swapper/223:57:382
3604848993625,6cyclictest0-21swapper/100:30:271
3604848993618,5cyclictest131rcu_preempt20:25:161
3604848993618,3cyclictest131rcu_preempt23:05:131
360484499363,5cyclictest3698591-21apt-get23:40:160
360484499361,8cyclictest3643118-21munin-run21:00:000
360484499361,8cyclictest3638041-21/usr/sbin/munin20:45:160
360484499361,6cyclictest3705744-21diskstats00:00:170
360484499361,6cyclictest3633127-21fschecks_time20:30:170
3604856993520,4cyclictest131rcu_preempt00:18:203
3604856993519,5cyclictest131rcu_preempt23:25:203
3604856993518,4cyclictest131rcu_preempt19:33:163
3604853993526,6cyclictest3614354-21fschecks_time19:35:182
3604848993525,6cyclictest0-21swapper/120:05:261
3604848993525,5cyclictest0-21swapper/122:20:001
3604856993428,4cyclictest3617214-21kworker/u8:1+events_unbound19:51:573
3604856993416,5cyclictest131rcu_preempt22:15:263
3604856993415,4cyclictest131rcu_preempt22:06:043
3604856993415,11cyclictest131rcu_preempt23:45:163
3604856993415,11cyclictest131rcu_preempt23:45:153
3604856993414,10cyclictest131rcu_preempt21:31:033
3604856993410,14cyclictest40-21ksoftirqd/300:00:013
3604856993410,12cyclictest0-21swapper/319:15:163
3604853993427,4cyclictest0-21swapper/200:20:262
3604848993428,4cyclictest3684400-21kworker/u8:0+events_unbound22:59:411
3604848993428,4cyclictest3637815-21idleruntime-cro20:45:021
3604848993426,4cyclictest0-21swapper/121:20:261
3604848993424,6cyclictest0-21swapper/122:50:261
360485699339,15cyclictest40-21ksoftirqd/321:20:183
3604856993327,4cyclictest3717086-21kworker/u8:2+events_unbound00:32:453
3604856993327,4cyclictest3698016-21kworker/u8:0+events_unbound23:37:363
3604856993325,4cyclictest3629282-21sendmail20:20:003
3604856993325,4cyclictest3629282-21sendmail20:20:003
3604856993320,3cyclictest131rcu_preempt21:45:163
3604856993317,4cyclictest131rcu_preempt20:05:163
3604856993314,4cyclictest131rcu_preempt23:12:263
3604856993311,12cyclictest131rcu_preempt22:25:143
3604853993326,4cyclictest3702983-21unixbench_multi23:50:252
3604853993323,7cyclictest3611760-21unixbench_multi19:25:272
3604853993322,8cyclictest3628857-21grep20:15:312
3604853993322,8cyclictest3628857-21grep20:15:312
360484899333,10cyclictest24-21ksoftirqd/120:15:181
360484899333,10cyclictest24-21ksoftirqd/120:15:181
3604848993325,4cyclictest3664034-21idleruntime-cro22:00:001
3604848993325,4cyclictest3664034-21idleruntime-cro22:00:001
3604848993315,7cyclictest131rcu_preempt21:35:181
360485699327,15cyclictest40-21ksoftirqd/323:45:013
3604856993225,5cyclictest3602715-21kworker/u8:1+events_unbound19:11:493
3604856993224,6cyclictest3710724-21kworker/u8:1+events_unbound00:25:433
3604856993224,4cyclictest0-21swapper/322:35:283
3604856993220,9cyclictest391ktimers/322:45:303
3604856993219,5cyclictest131rcu_preempt19:23:533
3604856993216,5cyclictest131rcu_preempt23:30:253
3604856993215,9cyclictest131rcu_preempt21:10:183
3604856993213,5cyclictest131rcu_preempt20:20:133
3604853993223,4cyclictest0-21swapper/223:00:012
360484899326,11cyclictest131rcu_preempt22:00:301
360484899322,7cyclictest24-21ksoftirqd/100:25:141
3604848993223,6cyclictest231ktimers/123:20:131
3604848993223,5cyclictest0-21swapper/122:20:271
3604848993222,6cyclictest0-21swapper/123:10:341
360484499321,4cyclictest3604877-21grep19:10:000
360485699318,20cyclictest391ktimers/320:36:243
360485699318,14cyclictest40-21ksoftirqd/320:50:013
3604856993120,7cyclictest391ktimers/300:05:283
3604856993117,3cyclictest131rcu_preempt19:26:103
3604856993115,9cyclictest131rcu_preempt22:20:253
3604856993112,11cyclictest131rcu_preempt19:55:143
3604856993111,16cyclictest391ktimers/322:03:193
3604856993110,12cyclictest131rcu_preempt23:15:153
360485399315,5cyclictest131rcu_preempt20:55:142
3604853993124,4cyclictest0-21swapper/219:40:252
3604853993123,4cyclictest0-21swapper/223:02:292
3604853993120,7cyclictest0-21swapper/222:40:282
3604848993121,6cyclictest0-21swapper/119:25:281
3604848993120,4cyclictest131rcu_preempt20:59:511
3604848993118,9cyclictest1-21systemd19:31:511
3604856993021,5cyclictest0-21swapper/320:50:293
3604856993018,8cyclictest1-21systemd21:42:113
3604856993016,6cyclictest3701567-21kworker/u8:1+events_unbound23:53:393
3604856993015,4cyclictest131rcu_preempt21:18:023
3604856993012,9cyclictest131rcu_preempt23:00:053
3604853993022,4cyclictest0-21swapper/200:05:162
3604853993020,6cyclictest0-21swapper/222:15:272
3604853993019,7cyclictest0-21swapper/223:35:292
3604848993020,6cyclictest0-21swapper/121:10:271
3604848993019,7cyclictest0-21swapper/100:15:271
360485699299,11cyclictest40-21ksoftirqd/323:20:213
3604856992914,4cyclictest131rcu_preempt00:20:263
3604856992913,4cyclictest131rcu_preempt21:52:033
3604856992912,5cyclictest131rcu_preempt22:55:173
3604856992912,4cyclictest131rcu_preempt19:40:273
3604853992923,4cyclictest3648087-21kworker/u8:1+events_unbound21:32:102
3604853992923,3cyclictest0-21swapper/200:10:282
3604853992922,4cyclictest3650375-21munin-plugin-st21:20:012
3604853992921,4cyclictest0-21swapper/221:40:172
3604853992921,4cyclictest0-21swapper/219:46:552
3604853992920,5cyclictest0-21swapper/222:20:282
3604853992919,7cyclictest3641200-21auditctl20:55:002
360484899296,5cyclictest131rcu_preempt19:20:151
3604848992922,4cyclictest0-21swapper/123:45:261
3604848992922,4cyclictest0-21swapper/123:45:261
3604848992921,6cyclictest3684400-21kworker/u8:0+events_unbound23:19:561
3604848992919,6cyclictest0-21swapper/121:45:191
3604848992917,7cyclictest0-21swapper/119:35:271
3604848992914,5cyclictest3698016-21kworker/u8:0+events_unbound23:50:431
3604848992913,8cyclictest131rcu_preempt22:40:201
360485699281,7cyclictest0-21swapper/319:35:153
3604853992822,4cyclictest3646768-21ssh21:10:012
3604853992820,5cyclictest3668713-21unixbench_singl22:10:272
3604853992820,5cyclictest0-21swapper/222:50:272
3604853992820,4cyclictest0-21swapper/223:22:342
3604853992819,7cyclictest3604837-21cyclictest00:30:262
3604853992819,6cyclictest3706638-21unixbench_multi00:00:262
3604853992818,6cyclictest0-21swapper/223:45:002
3604853992818,6cyclictest0-21swapper/220:40:002
3604853992817,5cyclictest3606227-21kworker/u8:2+events_unbound19:21:512
360484899289,12cyclictest24-21ksoftirqd/119:45:251
3604848992820,4cyclictest0-21swapper/120:10:201
3604848992820,4cyclictest0-21swapper/120:10:201
3604848992819,6cyclictest0-21swapper/122:10:271
3604848992819,6cyclictest0-21swapper/120:35:181
3604848992811,4cyclictest131rcu_preempt20:30:131
3604853992722,3cyclictest0-21swapper/222:25:142
3604853992721,3cyclictest0-21swapper/220:25:002
3604853992719,5cyclictest3622438-21munin-plugin-st20:00:002
3604853992719,5cyclictest0-21swapper/221:35:262
3604853992718,6cyclictest311ktimers/221:20:132
3604853992717,7cyclictest0-21swapper/219:15:012
3604848992721,4cyclictest3606606-21missed_timers19:15:011
3604848992721,4cyclictest3606227-21kworker/u8:2+events_unbound19:15:111
3604848992720,4cyclictest0-21swapper/121:30:291
3604848992720,3cyclictest0-21swapper/119:50:191
360484899271,9cyclictest24-21ksoftirqd/120:20:131
3604848992719,5cyclictest3691762-21kworker/u8:1+events_unbound23:29:481
3604848992717,7cyclictest3640737-21grep20:50:281
3604848992717,6cyclictest0-21swapper/122:25:261
3604848992715,5cyclictest3710724-21kworker/u8:1+events_unbound00:35:151
360485399268,5cyclictest131rcu_preempt23:45:142
360485399268,5cyclictest131rcu_preempt23:45:132
360485399263,16cyclictest0-21swapper/200:35:152
3604853992618,4cyclictest0-21swapper/223:30:262
3604853992618,4cyclictest0-21swapper/221:12:052
3604853992611,11cyclictest32-21ksoftirqd/219:15:042
3604848992620,4cyclictest3646370-21kworker/u8:0+events_unbound21:07:061
3604848992619,5cyclictest3655274-21kworker/u8:2+events_unbound21:40:211
3604848992619,3cyclictest0-21swapper/100:20:271
3604848992617,7cyclictest3691762-21kworker/u8:1+events_unbound23:35:421
3604848992616,6cyclictest0-21swapper/120:50:011
3604848992615,6cyclictest3650014-21kworker/u8:0+events_unbound21:25:251
3604853992519,3cyclictest0-21swapper/221:50:282
3604853992518,4cyclictest3664013-21munin-plugin-st22:00:012
3604853992518,4cyclictest3664013-21munin-plugin-st22:00:012
3604853992516,6cyclictest3711154-21cat00:15:212
3604853992514,6cyclictest0-21swapper/222:05:272
360484899258,9cyclictest24-21ksoftirqd/119:55:271
360484899258,10cyclictest24-21ksoftirqd/119:40:221
3604848992519,4cyclictest3660520-21kworker/u8:2+events_unbound21:50:181
3604848992518,4cyclictest0-21swapper/123:35:011
3604848992518,4cyclictest0-21swapper/121:00:171
3604848992517,5cyclictest3696289-21kworker/u8:2+events_unbound23:44:201
3604848992516,7cyclictest231ktimers/100:05:131
3604848992510,9cyclictest231ktimers/122:05:141
3604853992417,3cyclictest0-21swapper/223:10:252
3604853992416,5cyclictest0-21swapper/222:05:002
3604853992416,5cyclictest0-21swapper/219:35:012
360485399241,20cyclictest3626173-21awk20:10:192
360485399241,20cyclictest3626173-21awk20:10:192
360484899247,10cyclictest24-21ksoftirqd/123:55:301
360484899247,10cyclictest24-21ksoftirqd/122:35:231
360484899246,11cyclictest24-21ksoftirqd/100:00:191
3604848992410,8cyclictest131rcu_preempt22:31:321
3604853992317,3cyclictest0-21swapper/220:45:012
3604853992314,6cyclictest3715205-21unixbench_singl00:25:282
3604853992313,7cyclictest3677676-21sendmail22:40:002
360485399231,13cyclictest0-21swapper/220:00:172
360485399225,8cyclictest131rcu_preempt23:28:542
3604853992216,4cyclictest3675644-21kworker/u8:2+events_unbound22:48:492
3604853992216,3cyclictest0-21swapper/223:05:182
3604853992214,4cyclictest0-21swapper/223:15:292
3604853992213,5cyclictest0-21swapper/222:30:272
360485399212,8cyclictest131rcu_preempt21:30:002
3604853992015,3cyclictest3659450-21if_eth121:45:202
3604853992010,7cyclictest1-21systemd19:51:552
360485399200,5cyclictest0-21swapper/220:32:342
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional