You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 12:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Sun Nov 30, 2025 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3693162995441,10cyclictest121ktimers/023:00:140
3693175995138,8cyclictest3732727-21grep21:00:273
369317599492,3cyclictest3748953-21apt-get21:50:133
3693175994837,7cyclictest391ktimers/322:10:133
3693175994736,7cyclictest3741416-21grep21:25:253
3693175994734,9cyclictest391ktimers/300:15:123
3693171994739,5cyclictest3703926-21cron19:40:002
3693171994634,8cyclictest3806941-21sendmail00:40:002
3693166994637,5cyclictest3738272-21munin-plugin-st21:20:001
3693162994636,6cyclictest3751702-21grep21:55:260
3693175994415,25cyclictest3796210-21tail00:05:263
3693171994431,10cyclictest311ktimers/221:55:152
3693166994433,6cyclictest3802266-21fschecks_time00:25:181
3693162994416,7cyclictest131rcu_preempt21:25:120
369317599433,9cyclictest3745894-21/usr/sbin/munin21:40:213
369317599433,9cyclictest3745894-21/usr/sbin/munin21:40:203
369317599431,9cyclictest3806869-21sshd00:39:313
3693175994312,22cyclictest3733388-21/usr/sbin/munin21:05:233
3693171994334,5cyclictest3801828-21munin-plugin-st00:25:002
3693171994334,5cyclictest3801828-21munin-plugin-st00:24:592
3693175994235,4cyclictest3776010-21sh23:10:013
3693175994232,7cyclictest3786026-21grep23:35:243
369317599421,9cyclictest3773141-21/usr/sbin/munin23:00:193
369317599421,11cyclictest3698436-21users19:20:273
3693171994235,4cyclictest3714169-21sh20:10:012
3693166994235,4cyclictest3782584-21unixbench_multi23:25:281
369317599416,32cyclictest3780640-21/usr/sbin/munin23:20:253
369317599411,9cyclictest3793903-21/usr/sbin/munin00:00:243
369317599411,7cyclictest3761563-21/usr/sbin/munin22:25:223
369317599411,10cyclictest3730812-21/usr/sbin/munin20:55:273
369317599411,10cyclictest3706461-21/usr/sbin/munin19:45:243
3693162994130,6cyclictest0-21swapper/021:50:010
369317599402,5cyclictest3711846-21/usr/sbin/munin20:00:293
369317599401,9cyclictest3792336-21ntp_kernel_pll_23:55:213
369317599401,8cyclictest3707586-21/usr/sbin/munin19:50:143
369317599401,7cyclictest3723062-21/usr/sbin/munin20:35:163
369317599401,6cyclictest3778053-21df_inode23:15:153
369317599401,6cyclictest3700217-21sshd19:26:273
369317599401,5cyclictest3781694-21df_inode23:25:173
369317599401,4cyclictest3790826-21/usr/sbin/munin23:50:263
369317599401,3cyclictest3763160-21/usr/sbin/munin22:30:213
369317599401,10cyclictest3715128-21/usr/sbin/munin20:10:263
369317599401,10cyclictest3715128-21/usr/sbin/munin20:10:263
3693166994033,4cyclictest3734860-21cron21:10:011
369316699402,3cyclictest131rcu_preempt19:15:161
3693162994033,4cyclictest3769559-21fschecks_count22:50:170
369317599393,11cyclictest3728174-21diskstats20:50:173
369317599392,4cyclictest3693775-21memory19:10:213
369317599392,13cyclictest3789406-21tail23:45:273
369317599392,12cyclictest3803096-21grep00:25:273
369317599391,9cyclictest3804192-21/usr/sbin/munin00:30:193
369317599391,9cyclictest3758043-21ntp_states22:15:253
369317599391,9cyclictest3725636-21/usr/sbin/munin20:40:283
369317599391,9cyclictest3718162-21/usr/sbin/munin20:20:193
369317599391,7cyclictest3776237-21/usr/sbin/munin23:10:263
369317599391,5cyclictest3797685-21/usr/sbin/munin00:10:243
369317599391,4cyclictest3695634-21apt-get19:15:163
369317599391,12cyclictest3742702-21ntp_states21:30:213
369317599391,11cyclictest3726691-21irqstats20:45:203
369317599391,11cyclictest3726691-21irqstats20:45:193
3693166993931,4cyclictest0-21swapper/122:20:001
3693162993929,7cyclictest3775571-21unixbench_singl23:05:250
369317599383,3cyclictest3705440-21sshd19:42:413
369317599383,30cyclictest3783384-21df23:30:133
369317599382,6cyclictest3702697-21/usr/sbin/munin19:35:203
369317599381,8cyclictest3771998-21timerandwakeup22:55:263
369317599381,8cyclictest3759646-21memory22:20:203
369317599381,7cyclictest3710405-21cron19:56:003
369317599381,3cyclictest3769940-21/usr/sbin/munin22:50:213
369317599381,12cyclictest3748726-21ls21:50:013
369317599381,11cyclictest3800495-21grep00:20:173
369317599381,11cyclictest3800495-21grep00:20:173
369317599381,10cyclictest3737091-21if_eth021:15:193
369317599381,10cyclictest3735659-21ntp_states21:10:233
3693166993829,5cyclictest3724389-21cron20:40:001
369316299387,4cyclictest131rcu_preempt20:55:140
369317599372,3cyclictest3787376-21/usr/sbin/munin23:40:273
369317599371,9cyclictest53-21kcompactd022:35:043
369317599371,9cyclictest3717115-21tail20:15:263
369317599371,8cyclictest3713611-21grep20:05:283
369317599371,6cyclictest3739503-21/usr/sbin/munin21:20:273
369317599371,4cyclictest3753660-21cron22:03:003
369317599371,4cyclictest3701100-21/usr/sbin/munin19:30:223
369317599371,3cyclictest3754679-21/usr/sbin/munin22:05:253
369317599371,10cyclictest3721792-21ntp_states20:30:243
3693171993729,4cyclictest0-21swapper/200:00:322
3693162993729,6cyclictest3753473-21kworker/u8:0+events_unbound22:18:130
369316299371,3cyclictest131rcu_preempt23:10:120
369316299370,5cyclictest131rcu_preempt21:20:150
369316299370,5cyclictest131rcu_preempt19:40:140
369317599361,9cyclictest3766135-21diskstats22:40:163
369317599361,9cyclictest3750773-21diskstats21:55:163
369317599361,8cyclictest3719869-21kernelversion20:25:203
369317599361,4cyclictest3768119-21/usr/sbin/munin22:45:203
369317599361,4cyclictest3768119-21/usr/sbin/munin22:45:203
369317599361,4cyclictest3744280-21memory21:35:193
3693171993627,6cyclictest3745291-21cron21:40:002
3693171993624,7cyclictest0-21swapper/220:05:002
3693166993629,5cyclictest3787876-21kworker/u8:1+events_unbound23:53:361
369316299362,4cyclictest131rcu_preempt20:40:150
369316299362,4cyclictest0-21swapper/020:35:120
3693171993526,5cyclictest0-21swapper/221:00:182
3693166993527,4cyclictest0-21swapper/123:10:001
3693166993527,4cyclictest0-21swapper/100:05:261
3693171993427,4cyclictest3758951-21cron22:20:012
3693171993426,4cyclictest0-21swapper/223:15:012
3693171993426,4cyclictest0-21swapper/219:10:172
3693166993427,5cyclictest3712092-21kworker/u8:2+events_unbound20:22:501
3693166993426,4cyclictest0-21swapper/122:00:011
3693166993426,4cyclictest0-21swapper/120:05:181
3693166993424,6cyclictest0-21swapper/121:25:251
3693166993417,5cyclictest3762118-21kworker/u8:0+events_unbound23:02:001
3693166993416,11cyclictest3787876-21kworker/u8:1+events_unbound00:03:431
369316299348,3cyclictest131rcu_preempt21:40:000
3693162993426,4cyclictest0-21swapper/020:15:000
3693162993426,4cyclictest0-21swapper/020:15:000
3693162993426,4cyclictest0-21swapper/019:55:270
3693171993325,4cyclictest0-21swapper/222:25:162
3693171993324,5cyclictest0-21swapper/223:00:272
3693166993325,6cyclictest3725944-21kworker/u8:1+events_unbound20:57:561
3693166993311,8cyclictest131rcu_preempt20:50:131
3693171993224,4cyclictest0-21swapper/219:50:272
369316699322,5cyclictest131rcu_preempt21:10:161
3693166993223,4cyclictest0-21swapper/121:35:261
369316299323,9cyclictest11-21ksoftirqd/020:25:130
3693162993225,3cyclictest0-21swapper/020:45:280
3693162993225,3cyclictest0-21swapper/020:45:280
3693162993223,6cyclictest0-21swapper/022:05:000
3693162993222,6cyclictest3787715-21tail23:40:290
3693162993216,6cyclictest3727681-21kworker/u8:2+events_unbound20:53:040
3693162993212,5cyclictest3735078-21kworker/u8:0+events_unbound21:15:220
369316299320,4cyclictest131rcu_preempt19:20:130
3693171993123,4cyclictest0-21swapper/221:50:272
3693171993122,5cyclictest3724409-21munin-plugin-st20:40:012
3693171993119,8cyclictest0-21swapper/222:59:592
369316699317,5cyclictest131rcu_preempt00:30:131
3693166993123,5cyclictest0-21swapper/100:00:001
3693166993123,3cyclictest0-21swapper/119:10:261
369316699312,24cyclictest3777287-21grep23:10:251
3693166993112,7cyclictest0-21swapper/123:45:221
3693162993124,4cyclictest3762370-21latency_hist22:30:000
3693162993112,12cyclictest131rcu_preempt22:40:010
3693171993023,5cyclictest3782949-21kworker/u8:0+events_unbound23:38:322
3693171993022,4cyclictest0-21swapper/223:40:202
3693171993022,4cyclictest0-21swapper/219:55:262
3693171993021,5cyclictest0-21swapper/221:30:162
369316699307,14cyclictest131rcu_preempt22:00:191
3693166993023,5cyclictest3729419-21kworker/u8:0+events_unbound21:02:591
3693166993022,6cyclictest3765562-21kworker/u8:2+events_unbound22:48:211
3693166993022,6cyclictest3765562-21kworker/u8:2+events_unbound22:48:201
3693166993022,6cyclictest3712092-21kworker/u8:2+events_unbound20:01:411
3693166993022,3cyclictest0-21swapper/123:30:141
3693166993021,5cyclictest0-21swapper/122:25:251
3693166993021,4cyclictest0-21swapper/122:35:161
3693166993016,3cyclictest131rcu_preempt22:30:171
369316299303,14cyclictest0-21swapper/000:30:120
3693162993023,4cyclictest0-21swapper/019:15:280
3693162993018,8cyclictest3779435-21sendmail23:20:000
3693162993017,4cyclictest3712092-21kworker/u8:2+events_unbound20:15:120
3693171992921,5cyclictest3757243-21auditctl22:15:002
3693171992921,5cyclictest0-21swapper/221:05:192
3693171992918,7cyclictest3715862-21cron20:15:012
3693171992918,7cyclictest3715862-21cron20:15:012
369316699299,4cyclictest131rcu_preempt23:40:001
369316699297,3cyclictest131rcu_preempt19:35:271
3693166992922,5cyclictest3719277-21kworker/u8:1+events_unbound20:31:351
3693166992920,4cyclictest0-21swapper/119:50:281
3693166992918,8cyclictest3770271-21grep22:50:281
3693166992911,6cyclictest24-21ksoftirqd/122:40:251
369316299298,9cyclictest131rcu_preempt22:30:260
369316299296,5cyclictest131rcu_preempt23:45:130
3693162992922,5cyclictest3722066-21kworker/u8:2+events_unbound20:32:290
3693162992921,4cyclictest0-21swapper/022:10:260
3693162992918,8cyclictest3700520-21sh19:30:000
3693162992915,8cyclictest131rcu_preempt19:10:160
3693162992914,9cyclictest131rcu_preempt23:53:150
3693171992821,4cyclictest3687724-21kworker/u8:2+events_unbound19:32:402
3693171992820,4cyclictest0-21swapper/219:27:352
369316699284,15cyclictest24-21ksoftirqd/120:10:211
369316699284,15cyclictest24-21ksoftirqd/120:10:201
3693166992822,4cyclictest3725944-21kworker/u8:1+events_unbound20:45:341
3693166992822,4cyclictest3725944-21kworker/u8:1+events_unbound20:45:331
369316699282,15cyclictest24-21ksoftirqd/122:12:281
3693166992820,6cyclictest3737971-21kworker/u8:2+events_unbound21:23:261
3693166992820,5cyclictest3799977-21kworker/u8:0+events_unbound00:24:591
3693166992820,5cyclictest3799977-21kworker/u8:0+events_unbound00:24:591
3693166992817,5cyclictest3709270-21kworker/u8:1+events_unbound19:56:091
3693162992823,3cyclictest3755599-21latency_hist22:10:010
3693162992821,4cyclictest3749944-21unixbench_multi21:50:260
3693162992815,7cyclictest121ktimers/019:45:140
3693162992814,6cyclictest3739942-21kworker/u8:0-events_freezable_power_21:31:090
3693171992720,4cyclictest3706096-21fschecks_time19:45:192
3693171992720,4cyclictest0-21swapper/221:00:012
3693171992719,4cyclictest0-21swapper/219:15:202
3693171992718,5cyclictest0-21swapper/220:45:272
3693171992718,5cyclictest0-21swapper/220:45:272
3693171992717,7cyclictest3736588-21cat21:15:002
369316699276,4cyclictest131rcu_preempt20:15:171
3693166992720,5cyclictest3805443-21kworker/u8:0+events_unbound00:38:401
3693166992710,3cyclictest131rcu_preempt20:30:001
369316699270,12cyclictest0-21swapper/119:20:121
369316299279,3cyclictest131rcu_preempt20:05:160
369316299277,13cyclictest131rcu_preempt00:25:270
3693162992721,4cyclictest3780701-21unixbench_singl23:20:250
3693162992721,3cyclictest0-21swapper/019:35:280
3693162992716,7cyclictest0-21swapper/019:30:290
3693162992715,4cyclictest131rcu_preempt00:15:180
3693162992711,9cyclictest131rcu_preempt00:10:000
3693171992619,4cyclictest3793290-21idleruntime-cro00:00:002
3693171992619,4cyclictest0-21swapper/221:45:272
3693171992619,3cyclictest0-21swapper/200:19:592
3693171992618,5cyclictest0-21swapper/223:25:272
3693171992618,4cyclictest0-21swapper/221:30:012
3693171992617,5cyclictest3776004-21sh23:10:012
3693171992616,7cyclictest3802231-21grep00:25:192
3693171992615,7cyclictest0-21swapper/222:35:182
369316699269,5cyclictest131rcu_preempt21:45:011
369316699269,5cyclictest131rcu_preempt21:45:001
369316699266,7cyclictest131rcu_preempt22:05:131
3693166992620,4cyclictest3762118-21kworker/u8:0+events_unbound22:57:321
3693166992619,4cyclictest0-21swapper/121:50:261
369316299266,10cyclictest0-21swapper/000:35:150
369316299262,14cyclictest3688786-21kworker/0:1+events20:23:130
3693162992619,5cyclictest3779225-21kworker/u8:2+events_unbound23:35:130
369316299261,8cyclictest11-21ksoftirqd/021:05:150
3693162992614,9cyclictest215-21dbus-daemon00:13:390
3693171992518,5cyclictest3722066-21kworker/u8:2+events_unbound20:32:542
3693171992518,4cyclictest0-21swapper/222:50:012
3693171992518,4cyclictest0-21swapper/222:50:012
3693171992517,4cyclictest0-21swapper/222:03:092
369316699259,6cyclictest24-21ksoftirqd/123:40:301
369316699259,6cyclictest24-21ksoftirqd/121:30:261
3693166992517,5cyclictest630-21snmpd19:25:511
3693166992517,5cyclictest231ktimers/121:45:141
3693166992517,5cyclictest0-21swapper/119:30:281
369316299258,10cyclictest131rcu_preempt22:20:190
369316299257,10cyclictest131rcu_preempt00:20:180
369316299257,10cyclictest131rcu_preempt00:20:170
369316299256,11cyclictest131rcu_preempt23:30:160
3693162992519,4cyclictest3793102-21kworker/u8:2+events_unbound00:02:370
3693162992519,4cyclictest3765562-21kworker/u8:2+events_unbound22:40:150
3693162992519,3cyclictest0-21swapper/020:00:200
369316299251,20cyclictest0-21swapper/019:53:220
3693171992417,4cyclictest0-21swapper/221:40:282
3693171992417,4cyclictest0-21swapper/221:40:282
3693171992415,6cyclictest0-21swapper/219:20:282
3693171992415,5cyclictest0-21swapper/220:40:192
369316699248,7cyclictest24-21ksoftirqd/100:15:191
3693166992417,5cyclictest3779225-21kworker/u8:2+events_unbound23:20:131
3693166992417,4cyclictest3717550-21kworker/u8:0+events_unbound20:40:281
369316699240,12cyclictest0-21swapper/122:20:241
3693162992417,5cyclictest3789605-21kworker/u8:0+events_unbound23:55:160
369316299241,19cyclictest0-21swapper/023:28:120
369316299241,19cyclictest0-21swapper/021:12:050
369316299241,13cyclictest0-21swapper/021:03:520
369317199237,3cyclictest131rcu_preempt23:15:192
3693171992315,5cyclictest0-21swapper/200:34:592
3693171992315,4cyclictest0-21swapper/222:20:282
3693171992313,7cyclictest3717573-21auditctl20:20:002
369316699238,5cyclictest131rcu_preempt00:10:171
3693166992317,4cyclictest3762118-21kworker/u8:0+events_unbound23:17:311
369316699231,13cyclictest0-21swapper/119:42:401
369316299237,6cyclictest11-21ksoftirqd/022:55:190
369316299231,18cyclictest0-21swapper/021:40:420
369316299231,18cyclictest0-21swapper/021:40:420
3693171992216,3cyclictest0-21swapper/222:05:272
3693171992215,4cyclictest0-21swapper/220:25:252
3693171992215,3cyclictest3738292-21sendmail21:20:002
3693171992214,5cyclictest0-21swapper/223:55:002
3693171992214,5cyclictest0-21swapper/222:40:252
3693171992214,4cyclictest0-21swapper/223:20:252
3693171992213,4cyclictest0-21swapper/223:30:242
3693171992211,8cyclictest3705064-21sed19:40:272
369316699222,5cyclictest0-21swapper/119:45:141
3693171992117,2cyclictest3737971-21kworker/u8:2+events_unbound21:23:032
3693171992113,4cyclictest0-21swapper/222:30:252
369317199211,17cyclictest311ktimers/200:11:592
3693171992110,8cyclictest3789241-21threads23:45:262
3693171992110,7cyclictest0-21swapper/220:50:192
369317199202,13cyclictest3770876-21cron22:54:592
3693171991913,4cyclictest311ktimers/220:20:182
3693171991913,4cyclictest311ktimers/200:07:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional