You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 16:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot5s.osadl.org (updated Tue Aug 26, 2025 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4087499995022,24cyclictest0-21swapper/312:15:253
4087499994940,6cyclictest4153472-21grep10:20:013
4087499994921,24cyclictest0-21swapper/309:00:413
4087499994921,24cyclictest0-21swapper/309:00:413
4087491994943,4cyclictest4180594-21kworker/u8:1+events_unbound11:51:081
4087496994834,11cyclictest311ktimers/208:25:122
4087499994626,17cyclictest4156888-21kthreadd10:30:003
4087487994639,4cyclictest4099506-21unixbench_multi07:40:250
408749999451,9cyclictest4140070-21/usr/sbin/munin09:40:123
408749999451,8cyclictest4157575-21/usr/sbin/munin10:30:263
4087487994538,5cyclictest4190856-21kworker/u8:0+events_unbound12:11:130
4087496994430,10cyclictest311ktimers/211:00:162
408749999432,10cyclictest630-21snmpd11:09:523
408749999431,11cyclictest4162578-21irqstats10:45:193
4087491994319,11cyclictest0-21swapper/107:22:291
4087487994337,4cyclictest4170489-21kworker/u8:3+events_unbound11:16:020
408749999421,9cyclictest4159851-21/usr/sbin/munin10:35:303
408749999421,9cyclictest4100328-21/usr/sbin/munin07:45:163
4087499994214,25cyclictest0-21swapper/311:15:253
408749999421,10cyclictest4891-21/usr/sbin/munin12:30:263
408749999421,10cyclictest4105649-21/usr/sbin/munin08:00:203
408749999420,9cyclictest4163936-21/usr/sbin/munin10:50:263
4087496994235,5cyclictest4096478-21kworker/u8:0+events_unbound07:45:272
4087496994235,4cyclictest4132584-21munin-plugin-st09:20:012
408749999413,8cyclictest4118457-21cron08:37:003
408749999411,9cyclictest4180473-21/usr/sbin/munin11:35:283
408749999411,9cyclictest4150742-21/usr/sbin/munin10:10:203
408749999411,9cyclictest2787-21/usr/sbin/munin12:25:163
408749999411,8cyclictest4165945-21/usr/sbin/munin10:55:183
408749999411,8cyclictest4165945-21/usr/sbin/munin10:55:183
4087499994117,20cyclictest0-21swapper/307:40:023
408749999411,11cyclictest4145287-21df_inode09:55:183
4087496994135,4cyclictest4123614-21kworker/u8:2+events_unbound09:00:412
4087496994135,4cyclictest4123614-21kworker/u8:2+events_unbound09:00:412
4087496994129,5cyclictest4170489-21kworker/u8:3+events_unbound11:11:002
4087491994132,4cyclictest0-21swapper/109:25:001
4087491994131,5cyclictest0-21swapper/108:00:291
4087491994131,4cyclictest0-21swapper/107:45:011
408749999402,6cyclictest4174718-21tr11:20:203
408749999402,6cyclictest4120422-21cron08:45:013
408749999401,9cyclictest4181822-21/usr/sbin/munin11:40:233
408749999401,9cyclictest4171209-21meminfo11:10:223
408749999401,9cyclictest4142222-21memory09:45:213
408749999401,9cyclictest4142222-21memory09:45:203
408749999401,9cyclictest4096086-21grep07:30:253
408749999401,8cyclictest4186920-21/usr/sbin/munin11:55:223
4087499994013,24cyclictest0-21swapper/308:14:103
4087499994013,23cyclictest0-21swapper/309:15:003
4087499994011,3cyclictest4089602-21ntp_states07:15:023
408749999401,12cyclictest4099497-21grep07:40:253
408749999401,10cyclictest4183540-21/usr/sbin/munin11:45:273
408749999401,10cyclictest4138563-21cut09:35:163
4087499994010,4cyclictest4089237-21latency_hist07:15:013
408749699401,16cyclictest0-21swapper/209:05:182
4087491994034,4cyclictest1996-21kworker/u8:1+events_unbound12:31:181
408748799408,6cyclictest131rcu_preempt10:40:150
4087487994032,4cyclictest4117019-21idleruntime-cro08:35:010
4087487994032,4cyclictest4117019-21idleruntime-cro08:35:000
408749999393,6cyclictest4175650-21apt-get11:25:123
408749999393,5cyclictest4103594-21/usr/sbin/munin07:55:243
408749999393,10cyclictest4124278-21apt-get08:55:153
408749999391,9cyclictest4154310-21/usr/sbin/munin10:20:233
408749999391,9cyclictest4143821-21/usr/sbin/munin09:50:173
408749999391,9cyclictest4132110-21tail09:15:263
408749999391,9cyclictest4120772-21df08:45:143
408749999391,9cyclictest4115672-21/usr/sbin/munin08:30:163
408749999391,9cyclictest4115672-21/usr/sbin/munin08:30:163
408749999391,9cyclictest4094217-21/usr/sbin/munin07:25:253
408749999391,6cyclictest630-21snmpd11:04:503
408749999391,6cyclictest4147901-21unixbench_multi10:00:263
408749999391,10cyclictest4192115-21gunzip12:10:273
408749999391,10cyclictest4133039-21apt-get09:20:133
4087499993910,3cyclictest0-21swapper/308:50:243
4087496993933,4cyclictest4143008-21kworker/u8:1+events_unbound09:50:482
4087496993927,8cyclictest0-21swapper/207:15:282
4087496993919,12cyclictest4180594-21kworker/u8:1+events_unbound12:04:322
4087487993933,4cyclictest4113390-21kworker/u8:2+events_unbound08:40:380
4087487993930,6cyclictest4151296-21grep10:10:260
408749999381,9cyclictest4184725-21df_inode11:50:153
408749999381,9cyclictest4148684-21/usr/sbin/munin10:05:153
408749999381,9cyclictest2005-21sh12:21:003
408749999381,8cyclictest4107247-21fschecks_time08:05:193
4087499993811,23cyclictest4103473-21latency_hist07:55:013
408749999381,11cyclictest4190721-21tail12:05:253
408749999381,11cyclictest4190721-21tail12:05:253
4087491993821,4cyclictest131rcu_preempt10:57:341
4087491993821,4cyclictest131rcu_preempt10:57:331
4087491993819,4cyclictest0-21swapper/109:54:551
4087487993832,4cyclictest7453-21cron12:40:000
408749999372,9cyclictest4134851-21df_inode09:25:153
408749999372,7cyclictest4110760-21irqstats08:15:213
408749999372,6cyclictest4179252-21cron11:35:003
408749999371,9cyclictest4127819-21df_inode09:05:193
408749999371,9cyclictest4112239-21/usr/sbin/munin08:20:143
4087499993711,22cyclictest0-21swapper/309:35:013
408749999371,11cyclictest4113961-21diskstats08:25:153
4087496993729,4cyclictest0-21swapper/210:30:222
4087491993731,4cyclictest4166087-21kworker/u8:0+events_unbound11:00:581
4087491993730,5cyclictest4113390-21kworker/u8:2+events_unbound08:35:381
4087491993726,6cyclictest0-21swapper/112:10:001
4087491993726,6cyclictest0-21swapper/112:10:001
4087491993726,4cyclictest131rcu_preempt09:15:171
4087491993720,3cyclictest0-21swapper/111:36:401
408749999361,8cyclictest6427-21meminfo12:35:193
408749999361,8cyclictest4188477-21/usr/sbin/munin12:00:183
408749999361,11cyclictest4160985-21meminfo10:40:203
4087496993624,8cyclictest4173931-21apt11:20:012
408749699360,14cyclictest0-21swapper/209:10:162
4087491993630,4cyclictest4103586-21kworker/u8:2+events_unbound07:55:311
4087491993620,5cyclictest131rcu_preempt11:06:071
4087491993619,10cyclictest131rcu_preempt12:15:231
4087491993618,5cyclictest131rcu_preempt12:20:221
4087491993618,4cyclictest0-21swapper/110:29:461
4087487993629,3cyclictest0-21swapper/009:45:000
4087487993627,5cyclictest0-21swapper/008:05:000
4087487993617,16cyclictest0-21swapper/009:20:010
4087491993529,4cyclictest4103586-21kworker/u8:2+events_unbound08:05:321
4087491993525,5cyclictest0-21swapper/111:16:001
4087491993522,4cyclictest131rcu_preempt08:48:111
4087491993516,12cyclictest131rcu_preempt09:25:211
4087491993516,12cyclictest131rcu_preempt07:50:011
4087491993515,4cyclictest131rcu_preempt07:52:071
4087491993515,10cyclictest131rcu_preempt11:48:231
4087491993514,13cyclictest131rcu_preempt09:05:211
4087487993529,4cyclictest4161945-21kworker/u8:1+events_unbound10:52:420
4087496993423,7cyclictest4487-21tune2fs12:30:202
408749199348,22cyclictest231ktimers/112:10:291
4087491993428,4cyclictest4134349-21kworker/u8:2+events_unbound09:31:411
4087491993424,7cyclictest4158597-21cut10:35:011
4087491993419,4cyclictest131rcu_preempt10:46:001
4087487993423,8cyclictest4186082-21auditctl11:55:000
4087496993325,4cyclictest0-21swapper/208:55:402
4087496993322,6cyclictest0-21swapper/210:55:262
4087496993322,6cyclictest0-21swapper/210:55:262
4087491993327,4cyclictest4146351-21kworker/u8:0+events_unbound10:05:491
4087491993326,5cyclictest68-21kswapd011:25:151
4087491993325,4cyclictest0-21swapper/111:30:171
4087491993320,5cyclictest131rcu_preempt09:45:011
4087491993320,5cyclictest131rcu_preempt07:30:231
4087491993319,4cyclictest131rcu_preempt09:00:171
4087491993319,4cyclictest131rcu_preempt09:00:171
4087491993316,5cyclictest131rcu_preempt11:40:251
4087491993315,4cyclictest131rcu_preempt10:03:481
4087491993315,4cyclictest131rcu_preempt08:51:011
4087491993314,9cyclictest131rcu_preempt10:40:581
4087491993314,9cyclictest131rcu_preempt07:11:011
4087491993314,6cyclictest131rcu_preempt09:55:061
4087491993313,5cyclictest131rcu_preempt08:40:141
4087487993326,4cyclictest0-21swapper/010:00:260
4087487993324,6cyclictest0-21swapper/009:05:210
408749699325,5cyclictest131rcu_preempt08:50:142
4087496993224,4cyclictest0-21swapper/207:55:272
4087496993222,6cyclictest4120006-21grep08:40:262
4087496993213,9cyclictest4127145-21kworker/u8:0+events_unbound09:21:582
4087491993222,6cyclictest0-21swapper/109:15:011
4087491993222,6cyclictest0-21swapper/108:20:261
4087491993221,4cyclictest131rcu_preempt12:38:081
4087491993219,5cyclictest131rcu_preempt08:30:241
4087491993219,5cyclictest131rcu_preempt08:30:241
4087491993217,9cyclictest131rcu_preempt11:13:001
4087491993217,4cyclictest131rcu_preempt08:25:111
4087491993216,5cyclictest131rcu_preempt09:45:211
4087491993216,5cyclictest131rcu_preempt09:45:211
4087491993215,9cyclictest131rcu_preempt08:10:241
4087491993215,10cyclictest131rcu_preempt09:35:181
4087491993214,10cyclictest131rcu_preempt07:19:381
4087491993213,5cyclictest131rcu_preempt07:35:151
4087491993211,5cyclictest131rcu_preempt10:20:181
4087487993224,5cyclictest0-21swapper/007:35:010
4087487993224,4cyclictest0-21swapper/012:20:000
4087487993223,5cyclictest0-21swapper/011:05:000
4087487993221,7cyclictest0-21swapper/007:20:000
4087496993124,5cyclictest4099620-21kworker/u8:1+events_unbound08:00:312
4087496993117,5cyclictest4194236-21kworker/u8:3+events_unbound12:17:102
4087496993116,4cyclictest4096478-21kworker/u8:0+events_unbound07:52:392
4087496993113,5cyclictest4180594-21kworker/u8:1+events_unbound11:49:342
4087496993112,10cyclictest4108121-21kworker/u8:0+events_unbound08:19:432
408749199313,6cyclictest24-21ksoftirqd/110:10:261
4087491993118,4cyclictest131rcu_preempt11:23:051
4087491993118,4cyclictest131rcu_preempt10:36:091
4087491993116,4cyclictest131rcu_preempt10:53:091
4087491993113,10cyclictest131rcu_preempt10:17:181
4087491993110,5cyclictest131rcu_preempt12:00:211
408748799316,4cyclictest131rcu_preempt09:55:160
4087487993124,5cyclictest4187590-21kworker/u8:2+events_unbound12:01:110
4087487993121,6cyclictest4094421-21grep07:25:260
408748799310,4cyclictest131rcu_preempt10:05:140
408749699309,9cyclictest4146351-21kworker/u8:0+events_unbound10:21:512
4087496993023,5cyclictest4134349-21kworker/u8:2+events_unbound09:40:462
4087496993023,4cyclictest4098236-21cron07:40:012
4087496993018,4cyclictest4185832-21kworker/u8:0+events_unbound11:57:222
4087491993016,4cyclictest131rcu_preempt08:55:191
4087491993015,5cyclictest131rcu_preempt08:15:161
4087491993013,4cyclictest131rcu_preempt11:55:201
4087491993010,3cyclictest131rcu_preempt12:25:131
408748799304,4cyclictest131rcu_preempt10:45:140
4087487993024,4cyclictest4190856-21kworker/u8:0+events_unbound12:06:030
4087487993024,4cyclictest4190856-21kworker/u8:0+events_unbound12:06:030
4087487993024,4cyclictest4166087-21kworker/u8:0+events_unbound11:08:210
4087487993024,3cyclictest13650irq/42-30be0000.ethernet11:35:270
4087487993023,3cyclictest0-21swapper/012:30:200
4087487993022,5cyclictest4156475-21unixbench_singl10:25:250
4087487993022,4cyclictest0-21swapper/008:40:000
4087487993018,8cyclictest4160292-21run-parts10:40:000
4087487993018,7cyclictest4166658-21grep10:55:260
4087487993018,7cyclictest4166658-21grep10:55:260
4087496992923,4cyclictest4146351-21kworker/u8:0+events_unbound10:39:422
4087496992921,5cyclictest0-21swapper/208:35:182
4087496992915,4cyclictest4094759-21kworker/u8:2+events_unbound07:29:412
4087496992913,9cyclictest32-21ksoftirqd/211:30:262
4087496992913,10cyclictest32-21ksoftirqd/212:25:172
408749699290,7cyclictest4176267-21cut11:25:262
4087491992914,5cyclictest131rcu_preempt07:25:211
4087487992922,4cyclictest0-21swapper/011:10:280
4087487992922,3cyclictest0-21swapper/011:45:290
4087487992921,4cyclictest0-21swapper/008:15:280
408749699287,7cyclictest131rcu_preempt08:30:192
408749699287,7cyclictest131rcu_preempt08:30:192
408749699287,5cyclictest4148089-21kworker/u8:1+events_unbound10:04:072
4087496992822,4cyclictest4161945-21kworker/u8:1+events_unbound10:46:402
4087496992821,4cyclictest0-21swapper/212:15:002
4087496992820,4cyclictest4093138-21sh07:25:022
4087496992819,6cyclictest4109733-21unixbench_multi08:10:282
4087496992814,7cyclictest32-21ksoftirqd/209:45:172
4087496992814,7cyclictest32-21ksoftirqd/209:45:172
408749699281,13cyclictest0-21swapper/208:20:332
4087496992811,11cyclictest32-21ksoftirqd/209:28:072
408748799283,5cyclictest131rcu_preempt08:10:140
408748799281,9cyclictest4124616-21irqstats08:55:200
4087487992819,6cyclictest0-21swapper/008:05:180
4087487992818,7cyclictest0-21swapper/010:20:010
408749699278,8cyclictest131rcu_preempt11:05:202
408749699278,7cyclictest131rcu_preempt09:55:192
408749699278,11cyclictest199-21systemd-network10:52:492
408749699277,8cyclictest131rcu_preempt07:40:172
4087496992720,5cyclictest4146351-21kworker/u8:0+events_unbound10:25:252
4087496992720,3cyclictest0-21swapper/211:20:252
4087496992720,3cyclictest0-21swapper/210:05:272
4087496992720,3cyclictest0-21swapper/209:40:002
4087496992720,3cyclictest0-21swapper/208:45:242
4087496992718,4cyclictest0-21swapper/211:50:252
4087496992714,5cyclictest4099620-21kworker/u8:1+events_unbound08:09:192
408748799278,3cyclictest131rcu_preempt07:55:010
4087487992721,4cyclictest4156764-21kworker/u8:3+events_unbound10:33:440
4087487992720,4cyclictest0-21swapper/009:20:280
408748799271,3cyclictest131rcu_preempt09:35:000
408748799271,21cyclictest0-21swapper/011:25:290
408749699266,8cyclictest131rcu_preempt10:15:182
4087496992620,4cyclictest4190856-21kworker/u8:0+events_unbound12:24:582
4087496992613,7cyclictest32-21ksoftirqd/210:40:172
4087496992612,6cyclictest32-21ksoftirqd/212:35:252
4087487992620,4cyclictest4194236-21kworker/u8:3+events_unbound12:20:210
4087487992620,4cyclictest4174750-21kworker/u8:1+events_unbound11:24:000
4087487992619,5cyclictest4173333-21kworker/u8:0+events_unbound11:40:180
4087487992618,4cyclictest0-21swapper/007:10:270
4087487992610,6cyclictest131rcu_preempt07:20:150
408749699257,7cyclictest131rcu_preempt11:40:202
4087496992512,7cyclictest32-21ksoftirqd/212:07:442
4087496992512,7cyclictest32-21ksoftirqd/212:07:432
4087487992518,5cyclictest4096478-21kworker/u8:0+events_unbound07:45:210
4087487992517,6cyclictest4190856-21kworker/u8:0+events_unbound12:26:160
4087487992516,5cyclictest0-21swapper/008:47:590
4087487992515,6cyclictest0-21swapper/007:39:290
408748799251,3cyclictest4141773-21apt-get09:45:140
408748799251,3cyclictest4141773-21apt-get09:45:130
408749699246,7cyclictest131rcu_preempt10:10:232
4087496992417,5cyclictest311ktimers/209:30:062
4087496992417,4cyclictest0-21swapper/207:10:192
408748799243,3cyclictest131rcu_preempt08:25:270
4087487992418,4cyclictest4134349-21kworker/u8:2+events_unbound09:26:520
4087487992418,4cyclictest4103586-21kworker/u8:2+events_unbound07:59:260
4087487992418,4cyclictest151rcuc/011:30:260
4087487992418,2cyclictest0-21swapper/008:50:240
4087487992417,5cyclictest0-21swapper/009:35:240
4087487992417,3cyclictest0-21swapper/011:55:540
408748799241,19cyclictest0-21swapper/010:21:240
4087496992317,4cyclictest4094759-21kworker/u8:2+events_unbound07:34:402
408748799238,3cyclictest131rcu_preempt09:00:260
408748799238,3cyclictest131rcu_preempt09:00:260
408748799230,19cyclictest0-21swapper/009:11:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional