You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-11 - 15:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot5s.osadl.org (updated Tue Nov 11, 2025 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
335082994715,13cyclictest412014-21/usr/sbin/munin10:50:240
335089994622,20cyclictest0-21swapper/208:05:002
33508999462,11cyclictest367891-21sshd08:40:502
335089994616,3cyclictest0-21swapper/209:35:192
335089994614,4cyclictest339031-21sendmail07:20:012
335089994538,4cyclictest388920-21cron09:45:002
335086994534,7cyclictest349011-21unixbench_multi07:45:271
335082994517,10cyclictest421402-21taskset11:18:320
335089994415,25cyclictest0-21swapper/212:05:262
335089994332,7cyclictest390217-21grep09:45:252
335089994317,4cyclictest349892-21fschecks_time07:50:172
335089994317,4cyclictest349892-21fschecks_time07:50:172
335089994315,4cyclictest368636-21fschecks_count08:45:202
335089994312,6cyclictest421546-21sendmail11:20:012
335082994327,11cyclictest121ktimers/012:30:140
335089994216,3cyclictest0-21swapper/211:20:122
33508999421,12cyclictest411090-21cron10:50:002
335086994234,4cyclictest0-21swapper/107:20:011
335082994215,10cyclictest423458-21latency_hist11:25:010
335093994128,9cyclictest440-21cron12:10:003
335089994111,5cyclictest0-21swapper/207:10:232
335093994033,5cyclictest344006-21kworker/u8:0+events_unbound07:55:423
335089994028,8cyclictest394056-21sendmail10:00:002
33508999401,11cyclictest366540-21sendmail08:40:002
33508299405,19cyclictest0-21swapper/011:00:010
335086993931,5cyclictest357991-21cron08:15:011
335082993929,6cyclictest390172-21grep09:45:240
335093993827,6cyclictest0-21swapper/311:30:003
33509399381,6cyclictest0-21swapper/312:10:123
335089993825,9cyclictest373339-21cron09:00:002
335082993830,4cyclictest0-21swapper/011:10:010
335082993825,8cyclictest121ktimers/010:35:130
335093993729,4cyclictest0-21swapper/309:20:013
335093993727,5cyclictest0-21swapper/307:15:283
335093993724,8cyclictest359726-21sendmail08:20:023
335089993730,4cyclictest0-21swapper/212:35:002
335089993726,7cyclictest342649-21cron07:30:012
335089993726,7cyclictest0-21swapper/207:35:262
335089993726,6cyclictest0-21swapper/210:20:262
335089993715,4cyclictest311ktimers/209:15:152
335086993727,6cyclictest0-21swapper/111:00:001
33508299378,9cyclictest436079-21perl12:00:160
33508299378,12cyclictest381974-21latency_hist09:25:010
335082993725,8cyclictest395984-21apt-get10:05:120
335082993724,9cyclictest121ktimers/009:15:150
335082993723,11cyclictest448411-21/usr/sbin/munin12:35:240
33508999368,7cyclictest1-21systemd10:36:032
335089993628,4cyclictest0-21swapper/208:05:172
33508999362,11cyclictest375060-21cron09:05:002
33508999362,11cyclictest375060-21cron09:05:002
335082993624,8cyclictest350156-21meminfo07:50:200
335082993624,8cyclictest350156-21meminfo07:50:200
335082993622,11cyclictest393635-21/usr/sbin/munin09:55:280
335082993622,11cyclictest366125-21/usr/sbin/munin08:35:260
335082993622,11cyclictest358955-21/usr/sbin/munin08:15:250
335082993622,11cyclictest343419-21/usr/sbin/munin07:30:220
335082993620,12cyclictest431075-21/usr/sbin/munin11:45:210
335082993620,12cyclictest382478-21/usr/sbin/munin09:25:190
335082993619,13cyclictest0-21swapper/012:10:120
335082993616,4cyclictest342662-21munin-run07:30:010
335093993527,4cyclictest376773-21sh09:10:003
335093993527,4cyclictest0-21swapper/312:15:153
335093993525,6cyclictest0-21swapper/308:35:263
335093993524,7cyclictest0-21swapper/312:00:013
335093993523,7cyclictest0-21swapper/307:35:173
335089993526,6cyclictest430284-21auditctl11:45:012
335089993524,7cyclictest0-21swapper/209:25:292
335089993524,7cyclictest0-21swapper/209:10:282
335086993528,3cyclictest0-21swapper/108:30:011
335086993527,5cyclictest0-21swapper/111:20:001
335086993525,6cyclictest0-21swapper/110:00:281
335086993523,7cyclictest0-21swapper/111:30:261
335082993523,9cyclictest340018-21/usr/sbin/munin07:20:230
335082993522,9cyclictest401371-21/usr/sbin/munin10:20:180
335082993520,12cyclictest402943-21/usr/sbin/munin10:25:170
335093993426,3cyclictest0-21swapper/312:00:163
335093993425,5cyclictest0-21swapper/310:25:173
335093993425,4cyclictest0-21swapper/308:10:273
335093993425,4cyclictest0-21swapper/307:25:023
335093993424,6cyclictest354592-21sh08:05:003
335089993426,4cyclictest0-21swapper/210:25:282
33508699347,5cyclictest24-21ksoftirqd/110:25:261
335086993427,4cyclictest0-21swapper/110:50:261
335086993425,6cyclictest0-21swapper/108:25:011
335086993424,6cyclictest0-21swapper/110:10:281
335082993426,5cyclictest378057-21unixbench_singl09:10:270
335082993422,8cyclictest428944-21/usr/sbin/munin11:40:170
335082993420,11cyclictest445099-21/usr/sbin/munin12:25:240
335082993420,11cyclictest399499-21/usr/sbin/munin10:15:180
335082993420,11cyclictest376512-21cron09:07:000
335082993420,11cyclictest357053-21/usr/sbin/munin08:10:230
335082993420,11cyclictest352352-21/usr/sbin/munin07:55:290
335082993419,12cyclictest374270-21/usr/sbin/munin09:00:240
335082993419,12cyclictest374270-21/usr/sbin/munin09:00:230
335082993419,11cyclictest361656-21/usr/sbin/munin08:25:140
33509399333,6cyclictest131rcu_preempt10:10:143
335093993326,4cyclictest361435-21munin-plugin-st08:25:013
335093993322,7cyclictest342650-21cron07:30:013
335089993325,4cyclictest0-21swapper/207:25:012
335089993324,5cyclictest0-21swapper/210:05:252
335089993312,3cyclictest0-21swapper/210:45:012
335086993323,6cyclictest0-21swapper/112:25:251
335086993323,6cyclictest0-21swapper/107:55:001
335086993323,6cyclictest0-21swapper/107:55:001
335086993323,6cyclictest0-21swapper/107:40:301
335086993322,7cyclictest340495-21grep07:20:331
335086993322,6cyclictest0-21swapper/112:30:261
335082993327,3cyclictest440953-21apt-get12:15:120
335082993327,3cyclictest344590-21apt-get07:35:140
335082993326,4cyclictest433919-21apt-get11:55:150
335082993326,4cyclictest384043-21apt-get09:30:160
335082993323,7cyclictest397683-21munin-node10:10:130
335082993323,7cyclictest368203-21sshd08:44:390
335082993323,7cyclictest363905-21/usr/sbin/munin08:30:220
335082993323,7cyclictest360062-21/usr/sbin/munin08:20:150
335082993322,8cyclictest337548-21munin-node07:15:140
335082993322,7cyclictest437998-21meminfo12:05:190
335082993321,8cyclictest394287-21apt-get10:00:140
335082993321,8cyclictest355586-21/usr/sbin/munin08:05:250
335082993320,10cyclictest368691-21/usr/sbin/munin08:45:210
335082993320,10cyclictest336592-21sshd07:13:400
335082993319,8cyclictest409743-21df_inode10:45:150
33509399324,5cyclictest131rcu_preempt12:35:143
33509399323,4cyclictest131rcu_preempt08:25:153
335093993224,4cyclictest0-21swapper/312:30:273
335093993224,4cyclictest0-21swapper/310:20:263
335093993218,4cyclictest393804-21kworker/u8:0+events_unbound10:06:013
335089993224,5cyclictest431992-21cut11:49:592
335086993226,3cyclictest431989-21idleruntime-cro11:50:001
335086993224,4cyclictest0-21swapper/110:50:001
335086993224,4cyclictest0-21swapper/110:05:261
335086993222,6cyclictest0-21swapper/108:55:271
335082993225,4cyclictest427097-21apt-get11:35:130
335082993225,4cyclictest427097-21apt-get11:35:130
335082993222,7cyclictest415479-21/usr/sbin/munin11:00:200
335082993222,7cyclictest388552-21sshd09:40:380
335082993222,7cyclictest348122-21df_inode07:45:170
335082993221,8cyclictest418487-21/usr/sbin/munin11:10:160
335082993221,7cyclictest372327-21/usr/sbin/munin08:55:200
335082993220,9cyclictest370509-21/usr/sbin/munin08:50:160
335082993220,7cyclictest432504-21/usr/sbin/munin11:50:170
335082993219,10cyclictest426238-21/usr/sbin/munin11:30:260
335082993219,10cyclictest424561-21/usr/sbin/munin11:25:270
335082993215,13cyclictest391210-21kernelversion09:50:210
335082993215,13cyclictest391210-21kernelversion09:50:210
33509399312,6cyclictest131rcu_preempt11:35:143
33509399312,6cyclictest131rcu_preempt11:35:133
335093993125,4cyclictest412653-21kworker/u8:2+events_unbound11:31:073
335093993121,4cyclictest0-21swapper/311:00:263
33509399311,5cyclictest131rcu_preempt11:45:133
335089993123,5cyclictest349024-21unixbench_multi07:45:272
335089993121,7cyclictest343090-21tune2fs07:30:192
33508699314,8cyclictest24-21ksoftirqd/109:20:221
335086993123,4cyclictest0-21swapper/112:15:001
335086993123,4cyclictest0-21swapper/111:25:281
335086993122,6cyclictest0-21swapper/108:50:251
335086993121,6cyclictest0-21swapper/112:00:001
335086993118,10cyclictest231ktimers/109:40:131
335086993112,8cyclictest131rcu_preempt09:25:161
335082993120,8cyclictest442785-21df12:20:150
335082993120,7cyclictest404589-21/usr/sbin/munin10:30:140
335082993119,8cyclictest408037-21df10:40:170
335082993116,9cyclictest386733-21tail09:35:310
335093993024,4cyclictest410927-21kworker/u8:1+events_unbound10:51:033
335093993022,4cyclictest0-21swapper/311:15:013
335089993023,4cyclictest0-21swapper/208:30:262
335089993022,4cyclictest0-21swapper/211:10:272
335089993022,4cyclictest0-21swapper/210:30:162
33508999301,10cyclictest3957652awk10:05:012
33508699306,6cyclictest0-21swapper/108:00:241
33508699305,7cyclictest24-21ksoftirqd/110:25:001
335086993020,7cyclictest376298-21grep09:05:261
335086993018,5cyclictest131rcu_preempt09:57:411
335086993017,4cyclictest131rcu_preempt11:07:091
335086993014,9cyclictest131rcu_preempt08:42:191
335082993020,7cyclictest353576-21/usr/sbin/munin08:00:200
335082993019,8cyclictest346431-21diskstats07:40:190
335093992918,8cyclictest336801-21munin-node07:15:013
335089992921,4cyclictest0-21swapper/212:20:272
335089992921,3cyclictest0-21swapper/211:00:182
335089992918,8cyclictest311ktimers/211:30:122
335089992914,10cyclictest435408-21tr12:00:002
33508699298,8cyclictest428035-21timerwakeupswit11:35:251
33508699298,8cyclictest428035-21timerwakeupswit11:35:251
33508699297,5cyclictest0-21swapper/111:40:261
33508699295,8cyclictest24-21ksoftirqd/107:35:171
335086992915,5cyclictest131rcu_preempt09:03:571
335086992915,5cyclictest131rcu_preempt09:03:571
335086992913,4cyclictest131rcu_preempt12:35:471
335093992822,4cyclictest412653-21kworker/u8:2+events_unbound11:19:433
335093992822,4cyclictest405849-21kworker/u8:0+events_unbound10:41:073
335093992821,3cyclictest0-21swapper/307:50:003
335093992819,4cyclictest0-21swapper/309:30:013
33508699285,5cyclictest0-21swapper/107:55:181
33508699284,7cyclictest0-21swapper/107:25:191
335086992821,4cyclictest433694-21munin-plugin-st11:55:001
335086992820,4cyclictest0-21swapper/109:30:291
335086992816,4cyclictest131rcu_preempt12:20:161
335086992815,4cyclictest131rcu_preempt08:31:401
335086992810,4cyclictest131rcu_preempt12:15:121
335093992720,3cyclictest0-21swapper/309:00:003
335093992719,5cyclictest346860-21ntpq07:40:243
335093992718,5cyclictest0-21swapper/308:45:203
33509399271,4cyclictest131rcu_preempt09:30:163
335089992720,4cyclictest0-21swapper/208:00:012
335086992718,6cyclictest231ktimers/111:14:241
335086992717,4cyclictest131rcu_preempt08:48:261
335086992711,4cyclictest131rcu_preempt09:16:481
33509399263,4cyclictest3898432gzip09:45:223
335093992620,4cyclictest433348-21kworker/u8:0+events_unbound12:25:253
335093992618,4cyclictest0-21swapper/309:35:313
335093992616,6cyclictest400876-21sendmail10:20:013
33509399260,4cyclictest131rcu_preempt09:55:143
335089992617,6cyclictest0-21swapper/208:50:262
335089992616,6cyclictest400437-21grep10:15:292
33508699268,4cyclictest131rcu_preempt10:34:151
33508699265,4cyclictest221rcuc/112:00:131
33508699263,6cyclictest24-21ksoftirqd/109:35:281
335086992621,3cyclictest231ktimers/112:09:311
335086992618,6cyclictest410927-21kworker/u8:1+events_unbound11:04:391
33508699261,5cyclictest24-21ksoftirqd/108:05:221
335086992615,7cyclictest0-21swapper/108:20:011
335086992610,9cyclictest131rcu_preempt09:10:181
335093992517,3cyclictest0-21swapper/310:30:263
335093992516,5cyclictest0-21swapper/307:30:173
335093992514,7cyclictest371175-21grep08:50:253
335089992519,4cyclictest311ktimers/209:54:202
335089992519,4cyclictest311ktimers/209:54:202
335089992518,4cyclictest0-21swapper/211:00:002
335089992517,4cyclictest428558-21cron11:40:012
335089992517,4cyclictest428558-21cron11:40:002
335089992517,4cyclictest0-21swapper/208:25:192
33508699254,8cyclictest24-21ksoftirqd/107:30:211
33508699254,11cyclictest407929-21apt-get10:40:131
33508699253,8cyclictest24-21ksoftirqd/109:48:541
33508699252,8cyclictest24-21ksoftirqd/110:15:191
335086992517,5cyclictest0-21swapper/107:10:261
335093992418,4cyclictest412653-21kworker/u8:2+events_unbound11:21:233
335093992416,4cyclictest0-21swapper/309:50:283
335093992416,4cyclictest0-21swapper/309:50:273
335093992415,5cyclictest0-21swapper/311:50:163
335089992418,3cyclictest0-21swapper/207:40:262
335089992417,3cyclictest0-21swapper/212:15:262
335089992416,5cyclictest0-21swapper/212:25:162
335089992416,4cyclictest357990-21cron08:15:012
33508999241,12cyclictest0-21swapper/211:08:422
33508699243,11cyclictest24-21ksoftirqd/109:53:591
33508699243,11cyclictest24-21ksoftirqd/109:53:591
335086992414,7cyclictest231ktimers/108:36:221
335093992317,3cyclictest0-21swapper/311:40:183
335093992317,3cyclictest0-21swapper/308:30:223
335093992316,5cyclictest373100-21kworker/u8:2+events_unbound09:00:563
335093992316,5cyclictest373100-21kworker/u8:2+events_unbound09:00:553
335093992314,5cyclictest391ktimers/308:05:183
335093992314,4cyclictest0-21swapper/308:40:223
335089992317,3cyclictest0-21swapper/211:50:162
335089992316,3cyclictest0-21swapper/208:20:012
335086992311,5cyclictest131rcu_preempt10:35:231
335089992216,3cyclictest0-21swapper/211:25:272
335089992213,4cyclictest0-21swapper/208:20:472
335093992115,4cyclictest400537-21kworker/u8:2+events_unbound10:47:343
335093992115,3cyclictest381899-21idleruntime-cro09:25:013
335093992111,7cyclictest387606-21cut09:40:173
335089992116,3cyclictest385415-21runrttasks09:33:592
335089992115,4cyclictest311ktimers/210:14:562
335089992115,4cyclictest146-21jbd2/mmcblk1p2-812:10:232
33508999210,19cyclictest311ktimers/212:35:132
335093992015,2cyclictest0-21swapper/309:15:003
335093992011,6cyclictest416986-21/usr/sbin/munin11:05:193
335093992011,6cyclictest413959-21tr10:55:293
33509399201,13cyclictest350337-21ps07:50:243
33509399201,13cyclictest350337-21ps07:50:233
33508999202,13cyclictest436880-21head12:00:252
335089992014,4cyclictest311ktimers/210:52:222
335089992010,7cyclictest376391-21seq09:05:372
335093991913,2cyclictest0-21swapper/312:20:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional