You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 04:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot5s.osadl.org (updated Wed Dec 03, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3352821995641,7cyclictest3393411-21kworker/u8:0+events_unbound21:17:592
3352812995546,5cyclictest3363808-21awk19:40:000
3352824995446,5cyclictest3372547-21munin-plugin-st20:05:003
3352824995244,4cyclictest3445460-21munin-plugin-st23:35:003
3352821995245,5cyclictest3448732-21kworker/u8:2+events_unbound23:48:162
3352812995038,8cyclictest121ktimers/019:30:130
3352816994736,7cyclictest3372137-21grep20:00:271
3352812994737,8cyclictest3462937-21apt00:25:000
3352812994635,8cyclictest3356760-21munin-plugin-st19:20:010
3352812994635,8cyclictest3356760-21munin-plugin-st19:20:000
3352824994337,4cyclictest3461206-21kworker/u8:0+events_unbound00:28:233
3352821994233,6cyclictest299-21systemd-logind19:22:252
3352821994226,4cyclictest131rcu_preempt22:40:152
3352812994232,6cyclictest3436479-21unixbench_multi23:05:260
3352812994231,8cyclictest121ktimers/019:45:130
3352824994133,5cyclictest3407289-21cron21:45:003
3352824994133,5cyclictest3395533-21fschecks_count21:10:163
3352821994126,4cyclictest131rcu_preempt23:20:232
3352824994030,6cyclictest3453993-21idleruntime-cro23:59:593
3352824994029,7cyclictest3374512-21apt-config20:10:003
3352821994025,5cyclictest3367030-21kworker/u8:1-events_freezable_power_20:08:122
3352824993930,6cyclictest3429427-21unixbench_multi22:45:253
3352824993918,11cyclictest40-21ksoftirqd/321:15:123
3352816993932,3cyclictest0-21swapper/123:40:011
3352816993931,4cyclictest3426078-21idleruntime-cro22:40:011
3352816993931,4cyclictest0-21swapper/123:48:161
3352812993913,11cyclictest0-21swapper/020:15:020
3352824993826,6cyclictest0-21swapper/320:15:283
3352824993822,5cyclictest3446986-21kworker/u8:0+events_unbound00:04:043
3352824993730,4cyclictest3418283-21ntp_kernel_err22:15:223
3352824993728,5cyclictest3363808-21idleruntime-cro19:40:003
3352821993715,12cyclictest131rcu_preempt21:25:232
3352821993715,12cyclictest131rcu_preempt20:10:192
3352816993711,4cyclictest34675612grep00:35:251
3352824993628,4cyclictest0-21swapper/321:35:013
3352824993626,6cyclictest3356304-21unixbench_multi19:15:263
3352824993626,6cyclictest3356304-21unixbench_multi19:15:263
3352824993626,5cyclictest0-21swapper/323:50:193
3352824993626,5cyclictest0-21swapper/323:50:193
3352821993627,6cyclictest0-21swapper/222:40:012
3352821993625,7cyclictest0-21swapper/220:25:282
3352821993624,8cyclictest3391238-21grep20:55:242
3352821993622,4cyclictest131rcu_preempt22:25:372
3352821993621,6cyclictest3441747-21kworker/u8:2+events_unbound23:26:442
3352821993618,4cyclictest131rcu_preempt23:00:242
3352821993616,5cyclictest131rcu_preempt00:33:232
3352821993615,12cyclictest131rcu_preempt23:00:022
3352821993615,12cyclictest131rcu_preempt19:10:232
3352821993615,12cyclictest131rcu_preempt19:10:232
3352816993628,5cyclictest3425937-21kworker/u8:1+events_unbound22:58:101
3352816993628,5cyclictest0-21swapper/121:40:001
335281299366,14cyclictest3443750-21latency_hist23:30:000
3352812993624,8cyclictest0-21swapper/020:25:000
335281299361,11cyclictest33775412timerwakeupswit20:15:260
3352824993517,11cyclictest3367030-21kworker/u8:1+events_unbound19:55:493
3352821993524,7cyclictest0-21swapper/221:45:252
3352821993524,7cyclictest0-21swapper/221:45:252
3352821993523,8cyclictest311ktimers/221:20:192
3352821993521,5cyclictest3364229-21kworker/u8:0+events_unbound19:45:332
3352821993520,4cyclictest131rcu_preempt20:33:412
3352821993517,6cyclictest3425937-21kworker/u8:1+events_unbound22:53:382
3352821993516,11cyclictest131rcu_preempt20:15:162
3352821993515,4cyclictest131rcu_preempt22:45:162
3352816993528,4cyclictest3402172-21cron21:30:011
3352816993527,5cyclictest0-21swapper/121:05:161
3352816993523,7cyclictest0-21swapper/122:55:001
335281299359,10cyclictest3394341-21rt-features21:05:240
3352812993524,6cyclictest3387888-21grep20:45:280
3352824993426,6cyclictest3381308-21kworker/u8:0+events_unbound20:36:443
3352824993426,4cyclictest0-21swapper/321:00:003
3352824993423,7cyclictest3441583-21grep23:20:253
3352824993423,7cyclictest0-21swapper/300:20:253
3352824993423,6cyclictest0-21swapper/320:40:173
3352821993427,4cyclictest0-21swapper/221:10:002
3352821993425,6cyclictest311ktimers/200:05:132
3352821993425,6cyclictest311ktimers/200:05:132
3352821993420,4cyclictest131rcu_preempt22:00:182
3352821993416,5cyclictest131rcu_preempt21:55:202
3352821993416,4cyclictest131rcu_preempt20:43:092
3352821993415,11cyclictest3427906-21kworker/u8:0+events_unbound23:18:112
3352821993414,4cyclictest131rcu_preempt19:40:202
3352824993325,4cyclictest0-21swapper/321:00:273
3352824993325,4cyclictest0-21swapper/321:00:263
3352824993325,4cyclictest0-21swapper/320:45:293
3352824993324,7cyclictest3389940-21kworker/u8:2+events_unbound21:09:323
3352824993324,6cyclictest3426052-21munin-plugin-st22:40:003
3352824993324,6cyclictest0-21swapper/323:45:003
3352824993323,6cyclictest0-21swapper/322:00:253
335282199333,8cyclictest32-21ksoftirqd/220:05:012
3352821993328,3cyclictest301rcuc/220:50:142
3352821993327,4cyclictest3379599-21kworker/u8:2+events_unbound20:39:432
3352821993318,4cyclictest131rcu_preempt21:00:262
3352821993318,4cyclictest131rcu_preempt21:00:262
3352821993315,4cyclictest131rcu_preempt19:50:092
3352821993312,13cyclictest131rcu_preempt00:00:232
3352821993311,4cyclictest131rcu_preempt22:10:572
3352821993311,13cyclictest131rcu_preempt22:20:252
3352821993310,15cyclictest0-21swapper/219:55:262
3352812993327,4cyclictest3412403-21munin-run22:00:000
3352812993325,6cyclictest121ktimers/022:50:140
3352812993325,4cyclictest3372615-21latency_hist20:05:010
3352812993323,7cyclictest121ktimers/020:55:120
3352812993322,7cyclictest3384904-21sendmail20:40:010
3352824993224,4cyclictest0-21swapper/320:10:183
3352824993222,6cyclictest0-21swapper/319:50:283
3352824993222,6cyclictest0-21swapper/319:35:013
3352824993221,6cyclictest0-21swapper/322:55:253
335282499320,7cyclictest131rcu_preempt23:45:133
3352821993218,4cyclictest3437140-21kworker/u8:2+events_unbound23:10:302
3352821993216,5cyclictest131rcu_preempt19:35:242
3352821993214,6cyclictest131rcu_preempt23:10:012
3352821993214,4cyclictest131rcu_preempt19:29:052
3352816993221,7cyclictest0-21swapper/121:35:001
3352816993220,8cyclictest24-21ksoftirqd/123:25:111
3352812993226,3cyclictest0-21swapper/020:25:270
3352812993222,6cyclictest0-21swapper/023:40:250
3352812993222,6cyclictest0-21swapper/020:05:270
335281299321,11cyclictest3427232-21mailstats22:40:250
335281299320,19cyclictest0-21swapper/019:10:220
335281299320,19cyclictest0-21swapper/019:10:210
3352824993124,3cyclictest0-21swapper/320:50:183
3352824993121,6cyclictest0-21swapper/300:10:243
3352824993121,6cyclictest0-21swapper/300:10:233
335282199319,11cyclictest32-21ksoftirqd/222:15:272
335282199316,8cyclictest3407439-21latency_hist21:45:012
335282199313,12cyclictest32-21ksoftirqd/221:30:122
3352821993123,4cyclictest0-21swapper/223:30:272
3352821993121,6cyclictest3356749-21sh19:20:002
3352821993121,6cyclictest3356749-21sh19:20:002
3352821993120,7cyclictest311ktimers/221:10:172
3352821993117,5cyclictest131rcu_preempt20:22:042
3352821993112,4cyclictest0-21swapper/221:50:182
3352821993112,11cyclictest131rcu_preempt23:50:242
3352821993112,11cyclictest131rcu_preempt23:50:232
3352821993111,4cyclictest0-21swapper/222:30:182
3352816993121,6cyclictest0-21swapper/123:20:241
3352812993124,4cyclictest3453980-21munin-plugin-st23:59:590
3352824993023,5cyclictest3425937-21kworker/u8:1+events_unbound22:53:443
3352824993015,5cyclictest3356675-21kworker/u8:2+events_unbound19:20:593
335282199309,11cyclictest0-21swapper/220:48:122
335282199309,10cyclictest0-21swapper/223:44:552
335282199308,7cyclictest32-21ksoftirqd/222:10:012
335282199307,14cyclictest32-21ksoftirqd/219:30:172
3352821993023,5cyclictest3464390-21kworker/u8:3+events_unbound00:39:382
3352821993016,4cyclictest131rcu_preempt00:15:282
3352821993013,10cyclictest131rcu_preempt00:10:112
3352821993013,10cyclictest131rcu_preempt00:10:112
3352821993011,11cyclictest131rcu_preempt21:35:212
3352816993022,6cyclictest3423401-21tr22:30:191
3352816993020,7cyclictest3374449-21idleruntime-cro20:10:011
335281299303,10cyclictest0-21swapper/022:08:110
3352812993023,4cyclictest0-21swapper/023:00:000
3352812993022,5cyclictest0-21swapper/023:40:000
3352812993022,4cyclictest0-21swapper/023:50:280
3352812993022,4cyclictest0-21swapper/023:50:270
335281299302,23cyclictest3420935-21cron22:25:010
3352812993018,8cyclictest3405586-21idleruntime-cro21:40:000
3352812993015,10cyclictest0-21swapper/019:50:190
3352824992923,4cyclictest3376118-21kworker/u8:3+events_unbound20:22:383
3352824992922,5cyclictest3458977-21kworker/u8:2+events_unbound00:17:233
3352824992922,5cyclictest3437140-21kworker/u8:2+events_unbound23:10:183
3352824992921,5cyclictest3450560-21kworker/u8:1+events_unbound00:05:023
3352824992921,5cyclictest3450560-21kworker/u8:1+events_unbound00:05:013
3352824992920,5cyclictest0-21swapper/323:15:253
3352824992919,6cyclictest0-21swapper/321:55:263
3352824992918,8cyclictest3422978-21seq22:30:133
3352824992916,5cyclictest3464390-21kworker/u8:3+events_unbound00:31:093
3352821992917,5cyclictest3450560-21kworker/u8:1+events_unbound00:22:352
3352821992913,5cyclictest131rcu_preempt00:28:222
3352821992912,4cyclictest131rcu_preempt23:35:252
3352816992923,4cyclictest3395363-21kworker/u8:3+events_unbound21:10:241
3352816992922,3cyclictest0-21swapper/120:25:281
3352816992922,3cyclictest0-21swapper/119:40:021
3352816992921,4cyclictest0-21swapper/121:00:011
3352816992918,8cyclictest3458561-21cat00:10:221
3352816992918,8cyclictest3458561-21cat00:10:221
3352812992922,4cyclictest0-21swapper/023:20:250
3352812992922,4cyclictest0-21swapper/000:04:590
3352812992921,4cyclictest13650irq/42-30be0000.ethernet00:15:000
3352812992921,4cyclictest13650irq/42-30be0000.ethernet00:15:000
335282499288,6cyclictest0-21swapper/319:25:263
3352824992820,4cyclictest0-21swapper/322:10:163
3352824992820,4cyclictest0-21swapper/319:45:253
3352824992817,5cyclictest3408867-21kworker/u8:2+events_unbound21:50:173
335282499281,5cyclictest131rcu_preempt21:45:133
335282499281,5cyclictest131rcu_preempt21:45:133
335282499280,13cyclictest0-21swapper/323:35:213
3352821992815,5cyclictest131rcu_preempt23:55:152
3352816992821,4cyclictest0-21swapper/100:30:241
3352816992821,4cyclictest0-21swapper/100:15:261
3352816992821,3cyclictest0-21swapper/123:59:591
3352816992820,5cyclictest0-21swapper/119:25:261
3352816992820,4cyclictest0-21swapper/121:50:271
3352812992821,3cyclictest0-21swapper/022:50:010
3352812992820,5cyclictest0-21swapper/020:40:250
3352812992820,4cyclictest0-21swapper/022:15:180
3352812992812,11cyclictest215-21dbus-daemon22:28:070
335281299280,10cyclictest3392034-21cpuspeed21:00:160
335281299280,10cyclictest3392034-21cpuspeed21:00:160
3352824992721,4cyclictest3419085-21kworker/u8:1+events_unbound22:20:253
3352824992717,6cyclictest0-21swapper/321:40:013
3352824992713,5cyclictest3435203-21kworker/u8:1+events_unbound23:05:393
3352816992720,4cyclictest3451820-21unixbench_multi23:50:271
3352816992720,4cyclictest3451820-21unixbench_multi23:50:271
3352816992720,4cyclictest3386597-21cron20:45:001
3352816992720,4cyclictest0-21swapper/123:30:261
3352816992720,4cyclictest0-21swapper/121:15:261
3352816992719,5cyclictest231ktimers/119:15:141
3352816992719,5cyclictest231ktimers/119:15:131
3352816992719,4cyclictest3420941-21sh22:25:001
3352816992718,7cyclictest231ktimers/100:05:061
3352816992718,7cyclictest231ktimers/100:05:051
3352812992720,4cyclictest3390014-21idleruntime-cro20:55:000
3352812992719,4cyclictest0-21swapper/022:38:070
3352812992718,5cyclictest3413695-21grep22:00:250
3352812992717,8cyclictest3369653-21if_eth119:55:180
3352812992717,7cyclictest0-21swapper/022:30:150
3352812992717,6cyclictest0-21swapper/019:25:170
3352812992714,10cyclictest121ktimers/021:20:130
335281299271,21cyclictest3468276-21latency_hist00:40:000
3352812992712,11cyclictest3464795-21date00:30:010
335282499268,11cyclictest40-21ksoftirqd/322:25:263
3352824992620,4cyclictest3415652-21kworker/u8:3+events_unbound22:08:043
3352824992620,4cyclictest3379599-21kworker/u8:2+events_unbound20:25:103
3352816992620,4cyclictest299-21systemd-logind00:01:421
3352816992619,4cyclictest0-21swapper/100:20:161
3352812992622,2cyclictest13650irq/42-30be0000.ethernet00:15:010
3352812992619,4cyclictest0-21swapper/021:30:270
3352812992616,7cyclictest3406834-21grep21:40:270
335282499252,16cyclictest3426764-21basename22:40:173
3352824992517,6cyclictest3401877-21kworker/u8:0+events_unbound21:26:443
3352824992511,8cyclictest131rcu_preempt19:10:233
3352824992511,8cyclictest131rcu_preempt19:10:223
335282499251,16cyclictest3381835-21diskstats20:30:153
335282499251,12cyclictest3434562-21wc23:00:253
335282499251,12cyclictest0-21swapper/300:35:263
3352812992519,3cyclictest0-21swapper/023:45:160
3352812992516,5cyclictest3396377-21sed21:10:250
3352812992516,5cyclictest121ktimers/022:10:150
335282499247,9cyclictest131rcu_preempt21:20:193
335282499245,9cyclictest40-21ksoftirqd/323:25:233
3352816992417,4cyclictest0-21swapper/119:20:271
3352816992416,4cyclictest3440333-21sh23:20:001
335281699241,19cyclictest3365296-21grep19:40:271
3352816992411,9cyclictest215-21dbus-daemon20:22:361
3352812992418,3cyclictest0-21swapper/023:33:000
3352812992416,6cyclictest0-21swapper/023:00:150
3352812992416,4cyclictest3346088-1kworker/0:1H+mmc_complete21:20:000
3352812992415,7cyclictest3438800-21latency_hist23:15:010
335281299241,19cyclictest0-21swapper/021:54:010
335281299241,19cyclictest0-21swapper/000:09:260
335281299241,19cyclictest0-21swapper/000:09:260
335282499238,9cyclictest131rcu_preempt19:42:273
3352816992317,3cyclictest0-21swapper/122:45:221
3352816992317,3cyclictest0-21swapper/122:40:181
3352816992316,4cyclictest0-21swapper/121:00:201
3352816992316,4cyclictest0-21swapper/121:00:191
3352816992315,4cyclictest0-21swapper/122:25:201
3352816992315,4cyclictest0-21swapper/122:08:031
3352812992318,3cyclictest13650irq/42-30be0000.ethernet19:20:200
3352812992317,3cyclictest0-21swapper/021:25:230
3352812992317,3cyclictest0-21swapper/019:40:180
3352812992316,5cyclictest0-21swapper/023:15:150
3352812992314,6cyclictest0-21swapper/000:35:010
335281299231,18cyclictest0-21swapper/021:47:220
335281299231,18cyclictest0-21swapper/021:47:210
3352816992216,4cyclictest3410574-21kworker/u8:0+events_unbound22:13:071
3352816992216,3cyclictest0-21swapper/121:50:011
3352816992216,3cyclictest0-21swapper/121:50:011
3352816992216,3cyclictest0-21swapper/120:55:011
3352816992215,4cyclictest0-21swapper/121:55:251
3352816992214,5cyclictest0-21swapper/120:35:171
3352816992211,8cyclictest3369854-21cat19:55:201
335281699221,13cyclictest0-21swapper/123:04:031
335281699220,12cyclictest0-21swapper/120:31:031
3352812992216,3cyclictest0-21swapper/020:30:160
3352816992115,3cyclictest0-21swapper/123:45:011
3352816992115,3cyclictest0-21swapper/122:15:171
3352816992115,3cyclictest0-21swapper/119:10:181
3352816992115,3cyclictest0-21swapper/119:10:171
3352816992115,3cyclictest0-21swapper/100:25:181
3352816992113,5cyclictest231ktimers/119:52:551
3352816992113,5cyclictest0-21swapper/121:20:181
3352816992113,5cyclictest0-21swapper/120:15:271
3352816992112,6cyclictest0-21swapper/119:35:011
3352816992111,6cyclictest0-21swapper/121:40:271
3352816992014,4cyclictest0-21swapper/122:05:011
3352816992014,3cyclictest0-21swapper/119:45:161
3352816992013,4cyclictest3375884-21unixbench_multi20:10:261
3352816992011,6cyclictest3435613-21fschecks_count23:05:171
3352816991913,3cyclictest0-21swapper/120:45:231
3352816991912,5cyclictest231ktimers/123:10:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional