You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 06:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Thu Mar 26, 2026 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3110740996558,4cyclictest1056-21mta-sts-daemon20:50:282
311074099578,4cyclictest0-21swapper/223:40:242
3110740995748,6cyclictest3272821-21systemd-run23:40:012
3110739995241,4cyclictest0-21swapper/121:45:281
3110739995241,4cyclictest0-21swapper/121:45:281
3110740994939,6cyclictest1051-21mta-sts-daemon21:55:412
3110738994927,17cyclictest151ktimers/020:05:250
3110740994738,5cyclictest421-21dbus-daemon21:25:012
3110739994738,5cyclictest3159431-21ssh21:25:011
3110738994718,24cyclictest3303153-21ssh00:15:260
311074099461,31cyclictest3299076-21sh00:10:282
311074099461,31cyclictest3299076-21sh00:10:282
3110739994538,4cyclictest0-21swapper/122:40:351
3110738994534,6cyclictest12350irq/32-eth022:00:300
3110740994435,5cyclictest3150939-21munin-plugin-st21:15:002
3110740994416,3cyclictest171rcu_preempt22:05:282
3110739994420,19cyclictest1144518-21cups-browsed21:55:321
3110738994432,7cyclictest12350irq/32-eth022:10:260
3110738994420,6cyclictest12450irq/33-eth023:20:290
311074099435,27cyclictest3190320-21ssh22:00:322
3110740994331,7cyclictest3268621-21systemd-run23:35:012
3110739994333,6cyclictest1-21systemd19:30:011
311073899434,31cyclictest3211220-21ssh22:25:290
311074099427,5cyclictest171rcu_preempt00:30:252
3110740994234,4cyclictest3187134-21kworker/u8:6-xprtiod22:45:012
3110740994233,6cyclictest3137739-21systemd-run20:40:012
3110739994234,5cyclictest421-21dbus-daemon19:10:011
3110739994233,5cyclictest3128729-21sh20:10:011
311073899423,28cyclictest3219511-21ssh22:35:280
3110738994225,14cyclictest3235481-21aten_r2power_cu22:55:140
311074099411,35cyclictest36-21ksoftirqd/221:30:242
3110739994133,5cyclictest3115259-21systemd-run19:25:021
3110738994135,3cyclictest0-21swapper/021:10:010
3110738994131,4cyclictest3286025-21fschecks_time23:55:180
3110740994032,5cyclictest3133236-21systemd-run20:25:012
3110740994031,6cyclictest3210017-21systemd-run22:25:012
3110740994027,10cyclictest3203122-21switchtime22:15:362
311074099401,31cyclictest1144518-21cups-browsed22:25:252
3110739994032,5cyclictest3299617-21sed00:10:381
3110739994032,5cyclictest3299617-21sed00:10:381
3110739994032,5cyclictest3264401-21sh23:30:011
3110739994032,5cyclictest3121249-21systemd-run19:45:011
3110739994031,6cyclictest3290374-21kernelversion00:00:191
3110738994032,5cyclictest1144515-21cupsd21:46:110
3110738994032,5cyclictest1144515-21cupsd21:46:110
3110738994031,6cyclictest3299138-21ntp_states00:10:310
3110738994031,6cyclictest3299138-21ntp_states00:10:310
311074099395,28cyclictest3303153-21ssh00:15:262
3110740993933,3cyclictest0-21swapper/223:25:242
311074099393,26cyclictest3289533-21cups-browsed00:05:272
311074099393,26cyclictest3289533-21cups-browsed00:05:272
3110740993930,5cyclictest3235091-21cron22:55:012
3110740993928,7cyclictest1144518-21cups-browsed22:45:332
3110740993926,9cyclictest3290474-21missed_timers00:00:202
311073999393,27cyclictest3236336-21ssh22:55:301
3110740993832,3cyclictest0-21swapper/219:45:012
3110740993831,3cyclictest0-21swapper/223:05:012
3110740993831,3cyclictest0-21swapper/223:05:012
3110740993815,5cyclictest171rcu_preempt20:05:272
311073999384,27cyclictest3240439-21ssh23:00:281
311073999384,27cyclictest3240439-21ssh23:00:281
311073999383,26cyclictest3274016-21ssh23:40:301
3110739993828,7cyclictest3312056-21timerwakeupswit00:25:371
3110739993828,7cyclictest3298028-21latency_hist00:10:021
3110739993828,7cyclictest3298028-21latency_hist00:10:021
3110739993825,10cyclictest3146715-21diskmemload22:25:261
3110738993829,7cyclictest3316466-21sshd00:30:450
3110738993817,13cyclictest151ktimers/000:00:290
311074099376,23cyclictest3307344-21ssh00:20:252
311074099376,21cyclictest659-21snmpd21:45:282
311074099376,21cyclictest659-21snmpd21:45:282
3110740993728,6cyclictest3176402-21latency_hist21:45:022
3110740993727,6cyclictest3320556-21unixbench_singl00:35:362
3110740993726,5cyclictest0-21swapper/200:25:302
3110740993722,2cyclictest171rcu_preempt19:45:232
3110740993722,2cyclictest171rcu_preempt19:45:232
3110739993730,3cyclictest0-21swapper/122:20:011
3110739993730,3cyclictest0-21swapper/100:30:201
3110739993729,4cyclictest0-21swapper/122:45:311
3110739993729,3cyclictest0-21swapper/121:10:001
3110739993727,6cyclictest3270336-21unixbench_singl23:35:371
3110739993726,7cyclictest421-21dbus-daemon22:25:031
3110738993731,4cyclictest3165390-21unixbench_singl21:30:370
311073899372,31cyclictest3282371-21ssh23:50:290
311073899372,28cyclictest3181697-21ssh21:50:270
311073899372,28cyclictest3181697-21ssh21:50:270
3110740993628,4cyclictest3234308-21kworker/u8:13-xprtiod00:00:002
3110740993626,6cyclictest3252510-21fschecks_count23:15:182
3110740993626,5cyclictest0-21swapper/223:50:292
3110740993625,7cyclictest3145737-21fschecks_count21:05:172
3110740993625,4cyclictest0-21swapper/219:50:232
3110740993625,4cyclictest0-21swapper/219:50:232
3110740993623,8cyclictest3236179-21sh22:55:242
311074099362,27cyclictest3152131-21ssh21:15:292
3110739993629,4cyclictest0-21swapper/122:00:291
3110739993628,5cyclictest3218353-21cron22:35:011
3110739993628,5cyclictest3134737-21systemd-run20:30:011
3110739993627,5cyclictest0-21swapper/100:15:281
311073899365,21cyclictest12350irq/32-eth021:25:280
3110738993628,5cyclictest3176351-21cron21:45:010
3110738993626,8cyclictest1-21systemd22:00:010
3110738993624,10cyclictest3152280-21sendmail_mailqu21:15:330
3110738993614,18cyclictest3277492-21/usr/sbin/munin23:45:160
3110740993528,4cyclictest1056-21mta-sts-daemon21:27:412
3110740993528,4cyclictest1055-21mta-sts-daemon23:23:102
3110740993526,3cyclictest0-21swapper/223:45:272
3110740993521,6cyclictest1055-21mta-sts-daemon23:05:382
3110740993521,6cyclictest1055-21mta-sts-daemon23:05:382
3110740993521,3cyclictest171rcu_preempt20:00:282
3110739993527,5cyclictest3133409-21latency_hist20:25:031
311073899353,26cyclictest3248793-21ssh23:10:260
3110738993529,4cyclictest3214841-21sort22:30:170
3110738993526,6cyclictest3205784-21latency_hist22:20:020
3110738993524,8cyclictest3201634-21kworker/u8:1-nfsiod23:05:010
3110738993524,8cyclictest3201634-21kworker/u8:1-nfsiod23:05:010
311073899351,29cyclictest151ktimers/020:25:250
3110740993428,3cyclictest0-21swapper/221:40:012
3110740993425,4cyclictest0-21swapper/222:30:242
3110740993423,7cyclictest3219317-21missed_timers22:35:222
3110739993428,3cyclictest0-21swapper/123:15:271
3110739993428,3cyclictest0-21swapper/122:05:261
3110739993427,3cyclictest0-21swapper/122:10:271
3110739993425,7cyclictest3136226-21kworker/u8:16-xprtiod20:40:351
3110739993425,7cyclictest0-21swapper/121:25:211
3110738993429,3cyclictest3148425-21unixbench_multi21:10:370
3110738993427,5cyclictest3245187-21grep23:05:370
3110738993427,5cyclictest3245187-21grep23:05:370
3110738993425,6cyclictest420-21avahi-daemon00:24:200
3110738993420,11cyclictest420-21avahi-daemon22:46:320
3110740993327,3cyclictest0-21swapper/223:15:022
3110740993325,5cyclictest3181590-21needreboot21:50:232
3110740993325,5cyclictest3181590-21needreboot21:50:232
3110740993324,5cyclictest261-21systemd-journal20:00:002
3110740993320,3cyclictest171rcu_preempt19:30:232
3110739993326,4cyclictest3126746-21processes20:00:321
3110739993325,5cyclictest3155411-21latency_hist21:20:031
3110739993323,7cyclictest1144518-21cups-browsed21:33:291
3110738993326,4cyclictest1055-21mta-sts-daemon22:40:340
3110738993315,11cyclictest3311561-21ssh00:25:260
3110740993225,4cyclictest3167897-21kworker/u8:11-xprtiod22:15:022
3110740993224,5cyclictest3131709-21munin-run20:20:012
3110740993223,6cyclictest439-21systemd-logind21:05:012
3110739993225,4cyclictest0-21swapper/100:20:281
3110739993225,3cyclictest0-21swapper/122:50:311
3110739993224,5cyclictest3320450-21unixbench-2d00:35:361
3110739993224,5cyclictest3169396-21threads21:35:361
3110739993223,6cyclictest3145214-21cron21:05:001
311073999322,23cyclictest3257132-21ssh23:20:241
311073899322,11cyclictest548-21gdbus23:41:010
3110738993214,3cyclictest0-21swapper/020:50:230
311074099317,17cyclictest753-21ntpd19:15:302
311074099313,20cyclictest659-21snmpd20:25:292
3110739993122,6cyclictest420-21avahi-daemon23:50:331
3110739993120,8cyclictest3218903-21cpu22:35:161
3110738993125,3cyclictest1055-21mta-sts-daemon23:25:240
3110738993125,3cyclictest0-21swapper/023:35:350
3110738993122,6cyclictest0-21swapper/021:35:160
311073899311,26cyclictest3207037-21ssh22:20:300
311074099309,16cyclictest1144518-21cups-browsed19:20:302
3110740993021,5cyclictest3118742-21fschecks_time19:35:192
3110739993023,3cyclictest0-21swapper/123:50:021
3110739993022,6cyclictest0-21swapper/121:45:021
3110739993020,7cyclictest3118237-21true19:35:011
3110739993020,6cyclictest420-21avahi-daemon21:10:451
3110739993019,8cyclictest3251834-21sh23:15:001
3110739993018,9cyclictest3244447-21missed_timers23:05:221
3110739993018,9cyclictest3244447-21missed_timers23:05:221
311073899303,20cyclictest3119185-21sendmail-mta19:35:300
3110738993025,2cyclictest0-21swapper/019:30:010
3110738993024,3cyclictest0-21swapper/000:35:160
3110738993023,5cyclictest1144518-21cups-browsed19:42:190
3110740992920,6cyclictest421-21dbus-daemon19:25:162
3110739992922,3cyclictest0-21swapper/123:30:321
311073999292,20cyclictest3286471-21ssh23:55:261
3110739992921,5cyclictest3112211-21cron19:15:011
311073899292,24cyclictest3156300-21ssh21:20:260
311074099284,15cyclictest420-21avahi-daemon20:45:262
3110740992816,3cyclictest171rcu_preempt20:40:222
3110740992810,3cyclictest171rcu_preempt20:30:332
3110739992822,3cyclictest0-21swapper/119:35:251
3110739992820,5cyclictest3143732-21sendmail-msp21:00:011
3110738992823,3cyclictest3130663-21cpuspeed20:15:170
3110738992820,5cyclictest420-21avahi-daemon23:32:240
3110738992817,9cyclictest3139231-21kworker/u8:2+rpciod23:15:300
3110738992815,9cyclictest261-21systemd-journal19:45:550
3110738992815,9cyclictest261-21systemd-journal19:45:550
311073899281,22cyclictest3232025-21ssh22:50:260
3110740992720,4cyclictest57-21kcompactd020:55:252
3110740992713,4cyclictest171rcu_preempt19:10:272
3110739992721,3cyclictest0-21swapper/119:20:011
3110739992719,5cyclictest3136207-21idleruntime-cro20:35:011
3110739992719,5cyclictest1-21systemd21:55:011
3110739992719,5cyclictest1-21systemd21:55:011
3110739992719,5cyclictest0-21swapper/120:00:001
3110738992722,3cyclictest2335380-21kworker/u8:0+rpciod19:25:010
3110738992721,4cyclictest1144518-21cups-browsed20:23:130
3110738992720,5cyclictest421-21dbus-daemon19:35:020
3110738992718,6cyclictest1055-21mta-sts-daemon00:05:230
3110738992718,6cyclictest1055-21mta-sts-daemon00:05:230
3110738992714,10cyclictest0-21swapper/019:20:010
3110740992620,3cyclictest0-21swapper/220:10:292
311073899263,3cyclictest3144671-21ntpq21:00:280
3110738992621,3cyclictest0-21swapper/019:50:340
3110738992621,3cyclictest0-21swapper/019:50:340
3110738992619,5cyclictest421-21dbus-daemon19:10:010
3110738992618,6cyclictest3142895-21latency20:55:210
3110738992616,7cyclictest3197495-21ssh22:10:000
3110739992518,4cyclictest1051-21mta-sts-daemon20:10:251
3110739992515,7cyclictest3131711-21munin-plugin-st20:20:011
3110740992417,4cyclictest3110691-21munin-plugin-st19:10:032
3110739992418,3cyclictest0-21swapper/120:50:361
3110739992416,5cyclictest1144518-21cups-browsed20:38:381
311073899242,14cyclictest151ktimers/020:35:270
3110738992418,4cyclictest3130093-21tail20:10:360
3110739992317,3cyclictest0-21swapper/120:45:171
3110739992317,3cyclictest0-21swapper/119:50:211
3110739992317,3cyclictest0-21swapper/119:50:211
3110738992319,2cyclictest0-21swapper/020:40:170
3110738992318,2cyclictest0-21swapper/019:15:010
3110738992317,3cyclictest0-21swapper/019:55:310
3110738992214,5cyclictest12450irq/33-eth020:45:290
3110738992213,7cyclictest261-21systemd-journal20:33:350
311073999211,4cyclictest3122103-21ntpq19:45:251
311073999211,4cyclictest3122103-21ntpq19:45:251
3110738992116,3cyclictest1055-21mta-sts-daemon20:00:060
311074199163,6cyclictest0-21swapper/321:10:283
311074199162,9cyclictest0-21swapper/321:40:273
311074199162,6cyclictest0-21swapper/323:30:293
311074199162,5cyclictest0-21swapper/321:45:243
311074199162,5cyclictest0-21swapper/321:45:243
311074199161,5cyclictest0-21swapper/322:00:263
311074199152,7cyclictest0-21swapper/322:25:263
311074199151,7cyclictest0-21swapper/323:55:253
311074199151,7cyclictest0-21swapper/321:25:283
311074199151,6cyclictest0-21swapper/322:30:223
311074199151,5cyclictest0-21swapper/319:25:233
311074199144,6cyclictest0-21swapper/323:40:283
311074199143,5cyclictest0-21swapper/321:15:363
311074199142,7cyclictest0-21swapper/321:55:323
311074199142,5cyclictest0-21swapper/320:30:283
311074199142,4cyclictest0-21swapper/320:50:233
311074199141,7cyclictest0-21swapper/321:35:193
311074199141,7cyclictest0-21swapper/300:10:253
311074199141,7cyclictest0-21swapper/300:10:253
311074199141,6cyclictest0-21swapper/320:15:243
311074199141,6cyclictest0-21swapper/320:00:243
311074199141,6cyclictest0-21swapper/319:55:243
311074199140,7cyclictest0-21swapper/320:45:243
311074199132,7cyclictest0-21swapper/322:55:303
311074199131,8cyclictest0-21swapper/322:50:263
311074199131,8cyclictest0-21swapper/300:30:293
311074199131,7cyclictest0-21swapper/322:40:353
311074199131,7cyclictest0-21swapper/322:05:243
311074199131,6cyclictest0-21swapper/320:05:373
311074199131,6cyclictest0-21swapper/319:40:273
311074199131,6cyclictest0-21swapper/300:20:213
311074199131,5cyclictest0-21swapper/319:35:243
311074199131,3cyclictest0-21swapper/321:50:273
311074199131,3cyclictest0-21swapper/321:50:273
311074199122,6cyclictest0-21swapper/323:10:163
311074199122,6cyclictest0-21swapper/322:50:013
311074199122,6cyclictest0-21swapper/322:20:223
311074199122,6cyclictest0-21swapper/321:30:353
311074199122,4cyclictest0-21swapper/319:15:263
311074199121,7cyclictest0-21swapper/323:50:213
311074199121,7cyclictest0-21swapper/323:35:183
311074199121,7cyclictest0-21swapper/300:35:243
311074199121,6cyclictest0-21swapper/323:20:263
311074199121,6cyclictest0-21swapper/322:35:223
311074199121,6cyclictest0-21swapper/321:20:133
311074199121,6cyclictest0-21swapper/319:45:263
311074199121,6cyclictest0-21swapper/319:45:263
311074199121,5cyclictest0-21swapper/319:20:243
311074199112,6cyclictest0-21swapper/323:45:193
311074199112,6cyclictest0-21swapper/323:15:353
311074199112,5cyclictest0-21swapper/322:10:243
311074199112,5cyclictest0-21swapper/319:30:243
311074199112,5cyclictest0-21swapper/300:05:153
311074199112,5cyclictest0-21swapper/300:05:153
311074199112,5cyclictest0-21swapper/300:00:233
311074199111,6cyclictest0-21swapper/322:15:163
311074199111,6cyclictest0-21swapper/320:25:273
311074199102,6cyclictest0-21swapper/323:25:303
311074199102,5cyclictest0-21swapper/321:05:223
311074199102,5cyclictest0-21swapper/319:10:153
311074199102,4cyclictest0-21swapper/320:20:253
311074199101,6cyclictest0-21swapper/323:05:223
311074199101,6cyclictest0-21swapper/323:05:223
311074199101,5cyclictest0-21swapper/320:40:233
311074199101,5cyclictest0-21swapper/300:15:153
31107419993,4cyclictest0-21swapper/320:55:283
31107419992,5cyclictest0-21swapper/319:10:013
31107419991,7cyclictest0-21swapper/323:00:293
31107419991,7cyclictest0-21swapper/323:00:293
31107419991,5cyclictest0-21swapper/320:10:183
31107419991,5cyclictest0-21swapper/319:50:183
31107419991,5cyclictest0-21swapper/319:50:183
31107419991,4cyclictest0-21swapper/300:25:203
31107419990,5cyclictest0-21swapper/321:00:183
31107419981,5cyclictest0-21swapper/320:35:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional