You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 11:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Sun Jun 28, 2026 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2986682995343,7cyclictest3026450-21cut21:15:291
2986682994840,5cyclictest3030467-21unixbench-2d21:20:451
2986681994740,3cyclictest0-21swapper/021:45:320
298668399467,24cyclictest428-21mta-sts-daemon21:30:332
2986683994537,5cyclictest3031983-21idleruntime-cro21:25:012
2986681994538,5cyclictest3028853-21ssh21:20:010
2986681994537,6cyclictest1236058-21cupsd23:50:300
2986683994434,4cyclictest0-21swapper/222:10:322
2986683994434,4cyclictest0-21swapper/222:10:322
2986683994434,4cyclictest0-21swapper/222:00:392
2986683994434,4cyclictest0-21swapper/222:00:392
2986682994433,5cyclictest0-21swapper/122:30:311
2986682994432,9cyclictest0-21swapper/123:00:021
298668299435,30cyclictest3116159-21ssh23:40:321
2986682994234,5cyclictest2995669-21idleruntime-cro19:40:011
2986683994130,8cyclictest3102827-21idleruntime-cro23:20:012
2986683994130,8cyclictest3102827-21idleruntime-cro23:20:012
2986682994132,6cyclictest1003-21mta-sts-daemon21:53:281
2986682994132,5cyclictest0-21swapper/121:00:011
2986682994130,7cyclictest421-21dbus-daemon23:14:101
2986682994128,9cyclictest2997982-21needreboot19:45:281
2986681994134,4cyclictest3025896-21latency_hist21:15:020
2986683994022,11cyclictest3045444-21sshd21:45:332
298668399401,32cyclictest3110017-21ssh23:30:322
298668399401,32cyclictest3110017-21ssh23:30:322
298668299408,18cyclictest3067023-21ssh22:20:361
2986682994032,5cyclictest3145925-21ssh00:30:011
2986682994031,6cyclictest3022703-21systemd21:10:021
2986681994034,4cyclictest3093604-21ssh23:05:010
2986681994033,4cyclictest2986686-21systemd-run19:10:010
2986683993932,3cyclictest0-21swapper/220:45:012
2986683993930,6cyclictest3079642-21timerandwakeup22:40:432
2986683993929,6cyclictest0-21swapper/220:15:362
2986682993932,4cyclictest1000-21mta-sts-daemon00:15:381
2986682993932,4cyclictest1000-21mta-sts-daemon00:15:381
2986682993928,7cyclictest420-21avahi-daemon00:08:581
2986682993928,7cyclictest1001-21mta-sts-daemon20:20:391
2986681993930,7cyclictest3097963-21mailstats23:10:390
2986681993928,8cyclictest420-21avahi-daemon22:30:300
298668199391,4cyclictest171rcu_preempt23:20:370
298668199391,4cyclictest171rcu_preempt23:20:370
298668399383,29cyclictest3116233-21ssh23:40:352
2986683993832,3cyclictest0-21swapper/223:00:012
2986683993830,5cyclictest3096706-21systemd-run23:10:022
2986683993827,8cyclictest3127648-21latency_hist00:00:032
298668399382,28cyclictest3067104-21ssh22:20:412
298668299383,26cyclictest3100829-21ssh23:15:401
298668299383,26cyclictest3100829-21ssh23:15:401
2986682993830,5cyclictest3092110-21grep23:00:461
2986682993830,5cyclictest3053564-21munin-run22:00:011
2986682993830,5cyclictest3053564-21munin-run22:00:011
2986682993830,4cyclictest0-21swapper/122:40:361
2986682993828,6cyclictest3076328-21wc22:35:391
2986682993827,6cyclictest0-21swapper/122:50:021
298668299382,22cyclictest3060781-21ssh22:10:331
298668299382,22cyclictest3060781-21ssh22:10:331
2986683993730,4cyclictest1002-21mta-sts-daemon21:11:122
2986683993729,5cyclictest3020196-21fschecks_time21:00:252
298668399371,28cyclictest3070071-21ssh22:25:362
2986682993729,6cyclictest0-21swapper/121:40:031
2986682993728,6cyclictest421-21dbus-daemon23:10:021
2986682993728,4cyclictest0-21swapper/123:45:261
2986682993720,10cyclictest1003-21mta-sts-daemon23:32:591
2986682993720,10cyclictest1003-21mta-sts-daemon23:32:591
2986681993729,6cyclictest3139753-21sendmail00:20:010
2986681993729,6cyclictest3139753-21sendmail00:20:010
2986681993721,11cyclictest12650irq/32-eth023:30:330
2986681993721,11cyclictest12650irq/32-eth023:30:330
2986683993630,3cyclictest0-21swapper/219:30:012
2986682993629,3cyclictest0-21swapper/120:50:331
2986682993627,5cyclictest3136797-21latency_hist00:15:021
298668299362,26cyclictest3033058-21rm21:25:361
2986682993621,10cyclictest3127409-21latency_hist00:00:021
298668299361,26cyclictest3054598-21ssh22:00:331
298668299361,26cyclictest3054598-21ssh22:00:331
2986681993628,5cyclictest3090507-21cron23:00:010
2986681993627,6cyclictest3101078-21mailstats23:15:480
2986681993627,6cyclictest3101078-21mailstats23:15:480
2986683993527,5cyclictest3066094-21latency_hist22:20:022
2986683993524,8cyclictest1-21systemd00:10:342
298668399352,27cyclictest3026862-21ssh21:15:382
2986682993527,5cyclictest3087462-21systemd-run22:55:011
2986682993527,5cyclictest3000155-21cron19:55:011
2986682993526,5cyclictest3115363-21kworker/u8:7-events_unbound00:00:241
2986681993530,3cyclictest694-21cron21:40:010
2986681993528,5cyclictest420-21avahi-daemon22:44:490
2986681993527,5cyclictest3133631-21systemd-run00:10:020
2986681993527,4cyclictest2986680-21cyclictest22:36:100
2986681993524,8cyclictest0-21swapper/022:45:340
2986681993517,12cyclictest12650irq/32-eth021:40:340
298668199351,28cyclictest3060725-21ssh22:10:300
298668199351,28cyclictest3060725-21ssh22:10:300
2986683993425,6cyclictest420-21avahi-daemon23:20:422
2986683993425,6cyclictest420-21avahi-daemon23:20:422
2986683993425,6cyclictest3127484-21cups-browsed00:28:032
2986683993425,6cyclictest1000-21mta-sts-daemon00:30:252
2986683993422,9cyclictest3130055-21sshd00:03:332
298668399342,26cyclictest3033219-21ssh21:25:422
2986682993427,4cyclictest3044398-21kworker/u8:9+xprtiod21:45:021
2986682993426,5cyclictest2991179-21latency_hist19:25:011
2986682993426,3cyclictest0-21swapper/123:25:341
2986682993426,3cyclictest0-21swapper/123:25:341
2986682993424,5cyclictest1000-21mta-sts-daemon23:21:581
2986682993424,5cyclictest1000-21mta-sts-daemon23:21:581
2986682993423,3cyclictest0-21swapper/123:54:361
2986682993413,15cyclictest420-21avahi-daemon21:45:321
2986681993427,5cyclictest3029462-21sort21:20:250
2986683993326,3cyclictest0-21swapper/222:50:392
2986683993326,3cyclictest0-21swapper/222:35:322
2986683993325,4cyclictest1002-21mta-sts-daemon00:39:142
2986683993325,4cyclictest0-21swapper/221:40:342
2986683993322,7cyclictest420-21avahi-daemon20:50:022
298668299335,19cyclictest3147018-21sh00:30:361
298668299333,22cyclictest3023746-21ssh21:10:331
2986681993328,3cyclictest0-21swapper/022:00:020
2986681993328,3cyclictest0-21swapper/022:00:020
2986681993326,3cyclictest0-21swapper/022:25:010
2986681993325,6cyclictest3148823-21sshd00:34:340
2986681993325,5cyclictest151ktimers/019:50:330
2986681993311,14cyclictest151ktimers/023:05:350
2986683993226,3cyclictest0-21swapper/223:50:012
2986683993225,3cyclictest0-21swapper/223:40:012
2986683993224,5cyclictest3041254-21cron21:40:012
2986683993223,6cyclictest1236058-21cupsd23:10:302
2986683993223,5cyclictest3022672-21diskmemload23:25:352
2986683993223,5cyclictest3022672-21diskmemload23:25:352
2986682993225,3cyclictest0-21swapper/121:30:341
2986682993223,6cyclictest1002-21mta-sts-daemon19:10:021
2986682993223,5cyclictest3021180-21grep21:05:011
298668299322,24cyclictest3010095-21sshd20:25:331
2986681993225,5cyclictest3128132-21fschecks_time00:00:270
2986681993225,4cyclictest3084804-21aten_r2power_fr22:50:230
2986681993223,6cyclictest3057616-21needreboot22:05:310
298668399313,22cyclictest3092116-21ssh23:00:462
2986683993124,4cyclictest1003-21mta-sts-daemon23:50:352
298668399312,23cyclictest3131608-21ssh00:05:342
2986682993125,3cyclictest0-21swapper/120:15:011
2986682993124,3cyclictest0-21swapper/100:20:461
2986682993124,3cyclictest0-21swapper/100:20:461
298668299312,22cyclictest3070048-21ssh22:25:351
298668299312,21cyclictest3113149-21ssh23:35:361
2986682993121,7cyclictest1002-21mta-sts-daemon00:37:101
2986682993120,8cyclictest3011568-21ntp_kernel_pll_20:30:341
2986681993125,4cyclictest3115109-21idleruntime-cro23:40:000
2986681993124,4cyclictest1000-21mta-sts-daemon21:54:540
2986681993123,5cyclictest1236063-21cups-browsed21:33:250
298668199312,23cyclictest3140851-21ssh00:20:360
298668199312,23cyclictest3140851-21ssh00:20:360
2986681993119,9cyclictest3127537-21latency_hist00:00:020
2986681993119,8cyclictest3069883-21missed_timers22:25:300
2986683993024,3cyclictest0-21swapper/219:40:442
2986683993023,4cyclictest3053745-21latency_hist22:00:032
2986683993023,4cyclictest3053745-21latency_hist22:00:032
2986683993022,5cyclictest3018491-21aten_r2power_fr20:55:222
2986683993022,5cyclictest3000179-21systemd-run19:55:012
298668299306,5cyclictest271ktimers/119:25:341
298668299305,8cyclictest28-21ksoftirqd/119:10:281
2986682993023,3cyclictest0-21swapper/122:10:021
2986682993022,4cyclictest3007537-21unixbench_multi20:15:491
2986682993022,4cyclictest0-21swapper/120:40:331
298668199304,21cyclictest3144064-21sh00:25:400
2986681993024,3cyclictest3022648-21kworker/u8:3-xprtiod21:10:010
2986681993024,3cyclictest0-21swapper/000:10:240
2986681993022,5cyclictest1236063-21cups-browsed22:16:430
298668199301,27cyclictest0-21swapper/021:00:370
2986683992923,3cyclictest0-21swapper/220:00:012
2986683992922,4cyclictest3003178-21sh20:05:002
2986683992921,5cyclictest3006167-21munin-run20:15:002
298668299298,7cyclictest28-21ksoftirqd/122:15:301
2986682992920,6cyclictest3013663-21sendmail20:40:011
2986682992920,5cyclictest3001829-21latency_hist20:00:021
298668199295,20cyclictest14-21ksoftirqd/023:25:310
298668199295,20cyclictest14-21ksoftirqd/023:25:310
298668199294,16cyclictest151ktimers/020:40:360
2986681992924,2cyclictest0-21swapper/020:10:340
2986681992923,4cyclictest3007662-21latency_hist20:20:000
2986681992914,10cyclictest151ktimers/019:20:340
298668199291,21cyclictest2992059-21modprobe19:25:330
2986683992822,3cyclictest0-21swapper/222:35:012
2986683992820,5cyclictest3009481-21aten_r2power_en20:25:212
2986683992820,5cyclictest3008221-21gltestperf20:20:272
2986683992820,5cyclictest2994135-21sshd19:31:332
2986683992819,6cyclictest3004246-21mailstats20:05:472
2986683992819,6cyclictest3004246-21mailstats20:05:472
298668399281,24cyclictest3140993-21grep00:20:412
298668399281,24cyclictest3140993-21grep00:20:412
2986682992820,5cyclictest2988704-21gltestperf19:15:261
2986682992819,3cyclictest171rcu_preempt20:45:331
298668199283,6cyclictest171rcu_preempt19:15:350
298668199283,18cyclictest3054681-21ssh22:00:370
298668199283,18cyclictest3054681-21ssh22:00:370
2986681992820,5cyclictest3118536-21/usr/sbin/munin23:45:310
2986681992819,3cyclictest3009108-21unixbench_singl20:20:470
298668399274,12cyclictest3082415-21ssh22:45:362
2986683992721,3cyclictest3043895-21kworker/u8:8-xprtiod21:50:262
2986683992721,3cyclictest0-21swapper/220:55:012
2986683992718,7cyclictest3057529-21missed_timers22:05:302
298668399271,21cyclictest3137746-21ssh00:15:352
298668399271,21cyclictest3137746-21ssh00:15:352
298668299274,8cyclictest28-21ksoftirqd/120:05:331
298668299274,8cyclictest28-21ksoftirqd/120:05:331
2986682992714,6cyclictest171rcu_preempt19:30:331
2986682992713,3cyclictest171rcu_preempt19:45:011
2986681992717,8cyclictest151ktimers/020:25:320
2986683992621,2cyclictest0-21swapper/219:40:012
2986683992620,3cyclictest0-21swapper/220:40:012
2986683992619,4cyclictest2282494-21kworker/u8:3-nfsiod19:15:012
298668299269,3cyclictest171rcu_preempt20:00:301
298668199265,3cyclictest171rcu_preempt19:10:340
298668199264,17cyclictest428-21mta-sts-daemon19:40:330
298668199262,5cyclictest171rcu_preempt21:25:410
2986681992620,4cyclictest3016689-21systemd-run20:50:010
2986681992619,5cyclictest750-21ntpd19:35:260
2986681992616,5cyclictest0-21swapper/020:05:450
2986681992616,5cyclictest0-21swapper/020:05:450
2986681992613,3cyclictest0-21swapper/023:40:330
298668399251,11cyclictest3022770-21latency_hist21:10:022
2986683992415,5cyclictest0-21swapper/219:45:302
2986683992412,7cyclictest2989057-21sshd19:15:332
298668399241,17cyclictest1236063-21cups-browsed20:30:382
298668199249,10cyclictest151ktimers/020:50:320
298668199242,16cyclictest2993641-21ntp_states19:30:350
2986681992418,4cyclictest3013229-21sendmail_mailqu20:35:380
2986681992418,3cyclictest0-21swapper/021:00:010
2986681992418,3cyclictest0-21swapper/000:40:010
2986681992315,4cyclictest12650irq/32-eth020:30:220
2986683992215,3cyclictest0-21swapper/219:25:012
2986681992217,2cyclictest0-21swapper/020:00:010
298668199221,18cyclictest2998081-21ntpq19:45:330
2986683992115,2cyclictest0-21swapper/219:10:022
298668199211,14cyclictest3002620-21ntpq20:00:350
298668499182,9cyclictest0-21swapper/322:20:353
298668499182,8cyclictest0-21swapper/321:45:313
298668499182,8cyclictest0-21swapper/321:35:323
298668499181,8cyclictest0-21swapper/323:15:423
298668499181,8cyclictest0-21swapper/323:15:423
298668499181,6cyclictest0-21swapper/319:20:323
298668499173,8cyclictest0-21swapper/323:05:383
298668499173,7cyclictest0-21swapper/322:40:333
298668499172,8cyclictest0-21swapper/321:55:383
298668499172,8cyclictest0-21swapper/321:55:383
298668499171,9cyclictest0-21swapper/322:35:363
298668499161,9cyclictest0-21swapper/323:00:383
298668499161,9cyclictest0-21swapper/300:00:333
298668499161,8cyclictest0-21swapper/320:15:343
298668499161,8cyclictest0-21swapper/300:30:353
298668499161,6cyclictest0-21swapper/319:40:323
298668499153,8cyclictest0-21swapper/323:45:323
298668499153,7cyclictest0-21swapper/300:15:353
298668499153,7cyclictest0-21swapper/300:15:353
298668499152,9cyclictest0-21swapper/323:25:393
298668499152,9cyclictest0-21swapper/323:25:393
298668499152,9cyclictest0-21swapper/300:10:253
298668499152,8cyclictest0-21swapper/321:25:233
298668499152,8cyclictest0-21swapper/319:50:023
298668499152,7cyclictest0-21swapper/322:25:333
298668499152,7cyclictest0-21swapper/322:05:363
298668499152,7cyclictest0-21swapper/319:30:313
298668499152,6cyclictest0-21swapper/321:15:353
298668499152,5cyclictest0-21swapper/320:30:333
298668499151,8cyclictest0-21swapper/323:30:393
298668499151,8cyclictest0-21swapper/323:30:393
298668499151,8cyclictest0-21swapper/300:05:323
298668499151,5cyclictest0-21swapper/323:10:313
298668499142,8cyclictest0-21swapper/322:45:373
298668499142,7cyclictest0-21swapper/322:15:303
298668499142,7cyclictest0-21swapper/321:50:343
298668499142,7cyclictest0-21swapper/321:10:313
298668499141,9cyclictest0-21swapper/323:20:243
298668499141,9cyclictest0-21swapper/323:20:243
298668499141,8cyclictest0-21swapper/323:35:343
298668499141,8cyclictest0-21swapper/322:50:353
298668499141,8cyclictest0-21swapper/322:00:333
298668499141,8cyclictest0-21swapper/322:00:333
298668499141,8cyclictest0-21swapper/321:20:363
298668499141,7cyclictest0-21swapper/323:40:333
298668499141,7cyclictest0-21swapper/322:30:243
298668499141,7cyclictest0-21swapper/320:50:313
298668499141,7cyclictest0-21swapper/320:00:303
298668499133,8cyclictest0-21swapper/319:15:333
298668499133,5cyclictest0-21swapper/319:50:343
298668499132,6cyclictest0-21swapper/323:55:233
298668499132,6cyclictest0-21swapper/321:40:253
298668499132,6cyclictest0-21swapper/300:35:223
298668499132,5cyclictest0-21swapper/321:05:323
298668499131,7cyclictest0-21swapper/320:05:373
298668499131,7cyclictest0-21swapper/320:05:373
298668499131,7cyclictest0-21swapper/319:35:303
298668499131,7cyclictest0-21swapper/319:25:363
298668499131,7cyclictest0-21swapper/319:10:343
298668499131,7cyclictest0-21swapper/300:25:383
298668499131,6cyclictest0-21swapper/320:10:303
298668499123,6cyclictest0-21swapper/322:55:283
298668499122,7cyclictest0-21swapper/300:20:343
298668499122,7cyclictest0-21swapper/300:20:343
298668499122,6cyclictest0-21swapper/319:55:373
298668499122,5cyclictest0-21swapper/322:10:213
298668499122,5cyclictest0-21swapper/322:10:213
298668499122,5cyclictest0-21swapper/320:20:333
298668499121,7cyclictest0-21swapper/320:35:353
298668499121,6cyclictest0-21swapper/320:45:363
298668499121,6cyclictest0-21swapper/320:25:303
298668499121,5cyclictest0-21swapper/323:50:223
298668499120,6cyclictest0-21swapper/320:55:333
298668499113,6cyclictest0-21swapper/321:30:253
298668499112,5cyclictest0-21swapper/320:40:323
298668499111,6cyclictest0-21swapper/321:00:353
29866849970,4cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional