You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 09:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sun Mar 22, 2026 00:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2954800997058,8cyclictest3155757-21ssh00:20:012
2954799996458,3cyclictest0-21swapper/122:10:251
2954799995945,10cyclictest3001361-21grep21:20:351
2954800995840,10cyclictest0-21swapper/223:15:272
2954798995644,7cyclictest12350irq/32-eth021:30:240
2954798995232,10cyclictest12350irq/32-eth022:15:230
2954798995127,17cyclictest151ktimers/019:30:230
2954800995036,11cyclictest2983292-21/usr/sbin/munin20:45:022
295480099502,41cyclictest3061885-21ssh22:30:252
295480099502,41cyclictest3061885-21ssh22:30:252
2954798995039,5cyclictest3129943-21cron23:50:010
2954799994839,4cyclictest0-21swapper/123:45:281
2954800994638,5cyclictest2981801-21sendmail20:40:012
2954800994637,6cyclictest3052037-21grep22:20:012
2954799994638,5cyclictest0-21swapper/121:45:261
2954799994638,3cyclictest0-21swapper/100:20:291
2954799994637,6cyclictest421-21dbus-daemon20:30:011
2954799994635,4cyclictest0-21swapper/123:40:251
2954799994633,9cyclictest1055-21mta-sts-daemon21:40:231
2954798994631,13cyclictest1-21systemd21:55:020
2954799994536,6cyclictest2990825-21cron21:10:011
2954798994524,17cyclictest3083835-21ssh22:55:290
295480099444,34cyclictest3131108-21sh23:50:272
2954799994435,5cyclictest0-21swapper/122:30:281
2954799994435,5cyclictest0-21swapper/122:30:281
295479999441,35cyclictest3013940-21ssh21:35:241
2954800994334,7cyclictest3082586-21sh22:55:012
2954799994331,5cyclictest0-21swapper/100:30:291
295479899432,10cyclictest171rcu_preempt20:15:250
295479899432,10cyclictest171rcu_preempt20:15:250
295480099424,15cyclictest36-21ksoftirqd/222:00:242
2954800994234,5cyclictest3034582-21cron22:00:012
2954800994231,7cyclictest516-21in:imuxsock21:05:022
2954800994223,15cyclictest3105374-21ntpq23:20:292
2954799994231,7cyclictest421-21dbus-daemon00:16:491
2954799994230,5cyclictest0-21swapper/121:10:271
295479999422,32cyclictest3048899-21ssh22:15:291
295479999414,27cyclictest3026902-21ssh21:50:231
2954799994136,3cyclictest0-21swapper/121:30:011
2954799994134,3cyclictest0-21swapper/123:35:011
2954799994133,5cyclictest3082588-21systemd-run22:55:011
2954798994135,4cyclictest3138616-21idleruntime-cro00:00:010
2954798994135,4cyclictest3138616-21idleruntime-cro00:00:010
2954798994135,4cyclictest3091269-21cron23:05:010
2954798994128,10cyclictest0-21swapper/023:40:010
2954798994124,11cyclictest12350irq/32-eth021:45:270
2954798994120,15cyclictest151ktimers/000:00:230
295480099404,25cyclictest3026919-21ssh21:50:242
2954800994031,6cyclictest2966802-21systemd-run19:50:012
2954800994031,6cyclictest2966802-21systemd-run19:50:012
2954800994026,10cyclictest3172989-21sh00:40:012
295480099401,32cyclictest3096684-21ssh23:10:212
295479999405,31cyclictest3101058-21ssh23:15:291
295479999405,26cyclictest3139888-21grep00:00:301
2954799994034,3cyclictest0-21swapper/123:20:261
2954799994033,3cyclictest0-21swapper/123:15:011
2954798994032,5cyclictest420-21avahi-daemon23:20:130
295480099393,28cyclictest2990832-21diskmemload21:25:292
2954800993929,6cyclictest3052585-21wc22:20:142
295480099392,27cyclictest3018214-21ssh21:40:222
295480099391,28cyclictest3157066-21ssh00:20:312
295479999394,30cyclictest3083709-21ssh22:55:251
2954799993928,8cyclictest2957118-21ntp_kernel_err19:15:211
2954798993928,6cyclictest12350irq/32-eth000:10:270
295480099384,28cyclictest3165512-21ssh00:30:262
295480099383,27cyclictest3022645-21ssh21:45:262
2954800993830,4cyclictest0-21swapper/200:05:282
2954799993831,4cyclictest3172998-21systemd-run00:40:011
2954799993830,5cyclictest3053534-21switchtime22:20:341
2954799993829,6cyclictest516-21in:imuxsock20:05:001
2954798993832,4cyclictest421-21dbus-daemon00:30:010
2954800993728,6cyclictest548-21gdbus19:10:312
2954800993727,6cyclictest3066771-21unixbench-2d22:35:362
295480099372,30cyclictest3000801-21ssh21:20:262
295479999373,29cyclictest3109612-21ssh23:25:251
2954799993727,6cyclictest2962296-21systemd-run19:35:011
2954799993714,17cyclictest271ktimers/120:15:241
2954799993714,17cyclictest271ktimers/120:15:241
2954798993730,5cyclictest3060761-21systemd-run22:30:020
2954798993730,5cyclictest3060761-21systemd-run22:30:020
2954798993728,6cyclictest439-21systemd-logind23:15:010
295479899372,27cyclictest3152577-21ssh00:15:240
2954798993711,20cyclictest12450irq/33-eth021:40:250
2954800993629,3cyclictest0-21swapper/200:00:012
2954800993629,3cyclictest0-21swapper/200:00:012
2954800993628,5cyclictest3073849-21grep22:45:002
2954800993628,5cyclictest2990832-21diskmemload21:35:122
2954800993623,10cyclictest3122679-21mailstats23:40:302
295480099361,29cyclictest3044373-21ssh22:10:212
2954800993612,18cyclictest3139760-21sh00:00:262
2954799993628,5cyclictest3060745-21munin-run22:30:011
2954799993628,5cyclictest3060745-21munin-run22:30:011
2954799993626,7cyclictest998609-21cups-browsed23:03:541
295480099357,21cyclictest3083626-21ssh22:55:222
295480099353,23cyclictest3040097-21ssh22:05:262
2954800993526,5cyclictest3121327-21idleruntime-cro23:40:012
2954800993521,6cyclictest1054-21mta-sts-daemon00:10:202
295479999353,25cyclictest3092323-21ssh23:05:211
2954799993527,5cyclictest3069883-21aten_r2power_en22:40:121
2954799993527,5cyclictest2974282-21cut20:15:001
2954799993527,5cyclictest2967303-21fschecks_time19:50:161
2954798993530,3cyclictest0-21swapper/023:20:010
295479899352,30cyclictest3113979-21ssh23:30:290
2954798993519,12cyclictest151ktimers/022:05:290
2954800993425,6cyclictest3126170-21aten_r2power_vo23:45:142
2954800993425,5cyclictest2999629-21sh21:20:012
2954800993425,5cyclictest2999629-21sh21:20:012
2954800993422,9cyclictest3161659-21wc00:25:342
2954800993422,9cyclictest2990202-21ntp_kernel_pll_21:05:242
2954800993414,3cyclictest171rcu_preempt20:15:212
2954800993414,3cyclictest171rcu_preempt20:15:212
2954799993429,2cyclictest0-21swapper/100:00:011
2954799993429,2cyclictest0-21swapper/100:00:011
2954799993427,5cyclictest2977314-21systemd-run20:25:011
2954799993426,3cyclictest0-21swapper/121:55:271
2954799993424,7cyclictest2981783-21sendmail20:40:011
2954799993424,4cyclictest0-21swapper/122:35:261
295479999341,29cyclictest3074859-21ssh22:45:181
295479999341,28cyclictest3144014-21ssh00:05:241
295479999341,27cyclictest3118136-21ssh23:35:241
295479899343,26cyclictest3035701-21ssh22:00:240
2954798993426,6cyclictest2968282-21cut19:55:010
295479899342,24cyclictest3031356-21ssh21:55:250
295480099334,21cyclictest3009554-21ssh21:30:262
2954800993323,6cyclictest420-21avahi-daemon19:25:012
295480099332,24cyclictest3057628-21ssh22:25:282
295480099332,24cyclictest3057628-21ssh22:25:282
2954800993318,3cyclictest171rcu_preempt23:30:342
295480099331,22cyclictest3092484-21sh23:05:262
2954799993326,3cyclictest0-21swapper/100:10:261
2954799993325,5cyclictest2989361-21systemd-run21:05:011
2954798993329,2cyclictest0-21swapper/019:20:010
2954798993328,2cyclictest0-21swapper/019:10:010
2954798993325,6cyclictest3002285-21sh21:22:030
2954798993323,7cyclictest3156208-21aten_r2power_en00:20:140
2954798993323,7cyclictest12350irq/32-eth021:28:010
2954798993323,2cyclictest171rcu_preempt23:40:260
295479899332,26cyclictest3092352-21ssh23:05:220
295479899331,28cyclictest3169785-21ssh00:35:250
295479899331,25cyclictest3066300-21rm22:35:280
2954800993225,4cyclictest0-21swapper/220:05:002
2954800993221,7cyclictest0-21swapper/223:00:232
2954800993220,9cyclictest2979593-21needreboot20:30:212
2954800993218,3cyclictest171rcu_preempt21:10:232
2954800993212,17cyclictest351ktimers/223:25:242
295479999325,18cyclictest428-21mta-sts-daemon23:50:291
295479999324,19cyclictest428-21mta-sts-daemon19:10:281
2954799993224,5cyclictest3039654-21fschecks_time22:05:181
2954799993224,4cyclictest0-21swapper/100:25:231
295479999321,27cyclictest3035804-21ssh22:00:281
2954798993226,4cyclictest3052692-21ssh22:20:160
2954798993226,3cyclictest0-21swapper/023:50:240
2954798993225,5cyclictest2959317-21systemd-run19:25:010
2954798993224,5cyclictest2976320-21fschecks_count20:20:150
2954798993224,4cyclictest2954797-21cyclictest21:05:290
2954798993218,9cyclictest12350irq/32-eth022:40:240
2954798993215,10cyclictest12350irq/32-eth021:35:200
2954798993214,13cyclictest12350irq/32-eth022:45:180
2954800993123,4cyclictest1054-21mta-sts-daemon20:05:172
2954800993123,4cyclictest1054-21mta-sts-daemon20:05:172
2954800993118,7cyclictest171rcu_preempt22:45:122
2954800993114,5cyclictest36-21ksoftirqd/219:35:262
2954800993110,6cyclictest36-21ksoftirqd/220:10:232
2954798993124,5cyclictest2956146-21tail19:10:340
2954798993122,7cyclictest3166035-21unixbench-2d00:30:360
2954798993120,6cyclictest12350irq/32-eth000:05:240
2954800993021,7cyclictest2961624-21ntp_kernel_err19:30:212
2954800993013,5cyclictest171rcu_preempt19:50:222
2954800993011,3cyclictest171rcu_preempt20:20:242
2954799993025,2cyclictest0-21swapper/121:30:181
2954799993022,5cyclictest2968543-21aten_r2power_cu19:55:121
2954799993022,5cyclictest2964094-21aten_r2power_en19:40:121
2954799993022,5cyclictest2964094-21aten_r2power_en19:40:121
295479999302,21cyclictest2996312-21ssh21:15:221
295479999302,21cyclictest2996312-21ssh21:15:221
295479899302,22cyclictest659-21snmpd20:35:250
295480099298,8cyclictest36-21ksoftirqd/219:30:012
295480099295,8cyclictest36-21ksoftirqd/220:25:242
2954799992922,4cyclictest2987544-21timerwakeupswit20:55:341
2954799992919,7cyclictest2966161-21ntp_kernel_pll_19:45:221
2954799992919,7cyclictest2966161-21ntp_kernel_pll_19:45:221
295479899292,22cyclictest3044567-21ssh22:10:280
295480099285,7cyclictest36-21ksoftirqd/219:55:212
2954800992821,4cyclictest2985235-21cpuspeed20:50:132
2954799992822,3cyclictest0-21swapper/119:30:011
2954799992820,5cyclictest2972801-21systemd-run20:10:011
2954799992820,5cyclictest2972801-21systemd-run20:10:011
2954799992819,6cyclictest439-21systemd-logind19:25:011
2954799992819,5cyclictest2952088-21kworker/u8:5-nfsiod19:10:011
2954799992818,7cyclictest2983227-21unixbench_singl20:40:321
2954798992822,4cyclictest3079692-21cat22:50:340
2954798992822,4cyclictest2959643-21aten_r2power_po19:25:140
295479899282,23cyclictest3109516-21ssh23:25:220
2954798992822,3cyclictest2954805-21kworker/u8:6-xprtiod20:40:260
2954798992822,3cyclictest0-21swapper/019:40:160
2954798992822,3cyclictest0-21swapper/019:40:160
295480099279,6cyclictest2943639-21kworker/u8:8+xprtiod19:15:162
295480099273,7cyclictest36-21ksoftirqd/220:45:242
2954800992714,7cyclictest171rcu_preempt20:55:162
2954799992719,5cyclictest420-21avahi-daemon20:51:471
2954798992722,3cyclictest2988119-21telnet21:00:130
295479899272,20cyclictest2992036-21ssh21:10:270
2954798992721,3cyclictest0-21swapper/020:10:000
2954798992721,3cyclictest0-21swapper/020:10:000
2954798992720,4cyclictest3007255-21kworker/u8:0-nfsiod22:35:020
2954798992720,4cyclictest3007255-21kworker/u8:0-nfsiod22:35:020
2954800992611,3cyclictest171rcu_preempt19:40:272
2954800992611,3cyclictest171rcu_preempt19:40:272
295479999262,19cyclictest659-21snmpd20:30:241
2954799992619,4cyclictest2963467-21switchtime19:35:341
2954798992621,2cyclictest0-21swapper/019:55:180
295479899261,20cyclictest2996250-21ssh21:15:200
295479899261,20cyclictest2996250-21ssh21:15:200
2954800992516,3cyclictest171rcu_preempt19:10:012
295479899249,4cyclictest12450irq/33-eth020:50:130
295479899249,11cyclictest151ktimers/020:00:230
295479899247,13cyclictest420-21avahi-daemon20:25:220
2954798992418,4cyclictest2974161-21unixbench_multi20:10:340
2954798992417,5cyclictest0-21swapper/020:30:240
2954798992415,6cyclictest998609-21cups-browsed20:45:220
2954798992410,4cyclictest12450irq/33-eth020:55:220
2954799992212,7cyclictest2984184-21ntpq20:45:221
2954798992210,9cyclictest421-21dbus-daemon19:45:130
2954798992210,9cyclictest421-21dbus-daemon19:45:130
295479899211,15cyclictest2963130-21ntpq19:35:230
295480199172,9cyclictest0-21swapper/323:45:303
295480199171,7cyclictest0-21swapper/322:30:243
295480199171,7cyclictest0-21swapper/322:30:243
295480199171,7cyclictest0-21swapper/321:50:253
295480199162,7cyclictest0-21swapper/322:15:303
295480199162,4cyclictest0-21swapper/320:20:203
295480199161,7cyclictest0-21swapper/323:30:253
295480199161,7cyclictest0-21swapper/320:30:243
295480199154,6cyclictest0-21swapper/319:10:283
295480199153,7cyclictest0-21swapper/322:20:233
295480199153,6cyclictest0-21swapper/323:20:263
295480199153,5cyclictest0-21swapper/321:15:243
295480199153,5cyclictest0-21swapper/321:15:243
295480199152,7cyclictest0-21swapper/322:55:293
295480199152,5cyclictest0-21swapper/321:05:243
295480199151,7cyclictest0-21swapper/322:25:243
295480199151,7cyclictest0-21swapper/322:25:243
295480199151,7cyclictest0-21swapper/321:40:233
295480199151,6cyclictest0-21swapper/319:30:273
295480199151,6cyclictest0-21swapper/300:00:233
295480199142,5cyclictest0-21swapper/320:50:253
295480199141,8cyclictest0-21swapper/323:25:223
295480199141,8cyclictest0-21swapper/300:25:263
295480199141,7cyclictest0-21swapper/300:05:293
295480199141,6cyclictest0-21swapper/323:50:263
295480199141,6cyclictest0-21swapper/323:10:293
295480199141,6cyclictest0-21swapper/320:05:293
295480199141,6cyclictest0-21swapper/320:05:293
295480199141,6cyclictest0-21swapper/300:35:243
295480199141,3cyclictest0-21swapper/320:00:233
295480199140,7cyclictest0-21swapper/323:55:223
295480199140,7cyclictest0-21swapper/323:55:223
295480199133,4cyclictest0-21swapper/321:55:243
295480199132,8cyclictest0-21swapper/321:20:243
295480199132,6cyclictest0-21swapper/323:40:013
295480199132,6cyclictest0-21swapper/322:50:263
295480199132,6cyclictest0-21swapper/300:15:293
295480199132,5cyclictest0-21swapper/323:05:223
295480199132,5cyclictest0-21swapper/323:00:243
295480199131,7cyclictest0-21swapper/321:10:253
295480199131,7cyclictest0-21swapper/300:10:243
295480199131,6cyclictest0-21swapper/322:00:243
295480199131,6cyclictest0-21swapper/320:10:223
295480199131,5cyclictest0-21swapper/321:00:213
295480199131,5cyclictest0-21swapper/320:55:273
295480199131,5cyclictest0-21swapper/320:35:213
295480199131,5cyclictest0-21swapper/319:40:213
295480199131,5cyclictest0-21swapper/319:40:213
295480199123,5cyclictest0-21swapper/321:30:253
295480199122,5cyclictest0-21swapper/319:45:163
295480199122,5cyclictest0-21swapper/319:45:163
295480199121,5cyclictest0-21swapper/322:10:033
295480199121,5cyclictest0-21swapper/320:40:193
295480199121,5cyclictest0-21swapper/320:15:233
295480199121,5cyclictest0-21swapper/320:15:233
295480199112,6cyclictest0-21swapper/321:46:483
295480199112,6cyclictest0-21swapper/300:20:243
295480199111,6cyclictest0-21swapper/322:45:193
295480199111,6cyclictest0-21swapper/322:40:283
295480199111,6cyclictest0-21swapper/322:35:263
295480199111,6cyclictest0-21swapper/322:05:233
295480199111,6cyclictest0-21swapper/321:35:223
295480199111,6cyclictest0-21swapper/321:25:223
295480199111,6cyclictest0-21swapper/320:45:223
295480199111,6cyclictest0-21swapper/320:25:233
295480199111,6cyclictest0-21swapper/319:50:293
295480199111,6cyclictest0-21swapper/319:25:243
295480199101,6cyclictest0-21swapper/323:15:283
295480199101,6cyclictest0-21swapper/319:20:213
295480199101,5cyclictest0-21swapper/300:30:243
29548019992,5cyclictest0-21swapper/319:55:163
29548019991,5cyclictest0-21swapper/319:35:143
29548019990,3cyclictest0-21swapper/323:40:033
29548019981,3cyclictest0-21swapper/319:15:123
29548019971,4cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional