You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-18 - 13:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 18, 2025 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
301551923170,9sleep34497-21PK-Backend19:05:203
301515622680,13sleep234-21ksoftirqd/219:05:152
308391321240,7sleep30-21swapper/322:20:023
1973991070,9rtkit-daemon1972-21rtkit-daemon21:20:073
1973991010,9rtkit-daemon1972-21rtkit-daemon20:24:293
197399990,9rtkit-daemon1972-21rtkit-daemon20:47:413
197399970,9rtkit-daemon1972-21rtkit-daemon22:25:051
197399950,8rtkit-daemon1972-21rtkit-daemon20:35:543
197399920,9rtkit-daemon1972-21rtkit-daemon21:13:373
197399920,9rtkit-daemon1972-21rtkit-daemon21:13:373
30310242910,10sleep153-21kcompactd019:50:091
197399880,9rtkit-daemon1972-21rtkit-daemon20:42:553
197399880,6rtkit-daemon1972-21rtkit-daemon22:22:573
30884812770,9sleep342-21ksoftirqd/322:30:283
197399730,59rtkit-daemon0-21swapper/119:07:561
301626599700,65cyclictest1972-21rtkit-daemon21:30:292
30935462660,7sleep10-21swapper/122:45:281
301451026438,20sleep00-21swapper/019:05:020
30883672590,14sleep2341ksoftirqd/222:30:262
197399560,9rtkit-daemon1972-21rtkit-daemon20:36:422
197399560,8rtkit-daemon1972-21rtkit-daemon23:23:532
197399560,8rtkit-daemon1972-21rtkit-daemon20:25:402
197399540,8rtkit-daemon1972-21rtkit-daemon20:44:592
197399530,8rtkit-daemon1972-21rtkit-daemon22:20:502
197399520,8rtkit-daemon1972-21rtkit-daemon20:24:012
30280502510,7chrt0-21swapper/219:40:242
30280502510,7chrt0-21swapper/219:40:242
301626999500,46cyclictest1972-21rtkit-daemon20:58:183
301625599494,6cyclictest141rcu_preempt21:30:000
31187192480,7sleep2331rcuc/200:00:022
3016255994817,27cyclictest3101055-21cron23:10:000
31217552470,7sleep10-21swapper/100:05:321
301626099473,11cyclictest1-21systemd22:15:001
301626099472,39cyclictest3115875-21hddtemp_smartct23:50:161
3016255994629,6cyclictest13-21ksoftirqd/000:10:140
197399460,32rtkit-daemon0-21swapper/019:10:210
197399450,9rtkit-daemon931-21irqbalance21:03:072
30696092440,7sleep14497-21PK-Backend21:40:001
30696092440,7sleep14497-21PK-Backend21:40:001
3016269994421,19cyclictest3093269-21memory22:45:223
197399440,5rtkit-daemon0-21swapper/100:11:391
197399440,5rtkit-daemon0-21swapper/000:15:440
3016269994321,18cyclictest3129887-21expr00:30:143
3016255994325,6cyclictest13-21ksoftirqd/019:40:010
3016255994322,13cyclictest13-21ksoftirqd/020:30:020
30932712420,6sleep04497-21PK-Backend22:45:220
30638352420,7sleep2331rcuc/221:20:292
301626999422,23cyclictest3021208-21ps19:20:263
31161882410,8sleep00-21swapper/023:50:220
3016269994124,13cyclictest42-21ksoftirqd/321:35:303
3016269994124,13cyclictest42-21ksoftirqd/321:35:293
3016269994116,21cyclictest3090032-21processes22:35:273
3016269994114,7cyclictest141rcu_preempt20:30:003
3016269994113,8cyclictest42-21ksoftirqd/300:09:553
3016269994110,15cyclictest42-21ksoftirqd/323:20:323
3016265994131,6cyclictest3034626-21sort20:00:022
3016265994131,6cyclictest3034626-21sort20:00:012
3016265994127,9cyclictest3119758-21ps00:00:272
3016265994127,9cyclictest3119758-21ps00:00:272
301626599411,17cyclictest3030853-21cat19:50:022
3016260994122,14cyclictest26-21ksoftirqd/123:36:551
3016260994122,14cyclictest26-21ksoftirqd/123:36:551
3016255994129,8cyclictest3064403-21systemd21:25:010
3016255994120,16cyclictest13-21ksoftirqd/019:30:140
31083272400,7chrt4497-21PK-Backend23:30:020
30330222400,7sleep14497-21PK-Backend19:55:121
30330222400,7sleep14497-21PK-Backend19:55:121
3016269994024,12cyclictest923-21dbus-daemon22:59:573
301626999402,32cyclictest3126248-21df00:20:143
3016260994023,13cyclictest26-21ksoftirqd/119:35:301
3016255994021,14cyclictest13-21ksoftirqd/019:20:010
31126542390,6chrt0-21swapper/123:40:211
301626999398,17cyclictest0-21swapper/320:01:593
301626999398,17cyclictest0-21swapper/320:01:593
301626999398,14cyclictest3047059-21sh20:35:003
3016269993924,11cyclictest923-21dbus-daemon23:05:013
3016269993922,12cyclictest1033-21gdbus19:58:583
3016269993922,12cyclictest1033-21gdbus19:58:583
3016260993922,5cyclictest26-21ksoftirqd/119:45:171
3016260993921,14cyclictest26-21ksoftirqd/122:44:061
301626099391,10cyclictest141rcu_preempt00:25:011
301625599392,7cyclictest3021157-21ntp_states19:20:250
3016255993923,11cyclictest438-20systemd-journal21:30:300
30325862381,6sleep23032775-20journal-offline19:55:022
3016269993823,11cyclictest3019804-21sshd19:15:573
3016265993823,11cyclictest3104528-21apt-get23:20:022
3016265993820,13cyclictest34-21ksoftirqd/221:45:022
3016265993818,15cyclictest3022428-21cstates19:25:142
3016260993822,11cyclictest438-20systemd-journal20:40:561
301625599382,32cyclictest3060189-21ntp_states21:10:250
301625599382,32cyclictest3060189-21ntp_states21:10:250
31289632370,6sleep04497-21PK-Backend00:25:310
31289632370,6sleep04497-21PK-Backend00:25:300
301626999372,31cyclictest3129025-40lsb_release00:25:593
301626999372,31cyclictest3129025-40lsb_release00:25:593
3016269993723,10cyclictest42-21ksoftirqd/320:20:023
301626999372,21cyclictest3036638-21grep20:05:133
3016269993720,13cyclictest3026163-21cat19:35:213
3016269993719,13cyclictest3029049-21cat19:45:103
3016269993711,12cyclictest141rcu_preempt23:50:003
3016269993710,14cyclictest141rcu_preempt22:05:003
3016269993710,14cyclictest141rcu_preempt22:04:593
301626599372,6cyclictest3128315-21if_enp2s000:25:202
301626599372,6cyclictest3128315-21if_enp2s000:25:202
3016265993724,10cyclictest34-21ksoftirqd/219:30:162
3016265993720,13cyclictest34-21ksoftirqd/219:15:012
3016265993716,16cyclictest3131146-21unin-run00:35:002
3016260993725,8cyclictest3098016-21cpuspeed_turbos23:00:131
3016260993722,11cyclictest3047179-21date20:35:011
3016260993721,11cyclictest438-20systemd-journal20:55:551
3016260993719,14cyclictest3037501-21python320:05:291
3016260993719,14cyclictest26-21ksoftirqd/123:20:371
301626099371,12cyclictest141rcu_preempt19:25:291
3016255993724,9cyclictest3030656-21latency_hist19:50:010
3016255993713,18cyclictest3080460-21wc22:10:020
301626999366,7cyclictest141rcu_preempt23:44:503
301626999362,5cyclictest1054-21snmpd23:05:543
301626999362,20cyclictest3110493-21cat23:35:153
301626999362,20cyclictest3110493-21cat23:35:143
301626999362,19cyclictest42-21ksoftirqd/321:45:023
3016269993621,11cyclictest3028538-21sshd19:41:563
3016269993621,11cyclictest3028538-21sshd19:41:563
301626999361,6cyclictest3032005-21find19:50:293
301626999361,29cyclictest3053009-21irqstats20:50:203
301626999361,18cyclictest3080987-21sort22:10:193
301626999361,12cyclictest3079378-21sed22:05:223
3016265993626,6cyclictest1137-21runrttasks21:39:592
3016265993626,6cyclictest1137-21runrttasks21:39:582
3016265993623,9cyclictest34-21ksoftirqd/221:50:132
3016265993621,11cyclictest3036347-21cron20:09:552
3016265993620,12cyclictest3046735-21sshd20:30:562
3016265993620,12cyclictest1-21systemd20:49:592
3016265993619,14cyclictest3035342-21irqstats20:00:212
3016265993619,14cyclictest3035342-21irqstats20:00:212
301626099368,12cyclictest3017970-21kworker/1:120:30:001
3016260993626,6cyclictest3063081-21cpuspeed_turbos21:20:121
3016260993621,11cyclictest3027656-21expr19:40:151
3016260993621,11cyclictest3027656-21expr19:40:141
3016260993620,12cyclictest3095700-21sshd22:52:561
3016260993618,6cyclictest26-21ksoftirqd/122:56:181
301626099361,15cyclictest3084706-21needreboot22:20:231
301626099361,14cyclictest4497-21PK-Backend19:32:211
3016260993610,5cyclictest141rcu_preempt19:20:141
3016255993626,6cyclictest13-21ksoftirqd/023:35:020
3016255993625,8cyclictest13-21ksoftirqd/019:58:590
3016255993625,8cyclictest13-21ksoftirqd/019:58:590
3016255993621,9cyclictest2095-21JS10:49:090
3016255993621,12cyclictest13-21ksoftirqd/019:40:270
3016255993621,12cyclictest13-21ksoftirqd/019:40:260
3016255993621,11cyclictest3036684-21awk20:05:130
3016255993620,5cyclictest13-21ksoftirqd/023:20:010
3016255993620,5cyclictest13-21ksoftirqd/022:10:220
3016255993620,5cyclictest13-21ksoftirqd/000:30:290
3016255993619,12cyclictest438-20systemd-journal23:05:000
3016255993618,7cyclictest3082398-21sort22:15:120
3016255993618,13cyclictest13-21ksoftirqd/000:20:120
301625599361,12cyclictest3087872-21/usr/sbin/munin22:30:160
197399360,29rtkit-daemon0-21swapper/223:38:002
197399360,29rtkit-daemon0-21swapper/223:38:002
301626999359,14cyclictest141rcu_preempt21:55:133
301626999354,11cyclictest141rcu_preempt00:15:283
301626999352,27cyclictest3064639-21apt21:25:113
3016269993520,11cyclictest3132270-21awk00:35:303
3016269993520,11cyclictest3024599-21ntp_kernel_pll_19:30:243
3016269993518,12cyclictest3106967-21grep23:25:133
301626999351,23cyclictest0-21swapper/323:16:423
301626999351,13cyclictest3101913-21memory23:10:193
3016265993525,6cyclictest34-21ksoftirqd/221:10:022
3016265993524,6cyclictest34-21ksoftirqd/223:55:012
3016265993524,6cyclictest34-21ksoftirqd/221:45:022
3016265993520,11cyclictest3052634-21cat20:50:132
3016265993519,12cyclictest976-21rs:main2
3016265993519,12cyclictest3123791-21seq00:13:592
3016265993519,12cyclictest3065030-21awk21:25:182
3016265993518,12cyclictest3100325-21ntpq23:05:242
301626599351,30cyclictest3131861-21interrupts00:35:202
301626599351,11cyclictest3059594-21grep21:10:132
301626599351,11cyclictest3059594-21grep21:10:122
301626599351,10cyclictest141rcu_preempt23:00:132
301626099357,9cyclictest26-21ksoftirqd/123:32:471
3016260993523,8cyclictest3072060-21cpuspeed_turbos21:45:121
3016260993521,10cyclictest26-21ksoftirqd/100:30:261
3016260993520,11cyclictest3090666-21sshd22:39:551
301626099351,8cyclictest4497-21PK-Backend20:54:071
3016260993517,6cyclictest26-21ksoftirqd/123:55:131
3016260993517,14cyclictest3066739-21df_inode21:30:151
3016260993514,5cyclictest141rcu_preempt21:30:011
301626099351,13cyclictest4497-21PK-Backend20:16:001
301625599353,9cyclictest141rcu_preempt22:40:000
3016255993528,5cyclictest0-21swapper/022:27:380
3016255993524,6cyclictest13-21ksoftirqd/021:45:010
3016255993524,6cyclictest13-21ksoftirqd/020:55:270
3016255993520,11cyclictest3092405-21cron22:45:010
3016255993520,11cyclictest3042424-21memory20:20:220
3016255993519,5cyclictest13-21ksoftirqd/019:25:210
3016255993519,12cyclictest3039825-21munin-run20:15:000
3016255993517,13cyclictest13-21ksoftirqd/000:40:020
197399350,25rtkit-daemon0-21swapper/000:06:330
301626999345,11cyclictest141rcu_preempt21:30:283
301626999344,25cyclictest3038336-21cat20:10:093
3016269993422,8cyclictest3091435-21irqstats22:40:203
3016269993422,7cyclictest3070634-21ntp_states21:40:253
301626999342,19cyclictest3061554-21diskstats21:15:163
3016269993417,13cyclictest3095174-21awk22:50:263
301626999341,30cyclictest0-21swapper/321:51:313
3016265993423,6cyclictest34-21ksoftirqd/200:05:222
3016265993418,12cyclictest3038835-21sshd20:10:182
3016265993418,11cyclictest3077749-21cat22:00:232
3016265993418,11cyclictest3077749-21cat22:00:222
3016265993417,13cyclictest3127346-21apt-get00:25:012
3016265993417,12cyclictest976-21rs:main2
3016265993417,12cyclictest3096321-21grep22:55:132
3016265993416,13cyclictest3086880-21seq22:25:472
3016265993412,18cyclictest3124349-21cstates00:15:152
301626099347,5cyclictest141rcu_preempt20:46:341
301626099344,13cyclictest141rcu_preempt23:30:001
301626099343,12cyclictest141rcu_preempt21:10:011
3016260993421,8cyclictest3077260-21cut22:00:141
3016260993421,8cyclictest3077260-21cut22:00:141
301626099341,8cyclictest4497-21PK-Backend21:03:081
301626099341,8cyclictest4497-21PK-Backend20:24:251
3016260993418,12cyclictest3047534-21ls20:35:151
3016260993417,13cyclictest3102362-21sed23:10:281
301626099341,12cyclictest0-21swapper/121:12:581
301626099341,12cyclictest0-21swapper/121:12:581
301626099341,10cyclictest3082682-21entropy22:15:171
3016255993427,4cyclictest4497-21PK-Backend20:30:220
3016255993426,5cyclictest4497-21PK-Backend22:50:310
3016255993424,5cyclictest13-21ksoftirqd/021:10:010
3016255993423,6cyclictest13-21ksoftirqd/023:45:180
3016255993423,6cyclictest13-21ksoftirqd/023:35:250
3016255993423,6cyclictest13-21ksoftirqd/023:35:250
3016255993423,6cyclictest13-21ksoftirqd/023:20:070
3016255993423,6cyclictest13-21ksoftirqd/019:50:270
3016255993423,6cyclictest13-21ksoftirqd/000:00:010
301625599341,29cyclictest3036364-21turbostat20:05:000
301625599341,29cyclictest3036364-21turbostat20:05:000
301626999337,5cyclictest141rcu_preempt00:11:333
301626999332,26cyclictest3109431-21python323:30:293
301626999331,29cyclictest0-21swapper/322:25:543
301626999331,29cyclictest0-21swapper/319:14:323
301626599335,11cyclictest34-21ksoftirqd/222:17:032
3016265993322,6cyclictest34-21ksoftirqd/219:15:032
3016265993322,5cyclictest34-21ksoftirqd/223:42:022
3016265993321,6cyclictest34-21ksoftirqd/222:40:002
3016265993320,9cyclictest34-21ksoftirqd/222:40:222
3016265993320,9cyclictest3055846-21sed21:00:022
3016265993317,12cyclictest3094400-21find22:50:102
3016265993317,11cyclictest3076123-21ntpq21:55:232
3016265993316,13cyclictest3107495-21grep23:25:212
3016265993316,12cyclictest976-21rs:main2
301626099339,5cyclictest141rcu_preempt19:15:071
301626099339,4cyclictest141rcu_preempt23:05:141
301626099336,5cyclictest141rcu_preempt00:38:341
301626099335,9cyclictest26-21ksoftirqd/122:34:451
301626099335,5cyclictest141rcu_preempt00:29:261
301626099335,5cyclictest141rcu_preempt00:29:261
3016260993324,6cyclictest3035000-21cpuspeed_turbos20:00:141
3016260993324,6cyclictest3035000-21cpuspeed_turbos20:00:131
3016260993323,6cyclictest3124303-21cpuspeed_turbos00:15:131
301626099331,9cyclictest4497-21PK-Backend21:42:361
3016260993318,11cyclictest3016690-21cat19:10:101
301626099331,29cyclictest4497-21PK-Backend21:57:231
3016255993323,6cyclictest13-21ksoftirqd/021:00:230
3016255993323,5cyclictest13-21ksoftirqd/023:00:010
3016255993322,6cyclictest13-21ksoftirqd/021:45:190
3016255993322,6cyclictest13-21ksoftirqd/000:03:590
3016255993322,6cyclictest13-21ksoftirqd/000:03:590
3016255993321,6cyclictest13-21ksoftirqd/020:40:230
3016255993319,10cyclictest13-21ksoftirqd/023:40:210
3016255993317,12cyclictest923-21dbus-daemon20:40:000
301625599331,5cyclictest3040091-21apt20:15:130
301625599331,13cyclictest3076890-21latency_hist22:00:010
3016269993217,11cyclictest3058617-21ls21:05:293
3016269993217,11cyclictest3057119-21sshd21:00:563
3016269993216,12cyclictest3023026-21awk19:25:283
3016265993222,6cyclictest34-21ksoftirqd/223:49:592
3016265993222,6cyclictest34-21ksoftirqd/219:35:152
3016265993217,10cyclictest3081319-21seq22:10:252
3016265993216,12cyclictest3108907-21expr23:30:152
3016265993215,13cyclictest1-21systemd20:19:472
301626099328,5cyclictest141rcu_preempt00:00:251
301626099328,5cyclictest141rcu_preempt00:00:251
301626099327,5cyclictest141rcu_preempt21:15:211
3016260993218,10cyclictest3079073-21expr22:05:131
3016260993218,10cyclictest3039350-21timerandwakeup20:10:271
301626099321,28cyclictest4497-21PK-Backend21:54:531
301626099321,28cyclictest0-21swapper/123:47:281
301626099321,14cyclictest4497-21PK-Backend23:16:331
3016255993223,5cyclictest13-21ksoftirqd/022:05:010
3016255993223,5cyclictest13-21ksoftirqd/022:05:010
3016255993222,6cyclictest13-21ksoftirqd/021:35:280
3016255993222,6cyclictest13-21ksoftirqd/021:35:280
3016255993222,6cyclictest13-21ksoftirqd/020:50:270
3016255993222,6cyclictest13-21ksoftirqd/020:45:130
3016255993221,6cyclictest13-21ksoftirqd/021:15:100
3016255993216,12cyclictest3084740-21awk22:20:220
301625599321,4cyclictest0-21swapper/021:51:590
301626999312,7cyclictest3119704-21ntp_states00:00:243
301626999312,7cyclictest3119704-21ntp_states00:00:233
301626999312,25cyclictest3117208-21/usr/sbin/munin23:55:113
3016269993119,8cyclictest3116947-21latency_hist23:55:023
301626599317,20cyclictest3080289-21cron22:10:002
3016265993125,4cyclictest0-21swapper/223:10:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional