You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 01:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Fri Jun 12, 2026 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3099148996857,7cyclictest3191848-21munin-run10:40:010
3099148996515,41cyclictest627-21lldpd08:05:270
309914999622,50cyclictest3158419-21ssh09:45:291
3099150996150,8cyclictest2299980-21cups-browsed12:18:532
3099150995833,21cyclictest1003-21mta-sts-daemon09:35:272
3099150995830,7cyclictest3189963-21mailstats10:35:322
309915099562,45cyclictest3239945-21ssh11:55:252
309914899553,38cyclictest3192839-21ssh10:40:230
309914999541,44cyclictest3155214-21ssh09:40:281
3099148995436,12cyclictest12650irq/32-eth011:10:300
3099148995339,9cyclictest151ktimers/011:35:230
3099150995023,5cyclictest0-21swapper/210:40:282
3099148994812,12cyclictest3179311-21ssh10:20:010
3099149994638,5cyclictest3176159-21systemd-run10:15:011
3099149994635,8cyclictest3120160-21sh08:20:011
309914999463,36cyclictest3227369-21ssh11:35:251
3099148994529,14cyclictest3133549-21unixbench_singl09:00:370
309914899443,8cyclictest3125538-21ntpq08:35:270
3099148994432,9cyclictest0-21swapper/011:00:010
3099148994424,17cyclictest0-21swapper/012:35:210
309914899441,9cyclictest3208553-21ntpq11:05:210
3099150994335,5cyclictest3195779-21wc10:45:202
3099149994331,10cyclictest0-21swapper/110:40:271
3099148994334,7cyclictest3120217-21latency_hist08:20:020
3099148994334,6cyclictest2299980-21cups-browsed12:02:340
309914899431,12cyclictest3113572-21ntpq07:55:260
3099149994231,7cyclictest28-21ksoftirqd/109:05:251
3099149994230,7cyclictest3148458-21cut09:30:181
3099149994219,16cyclictest271ktimers/111:30:241
309914899423,30cyclictest3233617-21ssh11:45:240
309914899423,17cyclictest3161565-21ssh09:50:320
309914899422,8cyclictest3101554-21ntpq07:15:300
309914899421,12cyclictest3112034-21ntpq07:50:270
3099150994133,4cyclictest0-21swapper/207:40:012
309915099403,29cyclictest3246338-21ssh12:05:282
3099150994032,5cyclictest421-21dbus-daemon08:45:022
309914999404,29cyclictest3186547-21ssh10:30:221
309914999404,28cyclictest3202391-21ssh10:55:301
309914999403,27cyclictest3151999-21ssh09:35:271
309914899408,6cyclictest3200825-21sh10:53:410
309914899407,10cyclictest3155347-21processes09:40:320
309914899403,9cyclictest3214954-21ntpq11:15:290
3099148994036,2cyclictest0-21swapper/012:00:010
3099148994019,18cyclictest3103646-21sh07:25:000
3099148994016,21cyclictest0-21swapper/010:45:180
3099148994015,22cyclictest3139649-21switchtime09:15:350
309914899401,33cyclictest3183483-21rm10:25:250
3099148994012,5cyclictest0-21swapper/012:17:470
309914899401,15cyclictest3152006-21ntpq09:35:280
309914899401,11cyclictest3100089-21ntpq07:10:340
3099150993917,18cyclictest351ktimers/211:35:262
309914999392,29cyclictest3221085-21ssh11:25:211
309914899393,5cyclictest0-21swapper/012:10:260
309914899392,10cyclictest3258913-21ssh12:25:280
3099148993915,2cyclictest0-21swapper/012:25:010
3099148993913,4cyclictest3205572-21ntp_states11:00:310
309914899391,17cyclictest3217989-21ntpq11:20:240
3099149993831,3cyclictest3179314-21kworker/u8:1-xprtiod10:22:051
3099149993827,7cyclictest2299980-21cups-browsed10:16:551
3099149993827,7cyclictest1000-21mta-sts-daemon11:55:261
309914899387,10cyclictest3167792-21ssh10:00:280
3099148993831,4cyclictest3160021-21kworker/u8:10+xprtiod10:35:010
3099148993819,16cyclictest0-21swapper/011:25:360
3099148993818,17cyclictest3264065-21latency_hist12:35:030
3099148993812,2cyclictest0-21swapper/009:20:250
3099148993812,2cyclictest0-21swapper/009:20:250
309914899381,15cyclictest3224313-21ssh11:30:280
309914899381,15cyclictest3107570-21ntpq07:35:300
3099148993810,5cyclictest0-21swapper/010:24:130
309914899380,18cyclictest3121021-21ntpq08:20:250
309914899380,18cyclictest3121021-21ntpq08:20:250
309915099373,24cyclictest3155126-21ssh09:40:242
3099150993729,6cyclictest3150994-21/usr/sbin/munin09:35:012
3099150993729,5cyclictest3147838-21munin-run09:30:002
3099150993729,5cyclictest3147838-21munin-run09:30:002
3099150993729,5cyclictest3109662-21systemd-run07:45:012
309914999374,12cyclictest28-21ksoftirqd/111:20:261
3099149993730,4cyclictest3251502-21grep12:15:011
3099149993730,4cyclictest1003-21mta-sts-daemon12:02:451
309914899373,9cyclictest3164674-21ntpq09:55:330
309914899371,15cyclictest3145681-21ssh09:25:250
309914899371,15cyclictest3145681-21ssh09:25:250
309914899371,14cyclictest3236823-21ntpq11:50:260
309914899371,14cyclictest3170417-21awk10:05:150
3099148993711,3cyclictest0-21swapper/007:44:480
3099148993711,2cyclictest0-21swapper/008:10:160
309914899370,11cyclictest3148942-21ssh09:30:290
3099150993627,6cyclictest439-21systemd-logind11:43:352
3099150993627,4cyclictest0-21swapper/212:10:282
3099150993625,8cyclictest3221228-21sh11:25:302
309914999369,5cyclictest171rcu_preempt11:05:241
309914999363,25cyclictest3161551-21ssh09:50:301
3099149993628,5cyclictest3108119-21latency_hist07:40:011
3099149993627,5cyclictest3169849-21cron10:05:011
309914899361,6cyclictest3123671-21grep08:30:180
3099148993611,3cyclictest0-21swapper/007:45:320
309914899361,12cyclictest3136165-21ntpq09:10:260
3099150993528,5cyclictest3215427-21unixbench_multi11:15:362
3099150993528,4cyclictest1002-21mta-sts-daemon08:25:012
3099150993528,4cyclictest1002-21mta-sts-daemon08:25:012
3099150993527,5cyclictest3171457-21grep10:05:362
3099150993527,5cyclictest3167330-21fschecks_time10:00:182
3099150993527,5cyclictest2299980-21cups-browsed08:26:502
3099150993526,7cyclictest3213864-21/usr/sbin/munin11:15:032
3099150993526,6cyclictest421-21dbus-daemon09:20:002
3099150993523,6cyclictest3138453-21grep09:15:032
309914999355,7cyclictest171rcu_preempt11:43:361
3099149993528,3cyclictest0-21swapper/110:00:011
3099149993527,3cyclictest0-21swapper/111:50:241
3099149993526,5cyclictest3099664-21grep07:10:201
309914999352,19cyclictest3142632-21ssh09:20:301
309914999352,19cyclictest3142632-21ssh09:20:301
309914899355,9cyclictest421-21dbus-daemon09:10:010
3099148993529,4cyclictest3130619-21ls08:55:010
309914899352,12cyclictest3106632-21/usr/sbin/munin07:35:020
309914899352,10cyclictest1002-21mta-sts-daemon08:25:240
309914899351,11cyclictest3174623-21users10:10:350
309914899350,8cyclictest3114662-21fschecks_time08:00:170
3099149993419,9cyclictest271ktimers/109:10:271
3099149993419,10cyclictest3233648-21ntp_kernel_pll_11:45:261
3099148993429,3cyclictest3230113-21sort11:40:220
309914899341,12cyclictest3128481-21ntpq08:45:290
309914899340,13cyclictest3132008-21unixbench_multi08:55:350
309915099333,25cyclictest3233865-21ssh11:45:322
3099150993328,2cyclictest0-21swapper/208:00:012
3099150993326,3cyclictest0-21swapper/209:45:242
3099150993325,5cyclictest601-21gmain10:10:192
3099150993325,3cyclictest0-21swapper/212:35:012
3099150993324,6cyclictest420-21avahi-daemon12:25:182
309914999333,23cyclictest3205412-21ssh11:00:251
3099149993326,4cyclictest3189387-21if_eth010:35:181
3099149993325,5cyclictest3258400-21fschecks_count12:25:171
3099149993325,5cyclictest3139916-21grep09:15:371
3099149993324,6cyclictest3217028-21systemd-run11:20:011
3099149993323,6cyclictest3146200-21unixbench-2d09:25:361
3099149993323,6cyclictest3146200-21unixbench-2d09:25:361
3099149993320,9cyclictest3104542-21ntp_kernel_pll_07:25:301
309914899339,4cyclictest3246814-21tail12:05:350
309915099327,13cyclictest3255706-21ssh12:20:272
3099150993225,3cyclictest0-21swapper/209:00:012
3099150993223,6cyclictest485-21in:imuxsock11:35:012
309915099321,26cyclictest3161560-21ssh09:50:312
3099149993225,4cyclictest3124642-21systemd-run08:35:011
3099149993224,5cyclictest3115548-21head08:00:361
309914999322,25cyclictest3261894-21ssh12:30:221
3099149993221,8cyclictest3105943-21needreboot07:30:251
309914899323,7cyclictest3104525-21ntpq07:25:280
3099148993210,4cyclictest0-21swapper/009:45:310
3099150993125,3cyclictest0-21swapper/211:05:272
3099150993124,4cyclictest3205060-21fschecks_time11:00:192
3099150993123,5cyclictest3179896-21fschecks_count10:20:162
3099150993123,5cyclictest265-21systemd-journal11:00:012
3099150993122,6cyclictest420-21avahi-daemon07:30:352
3099150993122,6cyclictest3120122-21latency_hist08:20:022
3099150993121,6cyclictest421-21dbus-daemon09:58:342
3099150993120,8cyclictest3110445-21needreboot07:45:262
309915099311,22cyclictest3186751-21sh10:30:302
3099149993122,6cyclictest439-21systemd-logind07:55:011
3099149993122,6cyclictest421-21dbus-daemon10:25:431
3099149993120,8cyclictest3199098-21ntp_kernel_err10:50:231
309914999311,25cyclictest28-21ksoftirqd/112:15:241
309914899311,10cyclictest1002-21mta-sts-daemon08:40:270
309915099305,22cyclictest3265103-21ssh12:35:242
3099150993023,4cyclictest3218221-21processes11:20:322
3099150993022,5cyclictest3242678-21ssh12:00:172
3099150993022,5cyclictest3123652-21fschecks_time08:30:182
3099150993019,8cyclictest3135130-21diskmemload10:20:012
309915099301,25cyclictest3236773-21ssh11:50:242
3099149993024,3cyclictest0-21swapper/112:25:011
309914999301,12cyclictest62750lldpd10:45:301
3099150992922,4cyclictest3184004-21unixbench_multi10:25:342
3099150992922,4cyclictest0-21swapper/208:40:012
3099150992922,3cyclictest0-21swapper/209:20:362
3099150992922,3cyclictest0-21swapper/209:20:362
3099150992913,14cyclictest351ktimers/208:05:282
3099149992921,5cyclictest3132099-21cron09:00:011
3099150992819,6cyclictest3100648-21systemd-run07:15:012
3099149992822,3cyclictest0-21swapper/111:10:361
3099149992821,4cyclictest3126211-21latency_hist08:40:021
3099149992818,7cyclictest421-21dbus-daemon08:10:011
3099149992815,9cyclictest3113487-21ntp_kernel_pll_07:55:221
309914999281,22cyclictest1000-21mta-sts-daemon08:25:241
309914999281,21cyclictest3170904-21ssh10:05:261
3099150992722,2cyclictest0-21swapper/209:00:372
3099150992722,2cyclictest0-21swapper/208:10:282
309915099272,18cyclictest3199206-21ssh10:50:262
3099150992720,4cyclictest3115638-21ls08:05:012
3099150992719,5cyclictest3105065-21unixbench_singl07:25:402
3099149992721,3cyclictest0-21swapper/107:40:381
3099149992720,4cyclictest1003-21mta-sts-daemon08:10:281
3099149992719,5cyclictest3248364-21sh12:10:011
3099149992719,5cyclictest3129127-21munin-plugin-st08:50:011
3099149992719,5cyclictest3126649-21fschecks_count08:40:181
3099149992719,4cyclictest0-21swapper/107:15:161
3099149992713,10cyclictest2299980-21cups-browsed12:39:481
3099150992620,3cyclictest3086670-21kworker/u8:19+xprtiod07:25:002
3099150992616,7cyclictest3134301-21latency09:05:232
309914999253,4cyclictest171rcu_preempt09:00:261
3099150992418,3cyclictest0-21swapper/207:20:012
3099150992418,3cyclictest0-21swapper/207:10:022
3099150992416,5cyclictest3111460-21aten_r2power_fr07:50:162
3099150992415,6cyclictest3130417-21timerwakeupswit08:50:372
3099149992417,3cyclictest0-21swapper/108:20:251
3099149992417,3cyclictest0-21swapper/108:20:251
3099149992314,7cyclictest3110283-21kernelversion07:45:241
3099149992312,8cyclictest3103020-21ntp_kernel_pll_07:20:251
3099150992212,7cyclictest3129112-21sshd08:49:472
309914999222,15cyclictest3130044-21ntpq08:50:291
3099149992216,3cyclictest0-21swapper/107:10:021
309915199191,7cyclictest0-21swapper/312:00:263
309914899184,8cyclictest0-21swapper/007:10:010
309915199172,8cyclictest0-21swapper/310:22:063
309915199161,7cyclictest0-21swapper/311:35:263
309915199153,7cyclictest0-21swapper/310:35:263
309915199152,7cyclictest0-21swapper/310:55:303
309915199152,6cyclictest0-21swapper/309:30:303
309915199151,7cyclictest0-21swapper/312:25:253
309915199151,7cyclictest0-21swapper/312:10:223
309915199151,6cyclictest0-21swapper/310:00:243
309915199142,7cyclictest0-21swapper/309:25:253
309915199142,7cyclictest0-21swapper/309:25:253
309915199142,6cyclictest0-21swapper/312:30:233
309915199142,6cyclictest0-21swapper/311:25:243
309915199142,6cyclictest0-21swapper/309:50:263
309915199142,6cyclictest0-21swapper/309:35:313
309915199142,5cyclictest0-21swapper/311:10:173
309915199141,6cyclictest0-21swapper/308:30:263
309915199141,6cyclictest0-21swapper/308:25:273
309915199141,6cyclictest0-21swapper/308:05:303
309915199141,5cyclictest0-21swapper/308:40:243
309915199136,5cyclictest0-21swapper/307:15:163
309915199136,4cyclictest0-21swapper/309:10:283
309915199133,5cyclictest0-21swapper/311:50:013
309915199132,5cyclictest0-21swapper/310:50:243
309915199131,7cyclictest0-21swapper/312:05:233
309915199131,6cyclictest0-21swapper/312:35:213
309915199131,5cyclictest0-21swapper/307:30:273
309915199126,4cyclictest0-21swapper/311:00:383
309915199126,4cyclictest0-21swapper/309:05:393
309915199126,4cyclictest0-21swapper/308:05:023
309915199126,4cyclictest0-21swapper/307:50:013
309915199125,4cyclictest0-21swapper/309:55:273
309915199124,5cyclictest0-21swapper/309:20:303
309915199124,5cyclictest0-21swapper/309:20:303
309915199123,5cyclictest0-21swapper/308:20:243
309915199123,5cyclictest0-21swapper/308:20:243
309915199122,7cyclictest0-21swapper/308:55:013
309915199122,6cyclictest0-21swapper/311:55:303
309915199122,6cyclictest0-21swapper/311:50:153
309915199122,6cyclictest0-21swapper/311:05:233
309915199122,6cyclictest0-21swapper/310:45:263
309915199122,6cyclictest0-21swapper/310:25:223
309915199122,6cyclictest0-21swapper/310:10:223
309915199122,6cyclictest0-21swapper/308:10:283
309915199122,5cyclictest0-21swapper/309:15:283
309915199122,5cyclictest0-21swapper/308:35:173
309915199121,6cyclictest0-21swapper/311:40:213
309915199121,6cyclictest0-21swapper/310:40:183
309915199121,6cyclictest0-21swapper/310:15:223
309915199121,6cyclictest0-21swapper/307:40:013
309915199121,6cyclictest0-21swapper/307:25:353
309915199116,3cyclictest0-21swapper/307:25:013
309915199113,4cyclictest0-21swapper/309:45:143
309915199111,6cyclictest0-21swapper/312:20:143
309915199111,6cyclictest0-21swapper/311:30:223
309915199111,6cyclictest0-21swapper/311:20:163
309915199111,5cyclictest0-21swapper/310:35:023
309915199105,3cyclictest0-21swapper/312:15:253
309915199105,3cyclictest0-21swapper/309:40:133
309915199105,3cyclictest0-21swapper/307:10:143
309915199104,5cyclictest0-21swapper/311:15:303
309915199102,6cyclictest0-21swapper/310:05:143
309915199101,5cyclictest0-21swapper/308:45:133
30991519995,3cyclictest0-21swapper/307:45:013
30991519992,5cyclictest0-21swapper/308:55:293
30991519992,4cyclictest0-21swapper/307:10:023
30991519991,5cyclictest0-21swapper/307:55:123
30991519984,3cyclictest0-21swapper/309:04:393
30991519984,3cyclictest0-21swapper/307:50:163
30991519983,3cyclictest0-21swapper/308:15:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional