You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-06 - 03:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Mon Jul 06, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1230137996858,6cyclictest1001-21mta-sts-daemon21:25:382
1230135995134,11cyclictest0-21swapper/023:20:390
1230135995134,11cyclictest0-21swapper/023:20:390
1230137994738,6cyclictest1258651-21systemd-run20:45:012
1230137994636,6cyclictest1360881-21munin-plugin-st23:45:012
1230135994634,9cyclictest1330392-21idleruntime-cro22:55:010
1230137994536,6cyclictest0-21swapper/220:20:012
123013699457,8cyclictest171rcu_preempt00:25:371
123013699457,8cyclictest171rcu_preempt00:25:371
1230135994527,9cyclictest12650irq/32-eth000:05:390
1230136994439,2cyclictest0-21swapper/100:15:371
1230136994439,2cyclictest0-21swapper/100:15:371
123013699443,33cyclictest1349739-21rm23:25:361
123013699443,33cyclictest1349739-21rm23:25:361
1230135994438,4cyclictest1357814-21munin-run23:40:010
123013599443,35cyclictest1377267-21diskmemload00:10:390
1230137994317,17cyclictest420-21avahi-daemon22:10:382
1230136994337,3cyclictest0-21swapper/123:10:461
1230136994334,6cyclictest1296984-21latency_hist22:00:021
1230136994333,7cyclictest1261630-21systemd-run20:55:011
1230135994336,4cyclictest1313774-21unixbench_singl22:25:510
1230137994234,5cyclictest1296774-21idleruntime-cro22:00:002
1230137994231,7cyclictest420-21avahi-daemon00:14:242
123013799422,33cyclictest1291744-21ssh21:50:372
123013799422,33cyclictest1291744-21ssh21:50:372
1230136994235,3cyclictest0-21swapper/121:30:351
1230136994233,6cyclictest421-21dbus-daemon23:10:011
1230136994233,6cyclictest1282145-21fschecks_time21:35:291
1230136994229,10cyclictest1313780-21unixbench_singl22:25:511
1230137994132,4cyclictest0-21swapper/223:05:352
1230136994128,5cyclictest1002-21mta-sts-daemon23:51:461
1230136994121,14cyclictest271ktimers/122:11:331
1230135994132,6cyclictest12650irq/32-eth022:05:420
1230136994032,5cyclictest1321933-21fschecks_time22:40:301
1230135994032,6cyclictest1266121-21diskmemload21:20:010
1230135994026,2cyclictest171rcu_preempt20:35:410
1230135994026,2cyclictest171rcu_preempt20:35:410
1230137993932,3cyclictest0-21swapper/200:30:002
1230137993932,3cyclictest0-21swapper/200:30:002
1230137993931,4cyclictest0-21swapper/221:15:022
1230137993929,6cyclictest1003-21mta-sts-daemon21:40:412
1230137993929,6cyclictest1003-21mta-sts-daemon21:40:412
1230137993929,5cyclictest0-21swapper/200:10:012
1230136993931,5cyclictest1325953-21unixbench_singl22:45:481
1230136993930,6cyclictest1233212-21latency_hist19:20:021
1230135993932,4cyclictest1296806-21systemd-run22:00:010
1230135993932,4cyclictest1236147-21systemd-run19:30:010
1230137993831,4cyclictest1373110-21munin-plugin-st00:05:012
1230137993831,4cyclictest1354789-21systemd-run23:35:012
1230137993831,4cyclictest1354789-21systemd-run23:35:012
1230137993830,5cyclictest1365147-21proc_pri23:50:422
1230136993831,4cyclictest1394419-21idleruntime-cro00:40:011
1230136993825,10cyclictest1389183-21latency00:30:321
1230135993830,5cyclictest1283019-21timerwakeupswit21:35:480
1230135993828,3cyclictest1000-21mta-sts-daemon22:55:490
1230135993827,8cyclictest1380172-21needreboot00:15:340
1230135993827,8cyclictest1380172-21needreboot00:15:340
1230137993729,5cyclictest1392025-21fschecks_count00:35:302
1230137993729,5cyclictest1243613-21sh19:55:012
1230137993728,6cyclictest0-21swapper/222:55:392
123013799372,29cyclictest1282568-21sh21:35:382
123013799370,6cyclictest171rcu_preempt22:45:342
1230135993731,4cyclictest1290670-21cron21:50:000
1230135993731,4cyclictest1290670-21cron21:50:000
1230135993730,4cyclictest1373129-21systemd-run00:05:020
1230135993730,4cyclictest1339601-21latency_hist23:10:010
1230135993730,4cyclictest1291580-21missed_timers21:50:330
1230135993730,4cyclictest1291580-21missed_timers21:50:330
1230135993729,5cyclictest1353176-21awk23:30:490
1230135993729,5cyclictest1353176-21awk23:30:490
1230135993728,6cyclictest3694966-21cups-browsed23:29:470
1230135993728,6cyclictest3694966-21cups-browsed23:29:470
1230135993723,3cyclictest171rcu_preempt23:45:010
123013599371,12cyclictest14-21ksoftirqd/020:00:380
1230137993630,3cyclictest0-21swapper/200:00:012
1230137993629,3cyclictest0-21swapper/221:10:012
1230137993628,5cyclictest1245148-21systemd-run20:00:012
1230137993623,9cyclictest1-21systemd00:20:282
1230137993623,9cyclictest1-21systemd00:20:282
1230137993614,17cyclictest3694966-21cups-browsed19:10:362
1230136993629,3cyclictest0-21swapper/122:30:421
1230136993628,5cyclictest1360877-21awk23:45:011
1230136993628,5cyclictest1-21systemd19:50:011
1230135993630,4cyclictest1278443-21latency_hist21:30:020
1230135993611,21cyclictest1386434-21ssh00:25:420
1230135993611,21cyclictest1386434-21ssh00:25:420
1230137993527,5cyclictest1239123-21systemctl19:40:002
1230137993527,5cyclictest1239123-21systemctl19:40:002
1230137993526,5cyclictest1249638-21systemd-run20:15:012
1230137993525,7cyclictest1272289-21idleruntime-cro21:20:012
123013799351,26cyclictest1319314-21ssh22:35:372
1230136993529,4cyclictest1376170-21sh00:10:011
1230136993529,3cyclictest1255587-21kworker/u8:2-xprtiod21:05:261
1230136993529,3cyclictest0-21swapper/100:00:011
1230136993528,4cyclictest1363936-21systemd-run23:50:011
1230136993528,4cyclictest1003-21mta-sts-daemon22:00:361
1230136993527,5cyclictest1335100-21grep23:00:501
1230136993527,5cyclictest1335100-21grep23:00:501
1230136993527,5cyclictest1287635-21munin-plugin-st21:45:011
1230136993527,5cyclictest1287635-21munin-plugin-st21:45:011
1230136993525,6cyclictest3694966-21cups-browsed21:46:561
1230136993525,6cyclictest3694966-21cups-browsed21:46:561
1230136993525,6cyclictest1273199-21missed_timers21:20:341
1230136993524,8cyclictest1371108-21ntp_kernel_pll_00:00:351
1230135993529,3cyclictest1000-21mta-sts-daemon21:30:310
1230135993519,3cyclictest171rcu_preempt22:30:370
1230135993516,12cyclictest12650irq/32-eth019:40:410
1230135993512,20cyclictest665-21snmpd21:40:370
1230135993512,20cyclictest665-21snmpd21:40:370
1230137993422,9cyclictest1322131-21latency22:40:332
1230136993428,3cyclictest0-21swapper/121:50:391
1230136993428,3cyclictest0-21swapper/121:50:391
1230136993428,3cyclictest0-21swapper/121:10:351
1230136993427,4cyclictest1312103-21grep22:25:011
1230136993426,5cyclictest1357805-21cron23:40:001
1230136993425,6cyclictest1249913-21aten_r2power_en20:15:261
123013599343,25cyclictest1365150-21ssh23:50:430
1230135993428,4cyclictest1336503-21systemd-run23:05:010
1230135993428,4cyclictest1336503-21systemd-run23:05:010
1230135993419,2cyclictest171rcu_preempt20:55:400
1230135993419,13cyclictest1297777-21needreboot22:00:350
1230137993326,4cyclictest0-21swapper/223:35:472
1230137993326,3cyclictest0-21swapper/223:25:392
1230137993326,3cyclictest0-21swapper/223:25:392
1230137993325,5cyclictest1339934-21sh23:10:272
1230137993325,5cyclictest1252655-21systemd-run20:25:012
1230137993325,5cyclictest0-21swapper/220:30:012
1230137993323,6cyclictest1003-21mta-sts-daemon23:19:532
1230137993323,6cyclictest1003-21mta-sts-daemon23:19:532
1230136993326,4cyclictest1254643-21fschecks_time20:30:291
1230136993326,4cyclictest1254643-21fschecks_time20:30:291
1230136993326,4cyclictest1252624-21cron20:25:011
1230136993326,3cyclictest0-21swapper/123:15:321
1230136993326,3cyclictest0-21swapper/123:15:321
1230136993325,4cyclictest0-21swapper/121:30:011
1230136993325,3cyclictest0-21swapper/122:50:431
1230136993325,3cyclictest0-21swapper/122:18:011
123013599339,8cyclictest14-21ksoftirqd/019:20:360
1230135993321,9cyclictest0-21swapper/022:45:020
1230137993225,4cyclictest1003-21mta-sts-daemon23:48:532
1230137993225,4cyclictest0-21swapper/222:50:322
1230137993223,7cyclictest694-21cron21:35:012
1230137993219,10cyclictest1236932-21needreboot19:30:372
1230136993226,2cyclictest0-21swapper/121:15:321
1230136993224,5cyclictest1352766-21ntp_kernel_err23:30:361
1230136993224,5cyclictest1352766-21ntp_kernel_err23:30:361
1230136993224,5cyclictest1331082-21diskmemload22:55:341
1230135993226,3cyclictest0-21swapper/022:35:350
1230135993225,5cyclictest1389602-21mailstats00:30:430
1230135993222,3cyclictest171rcu_preempt23:15:380
1230135993222,3cyclictest171rcu_preempt23:15:380
1230135993219,3cyclictest171rcu_preempt00:20:460
1230135993219,3cyclictest171rcu_preempt00:20:460
1230135993215,10cyclictest3694966-21cups-browsed22:15:350
1230137993126,2cyclictest0-21swapper/222:05:012
1230137993124,5cyclictest1234633-21ls19:25:012
1230137993124,5cyclictest1230134-21cyclictest19:17:022
1230137993123,5cyclictest665-21snmpd22:15:432
1230137993123,5cyclictest1373345-21kworker/u8:11-rpciod00:30:322
1230137993123,3cyclictest0-21swapper/222:05:402
1230137993122,7cyclictest0-21swapper/222:35:022
123013699317,4cyclictest171rcu_preempt23:20:401
123013699317,4cyclictest171rcu_preempt23:20:401
123013599318,3cyclictest171rcu_preempt20:45:390
1230135993126,2cyclictest0-21swapper/020:35:010
1230135993126,2cyclictest0-21swapper/020:35:010
1230135993116,3cyclictest171rcu_preempt23:55:380
123013599311,23cyclictest1273459-21ssh21:20:420
123013799305,19cyclictest1310131-21ssh22:20:372
1230137993022,5cyclictest420-21avahi-daemon20:35:432
1230137993022,5cyclictest420-21avahi-daemon20:35:432
1230137993022,5cyclictest1334087-21fschecks_count23:00:302
1230137993022,5cyclictest1334087-21fschecks_count23:00:302
1230136993023,5cyclictest0-21swapper/120:40:011
1230136993023,5cyclictest0-21swapper/120:40:011
1230136993022,5cyclictest1370086-21cups-browsed00:10:111
1230136993020,7cyclictest1318992-21kernelversion22:35:301
123013599302,8cyclictest14-21ksoftirqd/023:10:420
1230135993025,3cyclictest1000-21mta-sts-daemon20:15:400
1230135993023,5cyclictest1303239-21/usr/sbin/munin22:10:470
1230135993020,7cyclictest151ktimers/019:30:400
1230135993019,3cyclictest171rcu_preempt21:10:340
1230135993015,3cyclictest171rcu_preempt23:45:420
123013799294,17cyclictest1288745-21ssh21:45:392
123013799294,17cyclictest1288745-21ssh21:45:392
1230137992921,3cyclictest0-21swapper/222:25:422
1230137992919,7cyclictest1273419-21ntp_offset21:20:412
1230136992923,4cyclictest1235089-21cpuspeed19:25:271
1230136992920,5cyclictest421-21dbus-daemon22:06:581
1230136992916,9cyclictest271ktimers/100:20:511
1230136992916,9cyclictest271ktimers/100:20:511
123013599298,12cyclictest171rcu_preempt22:45:410
1230135992923,4cyclictest1391782-21aten_r2power_en00:35:270
1230135992923,2cyclictest0-21swapper/019:10:360
1230135992921,6cyclictest1254098-21sshd20:27:320
1230135992919,3cyclictest171rcu_preempt21:00:370
1230135992918,8cyclictest1261525-21grep20:50:480
1230135992918,4cyclictest151ktimers/021:10:010
1230135992916,2cyclictest171rcu_preempt19:35:420
1230135992916,2cyclictest171rcu_preempt19:35:420
1230137992821,4cyclictest0-21swapper/221:00:012
1230137992820,5cyclictest1380866-21unixbench_singl00:15:512
1230137992820,5cyclictest1380866-21unixbench_singl00:15:512
1230137992820,5cyclictest1346325-21fschecks_time23:20:312
1230137992820,5cyclictest1346325-21fschecks_time23:20:312
1230137992820,5cyclictest1264545-21head21:00:512
1230136992819,6cyclictest1243140-21proc_pri19:50:421
123013599286,14cyclictest14-21ksoftirqd/020:40:310
1230135992822,4cyclictest1233157-21systemd-run19:20:010
1230135992822,2cyclictest0-21swapper/020:00:010
1230135992819,6cyclictest1247327-21latency20:05:310
1230135992813,3cyclictest171rcu_preempt20:10:360
123013799272,21cyclictest665-21snmpd19:40:382
1230137992720,4cyclictest1255557-21unixbench_singl20:30:472
1230137992720,4cyclictest1255557-21unixbench_singl20:30:472
1230136992721,3cyclictest0-21swapper/119:25:011
1230136992720,5cyclictest3694966-21cups-browsed19:31:091
1230136992720,4cyclictest439-21systemd-logind20:30:011
1230136992719,5cyclictest421-21dbus-daemon20:02:521
1230135992719,5cyclictest1243518-21unixbench_multi19:50:470
1230135992718,3cyclictest171rcu_preempt22:20:470
1230135992715,3cyclictest171rcu_preempt19:45:330
123013799265,14cyclictest428-21mta-sts-daemon20:05:412
1230137992619,4cyclictest1246628-21idleruntime-cro20:05:012
1230137992613,9cyclictest1235694-21grep19:25:422
123013699269,13cyclictest421-21dbus-daemon19:10:361
1230136992617,7cyclictest1240036-21collect219:40:391
1230135992613,3cyclictest171rcu_preempt20:20:370
1230137992518,4cyclictest1261604-21cron20:55:012
1230137992517,5cyclictest1241152-21fschecks_time19:45:272
1230136992519,3cyclictest0-21swapper/120:55:351
1230136992517,5cyclictest1243866-21aten_r2power_cu19:55:251
1230137992416,5cyclictest420-21avahi-daemon20:46:432
123013699243,7cyclictest1238721-21systemctl19:35:461
123013699243,7cyclictest1238721-21systemctl19:35:461
1230136992418,4cyclictest1000-21mta-sts-daemon20:45:341
123013699241,19cyclictest1263737-21awk21:00:341
123013599247,5cyclictest171rcu_preempt19:10:020
123013699232,15cyclictest1247548-21ntpq20:05:381
1230136992317,3cyclictest0-21swapper/120:10:271
1230136992316,4cyclictest1230147-21systemd-run19:10:021
1230136992313,7cyclictest1258528-21grep20:40:471
1230137992110,8cyclictest421-21dbus-daemon19:10:012
123013899192,8cyclictest0-21swapper/321:10:403
123013899182,7cyclictest0-21swapper/322:05:373
123013899181,11cyclictest0-21swapper/321:40:383
123013899181,11cyclictest0-21swapper/321:40:383
123013899172,9cyclictest0-21swapper/323:15:363
123013899172,9cyclictest0-21swapper/323:15:363
123013899172,9cyclictest0-21swapper/322:00:363
123013899172,9cyclictest0-21swapper/300:35:423
123013899163,8cyclictest0-21swapper/322:30:363
123013899162,8cyclictest0-21swapper/322:10:383
123013899162,7cyclictest0-21swapper/323:45:383
123013899162,7cyclictest0-21swapper/319:10:383
123013899162,7cyclictest0-21swapper/300:20:423
123013899162,7cyclictest0-21swapper/300:20:423
123013899161,9cyclictest0-21swapper/323:40:353
123013899161,8cyclictest0-21swapper/323:50:403
123013899155,6cyclictest0-21swapper/321:20:363
123013899154,7cyclictest0-21swapper/321:35:293
123013899153,7cyclictest0-21swapper/322:20:393
123013899152,8cyclictest0-21swapper/323:10:303
123013899152,8cyclictest0-21swapper/323:00:423
123013899152,8cyclictest0-21swapper/323:00:423
123013899152,8cyclictest0-21swapper/322:55:443
123013899152,8cyclictest0-21swapper/322:15:393
123013899152,8cyclictest0-21swapper/321:25:303
123013899152,8cyclictest0-21swapper/300:25:413
123013899152,8cyclictest0-21swapper/300:25:413
123013899152,8cyclictest0-21swapper/300:13:323
123013899152,8cyclictest0-21swapper/300:05:283
123013899152,6cyclictest0-21swapper/320:35:353
123013899152,6cyclictest0-21swapper/320:35:353
123013899152,5cyclictest0-21swapper/319:50:313
123013899151,8cyclictest0-21swapper/323:30:383
123013899151,8cyclictest0-21swapper/323:30:383
123013899151,8cyclictest0-21swapper/321:15:353
123013899151,8cyclictest0-21swapper/320:05:423
123013899151,7cyclictest0-21swapper/320:20:393
123013899151,6cyclictest0-21swapper/319:25:353
123013899144,6cyclictest0-21swapper/320:10:363
123013899144,6cyclictest0-21swapper/300:20:003
123013899144,6cyclictest0-21swapper/300:20:003
123013899143,5cyclictest0-21swapper/320:50:403
123013899142,7cyclictest0-21swapper/323:25:023
123013899142,7cyclictest0-21swapper/323:25:023
123013899142,7cyclictest0-21swapper/322:40:273
123013899142,7cyclictest0-21swapper/320:30:023
123013899142,7cyclictest0-21swapper/300:00:253
123013899142,5cyclictest0-21swapper/321:00:383
123013899142,5cyclictest0-21swapper/319:45:323
123013899141,8cyclictest0-21swapper/323:55:383
123013899141,8cyclictest0-21swapper/321:55:403
123013899141,7cyclictest0-21swapper/322:50:273
123013899141,6cyclictest0-21swapper/320:45:353
123013899141,6cyclictest0-21swapper/320:40:343
123013899133,5cyclictest0-21swapper/319:40:003
123013899133,5cyclictest0-21swapper/319:40:003
123013899132,8cyclictest0-21swapper/321:30:263
123013899132,6cyclictest0-21swapper/322:25:303
123013899132,6cyclictest0-21swapper/321:50:263
123013899132,6cyclictest0-21swapper/321:50:263
123013899131,7cyclictest0-21swapper/323:05:253
123013899131,7cyclictest0-21swapper/300:35:023
123013899131,6cyclictest0-21swapper/320:55:383
123013899131,6cyclictest0-21swapper/320:05:013
123013899131,6cyclictest0-21swapper/319:20:353
123013899123,6cyclictest0-21swapper/322:35:243
123013899122,7cyclictest0-21swapper/323:35:283
123013899122,6cyclictest0-21swapper/319:55:343
123013899121,7cyclictest0-21swapper/322:45:343
123013899121,7cyclictest0-21swapper/321:45:263
123013899121,7cyclictest0-21swapper/321:45:263
123013899121,7cyclictest0-21swapper/321:10:013
123013899121,6cyclictest0-21swapper/320:30:383
123013899121,6cyclictest0-21swapper/320:30:383
123013899121,6cyclictest0-21swapper/320:15:383
123013899112,6cyclictest0-21swapper/323:25:363
123013899112,6cyclictest0-21swapper/323:25:363
123013899102,5cyclictest0-21swapper/319:40:303
123013899102,5cyclictest0-21swapper/319:15:323
123013899101,5cyclictest0-21swapper/319:30:273
12301389991,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional