You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 15:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Thu May 14, 2026 12:43:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2957171996122,24cyclictest151ktimers/011:05:540
295717199601,45cyclictest2994661-21ssh09:11:020
2957171995824,31cyclictest2966140-21cron07:40:010
2957173995648,5cyclictest3136516-21unixbench_singl12:06:052
2957172995533,19cyclictest3136426-21sed12:06:041
295717399542,43cyclictest3144056-21ssh12:15:552
2957171995345,5cyclictest3114533-21latency_hist11:40:020
2957172995125,7cyclictest420-21avahi-daemon12:25:291
2957173994838,6cyclictest3013272-21cron09:35:002
2957173994836,9cyclictest3009299-21sh09:30:022
2957173994836,9cyclictest3009299-21sh09:30:022
2957173994636,6cyclictest3132494-21sed12:01:052
2957173994636,6cyclictest3132494-21sed12:01:052
2957173994535,6cyclictest2975170-21systemd-run08:10:012
2957173994535,6cyclictest2975170-21systemd-run08:10:012
2957172994535,7cyclictest420-21avahi-daemon10:40:531
2957171994525,4cyclictest3011218-21unixbench_singl09:31:030
2957172994436,5cyclictest3013287-21idleruntime-cro09:35:001
2957172994436,5cyclictest3005249-21cron09:25:011
2957172994335,5cyclictest2976668-21systemd-run08:15:011
2957172994334,6cyclictest3163077-21latency_hist12:40:051
2957172994333,7cyclictest3031015-21processes09:56:001
295717399427,7cyclictest171rcu_preempt10:05:532
2957173994232,6cyclictest3001397-21latency_hist09:20:022
2957173994231,7cyclictest420-21avahi-daemon10:31:002
2957172994234,5cyclictest0-21swapper/108:00:011
2957171994235,4cyclictest3038147-21aten_r2power_cu10:05:390
2957173994131,6cyclictest3075902-21unixbench_multi10:51:032
295717199412,35cyclictest3067331-21ssh10:40:560
2957173994032,5cyclictest2985651-21sh08:45:012
2957173994031,5cyclictest0-21swapper/212:20:422
2957173994029,4cyclictest0-21swapper/210:40:542
2957173994026,11cyclictest3054977-21needreboot10:25:502
2957172994034,3cyclictest0-21swapper/109:40:001
2957173993933,3cyclictest0-21swapper/210:15:012
2957173993931,5cyclictest3074016-21ssh10:50:012
2957172993931,4cyclictest0-21swapper/111:15:021
2957172993930,6cyclictest421-21dbus-daemon12:31:321
2957172993930,4cyclictest0-21swapper/109:25:571
2957172993930,4cyclictest0-21swapper/109:25:571
2957171993932,5cyclictest3131437-21fschecks_time12:00:460
2957171993932,5cyclictest3131437-21fschecks_time12:00:460
2957171993931,5cyclictest3037570-21idleruntime-cro10:05:010
2957171993928,8cyclictest421-21dbus-daemon12:24:340
2957171993928,8cyclictest12350irq/32-eth010:55:540
2957171993924,10cyclictest12350irq/32-eth009:40:540
2957171993917,6cyclictest12450irq/33-eth011:00:530
2957173993832,3cyclictest0-21swapper/211:30:532
2957173993825,10cyclictest2972975-21needreboot08:00:482
2957172993830,5cyclictest2985589-21unixbench_singl08:41:031
2957172993829,6cyclictest3049722-21sendmail10:20:011
2957172993826,9cyclictest3098258-21awk11:20:001
2957172993826,9cyclictest3098258-21awk11:20:001
2957171993830,5cyclictest2982669-21systemd-run08:35:010
2957171993823,12cyclictest2993147-21diskmemload11:30:440
295717199381,31cyclictest3022697-21ssh09:45:520
2957173993730,4cyclictest3151639-21fschecks_time12:25:442
2957173993727,6cyclictest261-21systemd-journal11:09:282
295717299373,27cyclictest2998512-21ssh09:15:521
2957172993728,4cyclictest0-21swapper/110:35:561
2957172993728,4cyclictest0-21swapper/110:35:561
295717299371,28cyclictest3103743-21ssh11:26:001
295717199374,26cyclictest3119863-21ssh11:45:590
2957171993732,4cyclictest2957170-21cyclictest10:40:020
2957171993732,4cyclictest2957170-21cyclictest10:40:020
2957171993730,4cyclictest3106336-21systemd-run11:30:010
2957171993729,6cyclictest2127843-21cups-browsed10:51:100
2957171993715,17cyclictest2127839-21cupsd09:05:560
295717199371,29cyclictest3115789-21sh11:40:550
2957173993628,5cyclictest3158802-21cron12:35:012
2957173993627,3cyclictest0-21swapper/210:35:582
2957173993627,3cyclictest0-21swapper/210:35:582
2957173993624,4cyclictest0-21swapper/209:44:292
2957173993618,13cyclictest1-21systemd11:15:022
2957172993628,5cyclictest3142650-21grep12:15:001
2957172993628,5cyclictest3122456-21cron11:50:011
2957172993628,4cyclictest0-21swapper/111:35:511
2957172993627,5cyclictest0-21swapper/111:30:521
295717199365,21cyclictest12350irq/32-eth012:15:530
2957171993629,4cyclictest421-21dbus-daemon09:16:030
2957171993627,6cyclictest2127843-21cups-browsed12:36:290
2957171993627,6cyclictest2127843-21cups-browsed11:24:020
2957171993627,3cyclictest0-21swapper/010:30:510
2957171993626,7cyclictest3140947-21sshd12:11:450
2957171993623,2cyclictest0-21swapper/009:35:450
2957173993528,4cyclictest2965184-21fschecks_time07:35:442
2957173993528,3cyclictest0-21swapper/209:55:552
2957173993527,5cyclictest1-21systemd11:16:002
2957173993527,5cyclictest1-21systemd11:16:002
2957173993526,5cyclictest0-21swapper/209:50:012
2957173993525,6cyclictest3139288-21aten_r2power_fr12:10:422
2957173993520,9cyclictest2993147-21diskmemload10:15:532
2957173993519,10cyclictest1054-21mta-sts-daemon11:20:512
2957173993515,15cyclictest2127843-21cups-browsed11:00:542
295717299357,19cyclictest2994742-21awk09:11:031
2957172993527,5cyclictest3074893-21fschecks_time10:50:451
2957172993527,5cyclictest3027394-21unixbench_singl09:51:031
2957172993526,6cyclictest3134597-21systemd12:05:021
2957172993526,6cyclictest3134597-21systemd12:05:021
2957172993526,6cyclictest2981177-21systemd-run08:30:021
2957172993526,6cyclictest2127843-21cups-browsed07:13:181
295717299352,27cyclictest3034929-21ssh10:00:541
295717199353,26cyclictest3051137-21ssh10:20:570
2957171993526,6cyclictest753-21ntpd10:25:570
2957171993526,4cyclictest12450irq/33-eth012:30:500
2957171993518,14cyclictest151ktimers/007:50:560
2957171993518,11cyclictest12350irq/32-eth007:55:530
2957173993426,4cyclictest0-21swapper/209:10:562
2957173993425,6cyclictest2127843-21cups-browsed11:27:102
2957173993424,6cyclictest2982663-21systemd-run08:35:012
2957173993422,9cyclictest2988042-21ntp_kernel_pll_08:50:552
295717399341,26cyclictest3127957-21ssh11:56:022
2957172993426,5cyclictest2966086-21unixbench_singl07:36:021
2957172993426,4cyclictest0-21swapper/110:45:581
2957172993425,6cyclictest0-21swapper/111:00:581
2957171993429,4cyclictest2957170-21cyclictest08:10:520
2957171993428,4cyclictest1055-21mta-sts-daemon09:51:110
2957171993427,5cyclictest2127843-21cups-browsed10:49:530
295717399336,18cyclictest428-21mta-sts-daemon08:25:542
2957173993325,5cyclictest2967666-21grep07:45:012
2957173993324,5cyclictest2970643-21cron07:55:012
2957173993323,6cyclictest3124392-21/usr/sbin/munin11:51:062
295717399332,23cyclictest3051072-21ssh10:20:552
2957172993327,3cyclictest0-21swapper/111:10:011
2957172993326,4cyclictest2970586-21sed07:51:031
2957172993325,6cyclictest3050285-21/usr/sbin/munin10:21:041
2957172993325,5cyclictest3082075-21munin-run11:00:011
2957172993325,4cyclictest0-21swapper/110:26:231
2957172993324,6cyclictest2127839-21cupsd08:25:021
2957171993326,4cyclictest0-21swapper/007:15:000
2957171993325,5cyclictest2990266-21latency_hist09:00:020
2957171993325,5cyclictest2990266-21latency_hist09:00:020
2957171993317,11cyclictest12350irq/32-eth012:25:560
295717399324,23cyclictest3014539-21ssh09:35:492
2957173993225,4cyclictest2960163-21systemctl07:20:002
2957173993224,4cyclictest2976668-21systemd-run08:15:012
2957173993223,5cyclictest3027161-21xargs09:51:022
2957173993221,8cyclictest3119094-21aten_r2power_en11:45:432
295717399321,26cyclictest3160090-21ssh12:35:522
2957172993225,4cyclictest3144006-21ntp_kernel_pll_12:15:531
2957172993224,5cyclictest3025450-21munin-plugin-st09:50:011
2957172993224,5cyclictest2963196-21latency_hist07:30:021
2957172993224,5cyclictest2959114-21cpuspeed07:15:431
2957172993223,6cyclictest516-21in:imuxsock10:10:011
2957171993224,6cyclictest2127843-21cups-browsed12:05:510
2957171993224,5cyclictest2127843-21cups-browsed09:26:420
2957171993224,5cyclictest2127843-21cups-browsed09:26:420
2957173993122,7cyclictest2993162-21sh09:10:012
2957173993120,8cyclictest3079642-21mailstats10:56:022
2957172993124,5cyclictest0-21swapper/111:41:001
2957172993122,6cyclictest2977161-21fschecks_count08:15:431
2957172993116,11cyclictest3061922-21latency_hist10:35:021
2957171993125,3cyclictest0-21swapper/007:20:440
2957171993122,5cyclictest0-21swapper/008:50:570
2957171993121,7cyclictest421-21dbus-daemon11:15:470
2957171993121,7cyclictest421-21dbus-daemon11:15:470
2957173993025,2cyclictest0-21swapper/208:20:002
2957173993023,4cyclictest2972155-21idleruntime-cro08:00:012
2957173993023,3cyclictest0-21swapper/211:40:012
2957173993022,5cyclictest2964600-21unixbench_singl07:31:022
2957172993025,2cyclictest0-21swapper/109:00:001
2957172993025,2cyclictest0-21swapper/109:00:001
2957172993024,3cyclictest0-21swapper/111:20:571
2957172993024,3cyclictest0-21swapper/108:40:011
2957172993023,3cyclictest0-21swapper/112:20:491
2957172993022,5cyclictest2987025-21unixbench_multi08:46:031
295717299302,23cyclictest3123928-21sh11:51:001
295717199303,20cyclictest600-21gmain08:45:550
2957171993024,3cyclictest0-21swapper/011:55:010
2957171993024,3cyclictest0-21swapper/010:10:580
2957171993024,3cyclictest0-21swapper/009:25:010
2957171993023,5cyclictest3031096-21/usr/sbin/munin09:56:020
2957171993021,4cyclictest0-21swapper/011:58:290
295717399292,19cyclictest3002696-21sh09:20:582
2957173992921,5cyclictest2983105-21cpuspeed08:35:432
2957173992920,5cyclictest3115808-21ntpq11:40:552
2957173992913,10cyclictest351ktimers/207:45:492
295717299299,14cyclictest271ktimers/108:00:531
2957172992922,4cyclictest2957271-21latency_hist07:10:031
2957172992922,4cyclictest2957271-21latency_hist07:10:031
2957172992922,3cyclictest0-21swapper/109:05:561
2957172992921,5cyclictest1056-21mta-sts-daemon10:10:531
2957172992918,8cyclictest271ktimers/107:20:531
2957171992922,5cyclictest2127843-21cups-browsed09:04:280
2957171992922,5cyclictest2127843-21cups-browsed09:04:280
295717399287,15cyclictest1054-21mta-sts-daemon09:00:512
295717399287,15cyclictest1054-21mta-sts-daemon09:00:512
2957173992821,3cyclictest0-21swapper/210:05:012
2957173992819,6cyclictest2958579-21grep07:11:042
2957172992822,3cyclictest0-21swapper/109:01:001
2957172992822,3cyclictest0-21swapper/109:01:001
2957172992821,4cyclictest1056-21mta-sts-daemon11:59:101
2957172992821,4cyclictest1056-21mta-sts-daemon09:44:591
295717199288,3cyclictest171rcu_preempt10:15:540
2957171992821,4cyclictest2980181-21fschecks_count08:25:460
295717199281,23cyclictest3091948-21sed11:11:010
2957173992720,5cyclictest2957170-21cyclictest07:20:562
2957173992719,5cyclictest439-21systemd-logind08:50:012
2957171992723,3cyclictest2957170-21cyclictest08:00:580
2957171992719,6cyclictest2959801-21switchtime07:16:010
2957173992619,3cyclictest0-21swapper/209:00:012
2957173992619,3cyclictest0-21swapper/209:00:012
2957173992614,10cyclictest351ktimers/208:20:542
2957172992619,3cyclictest0-21swapper/107:30:551
2957171992619,4cyclictest2978050-21unixbench_multi08:16:010
2957173992516,6cyclictest2957126-21/usr/sbin/munin07:10:012
2957173992516,6cyclictest2957126-21/usr/sbin/munin07:10:012
295717299253,14cyclictest2127843-21cups-browsed07:40:501
2957172992519,3cyclictest0-21swapper/108:05:461
2957172992519,3cyclictest0-21swapper/108:05:461
2957171992520,2cyclictest0-21swapper/007:50:010
2957171992519,3cyclictest0-21swapper/008:35:510
2957172992419,4cyclictest2957170-21cyclictest08:31:021
2957172992417,5cyclictest2969159-21munin-plugin-st07:50:011
2957172992416,5cyclictest2988661-21systemd-run08:55:011
295717199242,17cyclictest2979057-21ntpq08:20:520
295717199242,16cyclictest2967033-21ntpq07:40:510
2957171992411,10cyclictest151ktimers/007:25:500
2957171992411,10cyclictest0-21swapper/008:41:030
295717499232,6cyclictest0-21swapper/310:10:503
295717499232,5cyclictest0-21swapper/312:35:563
295717499231,5cyclictest0-21swapper/307:20:513
295717499231,4cyclictest0-21swapper/309:25:543
295717499231,4cyclictest0-21swapper/309:25:543
295717499231,4cyclictest0-21swapper/309:00:543
295717499231,4cyclictest0-21swapper/309:00:543
295717499231,3cyclictest0-21swapper/311:15:543
295717499231,3cyclictest0-21swapper/311:15:543
2957173992317,4cyclictest0-21swapper/207:30:012
2957171992312,7cyclictest151ktimers/007:30:500
295717499223,3cyclictest0-21swapper/311:05:543
295717499221,6cyclictest0-21swapper/308:05:523
295717499221,6cyclictest0-21swapper/308:05:523
295717499221,5cyclictest0-21swapper/309:11:023
295717499221,4cyclictest0-21swapper/309:50:533
295717499221,4cyclictest0-21swapper/308:25:573
295717499221,4cyclictest0-21swapper/308:15:503
295717499221,4cyclictest0-21swapper/308:10:493
295717499221,4cyclictest0-21swapper/307:40:513
295717499221,3cyclictest0-21swapper/311:40:543
295717499221,3cyclictest0-21swapper/311:20:523
295717499221,3cyclictest0-21swapper/310:55:583
295717499221,3cyclictest0-21swapper/310:50:573
295717499221,3cyclictest0-21swapper/309:55:533
295717499221,3cyclictest0-21swapper/309:40:553
295717499221,3cyclictest0-21swapper/308:30:533
295717499221,3cyclictest0-21swapper/307:15:533
295717499220,3cyclictest0-21swapper/310:25:573
2957171992214,6cyclictest2975153-21idleruntime-cro08:10:000
2957171992214,6cyclictest2975153-21idleruntime-cro08:10:000
295717499211,6cyclictest0-21swapper/309:15:513
295717499211,6cyclictest0-21swapper/308:55:533
295717499211,6cyclictest0-21swapper/308:55:533
295717499211,5cyclictest0-21swapper/311:35:523
295717499211,5cyclictest0-21swapper/310:30:533
295717499211,4cyclictest0-21swapper/312:20:583
295717499211,4cyclictest0-21swapper/311:25:583
295717499211,4cyclictest0-21swapper/311:00:553
295717499211,4cyclictest0-21swapper/310:40:523
295717499211,4cyclictest0-21swapper/310:35:523
295717499211,4cyclictest0-21swapper/310:35:523
295717499211,4cyclictest0-21swapper/310:00:453
295717499211,4cyclictest0-21swapper/309:35:523
295717499211,4cyclictest0-21swapper/309:05:503
295717499211,4cyclictest0-21swapper/308:20:493
295717499211,4cyclictest0-21swapper/308:00:513
295717499211,4cyclictest0-21swapper/307:50:503
295717499211,3cyclictest0-21swapper/311:55:563
295717499211,3cyclictest0-21swapper/311:50:523
295717499211,3cyclictest0-21swapper/311:30:513
295717499211,3cyclictest0-21swapper/310:45:583
295717499211,3cyclictest0-21swapper/310:20:593
295717499211,3cyclictest0-21swapper/310:05:483
295717499211,3cyclictest0-21swapper/308:45:523
295717499211,3cyclictest0-21swapper/308:40:513
295717499211,3cyclictest0-21swapper/308:35:503
295717499211,3cyclictest0-21swapper/307:55:533
295717499211,3cyclictest0-21swapper/307:30:563
295717499211,3cyclictest0-21swapper/307:25:513
295717499211,3cyclictest0-21swapper/307:10:583
295717499210,3cyclictest0-21swapper/312:30:493
295717499210,3cyclictest0-21swapper/311:45:553
295717499202,3cyclictest0-21swapper/309:45:503
295717499201,4cyclictest0-21swapper/312:25:553
295717499201,4cyclictest0-21swapper/312:10:493
295717499201,4cyclictest0-21swapper/312:05:303
295717499201,4cyclictest0-21swapper/312:00:523
295717499201,4cyclictest0-21swapper/312:00:523
295717499201,4cyclictest0-21swapper/308:50:533
295717499201,3cyclictest0-21swapper/312:15:503
295717499201,3cyclictest0-21swapper/311:10:403
295717499201,3cyclictest0-21swapper/310:15:503
295717499201,3cyclictest0-21swapper/307:45:513
295717499201,3cyclictest0-21swapper/307:35:513
295717499200,4cyclictest0-21swapper/309:30:423
295717499200,4cyclictest0-21swapper/309:20:453
2957171991915,2cyclictest2954636-21kworker/u8:1+xprtiod07:10:020
2957171991915,2cyclictest2954636-21kworker/u8:1+xprtiod07:10:020
295717499171,4cyclictest0-21swapper/307:10:023
295717499171,4cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional