You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-27 - 10:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Mon Jul 27, 2026 00:43:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3057145995829,21cyclictest265-21systemd-journal20:05:522
3057145995225,23cyclictest0-21swapper/223:55:492
305714599522,23cyclictest3157986-21ssh22:55:492
3057144995142,5cyclictest421-21dbus-daemon23:05:011
3057143995139,10cyclictest3207695-21ssh00:17:460
3057143995139,10cyclictest3207695-21ssh00:17:460
3057143995040,7cyclictest3064118-21sshd19:30:520
3057143994936,10cyclictest3126544-21munin-plugin-st22:05:010
3057145994840,5cyclictest3090145-21systemd-run21:00:012
3057145994840,5cyclictest3090145-21systemd-run21:00:012
3057145994822,6cyclictest3127818-21proc_pri22:05:522
3057143994841,4cyclictest3196310-21cups-browsed00:00:020
3057143994737,7cyclictest1-21systemd00:25:010
3057145994624,3cyclictest0-21swapper/200:00:462
305714399455,33cyclictest3112627-21ssh21:40:500
3057145994436,5cyclictest3096178-21munin-run21:15:012
3057144994438,3cyclictest0-21swapper/122:40:521
3057143994437,5cyclictest421-21dbus-daemon23:20:010
3057145994333,6cyclictest1287407-21cups-browsed22:30:412
305714599433,32cyclictest1287404-21cupsd22:45:462
3057145994327,12cyclictest3170295-21sendmail_mailqu23:15:542
3057144994335,5cyclictest3211481-21ssh00:25:011
3057143994335,6cyclictest3159885-21tr23:00:010
3057143994327,13cyclictest3147790-21latency_hist22:40:020
305714399432,37cyclictest151ktimers/019:25:520
3057145994230,9cyclictest3110000-21timerwakeupswit21:36:012
3057144994234,5cyclictest3199908-21aten2.4-expect00:05:391
3057144994233,6cyclictest3079633-21cron20:25:011
3057145994132,6cyclictest3219173-21unixbench-2d00:35:582
3057145994130,8cyclictest3147722-21cron22:40:012
3057145994130,7cyclictest3166105-21latency_hist23:10:022
305714599411,28cyclictest3154877-21ssh22:50:442
3057144994135,3cyclictest0-21swapper/100:35:521
305714399413,8cyclictest171rcu_preempt21:45:500
3057143994135,4cyclictest3220572-21cut00:40:010
3057145994029,8cyclictest3168557-21sh23:13:452
3057144994031,5cyclictest420-21avahi-daemon19:40:511
3057143994033,5cyclictest421-21dbus-daemon21:15:010
3057143994031,6cyclictest1287404-21cupsd23:09:060
3057143994019,17cyclictest151ktimers/019:50:450
3057143994019,17cyclictest151ktimers/019:50:450
3057143994015,20cyclictest665-21snmpd00:10:520
3057145993931,3cyclictest0-21swapper/220:20:522
3057144993933,4cyclictest3057142-21cyclictest21:35:381
3057144993933,3cyclictest0-21swapper/121:50:501
3057144993931,5cyclictest3175582-21aten_r2power_po23:25:381
3057144993931,5cyclictest3141697-21systemd-run22:30:011
3057144993930,6cyclictest420-21avahi-daemon22:56:041
3057145993832,3cyclictest0-21swapper/221:35:022
3057145993832,3cyclictest0-21swapper/221:35:022
3057145993830,5cyclictest3099241-21idleruntime-cro21:20:012
3057145993818,16cyclictest3136844-21sshd22:20:512
3057144993830,5cyclictest3215538-21needreboot00:30:451
3057144993830,5cyclictest3060479-21aten_r2power_fr19:20:351
3057143993830,5cyclictest421-21dbus-daemon22:30:090
305714399382,30cyclictest3118587-21diskmemload21:50:500
3057143993815,15cyclictest151ktimers/021:55:470
3057143993814,19cyclictest3182244-21sshd23:35:510
3057143993814,19cyclictest3182244-21sshd23:35:510
3057145993730,4cyclictest3193216-21sh23:55:012
3057145993729,5cyclictest3211479-21cron00:25:002
3057144993731,3cyclictest0-21swapper/123:55:541
3057144993730,4cyclictest3120594-21kworker/u8:9-xprtiod22:40:011
3057143993729,5cyclictest3155532-21unixbench_singl22:50:570
3057143993728,4cyclictest0-21swapper/023:20:460
3057143993726,8cyclictest421-21dbus-daemon22:23:220
3057143993724,9cyclictest0-21swapper/022:05:480
305714599369,18cyclictest428-21mta-sts-daemon20:35:462
3057145993628,5cyclictest3181275-21latency_hist23:35:032
3057145993625,8cyclictest3203791-21switchtime00:11:002
3057145993621,9cyclictest0-21swapper/222:15:492
305714599361,28cyclictest3121771-21ssh21:55:542
3057144993630,4cyclictest3064852-21kworker/u8:13+xprtiod19:50:011
3057144993629,3cyclictest0-21swapper/123:20:531
3057144993627,6cyclictest439-21systemd-logind20:20:021
3057144993627,6cyclictest3114447-21munin-run21:45:011
3057144993627,5cyclictest3179452-21mailstats23:30:571
3057143993629,4cyclictest0-21swapper/023:45:380
3057143993628,5cyclictest3097900-21grep21:16:010
3057143993626,7cyclictest3101591-21sshd21:22:520
3057145993527,5cyclictest3178248-21latency_hist23:30:032
3057145993527,5cyclictest1-21systemd21:45:562
3057145993527,4cyclictest3208447-21idleruntime-cro00:20:012
3057145993527,4cyclictest3208447-21idleruntime-cro00:20:012
3057144993527,5cyclictest3132582-21cron22:15:011
3057144993527,5cyclictest3132582-21cron22:15:011
3057144993527,5cyclictest1-21systemd00:20:021
3057144993527,5cyclictest1-21systemd00:20:021
3057144993526,6cyclictest3203086-21gltestperf00:10:431
3057144993526,6cyclictest3138679-21sh22:25:011
305714399353,29cyclictest0-21swapper/023:10:440
305714399353,28cyclictest3200420-21ssh00:05:490
3057143993527,5cyclictest1001-21mta-sts-daemon23:03:350
3057143993526,4cyclictest0-21swapper/022:43:280
3057143993524,5cyclictest0-21swapper/023:40:290
3057143993521,3cyclictest0-21swapper/022:10:480
3057143993521,3cyclictest0-21swapper/022:10:480
3057143993517,14cyclictest151ktimers/019:55:510
3057143993517,14cyclictest151ktimers/019:55:510
3057143993511,20cyclictest3176133-21ssh23:25:480
3057145993421,10cyclictest3216247-21unixbench_singl00:31:002
3057144993429,2cyclictest0-21swapper/122:00:011
3057144993428,3cyclictest0-21swapper/119:20:011
3057144993426,5cyclictest3129558-21ssh22:10:011
305714499341,26cyclictest3185221-21sh23:40:471
305714399343,13cyclictest665-21snmpd00:00:550
3057143993425,6cyclictest3212531-21needreboot00:25:470
305714599333,23cyclictest3118670-21ssh21:50:532
3057145993328,2cyclictest0-21swapper/220:00:002
3057145993328,2cyclictest0-21swapper/220:00:002
3057145993326,4cyclictest3062007-21aten_r2power_po19:25:372
3057145993325,4cyclictest3073790-21latency_hist20:05:022
3057145993325,4cyclictest3073790-21latency_hist20:05:022
3057145993322,7cyclictest0-21swapper/219:20:472
305714599331,25cyclictest3130812-21ssh22:10:542
305714599331,25cyclictest3130812-21ssh22:10:542
3057144993326,3cyclictest0-21swapper/123:35:401
3057144993326,3cyclictest0-21swapper/123:35:401
3057144993325,5cyclictest3152450-21unixbench_multi22:46:001
3057144993325,5cyclictest3090145-21systemd-run21:00:011
3057144993325,5cyclictest3090145-21systemd-run21:00:011
3057144993325,4cyclictest0-21swapper/119:55:501
3057144993325,4cyclictest0-21swapper/119:55:501
3057144993325,3cyclictest0-21swapper/120:40:521
3057144993324,6cyclictest3188842-21grep23:45:581
3057143993325,5cyclictest1287407-21cups-browsed21:33:440
3057143993325,5cyclictest1287407-21cups-browsed21:33:440
3057145993225,3cyclictest0-21swapper/223:45:012
305714499326,21cyclictest3106478-21sh21:30:481
305714499326,21cyclictest3106478-21sh21:30:481
3057144993226,3cyclictest3129597-21kworker/u8:6-rpciod22:30:521
3057144993226,3cyclictest0-21swapper/120:25:451
3057144993225,4cyclictest3133879-21proc_pri22:15:561
3057144993225,4cyclictest3091625-21idleruntime-cro21:05:001
3057144993225,4cyclictest1002-21mta-sts-daemon20:30:471
3057144993225,3cyclictest0-21swapper/123:15:491
3057144993224,5cyclictest3214520-21idleruntime-cro00:30:011
3057144993224,5cyclictest3197405-21ssh00:00:491
3057144993224,5cyclictest3097876-21unixbench_multi21:16:011
3057144993223,6cyclictest3088619-21munin-plugin-st20:55:001
3057144993223,6cyclictest3088619-21munin-plugin-st20:55:001
3057144993221,7cyclictest1287407-21cups-browsed23:53:201
305714499321,27cyclictest3167124-21ssh23:10:481
3057143993226,3cyclictest0-21swapper/021:10:000
3057143993219,9cyclictest12650irq/32-eth023:50:480
3057143993219,10cyclictest0-21swapper/022:20:000
3057143993216,11cyclictest12650irq/32-eth019:10:520
3057145993125,3cyclictest0-21swapper/222:05:022
3057145993124,3cyclictest0-21swapper/200:05:482
3057145993123,5cyclictest0-21swapper/221:45:012
305714599312,24cyclictest3100436-21ssh21:20:532
3057145993114,14cyclictest3160576-21entropy23:00:392
3057144993125,3cyclictest0-21swapper/121:15:011
3057144993124,4cyclictest0-21swapper/119:55:011
3057144993124,4cyclictest0-21swapper/119:55:011
3057144993123,4cyclictest3108394-21kworker/u8:0-events_unbound22:00:371
3057144993118,7cyclictest3103485-21sshd21:25:521
305714399313,21cyclictest1287407-21cups-browsed23:30:460
3057143993124,5cyclictest0-21swapper/020:40:010
3057143993124,4cyclictest1002-21mta-sts-daemon22:25:550
3057143993122,6cyclictest1287407-21cups-browsed21:38:530
3057143993121,4cyclictest3059633-21proc_pri19:15:540
3057143993120,8cyclictest421-21dbus-daemon20:10:480
3057145993024,3cyclictest0-21swapper/221:25:512
3057145993023,3cyclictest0-21swapper/223:20:412
3057145993022,5cyclictest3070618-21cron19:55:012
3057145993022,5cyclictest3070618-21cron19:55:012
3057143993024,3cyclictest3081320-21latency_hist20:30:020
3057143993021,5cyclictest0-21swapper/000:30:470
3057143993018,6cyclictest3093122-21diskmemload21:25:510
3057145992922,5cyclictest0-21swapper/219:35:012
3057145992917,9cyclictest3187761-21cpu23:45:382
3057144992923,3cyclictest0-21swapper/121:10:001
3057144992922,3cyclictest0-21swapper/122:50:441
3057144992921,3cyclictest0-21swapper/121:20:521
3057144992920,5cyclictest0-21swapper/123:05:391
3057144992919,5cyclictest439-21systemd-logind20:15:021
3057144992918,6cyclictest3064118-21sshd19:30:521
3057143992920,7cyclictest3093122-21diskmemload22:45:450
3057145992822,4cyclictest1001-21mta-sts-daemon22:42:152
3057145992820,5cyclictest3139692-21needreboot22:25:452
3057145992820,5cyclictest0-21swapper/200:25:402
305714499282,19cyclictest3115588-21ssh21:45:501
3057144992820,5cyclictest0-21swapper/119:35:491
3057143992823,2cyclictest0-21swapper/020:50:370
3057143992823,2cyclictest0-21swapper/020:50:370
3057143992822,4cyclictest3066000-21unixbench_multi19:36:010
3057143992816,7cyclictest3084994-21modprobe20:40:520
3057145992722,3cyclictest3064852-21kworker/u8:13-xprtiod20:30:012
3057145992721,3cyclictest0-21swapper/223:36:012
3057145992721,3cyclictest0-21swapper/223:36:012
3057145992720,5cyclictest3057142-21cyclictest19:10:522
3057145992720,4cyclictest3087066-21unixbench_singl20:45:572
3057145992720,4cyclictest3087066-21unixbench_singl20:45:572
3057145992720,4cyclictest0-21swapper/219:40:012
3057145992720,3cyclictest0-21swapper/219:40:512
3057144992721,3cyclictest0-21swapper/119:10:581
3057143992721,3cyclictest0-21swapper/020:05:000
3057143992721,3cyclictest0-21swapper/020:05:000
305714399272,11cyclictest3074587-21sshd20:05:520
3057143992720,5cyclictest1287407-21cups-browsed20:30:530
3057143992720,5cyclictest1000-21mta-sts-daemon19:25:030
3057143992719,5cyclictest1-21systemd19:45:020
305714599262,20cyclictest1287407-21cups-browsed20:50:482
305714599262,20cyclictest1287407-21cups-browsed20:50:482
3057145992618,5cyclictest3093119-21sh21:10:002
3057145992618,5cyclictest1287407-21cups-browsed19:45:582
3057145992617,6cyclictest3091629-21latency_hist21:05:022
3057145992616,7cyclictest3059433-21needreboot19:15:452
3057144992619,4cyclictest271ktimers/119:25:521
3057144992618,5cyclictest3075062-21unixbench_singl20:06:011
3057144992617,7cyclictest271ktimers/120:45:481
3057144992617,7cyclictest271ktimers/120:45:481
305714499260,18cyclictest3083580-21sshd20:35:521
3057143992621,3cyclictest3051799-21kworker/u8:8-rpciod19:10:020
3057144992519,3cyclictest0-21swapper/120:01:011
3057144992519,3cyclictest0-21swapper/120:01:011
3057143992519,4cyclictest151ktimers/019:45:510
3057143992519,3cyclictest0-21swapper/021:00:010
3057143992519,3cyclictest0-21swapper/021:00:010
3057143992518,4cyclictest1000-21mta-sts-daemon20:20:510
3057145992415,6cyclictest3075352-21/usr/sbin/munin20:10:512
305714399243,11cyclictest265-21systemd-journal21:05:000
3057143992419,2cyclictest0-21swapper/020:20:010
3057145992317,3cyclictest0-21swapper/220:30:572
3057145992315,5cyclictest3085562-21unixbench_singl20:41:202
3057143992317,2cyclictest0-21swapper/020:50:010
3057143992317,2cyclictest0-21swapper/020:50:010
305714599213,12cyclictest428-21mta-sts-daemon20:15:482
305714599211,10cyclictest421-21dbus-daemon19:10:012
305714699193,7cyclictest0-21swapper/323:00:503
305714499181,13cyclictest3057099-21/usr/sbin/munin19:10:021
305714699173,7cyclictest0-21swapper/300:05:493
305714699172,9cyclictest0-21swapper/321:25:493
305714699172,7cyclictest0-21swapper/322:35:513
305714699172,6cyclictest0-21swapper/323:35:483
305714699172,6cyclictest0-21swapper/323:35:483
305714699171,8cyclictest0-21swapper/321:40:493
305714699171,7cyclictest0-21swapper/323:55:513
305714699163,6cyclictest0-21swapper/300:35:483
305714699163,6cyclictest0-21swapper/300:30:473
305714699162,8cyclictest0-21swapper/323:20:483
305714699162,8cyclictest0-21swapper/321:50:533
305714699162,7cyclictest0-21swapper/323:40:553
305714699162,10cyclictest0-21swapper/321:10:493
305714699161,8cyclictest0-21swapper/322:00:473
305714699154,4cyclictest0-21swapper/320:00:503
305714699154,4cyclictest0-21swapper/320:00:503
305714699153,7cyclictest0-21swapper/321:45:513
305714699153,6cyclictest0-21swapper/323:45:493
305714699153,6cyclictest0-21swapper/321:55:493
305714699152,7cyclictest0-21swapper/323:25:473
305714699152,7cyclictest0-21swapper/322:05:483
305714699152,7cyclictest0-21swapper/300:15:493
305714699152,7cyclictest0-21swapper/300:15:493
305714699152,7cyclictest0-21swapper/300:10:573
305714699151,7cyclictest0-21swapper/321:20:493
305714699151,6cyclictest0-21swapper/322:55:453
305714699151,6cyclictest0-21swapper/319:15:483
305714699143,6cyclictest0-21swapper/322:20:513
305714699143,6cyclictest0-21swapper/320:00:013
305714699143,6cyclictest0-21swapper/320:00:013
305714699142,7cyclictest0-21swapper/323:15:483
305714699142,7cyclictest0-21swapper/322:30:473
305714699142,6cyclictest0-21swapper/323:30:463
305714699142,6cyclictest0-21swapper/322:50:483
305714699142,5cyclictest0-21swapper/319:25:453
305714699141,8cyclictest0-21swapper/322:25:533
305714699141,8cyclictest0-21swapper/320:30:013
305714699141,7cyclictest0-21swapper/322:45:473
305714699141,6cyclictest0-21swapper/320:55:453
305714699141,6cyclictest0-21swapper/320:55:453
305714699141,6cyclictest0-21swapper/320:30:453
305714699141,6cyclictest0-21swapper/319:40:473
305714699141,6cyclictest0-21swapper/319:35:523
305714699133,5cyclictest0-21swapper/319:50:523
305714699133,5cyclictest0-21swapper/319:50:523
305714699132,7cyclictest0-21swapper/322:40:503
305714699132,6cyclictest0-21swapper/322:15:473
305714699132,6cyclictest0-21swapper/321:35:513
305714699131,7cyclictest0-21swapper/320:05:523
305714699131,6cyclictest0-21swapper/321:30:363
305714699131,6cyclictest0-21swapper/321:30:363
305714699131,6cyclictest0-21swapper/321:15:473
305714699131,6cyclictest0-21swapper/320:50:483
305714699131,6cyclictest0-21swapper/320:50:483
305714699131,6cyclictest0-21swapper/320:20:443
305714699123,5cyclictest0-21swapper/323:50:463
305714699122,6cyclictest0-21swapper/300:25:443
305714699122,6cyclictest0-21swapper/300:20:463
305714699121,6cyclictest0-21swapper/320:45:453
305714699121,6cyclictest0-21swapper/320:45:453
305714699121,6cyclictest0-21swapper/320:40:573
305714699121,6cyclictest0-21swapper/319:30:463
305714699121,6cyclictest0-21swapper/319:10:463
305714699121,5cyclictest0-21swapper/320:35:383
305714699121,5cyclictest0-21swapper/320:15:453
305714699121,5cyclictest0-21swapper/319:45:463
305714699113,6cyclictest0-21swapper/300:00:473
305714699112,6cyclictest0-21swapper/323:05:393
305714699112,5cyclictest0-21swapper/322:10:363
305714699112,5cyclictest0-21swapper/322:10:363
305714699112,5cyclictest0-21swapper/320:10:483
305714699111,6cyclictest0-21swapper/323:10:383
305714699111,6cyclictest0-21swapper/321:05:013
305714699111,5cyclictest0-21swapper/321:05:463
30571469992,4cyclictest0-21swapper/319:20:353
30571469991,5cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional