You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-19 - 14:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack2slot7s.osadl.org (updated Sun Oct 19, 2025 12:40:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2480133995847,6cyclictest959-21mta-sts-daemon09:08:571
2480132995237,12cyclictest2507155-21latency_hist08:40:020
2480134995143,5cyclictest2516136-21unin-run09:10:022
2480133995124,24cyclictest2501957-21needreboot08:23:521
2480133994833,12cyclictest2579828-21needreboot10:48:561
248013399462,14cyclictest2656872-21ps12:39:041
248013399453,9cyclictest2573479-21ntpq10:38:551
2480133994525,16cyclictest0-21swapper/112:28:521
2480133994524,18cyclictest0-21swapper/112:10:021
248013399451,25cyclictest2622333-21ssh11:53:551
2480132994537,5cyclictest2615419-21missed_timers11:43:520
2480134994433,8cyclictest2532752-21/usr/sbin/munin09:35:012
248013399443,12cyclictest2516110-21diskmemload10:09:401
2480133994419,21cyclictest0-21swapper/111:40:011
248013399433,21cyclictest2605846-21ssh11:29:011
248013399433,21cyclictest2605846-21ssh11:29:011
2480133994316,24cyclictest2505612-21munin-plugin-st08:35:011
248013399431,15cyclictest2557346-21ssh10:13:531
2480134994234,5cyclictest453-21systemd-logind10:05:022
2480134994234,5cyclictest2613331-21idleruntime-cro11:40:012
2480133994234,6cyclictest2480131-21cyclictest12:34:091
2480133994234,6cyclictest2480131-21cyclictest12:34:091
2480133994229,9cyclictest1-21systemd12:00:021
248013399422,13cyclictest2531912-21ntp_states09:34:091
2480133994219,21cyclictest0-21swapper/111:08:571
2480133994217,21cyclictest2590737-21ssh11:05:011
2480132994234,6cyclictest2609953-21latency_hist11:35:020
2480132994234,6cyclictest2609953-21latency_hist11:35:020
2480132994234,5cyclictest2600530-21latency_hist11:20:030
2480134994135,3cyclictest0-21swapper/210:28:562
2480134994135,3cyclictest0-21swapper/210:28:562
248013399414,11cyclictest421-21avahi-daemon09:59:581
2480133994133,5cyclictest2587476-21sendmail-msp11:00:011
2480133994133,5cyclictest2504132-21systemd-run08:30:011
2480133994132,5cyclictest2577626-21systemd-run10:45:011
2480133994121,17cyclictest0-21swapper/109:40:001
2480133994116,22cyclictest2637944-21ssh12:15:021
2480132994133,3cyclictest0-21swapper/011:55:010
2480134994034,3cyclictest0-21swapper/212:40:012
2480134994031,6cyclictest2609904-21systemd-run11:35:012
2480134994031,6cyclictest2609904-21systemd-run11:35:012
2480134994030,6cyclictest1696603-21cups-browsed11:51:452
248013399407,11cyclictest422-21dbus-daemon09:41:391
248013399403,9cyclictest2615667-21ssh11:44:011
248013399402,15cyclictest2566845-21ssh10:28:531
248013399402,15cyclictest2566845-21ssh10:28:531
2480133994021,16cyclictest0-21swapper/112:03:451
2480133994014,4cyclictest0-21swapper/109:18:531
248013399401,14cyclictest2560044-21irqstats10:18:481
248013399401,13cyclictest2543639-21aten_r2power_cu09:53:431
248013299404,26cyclictest2579965-21ssh10:49:000
2480132994031,6cyclictest1696603-21cups-browsed09:51:460
248013299402,32cyclictest2522250-21ssh09:19:000
2480132994018,16cyclictest422-21dbus-daemon11:03:550
2480134993930,6cyclictest2619165-21mailstats11:49:032
2480134993930,6cyclictest2587479-21systemd-run11:00:012
2480133993930,6cyclictest2595894-21fschecks_time11:13:481
2480133993914,3cyclictest0-21swapper/109:50:021
2480133993913,4cyclictest2498366-21acpi08:13:431
248013399391,17cyclictest2618946-21ssh11:48:551
248013399391,15cyclictest2518695-21ntpq09:13:531
2480132993933,4cyclictest2548525-21idleruntime-cro10:00:010
2480132993931,5cyclictest422-21dbus-daemon11:10:020
2480134993830,5cyclictest2542202-21ssh09:50:012
2480134993829,6cyclictest2560959-21unixbench_singl10:19:072
2480134993828,7cyclictest1696603-21cups-browsed12:02:282
2480134993825,9cyclictest1696603-21cups-browsed11:03:552
248013499382,31cyclictest2633036-21ssh12:09:022
248013399383,10cyclictest437-21mta-sts-daemon09:26:501
2480133993830,4cyclictest0-21swapper/110:33:581
2480133993830,4cyclictest0-21swapper/110:33:581
2480133993826,8cyclictest2644442-21kernelversion12:23:481
248013399382,14cyclictest2487037-21ntpq07:33:551
2480133993818,17cyclictest0-21swapper/108:08:471
2480133993813,5cyclictest0-21swapper/111:24:031
2480133993813,5cyclictest0-21swapper/111:24:031
2480133993813,3cyclictest0-21swapper/110:03:461
248013399381,14cyclictest2526480-21latency_hist09:25:031
248013299382,31cyclictest2557312-21ssh10:13:510
2480134993729,5cyclictest2645873-21idleruntime-cro12:25:012
2480134993729,5cyclictest2637896-21cron12:15:012
2480134993729,5cyclictest2625141-21diskmemload11:58:452
2480134993729,5cyclictest2606722-21ls11:30:002
2480134993729,5cyclictest2606722-21ls11:30:002
2480134993728,6cyclictest2498119-21munin-plugin-st08:10:012
2480134993728,6cyclictest1-21systemd08:15:022
2480134993727,6cyclictest421-21avahi-daemon12:17:082
2480134993726,8cyclictest2593565-21timerwakeupswit11:09:022
2480134993725,9cyclictest2545306-21/usr/sbin/munin09:55:022
2480133993711,24cyclictest2600385-21latency_hist11:20:031
248013299373,28cyclictest2531788-21ssh09:34:030
2480132993729,5cyclictest2481639-21systemd-run07:15:020
2480132993727,7cyclictest2568105-21latency_hist10:30:030
2480132993727,7cyclictest2568105-21latency_hist10:30:030
2480134993629,4cyclictest2535914-21munin-plugin-st09:40:002
2480134993629,4cyclictest0-21swapper/208:00:012
2480134993629,3cyclictest0-21swapper/209:00:012
2480134993628,5cyclictest2577626-21systemd-run10:45:012
2480133993629,4cyclictest1-21systemd07:20:021
2480133993614,2cyclictest0-21swapper/108:20:031
2480132993624,9cyclictest2656267-21fschecks_count12:38:490
2480132993624,9cyclictest2495000-21tail07:59:020
2480132993614,18cyclictest2541179-21ssh09:48:530
2480134993527,5cyclictest2600390-21ls11:20:012
2480134993527,5cyclictest2580927-21munin-run10:50:012
2480134993527,5cyclictest2505994-21aten_r2power_po08:39:082
2480134993527,5cyclictest2481510-21unixbench_multi07:14:092
2480134993526,5cyclictest421-21avahi-daemon10:22:292
2480134993522,4cyclictest0-21swapper/210:38:572
248013399352,13cyclictest2494604-21ntpq07:58:561
2480133993512,4cyclictest2480629-21sh07:13:471
2480133993510,3cyclictest0-21swapper/108:03:521
248013299356,8cyclictest171rcu_preempt08:18:530
2480132993523,9cyclictest2490647-21kworker/u8:11+xprtiod08:25:010
2480132993522,9cyclictest0-21swapper/009:13:580
2480132993515,15cyclictest151ktimers/008:08:580
248013499343,24cyclictest2547471-21ssh09:58:542
2480134993428,3cyclictest0-21swapper/208:20:022
2480134993425,5cyclictest2558374-21ssh10:15:022
2480133993413,4cyclictest2640098-21aten2.4-expect12:18:441
248013399341,12cyclictest2609023-21ntpq11:34:151
248013399341,12cyclictest2609023-21ntpq11:34:151
248013399341,10cyclictest2516110-21diskmemload10:52:591
248013299343,25cyclictest2636862-21ssh12:13:580
2480132993428,3cyclictest0-21swapper/009:43:550
2480132993415,13cyclictest12450irq/32-eth012:23:530
2480134993325,5cyclictest2523176-21sendmail09:20:012
2480134993324,6cyclictest959-21mta-sts-daemon12:34:012
2480134993324,6cyclictest959-21mta-sts-daemon12:34:012
2480134993324,6cyclictest0-21swapper/210:55:032
2480134993324,3cyclictest0-21swapper/210:09:032
2480133993325,5cyclictest2514521-21unixbench_multi09:04:061
2480133993324,6cyclictest421-21avahi-daemon07:10:021
2480133993324,6cyclictest2511570-21unixbench_singl08:54:021
2480133993323,7cyclictest2490735-21latency_hist07:45:021
2480133993323,7cyclictest2490735-21latency_hist07:45:021
2480132993326,5cyclictest2513131-21systemd-run09:00:010
2480132993325,5cyclictest421-21avahi-daemon08:43:580
2480132993325,5cyclictest2585896-21aten_r2power_cu10:58:430
2480132993325,4cyclictest151ktimers/011:48:580
2480132993323,6cyclictest2629233-21ntp_offset12:03:550
2480132993310,15cyclictest2583269-21ssh10:53:580
248013499323,5cyclictest171rcu_preempt08:48:582
2480134993224,5cyclictest955-21mta-sts-daemon08:33:532
248013499322,20cyclictest2518723-21ssh09:13:542
248013399324,10cyclictest1994697-21sendmail-mta07:53:581
2480133993212,4cyclictest2468980-21kworker/u8:14+xprtiod07:30:021
2480132993226,3cyclictest955-21mta-sts-daemon12:27:360
2480132993223,7cyclictest2480131-21cyclictest07:44:110
2480132993223,7cyclictest2480131-21cyclictest07:44:110
2480132993221,8cyclictest2632740-21missed_timers12:08:520
2480132993210,13cyclictest151ktimers/010:23:560
2480134993124,4cyclictest2511223-21sendmail_mailtr08:53:582
2480134993124,4cyclictest0-21swapper/211:12:112
2480134993122,3cyclictest0-21swapper/209:28:582
2480134993121,6cyclictest2503162-21fschecks_count08:28:542
248013399311,13cyclictest2563533-21/usr/sbin/munin10:23:541
2480132993122,4cyclictest0-21swapper/009:23:550
2480132993120,6cyclictest2574606-21latency_hist10:40:020
248013499303,21cyclictest2648811-21ntpq12:28:572
2480134993025,2cyclictest0-21swapper/210:34:052
2480134993025,2cyclictest0-21swapper/210:34:052
2480134993024,3cyclictest0-21swapper/211:45:012
2480134993024,3cyclictest0-21swapper/209:23:422
2480134993021,6cyclictest2602265-21fschecks_count11:23:462
2480134993021,6cyclictest2602265-21fschecks_count11:23:462
2480134993021,4cyclictest0-21swapper/209:40:332
2480132993024,4cyclictest2653848-21ssh12:35:010
2480132993024,4cyclictest2653848-21ssh12:35:010
2480132993024,3cyclictest0-21swapper/009:39:000
2480132993023,4cyclictest2641327-21unixbench-2d12:19:070
2480132993018,8cyclictest151ktimers/008:33:540
2480132993016,9cyclictest12450irq/32-eth010:03:550
2480132993016,12cyclictest2569471-21aten_r2power_fr10:33:430
2480132993016,12cyclictest2569471-21aten_r2power_fr10:33:430
2480132993015,11cyclictest2560362-21ssh10:18:540
2480134992923,3cyclictest0-21swapper/209:03:582
2480134992921,5cyclictest2486092-21grep07:29:112
2480133992922,3cyclictest0-21swapper/107:23:451
2480133992919,8cyclictest2491814-21switchtime07:49:021
2480132992918,6cyclictest151ktimers/007:29:040
2480132992914,12cyclictest151ktimers/011:29:020
2480132992914,12cyclictest151ktimers/011:29:020
248013299291,25cyclictest657-21snmpd10:09:030
2480134992822,3cyclictest0-21swapper/207:34:012
2480134992817,8cyclictest2482853-21timerandwakeup07:19:022
2480132992822,4cyclictest2483249-21latency_hist07:20:020
2480132992813,11cyclictest12450irq/32-eth011:59:000
2480134992719,5cyclictest2502623-21munin-plugin-st08:25:012
2480132992722,2cyclictest0-21swapper/009:28:460
2480132992719,6cyclictest2612177-21needreboot11:38:500
2480132992718,7cyclictest266-21systemd-journal09:10:010
2480133992618,5cyclictest2507665-21fschecks_time08:43:451
248013299266,13cyclictest12450irq/32-eth010:43:540
2480132992620,4cyclictest2493598-21cron07:55:000
248013299261,20cyclictest2602757-21ssh11:23:580
248013299261,20cyclictest2602757-21ssh11:23:580
248013599255,9cyclictest0-21swapper/310:28:563
248013599255,9cyclictest0-21swapper/310:28:563
248013499256,14cyclictest437-21mta-sts-daemon08:43:542
2480134992519,3cyclictest0-21swapper/207:40:012
2480134992517,5cyclictest2496054-21ntpq08:03:552
2480134992514,8cyclictest2484023-21ntp_kernel_pll_07:23:552
248013399251,18cyclictest657-21snmpd08:58:531
2480132992520,3cyclictest956-21mta-sts-daemon07:33:510
2480132992519,3cyclictest0-21swapper/011:15:010
2480132992513,8cyclictest422-21dbus-daemon08:53:510
2480134992416,5cyclictest2492660-21fschecks_time07:53:452
2480133992418,3cyclictest0-21swapper/107:40:011
2480133992416,5cyclictest2510132-21systemd-run08:50:021
248013299249,10cyclictest422-21dbus-daemon07:38:560
248013299244,14cyclictest437-21mta-sts-daemon07:48:510
2480134992318,2cyclictest0-21swapper/207:48:432
2480134992313,6cyclictest2480143-21systemd-run07:10:022
2480133992316,4cyclictest957-21mta-sts-daemon08:37:061
248013299222,14cyclictest2499112-21ntpq08:13:590
2480132992212,7cyclictest2503539-21ntp_kernel_pll_08:29:020
248013299212,14cyclictest2514020-21ntpq09:03:540
2480132992116,2cyclictest0-21swapper/008:49:040
248013599203,10cyclictest0-21swapper/311:43:593
248013599203,10cyclictest0-21swapper/307:48:553
2480134992014,3cyclictest0-21swapper/207:41:112
2480134992014,3cyclictest0-21swapper/207:41:112
2480132992015,2cyclictest0-21swapper/007:24:070
248013599193,9cyclictest0-21swapper/312:18:583
248013599193,9cyclictest0-21swapper/310:13:533
248013599192,10cyclictest0-21swapper/310:38:533
248013599191,11cyclictest0-21swapper/309:18:573
248013599191,10cyclictest0-21swapper/311:08:513
2480132991915,2cyclictest0-21swapper/007:10:020
248013299191,13cyclictest2496164-21ps08:03:590
248013599183,6cyclictest0-21swapper/307:18:573
248013599183,10cyclictest0-21swapper/309:58:583
248013599182,9cyclictest0-21swapper/311:58:523
248013599182,8cyclictest0-21swapper/309:28:553
248013599182,8cyclictest0-21swapper/308:13:553
248013599182,10cyclictest0-21swapper/312:23:513
248013599182,10cyclictest0-21swapper/311:18:593
248013599182,10cyclictest0-21swapper/311:03:553
248013599181,7cyclictest0-21swapper/308:28:593
248013599174,8cyclictest0-21swapper/311:28:573
248013599174,8cyclictest0-21swapper/311:28:573
248013599174,5cyclictest0-21swapper/308:43:543
248013599173,9cyclictest0-21swapper/309:43:563
248013599173,8cyclictest0-21swapper/309:49:013
248013599173,7cyclictest0-21swapper/312:03:533
248013599172,9cyclictest0-21swapper/312:38:563
248013599172,9cyclictest0-21swapper/312:08:553
248013599172,9cyclictest0-21swapper/310:58:523
248013599172,8cyclictest0-21swapper/311:49:013
248013599172,8cyclictest0-21swapper/310:48:573
248013599171,8cyclictest0-21swapper/308:39:263
248013599171,7cyclictest0-21swapper/308:48:523
248013599171,7cyclictest0-21swapper/307:23:553
248013599164,7cyclictest0-21swapper/312:13:483
248013599163,8cyclictest0-21swapper/310:18:543
248013599163,7cyclictest0-21swapper/311:53:533
248013599162,9cyclictest0-21swapper/311:38:593
248013599162,8cyclictest0-21swapper/312:33:563
248013599162,8cyclictest0-21swapper/312:33:563
248013599162,8cyclictest0-21swapper/311:13:543
248013599162,8cyclictest0-21swapper/310:08:573
248013599162,8cyclictest0-21swapper/309:34:033
248013599162,8cyclictest0-21swapper/309:23:553
248013599162,10cyclictest0-21swapper/308:03:523
248013599161,9cyclictest0-21swapper/310:23:553
248013599161,8cyclictest0-21swapper/307:33:563
248013599154,6cyclictest0-21swapper/310:43:483
248013599154,4cyclictest0-21swapper/307:43:553
248013599154,4cyclictest0-21swapper/307:43:553
248013599153,7cyclictest0-21swapper/311:34:113
248013599153,7cyclictest0-21swapper/311:34:113
248013599153,6cyclictest0-21swapper/309:03:563
248013599152,9cyclictest0-21swapper/309:12:323
248013599152,8cyclictest0-21swapper/312:28:433
248013599152,8cyclictest0-21swapper/310:53:543
248013599152,8cyclictest0-21swapper/310:03:593
248013599152,7cyclictest0-21swapper/308:18:553
248013599152,6cyclictest0-21swapper/307:13:543
248013599151,7cyclictest0-21swapper/309:38:553
248013599151,7cyclictest0-21swapper/308:33:593
248013599143,6cyclictest0-21swapper/311:23:513
248013599143,6cyclictest0-21swapper/311:23:513
248013599142,8cyclictest0-21swapper/310:33:443
248013599142,8cyclictest0-21swapper/310:33:443
248013599142,7cyclictest0-21swapper/308:09:023
248013599141,8cyclictest0-21swapper/308:24:003
248013599141,6cyclictest0-21swapper/307:58:503
248013599132,6cyclictest0-21swapper/309:51:463
248013599132,6cyclictest0-21swapper/308:58:503
248013599132,6cyclictest0-21swapper/308:53:503
248013599131,6cyclictest0-21swapper/307:29:003
248013599122,7cyclictest0-21swapper/309:10:013
248013599121,6cyclictest0-21swapper/307:53:523
248013599121,6cyclictest0-21swapper/307:38:553
24801359982,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional