You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 05:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Mon Jul 07, 2025 00:41:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2466890998165,11cyclictest0-21swapper/222:08:022
2466889995244,5cyclictest2593992-21munin-run23:20:011
2466890995142,6cyclictest2468393-21systemd-run19:15:022
2466890994738,6cyclictest422-21dbus-daemon23:58:032
2466889994637,6cyclictest0-21swapper/100:00:021
2466888994638,6cyclictest2466887-21cyclictest21:12:580
2466888994638,6cyclictest2466887-21cyclictest21:12:580
2466890994537,4cyclictest0-21swapper/221:52:562
2466889994510,20cyclictest28-21ksoftirqd/121:48:021
246688999446,33cyclictest2540365-21ssh22:03:001
2466889994436,5cyclictest422-21dbus-daemon22:08:061
2466889994436,5cyclictest2557521-21/usr/sbin/munin22:27:511
2466890994335,4cyclictest0-21swapper/221:58:002
2466889994337,2cyclictest0-21swapper/120:20:011
2466890994235,4cyclictest2510344-21sendmail-msp21:20:012
2466889994234,5cyclictest2552231-21cron22:20:011
246688999422,35cyclictest2512487-21ssh21:23:061
2466888994236,4cyclictest422-21dbus-daemon23:35:010
2466888994234,5cyclictest2607863-21systemd-run23:40:020
246688899421,35cyclictest0-21swapper/022:22:570
2466889994133,5cyclictest2633832-21aten_r2power_po00:17:471
2466889994132,6cyclictest2510360-21ssh21:20:011
2466890994033,3cyclictest0-21swapper/223:12:562
2466890994032,5cyclictest2495373-21munin-plugin-st20:45:012
2466890994026,11cyclictest2603418-21sh23:33:092
2466889994032,4cyclictest955-21mta-sts-daemon23:35:021
2466889994028,9cyclictest2599125-21tr23:27:511
2466889994027,9cyclictest421-21avahi-daemon21:33:071
2466888994032,5cyclictest2628977-21cron00:10:010
2466888994028,9cyclictest0-21swapper/022:00:010
2466890993931,5cyclictest2638420-21unixbench_multi00:23:102
2466890993930,6cyclictest266-21systemd-journal21:25:012
2466889993932,3cyclictest0-21swapper/100:08:011
2466889993930,5cyclictest2583661-21debian-sa123:05:011
2466889993930,5cyclictest2522269-21ssh21:37:471
2466889993929,6cyclictest2475855-21cron19:40:011
2466888993931,5cyclictest2600918-21systemd-run23:30:010
2466890993829,6cyclictest491-21gdbus20:17:482
2466890993829,5cyclictest0-21swapper/223:28:062
2466889993832,3cyclictest0-21swapper/121:58:001
2466889993829,5cyclictest2503340-21sh21:10:301
2466889993829,5cyclictest2503340-21sh21:10:301
2466889993829,4cyclictest0-21swapper/121:43:091
2466889993825,8cyclictest421-21avahi-daemon21:27:571
2466888993833,2cyclictest0-21swapper/021:25:010
2466888993831,5cyclictest2527909-21latency_hist21:45:020
2466890993729,5cyclictest477-21rs:main2
2466890993729,5cyclictest477-21rs:main2
2466890993727,7cyclictest2606119-21ssh23:37:492
2466890993725,9cyclictest2523356-21timerwakeupswit21:38:092
246688999373,26cyclictest2624227-21ssh00:03:021
246688999373,25cyclictest2568407-21ssh22:43:021
2466889993731,3cyclictest0-21swapper/100:15:011
2466889993730,3cyclictest0-21swapper/123:45:011
2466889993729,4cyclictest0-21swapper/123:40:011
2466889993728,5cyclictest2590535-21systemd-run23:15:011
2466889993727,7cyclictest421-21avahi-daemon22:36:241
2466889993727,7cyclictest421-21avahi-daemon22:36:241
2466889993727,7cyclictest2644607-21/usr/sbin/munin00:35:021
2466889993726,8cyclictest684-21munin-node00:23:121
246688999372,25cyclictest2613431-21ssh23:47:581
246688899373,23cyclictest12450irq/32-eth022:08:030
2466888993729,5cyclictest421-21avahi-daemon21:18:100
246688899370,31cyclictest151ktimers/023:03:010
2466890993628,5cyclictest266-21systemd-journal20:00:002
2466890993628,5cyclictest2574581-21aten_r2power_cu22:52:442
2466890993627,6cyclictest422-21dbus-daemon00:25:102
2466890993627,4cyclictest0-21swapper/221:42:572
246688999364,25cyclictest2575372-21ssh22:52:581
2466889993624,10cyclictest2646981-21unixbench_singl00:38:121
2466888993629,4cyclictest2587218-21latency_hist23:10:020
2466888993628,5cyclictest421-21avahi-daemon23:46:390
2466888993622,11cyclictest2644593-21cron00:35:010
2466890993530,2cyclictest0-21swapper/223:20:012
2466890993530,2cyclictest0-21swapper/222:40:012
2466890993530,2cyclictest0-21swapper/222:40:012
2466890993530,2cyclictest0-21swapper/200:02:592
2466890993526,6cyclictest2515749-21needreboot21:27:552
2466889993526,6cyclictest271ktimers/120:27:561
2466889993526,5cyclictest0-21swapper/100:28:121
2466889993525,8cyclictest959-21mta-sts-daemon20:03:011
2466889993525,7cyclictest688540-21cups-browsed23:07:261
2466889993525,6cyclictest2579027-21mailstats22:58:061
2466889993525,6cyclictest2579027-21mailstats22:58:061
2466889993520,4cyclictest171rcu_preempt22:12:551
2466888993531,2cyclictest0-21swapper/020:00:010
2466888993529,4cyclictest959-21mta-sts-daemon00:00:320
2466888993528,5cyclictest0-21swapper/000:20:010
2466888993523,9cyclictest0-21swapper/000:15:010
246688899352,28cyclictest2529853-21diskmemload21:47:570
2466888993521,8cyclictest12450irq/32-eth023:43:030
246689099343,25cyclictest657-21snmpd23:53:022
2466890993429,2cyclictest0-21swapper/222:35:012
2466890993427,3cyclictest0-21swapper/200:12:582
2466890993426,5cyclictest2645996-21sort00:37:532
2466890993426,5cyclictest2550495-21if_eth022:17:512
2466890993426,5cyclictest2548818-21idleruntime-cro22:15:012
2466890993424,5cyclictest0-21swapper/223:07:582
2466890993423,8cyclictest2578712-21needreboot22:57:542
2466890993423,8cyclictest2578712-21needreboot22:57:542
246689099342,27cyclictest421-21avahi-daemon00:07:562
246689099342,24cyclictest657-21snmpd22:03:012
246689099342,23cyclictest2643443-21ssh00:32:582
2466889993427,3cyclictest0-21swapper/121:53:051
2466889993417,4cyclictest171rcu_preempt22:47:551
2466888993428,4cyclictest2580441-21latency_hist23:00:030
2466888993428,4cyclictest2580441-21latency_hist23:00:030
2466888993427,4cyclictest2523423-21unixbench_multi21:38:100
2466888993426,5cyclictest688540-21cups-browsed22:17:110
2466888993426,5cyclictest2641792-21systemd-run00:30:010
2466888993426,5cyclictest2559302-21unin-run22:30:020
2466888993425,6cyclictest2593056-21timerwakeupswit23:18:100
246688899342,24cyclictest2533450-21ssh21:53:040
2466890993327,3cyclictest2492364-21idleruntime-cro20:35:012
2466890993327,3cyclictest0-21swapper/222:50:012
2466890993325,5cyclictest2582190-21ssh23:02:592
2466890993325,5cyclictest2518896-21ssh21:32:502
2466890993325,5cyclictest2469924-21latency_hist19:20:022
2466890993324,6cyclictest2502871-21munin-plugin-st21:10:012
2466890993324,6cyclictest2483395-21systemd-run20:05:012
2466889993324,6cyclictest2496883-21systemd-run20:50:011
2466889993323,3cyclictest171rcu_preempt20:25:021
2466889993323,3cyclictest171rcu_preempt20:25:021
2466889993317,3cyclictest171rcu_preempt21:03:021
2466889993313,4cyclictest171rcu_preempt19:47:571
2466889993313,4cyclictest171rcu_preempt19:47:571
246688999331,25cyclictest2617118-21ssh23:53:041
2466888993326,4cyclictest2575558-21mailstats22:53:030
2466888993324,6cyclictest959-21mta-sts-daemon23:23:200
2466890993226,4cyclictest2476419-21fschecks_time19:42:522
2466890993226,3cyclictest0-21swapper/222:30:012
2466890993224,5cyclictest2610467-21unixbench-2d23:43:112
246688999323,21cyclictest2561280-21ssh22:32:551
2466889993224,5cyclictest657-21snmpd21:00:031
2466889993224,5cyclictest0-21swapper/120:00:001
2466889993223,4cyclictest0-21swapper/123:23:061
2466889993222,6cyclictest688540-21cups-browsed19:17:511
2466889993222,6cyclictest2492389-21systemd-run20:35:011
2466888993226,4cyclictest2496870-21grep20:50:010
246688899322,21cyclictest2564919-21ssh22:38:030
246688899322,21cyclictest2564919-21ssh22:38:030
2466888993213,15cyclictest2561388-21ssh22:33:000
2466890993124,4cyclictest2505838-21sed21:13:112
2466890993124,4cyclictest2505838-21sed21:13:112
2466889993123,4cyclictest0-21swapper/122:25:011
2466889993116,3cyclictest171rcu_preempt20:37:581
2466888993125,4cyclictest2590599-21latency_hist23:15:020
2466888993125,4cyclictest2568916-21grep22:43:100
2466888993125,4cyclictest2502858-21cron21:10:010
2466888993125,3cyclictest0-21swapper/022:13:040
2466888993125,3cyclictest0-21swapper/019:55:010
2466888993125,3cyclictest0-21swapper/019:55:010
2466888993124,5cyclictest2469963-21latency_hist19:20:020
2466888993123,5cyclictest688540-21cups-browsed21:29:470
2466888993123,5cyclictest2499730-21unixbench_multi20:58:070
2466888993121,7cyclictest2622257-21latency_hist00:00:030
2466888993121,2cyclictest0-21swapper/022:47:510
2466888993120,2cyclictest0-21swapper/000:40:030
2466890993024,3cyclictest0-21swapper/223:23:012
2466890993023,3cyclictest0-21swapper/222:42:592
246689099302,22cyclictest2502865-21diskmemload21:47:572
2466890993021,6cyclictest2613319-21missed_timers23:47:562
2466889993024,3cyclictest959-21mta-sts-daemon21:08:061
2466889993024,3cyclictest0-21swapper/119:25:011
2466889993020,7cyclictest271ktimers/119:52:561
2466889993020,7cyclictest271ktimers/119:52:561
2466889993017,3cyclictest171rcu_preempt20:43:011
246688899304,12cyclictest2502865-21diskmemload22:03:450
2466888993023,3cyclictest0-21swapper/021:33:060
246689099296,18cyclictest2554373-21ssh22:22:562
2466890992921,6cyclictest0-21swapper/219:50:022
2466890992921,6cyclictest0-21swapper/219:50:022
2466889992920,5cyclictest688540-21cups-browsed19:32:521
2466888992923,4cyclictest2639364-21latency_hist00:25:030
2466890992822,3cyclictest0-21swapper/220:10:012
2466890992822,3cyclictest0-21swapper/220:10:012
2466890992820,5cyclictest2471376-21cron19:25:012
246688999288,3cyclictest171rcu_preempt19:43:021
2466889992819,5cyclictest2472828-21unixbench_singl19:28:061
2466889992814,3cyclictest171rcu_preempt19:12:541
2466889992812,3cyclictest171rcu_preempt20:53:021
2466888992822,3cyclictest2566780-21kworker/u8:6-xprtiod23:53:100
2466888992820,5cyclictest453-21systemd-logind20:05:010
2466888992818,7cyclictest2475679-21timerwakeupswit19:38:140
246689099272,10cyclictest171rcu_preempt00:18:012
2466890992720,3cyclictest0-21swapper/219:33:022
2466890992719,5cyclictest688540-21cups-browsed19:36:432
246688999278,5cyclictest271ktimers/120:12:511
2466888992722,2cyclictest0-21swapper/019:30:010
2466888992721,4cyclictest2498356-21latency_hist20:55:010
2466888992719,5cyclictest2470198-21aten_r2power_en19:22:450
2466888992717,7cyclictest2501368-21cron21:05:010
2466888992716,8cyclictest2477328-21unixbench_singl19:43:100
2466890992620,3cyclictest0-21swapper/221:00:012
246688999264,7cyclictest28-21ksoftirqd/120:08:011
246688999264,7cyclictest28-21ksoftirqd/120:08:011
2466888992620,3cyclictest0-21swapper/020:22:570
2466888992620,3cyclictest0-21swapper/020:22:570
2466888992619,4cyclictest0-21swapper/020:08:010
2466888992619,4cyclictest0-21swapper/020:08:010
246688899261,8cyclictest2492360-21sh20:35:010
2466888992618,6cyclictest698-21cron20:45:010
2466888992612,9cyclictest266-21systemd-journal20:17:000
246689099256,11cyclictest437-21mta-sts-daemon19:27:582
246689099253,6cyclictest171rcu_preempt20:47:572
2466890992516,7cyclictest2501313-21unixbench_singl21:03:102
2466888992516,6cyclictest151ktimers/019:47:570
2466888992516,6cyclictest151ktimers/019:47:570
2466888992515,8cyclictest151ktimers/019:14:310
246689099243,14cyclictest437-21mta-sts-daemon20:13:012
2466890992418,3cyclictest0-21swapper/220:40:012
2466890992416,6cyclictest955-21mta-sts-daemon20:27:512
2466890992413,2cyclictest0-21swapper/219:10:012
246688899245,3cyclictest171rcu_preempt20:27:570
2466888992415,7cyclictest2473109-21/usr/sbin/munin19:32:520
2466888992414,9cyclictest0-21swapper/020:12:540
2466888992410,11cyclictest0-21swapper/020:40:010
246689099231,5cyclictest171rcu_preempt20:52:582
2466890992213,6cyclictest2480011-21/usr/sbin/munin19:53:082
2466890992213,6cyclictest2480011-21/usr/sbin/munin19:53:082
246689199202,9cyclictest0-21swapper/322:08:033
246688999200,3cyclictest2466985-21latency_hist19:10:021
2466888991911,6cyclictest0-21swapper/019:10:020
246689199162,5cyclictest0-21swapper/322:57:553
246689199162,5cyclictest0-21swapper/322:57:553
246689199161,7cyclictest0-21swapper/322:22:573
246689199153,3cyclictest0-21swapper/320:23:043
246689199153,3cyclictest0-21swapper/320:23:043
246689199152,8cyclictest0-21swapper/322:52:583
246689199151,7cyclictest0-21swapper/323:43:033
246689199151,7cyclictest0-21swapper/320:08:013
246689199151,7cyclictest0-21swapper/320:08:013
246689199151,7cyclictest0-21swapper/300:38:053
246689199143,3cyclictest0-21swapper/319:37:573
246689199142,7cyclictest0-21swapper/322:43:013
246689199141,6cyclictest0-21swapper/323:03:033
246689199141,6cyclictest0-21swapper/322:12:563
246689199141,6cyclictest0-21swapper/319:43:023
246689199141,6cyclictest0-21swapper/300:07:583
246689199141,5cyclictest0-21swapper/319:52:553
246689199141,5cyclictest0-21swapper/319:52:553
246689199133,7cyclictest0-21swapper/321:27:573
246689199132,7cyclictest0-21swapper/321:50:063
246689199132,6cyclictest0-21swapper/323:47:513
246689199132,5cyclictest0-21swapper/300:17:583
246689199131,7cyclictest0-21swapper/322:03:013
246689199131,6cyclictest0-21swapper/323:12:553
246689199131,6cyclictest0-21swapper/322:38:033
246689199131,6cyclictest0-21swapper/322:38:033
246689199131,6cyclictest0-21swapper/321:47:543
246689199131,6cyclictest0-21swapper/321:23:023
246689199131,6cyclictest0-21swapper/320:17:543
246689199131,6cyclictest0-21swapper/320:03:023
246689199131,6cyclictest0-21swapper/319:57:573
246689199131,5cyclictest0-21swapper/321:02:573
246689199131,5cyclictest0-21swapper/319:17:563
246689199131,5cyclictest0-21swapper/319:12:553
246689199122,6cyclictest0-21swapper/322:47:513
246689199122,6cyclictest0-21swapper/321:13:053
246689199122,6cyclictest0-21swapper/321:13:053
246689199122,5cyclictest0-21swapper/323:58:003
246689199122,5cyclictest0-21swapper/323:27:583
246689199122,5cyclictest0-21swapper/319:28:013
246689199121,7cyclictest0-21swapper/323:35:513
246689199121,6cyclictest0-21swapper/323:07:523
246689199121,5cyclictest0-21swapper/320:38:053
246689199121,5cyclictest0-21swapper/320:32:553
246689199121,4cyclictest0-21swapper/320:55:033
246689199112,4cyclictest0-21swapper/319:47:543
246689199112,4cyclictest0-21swapper/319:47:543
246689199111,6cyclictest0-21swapper/323:32:463
246689199111,6cyclictest0-21swapper/323:17:583
246689199111,6cyclictest0-21swapper/322:17:583
246689199111,6cyclictest0-21swapper/321:42:573
246689199111,6cyclictest0-21swapper/320:42:593
246689199111,6cyclictest0-21swapper/300:13:013
246689199111,5cyclictest0-21swapper/322:25:513
246689199111,5cyclictest0-21swapper/321:57:483
246689199111,5cyclictest0-21swapper/321:32:473
246689199111,5cyclictest0-21swapper/319:22:553
246689199110,6cyclictest0-21swapper/300:03:023
246689199102,5cyclictest0-21swapper/323:52:483
246689199101,5cyclictest0-21swapper/322:32:463
246689199101,5cyclictest0-21swapper/321:35:503
246689199101,5cyclictest0-21swapper/320:57:563
246689199101,5cyclictest0-21swapper/320:12:533
246689199101,5cyclictest0-21swapper/300:28:143
24668919994,3cyclictest0-21swapper/320:47:583
24668919992,4cyclictest0-21swapper/321:17:503
24668919991,5cyclictest0-21swapper/323:21:053
24668919991,5cyclictest0-21swapper/320:27:513
24668919991,5cyclictest0-21swapper/319:32:463
24668919991,5cyclictest0-21swapper/300:32:463
24668919991,5cyclictest0-21swapper/300:22:483
24668919981,4cyclictest0-21swapper/321:06:513
24668919981,4cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional