You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 14:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu May 21, 2026 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2695152991092,5cyclictest131rcu_preempt19:55:022
2695156998679,3cyclictest37-21ksoftirqd/319:45:023
2695146997419,26cyclictest12-21ksoftirqd/000:24:320
2695152996828,11cyclictest309-21in:imuxsock20:04:592
2695152996729,9cyclictest30-21ksoftirqd/200:19:242
2695152996729,9cyclictest30-21ksoftirqd/200:19:232
2695146996762,2cyclictest0-21swapper/019:30:020
2695146996750,13cyclictest2744887-21rm22:40:000
2695146996733,13cyclictest2783708-21ssh00:00:040
2695152996630,9cyclictest30-21ksoftirqd/222:22:512
2695146996630,17cyclictest12-21ksoftirqd/023:37:030
2695152996226,11cyclictest30-21ksoftirqd/222:25:082
269514699622,9cyclictest2718508-21latency_hist21:45:010
2695152996043,14cyclictest30-21ksoftirqd/220:30:032
2695146996052,6cyclictest12-21ksoftirqd/021:10:010
2695146996031,13cyclictest2799053-21ssh00:31:580
2695146996025,15cyclictest27363871ssh22:22:080
2695146995932,12cyclictest2741034-21ssh22:31:480
269514699582,8cyclictest0-21swapper/000:20:000
269514699582,8cyclictest0-21swapper/000:20:000
2695146995828,15cyclictest2723841-21ssh21:56:030
2695146995828,14cyclictest2796619-21ssh00:26:540
2695146995825,16cyclictest2726257-21ssh22:01:060
269514699575,46cyclictest2700781-21modprobe21:00:000
2695146995738,15cyclictest16850irq/134-800000021:51:420
2695146995729,13cyclictest2759591-21ssh23:10:150
2695146995726,15cyclictest2706961-21ssh21:21:080
2695152995647,6cyclictest2696159-21kworker/2:122:10:002
2695146995638,4cyclictest16850irq/134-800000021:20:010
269514699562,39cyclictest2713478-21/usr/sbin/munin21:35:010
2695146995613,25cyclictest12-21ksoftirqd/023:45:010
269514699558,44cyclictest12-21ksoftirqd/000:08:080
2695146995546,5cyclictest16950irq/224-800000023:00:010
2695146995528,12cyclictest2755768-21ssh23:02:230
2695146995528,12cyclictest2710129-21ssh21:27:530
2695146995528,12cyclictest2710129-21ssh21:27:520
2695146995526,8cyclictest12-21ksoftirqd/023:45:350
2695146995526,14cyclictest2746313-21ssh22:42:480
2695146995525,14cyclictest2714749-21ssh21:37:250
2695148995423,15cyclictest2752282-21ssh22:55:021
269514699545,45cyclictest12-21ksoftirqd/023:31:020
2695146995426,13cyclictest2780935-21ssh23:54:450
2695146995426,12cyclictest2719670-21ssh21:47:310
2695146995426,12cyclictest2719670-21ssh21:47:300
2695146995416,34cyclictest12-21ksoftirqd/023:30:010
2695146995416,34cyclictest12-21ksoftirqd/023:30:000
2695146995414,36cyclictest12-21ksoftirqd/022:19:460
269514699541,38cyclictest2751903-21ssh22:54:360
2695156995321,22cyclictest37-21ksoftirqd/323:17:283
2695152995345,5cyclictest2700293-21kworker/2:322:50:012
269514699533,13cyclictest1-21systemd00:05:010
2695146995324,14cyclictest2764943-21ssh23:21:210
2695152995244,5cyclictest2752119-21kworker/2:022:59:272
2695152995244,5cyclictest2752119-21kworker/2:000:03:582
2695152995243,5cyclictest2696159-21kworker/2:121:45:012
2695152995241,8cyclictest0-21swapper/219:10:022
2695152995239,7cyclictest2700293-21kworker/2:323:05:072
269514699528,5cyclictest131rcu_preempt22:07:070
2695146995240,9cyclictest0-21swapper/000:12:270
269514699523,43cyclictest2732784-21munin-run22:14:590
2695156995135,11cyclictest275-21dbus-daemon19:20:013
2695152995143,5cyclictest2752119-21kworker/2:023:40:282
2695152995143,5cyclictest2696159-21kworker/2:121:55:222
2695152995142,6cyclictest2696159-21kworker/2:121:27:382
2695152995142,6cyclictest2696159-21kworker/2:121:27:372
2695152995138,7cyclictest30-21ksoftirqd/200:40:022
2695152995132,14cyclictest3467602-21systemd-journal21:19:592
269514899519,6cyclictest131rcu_preempt23:43:281
2695148995133,15cyclictest1-21systemd00:15:001
2695148995112,5cyclictest131rcu_preempt00:23:061
2695146995126,10cyclictest2801118-21ssh00:36:050
2695146995125,11cyclictest2701657-21ssh21:10:130
2695156995034,10cyclictest1-21systemd21:50:003
2695156995034,10cyclictest1-21systemd21:50:003
2695152995042,5cyclictest2752119-21kworker/2:023:17:202
2695152995042,5cyclictest2700293-21kworker/2:323:37:552
2695152995041,6cyclictest2752119-21kworker/2:000:11:402
2695152995041,6cyclictest2700293-21kworker/2:322:35:022
2695152995039,6cyclictest2752119-21kworker/2:000:20:152
2695148995013,6cyclictest131rcu_preempt22:54:441
2695146995037,7cyclictest2696637-21sh19:40:010
2695146995032,13cyclictest275-21dbus-daemon20:20:010
2695152994941,5cyclictest2752119-21kworker/2:023:20:512
2695152994941,5cyclictest2752119-21kworker/2:023:10:562
2695152994941,5cyclictest2700293-21kworker/2:322:36:422
2695152994939,6cyclictest2696159-21kworker/2:121:36:162
269515299492,9cyclictest2699741-21cron20:40:002
2695148994933,12cyclictest275-21dbus-daemon21:50:011
2695148994933,12cyclictest275-21dbus-daemon21:50:001
269514699492,41cyclictest2759344-21systemd-run23:10:000
2695156994823,21cyclictest37-21ksoftirqd/322:01:083
2695152994840,5cyclictest2752119-21kworker/2:023:29:142
2695152994840,5cyclictest2752119-21kworker/2:023:29:132
2695152994840,5cyclictest2700293-21kworker/2:322:16:262
2695152994840,5cyclictest2686701-21kworker/2:220:10:012
2695152994839,6cyclictest2700293-21kworker/2:321:10:232
269514899485,23cyclictest2694915-21kworker/u8:219:15:021
2695146994832,12cyclictest12-21ksoftirqd/019:20:010
2695156994737,4cyclictest37-21ksoftirqd/300:00:003
2695156994723,5cyclictest37-21ksoftirqd/322:52:593
269515299478,4cyclictest131rcu_preempt22:53:252
2695152994741,3cyclictest2700293-21kworker/2:321:33:592
2695152994740,4cyclictest2696159-21kworker/2:121:54:332
2695152994739,5cyclictest2700293-21kworker/2:300:26:422
2695152994733,8cyclictest275-21dbus-daemon20:45:012
2695152994729,13cyclictest275-21dbus-daemon19:20:002
2695148994728,14cyclictest1-21systemd00:35:011
269514899472,13cyclictest2773463-21sh23:39:131
269514699474,8cyclictest2697191-21wc19:50:010
2695146994723,20cyclictest2710859-21kworker/u8:222:27:160
269515699464,3cyclictest131rcu_preempt21:58:053
2695156994630,12cyclictest37-21ksoftirqd/300:20:013
2695156994630,12cyclictest37-21ksoftirqd/300:20:003
2695156994627,14cyclictest275-21dbus-daemon20:15:023
2695156994627,14cyclictest275-21dbus-daemon20:15:013
2695156994621,21cyclictest37-21ksoftirqd/300:20:463
2695152994638,5cyclictest2752119-21kworker/2:023:49:442
2695152994638,5cyclictest2700293-21kworker/2:323:30:592
2695152994638,5cyclictest2700293-21kworker/2:323:02:262
2695152994638,5cyclictest2700293-21kworker/2:300:30:592
2695152994638,5cyclictest2696159-21kworker/2:121:45:452
2695152994638,5cyclictest2696159-21kworker/2:121:45:452
269514899463,31cyclictest23-21ksoftirqd/123:30:031
269514899463,31cyclictest23-21ksoftirqd/123:30:021
2695146994632,11cyclictest1-21systemd19:25:000
2695146994629,12cyclictest3467602-21systemd-journal20:50:010
2695146994617,14cyclictest2749460-21ssh22:49:310
269515699452,8cyclictest0-21swapper/322:07:163
269515699452,3cyclictest131rcu_preempt21:35:013
2695156994522,20cyclictest37-21ksoftirqd/300:10:163
2695156994522,19cyclictest37-21ksoftirqd/321:22:333
2695156994521,20cyclictest37-21ksoftirqd/323:28:293
2695156994521,20cyclictest37-21ksoftirqd/323:28:293
2695156994520,21cyclictest37-21ksoftirqd/300:09:003
2695148994531,11cyclictest2766667-21sed23:25:021
2695148994530,12cyclictest275-21dbus-daemon22:05:001
269514899451,9cyclictest2717464-21ssh21:43:051
2695146994533,9cyclictest1-21systemd19:55:010
2695146994532,9cyclictest16950irq/224-800000023:17:310
2695156994435,5cyclictest2699016-21latency_hist20:25:023
269515699443,3cyclictest131rcu_preempt21:28:273
269515699443,3cyclictest131rcu_preempt21:28:273
2695156994432,9cyclictest37-21ksoftirqd/323:40:033
269515699441,3cyclictest131rcu_preempt23:01:053
2695152994437,5cyclictest2696159-21kworker/2:122:03:592
2695152994435,5cyclictest2700293-21kworker/2:321:20:292
2695152994429,11cyclictest275-21dbus-daemon00:00:022
2695148994431,9cyclictest1-21systemd23:10:021
2695148994427,12cyclictest275-21dbus-daemon22:50:001
269514899441,3cyclictest131rcu_preempt19:25:001
2695148994410,6cyclictest131rcu_preempt22:27:591
2695146994440,2cyclictest0-21swapper/019:45:020
2695156994331,8cyclictest3467602-21systemd-journal00:28:093
269515699432,3cyclictest131rcu_preempt23:23:553
2695156994320,20cyclictest37-21ksoftirqd/322:55:543
2695156994319,20cyclictest37-21ksoftirqd/321:35:223
2695156994318,21cyclictest37-21ksoftirqd/322:26:083
2695156994318,20cyclictest37-21ksoftirqd/321:10:363
269515699431,3cyclictest131rcu_preempt00:40:023
2695152994335,5cyclictest2700293-21kworker/2:323:51:472
2695152994335,5cyclictest2700293-21kworker/2:322:44:552
2695152994334,5cyclictest2696159-21kworker/2:122:11:442
2695152994331,9cyclictest3467602-21systemd-journal20:17:102
269515299431,3cyclictest131rcu_preempt19:30:022
2695148994330,10cyclictest3467602-21systemd-journal21:10:011
2695148994330,10cyclictest23-21ksoftirqd/121:25:031
2695148994326,12cyclictest275-21dbus-daemon22:00:011
269514899431,3cyclictest131rcu_preempt23:58:351
2695148994313,18cyclictest1-21systemd00:30:011
269514699436,9cyclictest12-21ksoftirqd/019:10:010
2695146994336,4cyclictest210-21jbd2/mmcblk1p2-20:31:300
2695146994330,9cyclictest2701055-21cron21:05:010
269515699428,8cyclictest37-21ksoftirqd/323:45:013
2695156994230,10cyclictest37-21ksoftirqd/319:24:593
2695156994217,3cyclictest37-21ksoftirqd/323:35:013
2695152994230,9cyclictest30-21ksoftirqd/219:25:022
2695152994226,11cyclictest275-21dbus-daemon21:00:012
269514899429,5cyclictest131rcu_preempt23:49:571
2695148994230,9cyclictest2779822-21sshd23:52:091
269514699425,26cyclictest2698997-21munin-plugin-st20:25:000
2695146994235,4cyclictest16950irq/224-800000020:00:010
269515699414,3cyclictest131rcu_preempt22:40:023
2695156994130,8cyclictest2776972-21sshd23:46:103
2695156994129,9cyclictest37-21ksoftirqd/322:30:463
2695156994128,9cyclictest37-21ksoftirqd/320:59:593
2695156994127,10cyclictest1-21systemd22:15:003
2695156994119,19cyclictest37-21ksoftirqd/322:42:163
269515699411,3cyclictest131rcu_preempt19:15:023
2695148994126,11cyclictest2723334-21date21:55:021
2695148994126,11cyclictest2696747-21sed19:40:021
2695146994114,5cyclictest131rcu_preempt19:15:020
269515699404,3cyclictest131rcu_preempt21:10:023
2695156994031,5cyclictest37-21ksoftirqd/320:50:023
2695156994031,5cyclictest37-21ksoftirqd/320:00:023
2695156994024,12cyclictest309-21in:imuxsock20:16:593
269515699401,5cyclictest131rcu_preempt21:45:023
2695156994010,5cyclictest131rcu_preempt22:49:113
269515299405,8cyclictest30-21ksoftirqd/220:32:102
2695152994027,10cyclictest30-21ksoftirqd/200:07:322
269514899408,6cyclictest131rcu_preempt22:12:041
269514899404,3cyclictest131rcu_preempt21:38:051
2695148994030,7cyclictest23-21ksoftirqd/100:15:041
2695148994030,7cyclictest23-21ksoftirqd/100:15:041
2695148994028,10cyclictest23-21ksoftirqd/100:09:151
2695148994025,11cyclictest23-21ksoftirqd/123:03:101
269514899401,5cyclictest2785140-21ssh00:03:181
2695148994015,6cyclictest131rcu_preempt21:20:021
269515699396,4cyclictest131rcu_preempt00:00:213
2695156993927,9cyclictest2780216-21sshd23:53:093
2695156993926,9cyclictest37-21ksoftirqd/300:32:323
2695156993925,10cyclictest37-21ksoftirqd/321:15:373
2695156993918,18cyclictest37-21ksoftirqd/321:50:143
2695156993918,17cyclictest37-21ksoftirqd/322:16:183
269515699391,3cyclictest131rcu_preempt23:13:063
2695152993931,5cyclictest2700293-21kworker/2:321:10:032
2695152993930,6cyclictest2686701-21kworker/2:219:43:102
2695148993929,8cyclictest23-21ksoftirqd/122:33:371
2695148993929,8cyclictest23-21ksoftirqd/122:24:411
2695148993929,7cyclictest23-21ksoftirqd/123:10:231
2695148993928,7cyclictest3467602-21systemd-journal19:44:591
2695148993925,11cyclictest0-21swapper/122:10:021
2695148993921,14cyclictest3467602-21systemd-journal00:40:021
2695148993911,5cyclictest346771850lldpd19:59:421
2695146993925,9cyclictest2698205-21munin-run20:10:010
2695146993924,11cyclictest2700517-21sshd20:53:090
2695156993829,5cyclictest37-21ksoftirqd/320:40:023
2695156993821,13cyclictest3467602-21systemd-journal19:40:003
2695152993830,5cyclictest2696159-21kworker/2:120:45:132
2695152993814,17cyclictest275-21dbus-daemon19:15:022
2695148993829,7cyclictest23-21ksoftirqd/123:17:271
2695148993827,9cyclictest23-21ksoftirqd/119:35:021
2695148993823,10cyclictest23-21ksoftirqd/120:20:011
2695148993822,12cyclictest2697156-21cron19:50:011
2695148993820,13cyclictest275-21dbus-daemon20:10:001
2695148993817,8cyclictest23-21ksoftirqd/120:30:021
269514899381,3cyclictest131rcu_preempt22:16:031
269514899381,3cyclictest131rcu_preempt21:25:491
269514899381,3cyclictest131rcu_preempt21:25:481
269514699387,23cyclictest2699242-21modprobe20:30:000
269514699386,18cyclictest2696376-21modprobe19:35:000
2695156993729,4cyclictest37-21ksoftirqd/320:10:003
2695156993725,9cyclictest37-21ksoftirqd/323:09:483
2695156993723,12cyclictest37-21ksoftirqd/322:23:043
2695152993725,9cyclictest30-21ksoftirqd/220:53:092
2695152993725,7cyclictest2697673-21sh20:00:002
269514899374,4cyclictest131rcu_preempt23:31:321
2695148993727,7cyclictest2698434-21sshd20:13:091
2695148993727,7cyclictest2698434-21sshd20:13:081
2695148993726,9cyclictest23-21ksoftirqd/121:34:151
2695146993725,8cyclictest0-21swapper/020:42:530
2695146993712,19cyclictest3467602-21systemd-journal20:40:000
269515699363,3cyclictest131rcu_preempt20:30:013
2695156993623,9cyclictest1-21systemd19:35:013
2695156993622,11cyclictest37-21ksoftirqd/319:54:593
2695156993621,11cyclictest37-21ksoftirqd/320:50:093
2695152993625,9cyclictest30-21ksoftirqd/219:30:002
269514899367,4cyclictest131rcu_preempt22:41:411
2695148993624,6cyclictest2698051-21cat20:05:021
2695148993623,9cyclictest23-21ksoftirqd/120:57:081
2695148993623,10cyclictest23-21ksoftirqd/120:25:011
2695148993620,11cyclictest3467602-21systemd-journal20:40:021
269514899361,3cyclictest131rcu_preempt22:40:011
269514699362,7cyclictest1-21systemd20:00:080
2695152993521,10cyclictest30-21ksoftirqd/219:39:192
269515299351,3cyclictest131rcu_preempt20:15:012
269515299351,3cyclictest131rcu_preempt20:15:012
2695148993525,8cyclictest23-21ksoftirqd/119:55:001
2695148993523,9cyclictest23-21ksoftirqd/121:10:141
2695156993426,4cyclictest37-21ksoftirqd/321:05:023
2695156993423,9cyclictest37-21ksoftirqd/320:44:593
269515299343,3cyclictest131rcu_preempt20:25:012
2695152993421,10cyclictest30-21ksoftirqd/219:50:012
2695148993424,6cyclictest23-21ksoftirqd/119:20:031
2695148993423,8cyclictest23-21ksoftirqd/120:45:011
269514899341,3cyclictest131rcu_preempt20:50:021
269514699341,29cyclictest0-21swapper/020:15:030
269514699341,29cyclictest0-21swapper/020:15:020
269515299331,4cyclictest131rcu_preempt21:05:002
2695148993324,7cyclictest23-21ksoftirqd/119:30:011
269515699324,3cyclictest131rcu_preempt20:00:023
269515699323,3cyclictest131rcu_preempt19:10:013
269515699323,23cyclictest2699504-21grep20:34:593
269515699321,3cyclictest131rcu_preempt19:50:033
2695148993223,7cyclictest23-21ksoftirqd/120:55:011
2695156993121,6cyclictest37-21ksoftirqd/319:30:023
2695148993124,4cyclictest2701055-21cron21:05:011
269514899302,3cyclictest131rcu_preempt20:35:011
2695148993020,7cyclictest0-21swapper/119:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional