You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-03 - 05:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Jun 03, 2026 00:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
270159799813,17cyclictest2712344-21munin-run21:20:010
270159799811,78cyclictest0-21swapper/019:40:010
2701597998036,41cyclictest16850irq/134-800000022:05:000
270159799801,74cyclictest0-21swapper/000:00:590
2701597997824,50cyclictest2783988-21ssh23:58:310
2701597997819,42cyclictest16850irq/134-800000000:17:240
2701597997817,56cyclictest16850irq/134-800000021:39:210
2701597997816,6cyclictest16850irq/134-800000000:36:380
2701597997816,6cyclictest16850irq/134-800000000:36:380
2701597997540,31cyclictest16850irq/134-800000000:35:000
270161299742,41cyclictest2726362-21ssh21:50:463
270160899747,54cyclictest2707860-21ssh21:10:012
2701597997430,40cyclictest2788407-21ssh00:08:090
2701597997424,33cyclictest2760494-21ssh23:06:150
270159799741,54cyclictest0-21swapper/000:22:070
270159799721,54cyclictest2765938-21ssh23:18:250
2701600997124,16cyclictest1-21systemd22:50:001
2701597997136,31cyclictest16950irq/224-800000023:00:000
270159799701,53cyclictest0-21swapper/022:30:260
2701597996924,41cyclictest0-21swapper/021:46:200
2701597996916,6cyclictest16850irq/134-800000023:01:170
2701612996825,39cyclictest2707778-21diskmemload21:25:453
270161299681,42cyclictest2789709-21ssh00:10:533
2701600996832,10cyclictest23-21ksoftirqd/121:55:011
2701597996821,42cyclictest0-21swapper/000:27:230
2701597996821,42cyclictest0-21swapper/000:27:230
2701597996814,37cyclictest16850irq/134-800000021:42:400
270159799681,2cyclictest0-21swapper/023:51:100
270159799681,11cyclictest2734863-21systemd-run22:10:010
2701612996717,21cyclictest2707778-21diskmemload00:05:453
270160899673,48cyclictest2721608-21sh21:40:032
2701597996718,44cyclictest2737308-21ls22:15:020
2701597996717,15cyclictest2763512-21ssh23:13:010
2701597996712,5cyclictest16850irq/134-800000021:24:290
2701597996712,5cyclictest16850irq/134-800000021:24:290
270161299662,42cyclictest2707778-21diskmemload00:36:043
270161299662,42cyclictest2707778-21diskmemload00:36:043
270161299661,60cyclictest2718853-21ssh21:34:283
2701597996625,36cyclictest2790714-21ssh00:13:210
2701597996624,38cyclictest2728475-21sh21:55:290
2701612996529,32cyclictest2721370-21sendmail21:40:013
270161299652,58cyclictest2744367-21ssh22:30:273
2701600996530,12cyclictest3467602-21systemd-journal23:08:021
2701597996521,39cyclictest2726368-21ssh21:50:470
2701612996433,28cyclictest3467602-21systemd-journal23:21:303
270161299642,58cyclictest2769670-21ssh23:26:333
270161299642,56cyclictest2729189-21ssh21:57:143
270161299641,58cyclictest2751029-21ssh22:45:133
2701597996425,35cyclictest16950irq/224-800000021:35:010
270159799642,18cyclictest2702085-21sendmail-msp19:20:010
270159799642,10cyclictest1-21systemd21:03:270
270159799641,5cyclictest2739421-21munin-run22:20:000
270159799641,59cyclictest0-21swapper/022:43:560
2701612996326,33cyclictest2707778-21diskmemload21:45:593
270161299632,56cyclictest2772131-21ssh23:31:553
270161299632,56cyclictest2764881-21ssh23:15:543
2701612996319,40cyclictest2778019-21latency_hist23:45:013
270161299631,58cyclictest2786818-21ssh00:04:473
270161299631,57cyclictest2763827-21ssh23:13:453
2701597996311,12cyclictest2741469-21ssh22:24:220
270161299622,56cyclictest2758037-21ssh23:00:423
270161299622,56cyclictest2753794-21ssh22:51:333
270161299622,56cyclictest2753794-21ssh22:51:333
270161299622,56cyclictest2747716-21ssh22:38:133
270161299622,56cyclictest2732604-21munin-run22:05:013
270161299622,56cyclictest2714294-21ssh21:24:223
270161299622,56cyclictest2714294-21ssh21:24:223
270161299622,55cyclictest2778773-21ssh23:46:353
270161299622,55cyclictest2703389-21munin-run19:45:013
270161299621,57cyclictest2784084-21ssh23:58:453
270161299621,57cyclictest2756172-21ssh22:56:453
270161299621,57cyclictest2750053-21ssh22:43:233
270161299621,57cyclictest2740485-21ssh22:22:013
270161299621,57cyclictest2737421-21ssh22:15:083
270161299621,57cyclictest2735216-21ssh22:10:203
270161299621,56cyclictest2794359-21ssh00:21:103
270161299621,56cyclictest2792530-21ssh00:17:183
270161299621,56cyclictest2774063-21ssh23:36:073
270161299621,14cyclictest2733247-21ssh22:06:033
2701597996228,30cyclictest2715394-21ssh21:26:270
2701597996225,33cyclictest2748144-21rm22:39:110
2701597996224,34cyclictest2777343-21ssh23:43:320
2701597996219,38cyclictest2710104-21ssh21:15:020
2701597996211,37cyclictest16850irq/134-800000023:23:470
2701612996126,31cyclictest2702567-21sshd19:27:243
270161299611,56cyclictest2760568-21ssh23:06:253
270161299611,56cyclictest2741975-21ssh22:25:073
270161299611,56cyclictest2711753-21ssh21:18:463
270161299611,55cyclictest2797058-21ssh00:27:173
270161299611,55cyclictest2797058-21ssh00:27:173
270161299611,34cyclictest2709804-21ssh21:14:333
270159799611,2cyclictest0-21swapper/023:27:090
2701612996032,23cyclictest309-21in:imuxsock21:00:283
270161299602,53cyclictest2798794-21ssh00:31:003
270161299602,53cyclictest2705937-21sshd20:33:263
270161299601,55cyclictest2721743-21ssh21:40:303
2701597996026,30cyclictest12-21ksoftirqd/020:20:020
2701597996020,36cyclictest2747602-21kworker/u8:122:47:200
270159799601,4cyclictest0-21swapper/023:32:550
270159799601,19cyclictest0-21swapper/022:27:310
270161299591,53cyclictest2702855-21munin-run19:35:013
270161299591,12cyclictest2782012-21ssh23:54:073
270159799591,54cyclictest12-21ksoftirqd/023:49:450
270159799583,16cyclictest2704434-21sort20:05:020
2701597995828,15cyclictest2753716-21ssh22:51:220
2701597995828,15cyclictest2753716-21ssh22:51:220
270159799582,21cyclictest3467602-21systemd-journal20:40:000
270161299572,50cyclictest2706501-21munin-run20:45:013
2701608995742,11cyclictest1-21systemd19:20:002
2701597995720,33cyclictest2774434-21sh23:37:010
270159799571,16cyclictest2703851-21sshd19:52:050
270159799571,16cyclictest2703851-21sshd19:52:050
270161299561,4cyclictest2706179-21sshd20:37:183
2701608995645,7cyclictest2700802-21kworker/2:023:27:092
270159799562,17cyclictest2707244-21cron21:00:000
270159799562,17cyclictest2704158-21sendmail-msp20:00:000
2701597995619,4cyclictest2705120-21latency_hist20:15:020
270161299552,48cyclictest1-21systemd19:50:013
270161299551,50cyclictest2706966-21sshd20:52:273
2701608995545,7cyclictest2710024-21kworker/2:222:16:522
2701597995520,31cyclictest2703624-21latency_hist19:50:020
270159799551,23cyclictest0-21swapper/019:31:120
270161299542,47cyclictest2702351-21munin-run19:25:013
270161299541,49cyclictest2705432-21sshd20:23:253
270161299541,49cyclictest2704106-21sshd19:58:263
270161299541,49cyclictest2701678-21latency_hist19:10:013
270161299541,48cyclictest2703070-21sshd19:37:253
270160899546,8cyclictest30-21ksoftirqd/221:25:012
270160899546,8cyclictest30-21ksoftirqd/221:25:012
2701608995445,5cyclictest2700802-21kworker/2:022:07:272
2701608995435,14cyclictest3467602-21systemd-journal21:30:002
2701600995440,10cyclictest1-21systemd22:40:001
270159799542,17cyclictest2706740-21munin-run20:50:010
2701597995417,10cyclictest12-21ksoftirqd/020:25:010
2701597995411,7cyclictest0-21swapper/020:35:000
270161299532,46cyclictest364-21snmpd20:45:253
270161299532,46cyclictest2705212-21systemctl20:20:003
270161299532,46cyclictest2703858-21sshd19:53:253
270161299532,46cyclictest2703858-21sshd19:53:253
270161299531,48cyclictest364-21snmpd19:16:093
2701608995344,6cyclictest2755256-21kworker/2:123:14:222
2701608995342,7cyclictest2700802-21kworker/2:023:48:292
2701608995336,12cyclictest1-21systemd19:45:002
270161299521,47cyclictest364-21snmpd19:11:093
270161299521,46cyclictest2705706-21munin-run20:30:003
2701608995244,5cyclictest2700802-21kworker/2:023:01:472
2701608995243,6cyclictest2755256-21kworker/2:100:31:312
2701608995243,6cyclictest2755256-21kworker/2:100:14:292
2701608995243,6cyclictest2700802-21kworker/2:021:54:512
2701608995240,8cyclictest30-21ksoftirqd/200:30:002
2701608995240,8cyclictest30-21ksoftirqd/200:30:002
2701600995235,13cyclictest1-21systemd23:55:001
270161299513,44cyclictest364-21snmpd20:12:153
270161299511,46cyclictest2707225-21sshd20:59:273
2701608995143,5cyclictest2755256-21kworker/2:123:16:272
2701608995143,5cyclictest2755256-21kworker/2:100:36:102
2701608995143,5cyclictest2755256-21kworker/2:100:36:092
2701608995143,5cyclictest2710024-21kworker/2:222:11:232
2701608995143,5cyclictest2710024-21kworker/2:221:55:412
2701608995143,5cyclictest2710024-21kworker/2:221:33:032
2701608995143,5cyclictest2700802-21kworker/2:022:27:062
2701608995143,5cyclictest2700802-21kworker/2:000:22:222
2701608995142,6cyclictest2755256-21kworker/2:100:09:152
2701608995142,6cyclictest2700802-21kworker/2:022:52:252
2701608995142,6cyclictest2700802-21kworker/2:022:52:252
2701608995142,5cyclictest2700802-21kworker/2:023:07:122
2701608995141,6cyclictest2755256-21kworker/2:123:57:222
2701608995141,6cyclictest2700802-21kworker/2:023:22:512
270159799512,18cyclictest2705688-21sshd20:29:270
270159799511,18cyclictest0-21swapper/021:06:310
270161299501,44cyclictest2707787-21sh21:10:003
270161299501,11cyclictest48-21kcompactd020:09:403
2701608995043,4cyclictest2700802-21kworker/2:000:19:522
2701608995042,5cyclictest2755256-21kworker/2:123:42:402
2701608995042,5cyclictest2755256-21kworker/2:123:35:132
2701608995042,5cyclictest2700802-21kworker/2:022:55:132
2701608995042,5cyclictest2700802-21kworker/2:021:45:472
2701608995042,5cyclictest2700802-21kworker/2:021:17:112
2701608995042,5cyclictest2700802-21kworker/2:021:12:032
2701608995041,6cyclictest2710024-21kworker/2:222:40:262
2701608995041,6cyclictest2710024-21kworker/2:221:35:432
2701608995041,6cyclictest2700802-21kworker/2:023:51:542
2701608995040,6cyclictest2700802-21kworker/2:000:02:502
2701608995036,10cyclictest1-21systemd19:30:002
2701608995026,17cyclictest3467602-21systemd-journal23:35:002
2701608994941,5cyclictest2710024-21kworker/2:222:00:502
2701608994941,5cyclictest2700802-21kworker/2:022:47:332
2701608994941,5cyclictest2700802-21kworker/2:022:38:252
2701608994941,5cyclictest2700802-21kworker/2:022:33:052
2701608994941,5cyclictest2700802-21kworker/2:022:21:412
2701600994933,12cyclictest23-21ksoftirqd/121:30:001
270159799493,4cyclictest2701619-21latency_hist19:10:010
270159799492,14cyclictest2702577-21cron19:30:000
2701608994840,5cyclictest2700802-21kworker/2:020:40:442
2701608994839,6cyclictest2700802-21kworker/2:020:16:082
2701608994839,6cyclictest2700802-21kworker/2:020:02:282
2701608994839,5cyclictest2677749-21kworker/2:119:45:252
2701608994838,7cyclictest2700802-21kworker/2:020:08:432
2701597994815,9cyclictest12-21ksoftirqd/019:45:000
270161299473,41cyclictest2704399-21modprobe20:05:003
2701608994740,4cyclictest2700802-21kworker/2:019:50:032
2701608994740,4cyclictest2700802-21kworker/2:019:50:032
2701608994739,5cyclictest2700802-21kworker/2:021:04:402
2701608994739,5cyclictest2700802-21kworker/2:020:52:582
2701608994739,5cyclictest2700802-21kworker/2:020:30:062
2701608994739,5cyclictest2700802-21kworker/2:020:26:092
2701608994739,5cyclictest2700802-21kworker/2:020:21:262
2701608994739,5cyclictest2700802-21kworker/2:020:13:322
2701608994739,5cyclictest2700802-21kworker/2:019:55:172
2701608994739,5cyclictest2677749-21kworker/2:119:14:332
2701600994734,7cyclictest309-21in:imuxsock19:50:011
270160099472,11cyclictest2704925-21unin-run20:15:011
270159799471,18cyclictest2704662-21munin-plugin-st20:10:010
2701608994639,4cyclictest2700802-21kworker/2:020:47:582
2701608994639,4cyclictest2677749-21kworker/2:119:20:312
2701608994638,5cyclictest2700802-21kworker/2:020:55:162
2701608994638,5cyclictest2700802-21kworker/2:020:36:272
2701608994638,5cyclictest2700802-21kworker/2:019:34:432
2701600994532,9cyclictest1-21systemd00:35:001
2701600994531,10cyclictest2772780-21sshd23:33:291
2701600994524,16cyclictest275-21dbus-daemon23:05:011
2701597994514,10cyclictest2701868-21date19:15:010
2701608994436,5cyclictest2700802-21kworker/2:019:36:562
2701600994430,12cyclictest23-21ksoftirqd/123:00:011
2701600994428,12cyclictest1-21systemd19:55:011
2701600994428,12cyclictest1-21systemd19:55:011
2701600994425,14cyclictest23-21ksoftirqd/100:20:001
270159799441,39cyclictest0-21swapper/020:52:570
2701600994332,7cyclictest23-21ksoftirqd/122:41:411
2701600994331,9cyclictest2768598-21sshd23:24:301
2701600994330,9cyclictest2741063-21sshd22:23:271
2701600994329,11cyclictest23-21ksoftirqd/100:04:591
2701600994329,10cyclictest1-21systemd19:15:001
2701600994322,16cyclictest3467602-21systemd-journal20:00:011
270160099431,38cyclictest3467602-21systemd-journal20:49:281
2701600994230,8cyclictest2787733-21sshd00:06:291
2701600994229,9cyclictest2702070-21systemctl19:20:001
2701600994227,12cyclictest1-21systemd21:10:001
2701600994227,11cyclictest2784839-21latency_hist00:00:031
2701600994210,10cyclictest131rcu_preempt22:11:141
270159799421,22cyclictest0-21swapper/020:43:480
2701600994129,10cyclictest2705688-21sshd20:29:271
2701600994129,10cyclictest23-21ksoftirqd/122:53:031
2701600994129,10cyclictest23-21ksoftirqd/122:53:031
2701600994127,11cyclictest2796172-21sed00:25:021
2701600994126,11cyclictest1-21systemd23:50:011
2701600994126,10cyclictest23-21ksoftirqd/100:39:141
2701600994126,10cyclictest23-21ksoftirqd/100:39:141
2701600994124,12cyclictest23-21ksoftirqd/123:35:351
2701600994123,14cyclictest2770537-21sshd23:28:291
2701597994120,17cyclictest0-21swapper/019:20:240
270160099409,28cyclictest2764335-21sh23:15:001
2701600994030,8cyclictest23-21ksoftirqd/121:34:161
270160099401,3cyclictest131rcu_preempt22:34:101
270160099396,29cyclictest2712309-21munin-plugin-st21:20:001
270160099393,4cyclictest131rcu_preempt21:41:051
2701600993926,10cyclictest3467602-21systemd-journal20:06:321
2701600993926,10cyclictest23-21ksoftirqd/121:40:011
2701600993926,10cyclictest23-21ksoftirqd/120:45:001
2701600993925,11cyclictest23-21ksoftirqd/121:00:011
2701600993924,12cyclictest23-21ksoftirqd/120:15:021
2701600993924,10cyclictest309-21in:imuxsock23:45:001
270160099391,3cyclictest131rcu_preempt22:04:241
2701600993828,7cyclictest23-21ksoftirqd/122:17:251
2701600993827,9cyclictest23-21ksoftirqd/121:12:051
2701600993827,8cyclictest23-21ksoftirqd/121:20:401
2701600993827,8cyclictest23-21ksoftirqd/121:20:401
2701600993826,8cyclictest23-21ksoftirqd/100:14:091
2701600993825,10cyclictest23-21ksoftirqd/120:40:011
2701600993823,11cyclictest3467602-21systemd-journal21:55:021
2701600993823,10cyclictest23-21ksoftirqd/119:35:021
2701600993822,12cyclictest23-21ksoftirqd/119:21:251
2701600993821,13cyclictest3467602-21systemd-journal22:30:001
270160099374,4cyclictest131rcu_preempt00:27:371
270160099374,4cyclictest131rcu_preempt00:27:371
2701600993726,7cyclictest23-21ksoftirqd/121:50:021
2701600993724,11cyclictest23-21ksoftirqd/122:05:521
2701600993723,10cyclictest23-21ksoftirqd/119:41:241
270160099362,8cyclictest2703100-21modprobe19:40:011
2701600993523,9cyclictest23-21ksoftirqd/123:16:371
2701600993520,11cyclictest3467602-21systemd-journal19:27:251
2701600993422,9cyclictest23-21ksoftirqd/120:25:011
270160899324,4cyclictest131rcu_preempt19:10:002
2701600993220,9cyclictest2706065-21sed20:35:021
2701600993220,9cyclictest23-21ksoftirqd/121:05:011
2701600993220,10cyclictest23-21ksoftirqd/120:55:001
2701600993219,10cyclictest23-21ksoftirqd/120:05:021
2701600993123,5cyclictest2701660-21latency_hist19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional