You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 04:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Sep 14, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
292119899146137,7cyclictest37-21ksoftirqd/320:40:023
2921188991091,105cyclictest0-21swapper/021:00:020
2921188999355,28cyclictest19350irq/34-mmc100:00:020
2921188998232,24cyclictest12-21ksoftirqd/000:25:010
2921188997631,22cyclictest12-21ksoftirqd/021:40:000
2921198997132,6cyclictest2924756-21sendmail20:20:003
2921188996925,33cyclictest12-21ksoftirqd/021:45:290
2921188996635,23cyclictest12-21ksoftirqd/022:50:010
2921188996622,38cyclictest275-21dbus-daemon23:25:010
2921188996622,38cyclictest275-21dbus-daemon23:25:010
2921188996528,17cyclictest0-21swapper/021:18:560
2921188996523,21cyclictest2978296-21ssh23:01:270
2921188996425,20cyclictest12-21ksoftirqd/023:50:000
2921188996324,20cyclictest12-21ksoftirqd/023:25:290
2921188996223,18cyclictest3021799-21ssh00:35:360
292118899621,3cyclictest131rcu_preempt22:55:000
2921188996116,40cyclictest0-21swapper/019:50:010
292118899596,3cyclictest131rcu_preempt22:00:020
292118899594,35cyclictest131rcu_preempt19:15:010
2921188995932,23cyclictest16850irq/134-800000022:04:350
2921188995932,23cyclictest16850irq/134-800000022:04:350
2921188995926,3cyclictest131rcu_preempt23:09:170
2921188995914,5cyclictest131rcu_preempt22:45:000
2921188995914,12cyclictest131rcu_preempt21:28:250
2921188995912,42cyclictest16850irq/134-800000022:37:150
292119699577,4cyclictest131rcu_preempt23:00:022
2921188995728,14cyclictest2947128-21ssh21:53:300
2921188995728,14cyclictest2947128-21ssh21:53:300
2921188995727,14cyclictest3016779-21ssh00:25:050
2921188995726,15cyclictest2982442-21ssh23:10:290
2921188995726,15cyclictest2982442-21ssh23:10:290
2921188995715,34cyclictest16850irq/134-800000023:30:140
2921188995715,34cyclictest16850irq/134-800000023:30:140
2921196995635,8cyclictest2984247-21kworker/2:200:29:562
2921188995636,15cyclictest1-21systemd21:10:010
292118899561,37cyclictest131rcu_preempt00:00:520
292119699555,8cyclictest30-21ksoftirqd/200:37:152
2921196995545,6cyclictest2926285-21kworker/2:022:30:002
2921188995529,11cyclictest2976109-21ssh22:56:550
2921188995528,12cyclictest12-21ksoftirqd/023:45:010
2921188995526,14cyclictest2959839-21ssh22:21:010
2921188995519,18cyclictest16850irq/134-800000000:15:460
2921188995517,33cyclictest16850irq/134-800000022:20:010
2921188995517,19cyclictest131rcu_preempt22:08:030
2921196995445,6cyclictest2682999-21kworker/2:323:00:302
2921196995445,5cyclictest2926285-21kworker/2:023:15:022
2921196995445,5cyclictest2926285-21kworker/2:023:15:022
2921192995437,13cyclictest2931880-21ssh21:20:011
292118899542,4cyclictest131rcu_preempt21:22:470
2921188995419,17cyclictest16850irq/134-800000023:35:040
2921188995419,16cyclictest16850irq/134-800000022:25:040
2921188995418,18cyclictest16850irq/134-800000021:14:440
2921188995415,19cyclictest16850irq/134-800000023:18:340
292118899541,3cyclictest131rcu_preempt00:08:570
292118899541,3cyclictest131rcu_preempt00:08:570
2921188995411,13cyclictest131rcu_preempt00:31:110
292119899531,36cyclictest361rcuc/321:25:023
292119699539,5cyclictest131rcu_preempt00:22:572
2921196995345,5cyclictest2984247-21kworker/2:223:37:282
2921196995345,5cyclictest2926285-21kworker/2:022:05:032
2921196995345,5cyclictest2926285-21kworker/2:022:05:022
2921196995344,5cyclictest2682999-21kworker/2:321:32:232
2921196995313,10cyclictest131rcu_preempt23:55:012
2921188995326,18cyclictest3001989-21ssh23:53:420
2921188995324,14cyclictest2943126-21ssh21:44:530
292118899532,30cyclictest131rcu_preempt19:42:120
292118899532,16cyclictest2926322-21munin-plugin-st20:50:000
2921188995315,21cyclictest16850irq/134-800000022:10:080
2921188995315,21cyclictest16850irq/134-800000022:10:080
2921196995244,5cyclictest2682999-21kworker/2:321:38:282
292119699524,3cyclictest131rcu_preempt00:00:002
2921196995243,5cyclictest2984247-21kworker/2:223:29:112
2921196995234,7cyclictest2984247-21kworker/2:200:15:532
2921192995230,15cyclictest275-21dbus-daemon20:45:011
2921188995220,27cyclictest0-21swapper/021:31:260
2921188995218,17cyclictest16850irq/134-800000000:11:450
292119699514,9cyclictest30-21ksoftirqd/220:35:012
2921196995142,5cyclictest2926285-21kworker/2:022:31:162
2921196995134,13cyclictest275-21dbus-daemon00:00:042
2921196995119,7cyclictest30-21ksoftirqd/222:00:012
292119299516,14cyclictest23-21ksoftirqd/121:45:001
2921188995110,36cyclictest16850irq/134-800000022:33:350
292119699505,9cyclictest30-21ksoftirqd/219:55:012
2921196995042,5cyclictest2926285-21kworker/2:022:44:182
2921196995042,5cyclictest2926285-21kworker/2:021:53:292
2921196995042,5cyclictest2926285-21kworker/2:021:53:292
2921196995042,5cyclictest2682999-21kworker/2:321:05:012
2921196995042,5cyclictest2682999-21kworker/2:320:25:022
2921196995042,5cyclictest2682999-21kworker/2:320:25:022
2921196995041,6cyclictest2926285-21kworker/2:022:11:472
2921196995041,6cyclictest2926285-21kworker/2:022:11:472
2921196995041,6cyclictest2926285-21kworker/2:021:40:332
2921196995041,6cyclictest2682999-21kworker/2:321:19:182
2921196995037,10cyclictest2921629-21cron19:20:002
292119699503,3cyclictest131rcu_preempt23:35:012
292119699503,3cyclictest131rcu_preempt23:35:012
2921198994930,14cyclictest1-21systemd00:10:023
2921198994930,14cyclictest1-21systemd00:10:013
2921198994929,11cyclictest37-21ksoftirqd/322:00:003
292119699494,3cyclictest131rcu_preempt23:05:142
2921196994941,5cyclictest2984247-21kworker/2:200:31:382
2921196994941,5cyclictest2682999-21kworker/2:321:27:122
2921196994940,6cyclictest2984247-21kworker/2:223:46:292
2921196994940,6cyclictest2926285-21kworker/2:022:25:012
292119699492,4cyclictest131rcu_preempt20:55:022
292119699492,4cyclictest131rcu_preempt20:55:022
2921192994935,10cyclictest1-21systemd23:30:001
2921192994934,11cyclictest275-21dbus-daemon22:15:011
2921192994934,11cyclictest275-21dbus-daemon22:15:011
2921192994931,13cyclictest3467602-21systemd-journal22:25:011
2921192994919,17cyclictest23-21ksoftirqd/120:10:001
292119299491,13cyclictest2992962-21ssh23:34:021
292119299491,13cyclictest2992962-21ssh23:34:021
2921198994834,11cyclictest2947926-21wc21:55:023
2921198994834,11cyclictest2947926-21wc21:55:023
2921198994830,13cyclictest1-21systemd20:25:013
2921198994830,13cyclictest1-21systemd20:25:013
292119699484,5cyclictest131rcu_preempt23:20:552
292119699484,5cyclictest131rcu_preempt23:20:552
2921196994840,6cyclictest2926285-21kworker/2:022:18:332
2921196994840,5cyclictest2682999-21kworker/2:321:47:292
2921196994839,6cyclictest2682999-21kworker/2:321:15:012
2921196994839,5cyclictest2682999-21kworker/2:320:27:122
292119699483,3cyclictest131rcu_preempt22:52:532
2921192994833,12cyclictest1-21systemd19:19:591
292119299482,7cyclictest131rcu_preempt21:30:031
292118899482,3cyclictest131rcu_preempt20:30:000
2921198994731,11cyclictest3467602-21systemd-journal23:55:013
2921198994730,13cyclictest3467602-21systemd-journal21:05:013
292119699475,12cyclictest30-21ksoftirqd/222:09:302
2921196994739,5cyclictest2984247-21kworker/2:200:05:422
2921196994739,5cyclictest2984247-21kworker/2:200:05:422
2921196994739,5cyclictest2682999-21kworker/2:321:20:302
2921196994738,5cyclictest2926285-21kworker/2:023:18:272
2921196994737,6cyclictest2926285-21kworker/2:022:46:072
2921196994732,11cyclictest3467602-21systemd-journal19:30:012
2921196994730,13cyclictest3467602-21systemd-journal22:40:002
2921192994733,10cyclictest275-21dbus-daemon21:55:011
2921192994733,10cyclictest275-21dbus-daemon21:55:011
2921192994710,31cyclictest2977435-21modprobe23:00:001
292118899471,3cyclictest131rcu_preempt20:00:020
292118899471,3cyclictest131rcu_preempt20:00:020
292119699465,4cyclictest131rcu_preempt20:55:032
292119699465,3cyclictest131rcu_preempt23:41:532
2921196994638,5cyclictest2984247-21kworker/2:200:11:332
2921196994638,5cyclictest2682999-21kworker/2:319:57:582
2921196994638,5cyclictest2682999-21kworker/2:319:57:582
2921196994637,6cyclictest2682999-21kworker/2:319:40:012
2921196994629,12cyclictest30-21ksoftirqd/220:40:002
292119699461,3cyclictest131rcu_preempt20:05:012
2921192994633,9cyclictest3467602-21systemd-journal00:35:021
2921192994628,14cyclictest275-21dbus-daemon21:40:001
2921192994623,17cyclictest1-21systemd00:00:011
2921188994631,11cyclictest275-21dbus-daemon19:25:010
2921188994628,14cyclictest1-21systemd19:35:010
2921198994532,10cyclictest2977432-21cat23:00:003
2921198994529,12cyclictest2968335-21sendmail-msp22:40:003
2921198994518,17cyclictest0-21swapper/322:25:133
292119699458,5cyclictest131rcu_preempt20:45:032
2921196994530,10cyclictest275-21dbus-daemon21:10:022
2921196994519,5cyclictest30-21ksoftirqd/220:20:002
292119299453,4cyclictest131rcu_preempt22:08:291
2921192994528,13cyclictest275-21dbus-daemon19:35:001
292119299452,23cyclictest0-21swapper/100:03:551
292119299451,4cyclictest131rcu_preempt19:10:011
292119299451,3cyclictest131rcu_preempt00:20:001
292118899452,4cyclictest131rcu_preempt19:20:000
2921188994518,19cyclictest0-21swapper/019:25:200
292119899444,3cyclictest131rcu_preempt21:10:253
2921198994435,6cyclictest2922681-21sendmail19:40:003
2921198994433,8cyclictest2951312-21sshd22:02:323
2921198994433,8cyclictest2951312-21sshd22:02:323
2921198994429,11cyclictest1-21systemd20:00:013
2921198994429,11cyclictest1-21systemd20:00:013
2921198994428,11cyclictest37-21ksoftirqd/322:07:423
2921198994425,10cyclictest37-21ksoftirqd/321:35:023
2921198994422,17cyclictest275-21dbus-daemon23:35:023
2921198994422,17cyclictest275-21dbus-daemon23:35:013
292119699444,9cyclictest30-21ksoftirqd/220:08:102
292119699441,4cyclictest131rcu_preempt20:50:022
292119299449,6cyclictest131rcu_preempt00:23:561
2921192994427,12cyclictest275-21dbus-daemon22:55:001
2921192994421,4cyclictest23-21ksoftirqd/123:05:011
2921188994425,13cyclictest1-21systemd20:15:010
292118899441,4cyclictest131rcu_preempt20:20:000
292118899441,4cyclictest131rcu_preempt20:10:000
292119899435,5cyclictest131rcu_preempt23:10:003
2921198994328,12cyclictest37-21ksoftirqd/321:50:023
292119899432,4cyclictest131rcu_preempt22:48:513
292119899431,3cyclictest131rcu_preempt21:45:013
2921198994313,16cyclictest2975391-21cut22:55:023
292119699439,8cyclictest131rcu_preempt19:34:232
292119699435,4cyclictest131rcu_preempt19:25:022
292119299437,5cyclictest131rcu_preempt21:46:361
292119299435,3cyclictest131rcu_preempt23:55:011
292119299433,6cyclictest131rcu_preempt21:35:021
2921192994328,10cyclictest1-21systemd22:35:011
2921192994319,5cyclictest23-21ksoftirqd/119:50:011
2921192994311,4cyclictest131rcu_preempt00:09:251
2921192994311,4cyclictest131rcu_preempt00:09:251
2921188994316,4cyclictest12-21ksoftirqd/020:35:010
292118899431,39cyclictest2927099-21latency_hist21:05:000
2921198994227,11cyclictest3016555-21systemd-run00:25:013
2921198994227,11cyclictest2921659-21tr19:20:003
2921198994227,10cyclictest37-21ksoftirqd/319:55:033
2921198994227,10cyclictest1-21systemd22:45:013
2921198994226,12cyclictest37-21ksoftirqd/323:16:143
292119899422,5cyclictest131rcu_preempt00:35:003
292119899422,15cyclictest2925839-21grep20:40:013
292119899421,14cyclictest2999833-21ssh23:48:573
292119299429,4cyclictest131rcu_preempt22:01:031
292119299429,4cyclictest131rcu_preempt22:01:031
292119299428,6cyclictest131rcu_preempt21:55:041
292119299428,5cyclictest131rcu_preempt00:25:191
292119299426,7cyclictest23-21ksoftirqd/123:17:381
292119299424,5cyclictest131rcu_preempt23:45:021
292119299424,3cyclictest131rcu_preempt00:35:011
2921192994229,9cyclictest275-21dbus-daemon20:30:001
2921192994226,13cyclictest2982019-21systemctl23:10:011
292119299422,4cyclictest131rcu_preempt22:30:001
2921192994216,4cyclictest23-21ksoftirqd/123:22:171
2921192994216,4cyclictest23-21ksoftirqd/123:22:171
2921188994218,16cyclictest275-21dbus-daemon20:05:000
292119899415,7cyclictest37-21ksoftirqd/323:37:083
292119899414,3cyclictest131rcu_preempt23:56:133
2921198994125,12cyclictest37-21ksoftirqd/321:30:013
2921192994130,8cyclictest2925008-21sshd20:22:291
2921192994130,8cyclictest2925008-21sshd20:22:291
2921192994129,9cyclictest23-21ksoftirqd/100:14:081
2921192994129,8cyclictest2927072-21sshd21:01:291
2921192994128,10cyclictest2984254-21sed23:15:011
2921192994128,10cyclictest2984254-21sed23:15:011
292119299412,6cyclictest2928802-21ssh21:13:091
292119299412,4cyclictest131rcu_preempt22:45:031
292119299411,7cyclictest2932901-21ssh21:22:101
292119899409,7cyclictest2997942-21cron23:45:013
292119899408,19cyclictest37-21ksoftirqd/300:11:113
2921198994029,8cyclictest0-21swapper/300:20:033
2921198994026,11cyclictest37-21ksoftirqd/323:25:013
2921198994026,11cyclictest37-21ksoftirqd/323:25:013
2921198994012,17cyclictest37-21ksoftirqd/322:23:393
292119699401,6cyclictest131rcu_preempt19:14:362
292119299403,4cyclictest131rcu_preempt23:35:351
292119299403,4cyclictest131rcu_preempt22:36:531
2921192994026,11cyclictest2922762-21wc19:40:021
292119299402,3cyclictest131rcu_preempt20:55:021
292119299402,3cyclictest131rcu_preempt20:55:021
292119299401,4cyclictest131rcu_preempt22:20:011
292119299401,11cyclictest2999208-21ssh23:47:331
2921188994028,9cyclictest2921321-21sed19:10:020
2921188994024,11cyclictest3467718-21lldpd20:24:520
2921188994024,11cyclictest3467718-21lldpd20:24:520
292118899402,34cyclictest2923442-21sshd19:54:290
292119899395,4cyclictest131rcu_preempt00:40:013
2921198993928,9cyclictest37-21ksoftirqd/300:28:143
2921198993926,11cyclictest37-21ksoftirqd/321:35:253
2921198993925,10cyclictest37-21ksoftirqd/300:01:383
2921198993924,11cyclictest37-21ksoftirqd/321:17:163
292119899391,5cyclictest2925535-21modprobe20:35:003
292119699391,3cyclictest131rcu_preempt19:45:032
2921192993928,8cyclictest2921364-21sshd19:14:291
2921192993926,10cyclictest23-21ksoftirqd/122:41:231
2921192993921,13cyclictest275-21dbus-daemon20:15:011
2921198993820,16cyclictest361rcuc/321:10:023
292119899381,3cyclictest131rcu_preempt21:00:013
292119299383,4cyclictest131rcu_preempt19:30:001
292119299383,3cyclictest131rcu_preempt19:55:021
2921192993828,6cyclictest309-21in:imuxsock20:35:011
2921192993824,10cyclictest2925890-21cat20:40:011
292119299381,4cyclictest131rcu_preempt21:00:001
292119299381,4cyclictest131rcu_preempt20:00:011
292119299381,4cyclictest131rcu_preempt20:00:011
292119299381,3cyclictest131rcu_preempt19:45:031
292118899382,5cyclictest364-21snmpd20:37:230
292118899381,3cyclictest131rcu_preempt20:55:020
292118899381,3cyclictest131rcu_preempt20:55:020
2921198993730,4cyclictest2922173-21munin-plugin-st19:30:013
2921198993725,9cyclictest37-21ksoftirqd/320:50:293
2921198993725,9cyclictest37-21ksoftirqd/320:50:293
2921198993725,10cyclictest37-21ksoftirqd/323:28:573
2921198993723,10cyclictest275-21dbus-daemon20:30:003
292119899371,14cyclictest2924127-21cat20:05:023
2921198993710,19cyclictest0-21swapper/323:11:293
2921198993710,19cyclictest0-21swapper/323:11:293
292119699376,17cyclictest2923208-21cut19:50:002
292119699374,4cyclictest131rcu_preempt20:13:292
2921192993728,5cyclictest2927357-21latency_hist21:10:031
292119299372,31cyclictest2924843-21grep20:20:011
2921192993721,12cyclictest1-21systemd19:25:011
292119899362,3cyclictest131rcu_preempt19:25:023
2921198993612,17cyclictest37-21ksoftirqd/320:50:013
292119899361,13cyclictest2954974-21ssh22:10:213
292119899361,13cyclictest2954974-21ssh22:10:213
292119699363,3cyclictest131rcu_preempt19:10:022
2921192993626,7cyclictest2923958-21sshd20:01:291
2921188993624,9cyclictest275-21dbus-daemon19:40:010
292119899353,3cyclictest131rcu_preempt19:10:023
2921198993527,5cyclictest3467562-21sshd19:42:033
2921198993525,7cyclictest37-21ksoftirqd/322:20:023
2921198993521,11cyclictest37-21ksoftirqd/323:00:503
292119299351,3cyclictest131rcu_preempt20:50:021
292118899351,4cyclictest131rcu_preempt20:40:020
292119899342,13cyclictest2922427-21munin-run19:35:003
2921198993420,10cyclictest37-21ksoftirqd/322:30:033
2921198993320,9cyclictest275-21dbus-daemon19:15:013
2921198993317,12cyclictest3467602-21systemd-journal20:10:013
292119899331,11cyclictest150-21pr/ttyS219:47:043
292119899321,19cyclictest0-21swapper/320:14:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional