You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-10 - 12:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Jul 10, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1989980998028,14cyclictest2043447-21ssh22:49:160
1989980997816,33cyclictest16850irq/134-800000023:24:340
1989980997816,33cyclictest16850irq/134-800000023:24:330
1989980997723,26cyclictest2017681-21latency_hist21:55:010
1989980997719,34cyclictest16850irq/134-800000021:49:320
1989980997719,34cyclictest16850irq/134-800000021:49:310
1989980997719,34cyclictest16850irq/134-800000021:42:180
198998099769,32cyclictest16850irq/134-800000022:43:050
1989980997618,33cyclictest16850irq/134-800000023:10:090
1989980997616,34cyclictest16850irq/134-800000021:36:210
198999099751,4cyclictest131rcu_preempt20:00:012
1989980997516,30cyclictest16850irq/134-800000000:03:000
1989980997516,30cyclictest16850irq/134-800000000:02:590
1989980997514,6cyclictest16850irq/134-800000000:38:150
1989980997417,32cyclictest16850irq/134-800000000:24:460
1989980997417,32cyclictest16850irq/134-800000000:24:450
1989980997415,34cyclictest16850irq/134-800000023:37:500
1989980997414,35cyclictest16850irq/134-800000022:36:370
1989980997413,30cyclictest16850irq/134-800000021:10:070
1989990997335,9cyclictest3467602-21systemd-journal22:30:012
1989990997335,9cyclictest3467602-21systemd-journal22:30:002
1989980997327,10cyclictest2073300-21ssh23:51:380
1989980997324,12cyclictest2091055-21ssh00:28:530
1989980997216,33cyclictest16850irq/134-800000022:32:310
1989980997214,5cyclictest16850irq/134-800000000:11:540
1989980997214,5cyclictest16850irq/134-800000000:11:530
1989980997214,32cyclictest16850irq/134-800000000:34:010
1989980997214,31cyclictest16850irq/134-800000000:15:120
1989980997213,6cyclictest16850irq/134-800000023:01:250
1989980997115,35cyclictest16850irq/134-800000022:03:500
1989980997112,30cyclictest16850irq/134-800000022:55:050
1989980997111,31cyclictest16850irq/134-800000021:26:480
198998099681,42cyclictest150-21pr/ttyS220:05:010
1989980996813,6cyclictest16850irq/134-800000022:19:530
1989980996813,6cyclictest16850irq/134-800000022:19:530
1989980996813,32cyclictest16850irq/134-800000022:12:420
198998599672,14cyclictest2087829-21ssh00:21:491
198998599672,14cyclictest2087829-21ssh00:21:491
198998599671,15cyclictest2096497-21cat00:40:021
1989980996715,5cyclictest16850irq/134-800000023:26:450
198998599652,16cyclictest1994886-21munin-run20:45:011
1989985996426,34cyclictest2043787-21systemd-run22:50:011
198998599632,13cyclictest2016972-21ssh21:53:231
198998599632,13cyclictest1996187-21munin-run21:10:011
1989985996320,39cyclictest23-21ksoftirqd/123:47:371
198998599631,13cyclictest48-21kcompactd000:00:061
198998599631,13cyclictest48-21kcompactd000:00:051
198998599622,16cyclictest2093236-21ssh00:33:251
1989985996219,39cyclictest23-21ksoftirqd/120:25:011
198998599611,14cyclictest2027155-21munin-run22:15:001
1989980996120,12cyclictest1993304-21systemd-run20:15:000
198998599602,16cyclictest2047917-21ssh22:58:351
198998599593,13cyclictest1993054-21munin-run20:10:001
198998599592,13cyclictest2081196-21sshd00:08:061
198998599592,11cyclictest48-21kcompactd022:50:261
1989985995918,36cyclictest2057381-21rm23:18:271
198998599591,14cyclictest364-21snmpd21:00:431
198998599591,14cyclictest1991975-21sshd19:46:551
198998599591,13cyclictest2041186-21ssh22:44:301
1989980995918,21cyclictest16850irq/134-800000023:41:290
198998599582,16cyclictest2013654-21ssh21:46:181
198998599582,16cyclictest2013654-21ssh21:46:171
1989985995820,35cyclictest1996161-21diskmemload23:09:511
1989985995819,32cyclictest2082735-21sh00:11:101
1989985995819,32cyclictest2082735-21sh00:11:101
1989985995817,38cyclictest2008172-21latency_hist21:35:011
1989985995816,38cyclictest2035295-21sshd22:31:561
198998599581,15cyclictest364-21snmpd19:26:311
198998599581,13cyclictest364-21snmpd19:34:581
1989985995810,3cyclictest0-21swapper/119:17:041
1989980995816,37cyclictest16850irq/134-800000023:50:010
1989990995749,5cyclictest2027170-21kworker/2:222:42:182
198998599573,17cyclictest2089735-21ssh00:25:481
198998599572,15cyclictest2048860-21ssh23:00:111
198998599572,14cyclictest2029224-21ssh22:19:241
198998599572,14cyclictest2029224-21ssh22:19:231
1989985995717,36cyclictest23-21ksoftirqd/123:45:011
1989985995715,5cyclictest2000742-21sh21:19:361
1989985995714,38cyclictest2020138-21latency_hist22:00:021
1989985995714,38cyclictest2020138-21latency_hist22:00:011
198998599571,16cyclictest2032317-21ssh22:25:311
198998599571,16cyclictest2032317-21ssh22:25:301
198998599571,16cyclictest2004626-21ssh21:27:331
198998599571,16cyclictest1996969-21ssh21:11:111
198998599571,14cyclictest2029797-21ssh22:20:041
198998599571,13cyclictest2078389-21ssh00:02:091
198998599571,13cyclictest2078389-21ssh00:02:091
198998599571,12cyclictest2053958-21ssh23:11:001
1989980995720,33cyclictest16850irq/134-800000022:06:370
1989980995720,33cyclictest16850irq/134-800000022:06:370
1989980995719,34cyclictest16850irq/134-800000021:32:100
1989980995719,33cyclictest16850irq/134-800000021:15:250
1989980995718,30cyclictest16850irq/134-800000021:21:300
1989980995717,35cyclictest16850irq/134-800000023:16:530
198998599565,6cyclictest1993542-21cron20:20:001
198998599565,6cyclictest1993542-21cron20:20:001
198998599562,16cyclictest2062422-21ssh23:28:591
198998599562,14cyclictest2060619-21latency_hist23:25:031
198998599562,14cyclictest2060619-21latency_hist23:25:021
198998599562,14cyclictest2011200-21ssh21:41:001
198998599562,14cyclictest2002559-21ssh21:23:111
198998599562,11cyclictest364-21snmpd22:09:551
198998599562,11cyclictest364-21snmpd22:09:551
198998599562,11cyclictest1976151-21munin-node20:00:011
1989985995616,34cyclictest275-21dbus-daemon00:20:011
198998599561,14cyclictest2021483-21ssh22:02:531
198998599561,13cyclictest364-21snmpd21:36:431
1989980995618,33cyclictest16850irq/134-800000023:59:020
1989980995618,33cyclictest16850irq/134-800000023:59:010
1989980995618,33cyclictest16850irq/134-800000000:05:260
1989980995617,19cyclictest16850irq/134-800000023:08:360
1989980995616,32cyclictest16850irq/134-800000022:27:240
1989980995616,32cyclictest16850irq/134-800000022:27:230
198998599559,6cyclictest0-21swapper/119:25:031
198998599552,15cyclictest2065705-21ssh23:35:441
198998599552,12cyclictest364-21snmpd20:26:131
198998599552,12cyclictest364-21snmpd20:26:131
198998599552,12cyclictest2074355-21ssh23:53:581
198998599551,14cyclictest2065234-21ssh23:35:001
1989980995519,33cyclictest16850irq/134-800000022:22:360
1989980995518,32cyclictest16850irq/134-800000023:33:120
1989980995517,7cyclictest16850irq/134-800000021:58:480
1989980995517,7cyclictest16850irq/134-800000021:58:480
1989985995417,33cyclictest1-21systemd19:15:001
198998599541,13cyclictest2037508-21ssh22:36:321
1989980995416,34cyclictest16850irq/134-800000022:53:000
1989993995212,5cyclictest131rcu_preempt22:28:263
1989993995212,5cyclictest131rcu_preempt22:28:263
1989990995244,5cyclictest2043816-21kworker/2:322:55:472
1989990995243,5cyclictest1991509-21kworker/2:122:13:102
198998599522,14cyclictest1995223-21latency_hist20:50:021
198998599521,47cyclictest23-21ksoftirqd/119:55:021
1989993995132,13cyclictest275-21dbus-daemon20:25:013
1989990995143,5cyclictest2043816-21kworker/2:323:59:592
1989990995143,5cyclictest2043816-21kworker/2:323:53:492
1989990995143,5cyclictest2043816-21kworker/2:300:00:002
1989990995143,5cyclictest2027170-21kworker/2:223:13:192
1989990995143,5cyclictest1991509-21kworker/2:121:58:592
1989990995143,5cyclictest1991509-21kworker/2:121:58:592
1989990995142,5cyclictest2000985-21kworker/2:321:51:002
1989980995142,6cyclictest16950irq/224-800000021:10:010
198999399504,5cyclictest131rcu_preempt23:55:013
1989990995042,5cyclictest1991509-21kworker/2:121:18:522
1989990995041,6cyclictest2027170-21kworker/2:200:18:552
1989990995033,13cyclictest2045019-21sshd22:52:242
1989985995022,24cyclictest1992773-21cron20:05:011
198998599501,10cyclictest3467602-21systemd-journal20:40:001
198998099506,3cyclictest131rcu_preempt20:20:010
198998099506,3cyclictest131rcu_preempt20:20:000
1989980995031,15cyclictest3467602-21systemd-journal20:30:020
1989980995031,15cyclictest3467602-21systemd-journal20:30:010
198999399499,5cyclictest131rcu_preempt21:23:293
1989993994931,13cyclictest3467602-21systemd-journal19:15:003
1989993994911,10cyclictest131rcu_preempt23:09:093
1989990994941,5cyclictest2043816-21kworker/2:323:26:422
1989990994941,5cyclictest2027170-21kworker/2:200:27:162
1989990994941,5cyclictest2027170-21kworker/2:200:20:332
1989990994941,5cyclictest2027170-21kworker/2:200:20:322
1989990994941,5cyclictest1991509-21kworker/2:122:31:412
1989990994940,6cyclictest2027170-21kworker/2:200:09:172
1989990994940,6cyclictest2000985-21kworker/2:322:05:412
1989990994940,6cyclictest2000985-21kworker/2:322:05:412
1989990994940,5cyclictest2000985-21kworker/2:321:38:232
1989990994939,6cyclictest2027170-21kworker/2:223:39:592
198999399488,5cyclictest131rcu_preempt20:31:153
1989993994817,20cyclictest1992499-21modprobe20:00:003
1989990994840,5cyclictest1992799-21kworker/2:021:10:042
1989990994840,5cyclictest1991509-21kworker/2:122:21:502
1989990994840,5cyclictest1989582-21kworker/2:319:50:092
198999099483,3cyclictest131rcu_preempt21:22:562
198999099481,3cyclictest131rcu_preempt20:20:012
198999099481,3cyclictest131rcu_preempt20:20:002
1989990994811,7cyclictest131rcu_preempt23:50:002
198998599489,8cyclictest1989975-21cyclictest20:55:021
198998599483,8cyclictest1995635-21systemctl21:00:011
1989980994833,12cyclictest275-21dbus-daemon20:25:000
198999399475,3cyclictest131rcu_preempt23:26:253
1989990994739,5cyclictest2043816-21kworker/2:323:23:032
1989990994739,5cyclictest2043816-21kworker/2:323:23:032
1989990994739,5cyclictest2027170-21kworker/2:223:19:112
1989990994738,6cyclictest2027170-21kworker/2:223:30:302
1989990994738,5cyclictest2000985-21kworker/2:321:47:582
1989990994738,5cyclictest2000985-21kworker/2:321:47:582
198999099472,3cyclictest131rcu_preempt22:40:002
1989985994719,24cyclictest1994337-21latency_hist20:35:021
198999399463,4cyclictest131rcu_preempt23:00:123
1989993994630,11cyclictest309-21in:imuxsock21:35:003
1989993994627,11cyclictest2077261-21sh00:00:013
1989993994627,11cyclictest2077261-21sh00:00:003
198999399462,12cyclictest2036156-21ssh22:33:523
198999399461,4cyclictest131rcu_preempt00:34:563
1989993994610,6cyclictest131rcu_preempt00:04:483
1989993994610,6cyclictest131rcu_preempt00:04:473
1989990994638,5cyclictest1991509-21kworker/2:122:16:312
1989990994638,5cyclictest1991509-21kworker/2:122:16:302
1989990994637,5cyclictest1991509-21kworker/2:121:05:012
198999099463,3cyclictest131rcu_preempt00:35:022
1989993994530,10cyclictest3467602-21systemd-journal20:06:583
198999399452,4cyclictest131rcu_preempt22:22:553
1989990994537,5cyclictest1991509-21kworker/2:121:32:352
1989990994536,6cyclictest2027170-21kworker/2:223:43:112
198999099453,3cyclictest131rcu_preempt20:40:012
1989990994532,10cyclictest275-21dbus-daemon19:35:012
198999099451,3cyclictest131rcu_preempt00:10:452
198999099451,3cyclictest131rcu_preempt00:10:442
198998599452,12cyclictest1991471-21cron19:40:011
198998599451,12cyclictest1-21systemd19:45:001
198998099452,7cyclictest1994601-21munin-run20:40:000
198999399444,3cyclictest131rcu_preempt23:15:073
198999399442,3cyclictest131rcu_preempt22:02:263
198999399441,5cyclictest131rcu_preempt23:24:523
198999399441,5cyclictest131rcu_preempt23:24:523
198999399441,4cyclictest131rcu_preempt23:34:053
198999099447,10cyclictest30-21ksoftirqd/220:33:542
1989990994437,4cyclictest2000985-21kworker/2:322:03:102
1989990994436,5cyclictest2027170-21kworker/2:222:46:142
1989990994435,5cyclictest2043816-21kworker/2:323:02:212
198999099443,4cyclictest131rcu_preempt00:01:272
198999099443,4cyclictest131rcu_preempt00:01:262
1989990994434,6cyclictest1991509-21kworker/2:120:09:152
1989980994428,12cyclictest3467602-21systemd-journal19:16:280
198999399435,3cyclictest131rcu_preempt22:50:483
198999399434,3cyclictest131rcu_preempt00:07:323
198999399433,6cyclictest2023459-21ssh22:07:033
198999399433,6cyclictest2023459-21ssh22:07:023
198999399432,4cyclictest131rcu_preempt23:47:393
1989993994320,15cyclictest275-21dbus-daemon20:05:013
198999399431,3cyclictest131rcu_preempt21:51:503
198999399431,3cyclictest131rcu_preempt21:39:323
1989993994311,4cyclictest131rcu_preempt00:25:393
198999099434,4cyclictest131rcu_preempt00:35:212
198999099434,3cyclictest131rcu_preempt21:29:032
1989990994333,5cyclictest1991509-21kworker/2:120:57:562
1989990994328,12cyclictest30-21ksoftirqd/223:08:552
198998099433,3cyclictest131rcu_preempt19:45:020
198998099432,7cyclictest284-21systemd-logind20:35:000
1989980994326,12cyclictest12-21ksoftirqd/019:50:010
198999399429,21cyclictest2043772-21modprobe22:50:013
1989993994231,9cyclictest37-21ksoftirqd/300:20:463
1989993994231,9cyclictest37-21ksoftirqd/300:20:453
1989993994225,12cyclictest3467602-21systemd-journal22:15:003
198999399422,4cyclictest131rcu_preempt23:36:523
1989993994224,14cyclictest3467602-21systemd-journal21:10:013
1989993994224,13cyclictest37-21ksoftirqd/323:13:003
198999399422,3cyclictest131rcu_preempt22:17:533
198999399422,3cyclictest131rcu_preempt22:17:523
198999399422,3cyclictest131rcu_preempt21:26:123
198999399421,3cyclictest131rcu_preempt21:59:433
198999399421,3cyclictest131rcu_preempt21:59:423
1989990994234,5cyclictest2000985-21kworker/2:321:42:192
1989990994230,9cyclictest1-21systemd20:25:002
1989985994211,6cyclictest23-21ksoftirqd/120:15:031
198998099424,4cyclictest131rcu_preempt20:55:280
1989980994228,11cyclictest1990375-21cut19:15:020
1989980994224,14cyclictest1991189-21sshd19:31:280
198998099422,3cyclictest131rcu_preempt19:52:280
198998099421,4cyclictest131rcu_preempt20:50:290
198999399419,21cyclictest2048527-21ssh23:00:003
1989993994130,7cyclictest1993534-21sshd20:18:283
1989993994130,7cyclictest1993534-21sshd20:18:283
1989993994129,9cyclictest3467602-21systemd-journal19:30:003
198999399411,3cyclictest131rcu_preempt22:39:083
1989993994110,5cyclictest131rcu_preempt21:10:083
198999099415,3cyclictest131rcu_preempt20:30:012
198999099415,3cyclictest131rcu_preempt20:30:012
198999099413,3cyclictest131rcu_preempt20:53:282
198999099412,3cyclictest131rcu_preempt20:43:282
1989980994129,9cyclictest1-21systemd20:50:000
198999399405,8cyclictest0-21swapper/319:54:283
1989993994029,9cyclictest37-21ksoftirqd/321:40:293
1989993994027,10cyclictest37-21ksoftirqd/300:35:083
198999399402,4cyclictest131rcu_preempt00:14:153
198999399402,4cyclictest131rcu_preempt00:14:143
1989993994018,16cyclictest275-21dbus-daemon19:50:023
198999099408,6cyclictest131rcu_preempt19:50:002
1989990994029,8cyclictest1-21systemd19:40:012
1989990994025,12cyclictest30-21ksoftirqd/219:20:002
1989990994025,11cyclictest1990973-21systemd-run19:30:012
198999099402,3cyclictest131rcu_preempt19:25:002
1989980994026,11cyclictest3467602-21systemd-journal21:00:290
1989980994022,11cyclictest309-21in:imuxsock19:25:010
1989993993929,8cyclictest37-21ksoftirqd/319:10:033
1989993993927,9cyclictest37-21ksoftirqd/300:20:013
1989993993924,11cyclictest3467602-21systemd-journal20:40:003
1989993993920,14cyclictest275-21dbus-daemon21:20:003
1989990993926,9cyclictest30-21ksoftirqd/221:09:502
1989980993930,6cyclictest19350irq/34-mmc119:40:010
1989980993930,6cyclictest12-21ksoftirqd/019:30:010
1989980993926,10cyclictest1994978-21cat20:45:030
198999399385,6cyclictest131rcu_preempt21:48:303
198999399385,6cyclictest131rcu_preempt21:48:303
198999399382,9cyclictest2039323-21sh22:40:073
1989993993827,9cyclictest37-21ksoftirqd/323:43:403
1989993993827,8cyclictest37-21ksoftirqd/320:50:003
1989993993824,11cyclictest3467602-21systemd-journal20:55:003
198999099381,4cyclictest131rcu_preempt20:00:282
198998099382,5cyclictest1992547-21unin-run20:00:010
1989993993727,6cyclictest1991458-21cron19:40:003
1989993993726,8cyclictest37-21ksoftirqd/320:15:033
1989993993726,7cyclictest1995599-21sshd20:57:063
1989990993726,8cyclictest30-21ksoftirqd/219:13:282
198999099371,3cyclictest131rcu_preempt20:15:002
198998599371,13cyclictest1990065-21sort19:10:021
1989980993723,10cyclictest12-21ksoftirqd/020:10:000
198999399362,5cyclictest131rcu_preempt19:20:003
1989993993621,11cyclictest1-21systemd19:25:013
198999099362,3cyclictest131rcu_preempt19:45:002
198999399353,17cyclictest1991320-21cat19:35:033
1989993993526,7cyclictest37-21ksoftirqd/320:45:023
1989993993524,9cyclictest37-21ksoftirqd/320:30:003
1989993993524,9cyclictest37-21ksoftirqd/320:30:003
198999099354,3cyclictest131rcu_preempt20:48:272
1989980993425,6cyclictest12-21ksoftirqd/019:10:020
1989993993324,5cyclictest1991897-21latency_hist19:45:013
1989993993319,9cyclictest309-21in:imuxsock21:05:013
1989990992720,5cyclictest1990113-21latency_hist19:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional