You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-05 - 15:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Aug 05, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2111449998577,6cyclictest30-21ksoftirqd/219:30:022
2111449997535,32cyclictest30-21ksoftirqd/223:05:012
2111444996411,17cyclictest2126848-21cat23:50:021
2111444996311,21cyclictest2112168-21/usr/sbin/munin19:25:011
211145199604,17cyclictest37-21ksoftirqd/323:00:013
211144099603,7cyclictest12-21ksoftirqd/020:25:030
211144099601,43cyclictest0-21swapper/022:35:010
2111444995932,23cyclictest275-21dbus-daemon00:00:001
2111444995843,11cyclictest1-21systemd22:35:011
2111444995711,17cyclictest2113405-21latency_hist19:45:031
2111444995710,14cyclictest2118578-21ssh21:11:371
2111444995710,14cyclictest2118578-21ssh21:11:361
211145199565,9cyclictest37-21ksoftirqd/323:29:593
211144999565,9cyclictest30-21ksoftirqd/200:30:022
211144499568,17cyclictest2121624-21date22:10:021
211144499568,17cyclictest2121624-21date22:10:021
2111444995615,6cyclictest131rcu_preempt22:40:021
2111444995610,16cyclictest2113295-21cat19:45:021
2111440995620,32cyclictest16850irq/134-800000021:10:310
2111440995620,32cyclictest16850irq/134-800000021:10:310
211144099552,6cyclictest121ksoftirqd/020:27:550
211144099552,6cyclictest121ksoftirqd/020:27:540
211144999545,7cyclictest30-21ksoftirqd/219:25:002
2111440995415,11cyclictest12-21ksoftirqd/022:55:000
211145199539,6cyclictest131rcu_preempt19:25:003
211145199531,20cyclictest37-21ksoftirqd/320:50:023
211145199531,20cyclictest37-21ksoftirqd/320:50:023
211144999537,8cyclictest30-21ksoftirqd/221:00:012
211144999536,7cyclictest30-21ksoftirqd/221:20:022
2111444995329,18cyclictest2116785-21grep20:50:031
2111444995329,18cyclictest2116785-21grep20:50:021
2111444995314,7cyclictest23-21ksoftirqd/123:40:011
2111444995314,7cyclictest23-21ksoftirqd/123:40:011
211144999529,5cyclictest131rcu_preempt22:40:002
2111444995230,18cyclictest2115533-21munin-run20:30:011
2111444995230,18cyclictest2115533-21munin-run20:30:001
2111444995214,34cyclictest2111927-21systemd-run19:20:011
211145199516,14cyclictest37-21ksoftirqd/323:55:023
211145199516,14cyclictest37-21ksoftirqd/323:55:023
211145199516,13cyclictest37-21ksoftirqd/321:20:003
2111451995143,5cyclictest361rcuc/321:11:003
2111451995143,5cyclictest361rcuc/321:10:593
2111451995136,10cyclictest1-21systemd21:10:013
211145199513,14cyclictest37-21ksoftirqd/300:15:013
211145199504,43cyclictest3467602-21systemd-journal21:25:033
211144499509,7cyclictest23-21ksoftirqd/121:55:011
211144499509,7cyclictest23-21ksoftirqd/121:55:011
2111444995033,12cyclictest275-21dbus-daemon20:10:001
2111444995032,15cyclictest48-21kcompactd020:50:261
2111444995032,13cyclictest1-21systemd20:20:011
2111444995031,14cyclictest2124704-21munin-run23:10:001
2111444995028,18cyclictest2127827-21munin-run00:10:001
2111444995013,6cyclictest131rcu_preempt00:30:011
211144099503,10cyclictest2125761-21unin-run23:30:000
211144099502,13cyclictest3467602-21systemd-journal20:00:010
211145199499,6cyclictest131rcu_preempt19:50:013
211145199496,5cyclictest131rcu_preempt20:15:013
211145199496,5cyclictest131rcu_preempt20:00:023
211145199492,5cyclictest131rcu_preempt20:02:543
2111451994911,5cyclictest131rcu_preempt19:35:013
211144999498,6cyclictest131rcu_preempt20:45:022
211144999496,7cyclictest30-21ksoftirqd/219:40:002
211144999495,9cyclictest30-21ksoftirqd/221:31:012
2111449994939,8cyclictest291rcuc/219:54:452
2111449994937,9cyclictest30-21ksoftirqd/221:25:022
2111449994910,5cyclictest131rcu_preempt00:21:322
211144499498,14cyclictest2117454-21cat21:05:021
2111444994934,12cyclictest2113678-21sshd19:51:541
2111444994934,11cyclictest2123388-21sshd22:41:541
2111444994933,12cyclictest2126545-21munin-run23:45:011
2111444994933,12cyclictest2123670-21munin-run22:50:001
211144099495,9cyclictest12-21ksoftirqd/022:11:170
2111440994942,4cyclictest16950irq/224-800000023:40:010
2111440994942,4cyclictest16950irq/224-800000023:40:010
2111440994940,4cyclictest16950irq/224-800000021:09:590
2111440994926,9cyclictest12-21ksoftirqd/020:38:370
211145199489,6cyclictest131rcu_preempt00:30:013
2111451994833,11cyclictest1-21systemd00:25:003
211144999487,6cyclictest30-21ksoftirqd/200:35:022
211144999485,13cyclictest30-21ksoftirqd/220:10:012
2111449994840,5cyclictest2094954-21kworker/2:320:19:452
2111444994831,13cyclictest2122647-21munin-run22:30:011
2111444994810,4cyclictest2119267-21latency_hist21:25:011
211144099484,6cyclictest12-21ksoftirqd/019:10:020
2111440994841,4cyclictest16950irq/224-800000021:55:010
2111440994841,4cyclictest16950irq/224-800000021:55:000
2111440994841,4cyclictest16950irq/224-800000000:00:000
2111451994710,6cyclictest131rcu_preempt23:25:023
2111451994710,6cyclictest131rcu_preempt23:25:023
2111451994710,6cyclictest131rcu_preempt21:42:543
211144999479,8cyclictest30-21ksoftirqd/221:11:082
211144999479,8cyclictest30-21ksoftirqd/221:11:072
211144999479,4cyclictest131rcu_preempt20:48:542
211144999479,4cyclictest131rcu_preempt20:48:542
211144999476,6cyclictest30-21ksoftirqd/222:30:012
2111449994739,5cyclictest2123427-21kworker/2:323:37:292
2111449994739,5cyclictest2123427-21kworker/2:323:37:292
2111449994739,5cyclictest2123427-21kworker/2:300:15:362
2111449994739,5cyclictest2120269-21kworker/2:122:47:402
2111449994739,5cyclictest2120269-21kworker/2:121:49:032
2111449994739,5cyclictest2120269-21kworker/2:121:41:472
2111449994739,5cyclictest2118753-21kworker/2:023:46:412
2111449994739,5cyclictest2118753-21kworker/2:023:25:542
2111449994739,5cyclictest2118753-21kworker/2:000:38:562
2111449994738,6cyclictest30-21ksoftirqd/222:25:012
2111449994732,10cyclictest1-21systemd00:00:002
211144499477,34cyclictest2122524-21grep22:25:021
211144499472,11cyclictest2112837-21latency_hist19:35:031
211144099478,34cyclictest12-21ksoftirqd/019:30:000
2111440994739,4cyclictest16850irq/134-800000020:15:540
211144099473,8cyclictest2120034-21munin-run21:40:010
211144099473,7cyclictest12-21ksoftirqd/022:05:010
2111440994713,31cyclictest0-21swapper/021:00:010
211145199465,11cyclictest37-21ksoftirqd/322:15:003
211145199465,10cyclictest37-21ksoftirqd/322:55:003
2111451994637,6cyclictest37-21ksoftirqd/323:40:013
2111451994637,6cyclictest37-21ksoftirqd/323:40:003
211145199463,5cyclictest131rcu_preempt22:10:013
211145199463,5cyclictest131rcu_preempt22:10:003
211144999466,7cyclictest30-21ksoftirqd/219:35:032
211144999463,9cyclictest30-21ksoftirqd/221:05:022
2111449994639,4cyclictest2120269-21kworker/2:122:01:582
2111449994639,4cyclictest2120269-21kworker/2:121:52:322
2111449994639,4cyclictest2120269-21kworker/2:121:52:312
2111449994639,4cyclictest2118753-21kworker/2:023:42:022
2111449994639,4cyclictest2115056-21kworker/2:221:09:592
2111449994639,4cyclictest2112191-21kworker/2:020:30:002
2111449994639,4cyclictest2112191-21kworker/2:020:30:002
2111449994639,4cyclictest2112191-21kworker/2:019:46:222
2111449994639,4cyclictest2094954-21kworker/2:319:41:352
2111449994639,4cyclictest2094954-21kworker/2:319:41:352
2111449994638,5cyclictest2123427-21kworker/2:322:58:422
2111449994638,5cyclictest2118753-21kworker/2:023:54:032
2111449994638,5cyclictest2118753-21kworker/2:023:54:032
2111449994638,5cyclictest2118753-21kworker/2:023:22:522
2111449994638,5cyclictest2118753-21kworker/2:023:22:512
2111449994638,5cyclictest2115056-21kworker/2:220:36:592
2111449994638,5cyclictest2115056-21kworker/2:220:24:382
2111449994638,5cyclictest2112191-21kworker/2:020:32:342
2111449994638,5cyclictest2112191-21kworker/2:019:55:162
2111449994615,6cyclictest30-21ksoftirqd/200:10:142
2111444994631,12cyclictest364-21snmpd21:31:101
2111444994631,11cyclictest2126014-21munin-run23:35:001
2111444994629,14cyclictest2111594-21sshd19:10:551
2111444994629,13cyclictest2129106-21sshd00:31:351
211144099469,32cyclictest151rcuc/021:30:020
2111440994634,9cyclictest2112936-21systemctl19:40:000
2111440994617,5cyclictest131rcu_preempt22:35:530
211144099461,41cyclictest2125191-21sshd23:17:540
2111440994613,9cyclictest0-21swapper/020:49:180
2111440994613,9cyclictest0-21swapper/020:49:170
2111440994613,3cyclictest0-21swapper/021:30:550
211144099461,24cyclictest0-21swapper/000:35:010
211145199458,7cyclictest37-21ksoftirqd/321:00:003
211145199457,7cyclictest37-21ksoftirqd/300:35:023
211145199456,10cyclictest37-21ksoftirqd/322:45:543
211145199454,7cyclictest37-21ksoftirqd/320:35:013
211145199454,14cyclictest37-21ksoftirqd/300:38:553
211145199454,10cyclictest37-21ksoftirqd/300:00:013
2111451994512,7cyclictest131rcu_preempt19:20:003
2111449994538,5cyclictest2123427-21kworker/2:323:05:532
2111449994538,4cyclictest2123427-21kworker/2:300:09:102
2111449994538,4cyclictest2120269-21kworker/2:122:40:262
2111449994538,4cyclictest2116080-21kworker/2:320:50:352
2111449994538,4cyclictest2112191-21kworker/2:020:14:562
2111449994537,5cyclictest2123427-21kworker/2:300:03:362
2111449994537,5cyclictest2123427-21kworker/2:300:03:362
2111449994537,5cyclictest2120269-21kworker/2:121:55:432
2111444994534,7cyclictest2115751-21sshd20:30:511
2111444994532,9cyclictest2124941-21cron23:15:001
2111444994532,10cyclictest2123936-21munin-run22:55:011
2111444994531,10cyclictest3467602-21systemd-journal19:25:411
2111444994530,12cyclictest2118973-21sshd21:19:551
2111444994530,11cyclictest48-21kcompactd021:59:461
2111444994530,11cyclictest2120573-21munin-run21:50:001
2111444994529,13cyclictest2127555-21sshd00:03:541
2111444994529,13cyclictest2127555-21sshd00:03:541
2111444994529,12cyclictest2116331-21munin-run20:45:021
2111444994529,11cyclictest48-21kcompactd020:22:401
2111444994528,13cyclictest3467602-21systemd-journal00:20:001
2111444994528,13cyclictest2129370-21sshd00:38:551
2111444994525,14cyclictest2116025-21cron20:40:001
211144499451,13cyclictest2113966-21munin-run20:00:011
211144099457,9cyclictest12-21ksoftirqd/000:15:000
211144099456,5cyclictest131rcu_preempt23:43:530
2111440994538,4cyclictest16950irq/224-800000020:45:020
2111440994538,4cyclictest16950irq/224-800000020:05:000
2111440994536,6cyclictest151rcuc/021:19:590
2111440994535,7cyclictest3467602-21systemd-journal23:55:010
2111440994535,7cyclictest3467602-21systemd-journal23:55:000
211144099451,6cyclictest2128627-21systemd-run00:25:000
211145199446,9cyclictest37-21ksoftirqd/321:46:133
2111451994431,10cyclictest3467602-21systemd-journal23:42:543
211145199442,10cyclictest37-21ksoftirqd/320:23:543
2111451994410,5cyclictest131rcu_preempt22:20:023
2111451994410,5cyclictest131rcu_preempt21:05:013
211145199440,7cyclictest150-21pr/ttyS221:57:023
2111449994437,4cyclictest2123427-21kworker/2:323:10:502
2111449994437,4cyclictest2120269-21kworker/2:122:12:562
2111449994437,4cyclictest2094954-21kworker/2:320:02:322
2111449994437,4cyclictest2094954-21kworker/2:319:17:252
2111449994436,5cyclictest2120269-21kworker/2:122:32:402
2111449994436,5cyclictest2120269-21kworker/2:122:05:082
2111449994436,5cyclictest2120269-21kworker/2:122:05:082
2111449994436,5cyclictest2118753-21kworker/2:022:52:362
211144499446,4cyclictest131rcu_preempt23:15:031
2111444994432,9cyclictest2128092-21munin-run00:15:011
2111444994431,9cyclictest2117628-21sshd21:07:541
2111444994427,13cyclictest364-21snmpd19:46:571
2111444994422,18cyclictest364-21snmpd22:14:221
2111444994421,19cyclictest2114251-21unin-run20:05:011
2111444994411,5cyclictest131rcu_preempt21:40:011
211144099446,9cyclictest12-21ksoftirqd/019:50:450
2111440994435,6cyclictest16950irq/224-800000022:00:000
2111440994432,9cyclictest1-21systemd19:35:010
211144099442,34cyclictest2116868-21modprobe20:55:000
211145199439,5cyclictest131rcu_preempt00:05:023
211145199439,5cyclictest131rcu_preempt00:05:013
211145199437,9cyclictest37-21ksoftirqd/323:13:543
211145199437,8cyclictest37-21ksoftirqd/321:40:003
211145199435,8cyclictest37-21ksoftirqd/319:30:003
211145199435,7cyclictest37-21ksoftirqd/319:40:003
211145199433,10cyclictest37-21ksoftirqd/319:51:543
2111451994323,14cyclictest3467602-21systemd-journal22:05:003
211145199431,9cyclictest0-21swapper/323:00:493
211145199431,9cyclictest0-21swapper/322:24:163
211145199431,6cyclictest2126791-21systemd-run23:50:003
2111451994314,5cyclictest131rcu_preempt00:10:013
2111451994311,5cyclictest131rcu_preempt21:55:013
2111451994311,5cyclictest131rcu_preempt21:55:003
2111449994336,4cyclictest2123427-21kworker/2:323:33:152
2111449994335,5cyclictest2094954-21kworker/2:319:13:262
2111449994326,12cyclictest275-21dbus-daemon23:19:592
211144499438,6cyclictest131rcu_preempt22:05:011
211144499437,5cyclictest131rcu_preempt23:55:021
211144499437,5cyclictest131rcu_preempt23:55:021
2111444994330,9cyclictest2124436-21sshd23:04:541
2111444994328,11cyclictest364-21snmpd20:56:021
2111444994323,15cyclictest364-21snmpd21:42:241
2111444994320,17cyclictest1-21systemd23:00:001
211144099435,8cyclictest12-21ksoftirqd/000:17:010
211144099435,6cyclictest12-21ksoftirqd/019:49:350
211144099434,7cyclictest12-21ksoftirqd/023:04:540
211144099433,8cyclictest12-21ksoftirqd/019:20:020
2111440994326,12cyclictest275-21dbus-daemon23:25:010
2111440994326,12cyclictest275-21dbus-daemon23:25:010
2111440994318,16cyclictest16850irq/134-800000021:21:550
211144099431,12cyclictest12-21ksoftirqd/021:05:020
2111440994310,29cyclictest2124157-21sshd22:59:540
211145199428,5cyclictest131rcu_preempt22:45:003
211145199425,8cyclictest37-21ksoftirqd/320:45:013
2111451994233,5cyclictest2123140-21sendmail22:40:003
211145199422,7cyclictest37-21ksoftirqd/320:29:543
211145199422,7cyclictest37-21ksoftirqd/320:29:543
211145199421,8cyclictest0-21swapper/322:29:153
211144999426,7cyclictest30-21ksoftirqd/222:20:032
211144499425,6cyclictest131rcu_preempt22:20:001
2111444994222,16cyclictest364-21snmpd19:37:171
2111444994221,16cyclictest364-21snmpd23:21:061
2111444994221,16cyclictest364-21snmpd23:21:051
2111444994217,19cyclictest2114916-21latency_hist20:15:031
211144099426,7cyclictest12-21ksoftirqd/022:10:010
211144099426,7cyclictest12-21ksoftirqd/022:10:010
211144099425,5cyclictest12-21ksoftirqd/020:15:010
2111440994220,8cyclictest12-21ksoftirqd/000:05:410
211144099421,11cyclictest2120636-21cut21:50:020
211145199418,11cyclictest37-21ksoftirqd/319:13:403
2111449994127,11cyclictest30-21ksoftirqd/221:40:002
2111444994122,15cyclictest364-21snmpd00:21:441
211144099417,4cyclictest131rcu_preempt23:32:550
2111440994127,10cyclictest2124675-21sshd23:06:530
211144099412,6cyclictest1-21systemd23:15:010
2111440994126,10cyclictest275-21dbus-daemon20:10:010
211144099412,35cyclictest1-21systemd23:50:010
2111440994116,16cyclictest16850irq/134-800000022:24:550
2111440994111,7cyclictest0-21swapper/019:40:130
2111440994111,7cyclictest0-21swapper/019:40:130
211144099411,12cyclictest12-21ksoftirqd/022:50:000
211144099411,10cyclictest2128887-21munin-run00:30:010
211145199404,7cyclictest37-21ksoftirqd/323:20:003
211145199400,6cyclictest150-21pr/ttyS220:08:413
2111449994025,11cyclictest2111546-21date19:10:012
2111444994026,10cyclictest48-21kcompactd023:29:391
2111440994033,4cyclictest16950irq/224-800000022:30:000
2111440994010,5cyclictest131rcu_preempt00:02:540
2111440994010,5cyclictest131rcu_preempt00:02:530
211145199395,5cyclictest37-21ksoftirqd/321:31:093
211144099396,3cyclictest131rcu_preempt00:40:010
2111440993923,11cyclictest3467602-21systemd-journal22:45:000
211145199385,8cyclictest37-21ksoftirqd/319:45:023
211145199385,8cyclictest37-21ksoftirqd/319:45:013
211145199380,7cyclictest150-21pr/ttyS222:31:393
211145199380,6cyclictest150-21pr/ttyS200:15:253
2111444993825,10cyclictest364-21snmpd21:25:111
2111440993829,6cyclictest12-21ksoftirqd/021:45:010
211144099381,8cyclictest2122159-21sort22:20:020
2111440993812,8cyclictest0-21swapper/020:31:040
211145199372,8cyclictest37-21ksoftirqd/321:25:003
2111451993725,10cyclictest37-21ksoftirqd/320:20:003
2111451993724,9cyclictest3467602-21systemd-journal23:35:003
211145199371,9cyclictest0-21swapper/320:39:043
2111451993716,5cyclictest361rcuc/323:06:273
211144999378,5cyclictest131rcu_preempt21:27:362
211144099377,5cyclictest131rcu_preempt19:11:540
211145199361,9cyclictest0-21swapper/320:53:473
211144099357,4cyclictest131rcu_preempt19:22:170
2111444993426,5cyclictest2111446-21latency_hist19:10:021
2111451992916,10cyclictest2111493-21cat19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional