You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 09:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Jul 23, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
205949499118108,7cyclictest23-21ksoftirqd/123:10:021
2059486999488,3cyclictest0-21swapper/021:00:020
2059486999336,52cyclictest16850irq/134-800000021:40:010
2059486998735,45cyclictest1-21systemd23:10:010
205950099842,78cyclictest2087089-21sh21:55:023
2059486998230,4cyclictest16850irq/134-800000000:40:010
2059486998224,11cyclictest2138115-21ssh23:41:560
2059486998131,11cyclictest2108398-21cron22:40:010
2059486998131,11cyclictest2108398-21cron22:40:000
2059486998128,13cyclictest2130717-21ssh23:26:170
2059486998128,13cyclictest2130717-21ssh23:26:160
2059486998128,12cyclictest2142851-21ssh23:51:480
2059486998022,13cyclictest2093351-21ssh22:08:060
2059486997929,11cyclictest2088158-21ssh21:57:080
2059486997924,13cyclictest2149363-21ssh00:05:200
205948699782,35cyclictest1-21systemd00:00:010
2059486997822,14cyclictest2094794-21ssh22:11:000
2059486997728,13cyclictest2097185-21ssh22:15:540
2059486997628,10cyclictest2152344-21ssh00:11:490
2059486997627,8cyclictest2091806-21cron22:05:000
2059486997625,12cyclictest2071711-21ssh21:22:310
2059486997623,15cyclictest2080357-21ssh21:40:210
2059486997617,17cyclictest16850irq/134-800000021:54:140
205948699761,39cyclictest2155616-21ssh00:18:510
205948699761,39cyclictest2067916-21ssh21:14:500
205948699761,38cyclictest2148417-21ssh00:03:410
2059486997528,12cyclictest2083629-21ssh21:47:340
2059486997527,11cyclictest2110499-21ssh22:44:090
2059486997518,17cyclictest16850irq/134-800000023:45:530
2059486997518,17cyclictest16850irq/134-800000023:45:530
2059486997518,17cyclictest16850irq/134-800000000:34:130
2059486997517,17cyclictest16850irq/134-800000022:48:570
2059486997517,17cyclictest16850irq/134-800000022:48:570
2059486997419,18cyclictest16850irq/134-800000023:18:300
205949699731,68cyclictest2158326-21sh00:25:022
205949499733,49cyclictest2118151-21ssh23:00:021
2059486997317,17cyclictest16850irq/134-800000022:24:240
205949499728,47cyclictest2065667-21diskmemload21:20:021
2059486997227,11cyclictest2102254-21ssh22:26:380
2059486997216,18cyclictest16850irq/134-800000021:25:360
205948699721,14cyclictest2127922-21ssh23:20:120
205950099712,14cyclictest2062766-21sshd20:10:313
2059486997126,12cyclictest2105611-21ssh22:33:530
205948699712,38cyclictest2059662-21sshd19:12:250
2059486997026,9cyclictest2114888-21ssh22:53:150
2059486997023,12cyclictest2076276-21ssh21:31:580
205948699693,7cyclictest2118088-21/usr/sbin/munin23:00:010
2059486996917,18cyclictest16850irq/134-800000021:18:580
2059500996826,38cyclictest275-21dbus-daemon23:05:023
205948699682,8cyclictest2160621-21latency_hist00:30:020
205948699682,8cyclictest2160621-21latency_hist00:30:020
205950099672,11cyclictest2146595-40mandb00:00:063
2059486996736,26cyclictest0-21swapper/000:25:000
2059500996622,39cyclictest37-21ksoftirqd/323:06:433
205950099661,13cyclictest2062267-21sshd20:01:273
2059486996629,7cyclictest274-21cron21:10:000
2059486996627,6cyclictest2137113-21sendmail23:40:000
2059500996516,45cyclictest2065272-21sed21:00:023
2059486996535,26cyclictest16850irq/134-800000023:15:020
2059500996422,38cyclictest37-21ksoftirqd/323:56:103
205950099642,13cyclictest2153700-21munin-run00:15:013
2059500996413,47cyclictest3467602-21systemd-journal21:39:323
2059500996411,47cyclictest0-21swapper/322:42:373
2059500996323,35cyclictest2103177-21sh22:28:423
205950099632,14cyclictest2063577-21cron20:30:003
205950099632,11cyclictest2117752-21ssh22:59:223
205950099631,14cyclictest2064592-21sshd20:45:343
205950099631,14cyclictest2064592-21sshd20:45:343
205950099624,7cyclictest2059647-21sshd19:10:233
205950099623,51cyclictest2061029-21grep19:40:023
2059500996231,26cyclictest2160403-21sh00:29:463
2059500996231,26cyclictest2160403-21sh00:29:463
205950099622,15cyclictest2139433-21munin-run23:45:003
2059486996224,10cyclictest2120478-21munin-plugin-st23:05:000
205950099612,14cyclictest2151416-21munin-run00:10:013
205950099612,12cyclictest48-21kcompactd021:25:073
205950099612,11cyclictest2097657-21ssh22:16:533
205950099611,11cyclictest2154127-21ssh00:15:293
2059486996128,8cyclictest2060187-21sshd19:20:250
2059486996123,13cyclictest275-21dbus-daemon19:40:010
205950099603,6cyclictest364-21snmpd22:47:143
205950099603,6cyclictest364-21snmpd22:47:143
205950099602,16cyclictest2061511-21munin-run19:50:003
2059500996021,34cyclictest2093547-21sh22:08:303
205950099601,14cyclictest2106264-21ssh22:35:033
205950099601,13cyclictest2077358-21sshd21:34:263
205950099601,11cyclictest2059927-21sshd19:19:263
205950099592,17cyclictest2115603-21ssh22:54:523
205950099592,16cyclictest2162024-21ssh00:33:033
205950099592,15cyclictest2143940-21ssh23:54:233
205950099592,14cyclictest48-21kcompactd023:10:223
205950099592,13cyclictest2080392-21sshd21:40:263
205950099591,12cyclictest364-21snmpd19:53:543
205950099591,12cyclictest2064350-21sshd20:42:263
2059496995941,15cyclictest1-21systemd20:20:002
205949499594,12cyclictest23-21ksoftirqd/123:25:011
205950099582,12cyclictest48-21kcompactd021:58:313
2059500995820,33cyclictest37-21ksoftirqd/320:19:593
205950099581,12cyclictest48-21kcompactd023:23:433
205950099581,12cyclictest364-21snmpd20:38:283
205950099573,6cyclictest2096613-21ssh22:15:013
2059500995725,26cyclictest2090457-21sh22:01:533
205950099572,11cyclictest2060489-21munin-run19:30:013
2059500995716,35cyclictest2065649-21cron21:10:003
205950099571,15cyclictest2072846-21unin-run21:25:003
205950099571,13cyclictest2133454-21ssh23:32:073
2059496995723,25cyclictest30-21ksoftirqd/222:18:452
205949499574,9cyclictest23-21ksoftirqd/121:43:581
2059494995712,5cyclictest131rcu_preempt22:33:051
205950099565,6cyclictest364-21snmpd21:11:113
2059500995623,29cyclictest2158136-21latency_hist00:25:023
2059500995623,29cyclictest2099283-21sh22:20:113
205950099562,16cyclictest2083101-21ssh21:46:183
205950099562,15cyclictest2140529-21ssh23:46:593
205950099562,15cyclictest2140529-21ssh23:46:593
2059500995621,31cyclictest37-21ksoftirqd/321:16:163
205950099562,11cyclictest3467602-21systemd-journal00:39:393
2059496995647,6cyclictest2053655-21kworker/2:123:44:522
2059496995641,7cyclictest2065168-21kworker/2:200:13:142
2059496995619,15cyclictest30-21ksoftirqd/223:30:012
2059496995619,15cyclictest30-21ksoftirqd/223:30:012
2059494995613,5cyclictest131rcu_preempt21:39:051
2059494995612,6cyclictest131rcu_preempt22:49:181
2059494995612,6cyclictest131rcu_preempt22:49:181
205950099559,40cyclictest0-21swapper/321:05:003
205950099552,9cyclictest48-21kcompactd020:20:093
2059496995545,6cyclictest2160569-21kworker/2:300:31:382
2059496995535,14cyclictest3467602-21systemd-journal00:00:012
2059494995514,16cyclictest131rcu_preempt23:49:451
2059494995514,16cyclictest131rcu_preempt23:49:441
2059494995513,9cyclictest131rcu_preempt00:05:131
205948699553,35cyclictest2061485-21kworker/u8:020:05:020
2059486995533,2cyclictest0-21swapper/023:35:010
205950099543,7cyclictest2130777-21ssh23:26:253
205950099543,7cyclictest2130777-21ssh23:26:243
205950099542,8cyclictest2064874-21latency_hist20:55:003
205950099542,12cyclictest2108388-21rm22:39:553
205950099542,12cyclictest2108388-21rm22:39:553
2059500995412,37cyclictest2065667-21diskmemload23:40:003
205950099541,17cyclictest1-21systemd20:10:013
205950099541,17cyclictest1-21systemd20:10:013
205950099541,12cyclictest2060694-21sshd19:30:263
2059496995445,6cyclictest30-21ksoftirqd/223:20:012
2059496995445,6cyclictest2053655-21kworker/2:121:46:502
2059496995444,7cyclictest2053655-21kworker/2:121:19:202
2059496995444,6cyclictest2053655-21kworker/2:122:20:312
2059496995439,7cyclictest2065168-21kworker/2:222:40:342
2059496995426,7cyclictest30-21ksoftirqd/221:55:002
2059496995417,7cyclictest30-21ksoftirqd/222:55:012
205949499545,7cyclictest23-21ksoftirqd/123:01:131
2059494995446,5cyclictest23-21ksoftirqd/122:24:291
2059494995415,10cyclictest23-21ksoftirqd/122:10:011
2059494995413,6cyclictest131rcu_preempt00:00:011
205950099531,13cyclictest2126042-21ssh23:16:323
2059496995345,5cyclictest2065168-21kworker/2:222:27:372
2059496995344,6cyclictest2053655-21kworker/2:121:27:022
2059496995344,5cyclictest2065168-21kworker/2:223:03:432
205949499534,9cyclictest23-21ksoftirqd/123:37:181
205950099522,14cyclictest2062057-21munin-run20:00:013
205950099522,14cyclictest2060262-21cat19:25:023
2059496995245,4cyclictest2065168-21kworker/2:223:05:152
2059496995244,5cyclictest2065168-21kworker/2:223:21:082
2059496995244,5cyclictest2053655-21kworker/2:119:54:262
2059496995243,6cyclictest2053655-21kworker/2:123:45:252
2059496995243,6cyclictest2053655-21kworker/2:123:45:252
2059496995243,5cyclictest2053655-21kworker/2:123:38:472
2059496995243,5cyclictest2053655-21kworker/2:100:07:262
2059496995242,6cyclictest2053655-21kworker/2:121:43:342
2059496995214,10cyclictest30-21ksoftirqd/223:32:552
2059494995217,8cyclictest131rcu_preempt00:14:161
2059494995210,17cyclictest131rcu_preempt21:10:071
2059500995110,34cyclictest0-21swapper/319:10:023
2059496995143,5cyclictest2053655-21kworker/2:122:58:132
2059496995143,5cyclictest2053655-21kworker/2:122:04:112
2059496995143,5cyclictest2053655-21kworker/2:121:39:252
2059496995142,6cyclictest2160569-21kworker/2:300:39:452
2059496995142,6cyclictest2065168-21kworker/2:222:37:392
2059496995142,6cyclictest2065168-21kworker/2:222:37:392
2059496995134,13cyclictest3467602-21systemd-journal20:44:262
2059496995134,13cyclictest1-21systemd21:25:002
205949499518,8cyclictest23-21ksoftirqd/121:45:541
205949499516,8cyclictest23-21ksoftirqd/123:18:591
205949499515,10cyclictest23-21ksoftirqd/123:53:131
2059494995112,9cyclictest131rcu_preempt21:34:531
205950099506,5cyclictest0-21swapper/319:45:013
2059496995042,5cyclictest2065168-21kworker/2:222:45:552
2059496995042,5cyclictest2065168-21kworker/2:222:45:552
2059496995042,5cyclictest2065168-21kworker/2:222:31:322
2059496995042,5cyclictest2053655-21kworker/2:123:51:422
2059496995042,5cyclictest2053655-21kworker/2:122:14:392
2059496995042,5cyclictest2053655-21kworker/2:120:50:202
2059496995041,6cyclictest2053655-21kworker/2:122:06:252
2059496995041,6cyclictest2053655-21kworker/2:121:58:592
2059496995041,6cyclictest2053655-21kworker/2:121:11:242
2059496995041,6cyclictest2053655-21kworker/2:100:00:542
205949499507,7cyclictest23-21ksoftirqd/100:26:391
205949499507,7cyclictest23-21ksoftirqd/100:26:381
205949499506,6cyclictest131rcu_preempt22:15:011
205949499505,9cyclictest23-21ksoftirqd/121:10:001
2059494995012,5cyclictest131rcu_preempt00:00:351
2059494995011,6cyclictest131rcu_preempt20:20:001
205949499501,15cyclictest0-21swapper/123:34:401
205948699503,9cyclictest2064082-21cron20:40:000
2059496994941,5cyclictest2065168-21kworker/2:200:27:232
2059496994941,5cyclictest2065168-21kworker/2:200:27:222
2059496994941,5cyclictest2065168-21kworker/2:200:15:502
2059496994941,5cyclictest2053655-21kworker/2:121:31:462
2059496994941,5cyclictest2053655-21kworker/2:120:49:012
2059496994941,5cyclictest2053655-21kworker/2:120:49:012
2059496994941,5cyclictest2053655-21kworker/2:120:27:542
2059496994941,5cyclictest2053655-21kworker/2:119:33:272
2059496994941,5cyclictest2047556-21kworker/2:219:21:002
205949699492,5cyclictest131rcu_preempt21:00:012
205949499497,5cyclictest131rcu_preempt21:55:051
2059494994940,6cyclictest23-21ksoftirqd/121:23:561
2059494994937,8cyclictest23-21ksoftirqd/119:30:021
2059494994915,9cyclictest131rcu_preempt00:33:491
205950099482,15cyclictest2063851-21systemd-run20:35:013
205950099482,15cyclictest2063851-21systemd-run20:35:013
2059496994840,5cyclictest2065168-21kworker/2:223:10:142
2059496994840,5cyclictest2053655-21kworker/2:119:49:272
2059496994840,5cyclictest2053655-21kworker/2:119:30:002
2059496994840,5cyclictest2047556-21kworker/2:219:12:282
2059496994839,6cyclictest2053655-21kworker/2:120:03:442
205949699481,24cyclictest0-21swapper/220:10:012
205949699481,24cyclictest0-21swapper/220:10:012
2059494994836,6cyclictest2060967-21cron19:40:001
2059494994813,8cyclictest131rcu_preempt00:16:051
2059494994813,5cyclictest131rcu_preempt23:28:511
2059494994813,5cyclictest131rcu_preempt23:28:511
2059494994812,6cyclictest131rcu_preempt00:25:001
2059494994812,5cyclictest131rcu_preempt21:00:011
2059494994810,6cyclictest131rcu_preempt22:51:061
2059496994739,5cyclictest2053655-21kworker/2:120:10:042
2059496994739,5cyclictest2053655-21kworker/2:119:58:292
205949499478,6cyclictest131rcu_preempt22:26:411
2059494994713,6cyclictest131rcu_preempt00:36:081
2059494994712,5cyclictest131rcu_preempt22:37:011
2059494994712,5cyclictest131rcu_preempt22:37:011
2059494994712,5cyclictest131rcu_preempt22:01:361
2059486994732,11cyclictest3467602-21systemd-journal20:05:020
2059486994732,11cyclictest3467602-21systemd-journal20:05:020
2059496994639,4cyclictest2053655-21kworker/2:121:05:062
2059496994639,4cyclictest2053655-21kworker/2:121:02:352
2059496994638,5cyclictest2053655-21kworker/2:120:35:312
2059496994638,5cyclictest2053655-21kworker/2:120:20:492
2059496994638,5cyclictest2053655-21kworker/2:119:40:232
2059496994638,5cyclictest2053655-21kworker/2:119:36:312
2059494994639,5cyclictest221rcuc/121:50:061
2059494994610,6cyclictest131rcu_preempt23:41:181
205948699462,39cyclictest284-21systemd-logind19:35:000
2059496994537,5cyclictest2053655-21kworker/2:120:30:432
2059496994537,5cyclictest2053655-21kworker/2:120:30:432
2059496994537,5cyclictest2047556-21kworker/2:219:15:592
2059494994530,11cyclictest3467602-21systemd-journal20:39:321
2059494994512,5cyclictest131rcu_preempt23:13:041
2059486994529,12cyclictest1-21systemd20:20:010
2059494994428,12cyclictest23-21ksoftirqd/119:50:021
2059494994415,5cyclictest131rcu_preempt22:16:291
2059494994414,5cyclictest131rcu_preempt21:26:581
2059494994411,5cyclictest131rcu_preempt22:42:481
2059494994410,5cyclictest131rcu_preempt20:00:011
205948699439,8cyclictest12-21ksoftirqd/020:52:260
205948699431,38cyclictest0-21swapper/020:15:000
205949499429,7cyclictest131rcu_preempt20:11:111
205949499423,7cyclictest23-21ksoftirqd/120:05:011
2059494994233,5cyclictest2059946-21latency_hist19:19:591
2059494994231,8cyclictest23-21ksoftirqd/120:35:011
2059494994231,8cyclictest23-21ksoftirqd/120:35:011
2059486994235,4cyclictest16950irq/224-800000019:20:000
2059486994228,10cyclictest3467602-21systemd-journal20:50:000
2059486994228,10cyclictest3467602-21systemd-journal20:50:000
2059494994132,7cyclictest23-21ksoftirqd/119:52:261
2059494994130,8cyclictest2061241-21sshd19:44:251
2059494994128,10cyclictest23-21ksoftirqd/119:25:021
2059494994112,5cyclictest131rcu_preempt20:25:021
2059486994131,7cyclictest12-21ksoftirqd/019:55:020
205948699411,35cyclictest2062016-21cron20:00:000
2059486994027,9cyclictest1-21systemd19:50:000
205949499399,10cyclictest23-21ksoftirqd/120:45:001
2059494993930,7cyclictest23-21ksoftirqd/121:05:031
2059494993923,12cyclictest275-21dbus-daemon20:55:001
205949499390,6cyclictest150-21pr/ttyS220:29:461
2059486993925,10cyclictest12-21ksoftirqd/020:25:010
2059486993921,14cyclictest364-21snmpd19:27:240
2059494993827,8cyclictest23-21ksoftirqd/119:10:021
2059486993825,10cyclictest2063790-21sed20:30:020
2059494993727,7cyclictest23-21ksoftirqd/120:10:011
2059494993727,7cyclictest23-21ksoftirqd/120:10:011
2059486993731,4cyclictest16950irq/224-800000020:32:260
2059486993731,4cyclictest16950irq/224-800000020:32:260
2059494993627,7cyclictest23-21ksoftirqd/120:45:341
2059494993627,7cyclictest23-21ksoftirqd/120:45:341
205949499361,6cyclictest150-21pr/ttyS219:32:051
2059486993622,9cyclictest12-21ksoftirqd/019:10:020
2059486993619,12cyclictest12-21ksoftirqd/021:05:010
2059496993522,8cyclictest2059608-21cat19:10:022
2059486993517,3cyclictest131rcu_preempt19:45:020
205948699351,30cyclictest3467602-21systemd-journal20:43:560
2059494993423,8cyclictest23-21ksoftirqd/119:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional