You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-07 - 18:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Sep 07, 2026 12:45:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2083180999360,30cyclictest0-21swapper/011:35:020
208319099872,47cyclictest2107499-21ssh09:48:262
2083180998426,10cyclictest2167428-21ssh11:55:130
2083190998327,22cyclictest2086398-21latency_hist08:10:022
2083180997922,13cyclictest2135623-21ssh10:48:060
2083180997911,41cyclictest2084660-21cron07:40:010
208319099782,39cyclictest2187582-21ssh12:38:292
208319099782,39cyclictest2187582-21ssh12:38:292
208319099772,66cyclictest2138824-21ssh10:55:022
208319099772,66cyclictest2138824-21ssh10:55:022
208319099772,37cyclictest2179664-21ssh12:21:262
2083186997767,6cyclictest2136541-21sh10:50:031
2083180997620,31cyclictest2148645-21sh11:15:290
2083190997529,42cyclictest3467602-21systemd-journal08:30:162
2083190997417,24cyclictest2089361-21diskmemload09:15:592
2083190997417,24cyclictest2089361-21diskmemload09:15:582
208319099741,43cyclictest2095463-21ssh09:22:412
208318099741,38cyclictest2180376-21ssh12:23:070
2083180997313,33cyclictest12-21ksoftirqd/009:51:000
208319099722,41cyclictest2125602-21ssh10:26:352
208319099722,37cyclictest2118431-21ssh10:11:232
208319099722,37cyclictest2118431-21ssh10:11:222
208319099717,35cyclictest2103298-21ssh09:39:372
208319099712,42cyclictest364-21snmpd11:18:382
208319099712,39cyclictest2145814-21munin-run11:10:002
2083190997118,20cyclictest2089361-21diskmemload09:40:202
2083190997118,20cyclictest2089361-21diskmemload09:40:202
208319099702,41cyclictest2147805-21ssh11:14:012
208319099702,32cyclictest2152415-21ssh11:23:462
208319099701,42cyclictest2115187-21rm10:04:552
208319099692,51cyclictest0-21swapper/209:15:022
208319099692,51cyclictest0-21swapper/209:15:022
2083180996822,14cyclictest2117300-21ssh10:09:190
208319099661,61cyclictest2182894-21ssh12:28:282
208319099661,61cyclictest2167942-21ssh11:56:312
208319099661,60cyclictest2083602-21cron07:17:002
208319099661,60cyclictest2083602-21cron07:17:002
2083180996653,4cyclictest19350irq/34-mmc108:25:010
2083190996522,39cyclictest2156723-21sh11:32:462
2083190996520,41cyclictest2173689-21sh12:09:022
208319099651,60cyclictest2167089-21ssh11:54:592
208319099651,60cyclictest2167089-21ssh11:54:592
208319099651,60cyclictest2158924-21ssh11:37:332
208319099651,60cyclictest2116605-21ssh10:07:422
208319099651,59cyclictest2124090-21ssh10:23:372
208319099651,59cyclictest2124090-21ssh10:23:362
208319099641,60cyclictest2161430-21ssh11:42:442
208319099641,60cyclictest2161430-21ssh11:42:442
208319099641,59cyclictest2176229-21ssh12:14:262
208319099641,59cyclictest2169966-21ssh12:00:342
208319099641,59cyclictest2163474-21ssh11:47:022
208319099641,59cyclictest2135598-21ssh10:48:042
208319099641,59cyclictest2127929-21ssh10:31:312
208319099641,59cyclictest2111158-21ssh09:56:022
208319099641,59cyclictest2097018-21ssh09:25:482
208319099641,59cyclictest2097018-21ssh09:25:472
2083195996344,14cyclictest275-21dbus-daemon07:40:013
208319099631,58cyclictest2184196-21ssh12:31:032
208319099631,58cyclictest2154027-21ssh11:26:592
208319099631,58cyclictest2142013-21ssh11:01:442
208319099631,58cyclictest2142013-21ssh11:01:442
208319099631,58cyclictest2139396-21ssh10:55:592
208319099631,58cyclictest2132114-21ssh10:40:242
208319099631,58cyclictest2131342-21ssh10:39:062
208319099631,58cyclictest2109919-21ssh09:53:342
208319099622,56cyclictest2088052-21sshd08:44:172
208319099622,56cyclictest2088052-21sshd08:44:172
208319099621,57cyclictest2177072-21ssh12:15:552
208319099621,57cyclictest2120589-21ssh10:15:582
208319099621,57cyclictest2099187-21ssh09:30:232
2083180996232,14cyclictest2185549-21ssh12:34:090
2083180996157,2cyclictest0-21swapper/009:25:010
208319099592,52cyclictest3467602-21systemd-journal08:01:012
2083190995913,41cyclictest0-21swapper/207:26:312
2083180995915,14cyclictest0-21swapper/012:29:230
208319099582,51cyclictest364-21snmpd07:48:032
208319099581,52cyclictest48-21kcompactd008:27:572
208319099581,52cyclictest2088321-21munin-run08:50:002
2083186995838,14cyclictest3467602-21systemd-journal11:20:001
2083195995711,6cyclictest131rcu_preempt12:00:003
208319099572,50cyclictest2085471-21munin-run07:55:002
208319099571,52cyclictest2087010-21sshd08:23:172
208319099571,51cyclictest364-21snmpd08:18:032
208319099571,51cyclictest364-21snmpd07:24:412
2083186995738,14cyclictest3467602-21systemd-journal11:45:001
2083186995738,14cyclictest3467602-21systemd-journal11:45:001
2083180995727,14cyclictest2154526-21ssh11:28:130
2083180995719,19cyclictest2134092-21ssh10:45:000
208319099561,50cyclictest48-21kcompactd009:05:072
2083180995634,17cyclictest275-21dbus-daemon10:00:010
208319099552,48cyclictest364-21snmpd08:54:402
208319099552,48cyclictest2086454-21sshd08:10:172
208319099551,49cyclictest48-21kcompactd007:57:592
208318099551,38cyclictest2162367-21ssh11:44:550
208318099551,38cyclictest2162367-21ssh11:44:550
208319099542,47cyclictest364-21snmpd07:44:452
208319099542,47cyclictest364-21snmpd07:44:442
208319099541,48cyclictest364-21snmpd07:12:512
2083180995446,6cyclictest12-21ksoftirqd/009:25:520
2083180995446,6cyclictest12-21ksoftirqd/009:25:520
2083180995426,11cyclictest2173937-21ssh12:09:330
208318099541,30cyclictest131rcu_preempt10:14:340
208318099541,30cyclictest131rcu_preempt10:14:340
208319099532,47cyclictest2088798-21sshd08:56:172
208319099532,47cyclictest2088798-21sshd08:56:172
208319099531,47cyclictest48-21kcompactd008:36:502
208319099531,47cyclictest364-21snmpd07:34:262
2083180995345,4cyclictest16950irq/224-800000011:35:000
2083180995327,11cyclictest2175918-21ssh12:13:480
208318099531,3cyclictest131rcu_preempt09:11:020
208318099531,3cyclictest131rcu_preempt09:11:010
208318099531,36cyclictest2124983-21ssh10:25:080
208319099522,46cyclictest2089108-21munin-run09:04:592
2083186995234,12cyclictest1-21systemd12:25:011
2083180995231,15cyclictest275-21dbus-daemon09:20:010
2083180995231,15cyclictest275-21dbus-daemon09:20:010
2083180995225,12cyclictest2152882-21ssh11:24:500
208318099522,4cyclictest131rcu_preempt10:38:300
208318099521,36cyclictest2137350-21ssh10:51:380
208318099521,36cyclictest2137350-21ssh10:51:370
2083180995134,12cyclictest2164770-21ssh11:50:000
208318099512,29cyclictest131rcu_preempt10:22:030
208318099512,29cyclictest131rcu_preempt10:22:030
2083180995114,31cyclictest150-21pr/ttyS210:56:360
208319599502,34cyclictest2086469-21kworker/u8:308:45:023
208319599502,34cyclictest2086469-21kworker/u8:308:45:023
208318699508,9cyclictest23-21ksoftirqd/110:45:001
208318099501,34cyclictest2187577-21ssh12:38:280
208318099501,34cyclictest2187577-21ssh12:38:280
2083195994932,13cyclictest3467602-21systemd-journal09:14:183
2083195994932,13cyclictest3467602-21systemd-journal09:14:183
208319599492,11cyclictest2101166-21latency_hist09:35:013
208319599491,5cyclictest0-21swapper/309:45:033
208319599491,5cyclictest0-21swapper/309:45:023
2083186994930,15cyclictest3467602-21systemd-journal07:55:011
2083186994929,14cyclictest275-21dbus-daemon12:10:001
2083180994930,15cyclictest2107236-21ssh09:47:530
2083180994919,14cyclictest2165745-21ssh11:51:480
2083180994919,14cyclictest2165745-21ssh11:51:470
2083186994833,12cyclictest23-21ksoftirqd/109:40:031
2083180994830,13cyclictest275-21dbus-daemon10:35:010
2083180994826,8cyclictest2145704-21ssh11:09:440
2083180994823,10cyclictest2105203-21ssh09:43:370
2083180994823,10cyclictest2105203-21ssh09:43:370
2083180994822,11cyclictest2142984-21ssh11:03:570
2083180994822,11cyclictest2142984-21ssh11:03:570
208318099481,3cyclictest131rcu_preempt09:36:160
2083195994731,12cyclictest275-21dbus-daemon11:10:013
2083195994711,6cyclictest131rcu_preempt10:02:293
2083180994729,14cyclictest2100499-21rm09:33:300
2083180994727,16cyclictest2147039-21ssh11:12:170
2083180994717,5cyclictest12-21ksoftirqd/010:05:020
2083195994633,10cyclictest37-21ksoftirqd/310:15:013
2083195994633,10cyclictest37-21ksoftirqd/310:15:013
2083195994631,11cyclictest2094091-21wc09:20:023
2083195994631,11cyclictest2094091-21wc09:20:013
2083195994630,12cyclictest3467602-21systemd-journal11:30:013
2083195994629,14cyclictest275-21dbus-daemon07:59:593
2083195994626,14cyclictest275-21dbus-daemon11:20:003
2083195994624,5cyclictest37-21ksoftirqd/311:13:343
2083195994621,21cyclictest37-21ksoftirqd/311:33:363
208319099461,40cyclictest364-21snmpd07:36:142
208318699462,40cyclictest2145918-21sh11:10:021
208318699462,3cyclictest131rcu_preempt10:32:591
2083180994639,4cyclictest16950irq/224-800000012:17:230
2083180994638,5cyclictest16850irq/134-800000009:10:010
2083180994620,11cyclictest2121406-21ssh10:17:530
208319599459,16cyclictest2085979-21kworker/u8:108:30:033
2083195994531,10cyclictest1-21systemd12:40:013
2083195994531,10cyclictest1-21systemd12:40:013
2083195994527,14cyclictest37-21ksoftirqd/312:17:503
2083186994533,8cyclictest3467602-21systemd-journal08:20:171
2083186994532,10cyclictest2122651-21sshd10:20:181
2083186994532,10cyclictest2122651-21sshd10:20:171
2083186994530,12cyclictest23-21ksoftirqd/109:35:001
208318099453,9cyclictest2086754-21latency_hist08:20:000
2083180994537,5cyclictest16850irq/134-800000012:04:580
2083195994428,13cyclictest37-21ksoftirqd/310:51:523
2083195994428,13cyclictest37-21ksoftirqd/310:51:523
208319599442,5cyclictest131rcu_preempt10:19:593
2083195994420,5cyclictest37-21ksoftirqd/310:57:143
2083195994419,6cyclictest37-21ksoftirqd/311:53:233
2083195994419,6cyclictest37-21ksoftirqd/311:53:233
208319599441,6cyclictest0-21swapper/310:31:313
208319599441,5cyclictest131rcu_preempt10:45:003
208318699446,6cyclictest131rcu_preempt11:35:011
2083186994427,13cyclictest2083627-21cron07:20:011
2083186994427,13cyclictest2083627-21cron07:20:001
2083186994426,14cyclictest2094017-21tr09:20:001
2083186994426,14cyclictest2094017-21tr09:19:591
2083186994425,14cyclictest275-21dbus-daemon07:45:011
2083186994425,14cyclictest275-21dbus-daemon07:45:011
2083186994420,8cyclictest23-21ksoftirqd/110:28:431
208318699441,5cyclictest131rcu_preempt09:50:211
208318099446,30cyclictest2087802-21modprobe08:40:000
208318099443,7cyclictest2086480-21cron08:15:000
2083180994436,4cyclictest16950irq/224-800000009:00:010
2083180994436,4cyclictest16950irq/224-800000009:00:010
2083195994320,5cyclictest37-21ksoftirqd/311:20:293
2083195994319,5cyclictest37-21ksoftirqd/312:10:513
2083195994319,5cyclictest37-21ksoftirqd/310:46:263
2083195994319,5cyclictest37-21ksoftirqd/309:26:193
2083195994319,5cyclictest37-21ksoftirqd/309:26:193
208318699434,3cyclictest131rcu_preempt12:31:441
208318699433,5cyclictest131rcu_preempt10:13:331
208318699433,5cyclictest131rcu_preempt10:13:321
2083186994330,10cyclictest23-21ksoftirqd/108:10:031
2083186994326,14cyclictest1-21systemd10:55:011
2083186994326,14cyclictest1-21systemd10:55:001
208318699431,4cyclictest131rcu_preempt10:56:311
208318699431,3cyclictest131rcu_preempt08:30:001
208318699431,21cyclictest0-21swapper/111:00:531
208318699431,21cyclictest0-21swapper/111:00:521
2083180994327,10cyclictest309-21in:imuxsock07:50:010
208319599426,6cyclictest131rcu_preempt12:28:283
208319599426,5cyclictest131rcu_preempt08:50:013
2083195994224,4cyclictest37-21ksoftirqd/312:07:113
2083195994220,6cyclictest37-21ksoftirqd/309:48:293
2083195994220,5cyclictest37-21ksoftirqd/309:53:013
2083195994219,5cyclictest37-21ksoftirqd/311:03:293
2083195994219,5cyclictest37-21ksoftirqd/311:03:293
2083195994219,4cyclictest37-21ksoftirqd/309:38:443
2083195994218,5cyclictest37-21ksoftirqd/310:26:423
2083195994218,4cyclictest37-21ksoftirqd/310:06:193
2083195994217,7cyclictest37-21ksoftirqd/312:01:243
2083190994216,22cyclictest2083271-21latency_hist07:10:012
208318699425,8cyclictest23-21ksoftirqd/110:35:451
208318699424,3cyclictest131rcu_preempt10:10:001
208318699423,6cyclictest131rcu_preempt12:00:021
2083186994229,10cyclictest2087501-21sshd08:30:171
2083186994226,8cyclictest23-21ksoftirqd/112:11:411
2083186994226,8cyclictest23-21ksoftirqd/111:48:371
208318699422,3cyclictest131rcu_preempt09:28:511
208318699422,3cyclictest131rcu_preempt09:28:501
2083186994219,10cyclictest23-21ksoftirqd/111:37:561
208318699421,6cyclictest131rcu_preempt12:20:001
208318699421,3cyclictest131rcu_preempt11:21:581
208318699421,3cyclictest131rcu_preempt09:44:511
208318699421,3cyclictest131rcu_preempt09:44:511
2083180994220,18cyclictest2086236-21latency_hist08:10:020
208319599416,5cyclictest131rcu_preempt12:23:243
208319599414,6cyclictest131rcu_preempt07:45:173
2083195994121,5cyclictest37-21ksoftirqd/311:36:393
2083195994118,4cyclictest37-21ksoftirqd/310:23:523
2083195994118,4cyclictest37-21ksoftirqd/310:23:523
2083195994117,5cyclictest37-21ksoftirqd/311:42:173
2083195994117,5cyclictest37-21ksoftirqd/311:42:163
208319599411,6cyclictest0-21swapper/309:20:523
2083186994133,6cyclictest23-21ksoftirqd/109:22:321
2083186994127,10cyclictest275-21dbus-daemon10:00:001
208318699411,3cyclictest131rcu_preempt07:20:161
2083186994113,20cyclictest131rcu_preempt07:36:171
2083180994135,4cyclictest16950irq/224-800000008:31:170
208318099412,6cyclictest2083307-21ls07:10:010
208319599406,5cyclictest131rcu_preempt07:23:163
208319599403,7cyclictest131rcu_preempt07:45:023
2083195994031,5cyclictest37-21ksoftirqd/312:32:173
2083195994019,4cyclictest37-21ksoftirqd/307:19:173
2083195994019,4cyclictest37-21ksoftirqd/307:19:173
2083195994018,5cyclictest37-21ksoftirqd/311:46:103
2083195994018,5cyclictest37-21ksoftirqd/309:56:103
2083195994016,5cyclictest37-21ksoftirqd/310:38:353
2083195994016,5cyclictest37-21ksoftirqd/309:10:003
208318699408,6cyclictest131rcu_preempt10:02:311
208318699405,5cyclictest131rcu_preempt11:25:251
208318699403,6cyclictest131rcu_preempt09:15:001
208318699403,6cyclictest131rcu_preempt09:15:001
2083186994030,7cyclictest309-21in:imuxsock08:40:001
2083186994029,8cyclictest2083341-21sshd07:10:171
2083186994026,11cyclictest2089458-21cut09:10:031
208318699402,4cyclictest131rcu_preempt12:03:101
2083186994024,12cyclictest3467602-21systemd-journal12:40:001
2083186994024,12cyclictest3467602-21systemd-journal12:40:001
2083186994015,7cyclictest23-21ksoftirqd/110:18:001
2083180994029,7cyclictest2083379-21munin-run07:14:590
2083180994029,6cyclictest2088563-21cron08:55:000
2083180994017,5cyclictest131rcu_preempt08:05:010
208318099401,36cyclictest2088298-21sshd08:48:180
2083180994011,6cyclictest12-21ksoftirqd/008:30:000
2083195993927,10cyclictest37-21ksoftirqd/307:55:013
208318699394,3cyclictest131rcu_preempt07:30:021
2083186993925,11cyclictest23-21ksoftirqd/112:28:121
2083186993925,10cyclictest1-21systemd11:55:011
2083186993925,10cyclictest1-21systemd11:55:011
208318699392,3cyclictest131rcu_preempt09:45:521
2083186993923,11cyclictest23-21ksoftirqd/108:20:001
208318699391,3cyclictest131rcu_preempt11:11:481
208318699391,3cyclictest131rcu_preempt08:05:011
2083180993923,12cyclictest275-21dbus-daemon07:35:000
2083195993817,5cyclictest37-21ksoftirqd/309:05:003
2083195993812,7cyclictest361rcuc/308:35:013
208318699383,6cyclictest131rcu_preempt08:50:011
208318699383,6cyclictest131rcu_preempt08:00:011
2083186993820,14cyclictest275-21dbus-daemon08:15:001
208318699381,5cyclictest131rcu_preempt09:00:001
208318699381,5cyclictest131rcu_preempt08:59:591
2083180993832,4cyclictest16950irq/224-800000007:17:180
2083180993832,4cyclictest16950irq/224-800000007:17:170
2083180993829,5cyclictest2085695-21cron08:00:000
2083180993826,10cyclictest0-21swapper/008:42:170
2083180993826,10cyclictest0-21swapper/008:42:160
2083180993824,10cyclictest12-21ksoftirqd/007:40:420
2083180993824,10cyclictest12-21ksoftirqd/007:40:420
208318099381,4cyclictest2083932-21munin-run07:25:010
208319599376,5cyclictest131rcu_preempt09:00:023
208319599376,5cyclictest131rcu_preempt09:00:013
208319599372,5cyclictest131rcu_preempt07:31:173
2083195993723,10cyclictest3467602-21systemd-journal08:25:013
208319599371,4cyclictest131rcu_preempt07:45:023
208318699373,5cyclictest131rcu_preempt08:44:171
208318699373,5cyclictest131rcu_preempt08:44:171
2083180993727,7cyclictest12-21ksoftirqd/009:01:170
2083195993626,6cyclictest2084156-21cron07:30:003
2083195993624,8cyclictest3467602-21systemd-journal08:32:173
2083195993618,4cyclictest37-21ksoftirqd/308:04:103
2083195993617,5cyclictest37-21ksoftirqd/308:19:003
208318699361,5cyclictest131rcu_preempt07:49:591
208318699361,11cyclictest2088562-21cron08:55:001
2083180993626,7cyclictest3467602-21systemd-journal07:50:170
208318699354,3cyclictest131rcu_preempt09:05:021
208318699352,4cyclictest131rcu_preempt07:35:021
2083180993524,8cyclictest2084135-21sshd07:25:180
2083195993421,10cyclictest37-21ksoftirqd/308:10:023
2083195993420,11cyclictest2083245-21cat07:10:023
2083195993419,12cyclictest275-21dbus-daemon08:10:023
2083195993324,6cyclictest2088587-21systemctl08:55:013
208319599294,20cyclictest2083380-21grep07:14:593
208318699268,13cyclictest0-21swapper/107:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional