You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-13 - 23:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Mar 13, 2026 12:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22354719910140,17cyclictest16850irq/134-800000011:30:010
223547899985,73cyclictest2320242-21rm12:05:031
2235484999231,30cyclictest2318274-21sh12:00:453
2235471998985,2cyclictest0-21swapper/009:15:020
2235471998879,6cyclictest16950irq/224-800000010:45:020
223548499791,37cyclictest2253304-21ssh09:35:293
2235484997827,18cyclictest2292498-21sh11:03:233
223548499742,40cyclictest2317473-21sshd11:59:233
2235484997213,23cyclictest2241645-21diskmemload10:43:193
2235482997227,12cyclictest30-21ksoftirqd/211:25:172
2235484997116,22cyclictest2315215-21rm11:54:213
223548499711,39cyclictest2241233-21date09:00:023
223548499711,39cyclictest2241233-21date09:00:023
223548499701,37cyclictest2328797-21ssh12:24:083
2235484996923,41cyclictest275-21dbus-daemon11:25:013
223548499692,33cyclictest2307418-21ssh11:36:393
223548499692,33cyclictest2304276-21ssh11:29:573
223548499691,63cyclictest2256519-21ssh09:42:513
223548499691,43cyclictest2297593-21munin-run11:15:013
223548499691,39cyclictest2326654-21rm12:19:323
223548499691,39cyclictest2326654-21rm12:19:313
223548499691,30cyclictest2298587-21ssh11:16:533
223548499681,62cyclictest2277910-21ssh10:30:263
223548499681,61cyclictest2240118-21munin-run08:40:013
223548499671,62cyclictest2252734-21ssh09:34:313
223548499671,62cyclictest2252734-21ssh09:34:313
223548499671,61cyclictest2280591-21ssh10:36:333
2235484996620,42cyclictest37-21ksoftirqd/310:26:233
223548499661,60cyclictest2322581-21ssh12:10:123
2235484996524,36cyclictest2284606-21sh10:45:213
223548499651,60cyclictest2320676-21ssh12:06:133
223548499651,60cyclictest2306004-21ssh11:33:393
223548499651,60cyclictest2262783-21ssh09:56:513
223548499651,60cyclictest2243570-21ssh09:14:123
2235482996527,9cyclictest30-21ksoftirqd/212:29:502
223548499642,58cyclictest2235892-21cron07:17:003
223548499642,58cyclictest2235892-21cron07:17:003
223548499642,31cyclictest2258600-21ssh09:47:303
223548499641,60cyclictest2244204-21ssh09:15:083
223548499641,59cyclictest2334762-21ssh12:36:573
223548499641,59cyclictest2331669-21ssh12:30:033
223548499641,59cyclictest2329635-21ssh12:25:343
223548499641,59cyclictest2293961-21ssh11:06:293
223548499641,59cyclictest2267390-21ssh10:07:023
223548499641,59cyclictest2265931-21ssh10:04:003
223548499641,59cyclictest2265931-21ssh10:03:593
223548499641,59cyclictest2260372-21ssh09:51:193
223548499641,58cyclictest2273367-21ssh10:20:153
223548499641,58cyclictest2273367-21ssh10:20:153
2235482996428,9cyclictest30-21ksoftirqd/209:35:472
2235482996426,8cyclictest30-21ksoftirqd/210:32:062
2235482996426,12cyclictest30-21ksoftirqd/210:44:462
223547199643,57cyclictest2270952-21ssh10:15:020
223547199643,57cyclictest2270952-21ssh10:15:020
2235471996432,29cyclictest121ksoftirqd/010:01:190
2235471996432,29cyclictest121ksoftirqd/010:01:190
223548499632,57cyclictest2239800-21sshd08:33:233
223548499631,58cyclictest2311564-21ssh11:45:433
223548499631,58cyclictest2309226-21ssh11:40:243
223548499631,58cyclictest2271410-21ssh10:15:573
223548499631,58cyclictest2269649-21ssh10:12:093
223548499631,58cyclictest2269649-21ssh10:12:093
223548499631,58cyclictest2246444-21ssh09:20:053
223548499622,56cyclictest2286724-21ssh10:50:113
223548499622,56cyclictest2286724-21ssh10:50:103
223548499621,57cyclictest2235644-21sshd07:13:233
2235482996228,10cyclictest30-21ksoftirqd/210:39:012
2235482996222,12cyclictest30-21ksoftirqd/209:31:432
2235482996222,12cyclictest30-21ksoftirqd/209:31:432
223548499612,55cyclictest2239776-21cat08:30:033
223548499612,55cyclictest2239776-21cat08:30:033
223548499612,53cyclictest2238282-21munin-run08:05:013
223548499612,53cyclictest2238282-21munin-run08:05:013
2235484996121,36cyclictest2238226-21date08:00:023
223548499611,56cyclictest2289439-21ssh10:56:273
223548499611,56cyclictest2249087-21ssh09:26:103
2235482996126,4cyclictest2265543-21sh10:03:032
2235482996126,4cyclictest2265543-21sh10:03:032
223548499602,53cyclictest2240610-21munin-run08:50:003
2235484996025,30cyclictest2238849-21latency_hist08:15:023
223548499601,55cyclictest2237718-21sshd07:51:233
223548499601,54cyclictest2238509-21sshd08:08:233
2235471996029,14cyclictest2314285-21ssh11:51:570
223548499592,52cyclictest2239336-21munin-run08:25:013
2235482995924,9cyclictest30-21ksoftirqd/211:34:382
2235482995922,10cyclictest30-21ksoftirqd/211:44:472
2235482995922,10cyclictest30-21ksoftirqd/211:37:382
2235482995920,8cyclictest309-21in:imuxsock12:05:012
223548499581,52cyclictest48-21kcompactd009:02:433
223548499581,52cyclictest2236211-21munin-run07:25:003
2235482995817,14cyclictest2238439-21cat08:05:022
2235482995817,14cyclictest2238439-21cat08:05:022
2235471995824,31cyclictest12-21ksoftirqd/007:24:230
2235471995818,36cyclictest0-21swapper/009:34:070
2235471995818,36cyclictest0-21swapper/009:34:060
223548499571,51cyclictest364-21snmpd08:41:463
223548499571,51cyclictest364-21snmpd07:33:253
223548499571,51cyclictest364-21snmpd07:33:253
223548499571,51cyclictest2239086-21munin-run08:20:013
2235482995723,8cyclictest30-21ksoftirqd/208:15:012
2235471995737,5cyclictest16850irq/134-800000009:59:230
223547199573,14cyclictest2299883-21sh11:19:590
223548499561,51cyclictest2240836-21sshd08:53:243
223548499561,51cyclictest2237120-21latency_hist07:40:013
223548499561,50cyclictest48-21kcompactd007:47:503
223548499561,49cyclictest48-21kcompactd007:26:123
223548299564,20cyclictest2238776-21kworker/u8:209:30:022
2235482995622,10cyclictest30-21ksoftirqd/209:47:372
2235471995628,13cyclictest2273580-21ssh10:20:460
2235471995628,13cyclictest2273580-21ssh10:20:450
223548499551,50cyclictest48-21kcompactd007:41:443
2235482995529,11cyclictest30-21ksoftirqd/210:22:162
2235482995529,11cyclictest30-21ksoftirqd/210:22:162
2235482995528,5cyclictest30-21ksoftirqd/209:56:582
2235471995526,12cyclictest2291196-21ssh11:00:080
2235471995426,13cyclictest2259004-21ssh09:48:280
2235471995426,12cyclictest2318451-21ssh12:01:100
2235471995426,12cyclictest2300906-21ssh11:22:020
2235471995424,14cyclictest2328767-21ssh12:24:040
2235482995345,5cyclictest2240119-21kworker/2:010:52:122
2235482995345,5cyclictest2240119-21kworker/2:010:52:122
2235482995345,5cyclictest2229616-21kworker/2:211:05:002
2235478995337,11cyclictest3467602-21systemd-journal11:15:231
2235478995332,16cyclictest3467602-21systemd-journal09:00:011
2235478995332,16cyclictest3467602-21systemd-journal09:00:011
2235471995321,23cyclictest19350irq/34-mmc108:30:020
2235471995321,23cyclictest19350irq/34-mmc108:30:020
223547199531,37cyclictest2257159-21ssh09:44:290
223547199531,36cyclictest2271209-21ssh10:15:270
2235482995234,13cyclictest275-21dbus-daemon10:50:002
2235478995214,7cyclictest131rcu_preempt12:05:021
2235471995244,5cyclictest16850irq/134-800000011:50:000
2235471995224,13cyclictest2260843-21ssh09:52:250
2235471995216,32cyclictest12-21ksoftirqd/009:35:590
223548499512,44cyclictest364-21snmpd09:06:313
2235482995042,5cyclictest2229616-21kworker/2:208:08:232
2235482995041,6cyclictest2240119-21kworker/2:010:05:522
2235478995036,10cyclictest23-21ksoftirqd/109:05:001
223547199507,26cyclictest151rcuc/008:15:020
2235471995043,4cyclictest16850irq/134-800000010:40:020
2235471995035,9cyclictest275-21dbus-daemon12:00:000
223547199503,17cyclictest2246192-21sendmail-msp09:20:000
2235471995026,10cyclictest2320531-21ssh12:05:470
2235471995020,19cyclictest0-21swapper/007:10:010
2235471995016,8cyclictest12-21ksoftirqd/010:57:370
223547199501,24cyclictest2309028-21latency_hist11:40:030
2235482994942,5cyclictest2318093-21kworker/2:112:15:282
2235482994942,5cyclictest2318093-21kworker/2:112:15:282
2235482994941,5cyclictest2240119-21kworker/2:011:19:072
2235482994941,5cyclictest2240119-21kworker/2:010:28:292
2235482994941,5cyclictest2229616-21kworker/2:207:40:022
2235482994940,6cyclictest2240119-21kworker/2:011:10:282
2235482994940,5cyclictest2229616-21kworker/2:211:46:052
2235482994917,8cyclictest2240119-21kworker/2:010:15:162
2235471994944,3cyclictest0-21swapper/008:20:010
2235471994933,10cyclictest3467602-21systemd-journal11:10:000
2235471994922,12cyclictest2325450-21ssh12:16:450
2235471994922,12cyclictest2325450-21ssh12:16:440
2235482994840,5cyclictest2318093-21kworker/2:112:39:372
2235482994840,5cyclictest2318093-21kworker/2:112:20:332
223547899485,3cyclictest131rcu_preempt09:52:271
2235478994831,13cyclictest23-21ksoftirqd/111:10:011
2235478994828,14cyclictest3467602-21systemd-journal10:15:011
2235478994828,14cyclictest3467602-21systemd-journal10:15:011
2235471994840,5cyclictest16950irq/224-800000009:10:000
2235471994828,16cyclictest2324492-21ssh12:14:510
2235471994820,12cyclictest2250031-21ssh09:28:230
2235471994817,15cyclictest2286462-21ssh10:50:010
2235482994736,6cyclictest2318093-21kworker/2:112:34:232
2235482994731,12cyclictest1-21systemd09:15:012
2235471994738,5cyclictest16950irq/224-800000012:40:000
2235471994729,12cyclictest309-21in:imuxsock10:30:000
2235471994728,15cyclictest2241645-21diskmemload10:10:020
2235471994728,13cyclictest275-21dbus-daemon07:45:010
2235471994710,32cyclictest12-21ksoftirqd/008:37:460
2235482994638,5cyclictest2240119-21kworker/2:011:21:102
2235482994638,5cyclictest2240119-21kworker/2:009:42:332
2235482994638,5cyclictest2229616-21kworker/2:212:09:272
223548299462,30cyclictest2229310-21kworker/u8:207:15:022
223548299461,41cyclictest2289409-21sshd10:56:232
223547899465,7cyclictest23-21ksoftirqd/109:35:331
2235478994629,13cyclictest23-21ksoftirqd/109:56:481
223547899462,41cyclictest2336165-21ssh12:40:021
2235478994618,16cyclictest2235466-21cyclictest10:35:021
2235471994638,5cyclictest16850irq/134-800000012:35:020
2235471994628,14cyclictest2295658-21ssh11:10:050
2235471994625,7cyclictest2234991-21kworker/u8:007:25:590
2235471994625,18cyclictest12-21ksoftirqd/011:44:460
223548299458,8cyclictest30-21ksoftirqd/207:51:242
2235482994537,5cyclictest2240119-21kworker/2:009:10:022
2235482994537,5cyclictest2229616-21kworker/2:211:59:462
2235482994535,6cyclictest2229616-21kworker/2:209:50:542
2235482994513,5cyclictest131rcu_preempt07:50:012
2235478994529,12cyclictest3467602-21systemd-journal11:55:001
2235478994528,12cyclictest3467602-21systemd-journal11:25:001
223547899451,3cyclictest131rcu_preempt09:29:461
2235478994513,6cyclictest131rcu_preempt07:47:231
2235471994527,13cyclictest275-21dbus-daemon07:50:010
223547199452,39cyclictest2248133-21ssh09:24:160
2235471994515,27cyclictest12-21ksoftirqd/010:30:590
2235471994514,27cyclictest12-21ksoftirqd/007:15:010
2235482994436,5cyclictest2229616-21kworker/2:211:54:122
223547899443,3cyclictest131rcu_preempt10:55:031
2235478994432,10cyclictest23-21ksoftirqd/110:41:591
2235478994427,13cyclictest2257466-21cat09:45:021
2235471994435,6cyclictest12-21ksoftirqd/010:51:340
2235471994435,6cyclictest12-21ksoftirqd/010:51:330
2235471994428,13cyclictest275-21dbus-daemon09:00:010
2235471994428,13cyclictest275-21dbus-daemon09:00:010
2235471994427,13cyclictest2330351-21ssh12:27:290
223547199442,39cyclictest2304874-21ssh11:30:500
2235482994334,6cyclictest2240119-21kworker/2:011:07:292
2235482994334,6cyclictest2229616-21kworker/2:210:11:182
2235482994334,6cyclictest2229616-21kworker/2:210:11:172
2235482994327,12cyclictest275-21dbus-daemon09:00:012
2235482994327,12cyclictest275-21dbus-daemon09:00:012
2235478994326,11cyclictest2317714-21modprobe12:00:001
223547899431,4cyclictest131rcu_preempt09:46:241
223547199433,36cyclictest2240358-21munin-run08:45:010
2235482994230,9cyclictest2239583-21systemd-run08:30:022
2235482994230,9cyclictest2239583-21systemd-run08:30:012
2235482994226,12cyclictest2238223-21sed08:00:022
2235478994229,9cyclictest2286510-21cat10:50:011
2235478994228,9cyclictest309-21in:imuxsock10:24:361
2235478994228,9cyclictest309-21in:imuxsock10:24:361
2235478994218,19cyclictest23-21ksoftirqd/109:25:001
2235471994235,4cyclictest16950irq/224-800000009:05:000
2235471994233,7cyclictest12-21ksoftirqd/007:52:140
223547199421,36cyclictest151rcuc/007:35:000
223547199421,36cyclictest151rcuc/007:35:000
2235482994122,14cyclictest1-21systemd07:30:012
2235482994119,16cyclictest275-21dbus-daemon08:25:012
2235478994132,7cyclictest23-21ksoftirqd/112:15:011
2235478994130,8cyclictest23-21ksoftirqd/111:00:251
2235478994128,10cyclictest3467602-21systemd-journal10:15:231
2235478994127,9cyclictest275-21dbus-daemon12:25:001
2235478994126,10cyclictest3467602-21systemd-journal08:15:001
2235471994133,5cyclictest16850irq/134-800000007:17:490
2235471994133,5cyclictest16850irq/134-800000007:17:490
2235471994131,6cyclictest2240846-21cron08:55:000
223548499402,35cyclictest2235523-21cat07:10:023
2235482994032,5cyclictest2318093-21kworker/2:112:14:052
2235482994032,5cyclictest2240119-21kworker/2:009:23:312
2235482994020,15cyclictest275-21dbus-daemon08:55:002
2235478994028,9cyclictest23-21ksoftirqd/110:53:181
2235478994028,9cyclictest23-21ksoftirqd/110:53:171
2235478994026,10cyclictest23-21ksoftirqd/112:16:111
2235478994026,10cyclictest23-21ksoftirqd/112:16:101
2235478994015,8cyclictest131rcu_preempt10:34:131
2235471994033,4cyclictest16950irq/224-800000007:40:000
2235471994026,11cyclictest309-21in:imuxsock08:00:020
2235471994026,11cyclictest309-21in:imuxsock08:00:020
2235471994026,10cyclictest12-21ksoftirqd/008:50:000
2235482993931,5cyclictest2229616-21kworker/2:209:20:012
2235482993931,5cyclictest2229616-21kworker/2:208:40:022
2235482993929,7cyclictest3467602-21systemd-journal08:33:232
2235478993926,8cyclictest275-21dbus-daemon07:40:001
2235478993925,10cyclictest23-21ksoftirqd/109:10:581
2235471993930,5cyclictest16950irq/224-800000008:00:000
2235482993821,12cyclictest309-21in:imuxsock07:20:012
2235482993821,12cyclictest309-21in:imuxsock07:20:012
2235482993819,15cyclictest275-21dbus-daemon08:20:002
2235478993826,10cyclictest23-21ksoftirqd/112:29:491
2235478993824,9cyclictest275-21dbus-daemon08:35:001
2235478993823,10cyclictest23-21ksoftirqd/110:30:011
2235478993818,11cyclictest23-21ksoftirqd/111:42:091
223547899381,3cyclictest131rcu_preempt11:36:411
2235471993829,4cyclictest12-21ksoftirqd/008:10:020
223547199381,4cyclictest2235239-21kworker/0:008:20:040
2235482993728,7cyclictest30-21ksoftirqd/209:05:022
2235482993724,9cyclictest1-21systemd07:45:012
223547899376,6cyclictest131rcu_preempt12:35:011
2235478993726,8cyclictest23-21ksoftirqd/108:45:021
2235478993725,9cyclictest23-21ksoftirqd/110:06:161
2235478993725,9cyclictest23-21ksoftirqd/109:31:301
2235478993725,9cyclictest23-21ksoftirqd/109:31:301
2235478993724,10cyclictest23-21ksoftirqd/107:55:021
223548299364,6cyclictest131rcu_preempt07:35:012
223548299364,6cyclictest131rcu_preempt07:35:012
2235478993628,5cyclictest2305198-21sh11:31:361
2235478993625,8cyclictest2240829-21sshd08:51:231
2235478993624,9cyclictest2239001-21sshd08:15:231
2235478993624,10cyclictest23-21ksoftirqd/110:03:471
2235478993624,10cyclictest23-21ksoftirqd/110:03:471
2235478993623,10cyclictest2240114-21systemd-run08:40:011
2235478993622,10cyclictest2235939-21systemctl07:20:001
2235478993622,10cyclictest2235939-21systemctl07:20:001
2235478993621,12cyclictest23-21ksoftirqd/108:25:001
223547899361,3cyclictest131rcu_preempt11:27:361
2235478993612,22cyclictest23-21ksoftirqd/111:15:021
2235482993526,6cyclictest30-21ksoftirqd/208:45:022
2235482993521,9cyclictest30-21ksoftirqd/208:50:002
2235478993528,4cyclictest2244947-21sh09:17:021
2235478993528,4cyclictest2238535-21/usr/sbin/munin08:10:001
2235478993527,5cyclictest2241648-21munin-plugin-st09:10:001
2235478993522,11cyclictest23-21ksoftirqd/111:46:521
2235478993424,5cyclictest23-21ksoftirqd/108:30:011
2235478993424,5cyclictest23-21ksoftirqd/108:30:011
2235478993418,12cyclictest3467602-21systemd-journal08:50:001
2235478993418,11cyclictest3467602-21systemd-journal08:00:011
2235478993416,14cyclictest275-21dbus-daemon07:35:011
2235478993416,14cyclictest275-21dbus-daemon07:35:011
223547199343,4cyclictest0-21swapper/008:31:070
223548299333,5cyclictest131rcu_preempt07:10:022
2235478993322,8cyclictest23-21ksoftirqd/108:01:291
2235478993322,8cyclictest23-21ksoftirqd/108:01:291
2235478993322,8cyclictest23-21ksoftirqd/107:26:351
2235478993317,13cyclictest3467602-21systemd-journal07:25:001
2235482993220,9cyclictest30-21ksoftirqd/207:25:002
2235478993223,6cyclictest2235656-21cron07:15:001
2235478993021,6cyclictest23-21ksoftirqd/107:10:021
2235478993018,9cyclictest23-21ksoftirqd/107:45:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional