You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 20:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Sep 14, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
330805399956,11cyclictest12-21ksoftirqd/011:30:010
3308065997730,10cyclictest3375704-21run-parts11:20:002
330805399775,5cyclictest121ksoftirqd/011:33:320
330806599734,7cyclictest131rcu_preempt09:40:012
330806599734,7cyclictest131rcu_preempt09:40:002
330806599732,59cyclictest3467602-21systemd-journal12:25:022
330806599732,59cyclictest3467602-21systemd-journal12:25:012
330806999699,55cyclictest3404052-21ssh12:20:023
330806199682,13cyclictest3390392-21ssh11:50:511
330806199681,14cyclictest3309306-21sshd07:31:301
3308053996828,35cyclictest12-21ksoftirqd/009:25:010
330805399682,43cyclictest12-21ksoftirqd/012:38:440
330805399682,43cyclictest12-21ksoftirqd/012:38:440
3308061996729,34cyclictest3348685-21sh10:22:251
330806199672,19cyclictest3377571-21ssh11:23:511
3308053996730,17cyclictest3403954-21ssh12:20:010
3308065996633,12cyclictest1-21systemd08:50:012
3308061996626,35cyclictest3359078-21sh10:44:451
3308061996616,44cyclictest275-21dbus-daemon12:30:001
330806199661,11cyclictest3383606-21ssh11:36:291
3308061996535,18cyclictest23-21ksoftirqd/109:25:021
330806199652,7cyclictest3313242-21munin-run08:50:001
330806199652,24cyclictest3337004-21ssh09:57:541
330806199652,16cyclictest3396891-21ssh12:04:481
330806199651,13cyclictest3363511-21ssh10:54:081
330806199651,10cyclictest3350943-21ssh10:27:221
330806199651,10cyclictest3350943-21ssh10:27:221
3308061996425,35cyclictest3328933-21sh09:40:341
330806199641,9cyclictest3328397-21ssh09:39:491
330806199641,9cyclictest3328397-21ssh09:39:481
330806199641,9cyclictest3321869-21ssh09:25:361
330806199641,9cyclictest3321869-21ssh09:25:361
330806199641,11cyclictest3467602-21systemd-journal09:51:421
3308053996418,8cyclictest12-21ksoftirqd/010:34:010
3308065996332,10cyclictest30-21ksoftirqd/212:40:022
3308065996332,10cyclictest30-21ksoftirqd/212:40:012
3308065996330,10cyclictest30-21ksoftirqd/209:48:172
3308065996322,11cyclictest275-21dbus-daemon07:50:002
330806199631,9cyclictest3373750-21ssh11:15:251
330805399634,7cyclictest121ksoftirqd/010:04:190
3308053996332,7cyclictest12-21ksoftirqd/012:09:120
3308053996326,12cyclictest12-21ksoftirqd/012:04:430
330806199622,18cyclictest3317882-21ssh09:17:271
330806199622,14cyclictest3355772-21ssh10:37:331
330806199621,9cyclictest3405378-21ssh12:22:541
330806199621,9cyclictest3405378-21ssh12:22:541
330806199621,9cyclictest3372223-21ssh11:12:151
330805399629,4cyclictest131rcu_preempt09:42:300
3308065996124,11cyclictest30-21ksoftirqd/207:55:032
330806199612,14cyclictest3344233-21ssh10:13:191
330806199612,14cyclictest3312461-21munin-run08:35:011
330806199612,12cyclictest3379802-21ssh11:28:341
3308061996120,37cyclictest3410513-21sh12:34:011
330806199611,7cyclictest3338921-21ssh10:01:431
330806199611,15cyclictest48-21kcompactd012:10:121
330806199611,15cyclictest3360431-21ssh10:47:141
3308053996123,34cyclictest12-21ksoftirqd/012:00:010
3308053996122,35cyclictest12-21ksoftirqd/009:30:240
330806199606,3cyclictest0-21swapper/109:03:021
330806199601,9cyclictest3381051-21ssh11:31:011
330806199601,8cyclictest3370112-21ssh11:07:581
330806199601,6cyclictest364-21snmpd11:03:301
330806199601,12cyclictest3315546-21ssh09:12:311
330806199601,11cyclictest3347252-21ssh10:19:501
330806199601,11cyclictest3347252-21ssh10:19:491
330805399607,5cyclictest131rcu_preempt10:57:190
330805399607,5cyclictest131rcu_preempt10:57:190
330805399602,55cyclictest0-21swapper/008:00:020
3308065995923,11cyclictest3467602-21systemd-journal10:09:592
330806199592,7cyclictest3312196-21munin-run08:30:011
330806199592,7cyclictest3312196-21munin-run08:30:001
330806199592,6cyclictest364-21snmpd08:02:501
3308061995922,32cyclictest23-21ksoftirqd/109:46:051
330806199592,17cyclictest3308560-21munin-run07:20:001
3308061995921,34cyclictest23-21ksoftirqd/110:33:491
330806199592,12cyclictest3413306-21sendmail12:40:001
330806199592,12cyclictest3413306-21sendmail12:39:591
330806199591,21cyclictest3388669-21ssh11:47:171
330806199591,12cyclictest364-21snmpd09:08:041
330806199591,12cyclictest3311706-21munin-run08:20:021
3308053995932,23cyclictest16850irq/134-800000009:51:360
330806199582,6cyclictest3310381-21munin-run07:55:011
330805399585,15cyclictest12-21ksoftirqd/011:50:030
3308053995831,12cyclictest3336846-21ssh09:57:220
3308053995827,14cyclictest3371416-21ssh11:10:260
3308053995827,14cyclictest3360176-21ssh10:46:420
3308053995826,7cyclictest12-21ksoftirqd/009:10:170
3308053995821,31cyclictest3391007-21sh11:52:170
330806199572,7cyclictest3309879-21munin-run07:45:001
3308061995726,27cyclictest23-21ksoftirqd/112:08:481
3308061995716,36cyclictest3386045-21sh11:41:331
330806199571,13cyclictest3325007-21ssh09:32:211
3308053995727,14cyclictest3407306-21ssh12:26:530
3308053995719,29cyclictest16850irq/134-800000012:11:340
3308053995717,7cyclictest12-21ksoftirqd/010:19:520
3308053995717,7cyclictest12-21ksoftirqd/010:19:510
3308069995637,14cyclictest1-21systemd10:00:003
3308061995632,19cyclictest3403966-21sh12:20:011
330806199562,8cyclictest3311417-21munin-run08:15:011
330806199561,9cyclictest3340710-21sshd10:05:321
330805399565,6cyclictest121ksoftirqd/010:40:530
330805399565,10cyclictest12-21ksoftirqd/012:31:490
330805399564,9cyclictest0-21swapper/009:47:440
3308053995639,13cyclictest3467602-21systemd-journal10:40:000
3308053995625,28cyclictest12-21ksoftirqd/011:39:120
330806199553,4cyclictest131rcu_preempt07:59:591
330806199552,8cyclictest3392794-21ssh11:56:051
330806199552,6cyclictest3312740-21munin-run08:40:011
330806199551,16cyclictest3364526-21rm10:55:581
330806199551,16cyclictest3364526-21rm10:55:581
330806199551,12cyclictest3312072-21latency_hist08:25:011
330805399556,5cyclictest16850irq/134-800000011:10:010
3308053995525,14cyclictest3406318-21ssh12:25:010
3308053995525,14cyclictest3406318-21ssh12:25:010
3308053995524,15cyclictest3363953-21ssh10:55:010
3308053995522,4cyclictest3342521-21sh10:09:390
3308053995518,18cyclictest16850irq/134-800000011:40:370
330806199542,7cyclictest3308831-21munin-run07:25:011
330805399547,7cyclictest12-21ksoftirqd/010:11:120
330805399546,11cyclictest12-21ksoftirqd/011:16:410
330805399545,4cyclictest131rcu_preempt09:16:080
3308053995427,12cyclictest3376593-21ssh11:21:310
3308053995418,5cyclictest16850irq/134-800000009:39:350
3308053995418,5cyclictest16850irq/134-800000009:39:350
3308065995344,6cyclictest3349784-21kworker/2:210:57:172
3308065995344,6cyclictest3349784-21kworker/2:210:57:172
3308065995341,7cyclictest3345004-21kworker/2:011:31:532
330806199535,18cyclictest3312978-21munin-run08:45:011
330806199534,7cyclictest3313541-21latency_hist08:55:021
330806199531,15cyclictest364-21snmpd07:47:471
330806199531,13cyclictest364-21snmpd07:10:201
330805399536,8cyclictest12-21ksoftirqd/008:35:020
3308065995244,5cyclictest3349784-21kworker/2:211:22:262
3308065995244,5cyclictest3311179-21kworker/2:309:16:222
3308065995243,6cyclictest3345004-21kworker/2:012:31:342
3308065995243,6cyclictest3345004-21kworker/2:011:07:492
3308065995242,6cyclictest3068121-21kworker/2:009:12:422
3308065995234,13cyclictest275-21dbus-daemon10:55:002
3308053995224,13cyclictest3323453-21ssh09:29:130
3308053995224,13cyclictest3323453-21ssh09:29:130
3308069995138,10cyclictest37-21ksoftirqd/311:30:033
3308065995143,5cyclictest3323806-21kworker/2:109:40:412
330806199513,5cyclictest364-21snmpd08:55:201
330806199513,10cyclictest3308064-21latency_hist07:10:011
330806199511,14cyclictest364-21snmpd07:36:331
330806199511,14cyclictest364-21snmpd07:36:331
330806199511,10cyclictest1-21systemd08:10:001
3308069995030,14cyclictest275-21dbus-daemon08:00:013
3308065995042,5cyclictest3349784-21kworker/2:211:27:092
3308065995041,6cyclictest3323806-21kworker/2:110:12:532
330806199502,13cyclictest3309047-21modprobe07:25:131
330805399506,8cyclictest12-21ksoftirqd/007:35:020
3308053995012,9cyclictest12-21ksoftirqd/010:25:090
3308053995012,9cyclictest12-21ksoftirqd/010:25:080
3308069994935,10cyclictest275-21dbus-daemon10:50:013
3308069994915,21cyclictest37-21ksoftirqd/311:10:023
3308065994941,5cyclictest3345004-21kworker/2:012:00:412
3308065994940,6cyclictest3345004-21kworker/2:010:48:272
3308065994935,11cyclictest275-21dbus-daemon11:15:012
330805399497,11cyclictest12-21ksoftirqd/011:01:150
330805399495,12cyclictest12-21ksoftirqd/008:45:010
3308053994937,9cyclictest12-21ksoftirqd/007:25:030
330805399491,7cyclictest0-21swapper/008:25:010
3308069994834,10cyclictest37-21ksoftirqd/307:25:023
3308069994833,11cyclictest275-21dbus-daemon09:35:013
3308065994840,5cyclictest3349784-21kworker/2:211:02:352
3308065994840,5cyclictest3345004-21kworker/2:011:50:582
3308065994840,5cyclictest3345004-21kworker/2:011:42:232
3308065994840,5cyclictest3340381-21kworker/2:310:16:322
3308065994840,5cyclictest3340381-21kworker/2:310:16:312
3308065994839,6cyclictest3345004-21kworker/2:010:35:492
3308065994839,6cyclictest3345004-21kworker/2:010:30:362
3308065994832,10cyclictest3467602-21systemd-journal11:40:002
330805399484,13cyclictest12-21ksoftirqd/008:10:010
3308053994837,7cyclictest12-21ksoftirqd/009:05:010
330805399481,42cyclictest3347678-21ssh10:20:120
3308069994726,15cyclictest275-21dbus-daemon08:10:003
3308069994713,5cyclictest131rcu_preempt09:25:013
3308065994739,5cyclictest3349784-21kworker/2:210:44:002
3308065994739,5cyclictest3345004-21kworker/2:010:20:072
3308065994739,5cyclictest3311179-21kworker/2:309:21:222
3308065994738,6cyclictest3068121-21kworker/2:009:33:262
3308053994737,7cyclictest3309864-21munin-plugin-st07:45:000
330806999461,10cyclictest37-21ksoftirqd/311:03:493
3308065994638,5cyclictest3345004-21kworker/2:012:05:532
3308065994638,5cyclictest3345004-21kworker/2:011:56:482
3308065994637,6cyclictest3068121-21kworker/2:007:40:032
3308065994637,6cyclictest3068121-21kworker/2:007:40:022
3308065994623,16cyclictest275-21dbus-daemon07:25:002
3308053994627,14cyclictest275-21dbus-daemon07:40:010
3308053994627,14cyclictest275-21dbus-daemon07:40:000
3308053994614,28cyclictest12-21ksoftirqd/008:48:000
3308069994535,5cyclictest3363927-21latency_hist10:55:023
3308065994537,5cyclictest3311179-21kworker/2:309:28:362
3308065994537,5cyclictest3311179-21kworker/2:309:28:362
3308065994529,12cyclictest3311666-21cron08:20:002
330805399456,4cyclictest131rcu_preempt08:30:010
330805399456,4cyclictest131rcu_preempt08:30:000
3308053994537,4cyclictest0-21swapper/007:20:000
3308053994534,7cyclictest12-21ksoftirqd/009:10:000
3308069994436,6cyclictest37-21ksoftirqd/308:46:453
3308069994431,10cyclictest37-21ksoftirqd/310:09:103
3308069994426,14cyclictest3309595-21systemctl07:40:013
3308069994426,14cyclictest3309595-21systemctl07:40:003
3308069994426,14cyclictest275-21dbus-daemon09:55:023
3308069994410,10cyclictest37-21ksoftirqd/312:10:003
3308065994434,6cyclictest3068121-21kworker/2:008:10:012
330806599442,4cyclictest131rcu_preempt12:27:542
330805399445,4cyclictest131rcu_preempt08:35:290
330805399444,9cyclictest12-21ksoftirqd/008:15:010
330805399443,39cyclictest12-21ksoftirqd/007:10:020
330805399443,11cyclictest12-21ksoftirqd/008:20:010
3308053994426,14cyclictest3313711-21sshd08:58:310
3308069994332,8cyclictest37-21ksoftirqd/309:18:123
3308069994322,16cyclictest275-21dbus-daemon08:30:013
3308069994322,16cyclictest275-21dbus-daemon08:30:013
3308069994312,7cyclictest37-21ksoftirqd/311:00:003
3308069994312,7cyclictest37-21ksoftirqd/310:59:593
3308065994335,5cyclictest3068121-21kworker/2:009:51:192
3308065994326,12cyclictest3467602-21systemd-journal07:30:032
3308065994311,6cyclictest131rcu_preempt09:56:462
3308069994228,11cyclictest275-21dbus-daemon09:00:013
3308069994227,12cyclictest37-21ksoftirqd/310:40:003
3308069994227,11cyclictest37-21ksoftirqd/310:24:323
330806999422,4cyclictest131rcu_preempt10:30:273
3308069994216,6cyclictest131rcu_preempt10:28:293
3308069994216,6cyclictest131rcu_preempt10:28:293
3308069994211,5cyclictest131rcu_preempt11:20:023
3308069994210,5cyclictest131rcu_preempt11:40:013
3308065994231,7cyclictest309-21in:imuxsock09:05:002
3308065994227,12cyclictest3345004-21kworker/2:011:45:172
3308065994227,11cyclictest275-21dbus-daemon08:15:002
3308065994222,13cyclictest309-21in:imuxsock07:45:012
330805399423,9cyclictest12-21ksoftirqd/008:55:010
330805399421,37cyclictest3308299-21systemd-run07:15:000
3308069994128,10cyclictest37-21ksoftirqd/312:25:443
3308069994125,11cyclictest37-21ksoftirqd/309:50:013
3308065994132,6cyclictest3335614-21kworker/2:210:01:112
3308065994127,11cyclictest275-21dbus-daemon08:25:012
330805399417,3cyclictest131rcu_preempt07:55:010
3308053994129,10cyclictest12-21ksoftirqd/007:30:010
3308053994126,11cyclictest12-21ksoftirqd/007:50:000
330806999408,6cyclictest131rcu_preempt12:39:143
330806999408,6cyclictest131rcu_preempt12:39:143
330806999406,5cyclictest131rcu_preempt11:53:583
3308069994029,8cyclictest3400148-21sshd12:11:303
3308069994028,9cyclictest37-21ksoftirqd/309:39:253
3308069994028,9cyclictest37-21ksoftirqd/309:39:243
3308069994026,8cyclictest309-21in:imuxsock07:15:313
3308069994026,10cyclictest3311677-21systemctl08:20:013
330806999401,8cyclictest37-21ksoftirqd/311:42:053
3308069994016,18cyclictest37-21ksoftirqd/312:20:213
3308069994016,18cyclictest37-21ksoftirqd/312:20:203
3308069994013,5cyclictest131rcu_preempt11:50:023
330806999401,15cyclictest131rcu_preempt11:14:093
3308065994031,6cyclictest3068121-21kworker/2:008:00:012
3308065994028,9cyclictest3467602-21systemd-journal08:25:032
3308065994028,9cyclictest3467602-21systemd-journal08:25:022
3308065994013,6cyclictest131rcu_preempt12:18:192
330806999398,5cyclictest131rcu_preempt12:05:013
330806999394,4cyclictest131rcu_preempt09:10:533
3308069993926,9cyclictest37-21ksoftirqd/310:14:333
3308069993915,5cyclictest131rcu_preempt11:20:183
3308069993912,6cyclictest131rcu_preempt12:34:443
3308069993912,18cyclictest37-21ksoftirqd/311:30:113
3308069993911,10cyclictest37-21ksoftirqd/308:35:013
3308065993929,6cyclictest3350202-21sh10:25:322
3308065993929,6cyclictest3350202-21sh10:25:312
330805399394,7cyclictest12-21ksoftirqd/008:05:000
3308069993829,6cyclictest37-21ksoftirqd/309:42:013
3308069993821,12cyclictest37-21ksoftirqd/310:40:223
3308069993818,5cyclictest361rcuc/311:56:193
330806599384,6cyclictest0-21swapper/212:13:042
3308065993826,10cyclictest30-21ksoftirqd/207:15:012
3308065993822,12cyclictest275-21dbus-daemon09:00:022
330806999378,6cyclictest131rcu_preempt10:03:143
3308069993726,9cyclictest37-21ksoftirqd/307:15:023
3308069993725,9cyclictest37-21ksoftirqd/308:05:023
3308069993724,9cyclictest37-21ksoftirqd/309:28:133
3308069993724,9cyclictest37-21ksoftirqd/309:28:123
3308065993728,6cyclictest3311179-21kworker/2:309:05:052
3308065993719,13cyclictest1-21systemd08:35:012
3308069993626,7cyclictest3311415-21systemctl08:15:013
3308069993626,6cyclictest309-21in:imuxsock08:45:023
3308069993622,10cyclictest309-21in:imuxsock09:10:013
330806999361,5cyclictest131rcu_preempt10:15:033
330806999361,5cyclictest131rcu_preempt10:15:033
330806599364,7cyclictest30-21ksoftirqd/207:15:122
3308065993527,5cyclictest3312974-21latency_hist08:45:022
3308065993522,9cyclictest3308185-21sed07:10:022
3308069993424,7cyclictest37-21ksoftirqd/307:50:023
3308069993422,10cyclictest37-21ksoftirqd/308:40:023
3308065993423,9cyclictest30-21ksoftirqd/207:30:022
3308069993323,7cyclictest3314170-21sed09:05:023
3308065993321,9cyclictest30-21ksoftirqd/208:38:342
3308065993315,13cyclictest3467602-21systemd-journal08:04:592
330806999326,22cyclictest3309083-21grep07:30:003
3308069993217,11cyclictest37-21ksoftirqd/307:50:003
330806999321,4cyclictest131rcu_preempt08:55:023
3308069993118,9cyclictest37-21ksoftirqd/307:35:013
3308069993022,5cyclictest3312120-21latency_hist08:25:023
3308069993016,11cyclictest275-21dbus-daemon07:45:013
3308065993020,7cyclictest30-21ksoftirqd/208:53:472
3308069992619,4cyclictest3308097-21latency_hist07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional