You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 20:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Apr 28, 2026 12:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3759001991167,4cyclictest131rcu_preempt09:45:010
3759001999328,13cyclictest3796747-21ssh10:22:460
375900199923,39cyclictest3808506-21munin-run10:50:010
375900199921,40cyclictest3850447-21ssh12:26:460
3759001998421,47cyclictest12-21ksoftirqd/009:15:180
3759001998351,30cyclictest0-21swapper/010:15:020
3759001998327,12cyclictest3820015-21ssh11:16:210
375900199831,34cyclictest3769916-21ssh09:20:190
3759001998028,12cyclictest3784669-21ssh09:54:520
3759001998025,12cyclictest3785563-21ssh09:56:370
3759001998025,12cyclictest3785563-21ssh09:56:370
3759001997825,14cyclictest3835124-21ssh11:51:160
3759001997825,13cyclictest3766722-21ssh09:13:150
3759001997729,12cyclictest3781936-21ssh09:48:270
3759001997724,10cyclictest3790922-21ssh10:09:170
3759001997724,10cyclictest3790922-21ssh10:09:160
3759001997628,11cyclictest3801151-21ssh10:32:550
3759001997619,14cyclictest3829226-21ssh11:37:520
375900199761,37cyclictest3837092-21ssh11:55:450
3759001997522,12cyclictest3853024-21ssh12:33:010
375900199731,37cyclictest3827531-21ssh11:34:050
375900199726,31cyclictest3759003-21cat07:10:020
375900199726,31cyclictest3759003-21cat07:10:020
3759001997121,11cyclictest3817528-21ssh11:10:320
3759001997120,14cyclictest3808751-21ssh10:50:070
3759001997029,3cyclictest16950irq/224-800000010:26:030
3759001997027,11cyclictest3841846-21ssh12:06:500
3759001997025,12cyclictest3839208-21ssh12:00:370
3759001996935,30cyclictest16850irq/134-800000011:05:550
375900199692,7cyclictest3812945-21latency_hist11:00:030
3759001996922,13cyclictest3834361-21ssh11:49:530
3759001996922,13cyclictest3834361-21ssh11:49:520
3759001996922,10cyclictest3846718-21ssh12:18:300
3759001996922,10cyclictest3846718-21ssh12:18:290
375900199683,45cyclictest131rcu_preempt09:10:020
3759001996824,9cyclictest16950irq/224-800000011:40:150
3759007996727,13cyclictest23-21ksoftirqd/112:35:011
3759001996629,4cyclictest16950irq/224-800000009:37:090
3759001996629,4cyclictest16950irq/224-800000009:37:080
3759001996434,27cyclictest12-21ksoftirqd/008:30:010
3759001996423,9cyclictest3824708-21rm11:27:130
3759001996418,17cyclictest16850irq/134-800000009:26:500
3759001996418,17cyclictest16850irq/134-800000009:26:500
3759011996325,34cyclictest1-21systemd11:50:012
3759011996325,34cyclictest1-21systemd11:50:012
375900199632,6cyclictest3805803-21sshd10:43:470
375900199632,13cyclictest3775048-21ssh09:32:340
3759001996216,17cyclictest16850irq/134-800000010:35:540
3759001996216,17cyclictest16850irq/134-800000010:35:530
375900799616,7cyclictest23-21ksoftirqd/107:35:011
3759011996049,6cyclictest3771750-21kworker/2:210:00:012
3759011996049,6cyclictest3771750-21kworker/2:210:00:002
375900799606,9cyclictest23-21ksoftirqd/110:20:001
375900799605,10cyclictest23-21ksoftirqd/107:20:001
375900799603,9cyclictest23-21ksoftirqd/109:20:001
375900799596,11cyclictest23-21ksoftirqd/111:29:561
3759007995911,11cyclictest23-21ksoftirqd/110:50:021
3759001995926,12cyclictest3849103-21ssh12:24:000
375900199592,32cyclictest379356950rm10:15:020
3759011995841,13cyclictest3467602-21systemd-journal10:55:002
375900799585,6cyclictest23-21ksoftirqd/111:45:011
3759001995825,9cyclictest12-21ksoftirqd/012:35:340
3759001995822,12cyclictest3814407-21rm11:03:390
3759001995816,6cyclictest16850irq/134-800000011:20:590
375900799578,14cyclictest23-21ksoftirqd/109:30:131
375900799577,7cyclictest23-21ksoftirqd/112:30:031
375900799575,14cyclictest23-21ksoftirqd/109:10:011
375900799574,7cyclictest23-21ksoftirqd/111:37:391
3759007995720,9cyclictest23-21ksoftirqd/109:25:181
3759007995720,9cyclictest23-21ksoftirqd/109:25:171
3759007995720,13cyclictest23-21ksoftirqd/109:40:011
3759007995720,13cyclictest23-21ksoftirqd/109:40:001
3759011995648,5cyclictest3789098-21kworker/2:010:16:032
375900799569,11cyclictest23-21ksoftirqd/109:45:471
375900799566,9cyclictest23-21ksoftirqd/111:00:021
375900799564,11cyclictest23-21ksoftirqd/112:12:561
375900799564,10cyclictest23-21ksoftirqd/111:05:421
3759007995614,11cyclictest23-21ksoftirqd/109:56:521
3759007995614,11cyclictest23-21ksoftirqd/109:56:511
3759007995614,10cyclictest23-21ksoftirqd/110:26:341
3759011995546,5cyclictest3789098-21kworker/2:011:05:242
375900799558,12cyclictest23-21ksoftirqd/108:50:021
3759007995536,14cyclictest1-21systemd12:00:001
375900799551,10cyclictest23-21ksoftirqd/110:10:011
375900799551,10cyclictest23-21ksoftirqd/110:10:011
375900199552,15cyclictest3845200-21ssh12:15:010
3759011995446,5cyclictest3802311-21kworker/2:111:24:582
3759011995446,5cyclictest3762673-21kworker/2:309:40:022
3759011995446,5cyclictest3762673-21kworker/2:309:40:012
3759011995438,10cyclictest3467602-21systemd-journal10:40:002
3759011995438,10cyclictest3467602-21systemd-journal10:40:002
375900799549,11cyclictest23-21ksoftirqd/110:34:041
375900799545,9cyclictest23-21ksoftirqd/111:34:241
3759007995414,10cyclictest23-21ksoftirqd/110:20:201
375900799537,21cyclictest3846613-21kworker/u8:312:25:011
375900799536,12cyclictest23-21ksoftirqd/112:39:081
375900799536,11cyclictest23-21ksoftirqd/111:25:021
375900799535,7cyclictest23-21ksoftirqd/112:15:031
375900799535,7cyclictest23-21ksoftirqd/112:15:021
3759007995333,15cyclictest275-21dbus-daemon08:55:011
3759007995333,15cyclictest275-21dbus-daemon08:55:011
375900799532,11cyclictest23-21ksoftirqd/111:50:011
375900799532,11cyclictest23-21ksoftirqd/111:50:001
375901199524,3cyclictest131rcu_preempt11:40:002
3759011995242,6cyclictest3802311-21kworker/2:111:52:142
375900799526,7cyclictest23-21ksoftirqd/107:10:021
375900799526,7cyclictest23-21ksoftirqd/107:10:021
375900799525,9cyclictest23-21ksoftirqd/109:44:141
375900799524,9cyclictest23-21ksoftirqd/109:15:031
3759007995234,12cyclictest3467602-21systemd-journal09:00:001
375900799523,16cyclictest23-21ksoftirqd/110:53:501
375900799523,10cyclictest23-21ksoftirqd/111:51:541
375900799522,6cyclictest23-21ksoftirqd/109:05:021
375900799522,6cyclictest23-21ksoftirqd/109:05:011
3759015995134,13cyclictest275-21dbus-daemon09:10:003
3759015995133,12cyclictest3467602-21systemd-journal11:35:003
3759015995111,6cyclictest131rcu_preempt07:15:003
3759011995143,5cyclictest3771750-21kworker/2:209:44:542
3759011995143,5cyclictest3762673-21kworker/2:309:49:132
3759011995142,5cyclictest3789098-21kworker/2:010:29:482
3759011995137,10cyclictest3789067-21systemd-run10:05:002
375900799518,11cyclictest23-21ksoftirqd/110:40:111
375900799516,14cyclictest23-21ksoftirqd/111:19:191
375900799514,7cyclictest23-21ksoftirqd/110:15:011
3759007995131,15cyclictest275-21dbus-daemon08:10:011
375900799512,8cyclictest23-21ksoftirqd/111:13:151
3759007995115,9cyclictest0-21swapper/108:16:471
375900799511,11cyclictest23-21ksoftirqd/108:30:001
3759007995110,4cyclictest131rcu_preempt12:03:181
3759015995014,7cyclictest131rcu_preempt11:30:023
3759015995012,6cyclictest131rcu_preempt09:00:003
3759011995044,3cyclictest3802311-21kworker/2:112:13:112
3759011995042,5cyclictest3802311-21kworker/2:112:06:272
3759011995042,5cyclictest3802311-21kworker/2:110:59:412
3759011995041,6cyclictest3789098-21kworker/2:010:34:192
3759011995040,6cyclictest3802311-21kworker/2:112:33:212
3759011995034,12cyclictest1-21systemd08:10:012
375900799506,9cyclictest23-21ksoftirqd/112:05:561
375900799506,8cyclictest221rcuc/109:51:121
375900799503,10cyclictest23-21ksoftirqd/111:00:531
375900799502,10cyclictest23-21ksoftirqd/109:21:191
3759007995019,10cyclictest23-21ksoftirqd/110:00:331
3759001995035,10cyclictest12-21ksoftirqd/008:40:010
375901599497,5cyclictest131rcu_preempt10:15:023
3759015994929,14cyclictest1-21systemd10:05:013
3759011994942,4cyclictest3789098-21kworker/2:010:20:492
3759011994941,6cyclictest3802311-21kworker/2:111:34:482
3759011994941,5cyclictest3802311-21kworker/2:111:28:172
3759011994941,5cyclictest3789098-21kworker/2:010:42:132
3759011994941,5cyclictest3771750-21kworker/2:210:05:352
3759011994941,5cyclictest3771750-21kworker/2:210:05:342
3759011994941,5cyclictest3762673-21kworker/2:309:19:002
3759011994940,5cyclictest3762673-21kworker/2:309:23:322
3759011994935,11cyclictest3764381-21cron08:55:002
3759011994935,11cyclictest3764381-21cron08:55:002
3759011994931,14cyclictest275-21dbus-daemon08:50:022
375900799495,8cyclictest23-21ksoftirqd/110:35:221
375900799495,8cyclictest23-21ksoftirqd/110:35:211
375900799492,12cyclictest23-21ksoftirqd/108:15:031
375900799491,9cyclictest3760524-21sendmail-msp07:40:001
375900799491,16cyclictest23-21ksoftirqd/108:40:001
375901599485,33cyclictest37-21ksoftirqd/310:30:033
3759015994819,25cyclictest37-21ksoftirqd/309:50:013
3759011994843,3cyclictest3762673-21kworker/2:309:50:502
3759011994840,5cyclictest3802311-21kworker/2:112:29:452
3759011994840,5cyclictest3802311-21kworker/2:112:02:162
3759011994840,5cyclictest3789098-21kworker/2:010:46:462
3759011994840,5cyclictest3789098-21kworker/2:010:14:502
375901199483,4cyclictest131rcu_preempt11:55:592
3759011994829,14cyclictest275-21dbus-daemon09:30:022
3759011994829,14cyclictest275-21dbus-daemon09:30:012
375901199481,3cyclictest131rcu_preempt07:20:022
375900799483,9cyclictest23-21ksoftirqd/107:55:011
3759001994832,12cyclictest3467602-21systemd-journal07:38:470
3759001994828,14cyclictest275-21dbus-daemon09:00:010
3759015994735,8cyclictest37-21ksoftirqd/308:00:033
375901599471,8cyclictest0-21swapper/311:05:413
375901599471,8cyclictest0-21swapper/309:36:363
375901599471,8cyclictest0-21swapper/309:36:363
3759011994741,3cyclictest3802311-21kworker/2:111:15:142
3759011994739,5cyclictest3802311-21kworker/2:112:36:222
3759011994739,5cyclictest3802311-21kworker/2:112:16:412
3759011994739,5cyclictest3802311-21kworker/2:112:16:402
3759011994739,5cyclictest3802311-21kworker/2:111:43:502
3759011994738,6cyclictest3789098-21kworker/2:011:02:112
375901199473,3cyclictest131rcu_preempt12:21:442
375901599468,5cyclictest131rcu_preempt09:05:023
375901599468,5cyclictest131rcu_preempt09:05:013
375901599461,9cyclictest0-21swapper/310:40:383
375901599461,8cyclictest0-21swapper/311:42:563
3759015994610,6cyclictest131rcu_preempt12:25:483
3759011994638,5cyclictest3763119-21kworker/2:008:41:522
3759011994632,10cyclictest3467602-21systemd-journal08:55:022
3759011994630,13cyclictest3467602-21systemd-journal07:55:022
375901199462,4cyclictest131rcu_preempt11:11:262
375900799466,7cyclictest23-21ksoftirqd/108:25:021
375900799466,6cyclictest23-21ksoftirqd/107:15:011
375900799466,11cyclictest23-21ksoftirqd/107:25:011
375900799462,10cyclictest23-21ksoftirqd/108:00:021
375900799461,7cyclictest0-21swapper/108:02:471
3759001994631,12cyclictest275-21dbus-daemon08:15:010
375901599454,5cyclictest131rcu_preempt11:05:023
3759015994529,12cyclictest3467602-21systemd-journal12:22:483
3759015994529,10cyclictest1-21systemd10:10:013
3759015994529,10cyclictest1-21systemd10:10:013
3759015994524,14cyclictest275-21dbus-daemon12:40:013
375901599451,8cyclictest0-21swapper/310:36:383
375901599451,8cyclictest0-21swapper/310:36:373
375901599451,8cyclictest0-21swapper/310:12:063
375901599451,8cyclictest0-21swapper/309:28:053
375901599451,8cyclictest0-21swapper/309:28:053
375901599451,7cyclictest0-21swapper/312:33:273
375901599451,6cyclictest0-21swapper/310:57:143
375901599451,5cyclictest0-21swapper/307:46:413
3759015994515,5cyclictest131rcu_preempt11:48:483
3759015994515,5cyclictest131rcu_preempt11:48:473
3759011994537,5cyclictest3762673-21kworker/2:309:10:022
3759011994536,6cyclictest3759252-21kworker/2:207:50:012
3759011994526,14cyclictest275-21dbus-daemon07:35:002
375900799454,7cyclictest23-21ksoftirqd/108:45:011
375900799451,6cyclictest0-21swapper/107:45:031
375900799451,10cyclictest23-21ksoftirqd/107:42:461
375900199451,38cyclictest16850irq/134-800000010:02:390
3759015994431,9cyclictest37-21ksoftirqd/307:45:033
3759015994429,11cyclictest275-21dbus-daemon12:20:023
3759015994429,11cyclictest275-21dbus-daemon12:20:023
3759015994428,12cyclictest3467602-21systemd-journal12:00:473
375901599441,7cyclictest0-21swapper/311:55:393
375901599441,7cyclictest0-21swapper/311:16:423
375901599441,7cyclictest0-21swapper/310:33:543
375901599441,7cyclictest0-21swapper/309:10:113
375901599441,6cyclictest0-21swapper/309:52:083
3759011994437,4cyclictest3771750-21kworker/2:209:30:552
375901199443,3cyclictest131rcu_preempt07:15:012
3759011994429,11cyclictest30-21ksoftirqd/208:15:002
375900799447,10cyclictest23-21ksoftirqd/107:25:031
375900799446,9cyclictest23-21ksoftirqd/108:35:011
3759001994431,10cyclictest12-21ksoftirqd/007:30:020
375900199442,5cyclictest3759549-21wc07:20:010
375901599431,8cyclictest0-21swapper/310:24:183
375901599431,6cyclictest0-21swapper/311:36:213
375901599431,6cyclictest0-21swapper/309:56:123
375901599431,6cyclictest0-21swapper/309:56:123
375901599431,6cyclictest0-21swapper/308:15:023
375901599431,5cyclictest0-21swapper/312:06:503
375901599431,5cyclictest0-21swapper/311:50:433
375901599431,5cyclictest0-21swapper/311:13:433
375901599431,5cyclictest0-21swapper/309:18:493
3759011994335,5cyclictest3751863-21kworker/2:107:25:002
375901199431,3cyclictest131rcu_preempt07:30:012
3759001994324,14cyclictest275-21dbus-daemon08:50:010
375901599421,8cyclictest0-21swapper/310:45:463
375901599421,7cyclictest0-21swapper/312:12:533
375901599421,7cyclictest0-21swapper/309:42:173
375901599421,7cyclictest0-21swapper/309:31:233
375901599421,6cyclictest0-21swapper/311:24:443
3759011994235,5cyclictest3762673-21kworker/2:309:14:512
3759011994229,9cyclictest275-21dbus-daemon07:55:012
3759001994235,4cyclictest16950irq/224-800000007:59:470
375901599419,6cyclictest131rcu_preempt08:35:023
375901599411,9cyclictest0-21swapper/309:22:053
375901599411,5cyclictest0-21swapper/310:52:283
3759015994112,5cyclictest131rcu_preempt08:20:003
3759011994133,5cyclictest3763092-21cron08:30:002
375900199413,5cyclictest3762625-21grep08:20:010
375901599401,5cyclictest131rcu_preempt08:10:013
375901199404,3cyclictest131rcu_preempt08:25:022
375900199402,8cyclictest3760235-21modprobe07:35:000
3759001994025,10cyclictest1-21systemd07:25:010
375900199402,10cyclictest3761798-21modprobe08:05:000
3759011993931,5cyclictest3751863-21kworker/2:108:01:342
3759011993926,9cyclictest3467602-21systemd-journal09:02:012
3759011993926,9cyclictest3467602-21systemd-journal09:02:012
375900199391,11cyclictest3467602-21systemd-journal07:55:010
375901599386,7cyclictest37-21ksoftirqd/307:30:003
3759015993825,9cyclictest37-21ksoftirqd/308:50:013
3759015993825,8cyclictest3759455-21modprobe07:20:003
3759015993815,6cyclictest131rcu_preempt07:35:003
3759011993810,5cyclictest131rcu_preempt08:16:472
375900199384,20cyclictest0-21swapper/007:45:010
375900199383,5cyclictest3763361-21cut08:35:000
3759001993832,4cyclictest16950irq/224-800000008:55:010
3759001993832,4cyclictest16950irq/224-800000008:55:010
3759001993830,6cyclictest19350irq/34-mmc107:10:080
375900199381,4cyclictest150-21pr/ttyS207:50:010
375901599375,5cyclictest131rcu_preempt08:45:003
3759015993725,7cyclictest309-21in:imuxsock08:30:013
3759011993726,7cyclictest30-21ksoftirqd/208:40:012
3759001993728,5cyclictest3762060-21munin-plugin-st08:10:010
375901599368,6cyclictest131rcu_preempt08:55:023
375901599368,6cyclictest131rcu_preempt08:55:013
3759011993620,12cyclictest30-21ksoftirqd/207:45:002
3759001993626,6cyclictest3763991-21wc08:45:010
375900199361,31cyclictest12-21ksoftirqd/008:25:020
3759011993525,6cyclictest309-21in:imuxsock07:40:012
375901199351,3cyclictest131rcu_preempt08:35:002
375901599348,5cyclictest131rcu_preempt08:40:023
375901599348,5cyclictest131rcu_preempt07:25:023
3759015993416,14cyclictest275-21dbus-daemon08:25:013
3759001993426,5cyclictest19350irq/34-mmc109:00:070
3759001993426,5cyclictest19350irq/34-mmc109:00:070
375901599333,8cyclictest37-21ksoftirqd/307:10:033
375901599333,8cyclictest37-21ksoftirqd/307:10:023
375901599332,7cyclictest37-21ksoftirqd/308:05:003
375901599332,4cyclictest131rcu_preempt07:50:393
375901599321,5cyclictest0-21swapper/307:36:273
375901199322,3cyclictest131rcu_preempt07:10:022
375901199322,3cyclictest131rcu_preempt07:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional