You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 05:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Mar 19, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3967566999692,2cyclictest0-21swapper/023:45:020
3967566998177,2cyclictest0-21swapper/019:15:020
396756699728,36cyclictest131rcu_preempt23:28:380
3967576997140,7cyclictest3978236-21kworker/2:100:20:002
3967576997140,7cyclictest3978236-21kworker/2:100:20:002
396756699682,8cyclictest121ksoftirqd/023:02:410
396756699675,8cyclictest121ksoftirqd/023:46:290
396758099667,11cyclictest37-21ksoftirqd/321:19:593
3967566996615,8cyclictest0-21swapper/021:50:020
3967576996513,4cyclictest131rcu_preempt19:15:022
396756699657,7cyclictest12-21ksoftirqd/020:40:020
396756699651,7cyclictest121ksoftirqd/021:05:010
396756699651,7cyclictest121ksoftirqd/021:05:010
396756699649,4cyclictest131rcu_preempt23:07:130
3967566996430,30cyclictest12-21ksoftirqd/022:55:580
396756699642,31cyclictest12-21ksoftirqd/022:25:130
3967580996213,13cyclictest37-21ksoftirqd/322:55:003
3967580996210,6cyclictest131rcu_preempt00:03:463
396756699626,7cyclictest121ksoftirqd/000:20:020
396756699626,7cyclictest121ksoftirqd/000:20:010
396756699625,7cyclictest12-21ksoftirqd/021:25:020
396756699621,31cyclictest12-21ksoftirqd/022:05:030
396756699619,14cyclictest12-21ksoftirqd/000:36:080
396756699617,6cyclictest121ksoftirqd/022:55:010
396758099602,10cyclictest37-21ksoftirqd/321:40:013
3967566996024,7cyclictest12-21ksoftirqd/019:25:020
3967566996023,33cyclictest4053722-21latency_hist00:10:030
3967566996023,33cyclictest4053722-21latency_hist00:10:030
3967566996015,14cyclictest12-21ksoftirqd/020:15:020
3967566996011,9cyclictest12-21ksoftirqd/000:11:260
3967580995918,10cyclictest37-21ksoftirqd/323:35:013
3967580995911,5cyclictest131rcu_preempt23:15:023
396756699596,11cyclictest12-21ksoftirqd/023:35:430
396756699594,10cyclictest12-21ksoftirqd/022:50:010
3967566995926,12cyclictest12-21ksoftirqd/021:31:180
3967566995911,13cyclictest12-21ksoftirqd/020:30:000
396758099587,7cyclictest37-21ksoftirqd/321:15:023
396758099587,7cyclictest37-21ksoftirqd/321:15:013
3967580995812,5cyclictest131rcu_preempt21:45:023
3967580995810,6cyclictest131rcu_preempt19:25:023
3967576995849,6cyclictest3978236-21kworker/2:100:10:592
396756699588,33cyclictest131rcu_preempt00:34:210
3967566995820,16cyclictest16850irq/134-800000021:39:170
396756699581,8cyclictest121ksoftirqd/022:37:470
396758099579,6cyclictest131rcu_preempt00:17:533
396758099579,6cyclictest131rcu_preempt00:17:523
396758099575,9cyclictest131rcu_preempt22:25:003
3967566995748,6cyclictest12-21ksoftirqd/020:20:000
396756699574,6cyclictest12-21ksoftirqd/020:00:000
396758099568,13cyclictest37-21ksoftirqd/321:05:013
396758099568,13cyclictest37-21ksoftirqd/321:05:003
396758099566,18cyclictest37-21ksoftirqd/320:35:003
396758099566,10cyclictest37-21ksoftirqd/321:25:033
396758099565,8cyclictest37-21ksoftirqd/323:05:013
396758099564,10cyclictest37-21ksoftirqd/323:00:003
3967580995625,9cyclictest37-21ksoftirqd/322:04:403
396758099562,10cyclictest37-21ksoftirqd/300:15:023
3967580995613,6cyclictest131rcu_preempt20:30:023
3967580995612,5cyclictest131rcu_preempt00:35:013
3967580995612,13cyclictest37-21ksoftirqd/300:25:003
3967580995611,6cyclictest131rcu_preempt21:33:493
3967580995611,6cyclictest131rcu_preempt00:35:303
3967580995610,6cyclictest131rcu_preempt22:14:313
396756699567,9cyclictest12-21ksoftirqd/021:17:350
3967566995619,19cyclictest16850irq/134-800000000:25:010
3967566995614,13cyclictest12-21ksoftirqd/022:22:530
3967566995612,8cyclictest12-21ksoftirqd/023:22:150
396758099559,5cyclictest131rcu_preempt23:53:553
396758099559,5cyclictest131rcu_preempt21:43:363
396758099556,6cyclictest37-21ksoftirqd/322:05:263
3967580995511,5cyclictest131rcu_preempt22:47:073
396758099551,12cyclictest4049047-40mandb00:00:023
3967580995510,5cyclictest131rcu_preempt22:30:093
3967576995512,6cyclictest131rcu_preempt21:55:012
396756699559,9cyclictest12-21ksoftirqd/022:20:010
396756699557,8cyclictest12-21ksoftirqd/020:59:590
396756699554,11cyclictest12-21ksoftirqd/023:10:280
396756699553,7cyclictest121ksoftirqd/023:58:450
3967566995511,12cyclictest12-21ksoftirqd/020:25:020
396758099548,5cyclictest131rcu_preempt23:24:153
396758099547,8cyclictest37-21ksoftirqd/300:09:473
396758099547,8cyclictest37-21ksoftirqd/300:09:463
396758099547,10cyclictest37-21ksoftirqd/321:59:053
3967580995412,6cyclictest131rcu_preempt23:07:583
3967580995412,4cyclictest131rcu_preempt23:35:463
3967580995410,10cyclictest37-21ksoftirqd/322:28:093
396756699547,8cyclictest12-21ksoftirqd/000:27:450
396756699546,8cyclictest12-21ksoftirqd/021:11:460
396756699546,8cyclictest12-21ksoftirqd/021:11:450
396756699546,7cyclictest12-21ksoftirqd/021:53:040
396756699544,12cyclictest12-21ksoftirqd/022:07:320
396756699543,8cyclictest12-21ksoftirqd/023:20:040
3967566995435,16cyclictest16850irq/134-800000021:10:020
3967566995415,11cyclictest12-21ksoftirqd/023:51:000
396758099539,6cyclictest131rcu_preempt21:53:173
396758099538,5cyclictest131rcu_preempt19:40:033
396758099538,5cyclictest131rcu_preempt19:40:023
396758099536,5cyclictest131rcu_preempt20:20:023
396758099535,7cyclictest37-21ksoftirqd/323:46:023
3967580995343,7cyclictest361rcuc/323:30:023
3967580995313,7cyclictest37-21ksoftirqd/323:43:573
3967580995310,8cyclictest37-21ksoftirqd/323:16:403
3967576995344,6cyclictest3959056-21kworker/2:021:05:032
3967576995344,6cyclictest3959056-21kworker/2:021:05:032
396757199533,4cyclictest131rcu_preempt19:40:031
396757199533,4cyclictest131rcu_preempt19:40:021
396756699539,8cyclictest12-21ksoftirqd/022:00:010
396756699535,7cyclictest12-21ksoftirqd/020:10:020
396756699535,7cyclictest12-21ksoftirqd/020:10:010
396756699534,13cyclictest12-21ksoftirqd/019:55:010
3967566995339,9cyclictest3467602-21systemd-journal19:30:020
3967566995319,31cyclictest16850irq/134-800000000:04:100
3967566995319,17cyclictest16850irq/134-800000023:31:000
3967566995318,6cyclictest16850irq/134-800000022:33:550
3967566995313,4cyclictest131rcu_preempt20:50:000
396758099529,5cyclictest131rcu_preempt19:30:033
396758099526,12cyclictest37-21ksoftirqd/320:24:323
396758099526,10cyclictest37-21ksoftirqd/320:00:013
396758099525,10cyclictest37-21ksoftirqd/321:10:013
396758099524,10cyclictest37-21ksoftirqd/319:55:013
3967580995210,5cyclictest131rcu_preempt00:26:263
3967576995244,5cyclictest4040244-21kworker/2:300:35:582
3967576995244,5cyclictest3959056-21kworker/2:022:15:152
3967576995244,5cyclictest3959056-21kworker/2:021:23:152
3967576995243,6cyclictest3978236-21kworker/2:123:48:142
3967576995243,5cyclictest3978236-21kworker/2:123:22:342
396756699526,8cyclictest12-21ksoftirqd/022:10:100
396756699525,9cyclictest12-21ksoftirqd/019:20:000
3967566995217,31cyclictest16850irq/134-800000022:43:580
3967566995217,17cyclictest16850irq/134-800000021:40:200
396756699521,17cyclictest12-21ksoftirqd/020:55:030
396758099515,7cyclictest37-21ksoftirqd/319:35:003
396758099511,8cyclictest3467602-21systemd-journal19:45:003
3967580995113,7cyclictest131rcu_preempt22:18:373
3967580995113,10cyclictest37-21ksoftirqd/321:25:573
3967576995144,4cyclictest3978236-21kworker/2:123:35:092
3967576995143,5cyclictest3978236-21kworker/2:123:30:362
3967576995143,5cyclictest3978236-21kworker/2:122:03:162
3967576995143,5cyclictest3978236-21kworker/2:100:28:182
3967576995143,5cyclictest3959056-21kworker/2:021:40:112
3967576995142,6cyclictest3978236-21kworker/2:122:55:412
3967576995142,6cyclictest3959056-21kworker/2:021:38:212
3967576995142,5cyclictest3978236-21kworker/2:122:28:192
3967576995132,7cyclictest3978236-21kworker/2:100:23:252
3967571995137,8cyclictest3984872-21munin-plugin-st21:35:001
396756699517,10cyclictest12-21ksoftirqd/021:26:180
3967566995110,13cyclictest12-21ksoftirqd/020:35:000
396758099506,8cyclictest37-21ksoftirqd/322:40:373
396758099506,11cyclictest37-21ksoftirqd/320:45:023
396758099505,10cyclictest37-21ksoftirqd/322:37:423
3967580995037,9cyclictest3467602-21systemd-journal20:05:033
3967580995037,9cyclictest3467602-21systemd-journal20:05:023
3967580995012,5cyclictest131rcu_preempt21:00:013
3967576995042,5cyclictest3978236-21kworker/2:123:52:082
3967576995042,5cyclictest3959056-21kworker/2:022:22:222
396758099497,10cyclictest37-21ksoftirqd/320:15:023
396758099492,14cyclictest37-21ksoftirqd/320:55:013
396758099491,7cyclictest37-21ksoftirqd/320:50:013
396758099491,5cyclictest131rcu_preempt20:40:023
3967576994941,5cyclictest3978236-21kworker/2:122:50:522
3967576994941,5cyclictest3978236-21kworker/2:100:32:202
3967576994941,5cyclictest3959056-21kworker/2:023:17:092
3967576994941,5cyclictest3959056-21kworker/2:023:13:042
3967576994941,5cyclictest3959056-21kworker/2:023:01:222
3967576994941,5cyclictest3959056-21kworker/2:021:27:092
3967576994941,5cyclictest3959056-21kworker/2:020:40:032
396757199493,4cyclictest131rcu_preempt22:14:361
396757199493,4cyclictest131rcu_preempt20:15:011
396757199493,3cyclictest131rcu_preempt21:40:011
396758099489,4cyclictest131rcu_preempt19:20:033
3967576994840,5cyclictest3978236-21kworker/2:123:29:592
3967576994840,5cyclictest3978236-21kworker/2:122:45:332
3967576994840,5cyclictest3978236-21kworker/2:122:40:562
3967576994840,5cyclictest3978236-21kworker/2:121:57:062
3967576994840,5cyclictest3959056-21kworker/2:023:06:302
3967576994840,5cyclictest3959056-21kworker/2:022:11:412
3967576994840,5cyclictest3959056-21kworker/2:020:24:342
3967576994839,5cyclictest3978236-21kworker/2:123:44:502
3967576994839,5cyclictest3978236-21kworker/2:122:35:422
3967576994740,4cyclictest3959056-21kworker/2:020:55:522
3967576994739,5cyclictest3978236-21kworker/2:121:46:502
3967576994739,5cyclictest3959056-21kworker/2:021:12:062
3967576994739,5cyclictest3959056-21kworker/2:021:12:052
3967576994738,5cyclictest3978236-21kworker/2:100:04:192
3967576994733,11cyclictest1-21systemd21:20:012
3967571994732,11cyclictest1-21systemd23:35:011
396757199471,4cyclictest131rcu_preempt22:45:021
396757199471,3cyclictest131rcu_preempt00:18:011
396757199471,3cyclictest131rcu_preempt00:18:011
396757199471,13cyclictest3968576-21unin-run19:30:021
3967566994733,11cyclictest12-21ksoftirqd/019:40:030
3967566994733,11cyclictest12-21ksoftirqd/019:40:030
396756699473,11cyclictest12-21ksoftirqd/019:35:010
396756699472,8cyclictest12-21ksoftirqd/020:45:020
396758099469,7cyclictest131rcu_preempt19:50:013
396758099468,5cyclictest131rcu_preempt20:05:013
3967580994627,15cyclictest3467602-21systemd-journal19:15:023
3967580994613,5cyclictest131rcu_preempt19:10:023
3967576994639,4cyclictest3978236-21kworker/2:100:06:282
3967576994639,4cyclictest3978236-21kworker/2:100:06:272
3967576994638,6cyclictest3978236-21kworker/2:122:31:362
3967576994638,5cyclictest3959056-21kworker/2:019:35:112
3967576994638,5cyclictest3959056-21kworker/2:019:35:102
396757199463,3cyclictest131rcu_preempt21:58:441
3967571994631,11cyclictest1-21systemd19:15:011
3967571994626,15cyclictest275-21dbus-daemon22:30:011
396756699461,8cyclictest12-21ksoftirqd/019:10:020
3967576994524,16cyclictest275-21dbus-daemon20:15:002
3967571994534,6cyclictest4040253-21latency_hist23:40:021
3967571994528,13cyclictest275-21dbus-daemon20:40:001
396757199452,3cyclictest131rcu_preempt21:42:341
396756699458,10cyclictest12-21ksoftirqd/019:45:090
3967576994436,5cyclictest3978236-21kworker/2:123:58:222
3967576994436,5cyclictest3959056-21kworker/2:021:10:012
3967576994435,5cyclictest3959056-21kworker/2:021:31:522
3967576994435,5cyclictest3959056-21kworker/2:020:43:352
396757199441,3cyclictest131rcu_preempt21:13:111
396757199441,3cyclictest131rcu_preempt21:13:101
396756699441,38cyclictest3970347-21sshd20:00:370
3967576994335,5cyclictest3959056-21kworker/2:019:43:002
3967571994329,10cyclictest1-21systemd00:40:011
396757199431,5cyclictest131rcu_preempt23:25:001
3967571994315,5cyclictest131rcu_preempt22:59:361
396757199431,3cyclictest131rcu_preempt21:49:251
396756699432,6cyclictest3969358-21unin-run19:45:000
3967576994231,6cyclictest3959056-21kworker/2:020:10:022
3967576994231,6cyclictest3959056-21kworker/2:020:10:012
3967571994233,6cyclictest3972716-21sshd20:45:201
3967571994233,6cyclictest23-21ksoftirqd/121:55:011
3967571994226,12cyclictest275-21dbus-daemon20:35:011
3967571994226,11cyclictest309-21in:imuxsock20:30:001
3967576994111,5cyclictest131rcu_preempt20:27:132
396757199415,5cyclictest131rcu_preempt00:10:021
396757199415,5cyclictest131rcu_preempt00:10:021
396757199414,3cyclictest131rcu_preempt23:07:041
3967571994129,10cyclictest23-21ksoftirqd/123:53:181
3967571994126,12cyclictest23-21ksoftirqd/123:05:021
3967571994119,14cyclictest4058890-21rm00:21:531
3967571994114,9cyclictest4043664-21sshd23:47:361
396757199411,3cyclictest131rcu_preempt21:00:031
396757199411,27cyclictest275-21dbus-daemon00:00:031
3967576994032,5cyclictest3959056-21kworker/2:020:16:132
3967571994029,8cyclictest3968784-21sshd19:31:361
3967571994028,9cyclictest4051424-21latency_hist00:05:011
3967571994028,8cyclictest23-21ksoftirqd/123:43:081
3967571994025,11cyclictest23-21ksoftirqd/123:12:351
396757199402,4cyclictest131rcu_preempt20:05:011
3967571994021,14cyclictest1-21systemd19:55:011
396757699396,7cyclictest131rcu_preempt20:35:002
3967576993932,4cyclictest3959056-21kworker/2:019:47:522
3967576993931,5cyclictest3959056-21kworker/2:019:56:472
396757199395,7cyclictest4055738-21cat00:15:011
3967571993928,9cyclictest23-21ksoftirqd/123:20:031
3967571993927,10cyclictest23-21ksoftirqd/100:26:281
3967571993926,9cyclictest3973534-21cron21:05:011
3967571993926,9cyclictest3973534-21cron21:05:011
3967571993926,10cyclictest3969530-21sed19:45:021
3967571993923,13cyclictest4004875-21munin-run22:20:011
396757199391,7cyclictest131rcu_preempt21:20:021
396757199391,3cyclictest131rcu_preempt22:55:021
396757699385,9cyclictest30-21ksoftirqd/219:22:332
396757699384,8cyclictest30-21ksoftirqd/219:25:362
3967576993831,4cyclictest3959056-21kworker/2:020:51:172
3967576993830,5cyclictest3959056-21kworker/2:019:55:022
396757199387,7cyclictest23-21ksoftirqd/119:20:021
396757199383,12cyclictest3970633-21munin-run20:10:011
396757199383,12cyclictest3970633-21munin-run20:10:001
3967571993829,5cyclictest3973005-21munin-plugin-st20:54:591
3967571993828,7cyclictest23-21ksoftirqd/122:50:011
3967571993826,10cyclictest23-21ksoftirqd/122:31:321
3967571993825,10cyclictest23-21ksoftirqd/121:10:011
3967571993824,9cyclictest309-21in:imuxsock20:40:031
396757199382,3cyclictest131rcu_preempt23:26:121
3967571993823,12cyclictest23-21ksoftirqd/122:22:141
396757699373,3cyclictest131rcu_preempt20:00:372
3967576993730,4cyclictest3978236-21kworker/2:122:10:022
3967576993728,5cyclictest3968090-21latency_hist19:20:012
396757199378,6cyclictest131rcu_preempt00:35:021
3967571993726,9cyclictest23-21ksoftirqd/119:10:021
3967571993726,8cyclictest3978624-21sshd21:20:271
3967571993723,10cyclictest4013808-21date22:40:031
396757199371,4cyclictest131rcu_preempt20:20:031
396757199371,3cyclictest131rcu_preempt22:01:331
3967576993628,5cyclictest3959056-21kworker/2:020:50:022
396757199362,3cyclictest131rcu_preempt22:05:461
396757199352,3cyclictest131rcu_preempt21:30:011
3967571993423,8cyclictest23-21ksoftirqd/119:47:361
396757199342,11cyclictest131rcu_preempt20:24:341
3967576993225,5cyclictest3959056-21kworker/2:019:10:022
3967576993215,13cyclictest275-21dbus-daemon19:35:002
396757199322,4cyclictest131rcu_preempt20:00:001
3967571993223,6cyclictest3968441-21latency_hist19:25:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional