You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-09 - 08:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Aug 09, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3122456999118,18cyclictest12-21ksoftirqd/019:50:020
3122456997622,27cyclictest12-21ksoftirqd/023:03:290
3122456997622,27cyclictest12-21ksoftirqd/023:03:290
3122456997517,38cyclictest12-21ksoftirqd/021:16:210
3122470997330,8cyclictest37-21ksoftirqd/322:45:013
3122470997330,8cyclictest37-21ksoftirqd/322:45:013
3122456997350,14cyclictest151rcuc/020:45:010
312245699701,5cyclictest131rcu_preempt23:40:000
312245699701,5cyclictest131rcu_preempt23:40:000
3122456996916,28cyclictest12-21ksoftirqd/022:13:590
3122465996845,5cyclictest3140318-21kworker/2:222:42:162
3122465996845,5cyclictest3140318-21kworker/2:222:42:162
3122470996453,7cyclictest37-21ksoftirqd/300:25:013
312247099643,56cyclictest3208209-40mandb00:00:033
3122465996444,8cyclictest3112456-21kworker/2:321:34:362
3122465996444,8cyclictest3112456-21kworker/2:321:34:362
3122456996415,40cyclictest12-21ksoftirqd/023:19:160
3122456996411,26cyclictest12-21ksoftirqd/021:59:470
312245699638,6cyclictest131rcu_preempt22:30:030
3122456996331,14cyclictest3226693-21ssh00:40:010
312245699631,28cyclictest131rcu_preempt23:14:110
312245699631,28cyclictest131rcu_preempt23:14:110
312247099621,12cyclictest37-21ksoftirqd/322:20:013
312247099616,8cyclictest37-21ksoftirqd/321:10:023
312246299601,54cyclictest3122463-21latency_hist19:10:021
312246299601,54cyclictest3122463-21latency_hist19:10:011
312247099594,11cyclictest37-21ksoftirqd/300:00:273
3122470995924,12cyclictest37-21ksoftirqd/323:45:003
3122456995938,16cyclictest1-21systemd22:40:010
312245699592,39cyclictest3208645-21ssh00:00:230
3122456995920,4cyclictest12-21ksoftirqd/021:45:240
312247099587,7cyclictest131rcu_preempt19:30:003
3122465995820,18cyclictest3203665-21ssh23:50:022
3122456995827,15cyclictest3155041-21ssh22:06:150
3122456995814,40cyclictest12-21ksoftirqd/022:30:050
3122470995723,7cyclictest37-21ksoftirqd/322:15:003
312247099571,11cyclictest37-21ksoftirqd/322:50:003
3122456995725,15cyclictest3135657-21ssh21:25:010
312245699572,39cyclictest3212015-21ssh00:07:580
312245699572,39cyclictest3139477-21ssh21:33:060
312245699572,39cyclictest3139477-21ssh21:33:060
312245699571,40cyclictest131rcu_preempt22:54:040
312245699571,40cyclictest131rcu_preempt22:54:040
3122470995610,5cyclictest131rcu_preempt21:30:023
3122465995648,5cyclictest3210560-21kworker/2:300:35:022
3122465995647,5cyclictest3140318-21kworker/2:223:10:092
3122465995647,5cyclictest3140318-21kworker/2:223:10:092
3122456995627,12cyclictest3213859-21ssh00:11:570
3122456995626,26cyclictest3199928-21ssh23:41:580
312247099558,6cyclictest131rcu_preempt22:02:583
312247099557,8cyclictest37-21ksoftirqd/323:34:243
3122470995538,12cyclictest275-21dbus-daemon22:55:013
3122470995538,12cyclictest275-21dbus-daemon22:55:013
3122465995547,5cyclictest3140318-21kworker/2:222:27:442
312246299551,9cyclictest284-21systemd-logind22:20:011
3122456995535,4cyclictest16850irq/134-800000022:43:330
3122456995535,4cyclictest16850irq/134-800000022:43:330
3122456995534,18cyclictest12-21ksoftirqd/000:30:340
312245699552,6cyclictest131rcu_preempt23:00:010
3122456995526,13cyclictest3182873-21ssh23:05:170
3122456995525,14cyclictest3190109-21ssh23:20:510
3122456995525,14cyclictest3129658-21ssh21:11:520
3122456995525,14cyclictest3129658-21ssh21:11:520
3122456995522,8cyclictest12-21ksoftirqd/000:00:010
3122456995517,34cyclictest12-21ksoftirqd/021:39:370
312247099547,5cyclictest131rcu_preempt21:14:553
312247099547,5cyclictest131rcu_preempt21:14:553
312247099546,6cyclictest131rcu_preempt19:35:013
3122470995424,5cyclictest131rcu_preempt23:40:013
3122470995424,5cyclictest131rcu_preempt23:40:003
312247099541,5cyclictest131rcu_preempt21:34:363
312247099541,5cyclictest131rcu_preempt21:34:363
312246599545,34cyclictest3149694-21sh21:55:022
3122465995446,5cyclictest3210560-21kworker/2:300:10:022
3122465995446,4cyclictest3210560-21kworker/2:300:24:352
3122465995444,5cyclictest3112456-21kworker/2:321:40:012
3122465995435,14cyclictest3467602-21systemd-journal21:45:002
3122462995438,12cyclictest1-21systemd00:30:011
312245699548,21cyclictest131rcu_preempt22:45:050
3122456995430,10cyclictest3217234-21ssh00:19:330
3122456995423,11cyclictest12-21ksoftirqd/022:20:010
312245699541,44cyclictest0-21swapper/023:25:490
312247099538,4cyclictest131rcu_preempt00:20:013
312247099536,8cyclictest37-21ksoftirqd/323:20:203
3122470995338,11cyclictest3192323-21sshd23:25:323
3122470995316,9cyclictest37-21ksoftirqd/300:12:093
3122470995316,7cyclictest37-21ksoftirqd/323:03:243
3122470995316,7cyclictest37-21ksoftirqd/323:03:243
3122470995314,6cyclictest131rcu_preempt00:32:083
3122465995345,5cyclictest3140318-21kworker/2:222:59:142
3122465995345,5cyclictest3140318-21kworker/2:222:35:072
3122465995345,5cyclictest3112456-21kworker/2:321:26:322
312245699536,5cyclictest131rcu_preempt21:44:580
3122456995327,11cyclictest3201906-21ssh23:46:110
3122456995322,15cyclictest3152519-21ssh22:00:420
3122456995321,23cyclictest12-21ksoftirqd/023:31:220
3122456995312,4cyclictest12-21ksoftirqd/000:25:530
3122470995214,6cyclictest131rcu_preempt21:40:003
312247099521,10cyclictest37-21ksoftirqd/321:45:013
3122465995246,4cyclictest3210560-21kworker/2:300:37:362
3122465995244,5cyclictest3140318-21kworker/2:223:19:512
3122465995244,5cyclictest3140318-21kworker/2:223:09:112
3122465995244,5cyclictest3140318-21kworker/2:222:10:322
3122465995243,6cyclictest3140318-21kworker/2:222:16:282
3122465995242,6cyclictest3210560-21kworker/2:300:11:232
3122456995229,14cyclictest12-21ksoftirqd/022:25:020
3122456995226,11cyclictest3137926-21ssh21:29:570
312245699521,36cyclictest3204803-21ssh23:52:280
312247099518,10cyclictest37-21ksoftirqd/322:00:023
3122470995141,6cyclictest37-21ksoftirqd/321:49:243
3122470995133,13cyclictest275-21dbus-daemon19:39:593
3122470995131,15cyclictest3467602-21systemd-journal23:15:013
3122470995131,15cyclictest3467602-21systemd-journal23:15:013
3122470995129,15cyclictest275-21dbus-daemon21:55:013
3122470995113,6cyclictest131rcu_preempt23:20:013
3122465995143,5cyclictest3177847-21kworker/2:023:35:032
3122465995143,5cyclictest3112456-21kworker/2:321:23:422
3122465995142,5cyclictest3140318-21kworker/2:223:25:132
3122465995140,7cyclictest3140318-21kworker/2:222:25:002
3122465995132,13cyclictest275-21dbus-daemon23:25:012
3122462995139,9cyclictest23-21ksoftirqd/123:34:011
3122456995124,13cyclictest3148426-21ssh21:52:050
312247099507,5cyclictest131rcu_preempt23:09:583
312247099505,7cyclictest37-21ksoftirqd/323:51:163
312247099505,7cyclictest37-21ksoftirqd/300:29:203
3122470995035,7cyclictest309-21in:imuxsock21:00:003
3122465995042,5cyclictest3140318-21kworker/2:223:44:132
3122465995042,5cyclictest3140318-21kworker/2:222:53:172
3122465995042,5cyclictest3140318-21kworker/2:222:53:172
3122465995042,5cyclictest3140318-21kworker/2:221:59:322
3122465995041,5cyclictest3210560-21kworker/2:300:20:012
3122465995041,5cyclictest3206153-21kworker/2:100:01:182
3122465995041,5cyclictest3140318-21kworker/2:222:30:312
3122456995041,5cyclictest16950irq/224-800000020:20:000
312247099494,9cyclictest37-21ksoftirqd/320:05:013
3122470994913,6cyclictest131rcu_preempt22:35:003
3122470994913,6cyclictest131rcu_preempt20:20:003
3122470994912,6cyclictest131rcu_preempt23:46:133
3122465994942,4cyclictest3140318-21kworker/2:223:57:012
3122465994941,5cyclictest3177847-21kworker/2:023:35:252
3122465994941,5cyclictest3177847-21kworker/2:023:35:252
3122465994941,5cyclictest3140318-21kworker/2:223:04:482
3122465994941,5cyclictest3140318-21kworker/2:223:04:482
3122465994941,5cyclictest3140318-21kworker/2:222:01:292
3122465994932,3cyclictest0-21swapper/220:45:022
3122462994930,13cyclictest275-21dbus-daemon23:10:001
3122462994913,12cyclictest3208217-40gzip00:00:031
312245699496,6cyclictest131rcu_preempt19:16:590
312247099487,12cyclictest37-21ksoftirqd/323:00:003
312247099486,11cyclictest37-21ksoftirqd/319:45:013
312247099484,5cyclictest131rcu_preempt00:07:283
3122470994835,10cyclictest37-21ksoftirqd/319:20:013
312247099481,8cyclictest0-21swapper/320:10:013
312247099481,8cyclictest0-21swapper/320:10:013
312247099481,4cyclictest131rcu_preempt22:27:523
3122465994840,5cyclictest3112456-21kworker/2:320:20:022
3122465994840,5cyclictest3112456-21kworker/2:319:25:012
3122465994834,10cyclictest1-21systemd19:40:012
312247099479,6cyclictest131rcu_preempt19:15:003
312247099477,9cyclictest37-21ksoftirqd/320:00:033
312247099472,11cyclictest37-21ksoftirqd/320:45:013
3122470994716,5cyclictest131rcu_preempt00:37:403
3122470994716,23cyclictest275-21dbus-daemon22:40:013
312247099471,4cyclictest131rcu_preempt21:16:373
3122470994711,6cyclictest37-21ksoftirqd/322:23:033
312246599476,26cyclictest291rcuc/221:10:022
3122465994740,4cyclictest3112456-21kworker/2:321:11:172
3122465994740,4cyclictest3112456-21kworker/2:321:11:172
3122465994739,5cyclictest3140318-21kworker/2:221:45:242
3122465994738,6cyclictest3112456-21kworker/2:320:40:022
3122465994738,5cyclictest3112456-21kworker/2:321:15:362
3122465994734,10cyclictest3467602-21systemd-journal20:02:312
312245699472,6cyclictest131rcu_preempt20:40:010
3122456994719,18cyclictest12-21ksoftirqd/000:24:390
3122470994614,10cyclictest37-21ksoftirqd/322:06:373
3122470994612,6cyclictest131rcu_preempt21:25:023
3122470994610,8cyclictest131rcu_preempt20:25:033
3122465994638,5cyclictest3210560-21kworker/2:300:29:552
3122465994638,5cyclictest3112456-21kworker/2:320:15:012
3122465994637,6cyclictest3140318-21kworker/2:223:51:252
3122462994630,12cyclictest3467602-21systemd-journal20:05:011
3122456994626,14cyclictest1-21systemd19:25:000
3122456994625,11cyclictest12-21ksoftirqd/020:35:010
312245699461,5cyclictest131rcu_preempt21:10:000
312247099452,15cyclictest37-21ksoftirqd/320:30:013
3122470994511,5cyclictest131rcu_preempt20:55:013
3122465994537,5cyclictest3140318-21kworker/2:222:46:082
3122465994536,6cyclictest3112456-21kworker/2:319:30:022
3122465994536,5cyclictest3140318-21kworker/2:222:08:452
3122462994537,6cyclictest23-21ksoftirqd/123:55:021
3122470994412,6cyclictest131rcu_preempt20:40:023
3122470994411,5cyclictest131rcu_preempt19:50:023
3122465994436,5cyclictest3112456-21kworker/2:319:49:142
3122465994436,5cyclictest3112456-21kworker/2:319:10:162
3122462994434,7cyclictest3182681-21cat23:05:021
3122462994434,7cyclictest3182681-21cat23:05:021
312246299443,11cyclictest23-21ksoftirqd/122:40:021
3122462994430,10cyclictest23-21ksoftirqd/100:25:021
3122462994424,14cyclictest3467602-21systemd-journal19:50:001
3122462994410,6cyclictest131rcu_preempt00:05:581
3122456994429,10cyclictest3122460-21latency_hist19:10:020
3122456994429,10cyclictest3122460-21latency_hist19:10:010
312247099437,11cyclictest37-21ksoftirqd/320:35:013
3122470994331,10cyclictest37-21ksoftirqd/320:12:323
3122470994327,11cyclictest3467602-21systemd-journal21:01:323
3122465994336,5cyclictest3112456-21kworker/2:319:51:092
312246299433,3cyclictest131rcu_preempt00:05:011
3122462994330,10cyclictest3198585-21sshd23:39:321
3122462994330,10cyclictest3198585-21sshd23:39:321
3122462994330,10cyclictest23-21ksoftirqd/121:40:021
3122462994329,10cyclictest3467602-21systemd-journal22:05:001
3122462994327,12cyclictest275-21dbus-daemon00:40:011
312246299432,3cyclictest131rcu_preempt21:29:031
3122456994334,4cyclictest16950irq/224-800000019:45:010
312245699432,37cyclictest3123420-21sshd19:27:130
3122456994319,20cyclictest12-21ksoftirqd/019:50:000
312247099421,6cyclictest0-21swapper/319:55:003
3122465994235,4cyclictest3112456-21kworker/2:319:15:142
3122465994234,5cyclictest3112456-21kworker/2:320:55:202
3122465994234,5cyclictest3112456-21kworker/2:320:33:382
3122465994233,5cyclictest3112456-21kworker/2:319:10:032
3122465994233,5cyclictest3112456-21kworker/2:319:10:032
3122465994227,11cyclictest3126360-21sed20:25:022
3122462994227,11cyclictest23-21ksoftirqd/123:29:081
3122462994226,12cyclictest23-21ksoftirqd/121:41:211
3122462994225,13cyclictest23-21ksoftirqd/121:22:271
312246299421,3cyclictest131rcu_preempt23:25:011
3122456994229,8cyclictest1-21systemd20:05:000
3122456994218,19cyclictest12-21ksoftirqd/019:55:030
3122456994214,21cyclictest0-21swapper/020:30:010
312245699421,35cyclictest3125510-21modprobe20:10:000
312245699421,35cyclictest3125510-21modprobe20:09:590
312247099416,8cyclictest37-21ksoftirqd/320:50:023
3122465994134,5cyclictest3112456-21kworker/2:319:45:022
312246299413,3cyclictest131rcu_preempt00:15:021
3122462994131,7cyclictest23-21ksoftirqd/121:13:441
3122462994131,7cyclictest23-21ksoftirqd/121:13:441
3122462994128,8cyclictest3133258-21cron21:20:001
3122462994128,10cyclictest23-21ksoftirqd/122:50:021
3122462994127,11cyclictest23-21ksoftirqd/123:42:321
3122462994126,12cyclictest23-21ksoftirqd/123:17:451
3122462994125,11cyclictest309-21in:imuxsock19:24:591
312246299412,3cyclictest131rcu_preempt22:30:491
3122462994122,14cyclictest3467602-21systemd-journal19:30:001
312246299411,22cyclictest0-21swapper/122:57:131
3122456994134,4cyclictest16950irq/224-800000021:05:010
312245699412,8cyclictest3128105-21munin-run21:00:000
312245699412,5cyclictest3127847-21munin-run20:55:000
3122456994119,18cyclictest12-21ksoftirqd/020:15:010
3122465994028,9cyclictest3467602-21systemd-journal21:00:022
312246599401,26cyclictest3125529-21unin-run20:10:012
312246599401,26cyclictest3125529-21unin-run20:10:012
312246299405,4cyclictest131rcu_preempt19:40:001
3122462994029,9cyclictest23-21ksoftirqd/119:15:011
3122462994028,10cyclictest23-21ksoftirqd/122:14:361
3122462994027,10cyclictest23-21ksoftirqd/100:30:091
3122462994025,12cyclictest23-21ksoftirqd/119:20:021
3122462994024,12cyclictest3124973-21cron20:00:011
312246299401,6cyclictest3164029-21ssh22:25:151
312245699402,6cyclictest3122666-21munin-run19:15:010
312247099399,5cyclictest131rcu_preempt19:25:013
312246599398,9cyclictest30-21ksoftirqd/220:30:012
3122465993923,11cyclictest30-21ksoftirqd/220:00:012
312246299398,5cyclictest131rcu_preempt22:05:391
312246299394,3cyclictest131rcu_preempt23:15:001
312246299394,3cyclictest131rcu_preempt23:15:001
3122462993927,9cyclictest3123726-21sed19:35:021
3122462993927,8cyclictest3126014-21cron20:20:001
3122462993924,11cyclictest3467602-21systemd-journal21:00:001
312246299391,3cyclictest131rcu_preempt21:57:281
312245699393,8cyclictest12-21ksoftirqd/019:40:000
3122456993923,12cyclictest3126285-21sshd20:22:310
3122465993830,5cyclictest3112456-21kworker/2:319:35:012
3122462993827,9cyclictest23-21ksoftirqd/122:52:101
3122462993827,9cyclictest23-21ksoftirqd/122:52:101
3122462993825,10cyclictest1-21systemd21:10:001
3122462993823,11cyclictest3467602-21systemd-journal20:10:031
312246299381,7cyclictest0-21swapper/121:55:021
312246299381,3cyclictest131rcu_preempt22:41:491
312246299381,3cyclictest131rcu_preempt22:41:491
312245699381,5cyclictest3127656-21wc20:50:020
312247099378,5cyclictest131rcu_preempt19:10:033
312247099378,5cyclictest131rcu_preempt19:10:033
312246299373,4cyclictest131rcu_preempt23:50:021
3122462993726,8cyclictest3147282-21cron21:50:001
3122462993724,10cyclictest3124211-21sshd19:43:141
312246299365,3cyclictest131rcu_preempt21:05:021
312246299364,4cyclictest131rcu_preempt20:45:021
3122462993629,4cyclictest3127064-21latency_hist20:40:031
3122462993622,11cyclictest23-21ksoftirqd/121:30:251
3122462993622,11cyclictest23-21ksoftirqd/121:30:251
312246299358,3cyclictest131rcu_preempt20:25:021
312246299353,3cyclictest131rcu_preempt20:35:021
3122462993523,9cyclictest23-21ksoftirqd/122:20:361
312245699351,30cyclictest0-21swapper/019:31:160
3122465993419,12cyclictest3127628-21cat20:50:012
312246299344,4cyclictest131rcu_preempt20:10:011
312246299344,4cyclictest131rcu_preempt20:10:011
3122462993423,8cyclictest23-21ksoftirqd/100:19:471
3122462993412,15cyclictest0-21swapper/120:55:001
3122465993325,5cyclictest3112456-21kworker/2:320:55:022
312246299334,4cyclictest131rcu_preempt20:29:591
312246299332,6cyclictest131rcu_preempt19:55:001
3122462993321,8cyclictest3127581-21cron20:50:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional