You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-24 - 13:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri Apr 24, 2026 00:45:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23784279910298,2cyclictest0-21swapper/019:50:020
237843699992,5cyclictest131rcu_preempt22:05:023
237842799849,69cyclictest364-21snmpd21:40:000
237842799821,64cyclictest2433591-21ssh23:03:130
2378427998027,49cyclictest2434753-21ssh23:05:450
2378427997818,54cyclictest16850irq/134-800000023:34:110
2378427997728,44cyclictest2420361-21ssh22:32:440
237842799772,5cyclictest3467602-21systemd-journal19:25:020
2378427997631,26cyclictest2389216-21ssh21:20:120
2378427997631,26cyclictest2389216-21ssh21:20:120
237842799761,11cyclictest2382758-21cron20:35:000
2378427997528,30cyclictest2386229-21ssh21:13:460
2378427997528,30cyclictest2386229-21ssh21:13:450
2378427997525,33cyclictest2392202-21ssh21:27:270
2378436997433,37cyclictest275-21dbus-daemon21:20:013
2378433997447,24cyclictest30-21ksoftirqd/223:50:022
2378427997329,28cyclictest2399301-21ssh21:44:100
2378427997323,45cyclictest2464692-21ssh00:15:090
2378427997319,49cyclictest275-21dbus-daemon22:59:590
2378427997230,26cyclictest2426535-21ssh22:46:570
237842799722,16cyclictest3467602-21systemd-journal19:40:000
2378427997128,24cyclictest2450128-21ssh23:41:240
2378427997128,24cyclictest2450128-21ssh23:41:240
237842799712,65cyclictest2410791-21sshd22:10:240
2378427997126,42cyclictest2406942-21ssh22:01:320
2378427997030,36cyclictest2454002-21ssh23:50:200
2378427997017,34cyclictest16850irq/134-800000000:05:230
2378436996925,40cyclictest2433721-21sh23:03:413
2378427996927,38cyclictest2384316-21sshd21:02:290
237842799692,18cyclictest2467709-21ssh00:22:220
2378427996921,45cyclictest2428940-21ssh22:52:340
2378427996921,44cyclictest2423747-21ssh22:40:150
2378427996913,41cyclictest2413099-21ssh22:15:480
2378427996911,53cyclictest2417651-21ssh22:26:160
237842799683,5cyclictest2416667-21ssh22:24:280
2378427996830,34cyclictest3467602-21systemd-journal21:20:000
237842799681,7cyclictest2439077-21ssh23:15:450
237842799674,53cyclictest16850irq/134-800000021:55:410
237842799673,59cyclictest16850irq/134-800000000:13:260
2378427996730,34cyclictest2384025-21sshd20:55:240
2378427996728,22cyclictest2444951-21ssh23:29:480
2378427996719,43cyclictest2437101-21ssh23:11:170
2378427996718,32cyclictest2441915-21ssh23:22:310
2378427996718,32cyclictest2441915-21ssh23:22:310
237842799671,61cyclictest2468977-21kworker/u8:000:38:590
2378436996624,38cyclictest2462589-21sh00:10:093
237842799662,59cyclictest2421597-21ssh22:35:190
237842799662,21cyclictest2400691-21ssh21:47:120
237842799661,11cyclictest48-21kcompactd000:30:270
2378436996530,31cyclictest2391329-21sh21:25:143
2378436996522,39cyclictest3467602-21systemd-journal22:57:233
2378427996532,29cyclictest2409840-21ssh22:08:330
2378427996524,36cyclictest2449091-21sh23:39:180
2378427996514,34cyclictest2394807-21ssh21:33:480
2378427996514,34cyclictest2394807-21ssh21:33:480
2378436996324,34cyclictest2440262-21sh23:18:513
2378436996322,38cyclictest2382649-21latency_hist20:30:013
2378436996320,5cyclictest2472915-21sh00:34:373
2378431996325,35cyclictest23-21ksoftirqd/123:50:021
237842799636,47cyclictest16850irq/134-800000023:45:480
237842799632,10cyclictest48-21kcompactd021:50:300
237843699622,18cyclictest364-21snmpd00:15:173
2378436996218,7cyclictest2402027-21latency_hist21:50:033
2378433996253,5cyclictest2371826-21kworker/2:222:15:512
2378427996212,47cyclictest150-21pr/ttyS200:29:360
2378427996212,47cyclictest150-21pr/ttyS200:29:360
237843699611,17cyclictest2445771-21ssh23:31:133
237842799611,6cyclictest2383810-21munin-run20:55:010
2378436996024,31cyclictest2475159-21sh00:39:493
2378436996023,33cyclictest2385650-21sh21:12:143
2378436996023,33cyclictest2385650-21sh21:12:143
237843699602,18cyclictest2380446-21munin-run19:50:003
2378436996021,7cyclictest2403949-21latency_hist21:55:023
237843699602,16cyclictest284-21systemd-logind23:10:003
2378427996016,39cyclictest364-21snmpd19:41:090
237842799601,47cyclictest16850irq/134-800000023:58:520
2378436995923,31cyclictest2409390-21sh22:07:183
2378436995922,33cyclictest2384598-21diskmemload23:22:243
2378436995922,33cyclictest2384598-21diskmemload23:22:243
2378436995920,4cyclictest2384598-21diskmemload21:58:033
2378436995914,10cyclictest275-21dbus-daemon22:35:013
2378433995950,6cyclictest2421292-21kworker/2:323:58:122
2378433995950,5cyclictest2371826-21kworker/2:222:07:062
2378433995950,5cyclictest2371826-21kworker/2:221:55:272
2378433995949,6cyclictest2371826-21kworker/2:220:04:232
2378427995927,28cyclictest2459807-21ssh00:04:090
237842799592,10cyclictest2380678-21sshd19:54:250
237842799592,10cyclictest2380678-21sshd19:54:240
237842799591,11cyclictest364-21snmpd20:06:280
237842799591,11cyclictest364-21snmpd20:06:270
2378436995826,27cyclictest2444634-21sh23:28:543
2378436995822,32cyclictest2384597-21latency_hist21:10:013
237843699582,18cyclictest2398412-21ssh21:41:523
2378436995819,6cyclictest2468034-21rm00:23:093
237842799581,10cyclictest2383277-21sshd20:42:240
237843699572,13cyclictest2455833-21munin-run23:55:003
237842799578,45cyclictest2381227-21munin-run20:05:010
2378427995739,14cyclictest16950irq/224-800000021:10:000
237842799572,5cyclictest2382535-21munin-run20:30:010
2378436995624,28cyclictest37-21ksoftirqd/322:43:543
237843699562,20cyclictest2449398-21munin-run23:40:003
2378436995614,9cyclictest37-21ksoftirqd/323:49:563
2378436995614,37cyclictest0-21swapper/320:15:013
2378433995649,4cyclictest2371826-21kworker/2:221:39:082
2378433995646,6cyclictest2383317-21kworker/2:021:27:182
237842799562,50cyclictest2378612-21latency_hist19:15:030
2378427995615,37cyclictest0-21swapper/020:15:010
2378436995521,6cyclictest37-21ksoftirqd/322:45:223
2378436995520,31cyclictest2384598-21diskmemload23:57:323
2378436995515,4cyclictest2413864-21sh22:17:443
2378433995548,4cyclictest2421292-21kworker/2:322:54:262
2378433995546,5cyclictest2371826-21kworker/2:219:30:252
237842799552,7cyclictest2378888-21munin-run19:20:010
237842799551,10cyclictest2380982-21systemd-run20:00:010
2378436995423,27cyclictest37-21ksoftirqd/322:10:243
237843699542,17cyclictest2461557-21ssh00:08:033
237843699542,16cyclictest2379837-21grep19:35:013
237843699542,16cyclictest2379046-21latency_hist19:20:013
2378436995416,4cyclictest2384598-21diskmemload23:10:293
2378433995446,6cyclictest2412608-21kworker/2:123:52:462
2378433995446,5cyclictest2466677-21kworker/2:200:23:242
2378433995444,5cyclictest2371826-21kworker/2:221:53:222
2378433995443,7cyclictest2412608-21kworker/2:122:27:312
237842799542,15cyclictest2378503-21grep19:10:020
237842799542,15cyclictest2378503-21grep19:10:010
2378427995420,31cyclictest0-21swapper/020:50:020
237843699532,16cyclictest2422860-21ssh22:38:373
2378436995317,5cyclictest2469314-21sh00:25:503
2378436995317,5cyclictest2469314-21sh00:25:503
2378433995345,5cyclictest2421292-21kworker/2:323:41:022
2378433995345,5cyclictest2421292-21kworker/2:323:41:022
2378433995345,5cyclictest2421292-21kworker/2:323:17:542
2378433995345,5cyclictest2421292-21kworker/2:322:58:302
2378433995345,5cyclictest2371826-21kworker/2:222:20:312
2378433995345,5cyclictest2371826-21kworker/2:219:55:022
2378433995345,5cyclictest2371826-21kworker/2:219:55:012
2378433995344,6cyclictest2412608-21kworker/2:122:36:542
2378433995344,6cyclictest2412608-21kworker/2:122:31:342
2378433995344,6cyclictest2371826-21kworker/2:222:00:302
2378433995344,5cyclictest2421292-21kworker/2:323:30:002
2378431995335,14cyclictest1-21systemd22:40:001
237842799531,9cyclictest2381973-21sshd20:19:230
2378427995317,32cyclictest2379628-21sshd19:31:250
237843699522,16cyclictest2381490-21unin-run20:10:013
237843699522,16cyclictest2381490-21unin-run20:10:003
237843699522,15cyclictest2381994-21modprobe20:20:003
237843699521,16cyclictest364-21snmpd00:04:073
2378433995244,5cyclictest2421292-21kworker/2:323:20:392
2378433995244,5cyclictest2421292-21kworker/2:323:20:392
2378433995244,5cyclictest2421292-21kworker/2:300:00:202
2378433995243,5cyclictest2371826-21kworker/2:222:11:122
2378433995242,6cyclictest2371826-21kworker/2:221:46:252
2378433995232,13cyclictest2371826-21kworker/2:221:43:442
2378436995113,5cyclictest0-21swapper/320:05:013
2378436995112,11cyclictest275-21dbus-daemon22:25:013
2378433995143,5cyclictest2421292-21kworker/2:323:31:092
2378433995143,5cyclictest2421292-21kworker/2:322:42:252
2378433995143,5cyclictest2412608-21kworker/2:100:15:562
2378433995143,5cyclictest2383317-21kworker/2:021:12:232
2378433995143,5cyclictest2383317-21kworker/2:021:12:232
2378433995143,5cyclictest2371826-21kworker/2:221:31:562
2378433995143,5cyclictest2371826-21kworker/2:221:31:562
2378433995143,5cyclictest2371826-21kworker/2:221:00:012
2378433995143,5cyclictest2371826-21kworker/2:220:10:032
2378433995143,5cyclictest2371826-21kworker/2:220:10:032
2378433995142,5cyclictest2421292-21kworker/2:323:00:272
2378433995140,7cyclictest2412608-21kworker/2:100:27:422
2378433995140,7cyclictest2412608-21kworker/2:100:27:422
237843199517,6cyclictest131rcu_preempt20:10:011
237843199517,6cyclictest131rcu_preempt20:10:001
237842799512,12cyclictest2379390-21sshd19:29:250
2378436995023,24cyclictest37-21ksoftirqd/322:27:433
237843699502,17cyclictest2390321-21ssh21:23:103
237843699502,17cyclictest2390321-21ssh21:23:093
2378436995013,13cyclictest275-21dbus-daemon20:00:003
2378436995012,13cyclictest275-21dbus-daemon20:50:003
237843699501,15cyclictest2382283-21munin-run20:25:013
237843699501,15cyclictest2382283-21munin-run20:25:013
2378433995042,5cyclictest2421292-21kworker/2:323:36:122
2378433995042,5cyclictest2421292-21kworker/2:323:08:272
2378433995042,5cyclictest2412608-21kworker/2:100:05:292
2378433995042,5cyclictest2383317-21kworker/2:021:23:162
2378433995042,5cyclictest2383317-21kworker/2:021:23:162
2378433995042,5cyclictest2371826-21kworker/2:220:50:022
2378433995041,5cyclictest2412608-21kworker/2:100:13:182
2378433995012,22cyclictest2382785-21kworker/2:320:40:032
2378433995012,22cyclictest2382785-21kworker/2:320:40:032
237842799503,7cyclictest2382271-21modprobe20:25:010
237842799503,7cyclictest2382271-21modprobe20:25:010
237842799501,14cyclictest2383008-21sshd20:39:240
237842799501,14cyclictest2383008-21sshd20:39:230
237843699492,17cyclictest2396516-21ssh21:37:383
2378436994920,11cyclictest37-21ksoftirqd/320:40:013
2378436994920,11cyclictest37-21ksoftirqd/320:40:013
237843699491,18cyclictest2451102-21ssh23:43:543
237843699491,18cyclictest2451102-21ssh23:43:543
2378436994911,8cyclictest2379338-21sort19:25:033
2378433994941,5cyclictest2412608-21kworker/2:123:12:242
2378433994941,5cyclictest2412608-21kworker/2:100:30:202
2378433994941,5cyclictest2383317-21kworker/2:021:16:032
2378433994941,5cyclictest2371826-21kworker/2:220:22:172
2378433994941,5cyclictest2371826-21kworker/2:220:22:162
2378433994940,6cyclictest2421292-21kworker/2:322:45:362
2378433994914,6cyclictest131rcu_preempt21:10:012
2378431994932,12cyclictest275-21dbus-daemon21:35:011
2378431994932,12cyclictest275-21dbus-daemon21:35:001
2378431994927,17cyclictest3467602-21systemd-journal19:55:011
2378431994927,17cyclictest3467602-21systemd-journal19:55:011
2378436994822,7cyclictest37-21ksoftirqd/321:00:013
2378433994840,5cyclictest2371826-21kworker/2:219:30:012
2378433994840,5cyclictest2371826-21kworker/2:219:10:022
2378433994840,5cyclictest2371826-21kworker/2:219:10:022
2378431994818,14cyclictest0-21swapper/123:00:021
237843199481,12cyclictest2457998-21true00:00:001
2378436994717,5cyclictest2429300-21rm22:53:263
2378433994738,5cyclictest2371826-21kworker/2:219:15:012
2378433994737,6cyclictest2371826-21kworker/2:219:25:012
237843199471,8cyclictest2410423-21kworker/u8:022:10:011
2378436994616,5cyclictest2382766-21modprobe20:35:003
2378433994639,5cyclictest2412608-21kworker/2:100:35:252
2378433994637,5cyclictest2371826-21kworker/2:219:58:062
2378431994637,6cyclictest2381866-21cat20:15:021
2378431994628,14cyclictest275-21dbus-daemon20:20:011
237843199461,5cyclictest131rcu_preempt21:10:011
237843199461,13cyclictest2425633-21ssh22:45:021
237843699452,16cyclictest2393539-21ssh21:30:243
237843699452,16cyclictest2393539-21ssh21:30:233
237843199458,29cyclictest2451568-21ssh23:45:021
237843199458,29cyclictest2451568-21ssh23:45:021
2378431994535,7cyclictest2384598-21diskmemload23:30:011
2378431994532,9cyclictest2414815-21systemd-run22:20:011
2378431994532,10cyclictest23-21ksoftirqd/100:05:461
2378431994528,12cyclictest23-21ksoftirqd/119:20:021
2378431994516,26cyclictest221rcuc/123:35:031
2378433994335,5cyclictest2371826-21kworker/2:220:52:272
2378431994327,13cyclictest2384423-21sed21:05:021
2378431994327,12cyclictest23-21ksoftirqd/119:37:211
2378431994327,11cyclictest2380957-21systemctl19:59:591
2378431994324,14cyclictest275-21dbus-daemon20:40:011
2378431994324,14cyclictest275-21dbus-daemon20:40:001
237843699425,9cyclictest37-21ksoftirqd/319:30:013
237843399424,5cyclictest131rcu_preempt20:30:012
2378433994222,5cyclictest30-21ksoftirqd/219:49:262
237843199425,4cyclictest131rcu_preempt20:45:001
2378431994232,6cyclictest2399777-21cat21:45:011
2378431994231,8cyclictest2379374-21sshd19:26:261
2378431994229,10cyclictest2393203-21sed21:30:011
2378431994229,10cyclictest23-21ksoftirqd/123:16:511
2378431994228,11cyclictest2455838-21systemd-run23:55:011
2378431994228,10cyclictest23-21ksoftirqd/122:46:531
2378431994224,14cyclictest3467602-21systemd-journal23:06:231
2378431994212,26cyclictest131rcu_preempt00:26:081
2378431994212,26cyclictest131rcu_preempt00:26:081
2378433994132,5cyclictest2371826-21kworker/2:220:30:122
237843199414,5cyclictest131rcu_preempt21:50:031
237843199414,3cyclictest131rcu_preempt00:10:241
237843199412,34cyclictest275-21dbus-daemon22:05:011
2378431994112,25cyclictest364-21snmpd21:55:011
2378436994023,11cyclictest275-21dbus-daemon20:45:003
237843399406,29cyclictest2379913-21sh19:40:002
237843199405,4cyclictest131rcu_preempt19:25:011
237843199404,3cyclictest131rcu_preempt00:23:591
237843199403,3cyclictest131rcu_preempt22:30:581
2378431994027,10cyclictest2466680-21systemd-run00:20:011
237843199402,6cyclictest131rcu_preempt21:20:001
2378431994026,11cyclictest23-21ksoftirqd/121:59:171
237843199402,4cyclictest131rcu_preempt22:55:011
2378431994024,10cyclictest275-21dbus-daemon20:50:011
2378436993926,9cyclictest3467602-21systemd-journal20:52:283
2378433993923,4cyclictest30-21ksoftirqd/220:20:002
237843199395,3cyclictest131rcu_preempt23:32:571
237843199395,3cyclictest131rcu_preempt22:27:221
2378431993930,6cyclictest2381222-21sh20:05:001
2378431993929,7cyclictest2416657-21sshd22:24:241
2378431993928,8cyclictest2472452-21sshd00:33:221
2378431993928,8cyclictest23-21ksoftirqd/121:24:561
2378431993928,8cyclictest23-21ksoftirqd/121:24:561
2378431993928,8cyclictest23-21ksoftirqd/100:36:431
2378431993927,9cyclictest23-21ksoftirqd/123:20:201
2378431993927,9cyclictest23-21ksoftirqd/123:20:201
2378431993927,8cyclictest23-21ksoftirqd/120:56:291
2378431993927,8cyclictest23-21ksoftirqd/119:35:021
2378431993925,11cyclictest23-21ksoftirqd/123:00:511
2378431993924,11cyclictest23-21ksoftirqd/121:36:081
2378431993828,6cyclictest2380433-21cron19:49:591
237843199381,3cyclictest131rcu_preempt22:14:431
237843699373,7cyclictest131rcu_preempt19:40:013
2378436993726,9cyclictest37-21ksoftirqd/319:55:013
2378436993726,9cyclictest37-21ksoftirqd/319:55:013
2378433993727,7cyclictest3467602-21systemd-journal20:10:232
2378431993724,10cyclictest23-21ksoftirqd/100:00:501
237843199372,3cyclictest131rcu_preempt20:30:001
2378431993723,10cyclictest2438535-21cron23:15:001
2378431993718,14cyclictest275-21dbus-daemon20:35:011
237843199371,3cyclictest131rcu_preempt21:11:381
237843199371,3cyclictest131rcu_preempt21:11:371
237843199363,7cyclictest131rcu_preempt19:10:021
237843199363,7cyclictest131rcu_preempt19:10:011
2378436993526,6cyclictest2378718-21latency_hist19:15:023
2378436993518,13cyclictest2384085-21tr21:00:003
2378436993425,7cyclictest37-21ksoftirqd/319:45:033
2378433993422,8cyclictest271irq_work/219:16:262
2378433993421,9cyclictest2383501-21cat20:45:012
237843199342,3cyclictest131rcu_preempt20:55:021
237843199342,3cyclictest131rcu_preempt19:11:251
2378431993420,10cyclictest275-21dbus-daemon19:45:011
2378431993319,11cyclictest2382453-21cat20:25:031
2378431993319,11cyclictest2382453-21cat20:25:021
2378433993224,4cyclictest2371826-21kworker/2:221:00:052
2378433993219,10cyclictest275-21dbus-daemon19:45:002
2378436993020,7cyclictest37-21ksoftirqd/319:10:033
2378436993020,7cyclictest37-21ksoftirqd/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional