You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 19:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Fri Apr 10, 2026 12:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2477520991071,8cyclictest0-21swapper/010:40:030
2477524999914,39cyclictest1-21systemd11:15:011
247752499902,71cyclictest0-21swapper/107:25:021
247752699811,52cyclictest2537557-21ssh11:14:422
247752699811,51cyclictest2552724-21ssh11:49:392
247752699798,53cyclictest2483754-21ssh09:10:022
2477520997712,43cyclictest19350irq/34-mmc111:30:030
2477520997712,43cyclictest19350irq/34-mmc111:30:020
247752499762,43cyclictest2555870-21ssh11:56:391
247752499752,37cyclictest2567589-21ssh12:24:041
2477529997127,30cyclictest2477513-21cyclictest09:25:033
2477524997128,38cyclictest2541100-21sh11:22:391
247752499711,45cyclictest2544777-21ssh11:30:581
247752499711,45cyclictest2544777-21ssh11:30:581
247752699702,44cyclictest2483684-21diskmemload12:17:372
247752499702,38cyclictest2484023-21ssh09:10:151
247752699691,44cyclictest2483684-21diskmemload12:28:232
247752699691,44cyclictest2483684-21diskmemload12:28:222
247752699691,43cyclictest2483684-21diskmemload10:21:512
247752499672,36cyclictest2492813-21ssh09:30:431
2477526996625,36cyclictest2523779-21sh10:42:242
247752699661,61cyclictest2510481-21ssh10:11:412
247752699661,43cyclictest2534301-21ssh11:06:492
2477520996612,38cyclictest151rcuc/012:05:020
247752699652,59cyclictest2533229-21ssh11:04:422
247752699652,59cyclictest2533229-21ssh11:04:412
247752699652,58cyclictest364-21snmpd10:04:262
2477526996525,36cyclictest2478245-21latency_hist07:25:022
247752699651,60cyclictest2545629-21ssh11:33:082
247752699651,60cyclictest2545629-21ssh11:33:072
247752699651,60cyclictest2540732-21ssh11:21:382
247752699651,60cyclictest2525043-21ssh10:45:092
247752699651,60cyclictest2495520-21ssh09:37:082
2477520996561,2cyclictest0-21swapper/011:50:020
2477526996426,34cyclictest2543457-21sshd11:28:132
247752699641,59cyclictest2571809-21ssh12:33:442
247752699641,59cyclictest2566477-21ssh12:21:072
247752699641,59cyclictest2528075-21ssh10:52:252
247752699641,59cyclictest2518847-21ssh10:30:502
247752699641,59cyclictest2509406-21ssh10:09:252
247752699641,59cyclictest2498627-21ssh09:44:372
247752699641,59cyclictest2492093-21ssh09:29:262
247752699641,59cyclictest2486789-21ssh09:16:492
2477524996426,34cyclictest2572188-21sh12:34:381
2477520996434,25cyclictest2565817-21cron12:20:000
247752699632,57cyclictest2502463-21ssh09:53:192
247752699631,58cyclictest2574234-21ssh12:39:262
247752699631,58cyclictest2562254-21ssh12:11:252
247752699631,58cyclictest2557028-21ssh11:59:452
247752699631,58cyclictest2554428-21ssh11:53:282
247752699631,58cyclictest2549539-21ssh11:42:022
247752699631,58cyclictest2530349-21ssh10:57:432
247752699631,58cyclictest2521357-21ssh10:36:482
247752699631,58cyclictest2517276-21ssh10:27:232
247752699631,58cyclictest2505050-21ssh09:59:232
247752699631,58cyclictest2500286-21ssh09:48:202
247752699631,58cyclictest2492900-21ssh09:30:572
247752699631,58cyclictest2485365-21ssh09:13:512
247752499631,39cyclictest2549924-21ssh11:43:051
2477520996333,20cyclictest3467602-21systemd-journal07:55:000
247752699621,58cyclictest2546636-21ssh11:35:102
247752699621,57cyclictest2559924-21ssh12:05:572
247752699621,57cyclictest2557713-21ssh12:00:452
247752699621,57cyclictest2539299-21ssh11:18:332
247752699621,57cyclictest2513118-21ssh10:17:522
247752699621,57cyclictest2489099-21ssh09:22:102
247752699621,57cyclictest2478472-21sshd07:27:122
2477524996224,34cyclictest2500329-21sh09:48:251
247752499621,33cyclictest2483684-21diskmemload11:08:291
2477520996210,47cyclictest12-21ksoftirqd/009:12:220
247752699611,56cyclictest2481588-21sshd08:28:132
247752499619,25cyclictest0-21swapper/112:29:511
247752499619,25cyclictest0-21swapper/112:29:501
2477524996130,27cyclictest1-21systemd07:45:001
2477524996125,32cyclictest2502177-21sh09:52:321
2477524996123,32cyclictest2483684-21diskmemload10:03:421
2477524996122,35cyclictest23-21ksoftirqd/110:31:431
247752699601,55cyclictest2481066-21sshd08:20:002
247752699601,54cyclictest2480784-21sshd08:10:122
247752499602,53cyclictest2565367-21ssh12:18:551
247752499602,52cyclictest2531554-21ssh11:00:211
247752499602,52cyclictest2531554-21ssh11:00:201
247752499601,55cyclictest2518425-21cut10:30:021
247752699591,54cyclictest2479259-21sshd07:43:142
247752499593,24cyclictest2483684-21diskmemload10:45:291
247752499592,53cyclictest2560702-21ssh12:07:551
2477524995923,32cyclictest2483684-21diskmemload12:10:111
247752499591,54cyclictest2558016-21ssh12:01:381
247752499591,54cyclictest2551042-21ssh11:45:231
247752499591,30cyclictest0-21swapper/111:37:541
247752099595,16cyclictest2488089-21cat09:20:020
2477520995925,31cyclictest12-21ksoftirqd/009:23:330
2477520995913,41cyclictest12-21ksoftirqd/010:27:530
247752699582,52cyclictest2480546-21sshd08:08:142
247752699582,51cyclictest2483506-21cat09:05:032
247752699581,53cyclictest2482088-21sshd08:37:122
247752699581,52cyclictest2481374-21munin-run08:25:012
247752499582,51cyclictest2492288-21ssh09:29:541
247752499582,51cyclictest2478723-21sshd07:32:131
247752499581,53cyclictest2513341-21ssh10:18:241
247752499581,52cyclictest2543304-21ssh11:27:421
247752499581,52cyclictest2496905-21sh09:40:061
247752499581,52cyclictest2487392-21sh09:18:241
2477520995817,37cyclictest12-21ksoftirqd/012:20:560
2477529995728,24cyclictest2487998-21systemctl09:20:003
247752699571,50cyclictest48-21kcompactd007:58:062
247752499572,51cyclictest2524442-21sh10:44:091
247752499572,50cyclictest2538854-21ssh11:17:281
247752499572,50cyclictest2479799-21munin-run07:55:021
247752499572,48cyclictest2481849-21cron08:35:001
247752499572,48cyclictest2481849-21cron08:35:001
247752499571,52cyclictest2530047-21sh10:57:001
247752499571,52cyclictest2527685-21ssh10:51:271
247752499571,51cyclictest2505307-21ssh09:59:591
247752499571,50cyclictest2482380-21cron08:45:001
2477520995730,11cyclictest2573326-21ssh12:37:060
2477520995729,12cyclictest2525714-21ssh10:46:480
2477520995710,43cyclictest12-21ksoftirqd/009:45:400
247752699563,45cyclictest2481848-21cron08:35:012
247752699563,45cyclictest2481848-21cron08:35:002
247752699562,48cyclictest2479773-21cron07:55:002
247752699561,52cyclictest2482869-21sshd08:51:142
247752499562,50cyclictest2554039-21ssh11:52:331
247752499562,49cyclictest2483669-21cron09:09:591
247752499561,51cyclictest2572943-21ssh12:36:011
247752499561,51cyclictest2515062-21ssh10:22:121
247752499561,51cyclictest2510117-21ssh10:10:411
247752499561,51cyclictest2494979-21ssh09:35:421
247752499561,51cyclictest2482944-21cat08:55:011
2477520995636,15cyclictest275-21dbus-daemon10:15:010
247752699552,48cyclictest2482369-21sshd08:42:142
247752699552,47cyclictest2478880-21latency_hist07:35:022
247752699551,49cyclictest2478974-21sshd07:37:122
247752499551,50cyclictest2521781-21ssh10:38:001
247752499551,50cyclictest2488279-21ssh09:20:061
247752699541,49cyclictest2479513-21sshd07:49:122
247752499541,49cyclictest2507795-21ssh10:05:141
247752099545,35cyclictest3467602-21systemd-journal07:40:000
2477520995450,2cyclictest0-21swapper/008:15:010
247752099543,44cyclictest2505353-21sendmail-msp10:00:000
2477520995421,14cyclictest2544183-21ssh11:30:020
2477529995334,15cyclictest1-21systemd10:30:003
247752499532,46cyclictest2481133-21wc08:20:021
247752499532,46cyclictest2480331-21munin-run08:05:021
247752499532,46cyclictest2480331-21munin-run08:05:021
247752499531,47cyclictest2477684-21sshd07:13:131
2477520995323,14cyclictest2553890-21ssh11:52:110
2477529995235,12cyclictest1-21systemd09:00:013
247752699521,47cyclictest48-21kcompactd007:13:092
247752699521,47cyclictest364-21snmpd08:55:062
247752499522,45cyclictest48-21kcompactd008:45:151
247752499522,45cyclictest48-21kcompactd008:45:141
247752499522,45cyclictest2483113-21sshd08:55:071
2477524995213,35cyclictest2481354-21latency_hist08:24:591
247752099524,9cyclictest2529059-21munin-run10:55:010
2477520995225,12cyclictest2506055-21ssh10:01:200
247752699512,45cyclictest2477928-21sshd07:16:142
247752699511,46cyclictest2480294-21sshd08:03:122
247752699511,46cyclictest2480294-21sshd08:03:122
2477524995125,22cyclictest2477971-21latency_hist07:20:011
247752499512,44cyclictest275-21dbus-daemon08:15:011
247752499512,44cyclictest2478261-21munin-run07:25:021
247752499511,47cyclictest2480001-21sshd07:55:131
247752499511,46cyclictest2481664-21cut08:30:021
247752499511,46cyclictest0-21swapper/107:47:121
247752499511,45cyclictest364-21snmpd08:09:111
247752099519,6cyclictest131rcu_preempt11:01:200
247752099519,6cyclictest131rcu_preempt11:01:190
2477520995126,21cyclictest2493780-21sshd09:33:130
247752099512,16cyclictest2522585-21munin-run10:40:000
2477520995119,27cyclictest12-21ksoftirqd/007:20:030
2477529995038,8cyclictest2565815-21latency_hist12:20:003
247752699503,43cyclictest3467602-21systemd-journal08:50:012
247752699503,43cyclictest3467602-21systemd-journal08:50:002
247752499501,46cyclictest2482096-21sshd08:39:141
2477520995041,5cyclictest16950irq/224-800000011:40:000
2477520995034,12cyclictest309-21in:imuxsock09:40:000
2477520995019,8cyclictest12-21ksoftirqd/010:15:050
2477520994942,4cyclictest16850irq/134-800000009:45:010
2477520994931,14cyclictest2539518-21ssh11:19:130
2477520994923,11cyclictest2518654-21ssh10:30:220
2477520994922,11cyclictest2535953-21ssh11:10:310
2477520994918,28cyclictest12-21ksoftirqd/010:57:180
2477529994832,10cyclictest2511929-21cat10:15:013
247752499482,40cyclictest2478990-21cron07:40:011
247752099483,9cyclictest1-21systemd09:10:010
2477520994838,6cyclictest12-21ksoftirqd/009:30:010
2477520994739,5cyclictest16850irq/134-800000010:10:010
2477520994733,7cyclictest309-21in:imuxsock10:25:000
2477520994728,16cyclictest275-21dbus-daemon07:25:010
2477520994725,9cyclictest2556076-21ssh11:57:100
247752099472,31cyclictest19350irq/34-mmc108:30:030
247752099472,31cyclictest19350irq/34-mmc108:30:020
2477520994719,12cyclictest2562820-21ssh12:12:530
247752099471,42cyclictest25576551ssh12:00:380
2477529994631,12cyclictest2482624-21sshd08:48:133
2477529994631,12cyclictest2482624-21sshd08:48:123
2477529994631,12cyclictest1-21systemd08:20:003
2477529994629,12cyclictest275-21dbus-daemon10:10:013
2477529994626,15cyclictest275-21dbus-daemon07:30:003
247752499461,41cyclictest364-21snmpd09:03:281
2477520994638,5cyclictest12-21ksoftirqd/011:22:590
2477529994529,11cyclictest309-21in:imuxsock11:00:013
2477529994529,11cyclictest309-21in:imuxsock11:00:013
247752699451,2cyclictest0-21swapper/207:10:012
247752499451,38cyclictest2477647-21latency_hist07:10:011
247752099454,31cyclictest19350irq/34-mmc108:30:010
247752099453,9cyclictest2481088-21munin-plugin-st08:20:000
2477520994532,11cyclictest12-21ksoftirqd/009:00:000
2477520994527,13cyclictest275-21dbus-daemon07:15:000
2477520994526,5cyclictest12-21ksoftirqd/012:30:130
247752999443,3cyclictest131rcu_preempt12:05:353
2477529994429,12cyclictest2570159-21systemd-run12:30:013
2477529994429,12cyclictest2570159-21systemd-run12:30:013
2477529994428,11cyclictest3467602-21systemd-journal09:45:003
2477529994425,13cyclictest275-21dbus-daemon08:30:013
247752999442,3cyclictest131rcu_preempt10:24:173
247752999442,36cyclictest2478236-21cron07:25:013
247752099443,6cyclictest131rcu_preempt12:30:020
247752099443,6cyclictest131rcu_preempt12:30:020
2477520994427,13cyclictest2549304-21ssh11:41:280
2477520994419,10cyclictest2501739-21ssh09:51:300
2477520994413,5cyclictest2479313-21kworker/0:108:06:200
2477529994333,8cyclictest37-21ksoftirqd/311:45:303
2477529994330,10cyclictest37-21ksoftirqd/311:08:463
2477529994328,10cyclictest3467602-21systemd-journal09:50:013
247752099432,13cyclictest2479536-21modprobe07:50:010
247752099431,12cyclictest2534341-21ssh11:06:560
2477529994230,8cyclictest3467602-21systemd-journal12:20:143
2477529994230,10cyclictest37-21ksoftirqd/310:20:013
2477529994226,12cyclictest37-21ksoftirqd/309:56:453
2477529994226,12cyclictest275-21dbus-daemon08:40:003
2477529994224,13cyclictest3467602-21systemd-journal11:55:023
2477529994134,5cyclictest37-21ksoftirqd/310:04:563
2477529994130,9cyclictest37-21ksoftirqd/309:10:593
2477529994125,12cyclictest2483582-21sed09:05:023
247752999412,3cyclictest131rcu_preempt10:55:003
2477529994112,9cyclictest37-21ksoftirqd/310:46:113
2477520994117,20cyclictest2482642-21latency_hist08:50:020
2477520994117,20cyclictest2482642-21latency_hist08:50:020
247752099411,36cyclictest0-21swapper/008:37:300
247752999408,5cyclictest131rcu_preempt08:05:013
247752999408,5cyclictest131rcu_preempt08:05:003
2477529994030,7cyclictest309-21in:imuxsock09:29:593
2477529994029,6cyclictest309-21in:imuxsock07:15:003
2477529994028,10cyclictest37-21ksoftirqd/310:30:493
2477529994027,9cyclictest37-21ksoftirqd/310:35:253
2477529994027,10cyclictest37-21ksoftirqd/308:10:003
2477529994026,11cyclictest37-21ksoftirqd/311:42:473
2477529994026,10cyclictest37-21ksoftirqd/311:10:303
2477529994025,12cyclictest37-21ksoftirqd/311:39:493
2477529994024,12cyclictest37-21ksoftirqd/312:32:423
2477529994024,12cyclictest2480262-21cat08:00:023
2477520994025,11cyclictest2480014-21sshd07:58:140
2477529993928,9cyclictest37-21ksoftirqd/312:11:023
2477529993928,6cyclictest2479004-21sh07:40:013
2477529993927,9cyclictest37-21ksoftirqd/311:20:023
2477529993926,9cyclictest37-21ksoftirqd/311:33:023
2477529993926,9cyclictest37-21ksoftirqd/311:33:023
2477529993926,9cyclictest1-21systemd11:25:013
247752999392,4cyclictest131rcu_preempt12:40:013
2477529993924,10cyclictest275-21dbus-daemon08:25:003
2477520993927,9cyclictest12-21ksoftirqd/008:55:020
2477529993826,9cyclictest37-21ksoftirqd/308:45:003
2477529993825,9cyclictest37-21ksoftirqd/307:45:013
247752999381,3cyclictest131rcu_preempt10:59:313
2477520993831,4cyclictest16950irq/224-800000007:35:010
2477520993822,12cyclictest2479269-21sshd07:44:540
2477529993724,10cyclictest37-21ksoftirqd/311:30:003
2477529993724,10cyclictest37-21ksoftirqd/309:39:403
2477529993724,10cyclictest37-21ksoftirqd/308:15:013
2477529993723,11cyclictest37-21ksoftirqd/309:53:443
2477520993726,7cyclictest309-21in:imuxsock08:45:010
2477520993725,9cyclictest2478476-21sshd07:28:130
247752099371,32cyclictest2483584-21latency_hist09:05:020
2477529993627,6cyclictest2483747-21sh09:10:023
2477529993626,8cyclictest37-21ksoftirqd/310:40:523
2477529993625,8cyclictest37-21ksoftirqd/312:03:333
2477529993624,10cyclictest37-21ksoftirqd/311:52:243
2477520993612,19cyclictest3467602-21systemd-journal08:24:300
247752999353,5cyclictest37-21ksoftirqd/309:31:473
247752999353,5cyclictest131rcu_preempt08:54:593
2477529993523,7cyclictest2477956-21cron07:20:003
2477529993424,6cyclictest2481924-21latency_hist08:35:023
2477529993424,6cyclictest2481924-21latency_hist08:35:013
2477529993422,9cyclictest37-21ksoftirqd/307:35:003
247752099341,31cyclictest0-21swapper/008:05:020
247752099341,31cyclictest0-21swapper/008:05:010
2477529993322,8cyclictest37-21ksoftirqd/307:51:123
2477529993123,5cyclictest2477623-21latency_hist07:10:023
2477529993119,9cyclictest2479498-21sshd07:45:143
247752099304,6cyclictest131rcu_preempt07:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional