You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-24 - 19:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Sep 24, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2692965998319,6cyclictest16850irq/134-800000011:40:000
269296599809,52cyclictest2717717-21ssh09:50:020
2692965997427,12cyclictest2783758-21ssh12:12:460
2692965997417,6cyclictest16850irq/134-800000010:29:220
2692965997325,11cyclictest2752794-21ssh11:05:450
2692965997325,11cyclictest2752794-21ssh11:05:440
2692965997317,7cyclictest16850irq/134-800000009:56:140
2692965997316,31cyclictest16850irq/134-800000011:25:530
2692965997315,33cyclictest16850irq/134-800000011:48:100
2692965997229,11cyclictest2794423-21ssh12:35:510
2692965997227,11cyclictest2751260-21ssh11:02:360
2692965997218,32cyclictest16850irq/134-800000012:04:370
2692965997218,31cyclictest16850irq/134-800000012:31:070
2692965997218,31cyclictest16850irq/134-800000011:19:190
2692965997217,31cyclictest16850irq/134-800000010:20:220
2692965997215,32cyclictest16850irq/134-800000012:19:570
2692965997215,32cyclictest16850irq/134-800000010:39:400
2692965997117,32cyclictest16850irq/134-800000011:22:220
2692965997114,18cyclictest16850irq/134-800000009:21:220
2692975997057,10cyclictest0-21swapper/207:10:022
2692965997017,30cyclictest16850irq/134-800000009:32:510
2692965996915,32cyclictest16850irq/134-800000011:12:330
2692965996717,31cyclictest16850irq/134-800000010:54:150
2692965996626,35cyclictest2694735-21cron07:45:010
2692965996626,35cyclictest2694735-21cron07:45:000
2692965996561,2cyclictest0-21swapper/008:35:020
2692965996515,33cyclictest16850irq/134-800000010:04:240
2692970996339,17cyclictest3467602-21systemd-journal12:20:001
2692965996254,3cyclictest0-21swapper/010:15:030
2692975996125,12cyclictest30-21ksoftirqd/211:55:102
2692970995914,6cyclictest131rcu_preempt10:12:211
2692970995838,14cyclictest275-21dbus-daemon08:40:001
269296599583,36cyclictest2693464-21kworker/u8:307:20:020
2692965995830,13cyclictest2790239-21ssh12:26:500
2692965995728,13cyclictest2726603-21ssh10:09:180
2692965995726,15cyclictest2700638-21ssh09:12:470
2692970995625,10cyclictest23-21ksoftirqd/110:26:481
2692970995620,11cyclictest23-21ksoftirqd/112:10:021
2692965995620,32cyclictest12-21ksoftirqd/010:57:090
2692965995620,32cyclictest12-21ksoftirqd/010:57:080
269297099554,31cyclictest23-21ksoftirqd/111:30:021
2692970995540,11cyclictest23-21ksoftirqd/107:40:011
2692970995516,30cyclictest23-21ksoftirqd/111:10:031
2692970995516,30cyclictest23-21ksoftirqd/111:10:021
2692965995528,12cyclictest2769967-21ssh11:43:070
2692965995526,14cyclictest2703687-21ssh09:19:400
2692965995518,33cyclictest16850irq/134-800000010:34:130
2692965995518,32cyclictest16850irq/134-800000012:07:310
269296599551,38cyclictest2742559-21ssh10:43:410
2692975995429,5cyclictest2697129-21kworker/2:310:07:542
2692970995433,16cyclictest3467602-21systemd-journal11:00:131
2692970995421,11cyclictest23-21ksoftirqd/109:21:491
2692965995426,13cyclictest2714496-21ssh09:42:540
269296599542,3cyclictest131rcu_preempt09:37:370
2692965995419,31cyclictest16850irq/134-800000012:22:340
2692965995418,6cyclictest16850irq/134-800000009:25:080
2692965995418,32cyclictest16850irq/134-800000010:14:470
2692979995334,15cyclictest275-21dbus-daemon07:25:013
2692975995345,5cyclictest2724664-21kworker/2:111:08:072
2692975995345,5cyclictest2724664-21kworker/2:111:08:072
2692975995345,5cyclictest2697129-21kworker/2:309:29:082
2692975995344,6cyclictest2724664-21kworker/2:110:15:312
2692975995344,5cyclictest2724664-21kworker/2:111:37:262
2692970995320,10cyclictest23-21ksoftirqd/108:00:021
2692970995313,6cyclictest131rcu_preempt09:30:121
2692965995326,11cyclictest2775731-21ssh11:55:140
2692965995317,31cyclictest16850irq/134-800000010:48:450
2692965995316,33cyclictest16850irq/134-800000011:50:150
2692979995236,12cyclictest1-21systemd09:20:013
2692979995235,13cyclictest3467602-21systemd-journal12:35:013
2692975995244,5cyclictest2793909-21kworker/2:012:37:472
2692975995244,5cyclictest2697129-21kworker/2:310:47:282
2692975995243,6cyclictest2697129-21kworker/2:309:37:532
269297099528,6cyclictest131rcu_preempt12:04:411
2692970995240,8cyclictest23-21ksoftirqd/108:20:011
2692965995218,30cyclictest2697135-21latency_hist08:30:020
2692975995143,5cyclictest2697129-21kworker/2:310:12:022
2692975995142,5cyclictest2697129-21kworker/2:312:25:062
2692975995142,5cyclictest2697129-21kworker/2:312:02:322
2692970995125,10cyclictest23-21ksoftirqd/111:46:011
2692970995110,6cyclictest131rcu_preempt10:00:001
2692965995128,7cyclictest2764056-21ssh11:30:040
2692975995042,5cyclictest2724664-21kworker/2:112:30:282
2692975995042,5cyclictest2724664-21kworker/2:111:35:032
2692975995042,5cyclictest2724664-21kworker/2:111:22:272
2692975995042,5cyclictest2724664-21kworker/2:111:12:122
2692975995042,5cyclictest2724664-21kworker/2:111:00:322
2692975995042,5cyclictest2724664-21kworker/2:110:59:282
2692975995042,5cyclictest2724664-21kworker/2:110:59:282
2692975995042,5cyclictest2697129-21kworker/2:311:26:522
2692975995042,5cyclictest2695784-21kworker/2:009:46:502
2692975995041,6cyclictest2697129-21kworker/2:311:45:222
2692975995041,6cyclictest2697129-21kworker/2:309:44:502
2692975995034,12cyclictest2739208-21sshd10:36:132
269297599502,3cyclictest131rcu_preempt09:20:012
269297099507,5cyclictest131rcu_preempt11:23:511
269297099506,6cyclictest131rcu_preempt12:24:451
269297099505,7cyclictest23-21ksoftirqd/112:35:121
2692970995037,9cyclictest23-21ksoftirqd/111:35:031
2692970995014,12cyclictest23-21ksoftirqd/111:14:031
2692970995014,10cyclictest131rcu_preempt09:20:001
2692979994935,10cyclictest3467602-21systemd-journal12:30:003
2692975994941,5cyclictest2724664-21kworker/2:110:53:122
2692975994941,5cyclictest2697129-21kworker/2:312:08:242
2692975994941,5cyclictest2697129-21kworker/2:309:14:052
2692975994941,5cyclictest2693926-21kworker/2:108:00:022
269297099496,4cyclictest131rcu_preempt10:40:381
269297099496,12cyclictest23-21ksoftirqd/108:05:011
269297099495,6cyclictest131rcu_preempt10:56:471
269297099495,6cyclictest131rcu_preempt10:56:461
2692970994940,6cyclictest23-21ksoftirqd/110:37:291
2692970994923,17cyclictest275-21dbus-daemon11:45:001
2692970994913,6cyclictest131rcu_preempt09:11:591
2692965994915,30cyclictest16850irq/134-800000009:54:410
2692965994915,30cyclictest16850irq/134-800000009:54:400
2692979994831,13cyclictest2736205-21ssh10:30:013
269297599485,10cyclictest30-21ksoftirqd/210:25:012
2692975994841,4cyclictest2695784-21kworker/2:009:54:042
2692975994841,4cyclictest2695784-21kworker/2:009:54:042
2692975994840,5cyclictest2724664-21kworker/2:112:22:462
2692975994840,5cyclictest2697129-21kworker/2:312:13:142
2692975994840,5cyclictest2697129-21kworker/2:311:54:472
2692975994840,5cyclictest2697129-21kworker/2:310:41:372
2692975994840,5cyclictest2697129-21kworker/2:309:32:552
2692975994839,5cyclictest2697129-21kworker/2:311:18:152
2692975994838,7cyclictest2697129-21kworker/2:308:45:022
2692975994838,7cyclictest2697129-21kworker/2:308:45:022
269297099486,5cyclictest131rcu_preempt09:40:131
269297099485,6cyclictest131rcu_preempt10:19:131
269297099485,6cyclictest131rcu_preempt09:55:021
269297099485,6cyclictest131rcu_preempt09:55:011
269297099484,8cyclictest23-21ksoftirqd/110:20:041
2692970994816,5cyclictest131rcu_preempt12:31:431
2692970994814,6cyclictest131rcu_preempt09:30:021
2692970994810,6cyclictest131rcu_preempt12:00:001
269297999472,3cyclictest131rcu_preempt08:35:003
269297599475,4cyclictest131rcu_preempt09:25:012
2692975994739,5cyclictest2697129-21kworker/2:310:29:492
2692975994739,5cyclictest2695784-21kworker/2:009:56:482
2692975994739,5cyclictest2693926-21kworker/2:108:10:022
2692975994739,5cyclictest2693926-21kworker/2:108:10:022
2692970994733,10cyclictest23-21ksoftirqd/110:34:281
2692970994732,9cyclictest2712968-21cron09:40:001
2692970994732,11cyclictest1-21systemd08:45:011
2692970994732,11cyclictest1-21systemd08:45:011
2692970994717,6cyclictest131rcu_preempt07:10:021
2692970994710,5cyclictest131rcu_preempt09:49:291
269296599477,7cyclictest12-21ksoftirqd/008:20:000
2692965994733,9cyclictest275-21dbus-daemon08:15:000
2692979994634,5cyclictest309-21in:imuxsock11:15:003
2692975994638,5cyclictest2724664-21kworker/2:111:40:502
2692975994638,5cyclictest2724664-21kworker/2:110:34:542
2692975994635,8cyclictest3467602-21systemd-journal08:56:132
2692975994632,10cyclictest30-21ksoftirqd/207:55:012
2692970994636,7cyclictest23-21ksoftirqd/111:19:131
2692970994636,7cyclictest23-21ksoftirqd/110:52:551
2692970994622,17cyclictest275-21dbus-daemon07:45:011
2692970994622,17cyclictest275-21dbus-daemon07:45:011
2692970994614,7cyclictest23-21ksoftirqd/111:50:141
2692970994610,7cyclictest131rcu_preempt11:33:131
2692970994610,6cyclictest131rcu_preempt12:27:181
2692965994639,4cyclictest16950irq/224-800000009:10:010
2692965994639,4cyclictest16950irq/224-800000009:10:010
2692965994630,12cyclictest2694461-21cat07:40:000
269296599462,40cyclictest2695999-21sshd08:06:130
269296599462,40cyclictest2695999-21sshd08:06:130
269297999458,6cyclictest0-21swapper/312:17:033
2692979994532,9cyclictest2712989-21systemctl09:40:003
2692979994532,7cyclictest2698135-21cron08:50:003
2692979994532,10cyclictest37-21ksoftirqd/311:37:323
2692979994527,14cyclictest275-21dbus-daemon09:30:023
269297999452,4cyclictest131rcu_preempt11:43:403
269297099453,5cyclictest131rcu_preempt12:14:071
2692970994535,7cyclictest23-21ksoftirqd/110:05:071
2692970994518,6cyclictest131rcu_preempt08:35:011
2692970994513,5cyclictest131rcu_preempt10:47:041
2692970994513,5cyclictest131rcu_preempt07:25:001
2692970994510,6cyclictest131rcu_preempt08:55:001
2692979994432,9cyclictest275-21dbus-daemon10:15:013
2692979994425,15cyclictest2761704-21wc11:25:023
2692975994437,5cyclictest2697129-21kworker/2:308:47:142
2692975994435,5cyclictest2695784-21kworker/2:010:02:282
2692975994421,7cyclictest30-21ksoftirqd/208:35:012
269297599441,3cyclictest131rcu_preempt07:30:002
269297099449,7cyclictest131rcu_preempt10:05:021
269297099447,6cyclictest131rcu_preempt08:30:011
269297999437,25cyclictest2729206-21kworker/u8:011:10:033
269297999437,25cyclictest2729206-21kworker/u8:011:10:023
269297999435,33cyclictest2740791-21munin-plugin-st10:40:003
269297999433,4cyclictest131rcu_preempt09:56:113
2692979994332,8cyclictest37-21ksoftirqd/312:40:023
2692979994332,8cyclictest1-21systemd11:20:023
269297999432,3cyclictest131rcu_preempt10:17:193
269297999431,4cyclictest131rcu_preempt10:32:113
269297999431,3cyclictest131rcu_preempt07:35:033
269297999431,3cyclictest131rcu_preempt07:35:023
2692975994335,5cyclictest2683250-21kworker/2:207:17:132
269297599431,3cyclictest131rcu_preempt12:16:132
2692970994320,16cyclictest3467602-21systemd-journal08:10:001
2692970994320,16cyclictest3467602-21systemd-journal08:10:001
2692979994233,5cyclictest274-21cron09:00:003
2692979994229,10cyclictest37-21ksoftirqd/310:48:303
2692979994229,10cyclictest0-21swapper/307:30:013
2692979994227,10cyclictest1-21systemd08:20:013
2692979994222,14cyclictest1-21systemd10:55:013
2692975994228,10cyclictest275-21dbus-daemon07:25:002
2692970994234,5cyclictest23-21ksoftirqd/107:55:011
2692970994227,11cyclictest275-21dbus-daemon07:50:001
2692970994211,6cyclictest131rcu_preempt07:10:011
2692970994210,5cyclictest131rcu_preempt08:50:011
2692965994236,3cyclictest16950irq/224-800000008:55:020
2692965994229,9cyclictest309-21in:imuxsock08:55:020
269297999413,3cyclictest131rcu_preempt10:40:433
2692979994131,7cyclictest37-21ksoftirqd/312:10:023
2692979994126,11cyclictest1-21systemd08:15:003
269297999412,3cyclictest131rcu_preempt11:56:153
269297999411,3cyclictest131rcu_preempt11:51:143
269297999411,3cyclictest131rcu_preempt10:24:183
2692975994133,5cyclictest2695784-21kworker/2:008:28:012
2692975994127,9cyclictest1-21systemd08:05:002
269297599411,3cyclictest131rcu_preempt08:40:022
269297099418,6cyclictest131rcu_preempt09:10:031
269297099418,6cyclictest131rcu_preempt09:10:021
2692970994127,10cyclictest23-21ksoftirqd/107:20:001
2692965994135,3cyclictest16950irq/224-800000008:35:000
2692965994126,11cyclictest2696794-21sshd08:22:130
269296599411,36cyclictest0-21swapper/008:50:000
269297999409,6cyclictest131rcu_preempt09:48:043
269297999404,3cyclictest131rcu_preempt11:31:203
2692979994029,9cyclictest37-21ksoftirqd/308:40:013
2692975994024,12cyclictest275-21dbus-daemon07:40:002
269297599402,3cyclictest131rcu_preempt07:45:002
269297599402,3cyclictest131rcu_preempt07:45:002
2692975994022,14cyclictest275-21dbus-daemon09:10:022
2692975994022,14cyclictest275-21dbus-daemon09:10:012
269297599401,3cyclictest131rcu_preempt08:20:022
269297099408,5cyclictest131rcu_preempt09:00:001
269297099405,6cyclictest131rcu_preempt08:25:011
2692965994033,4cyclictest16950irq/224-800000008:05:020
2692965994028,9cyclictest2695494-21sendmail08:00:000
269297999394,3cyclictest131rcu_preempt08:45:023
269297999394,3cyclictest131rcu_preempt08:45:013
269297999393,3cyclictest131rcu_preempt07:50:003
2692979993930,5cyclictest2696810-21sh08:25:003
2692979993929,6cyclictest2693442-21sendmail07:20:003
2692979993928,8cyclictest2773141-21sed11:50:013
2692979993927,8cyclictest275-21dbus-daemon11:05:013
2692979993926,10cyclictest37-21ksoftirqd/309:00:423
2692979993924,11cyclictest37-21ksoftirqd/307:45:003
2692979993924,11cyclictest37-21ksoftirqd/307:45:003
269297999392,34cyclictest2694472-21readlink07:40:003
2692979993923,12cyclictest1-21systemd09:15:013
2692979993920,13cyclictest1-21systemd07:15:013
269297099396,6cyclictest131rcu_preempt07:35:001
269297099396,6cyclictest131rcu_preempt07:35:001
2692965993933,4cyclictest16950irq/224-800000007:29:130
2692965993921,14cyclictest3467602-21systemd-journal07:25:000
269297999385,6cyclictest2741888-1kworker/u9:012:04:153
2692979993827,7cyclictest2698402-21grep08:55:013
2692979993825,9cyclictest2727115-21sed10:10:023
2692979993824,10cyclictest2763845-21systemd-run11:30:013
2692979993820,14cyclictest275-21dbus-daemon12:25:003
269297999381,3cyclictest131rcu_preempt09:53:353
269297999381,3cyclictest131rcu_preempt09:53:353
2692975993830,5cyclictest2697129-21kworker/2:309:00:382
2692975993827,8cyclictest1-21systemd07:35:012
2692975993827,8cyclictest1-21systemd07:35:012
2692975993823,11cyclictest3467602-21systemd-journal08:15:002
269297099387,9cyclictest23-21ksoftirqd/109:02:121
269296599383,10cyclictest3467602-21systemd-journal09:05:010
2692965993824,10cyclictest2694961-21sshd07:45:140
269297999375,4cyclictest131rcu_preempt10:55:233
269297999375,4cyclictest131rcu_preempt10:55:223
2692979993728,7cyclictest37-21ksoftirqd/309:43:013
2692979993726,7cyclictest2699184-21modprobe09:10:003
2692979993726,7cyclictest2699184-21modprobe09:10:003
2692979993725,8cyclictest3467602-21systemd-journal12:10:123
269297599374,3cyclictest131rcu_preempt08:25:022
2692970993728,6cyclictest23-21ksoftirqd/108:15:001
2692965993727,8cyclictest12-21ksoftirqd/007:35:010
2692965993727,8cyclictest12-21ksoftirqd/007:35:010
2692965993726,7cyclictest12-21ksoftirqd/007:15:020
2692965993725,9cyclictest12-21ksoftirqd/008:45:000
2692965993725,9cyclictest12-21ksoftirqd/008:45:000
269297999365,4cyclictest131rcu_preempt09:31:093
2692979993626,7cyclictest37-21ksoftirqd/310:01:103
269297999362,4cyclictest131rcu_preempt09:21:583
2692979993624,9cyclictest37-21ksoftirqd/308:10:023
2692979993624,9cyclictest37-21ksoftirqd/308:10:023
2692979993620,11cyclictest275-21dbus-daemon08:00:013
2692979993617,13cyclictest275-21dbus-daemon08:05:013
269297599361,3cyclictest131rcu_preempt07:50:012
2692979993527,5cyclictest2697113-21latency_hist08:30:023
2692975993524,8cyclictest3467602-21systemd-journal08:51:132
269297099359,5cyclictest131rcu_preempt07:26:221
269296599351,31cyclictest0-21swapper/007:53:480
269297999344,3cyclictest131rcu_preempt07:10:013
2692965993317,11cyclictest2692971-21latency_hist07:10:010
2692979993217,10cyclictest275-21dbus-daemon07:55:003
269297599315,5cyclictest131rcu_preempt07:13:512
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional