You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-22 - 22:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Aug 22, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3352815998278,2cyclictest0-21swapper/011:55:020
3352819997532,39cyclictest3360774-21sshd09:13:551
3352815997126,19cyclictest3422548-21ssh11:25:080
3352819997032,33cyclictest309-21in:imuxsock12:20:001
3352821996840,9cyclictest3358004-21kworker/2:211:33:582
335281999681,4cyclictest3453592-21ssh12:31:431
335281999681,4cyclictest3453592-21ssh12:31:421
3352821996740,9cyclictest3377709-21kworker/2:112:03:032
3352821996740,9cyclictest3377709-21kworker/2:112:03:022
3352815996726,19cyclictest3385722-21ssh10:06:510
335281599661,3cyclictest131rcu_preempt11:24:430
3352821996546,9cyclictest3377709-21kworker/2:112:15:042
3352819996526,34cyclictest3467562-21sshd09:05:561
3352819996526,34cyclictest3467562-21sshd09:05:561
3352819996523,37cyclictest3441432-21sh12:05:431
3352819996424,36cyclictest3383180-21sh10:01:231
3352819996326,32cyclictest150-21pr/ttyS211:51:281
335281999631,17cyclictest3366714-21ssh09:26:191
335281999631,15cyclictest3408396-21cat10:55:031
335281999631,15cyclictest3408396-21cat10:55:021
3352815996359,2cyclictest0-21swapper/011:10:020
335281599631,41cyclictest131rcu_preempt12:39:550
3352821996254,5cyclictest30-21ksoftirqd/212:35:022
3352821996254,5cyclictest30-21ksoftirqd/212:35:012
3352821996253,5cyclictest30-21ksoftirqd/207:50:032
3352819996222,36cyclictest3397914-21sh10:32:431
3352819996220,37cyclictest3438278-1kworker/u9:112:24:091
3352819996220,37cyclictest3438278-1kworker/u9:112:24:081
3352819996213,4cyclictest0-21swapper/108:15:101
3352821996153,5cyclictest3358004-21kworker/2:209:48:552
3352821996136,8cyclictest3354380-21kworker/2:009:21:412
3352819996124,33cyclictest23-21ksoftirqd/112:13:471
3352819996123,34cyclictest3392838-21sshd10:21:561
3352819996123,34cyclictest3392838-21sshd10:21:551
3352819996118,9cyclictest3398649-1kworker/u9:110:57:081
3352819996118,9cyclictest3383866-1kworker/u9:210:08:031
3352819996020,35cyclictest3430293-21sh11:41:411
3352821995932,7cyclictest3377709-21kworker/2:110:06:402
3352819995923,31cyclictest23-21ksoftirqd/112:39:541
3352819995922,33cyclictest3390944-21sh10:18:021
3352819995921,33cyclictest3433744-21sh11:49:211
3352819995920,35cyclictest3404203-21sshd10:45:551
3352819995913,6cyclictest3467602-21systemd-journal09:00:011
3352819995913,40cyclictest0-21swapper/111:20:011
3352815995921,34cyclictest12-21ksoftirqd/009:17:440
3352815995919,31cyclictest16850irq/134-800000012:20:010
3352819995826,29cyclictest3427348-21sh11:35:171
3352819995822,32cyclictest3422684-21sh11:25:251
335281999582,16cyclictest3415303-21munin-run11:10:021
3352819995820,32cyclictest3362528-21sh09:17:291
3352815995824,17cyclictest12-21ksoftirqd/012:24:470
3352815995824,17cyclictest12-21ksoftirqd/012:24:470
3352815995819,30cyclictest16850irq/134-800000012:09:590
3352815995816,33cyclictest16850irq/134-800000011:35:030
3352819995721,30cyclictest3440951-21cron12:05:011
3352819995721,30cyclictest3440951-21cron12:05:001
3352819995714,12cyclictest1-21systemd09:55:001
3352819995711,6cyclictest3355909-21munin-plugin-st08:10:011
3352815995736,16cyclictest275-21dbus-daemon09:00:000
3352815995723,16cyclictest0-21swapper/010:21:500
3352815995723,16cyclictest0-21swapper/010:21:490
3352821995647,5cyclictest3377709-21kworker/2:112:37:202
3352821995646,5cyclictest3377709-21kworker/2:111:20:032
3352819995624,27cyclictest3370939-21sh09:35:141
3352819995623,28cyclictest3394487-21sh10:25:121
3352819995621,9cyclictest23-21ksoftirqd/111:04:271
3352819995620,32cyclictest3417627-21latency_hist11:15:021
3352819995616,5cyclictest3421175-21rm11:22:221
3352819995615,7cyclictest3355757-21grep08:05:011
3352815995628,12cyclictest3387693-21ssh10:10:500
3352815995627,13cyclictest3451314-21ssh12:26:560
3352821995547,5cyclictest3377709-21kworker/2:112:09:232
3352819995525,27cyclictest23-21ksoftirqd/108:38:541
3352819995525,26cyclictest23-21ksoftirqd/109:32:291
3352819995524,26cyclictest3377100-21sh09:48:421
335281999552,16cyclictest3437395-21ssh11:57:001
3352819995516,8cyclictest23-21ksoftirqd/109:44:311
3352819995514,4cyclictest3388582-21sh10:13:051
335281599558,31cyclictest3440961-21latency_hist12:05:030
335281599558,31cyclictest3440961-21latency_hist12:05:030
3352815995534,5cyclictest16850irq/134-800000010:55:020
3352815995534,5cyclictest16850irq/134-800000010:55:010
3352815995528,12cyclictest3374111-21ssh09:42:130
3352815995527,12cyclictest3366421-21ssh09:25:300
3352815995519,28cyclictest12-21ksoftirqd/010:47:410
3352815995513,32cyclictest16850irq/134-800000010:42:080
3352815995513,32cyclictest16850irq/134-800000010:42:080
3352821995444,5cyclictest3377709-21kworker/2:111:30:022
3352821995430,8cyclictest3377709-21kworker/2:111:59:542
335281999542,17cyclictest3365975-21cut09:24:591
3352819995410,7cyclictest3249657-1kworker/u9:010:00:031
3352815995428,8cyclictest12-21ksoftirqd/009:24:590
3352815995424,25cyclictest3397417-21ssh10:31:290
3352815995424,13cyclictest3382648-21ssh10:00:060
3352815995423,15cyclictest3416038-21ssh11:11:180
3352815995415,30cyclictest16850irq/134-800000009:30:210
3352815995414,32cyclictest16850irq/134-800000012:14:490
3352815995414,31cyclictest16850irq/134-800000011:45:490
3352815995414,31cyclictest16850irq/134-800000011:41:530
3352815995414,31cyclictest16850irq/134-800000010:27:290
3352815995413,31cyclictest16850irq/134-800000009:52:270
3352821995345,5cyclictest3377709-21kworker/2:110:50:322
3352821995345,5cyclictest3377709-21kworker/2:110:50:312
3352821995344,6cyclictest3377709-21kworker/2:111:50:162
3352821995344,6cyclictest3377709-21kworker/2:111:17:252
3352821995344,6cyclictest3358004-21kworker/2:209:59:472
3352821995344,6cyclictest3358004-21kworker/2:209:52:352
3352821995344,5cyclictest3377709-21kworker/2:112:21:072
3352821995344,5cyclictest3377709-21kworker/2:112:21:072
3352821995343,7cyclictest3377709-21kworker/2:111:45:132
3352821995343,6cyclictest3358004-21kworker/2:211:35:312
3352821995343,6cyclictest3354380-21kworker/2:009:25:342
3352819995322,27cyclictest23-21ksoftirqd/111:32:081
3352819995316,8cyclictest3356141-21sshd08:13:551
3352819995313,10cyclictest275-21dbus-daemon08:25:001
3352819995313,10cyclictest275-21dbus-daemon08:25:001
335281599535,4cyclictest12-21ksoftirqd/009:56:390
3352815995334,14cyclictest3391923-21date10:20:020
3352815995313,32cyclictest16850irq/134-800000012:31:110
3352815995313,32cyclictest16850irq/134-800000012:31:110
3352815995312,32cyclictest16850irq/134-800000009:11:180
3352825995232,14cyclictest3467602-21systemd-journal10:25:013
3352825995232,14cyclictest3467602-21systemd-journal10:25:013
3352821995244,5cyclictest3377709-21kworker/2:111:09:302
3352821995244,5cyclictest3358004-21kworker/2:210:20:422
3352821995244,5cyclictest3358004-21kworker/2:210:20:412
3352821995244,5cyclictest3354380-21kworker/2:009:15:502
3352821995244,5cyclictest3340132-21kworker/2:107:40:012
3352821995243,6cyclictest3358004-21kworker/2:210:40:192
3352821995243,6cyclictest3358004-21kworker/2:210:40:182
3352821995242,6cyclictest3377709-21kworker/2:110:02:022
3352821995241,6cyclictest3377709-21kworker/2:110:47:032
3352821995227,19cyclictest3467602-21systemd-journal08:29:592
3352819995243,7cyclictest23-21ksoftirqd/109:05:011
3352819995216,30cyclictest0-21swapper/107:30:021
3352819995216,30cyclictest0-21swapper/107:30:011
3352819995215,6cyclictest3403688-21sh10:45:041
3352815995212,32cyclictest16850irq/134-800000010:57:280
3352821995143,5cyclictest3377709-21kworker/2:111:13:582
3352821995143,5cyclictest3377709-21kworker/2:110:57:132
3352821995143,5cyclictest3358004-21kworker/2:210:35:292
3352821995143,5cyclictest3354380-21kworker/2:009:30:592
3352821995143,5cyclictest3340132-21kworker/2:107:45:032
3352821995143,5cyclictest3340132-21kworker/2:107:45:022
3352821995142,6cyclictest3358004-21kworker/2:210:31:232
3352821995142,6cyclictest3354380-21kworker/2:009:43:412
3352821995141,5cyclictest3377709-21kworker/2:112:29:012
3352821995138,9cyclictest3412873-21sshd11:04:562
3352819995115,9cyclictest23-21ksoftirqd/110:42:161
3352819995113,8cyclictest3394527-21kworker/u8:210:39:241
3352815995147,2cyclictest0-21swapper/010:35:020
3352815995126,10cyclictest3436732-21ssh11:55:260
3352815995112,31cyclictest16850irq/134-800000011:35:250
3352815995112,30cyclictest16850irq/134-800000009:35:110
3352815995111,31cyclictest16850irq/134-800000009:49:180
3352821995042,5cyclictest3340132-21kworker/2:108:16:592
3352821995041,6cyclictest3354380-21kworker/2:009:36:352
3352821995040,6cyclictest3377709-21kworker/2:112:11:302
3352821995040,6cyclictest3377709-21kworker/2:110:10:292
335281999502,17cyclictest3353603-21munin-run07:25:001
3352819995018,8cyclictest23-21ksoftirqd/112:25:421
3352819995014,4cyclictest3354794-21latency_hist07:45:031
3352819995014,4cyclictest3354794-21latency_hist07:45:021
3352815995012,30cyclictest16850irq/134-800000011:18:160
3352815995010,31cyclictest16850irq/134-800000011:00:320
3352825994933,13cyclictest37-21ksoftirqd/310:03:553
3352821994941,5cyclictest3377709-21kworker/2:110:26:562
3352821994941,5cyclictest3354380-21kworker/2:009:11:272
3352821994941,5cyclictest3354380-21kworker/2:008:53:552
3352821994940,6cyclictest3358004-21kworker/2:210:16:212
3352821994940,5cyclictest3377709-21kworker/2:111:40:502
3352821994934,11cyclictest1-21systemd09:10:012
3352821994934,11cyclictest1-21systemd09:10:002
335282599481,4cyclictest131rcu_preempt10:48:223
3352821994839,5cyclictest3340132-21kworker/2:108:15:032
3352819994814,9cyclictest23-21ksoftirqd/108:50:001
3352815994827,13cyclictest3467602-21systemd-journal09:05:000
3352821994720,4cyclictest30-21ksoftirqd/208:00:012
3352821994720,4cyclictest30-21ksoftirqd/208:00:002
3352819994724,19cyclictest3353236-21date07:15:031
3352819994718,8cyclictest23-21ksoftirqd/107:56:131
3352819994718,8cyclictest23-21ksoftirqd/107:56:121
3352825994634,9cyclictest1-21systemd12:00:023
335282599463,3cyclictest131rcu_preempt12:01:143
335282599463,3cyclictest131rcu_preempt12:01:133
3352821994638,5cyclictest3354380-21kworker/2:008:37:132
3352821994638,5cyclictest3340132-21kworker/2:107:30:142
3352821994638,5cyclictest3340132-21kworker/2:107:20:312
335281599462,7cyclictest3354612-21modprobe07:45:010
335281599462,7cyclictest3354612-21modprobe07:45:000
335282599453,3cyclictest131rcu_preempt10:20:003
3352825994529,12cyclictest275-21dbus-daemon12:35:013
3352825994529,12cyclictest275-21dbus-daemon12:35:003
3352825994528,12cyclictest275-21dbus-daemon11:45:013
3352825994527,13cyclictest3467602-21systemd-journal09:13:553
335282199459,6cyclictest131rcu_preempt07:12:552
3352821994527,13cyclictest1-21systemd07:50:002
335281599453,6cyclictest3467602-21systemd-journal07:31:090
3352815994536,5cyclictest3355088-21sshd07:50:140
3352815994527,14cyclictest275-21dbus-daemon07:40:000
3352825994433,7cyclictest3457386-21sendmail-msp12:40:003
3352825994430,10cyclictest275-21dbus-daemon07:30:013
3352825994430,10cyclictest275-21dbus-daemon07:30:013
3352825994428,10cyclictest37-21ksoftirqd/310:59:593
3352825994427,12cyclictest1-21systemd09:50:013
3352821994436,5cyclictest3340132-21kworker/2:108:07:112
3352821994436,5cyclictest3340132-21kworker/2:108:00:552
3352821994434,6cyclictest3354380-21kworker/2:008:45:012
3352821994422,5cyclictest30-21ksoftirqd/207:30:012
3352821994422,5cyclictest30-21ksoftirqd/207:30:002
3352815994431,7cyclictest3357460-21sh08:40:000
3352825994331,10cyclictest37-21ksoftirqd/309:24:523
3352825994327,12cyclictest37-21ksoftirqd/312:25:033
3352825994327,12cyclictest37-21ksoftirqd/312:25:023
3352825994327,12cyclictest3467602-21systemd-journal07:47:553
335282599431,7cyclictest0-21swapper/309:50:333
335282599431,3cyclictest131rcu_preempt09:57:273
3352821994335,5cyclictest3354380-21kworker/2:008:34:022
3352821994319,6cyclictest30-21ksoftirqd/208:55:272
335281999433,15cyclictest3467602-21systemd-journal08:55:001
335281999432,16cyclictest3354367-21sendmail-msp07:40:001
335281999431,16cyclictest3353337-21sendmail-msp07:19:591
335281599433,3cyclictest131rcu_preempt07:25:010
3352815994324,15cyclictest3355872-21sshd08:05:180
335281599432,36cyclictest3353295-21cron07:19:590
335282599427,8cyclictest37-21ksoftirqd/311:40:013
3352825994229,9cyclictest37-21ksoftirqd/309:20:013
3352825994228,10cyclictest3358505-21systemctl09:00:003
3352825994227,12cyclictest3467602-21systemd-journal08:45:003
335282599421,4cyclictest131rcu_preempt11:05:193
335281599424,3cyclictest131rcu_preempt07:10:020
335281599423,35cyclictest3355388-21systemctl08:00:000
335281599423,35cyclictest3355388-21systemctl08:00:000
335281599421,36cyclictest3359031-21diskmemload09:10:000
335281599421,36cyclictest3359031-21diskmemload09:10:000
335282599418,6cyclictest131rcu_preempt10:09:143
3352825994127,10cyclictest3445169-21sshd12:13:553
3352825994127,10cyclictest309-21in:imuxsock10:40:013
3352825994124,14cyclictest3467602-21systemd-journal07:35:023
335281999411,12cyclictest3467602-21systemd-journal08:35:001
3352815994128,10cyclictest12-21ksoftirqd/008:50:020
3352815994120,15cyclictest275-21dbus-daemon08:15:020
3352825994032,5cyclictest3356386-21cron08:17:003
3352825994031,5cyclictest3368291-21cron09:30:003
3352825994029,8cyclictest3358206-21sshd08:50:273
3352825994027,9cyclictest309-21in:imuxsock08:48:553
3352825994025,10cyclictest309-21in:imuxsock08:29:553
3352825994020,16cyclictest3408266-21ssh10:55:023
3352825994020,16cyclictest3408266-21ssh10:55:023
3352825994013,8cyclictest37-21ksoftirqd/312:26:153
3352819994028,9cyclictest23-21ksoftirqd/107:50:021
3352815994028,9cyclictest1-21systemd08:50:020
3352815994023,13cyclictest275-21dbus-daemon08:30:000
3352815994022,14cyclictest275-21dbus-daemon08:20:000
3352825993930,6cyclictest3352764-21latency_hist07:10:023
3352825993930,5cyclictest309-21in:imuxsock11:50:013
3352825993928,8cyclictest37-21ksoftirqd/309:35:493
3352825993927,8cyclictest37-21ksoftirqd/309:45:013
3352825993925,10cyclictest37-21ksoftirqd/311:50:533
3352825993925,10cyclictest309-21in:imuxsock11:05:013
3352825993922,13cyclictest37-21ksoftirqd/309:05:013
3352825993921,14cyclictest275-21dbus-daemon07:25:013
335282599391,3cyclictest131rcu_preempt11:19:063
335282599391,3cyclictest131rcu_preempt10:14:053
335282199391,5cyclictest131rcu_preempt08:25:032
335282199391,5cyclictest131rcu_preempt08:25:022
3352819993928,7cyclictest23-21ksoftirqd/107:35:011
335281599392,5cyclictest3355659-21munin-run08:05:000
335282599382,6cyclictest131rcu_preempt08:00:013
335282599382,6cyclictest131rcu_preempt08:00:003
3352825993826,10cyclictest37-21ksoftirqd/312:08:073
3352825993825,9cyclictest1-21systemd12:20:013
3352825993825,10cyclictest37-21ksoftirqd/309:30:203
3352825993823,11cyclictest37-21ksoftirqd/310:26:063
3352825993810,5cyclictest131rcu_preempt10:42:583
3352825993810,5cyclictest131rcu_preempt10:42:573
3352821993829,6cyclictest3340132-21kworker/2:107:15:122
335281599381,4cyclictest131rcu_preempt07:50:010
3352825993727,8cyclictest37-21ksoftirqd/311:30:123
3352825993727,8cyclictest37-21ksoftirqd/310:31:303
3352825993726,8cyclictest37-21ksoftirqd/311:15:013
3352825993725,8cyclictest37-21ksoftirqd/311:20:273
3352825993724,9cyclictest3467602-21systemd-journal08:03:323
3352825993724,10cyclictest37-21ksoftirqd/307:13:563
3352825993723,10cyclictest37-21ksoftirqd/309:10:013
3352825993723,10cyclictest37-21ksoftirqd/309:10:003
3352819993723,10cyclictest3467602-21systemd-journal08:45:021
3352815993729,5cyclictest19350irq/34-mmc108:32:040
335281599372,3cyclictest131rcu_preempt08:25:030
335281599372,3cyclictest131rcu_preempt08:25:020
335282599364,4cyclictest131rcu_preempt08:40:003
335282599362,4cyclictest131rcu_preempt07:19:593
3352825993621,9cyclictest275-21dbus-daemon08:35:003
3352821993629,4cyclictest3354380-21kworker/2:008:48:412
3352819993628,6cyclictest23-21ksoftirqd/107:55:001
3352819993614,7cyclictest3356933-21sshd08:28:551
335282599354,4cyclictest131rcu_preempt07:35:013
335282599353,6cyclictest37-21ksoftirqd/311:27:403
3352825993525,7cyclictest3355101-21sshd07:52:553
3352821993527,5cyclictest3354380-21kworker/2:009:00:282
335281599351,5cyclictest3353025-21sshd07:13:550
335281599351,30cyclictest0-21swapper/007:27:330
335281599351,30cyclictest0-21swapper/007:27:320
3352825993422,8cyclictest3467602-21systemd-journal08:10:003
335281599341,30cyclictest0-21swapper/008:40:580
3352825993323,7cyclictest37-21ksoftirqd/308:10:243
335282599313,3cyclictest131rcu_preempt08:25:013
335282599313,3cyclictest131rcu_preempt08:25:013
3352825993022,5cyclictest3354618-21latency_hist07:45:033
3352825993022,5cyclictest3354618-21latency_hist07:45:023
335282199307,14cyclictest3352894-21latency_hist07:10:012
3352819992922,5cyclictest3352764-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional