You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-25 - 09:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat Jul 25, 2026 00:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
271355599941,4cyclictest131rcu_preempt00:40:012
2713547999234,54cyclictest16850irq/134-800000021:35:010
271354799921,63cyclictest0-21swapper/000:18:000
271354799901,55cyclictest0-21swapper/023:40:030
2713547998928,57cyclictest2813915-21ssh00:28:530
271354799892,53cyclictest0-21swapper/022:51:490
2713550998522,33cyclictest2782993-21sh23:23:411
2713550998321,32cyclictest2720081-21sh21:10:151
2713547998322,32cyclictest2746975-21ssh22:07:100
271355599821,68cyclictest2786050-21rm23:30:022
271354799791,57cyclictest0-21swapper/023:31:040
2713547997915,46cyclictest2781064-21ssh23:19:520
271354799782,70cyclictest3467602-21systemd-journal21:57:060
271354799781,73cyclictest2745160-21ssh22:03:290
271354799781,60cyclictest0-21swapper/021:37:310
2713547997621,38cyclictest2734591-21ssh21:41:010
2713547997615,56cyclictest16850irq/134-800000022:59:500
2713547997614,27cyclictest16850irq/134-800000022:17:390
271355099756,39cyclictest2800910-21ssh00:01:111
2713547997527,43cyclictest2758913-21ssh22:32:330
2713547997527,43cyclictest2758913-21ssh22:32:330
271354799752,67cyclictest2793747-21ssh23:46:020
2713547997430,27cyclictest2782369-21ssh23:22:150
271354799741,71cyclictest0-21swapper/022:45:330
271354799741,69cyclictest2793050-21/usr/sbin/munin23:45:010
271354799741,55cyclictest0-21swapper/000:37:550
2713547997328,40cyclictest309-21in:imuxsock23:50:020
271354799732,67cyclictest2723188-21ssh21:17:090
2713547997323,33cyclictest2737417-21ssh21:47:010
271354799731,55cyclictest2772508-21ssh23:01:220
271355099721,42cyclictest2753552-21ssh22:21:021
271355099721,42cyclictest2753552-21ssh22:21:011
271355099712,38cyclictest2796513-21ssh23:51:551
271354799712,64cyclictest2777894-21ssh23:13:060
271354799711,66cyclictest0-21swapper/021:52:140
2713550997029,37cyclictest2779376-21sh23:15:541
2713547997030,23cyclictest2803680-21ssh00:07:110
2713547997029,26cyclictest2727767-21ssh21:26:480
2713547997029,26cyclictest2727767-21ssh21:26:470
271354799701,65cyclictest0-21swapper/023:55:180
271354799701,65cyclictest0-21swapper/022:24:290
271354799701,65cyclictest0-21swapper/022:24:290
271355099691,38cyclictest2813318-21sh00:27:291
271355099691,35cyclictest2789682-21ssh23:37:411
271354799692,63cyclictest2816776-21munin-run00:35:010
271355099683,33cyclictest2739552-21ssh21:51:271
2713547996827,23cyclictest2805735-21ssh00:11:220
2713547996818,46cyclictest16850irq/134-800000021:14:270
271354799681,63cyclictest0-21swapper/000:05:020
271354799681,61cyclictest0-21swapper/023:10:030
271354799681,34cyclictest2783843-21ssh23:25:130
2713550996726,37cyclictest2774323-21sh23:05:121
2713550996726,37cyclictest2722758-21sh21:16:061
2713550996726,36cyclictest2771638-21sh22:59:551
271354799671,62cyclictest0-21swapper/022:45:010
271355099663,29cyclictest2768214-21ssh22:52:181
2713550996628,34cyclictest2742588-21sh21:58:001
271355099662,31cyclictest2728862-21ssh21:29:131
271355099662,31cyclictest2728862-21ssh21:29:121
271355099661,33cyclictest2784982-21ssh23:27:511
2713547996623,39cyclictest2724722-21ssh21:20:120
2713547996621,40cyclictest16850irq/134-800000022:25:280
2713547996619,43cyclictest2811107-21ssh00:22:550
2713547996619,43cyclictest2811107-21ssh00:22:550
2713550996525,35cyclictest2810781-21sh00:22:051
2713550996525,35cyclictest2810781-21sh00:22:041
2713550996524,37cyclictest2805447-21sh00:10:461
2713550996517,44cyclictest364-21snmpd20:55:351
271355099651,33cyclictest2719724-21diskmemload22:30:081
271355099651,33cyclictest2719724-21diskmemload22:30:071
271355099651,31cyclictest2738253-21ssh21:49:031
271354799651,59cyclictest2749946-21ssh22:13:380
271355099643,4cyclictest131rcu_preempt23:50:021
271355099642,57cyclictest2751857-21ssh22:17:401
271354799643,55cyclictest2714780-21munin-run19:35:000
2713547996425,36cyclictest2717964-21latency_hist20:35:020
271354799641,33cyclictest0-21swapper/022:38:320
2713555996315,11cyclictest30-21ksoftirqd/200:19:052
2713550996326,33cyclictest2804433-21sh00:08:561
2713550996323,36cyclictest2725653-21sh21:22:201
2713550996322,36cyclictest2777314-21sh23:11:441
271355099631,59cyclictest2747410-21ssh22:08:151
271355099622,34cyclictest2772997-21sh23:02:331
271355099621,57cyclictest2815014-21ssh00:31:031
271355099621,43cyclictest2719724-21diskmemload22:35:351
271354799621,57cyclictest0-21swapper/020:55:010
271354799621,57cyclictest0-21swapper/019:10:010
2713557996138,12cyclictest2719724-21diskmemload22:55:013
2713557996131,26cyclictest37-21ksoftirqd/321:35:023
271354799611,56cyclictest0-21swapper/020:01:260
271355099603,51cyclictest2714295-21wc19:25:011
2713550996020,36cyclictest2735413-21sh21:42:571
271355099601,55cyclictest2807866-21ssh00:15:521
271355099601,55cyclictest2732440-21ssh21:36:371
271355099601,55cyclictest2729777-21ssh21:30:491
271355099601,31cyclictest2719724-21diskmemload22:29:131
2713555995915,10cyclictest30-21ksoftirqd/222:08:082
271355099591,54cyclictest2766379-21ssh22:48:301
271355099581,53cyclictest2792792-21ssh23:44:191
271355099581,53cyclictest2763664-21ssh22:42:341
271354799582,28cyclictest1-21systemd19:46:270
2713555995716,10cyclictest30-21ksoftirqd/223:14:262
2713555995713,9cyclictest30-21ksoftirqd/223:24:022
271355099572,50cyclictest2716856-21munin-run20:15:011
271355099571,52cyclictest2817877-21ssh00:37:031
271355099571,52cyclictest2744459-21ssh22:01:551
271354799573,49cyclictest2719471-21munin-run21:05:010
271354799573,49cyclictest2719471-21munin-run21:05:000
271354799571,51cyclictest2716585-21sshd20:09:280
271354799571,51cyclictest12-21ksoftirqd/019:40:010
2713555995618,12cyclictest30-21ksoftirqd/222:42:382
2713555995614,9cyclictest30-21ksoftirqd/222:00:222
271355099562,50cyclictest364-21snmpd19:41:411
2713550995619,31cyclictest275-21dbus-daemon21:05:021
2713550995619,31cyclictest275-21dbus-daemon21:05:011
271355099561,51cyclictest364-21snmpd19:11:351
271355099561,51cyclictest2798114-21ssh23:55:081
271355099561,51cyclictest2786985-21ssh23:32:021
271355099561,51cyclictest2749765-21ssh22:13:151
271354799563,47cyclictest2715827-21unin-run19:55:010
271354799561,51cyclictest0-21swapper/021:05:290
2713557995541,11cyclictest2733932-21systemctl21:40:003
2713555995514,10cyclictest30-21ksoftirqd/200:34:012
2713555995512,9cyclictest30-21ksoftirqd/200:03:162
271355099553,46cyclictest2715195-21cat19:40:011
271355099551,49cyclictest364-21snmpd20:54:261
271354799552,48cyclictest2718696-21munin-run20:50:020
271354799552,48cyclictest2718696-21munin-run20:50:010
271354799552,44cyclictest2718442-21systemd-run20:45:010
2713547995520,31cyclictest12-21ksoftirqd/020:20:020
2713547995519,31cyclictest1-21systemd19:30:010
2713547995510,40cyclictest2713995-21cron19:20:000
2713557995414,6cyclictest3467718-21lldpd21:13:513
2713555995413,8cyclictest30-21ksoftirqd/223:47:032
2713550995428,21cyclictest23-21ksoftirqd/121:10:001
271355099542,48cyclictest2714750-21sshd19:31:451
271354799541,48cyclictest0-21swapper/019:45:020
2713555995344,6cyclictest2712734-21kworker/2:021:25:002
2713555995312,11cyclictest30-21ksoftirqd/221:14:042
2713555995311,11cyclictest30-21ksoftirqd/221:19:142
271355099532,47cyclictest3467602-21systemd-journal20:42:521
271355099532,47cyclictest2715781-21sshd19:51:261
271355099532,47cyclictest2714002-21munin-run19:20:001
271355099532,46cyclictest364-21snmpd19:28:121
271355099531,48cyclictest2718125-21sshd20:37:291
271354799532,46cyclictest364-21snmpd20:24:200
271354799531,48cyclictest0-21swapper/021:00:010
2713555995214,9cyclictest30-21ksoftirqd/200:11:492
2713555995213,9cyclictest30-21ksoftirqd/221:56:382
271355099523,44cyclictest2717647-21munin-run20:30:011
271355099521,46cyclictest364-21snmpd20:33:191
271355099521,46cyclictest364-21snmpd20:24:161
271355099521,46cyclictest364-21snmpd19:57:491
2713547995232,15cyclictest1-21systemd19:15:010
2713547995231,18cyclictest2714453-21latency_hist19:25:020
271354799521,46cyclictest2716967-21latency_hist20:15:010
271354799512,44cyclictest2717660-21munin-run20:30:010
2713555995019,8cyclictest30-21ksoftirqd/222:35:372
2713555995012,7cyclictest30-21ksoftirqd/222:17:432
271355099502,44cyclictest2716582-21sshd20:08:271
271355099501,45cyclictest364-21snmpd19:47:161
271354799505,39cyclictest2716240-21latency_hist20:00:030
271354799501,46cyclictest364-21snmpd20:35:260
2713557994932,12cyclictest1-21systemd20:30:013
271355599494,14cyclictest2719724-21diskmemload22:34:372
271355599494,14cyclictest2719724-21diskmemload22:34:362
2713555994913,7cyclictest30-21ksoftirqd/200:26:482
271355099492,43cyclictest2718683-21munin-run20:50:011
271355099492,43cyclictest2718683-21munin-run20:50:001
2713557994830,10cyclictest309-21in:imuxsock22:25:013
2713557994830,10cyclictest309-21in:imuxsock22:25:003
2713557994826,13cyclictest1-21systemd00:00:003
2713555994840,5cyclictest2712734-21kworker/2:021:27:122
2713555994840,5cyclictest2712734-21kworker/2:021:27:112
2713555994829,15cyclictest3467602-21systemd-journal22:30:002
271355099482,43cyclictest364-21snmpd20:01:461
271355099481,43cyclictest2717156-21unin-run20:20:011
2713557994734,10cyclictest3467602-21systemd-journal21:40:283
2713557994728,14cyclictest3467602-21systemd-journal20:15:013
2713555994740,4cyclictest2716615-21kworker/2:221:40:122
2713555994738,5cyclictest2716615-21kworker/2:221:50:022
271355799464,3cyclictest131rcu_preempt23:28:233
2713557994629,14cyclictest275-21dbus-daemon21:10:013
2713557994629,12cyclictest1-21systemd23:05:013
2713557994629,10cyclictest3467602-21systemd-journal23:25:003
2713557994626,14cyclictest275-21dbus-daemon00:15:013
2713557994624,17cyclictest275-21dbus-daemon21:25:023
2713555994637,5cyclictest2778747-21kworker/2:000:22:192
2713555994637,5cyclictest2778747-21kworker/2:000:22:182
271355799452,3cyclictest131rcu_preempt23:40:383
271355799451,4cyclictest131rcu_preempt23:39:403
2713555994534,8cyclictest30-21ksoftirqd/222:55:022
2713555994510,8cyclictest30-21ksoftirqd/223:06:372
271354799452,37cyclictest2713653-21latency_hist19:10:010
2713557994430,11cyclictest37-21ksoftirqd/322:10:203
2713557994427,11cyclictest309-21in:imuxsock20:00:013
2713557994424,15cyclictest2797834-21cron23:55:003
2713555994436,5cyclictest2712734-21kworker/2:021:35:472
2713555994430,10cyclictest2790715-21munin-run23:40:022
2713555994427,13cyclictest1-21systemd22:25:012
2713555994427,13cyclictest1-21systemd22:25:002
2713550994435,7cyclictest0-21swapper/119:10:021
2713550994435,7cyclictest0-21swapper/119:10:011
2713557994331,8cyclictest2719724-21diskmemload00:25:013
2713557994331,8cyclictest2719724-21diskmemload00:25:013
2713557994331,8cyclictest2718139-21cron20:40:013
2713557994329,10cyclictest309-21in:imuxsock00:35:013
2713557994328,12cyclictest309-21in:imuxsock21:50:013
2713557994317,5cyclictest37-21ksoftirqd/300:29:373
271355599439,5cyclictest131rcu_preempt23:58:122
2713555994327,11cyclictest1-21systemd20:25:002
2713555994324,12cyclictest2714260-21cron19:25:002
2713555994310,8cyclictest0-21swapper/200:08:512
2713555994310,7cyclictest30-21ksoftirqd/221:32:572
2713555994310,7cyclictest0-21swapper/223:16:362
2713555994233,6cyclictest2738742-21kworker/2:322:57:192
2713555994233,4cyclictest309-21in:imuxsock19:45:002
2713555994227,11cyclictest3467602-21systemd-journal19:50:272
2713555994222,14cyclictest275-21dbus-daemon19:35:002
2713555994211,7cyclictest0-21swapper/222:45:272
271355799413,4cyclictest131rcu_preempt23:31:023
2713557994131,8cyclictest0-21swapper/323:20:023
271355799412,4cyclictest131rcu_preempt22:18:213
2713557994124,12cyclictest3467602-21systemd-journal22:40:013
271355799412,3cyclictest131rcu_preempt21:16:263
2713557994122,14cyclictest275-21dbus-daemon19:55:003
2713557994120,15cyclictest275-21dbus-daemon19:30:003
2713557994119,12cyclictest0-21swapper/320:20:003
271355799411,3cyclictest131rcu_preempt22:45:003
271355799411,12cyclictest3467602-21systemd-journal19:20:013
2713555994129,9cyclictest30-21ksoftirqd/219:40:002
2713555994129,9cyclictest2716067-21cron20:00:002
2713555994129,10cyclictest30-21ksoftirqd/223:03:432
2713557994032,5cyclictest2803431-21sshd00:06:273
2713557994029,6cyclictest2729173-21munin-plugin-st21:30:003
2713557994029,6cyclictest2729173-21munin-plugin-st21:30:003
2713557994028,8cyclictest3467602-21systemd-journal19:31:453
2713557994027,10cyclictest37-21ksoftirqd/320:50:023
2713557994027,10cyclictest37-21ksoftirqd/320:50:023
2713557994026,11cyclictest37-21ksoftirqd/321:51:493
2713557994022,13cyclictest275-21dbus-daemon21:00:013
271355799401,3cyclictest131rcu_preempt00:19:063
271355599409,5cyclictest0-21swapper/222:12:392
2713555994032,5cyclictest2738742-21kworker/2:323:54:532
2713555994028,8cyclictest2788311-21cron23:35:002
2713555994027,9cyclictest3467602-21systemd-journal20:05:012
271355799394,3cyclictest131rcu_preempt23:15:003
271355799394,3cyclictest131rcu_preempt22:33:473
271355799394,3cyclictest131rcu_preempt22:33:463
2713557993930,7cyclictest37-21ksoftirqd/300:01:173
271355799392,3cyclictest131rcu_preempt22:01:113
2713557993923,9cyclictest2715018-21cron19:39:593
2713557993916,17cyclictest275-21dbus-daemon19:15:003
2713557993910,22cyclictest2771681-21sendmail23:00:013
2713555993932,5cyclictest2778747-21kworker/2:023:41:172
2713555993927,9cyclictest2741117-21systemctl21:55:002
2713555993924,11cyclictest1-21systemd20:45:012
2713555993912,23cyclictest2713862-21latency_hist19:15:022
271355799387,5cyclictest131rcu_preempt22:05:383
2713557993829,6cyclictest2765511-21sh22:46:203
2713557993824,11cyclictest2716762-21sed20:10:013
2713557993824,10cyclictest3467602-21systemd-journal23:48:293
2713557993823,11cyclictest37-21ksoftirqd/322:26:013
2713557993820,11cyclictest309-21in:imuxsock20:30:023
2713555993824,11cyclictest3467602-21systemd-journal19:47:272
2713555993817,16cyclictest2717141-21sendmail-msp20:20:012
2713555993810,23cyclictest2716629-21latency_hist20:10:022
2713557993724,9cyclictest2719457-21cron21:05:013
2713557993724,9cyclictest2719457-21cron21:05:003
2713557993723,11cyclictest37-21ksoftirqd/319:25:023
2713557993720,13cyclictest2743413-21tr22:00:003
271355799371,6cyclictest0-21swapper/300:39:253
2713555993726,9cyclictest30-21ksoftirqd/220:30:012
2713557993630,3cyclictest371ksoftirqd/320:05:013
271355799362,4cyclictest131rcu_preempt23:06:203
2713555993627,6cyclictest30-21ksoftirqd/221:00:022
2713555993626,6cyclictest2713981-21cron19:20:002
2713555993624,10cyclictest30-21ksoftirqd/221:10:002
2713555993619,13cyclictest1-21systemd20:50:012
2713555993619,13cyclictest1-21systemd20:50:002
2713555993523,9cyclictest2719653-21cat21:05:032
2713555993523,9cyclictest2719653-21cat21:05:022
2713555993523,9cyclictest1-21systemd20:15:002
2713555993523,10cyclictest30-21ksoftirqd/220:55:022
2713555993512,19cyclictest2713648-21latency_hist19:10:012
2713555993512,19cyclictest2713648-21latency_hist19:10:012
2713557993425,5cyclictest309-21in:imuxsock20:54:593
2713557993326,4cyclictest2715674-21latency_hist19:50:023
2713555993324,5cyclictest2718151-21sh20:40:002
2713555993321,9cyclictest30-21ksoftirqd/220:30:032
2713557993123,5cyclictest2717391-21latency_hist20:25:033
2713557993123,5cyclictest2715311-21/usr/sbin/munin19:45:013
2713555993116,12cyclictest2714500-21sshd19:27:272
271355799303,21cyclictest2718420-21modprobe20:45:003
2713557992922,4cyclictest2713617-21latency_hist19:10:023
2713557992922,4cyclictest2713617-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional