You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-09 - 10:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat May 09, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2929281999411,4cyclictest131rcu_preempt19:25:023
2929272997825,20cyclictest2974562-21ssh22:31:061
2929272997624,15cyclictest2993041-21ssh23:09:481
2929272997528,18cyclictest2981057-21munin-run22:45:011
2929272997422,15cyclictest3002783-21munin-run23:30:011
2929272997325,16cyclictest2995957-21ssh23:15:241
2929272997224,15cyclictest2937655-21rm21:14:511
2929272997223,15cyclictest2971167-21rm22:24:241
2929272997128,7cyclictest2964592-21ssh22:10:241
2929272997128,7cyclictest2964592-21ssh22:10:241
2929272997122,13cyclictest3018824-21sh00:03:131
2929272997119,14cyclictest3021203-21ssh00:08:021
2929272997030,18cyclictest2984056-21ssh22:50:501
2929272997023,14cyclictest3036141-21ssh00:39:251
2929272997020,14cyclictest48-21kcompactd021:52:091
2929272997020,14cyclictest48-21kcompactd021:52:091
2929281996932,32cyclictest1-21systemd20:05:003
2929281996914,6cyclictest131rcu_preempt20:15:023
2929281996914,6cyclictest131rcu_preempt20:15:023
2929276996919,32cyclictest3467602-21systemd-journal21:50:012
2929272996933,14cyclictest3029791-21ssh00:25:421
292927699676,12cyclictest2969021-21modprobe22:20:002
292927699675,17cyclictest3017176-40mandb00:00:012
292927699674,14cyclictest3007763-21cat23:40:022
2929276996725,6cyclictest3008396-21sh23:41:212
2929276996725,6cyclictest2969297-21sh22:20:032
2929276996724,10cyclictest275-21dbus-daemon22:30:012
2929272996731,13cyclictest2938119-21ssh21:15:201
2929272996722,15cyclictest3003870-21ssh23:32:071
292928199662,17cyclictest2997938-21ssh23:20:013
2929276996629,12cyclictest30-21ksoftirqd/219:20:012
2929272996654,9cyclictest2973333-21ssh22:28:491
2929272996649,14cyclictest2990120-21ssh23:03:481
2929272996649,14cyclictest2963622-21ssh22:08:421
2929272996646,16cyclictest2932827-21cron20:20:001
292928199651,13cyclictest3024460-21/usr/sbin/munin00:15:013
292928199651,11cyclictest2939674-21ssh21:18:583
2929272996552,10cyclictest2948149-21ssh21:36:201
2929272996520,13cyclictest3032776-21ssh00:32:061
2929276996427,5cyclictest2966656-21sh22:15:022
2929276996427,5cyclictest2966656-21sh22:15:012
2929276996425,8cyclictest30-21ksoftirqd/221:19:202
2929272996449,12cyclictest2994849-21ssh23:13:251
2929272996449,12cyclictest2945979-21ssh21:31:471
2929272996447,14cyclictest2943481-21ssh21:26:321
2929272996425,8cyclictest2951426-21ssh21:43:191
2929272996425,8cyclictest2951426-21ssh21:43:191
2929272996422,12cyclictest2935517-21latency_hist21:10:021
2929281996331,27cyclictest3000607-21sshd23:25:143
292928199632,18cyclictest2985882-21wc22:55:023
292928199631,13cyclictest2993767-21ssh23:11:003
292927699633,25cyclictest2981223-21latency_hist22:45:022
2929276996310,17cyclictest2932698-21latency_hist20:15:022
2929276996310,17cyclictest2932698-21latency_hist20:15:022
2929272996350,10cyclictest2960672-21ssh22:02:201
2929272996349,11cyclictest3022674-21ssh00:11:031
2929272996349,11cyclictest2957648-21ssh21:56:031
2929272996345,14cyclictest3006481-21ssh23:37:371
2929267996329,30cyclictest3015012-21latency_hist23:55:030
2929276996231,5cyclictest2933914-21kworker/2:121:41:512
2929276996231,5cyclictest2933914-21kworker/2:121:41:502
2929272996249,10cyclictest3009281-21ssh23:43:171
2929272996248,11cyclictest3014233-21ssh23:53:431
2929272996248,11cyclictest3014233-21ssh23:53:431
2929272996246,13cyclictest2977432-21ssh22:37:191
2929272996243,15cyclictest3026269-21ssh00:18:391
2929272996225,6cyclictest2941286-21ssh21:22:031
2929272996225,6cyclictest2941286-21ssh21:22:031
292928199613,7cyclictest364-21snmpd22:00:153
2929281996132,25cyclictest2980430-21sh22:43:343
292928199612,18cyclictest2966652-21munin-run22:15:013
292928199612,18cyclictest2966652-21munin-run22:15:013
292928199611,13cyclictest2976277-21munin-run22:35:023
2929276996126,6cyclictest2987526-21sh22:58:232
2929276996125,11cyclictest30-21ksoftirqd/222:08:452
2929276996120,11cyclictest30-21ksoftirqd/223:04:522
2929276996112,13cyclictest275-21dbus-daemon21:15:012
2929272996144,14cyclictest3015148-21ssh23:55:171
2929272996144,14cyclictest3010606-21ssh23:45:511
2929272996144,14cyclictest2966903-21ssh22:15:071
2929272996144,13cyclictest3028163-21ssh00:22:281
2929272996144,13cyclictest2953032-21ssh21:46:221
2929272996131,7cyclictest2981624-21ssh22:45:451
292926799617,5cyclictest131rcu_preempt21:20:000
292928199609,15cyclictest275-21dbus-daemon19:35:013
292928199601,12cyclictest2962505-21ssh22:06:043
292928199601,10cyclictest3026173-21rm00:18:273
2929272996044,13cyclictest2986115-21ssh22:55:071
2929272996044,12cyclictest2999665-21ssh23:23:351
2929272996042,14cyclictest2931521-21sshd19:53:141
292926799606,10cyclictest12-21ksoftirqd/021:30:000
292928199592,9cyclictest2931008-21sshd19:41:143
2929281995920,34cyclictest2935447-21diskmemload00:38:323
2929281995918,36cyclictest3017320-21sh00:00:023
2929276995925,13cyclictest30-21ksoftirqd/222:05:012
2929267995939,4cyclictest16850irq/134-800000023:20:000
2929267995930,14cyclictest2960272-21ssh22:01:290
2929267995928,15cyclictest2995579-21ssh23:15:010
2929267995927,16cyclictest3013897-21ssh23:53:010
2929267995913,14cyclictest12-21ksoftirqd/000:40:000
292928199583,14cyclictest3005286-21wc23:35:023
2929281995823,30cyclictest2943029-21sh21:25:353
292928199582,15cyclictest3012410-21latency_hist23:50:013
292928199582,14cyclictest2933114-21sh20:25:003
292928199582,14cyclictest2933114-21sh20:25:003
292928199582,14cyclictest2929747-21tr19:20:013
292928199582,13cyclictest3007629-21munin-run23:40:013
292928199581,15cyclictest2942558-21ssh21:25:003
292928199581,15cyclictest2942558-21ssh21:25:003
292928199581,11cyclictest364-21snmpd23:22:513
292927699585,14cyclictest3026834-21cron00:20:012
2929272995846,9cyclictest2933358-21cron20:30:001
2929272995841,14cyclictest364-21snmpd20:36:401
2929267995848,8cyclictest12-21ksoftirqd/021:57:500
292928199574,8cyclictest2933610-21cron20:35:003
2929281995733,20cyclictest37-21ksoftirqd/320:30:003
292928199572,8cyclictest3019686-21latency_hist00:05:023
292928199572,13cyclictest2947849-21ssh21:35:333
292928199571,12cyclictest364-21snmpd19:11:043
292928199571,11cyclictest48-21kcompactd023:43:303
292928199571,11cyclictest364-21snmpd21:01:503
292928199571,11cyclictest2957702-21ssh21:56:103
292928199571,10cyclictest3467602-21systemd-journal22:59:243
2929267995733,19cyclictest275-21dbus-daemon22:50:000
2929267995728,13cyclictest2950745-21ssh21:41:420
2929267995728,13cyclictest2950745-21ssh21:41:420
2929267995721,10cyclictest12-21ksoftirqd/023:01:090
292928199562,9cyclictest2932314-21cron20:10:003
2929281995623,29cyclictest2978196-21sh22:39:043
292928199562,13cyclictest364-21snmpd22:26:373
292928199562,10cyclictest48-21kcompactd000:23:383
2929281995614,35cyclictest309-21in:imuxsock20:55:013
292928199561,18cyclictest2935122-21sort21:00:033
292928199561,12cyclictest364-21snmpd23:09:323
292928199561,10cyclictest48-21kcompactd021:30:253
2929276995640,11cyclictest1-21systemd22:55:012
292927699563,21cyclictest3467602-21systemd-journal19:55:002
2929276995623,9cyclictest30-21ksoftirqd/200:35:132
2929276995623,11cyclictest2976346-21cat22:35:022
2929272995642,11cyclictest2934633-21sshd20:52:141
2929272995640,12cyclictest2934136-21sshd20:43:141
2929272995640,11cyclictest48-21kcompactd019:14:241
2929267995629,12cyclictest2980488-21ssh22:43:430
292928199554,10cyclictest3467602-21systemd-journal21:50:003
292928199553,9cyclictest2955030-21ssh21:50:353
292928199553,9cyclictest2955030-21ssh21:50:353
2929281995525,26cyclictest2989233-21sh23:01:493
292928199552,11cyclictest3467602-21systemd-journal00:32:593
2929276995515,5cyclictest131rcu_preempt21:36:542
2929272995543,9cyclictest2932042-21sshd20:01:141
2929272995538,13cyclictest2930476-21sshd19:31:141
2929272995528,21cyclictest2935078-21cat21:00:021
292926799559,4cyclictest131rcu_preempt22:30:010
2929267995528,12cyclictest3024205-21ssh00:14:310
2929267995515,35cyclictest12-21ksoftirqd/000:19:370
2929281995422,28cyclictest37-21ksoftirqd/319:45:283
292928199542,15cyclictest2969054-21sendmail-msp22:20:003
292928199541,15cyclictest2937748-21/usr/sbin/munin21:15:003
292928199541,14cyclictest3467602-21systemd-journal00:25:043
292928199541,13cyclictest2951406-21ssh21:43:163
292928199541,13cyclictest2951406-21ssh21:43:163
2929272995442,9cyclictest2935169-21sshd21:03:141
2929272995441,9cyclictest364-21snmpd19:39:261
2929272995440,11cyclictest3467602-21systemd-journal19:23:431
2929267995427,13cyclictest2992148-21ssh23:07:470
2929267995426,13cyclictest2936519-21ssh21:12:070
2929267995424,14cyclictest2986427-21ssh22:55:460
2929281995322,28cyclictest37-21ksoftirqd/320:45:003
292928199532,11cyclictest3020446-21sh00:06:193
292928199531,9cyclictest2969384-21sshd22:20:143
292928199531,11cyclictest48-21kcompactd022:47:503
2929276995344,6cyclictest2930266-21kworker/2:220:05:012
2929272995345,4cyclictest2932578-21munin-run20:15:001
2929272995345,4cyclictest2932578-21munin-run20:15:001
2929272995341,9cyclictest48-21kcompactd019:44:541
2929272995340,10cyclictest364-21snmpd20:31:391
2929272995340,10cyclictest364-21snmpd20:06:341
2929272995340,10cyclictest2930218-21sshd19:25:141
2929267995327,11cyclictest2968047-21ssh22:17:510
2929267995325,13cyclictest3009343-21ssh23:43:250
2929267995317,32cyclictest2957171-21sed21:55:020
2929267995317,32cyclictest2957171-21sed21:55:020
2929267995312,5cyclictest131rcu_preempt23:45:120
2929267995311,12cyclictest12-21ksoftirqd/023:35:010
292928199522,10cyclictest2931510-21sshd19:51:143
292928199521,13cyclictest3014434-21ssh23:54:083
292928199521,13cyclictest3014434-21ssh23:54:083
2929276995227,5cyclictest2992331-21sh23:08:102
2929272995240,9cyclictest2931775-21sshd19:59:141
2929272995240,9cyclictest2931775-21sshd19:59:141
2929267995222,15cyclictest3017797-21ssh00:00:450
2929267995215,6cyclictest131rcu_preempt21:35:000
292928199511,12cyclictest2930423-21latency_hist19:30:013
2929276995143,5cyclictest2955239-21kworker/2:022:50:022
2929276995143,5cyclictest2935202-21kworker/2:200:05:152
2929272995143,4cyclictest2933120-21munin-run20:25:011
2929272995143,4cyclictest2933120-21munin-run20:25:011
292926799514,9cyclictest12-21ksoftirqd/019:50:020
2929267995138,11cyclictest12-21ksoftirqd/022:37:370
2929267995132,14cyclictest3467602-21systemd-journal20:35:000
2929267995126,11cyclictest3033014-21ssh00:32:360
2929267995125,11cyclictest2984441-21ssh22:51:470
2929267995120,28cyclictest0-21swapper/021:21:190
2929267995120,28cyclictest0-21swapper/021:21:180
2929267995114,33cyclictest19350irq/34-mmc123:30:020
292928199503,21cyclictest2935453-21munin-run21:10:003
292928199501,10cyclictest364-21snmpd19:36:293
2929276995042,5cyclictest2935202-21kworker/2:223:49:262
2929276995042,5cyclictest2935202-21kworker/2:223:32:332
2929276995042,5cyclictest2935202-21kworker/2:200:26:172
2929276995042,5cyclictest2935202-21kworker/2:200:12:272
2929276995033,12cyclictest275-21dbus-daemon00:05:012
2929272995042,4cyclictest2929734-21munin-run19:20:011
2929272995038,9cyclictest2931253-21sshd19:45:141
292926799507,5cyclictest131rcu_preempt21:39:130
292926799505,7cyclictest12-21ksoftirqd/023:22:490
2929267995041,6cyclictest16950irq/224-800000000:30:000
2929267995041,5cyclictest19350irq/34-mmc123:59:590
2929267995023,12cyclictest3006703-21ssh23:38:040
2929267995013,5cyclictest131rcu_preempt00:07:380
292928199495,6cyclictest2934547-21grep20:50:023
2929281994917,29cyclictest37-21ksoftirqd/320:40:023
2929276994940,5cyclictest2955239-21kworker/2:023:19:212
2929272994934,12cyclictest2934374-21sshd20:45:141
2929267994932,12cyclictest12-21ksoftirqd/022:05:070
292926799491,43cyclictest2935513-21ssh21:10:020
292928199481,11cyclictest364-21snmpd19:55:443
292928199481,11cyclictest364-21snmpd19:55:443
2929276994841,4cyclictest2935202-21kworker/2:222:38:482
2929276994840,5cyclictest2955239-21kworker/2:023:25:592
2929276994812,6cyclictest131rcu_preempt23:50:312
2929276994812,6cyclictest131rcu_preempt23:50:312
292926799483,9cyclictest12-21ksoftirqd/022:34:020
292926799483,6cyclictest131rcu_preempt21:45:200
2929276994733,9cyclictest3467602-21systemd-journal19:30:022
2929276994733,10cyclictest1-21systemd22:00:002
2929267994740,4cyclictest16950irq/224-800000019:20:010
2929267994739,4cyclictest16950irq/224-800000020:00:010
2929267994739,4cyclictest16950irq/224-800000020:00:010
2929267994738,6cyclictest12-21ksoftirqd/000:22:150
2929267994732,10cyclictest2971509-21latency_hist22:25:020
2929267994723,9cyclictest2966571-21ssh22:14:530
2929267994723,9cyclictest2966571-21ssh22:14:530
292928199462,8cyclictest2932928-21cat20:20:013
292928199461,12cyclictest2929313-21wc19:10:013
2929276994637,6cyclictest2955239-21kworker/2:023:13:342
2929276994632,10cyclictest3467602-21systemd-journal19:39:142
2929276994630,11cyclictest30-21ksoftirqd/219:15:022
292926799461,7cyclictest1-21systemd20:25:010
292926799461,7cyclictest1-21systemd20:25:010
2929276994535,6cyclictest2933914-21kworker/2:121:05:022
2929276994531,10cyclictest30-21ksoftirqd/221:33:422
2929267994527,12cyclictest3467602-21systemd-journal19:15:000
2929276994436,5cyclictest2955239-21kworker/2:023:22:512
2929276994431,9cyclictest3467602-21systemd-journal19:57:152
2929276994431,9cyclictest3467602-21systemd-journal19:57:152
2929276994427,10cyclictest2932828-21cron20:20:002
2929267994427,12cyclictest2934587-21latency_hist20:50:030
2929276994335,5cyclictest2935202-21kworker/2:221:28:472
2929276994320,16cyclictest275-21dbus-daemon21:00:012
292926799436,26cyclictest2929269-21cat19:10:010
2929267994331,8cyclictest12-21ksoftirqd/020:10:000
2929267994330,11cyclictest12-21ksoftirqd/021:05:010
292926799432,34cyclictest3467602-21systemd-journal20:20:000
2929267994312,6cyclictest131rcu_preempt19:25:020
2929276994234,5cyclictest2935202-21kworker/2:221:50:562
2929276994234,5cyclictest2935202-21kworker/2:221:50:552
2929276994233,6cyclictest2919849-21kworker/2:320:54:462
2929276994224,13cyclictest3467718-21lldpd00:34:292
2929267994235,4cyclictest16950irq/224-800000019:55:000
2929267994232,7cyclictest2930796-21systemd-run19:39:590
292926799421,37cyclictest3467602-21systemd-journal19:41:420
2929276994129,8cyclictest1-21systemd21:25:012
2929276994129,8cyclictest1-21systemd21:25:012
2929276994128,9cyclictest3467602-21systemd-journal21:10:002
2929276994123,13cyclictest3029344-21cat00:25:022
2929276994116,18cyclictest275-21dbus-daemon19:30:012
2929272994129,9cyclictest2929275-21sort19:10:011
292926799412,36cyclictest2934678-21latency_hist20:55:010
2929267994122,14cyclictest3467602-21systemd-journal19:25:000
292927699404,8cyclictest131rcu_preempt20:10:002
2929276994025,10cyclictest2933885-21systemctl20:40:002
2929276994023,13cyclictest1-21systemd20:25:012
2929276994023,13cyclictest1-21systemd20:25:012
2929276994022,12cyclictest275-21dbus-daemon20:35:002
2929267994024,12cyclictest2933838-21sshd20:35:140
2929267994011,8cyclictest131rcu_preempt20:15:010
2929267994011,8cyclictest131rcu_preempt20:15:010
2929276993923,11cyclictest275-21dbus-daemon20:50:002
292926799394,10cyclictest12-21ksoftirqd/020:42:280
2929267993926,10cyclictest12-21ksoftirqd/020:05:000
2929276993828,7cyclictest30-21ksoftirqd/220:45:022
2929276993814,17cyclictest275-21dbus-daemon19:45:012
292926799387,9cyclictest12-21ksoftirqd/020:56:370
292927699373,26cyclictest2930024-21latency_hist19:25:012
2929267993725,10cyclictest12-21ksoftirqd/020:30:010
2929267993723,10cyclictest2930488-21sshd19:33:150
2929276993625,8cyclictest0-21swapper/219:50:022
2929276993117,9cyclictest3467602-21systemd-journal20:25:032
2929276993021,7cyclictest2929318-21latency_hist19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional