You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 06:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sun Apr 26, 2026 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2994291991265,22cyclictest0-21swapper/000:00:030
299430599932,67cyclictest3073735-40mandb00:00:033
299430099911,49cyclictest3075271-21sh00:03:162
299429199787,5cyclictest16850irq/134-800000000:40:020
2994291997366,4cyclictest0-21swapper/021:05:020
2994300996821,42cyclictest3011318-21wc21:35:022
2994300996821,42cyclictest3011318-21wc21:35:012
299430099681,63cyclictest3062295-21ssh23:33:072
299430099681,62cyclictest3083647-21sshd00:22:422
299430099681,62cyclictest3080803-21ssh00:15:542
299430099681,61cyclictest3067852-21ssh23:45:462
299430099671,61cyclictest3023204-21ssh22:02:262
299430099671,61cyclictest2996601-21sshd19:54:422
2994300996713,50cyclictest2999080-21ls20:40:012
2994291996747,17cyclictest12-21ksoftirqd/023:37:060
2994300996623,40cyclictest30-21ksoftirqd/223:26:002
299430099661,61cyclictest3069842-21ssh23:50:262
299430099661,61cyclictest3037425-21ssh22:35:092
299430099652,59cyclictest3052776-21ssh23:10:442
299430099651,60cyclictest3089418-21ssh00:35:522
299430099651,60cyclictest3086239-21ssh00:28:502
299430099651,60cyclictest3079766-21ssh00:13:512
299430099651,60cyclictest3072431-21ssh23:56:332
299430099651,60cyclictest3063422-21ssh23:35:342
299430099651,60cyclictest3055117-21ssh23:16:182
299430099651,60cyclictest3046810-21ssh22:57:082
299430099651,60cyclictest3035517-21ssh22:30:522
299430099651,60cyclictest3031800-21ssh22:22:192
299430099651,60cyclictest3029848-21ssh22:17:512
299430099651,60cyclictest3012980-21ssh21:38:592
299430099651,60cyclictest3007915-21ssh21:26:552
299430099651,60cyclictest3004084-21ssh21:18:142
2994296996515,15cyclictest2998957-21munin-run20:40:011
2994291996523,38cyclictest150-21pr/ttyS221:18:230
299430099641,60cyclictest3006144-21ssh21:22:532
299430099641,59cyclictest3076707-21ssh00:06:302
299430099641,59cyclictest3066346-21ssh23:42:262
299430099641,59cyclictest3057725-21ssh23:22:312
299430099641,59cyclictest3048912-21ssh23:01:592
299430099641,59cyclictest3044938-21ssh22:52:512
299430099641,59cyclictest3042357-21ssh22:46:492
299430099641,59cyclictest3042357-21ssh22:46:482
299430099641,59cyclictest3040492-21ssh22:42:262
299430099641,59cyclictest3033424-21ssh22:25:532
299430099641,59cyclictest3028370-21ssh22:14:452
299430099641,59cyclictest3026183-21ssh22:09:392
299430099641,59cyclictest3021187-21ssh21:57:552
299430099641,59cyclictest3019764-21ssh21:54:462
299430099641,59cyclictest3015914-21ssh21:45:232
299430099641,59cyclictest3013927-21ssh21:40:462
299430099641,59cyclictest3013927-21ssh21:40:452
299430099641,59cyclictest3001263-21ssh21:11:282
299430099631,58cyclictest3051114-21ssh23:07:032
2994291996321,9cyclictest3082480-21munin-run00:20:000
299430099622,55cyclictest2997374-21sshd20:07:432
299430099621,58cyclictest3087183-21ssh00:30:412
299430099621,57cyclictest2999445-21sshd20:45:432
299430099621,57cyclictest2996292-21sshd19:48:422
299429199622,44cyclictest3031265-21ssh22:21:000
2994291996220,11cyclictest3056569-21munin-run23:20:000
2994291996217,10cyclictest12-21ksoftirqd/021:20:250
299430099612,54cyclictest2995041-21/usr/sbin/munin19:25:012
299430099611,56cyclictest48-21kcompactd020:20:062
299430099611,56cyclictest2994501-21munin-run19:15:012
2994291996123,8cyclictest12-21ksoftirqd/019:48:420
299430099602,53cyclictest2998712-21munin-run20:35:012
299430099602,53cyclictest2997632-21sshd20:13:422
299430099601,55cyclictest2999196-21sshd20:41:442
2994291996019,33cyclictest16850irq/134-800000021:47:320
2994305995946,10cyclictest37-21ksoftirqd/322:12:113
2994291995916,38cyclictest16850irq/134-800000000:00:050
2994291995915,35cyclictest16850irq/134-800000000:25:000
2994305995816,6cyclictest131rcu_preempt00:35:033
299430099582,52cyclictest364-21snmpd20:58:402
299430099582,52cyclictest2999747-21munin-run20:55:012
299430099581,53cyclictest2996841-21sshd19:57:422
299430099581,51cyclictest48-21kcompactd019:37:542
2994296995840,15cyclictest2999951-21sshd20:55:431
2994296995840,14cyclictest3048519-21ssh23:00:521
2994296995837,17cyclictest3025682-21ssh22:08:171
2994291995823,32cyclictest3010829-21ssh21:33:560
2994291995823,32cyclictest3010829-21ssh21:33:560
299430599573,9cyclictest37-21ksoftirqd/323:00:003
299430099571,52cyclictest364-21snmpd19:26:092
2994296995740,13cyclictest3034598-21ssh22:29:021
2994296995739,15cyclictest3064932-21ssh23:39:281
2994296995711,9cyclictest131rcu_preempt23:12:231
299429199575,21cyclictest3022025-21systemctl22:00:010
299429199573,37cyclictest3067902-21ssh23:45:540
2994291995718,30cyclictest16850irq/134-800000023:34:270
2994291995717,32cyclictest16850irq/134-800000022:50:010
2994291995717,32cyclictest16850irq/134-800000022:50:010
2994305995648,5cyclictest37-21ksoftirqd/323:05:023
2994305995647,7cyclictest37-21ksoftirqd/322:50:023
2994305995647,7cyclictest37-21ksoftirqd/322:50:023
299430099561,51cyclictest2996034-21sshd19:42:152
299430099561,50cyclictest48-21kcompactd020:02:212
299430099561,50cyclictest48-21kcompactd020:02:202
2994296995641,12cyclictest3003958-21ssh21:17:471
2994296995640,13cyclictest3062972-21ssh23:34:551
2994296995640,13cyclictest3013186-21ssh21:39:291
2994296995639,13cyclictest3065518-21ssh23:40:161
2994296995639,13cyclictest3046436-21ssh22:56:131
2994296995639,13cyclictest3008071-21ssh21:27:181
2994296995638,15cyclictest3030962-21ssh22:20:041
2994296995638,14cyclictest3078342-21ssh00:10:061
2994296995638,14cyclictest3027480-21ssh22:12:211
2994296995616,9cyclictest131rcu_preempt22:20:021
2994291995621,4cyclictest3013425-21munin-run21:40:000
2994291995618,30cyclictest16850irq/134-800000022:16:470
2994300995527,25cyclictest2995509-21sshd19:33:422
299430099551,50cyclictest2997867-21sshd20:15:442
299430099551,49cyclictest48-21kcompactd019:18:292
2994296995542,11cyclictest3005999-21ssh21:22:321
2994296995540,12cyclictest3089442-21ssh00:35:551
2994296995539,13cyclictest3085704-21ssh00:27:231
2994296995539,13cyclictest3057071-21ssh23:20:501
2994296995539,12cyclictest3040611-21ssh22:42:441
2994296995538,14cyclictest3059651-21ssh23:26:541
2994296995538,14cyclictest3022518-21ssh22:00:411
2994296995538,13cyclictest3081171-21ssh00:16:551
2994296995538,13cyclictest3070046-21ssh23:50:581
2994296995538,13cyclictest3018183-21ssh21:50:431
2994296995538,13cyclictest3014446-21ssh21:42:061
2994296995538,13cyclictest3014446-21ssh21:42:051
2994296995538,13cyclictest3000849-21ssh21:10:151
2994296995537,14cyclictest3076527-21ssh00:06:051
2994296995537,14cyclictest3074341-21ssh00:00:501
2994291995515,31cyclictest16850irq/134-800000022:58:470
2994296995441,10cyclictest3087006-21ssh00:30:161
2994296995441,10cyclictest3015913-21ssh21:45:231
2994296995440,11cyclictest3021561-21ssh21:58:491
2994296995439,12cyclictest3073213-21ssh23:58:351
2994296995439,12cyclictest3067658-21ssh23:45:171
2994296995439,12cyclictest3054618-21ssh23:15:081
2994296995439,12cyclictest3044019-21ssh22:50:301
2994296995439,12cyclictest3041747-21ssh22:45:101
2994296995439,12cyclictest3041747-21ssh22:45:101
2994296995439,12cyclictest3009541-21ssh21:30:361
2994296995439,12cyclictest3009541-21ssh21:30:351
2994296995439,11cyclictest3035817-21ssh22:31:341
2994296995438,13cyclictest3083178-21ssh00:21:231
2994296995438,13cyclictest3050579-21ssh23:05:441
2994296995438,13cyclictest3037440-21ssh22:35:121
2994296995437,13cyclictest2997112-21sshd20:00:441
2994296995437,13cyclictest2997112-21sshd20:00:431
2994296995436,14cyclictest364-21snmpd20:51:291
2994291995427,12cyclictest3039182-21ssh22:39:490
2994291995426,12cyclictest3076924-21ssh00:07:000
2994291995426,12cyclictest3036358-21ssh22:32:590
2994291995426,12cyclictest3014727-21ssh21:42:560
2994291995426,12cyclictest3014727-21ssh21:42:550
2994291995424,27cyclictest3033882-21ssh22:27:060
299429199541,38cyclictest3088094-21ssh00:32:580
299430099531,48cyclictest3000493-21sshd21:07:432
299430099531,47cyclictest364-21snmpd21:00:092
2994296995337,12cyclictest2999228-21munin-run20:45:001
2994291995329,20cyclictest3008227-21ssh21:27:490
299429199531,47cyclictest3018122-21ssh21:50:330
2994291995313,30cyclictest16850irq/134-800000023:06:130
299430599525,12cyclictest37-21ksoftirqd/300:20:013
299430099521,47cyclictest364-21snmpd20:25:202
2994296995237,12cyclictest2996542-21sshd19:52:411
2994296995235,12cyclictest3000266-21munin-run21:05:001
2994291995248,2cyclictest0-21swapper/000:15:020
299429199524,38cyclictest2994294-21grep19:10:020
299429199524,38cyclictest2994294-21grep19:10:020
2994291995225,12cyclictest3057086-21ssh23:20:510
2994291995221,16cyclictest16850irq/134-800000022:02:060
299430599512,10cyclictest3000585-21ssh21:10:023
2994305995120,10cyclictest37-21ksoftirqd/322:18:043
2994305995114,5cyclictest131rcu_preempt21:56:033
2994305995111,6cyclictest131rcu_preempt21:53:323
2994291995134,13cyclictest1-21systemd20:40:000
2994291995115,31cyclictest16850irq/134-800000023:10:090
2994305995038,9cyclictest37-21ksoftirqd/322:54:423
2994305995012,5cyclictest131rcu_preempt22:40:013
2994296995037,10cyclictest2998176-21sshd20:24:421
2994296995036,11cyclictest2994460-21sshd19:12:131
2994296995036,10cyclictest48-21kcompactd019:35:411
2994296995036,10cyclictest364-21snmpd20:11:191
2994296995032,14cyclictest364-21snmpd20:08:101
2994291995036,10cyclictest1-21systemd20:50:000
2994291995021,14cyclictest3028300-21ssh22:14:270
2994291995015,7cyclictest12-21ksoftirqd/020:20:010
299430599499,10cyclictest37-21ksoftirqd/321:36:143
2994305994941,5cyclictest37-21ksoftirqd/322:24:433
2994296994937,8cyclictest48-21kcompactd019:49:021
2994296994936,9cyclictest48-21kcompactd019:43:271
2994296994936,9cyclictest48-21kcompactd019:20:091
2994296994936,9cyclictest2995279-21munin-run19:30:001
2994296994925,19cyclictest2994783-21true19:20:001
2994291994923,10cyclictest3069922-21ssh23:50:380
2994291994919,14cyclictest3000819-21ssh21:10:120
2994291994917,28cyclictest16850irq/134-800000000:25:080
299430599483,6cyclictest131rcu_preempt21:45:013
299430599483,6cyclictest131rcu_preempt21:45:013
2994305994833,11cyclictest37-21ksoftirqd/322:05:263
2994305994817,12cyclictest37-21ksoftirqd/323:45:003
2994305994812,6cyclictest131rcu_preempt21:14:493
299430599480,7cyclictest150-21pr/ttyS222:43:423
299429199489,35cyclictest0-21swapper/020:00:000
2994291994815,29cyclictest16850irq/134-800000022:09:410
2994291994812,9cyclictest0-21swapper/022:42:060
299430599478,9cyclictest37-21ksoftirqd/323:45:473
299430599478,5cyclictest131rcu_preempt22:01:423
2994305994739,5cyclictest37-21ksoftirqd/321:32:053
2994305994739,5cyclictest37-21ksoftirqd/321:32:053
2994305994738,5cyclictest361rcuc/323:27:093
2994305994717,21cyclictest0-21swapper/321:17:383
299430599471,6cyclictest0-21swapper/321:29:593
2994305994711,7cyclictest131rcu_preempt00:15:003
2994296994734,10cyclictest364-21snmpd20:49:151
2994296994734,10cyclictest2995545-21munin-run19:35:011
2994296994728,15cyclictest364-21snmpd21:05:461
2994291994726,16cyclictest3059535-21rm23:26:380
2994291994711,33cyclictest3049877-21ssh23:04:250
299430599464,7cyclictest37-21ksoftirqd/300:00:063
2994305994631,11cyclictest1-21systemd20:55:003
299430599461,7cyclictest3076758-21sh00:06:383
2994305994610,5cyclictest131rcu_preempt23:51:013
2994305994610,5cyclictest131rcu_preempt23:21:303
299430599460,5cyclictest150-21pr/ttyS200:22:473
2994296994634,9cyclictest364-21snmpd19:56:161
2994296994626,15cyclictest2998664-21sshd20:31:431
2994291994631,12cyclictest3467602-21systemd-journal20:42:240
299430599455,9cyclictest37-21ksoftirqd/321:21:323
2994305994537,6cyclictest37-21ksoftirqd/323:18:173
2994305994435,6cyclictest37-21ksoftirqd/320:45:023
2994305994432,10cyclictest37-21ksoftirqd/320:20:013
2994305994431,9cyclictest37-21ksoftirqd/323:08:523
2994305994415,9cyclictest131rcu_preempt22:32:223
2994305994413,10cyclictest37-21ksoftirqd/323:35:023
2994305994410,11cyclictest37-21ksoftirqd/323:14:583
299430099443,34cyclictest2994336-21grep19:10:022
299430099443,34cyclictest2994336-21grep19:10:022
2994296994427,13cyclictest2998460-21munin-run20:30:011
2994296994424,16cyclictest2997955-21systemd-run20:20:011
2994291994427,14cyclictest3044198-21ssh22:50:550
2994291994410,30cyclictest16850irq/134-800000023:40:220
299430599432,9cyclictest2993671-21kworker/3:022:29:363
299430599431,7cyclictest0-21swapper/319:16:133
2994305994314,5cyclictest131rcu_preempt21:45:543
2994305994311,6cyclictest131rcu_preempt00:29:443
299430599430,6cyclictest150-21pr/ttyS219:57:413
2994305994233,6cyclictest37-21ksoftirqd/319:55:023
2994305994232,7cyclictest37-21ksoftirqd/300:35:553
2994305994227,12cyclictest37-21ksoftirqd/320:10:003
299430599421,7cyclictest0-21swapper/323:37:303
2994291994226,12cyclictest2999949-21sshd20:55:440
2994291994225,13cyclictest2998676-21sshd20:33:430
2994291994225,13cyclictest2995755-21sshd19:37:150
2994305994126,10cyclictest3467602-21systemd-journal20:35:013
2994305994118,16cyclictest275-21dbus-daemon21:00:013
299430599411,6cyclictest0-21swapper/320:37:093
2994291994128,10cyclictest3467602-21systemd-journal19:15:000
2994291994128,10cyclictest2994758-21munin-run19:20:000
2994291994118,18cyclictest1-21systemd21:05:000
299430599401,7cyclictest0-21swapper/319:34:373
2994296994018,18cyclictest2994332-21latency_hist19:10:021
2994296994018,18cyclictest2994332-21latency_hist19:10:021
299429199408,27cyclictest1-21systemd20:10:000
2994291994029,8cyclictest12-21ksoftirqd/020:30:020
2994291994024,12cyclictest12-21ksoftirqd/019:52:430
2994291994023,11cyclictest2997641-21cron20:15:000
299429199400,9cyclictest0-21swapper/019:30:010
299430599391,5cyclictest0-21swapper/319:47:103
2994291993927,9cyclictest3467602-21systemd-journal20:00:320
2994291993927,9cyclictest3467602-21systemd-journal20:00:320
2994291993925,10cyclictest2999703-21sshd20:52:260
2994291993923,11cyclictest2995028-21munin-run19:25:010
2994305993829,7cyclictest37-21ksoftirqd/320:50:013
2994305993828,7cyclictest37-21ksoftirqd/320:30:003
2994305993827,9cyclictest37-21ksoftirqd/321:02:113
2994291993831,4cyclictest16950irq/224-800000020:20:490
2994291993825,10cyclictest2995515-21sshd19:34:420
2994291993820,13cyclictest364-21snmpd19:40:040
299430599378,5cyclictest131rcu_preempt19:40:033
2994305993627,7cyclictest37-21ksoftirqd/319:15:013
299430599361,6cyclictest150-21pr/ttyS220:11:243
2994305993612,6cyclictest131rcu_preempt19:21:323
2994305993525,8cyclictest37-21ksoftirqd/319:25:323
299430599351,6cyclictest0-21swapper/319:40:423
2994305993322,9cyclictest37-21ksoftirqd/320:24:443
2994305993223,7cyclictest37-21ksoftirqd/320:05:013
2994305993223,7cyclictest37-21ksoftirqd/320:05:013
2994305993121,7cyclictest37-21ksoftirqd/319:10:023
2994305993121,7cyclictest37-21ksoftirqd/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional