You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 06:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Fri Jul 03, 2026 00:43:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
397286099544,38cyclictest4105011-21ssh23:45:430
3972860995136,9cyclictest12650irq/32-eth021:40:380
397286299503,39cyclictest4095663-21ssh23:30:342
3972861995042,5cyclictest4010231-21switchtime21:10:441
3972862994739,5cyclictest4100737-21ssh23:40:012
3972862994536,5cyclictest0-21swapper/221:00:402
3972862994534,8cyclictest4105150-21mailstats23:45:462
3972861994435,5cyclictest4097700-21basename23:35:011
3972861994433,7cyclictest1000-21mta-sts-daemon00:14:111
3972862994311,10cyclictest171rcu_preempt21:30:362
3972861994335,5cyclictest4091624-21latency_hist23:25:021
3972861994335,5cyclictest1-21systemd22:45:011
3972861994231,8cyclictest4112961-21idleruntime-cro00:00:011
3972861994133,5cyclictest4064045-21munin-run22:40:011
3972861994133,5cyclictest4051819-21ssh22:20:001
3972861994133,5cyclictest4051819-21ssh22:20:001
3972861994130,8cyclictest1003-21mta-sts-daemon21:37:121
397286199412,30cyclictest4071235-21ssh22:50:361
3972860994135,4cyclictest4039593-21ssh22:00:010
3972862994032,5cyclictest1003-21mta-sts-daemon23:57:392
3972862994030,6cyclictest3981884-21systemd-run19:40:012
3972861994032,5cyclictest4022354-21ssh21:30:401
3972861994032,3cyclictest0-21swapper/123:45:461
3972861994031,6cyclictest4015716-21ssh21:20:291
3972861994031,5cyclictest4088503-21ssh23:20:001
397286199401,29cyclictest4034601-21ssh21:50:411
3972860994032,5cyclictest4001377-21systemd-run20:45:010
397286299394,24cyclictest4108029-21ssh23:50:432
3972862993930,6cyclictest4018639-21aten_r2power_vo21:25:252
3972861993927,9cyclictest4123652-21timerwakeupswit00:15:471
3972861993926,4cyclictest0-21swapper/119:25:391
3972860993929,5cyclictest0-21swapper/000:15:400
3972862993829,6cyclictest4064045-21munin-run22:40:012
3972862993827,6cyclictest4123376-21proc_pri00:15:422
3972862993826,8cyclictest4088906-21aten_r2power_cu23:20:242
3972862993826,7cyclictest420-21avahi-daemon00:00:082
3972862993826,7cyclictest420-21avahi-daemon00:00:082
3972861993831,4cyclictest4015027-21munin-run21:20:011
3972861993829,5cyclictest4113014-21cups-browsed00:22:141
3972861993828,6cyclictest4079399-21systemd23:05:021
3972861993828,6cyclictest4053531-21grep22:20:491
3972861993817,4cyclictest171rcu_preempt21:05:361
3972860993830,5cyclictest4102175-21switchtime23:40:460
3972860993830,4cyclictest0-21swapper/021:15:280
3972862993728,5cyclictest0-21swapper/222:00:012
397286299372,4cyclictest0-21swapper/219:55:342
3972862993724,10cyclictest4116955-21missed_timers00:05:332
3972861993726,7cyclictest420-21avahi-daemon23:28:031
397286199372,24cyclictest4080421-21ssh23:05:351
3972860993727,4cyclictest4092718-21rm23:25:390
3972862993629,4cyclictest4082447-21systemd-run23:10:022
3972862993628,5cyclictest4128962-21ssh00:25:322
3972862993627,6cyclictest4070177-21/usr/sbin/munin22:50:012
3972862993627,6cyclictest4051819-21ssh22:20:002
3972862993627,6cyclictest4051819-21ssh22:20:002
3972862993627,6cyclictest4036529-21munin-plugin-st21:55:012
3972862993625,8cyclictest4126070-21latency00:20:312
397286299362,29cyclictest4135444-21ssh00:35:362
3972861993628,5cyclictest4076286-21munin-plugin-st23:00:011
3972861993620,3cyclictest171rcu_preempt21:30:011
3972860993624,5cyclictest12650irq/32-eth022:10:370
3972860993619,11cyclictest151ktimers/019:10:400
397286299353,20cyclictest4120118-21ssh00:10:342
3972862993527,5cyclictest4085476-21cut23:15:012
3972862993527,5cyclictest4042829-21latency_hist22:05:032
3972862993526,5cyclictest3983380-21systemd-run19:45:012
3972862993525,7cyclictest3999876-21systemd-run20:40:012
397286199353,24cyclictest4031557-21rm21:45:371
3972861993526,6cyclictest421-21dbus-daemon19:45:011
3972861993526,4cyclictest0-21swapper/122:45:401
3972861993524,7cyclictest4134472-21ls00:35:031
3972861993523,8cyclictest4040436-21latency22:00:301
3972860993527,4cyclictest4122143-21sh00:15:010
3972862993427,4cyclictest4033505-21systemd-run21:50:012
3972862993426,5cyclictest4055014-21latency_hist22:25:022
3972862993426,5cyclictest3974358-21systemd-run19:15:012
3972862993425,6cyclictest4076282-21idleruntime-cro23:00:012
3972862993414,4cyclictest171rcu_preempt22:11:232
397286199346,9cyclictest28-21ksoftirqd/100:35:351
3972861993426,5cyclictest4046444-21if_eth022:10:271
3972861993423,8cyclictest3985712-21ntp_kernel_pll_19:50:351
397286199342,27cyclictest4101824-21ssh23:40:341
3972861993415,3cyclictest171rcu_preempt19:55:341
3972860993428,4cyclictest4018774-21fschecks_count21:25:270
3972860993428,4cyclictest3986859-21fschecks_count19:55:270
3972860993426,5cyclictest151ktimers/022:35:410
397286099341,28cyclictest4126417-21ssh00:20:420
3972862993326,4cyclictest0-21swapper/221:40:412
3972862993325,5cyclictest4024756-21aten_r2power_po21:35:262
3972861993326,4cyclictest0-21swapper/123:35:291
3972861993326,3cyclictest0-21swapper/100:05:431
3972861993325,5cyclictest4027890-21aten_r2power_vo21:40:251
3972861993323,7cyclictest3990122-21missed_timers20:05:301
3972861993323,6cyclictest420-21avahi-daemon00:00:431
3972861993323,6cyclictest420-21avahi-daemon00:00:431
3972860993329,2cyclictest0-21swapper/000:00:010
3972860993327,3cyclictest0-21swapper/000:40:010
3972860993325,6cyclictest4085496-21sh23:15:020
3972860993322,8cyclictest4132665-21mailstats00:30:440
397286299325,14cyclictest4101777-21ssh23:40:332
3972862993225,3cyclictest0-21swapper/223:25:302
3972862993224,5cyclictest4077412-21ntp_kernel_pll_23:00:442
3972862993224,5cyclictest4045741-21systemd-run22:10:022
3972862993224,4cyclictest0-21swapper/221:20:292
397286199325,14cyclictest4055988-21ssh22:25:371
3972861993226,3cyclictest0-21swapper/119:35:011
3972861993223,4cyclictest0-21swapper/122:30:371
397286199322,22cyclictest4083434-21ssh23:10:331
397286099327,15cyclictest428-21mta-sts-daemon23:50:410
3972860993226,4cyclictest3998335-21cron20:35:010
3972860993224,5cyclictest2244171-21cups-browsed22:48:140
3972860993224,4cyclictest0-21swapper/022:50:480
3972860993221,8cyclictest4025800-21timerwakeupswit21:35:480
3972860993221,8cyclictest4024263-21sh21:35:010
3972860993216,9cyclictest151ktimers/020:20:370
397286299315,19cyclictest2343518-21munin-node20:10:352
3972862993123,6cyclictest4133004-21unixbench_singl00:30:482
3972862993123,5cyclictest439-21systemd-logind20:20:022
3972862993123,3cyclictest0-21swapper/221:10:382
397286199316,20cyclictest1003-21mta-sts-daemon20:00:341
3972861993125,3cyclictest0-21swapper/123:55:011
3972861993124,4cyclictest4131371-21systemd00:30:021
3972861993123,5cyclictest3998346-21sh20:35:011
3972861993123,5cyclictest2244171-21cups-browsed19:10:251
3972861993123,5cyclictest0-21swapper/121:00:011
3972861993120,5cyclictest271ktimers/120:15:331
3972861993116,2cyclictest171rcu_preempt19:35:411
397286199310,12cyclictest28-21ksoftirqd/120:35:401
3972860993123,6cyclictest4033828-21/usr/sbin/munin21:50:360
3972860993123,6cyclictest4031113-21fschecks_time21:45:280
3972860993123,6cyclictest3992860-21fschecks_count20:15:260
3972860993123,5cyclictest0-21swapper/023:20:000
397286099312,22cyclictest4089568-21ssh23:20:370
3972860993121,7cyclictest4067122-21latency_hist22:45:020
3972860993121,7cyclictest4054927-21ssh22:25:010
3972860993118,11cyclictest151ktimers/019:15:390
3972862993022,5cyclictest4086326-21latency23:15:322
397286299302,23cyclictest4013058-21ssh21:15:382
397286299302,21cyclictest2244166-21cupsd21:05:382
3972862993020,7cyclictest4059007-21ntp_kernel_pll_22:30:362
3972862993020,6cyclictest1001-21mta-sts-daemon22:50:402
3972861993023,5cyclictest4043057-21aten_r2power_en22:05:231
3972861993023,3cyclictest0-21swapper/119:20:011
3972861993018,6cyclictest171rcu_preempt20:50:441
3972861993018,3cyclictest171rcu_preempt20:25:321
3972861993017,6cyclictest171rcu_preempt19:20:351
3972860993023,4cyclictest4008797-21head21:05:470
397286099302,22cyclictest4129372-21rm00:25:400
3972860993020,8cyclictest0-21swapper/022:30:470
3972860993019,7cyclictest4043657-21ssh22:05:320
397286299292,22cyclictest4065103-21ssh22:40:342
3972862992921,3cyclictest0-21swapper/222:25:292
397286199298,5cyclictest28-21ksoftirqd/120:10:251
397286199292,9cyclictest28-21ksoftirqd/120:20:401
3972860992924,3cyclictest4015720-21cut21:20:290
3972860992923,4cyclictest3977885-21fschecks_time19:25:270
3972860992923,3cyclictest0-21swapper/023:40:010
3972860992923,3cyclictest0-21swapper/020:15:020
397286099292,22cyclictest2244166-21cupsd23:00:440
3972860992921,6cyclictest4002749-21unixbench_multi20:45:460
3972860992921,4cyclictest0-21swapper/022:25:380
3972860992919,7cyclictest4116054-21cut00:05:020
3972860992919,7cyclictest4116054-21cut00:05:020
3972862992820,5cyclictest0-21swapper/219:35:012
3972862992817,8cyclictest3984367-21proc_pri19:45:452
3972862992817,8cyclictest3984367-21proc_pri19:45:452
3972861992820,2cyclictest171rcu_preempt20:45:361
3972861992817,5cyclictest171rcu_preempt20:40:331
3972861992816,3cyclictest171rcu_preempt21:00:271
3972861992816,3cyclictest171rcu_preempt19:45:271
3972861992816,3cyclictest171rcu_preempt19:45:271
3972861992815,6cyclictest171rcu_preempt21:55:311
3972860992823,3cyclictest0-21swapper/022:55:290
3972860992821,5cyclictest0-21swapper/021:10:440
3972860992812,8cyclictest12650irq/32-eth023:05:410
397286299274,14cyclictest2244171-21cups-browsed20:20:412
3972862992722,3cyclictest1000-21mta-sts-daemon20:50:382
3972862992721,3cyclictest0-21swapper/221:00:012
3972862992721,3cyclictest0-21swapper/219:20:472
3972862992720,3cyclictest0-21swapper/220:30:012
3972862992720,3cyclictest0-21swapper/219:20:012
3972862992712,12cyclictest351ktimers/220:30:372
397286099272,22cyclictest665-21snmpd22:16:310
397286099272,22cyclictest665-21snmpd22:16:310
397286099272,19cyclictest4095715-21ntpq23:30:360
3972860992715,4cyclictest3999830-21cron20:40:010
3972862992619,4cyclictest3985376-21fschecks_time19:50:292
3972862992617,6cyclictest3989378-21systemd-run20:05:022
3972860992620,4cyclictest4117271-21proc_pri00:05:430
3972862992518,4cyclictest1001-21mta-sts-daemon20:05:312
397286199253,8cyclictest28-21ksoftirqd/119:10:011
397286099254,16cyclictest4040784-21sh22:00:420
3972860992518,4cyclictest151ktimers/020:05:410
3972862992417,5cyclictest265-21systemd-journal20:42:562
3972860992418,4cyclictest4007200-21unixbench-2d21:00:470
3972860992418,3cyclictest151ktimers/020:25:340
397286099241,13cyclictest4005726-21sed20:55:460
3972860992410,8cyclictest2244171-21cups-browsed19:45:400
3972860992410,8cyclictest2244171-21cups-browsed19:45:400
397286299235,8cyclictest171rcu_preempt19:25:442
3972862992313,3cyclictest171rcu_preempt19:10:012
3972860992317,3cyclictest151ktimers/019:30:340
3972860992317,3cyclictest0-21swapper/019:40:280
3972860992314,5cyclictest151ktimers/019:35:410
397286099231,18cyclictest3988462-21awk20:00:290
3972860992311,7cyclictest12650irq/32-eth019:50:370
397286299225,3cyclictest171rcu_preempt20:45:362
397286099221,16cyclictest4003857-21ps20:50:410
397286399203,9cyclictest0-21swapper/323:00:443
397286399192,9cyclictest0-21swapper/323:40:363
397286399192,8cyclictest0-21swapper/322:05:343
397286399192,6cyclictest0-21swapper/321:05:343
397286399192,10cyclictest0-21swapper/323:35:393
3972860991914,3cyclictest0-21swapper/019:20:470
397286399183,9cyclictest0-21swapper/321:30:393
397286399183,8cyclictest0-21swapper/323:30:363
397286399182,8cyclictest0-21swapper/323:45:383
397286399181,7cyclictest0-21swapper/319:20:413
397286399173,9cyclictest0-21swapper/300:30:423
397286399173,8cyclictest0-21swapper/322:55:333
397286399173,7cyclictest0-21swapper/319:15:433
397286399172,9cyclictest0-21swapper/323:05:373
397286399172,9cyclictest0-21swapper/321:10:343
397286399172,8cyclictest0-21swapper/323:25:373
397286399172,8cyclictest0-21swapper/321:20:293
397286399171,8cyclictest0-21swapper/321:15:363
397286399171,7cyclictest0-21swapper/322:35:273
3972863991710,4cyclictest0-21swapper/320:45:403
397286399165,5cyclictest0-21swapper/321:45:393
397286399163,8cyclictest0-21swapper/322:40:413
397286399163,8cyclictest0-21swapper/321:50:303
397286399162,9cyclictest0-21swapper/323:55:433
397286399162,8cyclictest0-21swapper/322:20:243
397286399162,8cyclictest0-21swapper/300:35:383
397286399162,8cyclictest0-21swapper/300:25:373
397286399161,9cyclictest0-21swapper/322:15:293
397286399161,9cyclictest0-21swapper/322:15:293
397286399161,9cyclictest0-21swapper/320:10:023
397286399161,9cyclictest0-21swapper/300:05:403
397286399161,8cyclictest0-21swapper/322:10:383
397286099161,12cyclictest3972817-21/usr/sbin/munin19:10:010
397286399156,5cyclictest0-21swapper/323:13:263
397286399152,8cyclictest0-21swapper/322:30:333
397286399152,8cyclictest0-21swapper/322:00:333
397286399152,8cyclictest0-21swapper/321:40:323
397286399152,8cyclictest0-21swapper/321:25:273
397286399152,8cyclictest0-21swapper/300:01:523
397286399152,8cyclictest0-21swapper/300:01:523
397286399152,6cyclictest0-21swapper/319:40:333
397286399151,9cyclictest0-21swapper/323:50:383
397286399151,9cyclictest0-21swapper/322:25:283
397286399151,9cyclictest0-21swapper/300:15:253
397286399151,9cyclictest0-21swapper/300:10:293
397286399151,8cyclictest0-21swapper/323:20:353
397286399151,6cyclictest0-21swapper/320:00:353
397286399151,6cyclictest0-21swapper/319:45:373
397286399151,6cyclictest0-21swapper/319:45:373
397286399151,10cyclictest0-21swapper/321:35:273
397286399150,7cyclictest0-21swapper/319:10:383
397286399148,4cyclictest0-21swapper/319:25:433
397286399146,4cyclictest0-21swapper/319:35:483
397286399142,8cyclictest0-21swapper/322:45:353
397286399141,9cyclictest0-21swapper/321:55:313
397286399141,8cyclictest0-21swapper/323:15:273
397286399141,7cyclictest0-21swapper/321:00:373
397286399141,6cyclictest0-21swapper/319:50:343
397286399141,5cyclictest0-21swapper/320:20:353
397286399136,4cyclictest0-21swapper/320:30:473
397286399135,5cyclictest0-21swapper/320:38:293
397286399132,7cyclictest0-21swapper/322:50:263
397286399132,7cyclictest0-21swapper/320:10:363
397286399132,7cyclictest0-21swapper/300:20:253
397286399131,7cyclictest0-21swapper/319:30:353
397286399131,6cyclictest0-21swapper/320:55:343
397286399131,6cyclictest0-21swapper/320:40:393
397286399126,4cyclictest0-21swapper/320:25:453
397286399124,4cyclictest0-21swapper/320:50:343
397286399121,6cyclictest0-21swapper/320:15:333
397286399116,3cyclictest0-21swapper/319:55:453
39728639984,2cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional