You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 22:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Sun Jun 28, 2026 12:43:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3931383997030,18cyclictest12650irq/32-eth009:30:300
3931385994436,5cyclictest3985843-21munin-run09:40:012
3931384994437,3cyclictest0-21swapper/110:05:311
3931383994437,5cyclictest3992898-21needreboot09:50:290
3931383994429,13cyclictest3964404-21latency_hist09:00:020
3931385994334,6cyclictest421-21dbus-daemon08:05:012
3931384994332,7cyclictest3127484-21cups-browsed12:15:231
393138399431,36cyclictest4062860-21ssh11:45:350
3931385994233,6cyclictest421-21dbus-daemon11:31:492
3931383994236,4cyclictest4001159-21systemd-run10:05:020
3931383994232,8cyclictest3941879-21idleruntime-cro07:45:010
3931385994131,7cyclictest4092876-21missed_timers12:35:292
3931384994130,8cyclictest4048404-21sshd11:21:011
3931383994121,15cyclictest12650irq/32-eth012:05:330
3931385994034,3cyclictest0-21swapper/211:25:362
3931385994034,2cyclictest0-21swapper/210:00:012
3931385994032,4cyclictest4046640-21sendmail11:20:012
3931384994033,4cyclictest4025440-21idleruntime-cro10:45:011
3931383994019,14cyclictest12650irq/32-eth011:15:360
3931383994017,15cyclictest12650irq/32-eth009:15:430
3931385993934,2cyclictest0-21swapper/210:40:332
3931385993933,3cyclictest0-21swapper/212:00:262
3931385993930,5cyclictest4079892-21grep12:15:012
3931385993930,5cyclictest1003-21mta-sts-daemon10:19:582
3931385993928,7cyclictest4047638-21ntp_kernel_err11:20:312
3931384993931,5cyclictest1-21systemd12:35:011
3931383993929,8cyclictest4083767-21latency12:20:260
3931384993830,5cyclictest4070834-21idleruntime-cro12:00:011
3931384993830,4cyclictest0-21swapper/111:40:371
3931384993827,7cyclictest421-21dbus-daemon11:35:021
3931383993829,4cyclictest0-21swapper/012:30:320
3931385993730,4cyclictest4031557-21latency_hist10:55:022
3931385993730,3cyclictest0-21swapper/211:05:362
3931385993730,3cyclictest0-21swapper/209:45:202
3931384993730,4cyclictest4085956-21cron12:25:011
3931384993730,4cyclictest3991964-21/usr/sbin/munin09:50:011
3931384993729,5cyclictest4029880-21timerandwakeup10:50:421
3931384993728,6cyclictest3976764-21latency_hist09:25:021
3931384993728,6cyclictest3127484-21cups-browsed09:12:261
3931384993727,7cyclictest4017714-21switchtime10:30:431
3931384993727,6cyclictest0-21swapper/108:30:361
3931383993730,4cyclictest1002-21mta-sts-daemon11:20:380
3931383993729,5cyclictest4078142-21sendmail_mailst12:10:410
3931383993725,9cyclictest4019375-21munin-plugin-st10:35:010
3931385993630,3cyclictest0-21swapper/210:00:342
3931385993630,3cyclictest0-21swapper/209:25:262
3931385993630,3cyclictest0-21swapper/209:20:012
3931385993628,5cyclictest4037553-21munin-plugin-st11:05:002
3931385993628,5cyclictest4037553-21munin-plugin-st11:05:002
3931385993628,5cyclictest4028466-21cut10:50:012
3931385993628,5cyclictest3974336-21latency09:20:272
3931385993628,5cyclictest3940371-21systemctl07:40:012
3931385993627,6cyclictest420-21avahi-daemon10:55:412
3931385993627,6cyclictest1-21systemd11:40:022
3931385993624,8cyclictest421-21dbus-daemon09:12:042
3931384993629,4cyclictest4079880-21ssh12:15:011
3931384993629,3cyclictest0-21swapper/111:55:011
3931384993628,5cyclictest4095045-21basename12:40:011
3931384993628,5cyclictest4076870-21latency_hist12:10:011
3931384993628,5cyclictest3958366-21cut08:40:011
3931384993628,4cyclictest0-21swapper/109:15:361
3931383993626,7cyclictest0-21swapper/012:00:010
3931383993626,5cyclictest151ktimers/008:00:300
3931385993529,3cyclictest0-21swapper/210:35:322
3931385993529,3cyclictest0-21swapper/210:30:372
3931385993528,3cyclictest0-21swapper/211:40:412
3931385993528,3cyclictest0-21swapper/210:05:392
3931385993527,5cyclictest3967367-21diskmemload11:50:212
3931385993525,6cyclictest3127484-21cups-browsed12:20:102
3931385993524,6cyclictest485-21in:imuxsock07:45:012
393138499353,26cyclictest3986942-21ssh09:40:391
3931384993526,4cyclictest0-21swapper/110:55:311
3931383993528,4cyclictest4057301-21unixbench_multi11:35:440
3931385993427,3cyclictest0-21swapper/209:40:382
3931385993426,5cyclictest3948413-21fschecks_time08:05:252
3931385993425,5cyclictest3995030-21systemd-run09:55:012
3931385993425,5cyclictest3942393-21grep07:45:252
3931385993425,5cyclictest3942393-21grep07:45:252
393138499343,25cyclictest4071831-21ssh12:00:321
3931384993426,3cyclictest0-21swapper/112:25:371
3931384993425,6cyclictest627-21lldpd09:31:421
3931384993423,8cyclictest421-21dbus-daemon08:10:021
393138499341,26cyclictest3939836-21sshd07:35:361
393138399343,23cyclictest3974599-21ssh09:20:330
3931383993428,4cyclictest4061755-21cron11:45:010
3931383993428,4cyclictest3968807-21threads09:10:440
3931383993427,5cyclictest3957390-21fschecks_count08:35:300
3931385993327,3cyclictest0-21swapper/212:20:012
3931385993327,3cyclictest0-21swapper/209:30:372
3931385993326,4cyclictest4074522-21gltestperf12:05:262
3931385993325,5cyclictest3959821-21unixbench_singl08:40:442
3931384993326,4cyclictest4064784-21munin-plugin-st11:50:011
3931384993326,3cyclictest0-21swapper/111:15:321
393138499331,26cyclictest4050907-21diskmemload11:25:401
3931383993327,4cyclictest4007286-21latency_hist10:15:020
3931383993327,3cyclictest0-21swapper/012:01:350
3931383993326,4cyclictest1001-21mta-sts-daemon11:30:420
3931383993324,6cyclictest421-21dbus-daemon10:54:130
3931383993318,10cyclictest151ktimers/008:10:330
3931385993226,3cyclictest3934615-21kworker/u8:7-rpciod09:10:012
3931385993226,3cyclictest0-21swapper/207:35:012
3931385993225,4cyclictest4013845-21aten_r2power_vo10:25:252
3931385993223,5cyclictest4062433-21tr11:45:272
3931385993222,6cyclictest3127484-21cups-browsed08:25:012
3931385993221,8cyclictest4041594-21ntp_kernel_pll_11:10:332
3931385993221,8cyclictest4041594-21ntp_kernel_pll_11:10:332
3931384993226,3cyclictest0-21swapper/110:20:011
3931384993224,5cyclictest4028540-21latency_hist10:50:021
3931384993224,5cyclictest3965896-21systemd-run09:05:011
3931384993224,5cyclictest3127484-21cups-browsed08:02:341
393138499321,25cyclictest3941332-21sshd07:40:351
3931383993227,2cyclictest0-21swapper/009:40:350
3931383993225,3cyclictest0-21swapper/010:50:010
3931383993210,16cyclictest665-21snmpd10:39:520
3931385993123,4cyclictest0-21swapper/207:20:012
3931385993110,15cyclictest421-21dbus-daemon12:25:402
3931384993126,2cyclictest0-21swapper/110:40:011
3931384993126,2cyclictest0-21swapper/110:10:361
3931384993124,4cyclictest1000-21mta-sts-daemon09:37:461
3931384993124,3cyclictest0-21swapper/110:05:011
3931384993120,4cyclictest1001-21mta-sts-daemon10:20:451
3931384993118,8cyclictest1000-21mta-sts-daemon11:14:571
3931384993118,8cyclictest1000-21mta-sts-daemon11:14:571
393138399319,12cyclictest12650irq/32-eth011:25:380
393138399318,21cyclictest12650irq/32-eth010:40:320
3931383993125,3cyclictest0-21swapper/010:05:440
3931383993125,3cyclictest0-21swapper/008:00:000
3931383993122,5cyclictest0-21swapper/011:00:360
3931383993122,5cyclictest0-21swapper/011:00:360
393138599303,22cyclictest3127484-21cups-browsed08:10:382
3931385993024,3cyclictest0-21swapper/212:30:242
3931385993020,8cyclictest3944269-21ntp_kernel_pll_07:50:332
3931385993020,6cyclictest420-21avahi-daemon10:13:562
3931384993013,11cyclictest271ktimers/109:05:371
3931383993022,6cyclictest3989838-21missed_timers09:45:310
3931383993022,5cyclictest0-21swapper/012:25:410
393138399302,22cyclictest3996692-21users09:55:460
3931383993021,7cyclictest3953769-21tail08:20:430
393138399301,24cyclictest265-21systemd-journal10:20:470
3931385992922,3cyclictest0-21swapper/211:55:372
393138599292,21cyclictest3963756-21sshd08:55:352
3931385992921,5cyclictest3952369-21munin-run08:20:012
3931384992924,2cyclictest0-21swapper/111:40:021
3931384992922,3cyclictest0-21swapper/109:50:331
3931384992922,3cyclictest0-21swapper/109:25:341
3931384992921,5cyclictest3938867-21munin-plugin-st07:35:011
3931384992921,5cyclictest3127484-21cups-browsed07:21:491
3931384992920,7cyclictest3995580-21cpuspeed09:55:251
3931383992924,3cyclictest1001-21mta-sts-daemon10:17:100
3931383992924,2cyclictest0-21swapper/012:40:010
3931383992923,3cyclictest0-21swapper/010:25:240
3931385992820,5cyclictest4011896-21grep10:20:522
3931385992818,7cyclictest3935595-21threads07:20:432
3931384992823,2cyclictest0-21swapper/108:59:361
3931384992821,4cyclictest3936563-21latency07:25:291
3931384992821,4cyclictest1000-21mta-sts-daemon10:28:461
3931384992820,5cyclictest3953609-21timerandwakeup08:20:431
3931384992820,5cyclictest1003-21mta-sts-daemon11:03:121
3931384992820,5cyclictest1003-21mta-sts-daemon11:03:121
3931384992819,7cyclictest3952386-21sendmail08:20:011
3931383992823,2cyclictest0-21swapper/008:20:010
3931383992822,4cyclictest3977555-21grep09:25:300
3931383992822,3cyclictest1003-21mta-sts-daemon09:05:370
3931383992822,3cyclictest0-21swapper/011:15:000
3931383992822,3cyclictest0-21swapper/011:15:000
3931383992820,6cyclictest3954397-21uniq08:25:250
3931383992819,5cyclictest4033042-21unixbench_multi10:55:410
393138599273,19cyclictest428-21mta-sts-daemon08:35:432
3931385992720,3cyclictest0-21swapper/207:10:442
3931385992717,7cyclictest265-21systemd-journal07:30:012
3931384992720,4cyclictest3933382-21fschecks_count07:15:261
3931384992720,4cyclictest1003-21mta-sts-daemon11:06:421
3931384992720,4cyclictest0-21swapper/107:15:011
3931384992719,5cyclictest0-21swapper/108:50:441
3931383992718,7cyclictest3127484-21cups-browsed07:25:290
3931385992620,3cyclictest1001-21mta-sts-daemon09:02:542
3931384992619,4cyclictest3950858-21sh08:15:011
3931384992619,4cyclictest1001-21mta-sts-daemon08:40:261
3931383992621,3cyclictest3958901-21fschecks_count08:40:260
3931383992621,3cyclictest1002-21mta-sts-daemon07:15:340
3931383992620,3cyclictest4065289-21aten2.4-expect11:50:220
3931383992612,8cyclictest12650irq/32-eth009:35:330
393138599252,19cyclictest623-21sshd08:25:232
3931385992518,5cyclictest0-21swapper/208:00:012
3931385992512,10cyclictest3962263-21ntp_kernel_pll_08:50:342
3931384992514,8cyclictest3945874-21/usr/sbin/munin07:55:381
393138399255,15cyclictest4081018-21sshd12:15:350
393138399252,18cyclictest3956311-21sshd08:30:360
3931383992510,9cyclictest4038583-21ntpq11:05:320
3931385992418,3cyclictest3932883-21kworker/u8:1-xprtiod08:30:452
3931384992417,4cyclictest3942393-21fschecks_count07:45:251
3931384992417,4cyclictest3942393-21fschecks_count07:45:251
3931384992417,4cyclictest1003-21mta-sts-daemon07:50:211
393138499241,20cyclictest271ktimers/108:25:341
3931383992419,3cyclictest1002-21mta-sts-daemon07:20:270
3931383992418,4cyclictest3938581-21awk07:30:430
3931383992418,4cyclictest3932865-21latency_hist07:15:010
3931383992417,4cyclictest151ktimers/008:50:340
3931385992318,2cyclictest0-21swapper/207:10:022
3931385992313,7cyclictest3961151-21timerwakeupswit08:45:432
3931384992317,3cyclictest0-21swapper/108:50:011
393138399236,3cyclictest171rcu_preempt07:35:430
3931383992317,4cyclictest0-21swapper/009:00:360
3931383992317,3cyclictest3932883-21kworker/u8:1-rpciod08:10:020
393138399222,15cyclictest3960765-21ntpq08:45:340
393138399221,18cyclictest3944326-21ntpq07:50:350
393138399211,14cyclictest3942741-21ntpq07:45:310
393138399211,14cyclictest3942741-21ntpq07:45:310
393138699182,11cyclictest0-21swapper/312:25:403
393138699182,11cyclictest0-21swapper/311:40:373
3931384991812,4cyclictest3931333-21latency_hist07:10:031
3931383991814,2cyclictest0-21swapper/007:10:020
393138699173,9cyclictest0-21swapper/312:15:343
393138699173,9cyclictest0-21swapper/311:10:373
393138699173,9cyclictest0-21swapper/311:10:373
393138699172,11cyclictest0-21swapper/311:50:303
393138699164,7cyclictest0-21swapper/310:40:383
393138699163,9cyclictest0-21swapper/309:50:303
393138699163,8cyclictest0-21swapper/311:35:313
393138699162,9cyclictest0-21swapper/309:35:323
393138699162,7cyclictest0-21swapper/312:20:293
393138699162,10cyclictest0-21swapper/309:40:403
393138699161,9cyclictest0-21swapper/310:10:393
393138699161,8cyclictest0-21swapper/307:40:343
393138699153,7cyclictest0-21swapper/312:10:323
393138699153,6cyclictest0-21swapper/311:25:323
393138699153,6cyclictest0-21swapper/308:40:023
393138699152,9cyclictest0-21swapper/311:20:353
393138699152,9cyclictest0-21swapper/310:35:323
393138699152,9cyclictest0-21swapper/309:25:363
393138699152,8cyclictest0-21swapper/311:30:333
393138699152,8cyclictest0-21swapper/310:30:393
393138699152,8cyclictest0-21swapper/309:10:363
393138699152,8cyclictest0-21swapper/308:40:363
393138699152,7cyclictest0-21swapper/311:05:233
393138699152,7cyclictest0-21swapper/310:25:283
393138699152,7cyclictest0-21swapper/308:10:383
393138699151,9cyclictest0-21swapper/309:55:373
393138699151,8cyclictest0-21swapper/307:10:323
393138699144,5cyclictest0-21swapper/312:05:353
393138699143,7cyclictest0-21swapper/310:20:393
393138699143,7cyclictest0-21swapper/309:15:393
393138699143,6cyclictest0-21swapper/310:50:303
393138699142,9cyclictest0-21swapper/312:35:353
393138699142,9cyclictest0-21swapper/311:00:233
393138699142,9cyclictest0-21swapper/311:00:233
393138699142,9cyclictest0-21swapper/310:55:413
393138699142,8cyclictest0-21swapper/311:15:383
393138699142,8cyclictest0-21swapper/310:15:363
393138699142,8cyclictest0-21swapper/310:00:283
393138699142,7cyclictest0-21swapper/309:45:233
393138699142,7cyclictest0-21swapper/309:05:353
393138699142,6cyclictest0-21swapper/309:30:223
393138699141,9cyclictest0-21swapper/312:30:313
393138699141,8cyclictest0-21swapper/312:00:183
393138699141,8cyclictest0-21swapper/311:45:323
393138699141,8cyclictest0-21swapper/307:50:323
393138699141,6cyclictest0-21swapper/308:20:223
393138699141,6cyclictest0-21swapper/308:15:343
393138699141,6cyclictest0-21swapper/308:05:303
393138699137,4cyclictest0-21swapper/307:35:353
393138699132,8cyclictest0-21swapper/308:30:363
393138699132,7cyclictest0-21swapper/309:20:383
393138699132,6cyclictest0-21swapper/307:45:323
393138699132,6cyclictest0-21swapper/307:45:323
393138699132,5cyclictest0-21swapper/308:55:343
393138699131,8cyclictest0-21swapper/310:05:323
393138699131,7cyclictest0-21swapper/310:45:223
393138699131,7cyclictest0-21swapper/307:55:363
393138699131,6cyclictest0-21swapper/311:55:223
393138699131,6cyclictest0-21swapper/308:50:343
393138699123,6cyclictest0-21swapper/307:15:313
393138699122,6cyclictest0-21swapper/308:45:353
393138699121,7cyclictest0-21swapper/307:30:353
393138699121,6cyclictest0-21swapper/308:25:293
393138699111,6cyclictest0-21swapper/309:00:323
393138699111,6cyclictest0-21swapper/307:25:353
393138699111,6cyclictest0-21swapper/307:20:333
393138699102,4cyclictest0-21swapper/308:00:223
39313869991,4cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional