You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-11 - 04:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sat Apr 11, 2026 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3384638995851,3cyclictest0-21swapper/123:00:381
3384638995545,6cyclictest3563570-21ssh00:05:011
3384638995444,6cyclictest3387633-21systemctl19:20:011
338463999513,39cyclictest3466890-21ssh22:05:352
3384638995039,7cyclictest1461217-21cups-browsed22:31:471
3384637995020,8cyclictest3548413-21ntpq23:45:340
3384637994942,4cyclictest151ktimers/019:20:310
3384639994738,6cyclictest3559463-21cups-browsed00:34:292
3384638994739,5cyclictest1055-21mta-sts-daemon20:55:341
338463799472,38cyclictest659-21snmpd19:55:380
3384639994635,8cyclictest3531977-21latency23:25:302
338463899465,7cyclictest3420616-21diskmemload23:30:401
3384638994637,6cyclictest421-21dbus-daemon22:45:011
3384637994634,7cyclictest12350irq/32-eth021:55:340
3384637994621,7cyclictest3453180-21ssh21:49:390
3384639994536,3cyclictest0-21swapper/223:30:432
3384639994535,8cyclictest3519839-21missed_timers23:10:302
338463999453,35cyclictest3495599-21sh22:40:392
3384637994536,5cyclictest151ktimers/021:40:330
3384637994533,9cyclictest3571702-21cron00:15:010
3384637994530,10cyclictest3495497-21ssh22:40:350
3384639994431,9cyclictest3506561-21cron22:55:012
3384638994438,3cyclictest0-21swapper/122:25:001
3384638994430,10cyclictest3466626-21latency22:05:291
3384637994437,4cyclictest3504230-21tail22:50:450
3384637994419,7cyclictest3590155-21sh00:36:250
3384639994335,5cyclictest3571745-21ssh00:15:022
3384639994335,5cyclictest3559455-40systemctl00:00:002
3384639994335,5cyclictest3436992-21systemd-run21:30:012
3384639994335,5cyclictest3436992-21systemd-run21:30:012
338463899431,35cyclictest3520107-21rm23:10:401
3384637994332,8cyclictest0-21swapper/000:20:470
3384637994319,8cyclictest3472475-21ssh22:12:210
3384639994234,5cyclictest3567644-21ls00:10:012
3384637994227,10cyclictest3417014-21ntpq20:55:350
3384637994221,18cyclictest3456993-21sh21:54:080
3384637994218,7cyclictest3516559-21/usr/sbin/munin23:05:460
3384637994217,10cyclictest3492889-21sh22:37:240
3384637994216,10cyclictest3425080-21sh21:15:200
3384639994135,3cyclictest0-21swapper/223:45:462
3384639994133,5cyclictest3428791-21systemctl21:20:012
3384639994132,5cyclictest3457465-21cron21:55:002
3384639994131,7cyclictest261-21systemd-journal22:37:522
3384638994132,6cyclictest3441556-21aten_r2power_en21:35:231
3384638994132,6cyclictest3441080-21systemd-run21:35:011
3384638994132,6cyclictest3441080-21systemd-run21:35:011
3384638994131,6cyclictest3510651-21sendmail23:00:001
3384638994131,6cyclictest3510651-21sendmail23:00:001
3384637994134,5cyclictest421-21dbus-daemon21:15:010
3384637994134,4cyclictest3445206-21systemd-run21:40:010
3384637994120,18cyclictest3506876-21sh22:55:120
3384637994120,18cyclictest3506876-21sh22:55:120
3384637994117,7cyclictest3420616-21diskmemload00:10:000
3384637994112,21cyclictest3524083-21ssh23:15:360
3384639994030,7cyclictest420-21avahi-daemon00:19:262
3384638994031,5cyclictest3412151-21fschecks_time20:40:271
3384637994028,7cyclictest3560734-21ntpq00:00:380
3384637994026,8cyclictest3393080-21ntpq19:35:380
3384637994026,8cyclictest3393080-21ntpq19:35:380
3384637994021,8cyclictest3553235-21ssh23:50:540
3384637994020,4cyclictest3385602-21ntpq19:10:390
3384637994019,7cyclictest3420616-21diskmemload00:30:300
3384637994017,9cyclictest3468033-21diskmemload22:06:420
3384637994017,15cyclictest0-21swapper/019:50:350
3384637994015,7cyclictest3420616-21diskmemload21:22:120
3384637994015,7cyclictest3420616-21diskmemload21:22:120
3384637994014,9cyclictest3390041-21ntpq19:25:360
3384639993933,3cyclictest0-21swapper/200:40:012
3384639993930,6cyclictest1461217-21cups-browsed23:09:482
3384639993925,10cyclictest3540030-21kernelversion23:35:292
3384638993931,4cyclictest3410127-21kworker/u8:1-nfsiod22:20:021
3384638993931,4cyclictest3410127-21kworker/u8:1-nfsiod22:20:021
3384638993930,4cyclictest0-21swapper/121:45:421
3384638993928,6cyclictest3398135-21systemd-run19:55:011
3384637993930,6cyclictest3544455-21ntpq23:40:370
3384637993928,7cyclictest3420077-21ntpq21:05:380
3384637993928,7cyclictest3418556-21ntpq21:00:370
3384637993927,8cyclictest3407971-21ntpq20:25:330
3384637993926,8cyclictest3556588-21ntpq23:55:350
3384637993925,10cyclictest3520064-21ntpq23:10:380
3384637993924,13cyclictest3533384-21ssh23:26:460
3384637993923,6cyclictest3477080-21ssh22:18:210
3384637993923,6cyclictest3477080-21ssh22:18:210
3384637993923,13cyclictest3485782-21ssh22:29:210
3384637993920,16cyclictest3488362-21sh22:31:310
3384637993919,6cyclictest3572960-21ntpq00:15:370
3384637993917,11cyclictest3538473-21sh23:33:480
3384637993916,7cyclictest3502074-21sh22:49:150
3384637993915,7cyclictest3420616-21diskmemload23:36:080
3384639993831,3cyclictest0-21swapper/222:10:452
3384639993830,5cyclictest3405535-21sed20:15:452
3384639993827,7cyclictest3486133-21cron22:30:002
3384639993824,10cyclictest421-21dbus-daemon23:50:092
3384638993830,5cyclictest3535097-21ssh23:30:011
3384638993828,7cyclictest3585717-21unixbench_singl00:30:441
3384638993827,7cyclictest261-21systemd-journal22:02:521
338463899381,30cyclictest3540239-21ssh23:35:321
3384637993828,7cyclictest3401142-21sh20:05:010
3384637993826,8cyclictest3410958-21ntpq20:35:310
3384637993825,8cyclictest3479274-21ntpq22:20:400
3384637993825,8cyclictest3406488-21ntpq20:20:330
3384637993825,10cyclictest0-21swapper/020:20:010
3384637993824,9cyclictest3386968-21ntpq19:15:330
3384637993822,14cyclictest3580110-21ssh00:25:040
3384637993822,14cyclictest3580110-21ssh00:25:040
3384637993817,9cyclictest3527344-21sh23:20:220
3384637993817,7cyclictest3514132-21diskmemload23:03:530
3384637993817,6cyclictest3420616-21diskmemload22:02:120
3384639993730,4cyclictest0-21swapper/221:20:282
3384639993730,4cyclictest0-21swapper/221:20:282
3384639993729,5cyclictest3510631-21ssh23:00:002
3384639993729,5cyclictest3510631-21ssh23:00:002
3384639993729,4cyclictest3424706-21diskmemload21:15:012
3384639993726,8cyclictest3560558-21missed_timers00:00:322
338463999372,29cyclictest3487292-21ssh22:30:362
3384638993730,3cyclictest0-21swapper/122:40:011
3384638993729,5cyclictest3506552-21cron22:55:011
3384637993728,6cyclictest3394502-21ntpq19:40:340
3384637993727,6cyclictest3395976-21ntpq19:45:330
3384637993726,8cyclictest3415475-21ntpq20:50:320
3384637993725,9cyclictest3391511-21ntpq19:30:320
3384637993722,12cyclictest3412252-21irqstats20:40:290
3384637993717,15cyclictest0-21swapper/020:30:330
3384639993628,4cyclictest518-21rs:main2
3384639993628,4cyclictest0-21swapper/221:10:012
338463999361,30cyclictest3524146-21ssh23:15:382
338463999361,16cyclictest347502050ssh22:15:342
338463999361,16cyclictest347502050ssh22:15:342
3384638993629,4cyclictest1051-21mta-sts-daemon00:09:101
3384638993627,6cyclictest1-21systemd19:25:011
338463899362,27cyclictest3434056-21ssh21:25:351
338463899362,27cyclictest3434056-21ssh21:25:351
338463899362,26cyclictest3569012-21ssh00:10:421
3384637993628,6cyclictest3413957-21ntpq20:45:320
3384637993625,7cyclictest3402022-21ntpq20:05:350
3384637993622,12cyclictest3437272-21ssh21:30:100
3384637993622,12cyclictest3437272-21ssh21:30:100
3384637993618,15cyclictest3436241-21sh21:28:390
3384637993618,15cyclictest3436241-21sh21:28:390
338463999353,22cyclictest3576896-21ssh00:20:322
3384639993529,3cyclictest0-21swapper/221:45:302
3384639993527,5cyclictest421-21dbus-daemon20:45:012
3384639993527,5cyclictest3393652-21systemd-run19:40:012
3384639993527,5cyclictest3393652-21systemd-run19:40:012
3384639993524,9cyclictest3458259-21sh21:55:282
3384639993521,10cyclictest3581330-21mailstats00:25:432
3384639993521,10cyclictest3581330-21mailstats00:25:432
3384638993527,5cyclictest3528011-21needreboot23:20:311
3384638993527,5cyclictest1051-21mta-sts-daemon00:15:341
3384638993526,4cyclictest0-21swapper/121:55:331
338463899352,27cyclictest3446465-21ssh21:40:381
3384637993525,6cyclictest3403582-21ntp_states20:10:390
3384639993426,5cyclictest3478821-21if_wlan022:20:292
3384639993425,5cyclictest0-21swapper/220:05:342
3384638993426,5cyclictest3559488-21munin-run00:00:001
3384638993426,5cyclictest3421344-21sort21:10:261
3384638993426,5cyclictest3410112-21munin-plugin-st20:35:011
3384638993426,5cyclictest1054-21mta-sts-daemon23:42:281
3384638993425,6cyclictest3555492-21sshd23:55:021
3384639993325,5cyclictest261-21systemd-journal20:40:012
3384639993324,6cyclictest420-21avahi-daemon19:55:282
338463899334,19cyclictest428-21mta-sts-daemon22:10:361
3384638993327,4cyclictest1051-21mta-sts-daemon22:47:341
3384639993225,3cyclictest0-21swapper/222:45:372
3384639993223,4cyclictest0-21swapper/221:43:192
338463999322,22cyclictest3544416-21ssh23:40:352
3384639993217,10cyclictest421-21dbus-daemon19:35:012
338463899323,23cyclictest3454671-21sh21:50:391
3384638993226,4cyclictest0-21swapper/122:30:001
338463999315,18cyclictest428-21mta-sts-daemon19:40:332
3384639993124,4cyclictest1054-21mta-sts-daemon19:20:022
3384639993123,3cyclictest0-21swapper/221:30:502
3384639993123,3cyclictest0-21swapper/221:30:502
3384639993122,6cyclictest421-21dbus-daemon23:01:192
3384639993120,3cyclictest0-21swapper/222:00:242
338463999311,23cyclictest659-21snmpd19:25:332
3384639993112,16cyclictest351ktimers/219:10:342
338463899313,17cyclictest3515980-21ssh23:05:361
3384638993124,3cyclictest0-21swapper/121:15:361
3384638993123,5cyclictest3526977-21ssh23:20:021
338463899312,23cyclictest3580992-21ssh00:25:331
338463899312,23cyclictest3580992-21ssh00:25:331
3384637993122,6cyclictest3384649-21munin-run19:10:010
3384639993022,5cyclictest3445159-21cron21:40:002
3384639993022,5cyclictest3409205-21if_eth020:30:282
3384638993024,4cyclictest3384636-21cyclictest21:25:031
3384638993024,4cyclictest3384636-21cyclictest21:25:031
3384638993023,4cyclictest3398950-21ntp_kernel_err19:55:311
3384638992923,4cyclictest1051-21mta-sts-daemon19:35:331
3384638992923,4cyclictest1051-21mta-sts-daemon19:35:331
338463899292,21cyclictest3589129-21ssh00:35:351
3384638992921,5cyclictest3394151-21fschecks_count19:40:271
3384638992920,6cyclictest421-21dbus-daemon20:10:021
3384639992822,3cyclictest0-21swapper/220:55:012
3384639992820,5cyclictest3406147-21cut20:20:272
338463899286,14cyclictest3577056-21ssh00:20:371
3384638992819,6cyclictest3405614-21munin-run20:20:011
338463899281,23cyclictest3548503-21ssh23:45:371
3384639992721,3cyclictest0-21swapper/220:30:012
3384639992720,4cyclictest3417594-21latency_hist21:00:012
3384639992716,8cyclictest3396271-21switchtime19:45:442
3384638992720,4cyclictest3420612-21ls21:10:011
3384638992720,4cyclictest1056-21mta-sts-daemon19:45:351
3384638992715,7cyclictest3407274-21latency_hist20:25:021
338463899263,15cyclictest428-21mta-sts-daemon19:25:391
3384638992620,3cyclictest0-21swapper/120:15:011
3384638992618,5cyclictest3419107-21cron21:05:011
338464099253,3cyclictest0-21swapper/320:20:353
338464099252,6cyclictest0-21swapper/321:40:243
338464099251,9cyclictest0-21swapper/322:25:373
3384639992519,3cyclictest0-21swapper/220:05:022
3384639992518,4cyclictest1461217-21cups-browsed19:21:192
3384638992519,3cyclictest0-21swapper/120:45:451
338464099241,7cyclictest0-21swapper/321:50:363
338464099241,6cyclictest0-21swapper/319:30:323
338463999244,12cyclictest1573376-21munin-node20:10:352
338463899244,10cyclictest1055-21mta-sts-daemon20:35:331
3384638992418,4cyclictest1051-21mta-sts-daemon20:25:231
338464099235,4cyclictest0-21swapper/323:35:323
338464099232,8cyclictest0-21swapper/300:35:343
338464099232,7cyclictest0-21swapper/323:00:403
338464099232,6cyclictest0-21swapper/323:50:093
3384639992317,3cyclictest0-21swapper/219:10:012
3384639992316,4cyclictest3397829-21timerandwakeup19:50:442
338463899232,16cyclictest1461217-21cups-browsed20:50:311
3384638992316,6cyclictest271ktimers/119:10:321
3384638992316,4cyclictest0-21swapper/119:30:431
338464099223,6cyclictest0-21swapper/322:30:363
338464099222,8cyclictest0-21swapper/322:05:323
338464099222,7cyclictest0-21swapper/321:20:393
338464099222,7cyclictest0-21swapper/321:20:393
338464099222,6cyclictest0-21swapper/322:40:383
338464099221,9cyclictest0-21swapper/322:35:323
3384639992216,3cyclictest0-21swapper/220:50:002
3384638992213,5cyclictest0-21swapper/120:00:331
338464099213,4cyclictest0-21swapper/300:25:383
338464099213,4cyclictest0-21swapper/300:25:383
338464099212,7cyclictest0-21swapper/323:20:333
338464099201,8cyclictest0-21swapper/323:15:393
338464099201,7cyclictest0-21swapper/322:50:323
338463999201,15cyclictest3418456-21ntpq21:00:322
338464099192,10cyclictest0-21swapper/321:45:403
338464099191,10cyclictest0-21swapper/321:25:373
338464099191,10cyclictest0-21swapper/321:25:373
338464099183,8cyclictest0-21swapper/321:30:393
338464099183,8cyclictest0-21swapper/321:30:393
338464099182,9cyclictest0-21swapper/300:05:413
338464099181,9cyclictest0-21swapper/323:30:423
338464099174,6cyclictest0-21swapper/300:15:363
338464099173,7cyclictest0-21swapper/321:15:383
338464099172,8cyclictest0-21swapper/300:20:323
338464099172,8cyclictest0-21swapper/300:00:433
338464099171,8cyclictest0-21swapper/323:25:353
338464099163,7cyclictest0-21swapper/300:10:343
338464099163,6cyclictest0-21swapper/323:45:373
338464099162,6cyclictest0-21swapper/320:25:333
338464099162,6cyclictest0-21swapper/320:15:353
338464099161,8cyclictest0-21swapper/322:00:423
338464099161,8cyclictest0-21swapper/319:40:363
338464099161,8cyclictest0-21swapper/319:20:313
338464099161,7cyclictest0-21swapper/320:55:333
338464099161,7cyclictest0-21swapper/319:45:333
338464099161,6cyclictest0-21swapper/319:50:323
338464099153,7cyclictest0-21swapper/323:05:283
338464099152,9cyclictest0-21swapper/322:45:393
338464099152,8cyclictest0-21swapper/322:20:273
338464099152,7cyclictest0-21swapper/323:40:323
338464099152,7cyclictest0-21swapper/320:05:383
338464099151,8cyclictest0-21swapper/321:55:343
338464099151,7cyclictest0-21swapper/321:00:363
338464099151,7cyclictest0-21swapper/320:00:353
338464099151,7cyclictest0-21swapper/319:25:393
338464099151,6cyclictest0-21swapper/320:40:333
338464099143,6cyclictest0-21swapper/323:55:353
338464099142,7cyclictest0-21swapper/321:35:353
338464099141,6cyclictest0-21swapper/320:50:313
338464099141,10cyclictest0-21swapper/322:55:243
338464099141,10cyclictest0-21swapper/322:55:243
338463899141,10cyclictest3384592-21munin-run19:10:011
338464099133,6cyclictest0-21swapper/322:10:313
338464099132,6cyclictest0-21swapper/322:15:253
338464099132,6cyclictest0-21swapper/322:15:253
338464099131,7cyclictest0-21swapper/323:10:403
338464099123,5cyclictest0-21swapper/320:50:013
338464099121,7cyclictest0-21swapper/321:10:233
338464099114,5cyclictest0-21swapper/319:35:363
338464099114,5cyclictest0-21swapper/319:35:363
338464099113,6cyclictest0-21swapper/300:30:213
338464099112,6cyclictest0-21swapper/319:20:023
338464099111,6cyclictest0-21swapper/321:05:333
338464099102,5cyclictest0-21swapper/320:10:283
338464099101,5cyclictest0-21swapper/319:55:233
338464099101,5cyclictest0-21swapper/319:10:363
338464099101,4cyclictest0-21swapper/320:35:023
33846409981,5cyclictest0-21swapper/320:35:263
33846409971,3cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional