You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 06:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sat May 02, 2026 00:43:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3399336995747,8cyclictest12350irq/32-eth022:45:420
3399337995144,3cyclictest0-21swapper/122:20:011
3399338994940,6cyclictest0-21swapper/200:20:012
3399338994940,5cyclictest3524002-21readlink23:00:002
3399337994839,6cyclictest3435324-21diskmemload21:25:371
3399336994832,8cyclictest12350irq/32-eth000:15:470
339933799462,35cyclictest3436555-21ssh21:10:431
3399338994537,3cyclictest0-21swapper/221:40:022
3399338994536,6cyclictest3432312-21sendmail21:00:012
3399338994535,6cyclictest421-21dbus-daemon00:40:012
3399338994535,6cyclictest261-21systemd-journal21:54:192
3399337994534,7cyclictest3460583-21needreboot21:40:411
339933899445,32cyclictest3493016-21ssh22:20:462
3399338994437,4cyclictest1051-21mta-sts-daemon22:25:442
339933899443,33cyclictest3589906-21ssh00:20:512
3399337994438,3cyclictest0-21swapper/122:35:521
3399337994438,3cyclictest0-21swapper/100:20:011
339933799433,32cyclictest3517282-21ssh22:50:461
3399337994312,26cyclictest3593907-21ssh00:25:481
3399337994312,26cyclictest3593907-21ssh00:25:481
3399336994336,4cyclictest3443335-21ls21:20:010
3399338994233,5cyclictest3576495-21ssh00:05:012
3399337994236,3cyclictest0-21swapper/121:40:011
3399337994234,5cyclictest3604732-21latency_hist00:40:011
3399337994234,4cyclictest0-21swapper/123:00:471
3399337994233,6cyclictest3399358-21latency_hist19:10:021
3399336994221,15cyclictest151ktimers/019:20:460
339933899415,28cyclictest3485119-21ssh22:10:522
339933899415,28cyclictest3472831-21ssh21:55:452
3399337994134,4cyclictest70-21kswapd021:50:491
3399337994132,4cyclictest0-21swapper/123:30:471
3399336994133,5cyclictest1606395-21cups-browsed19:15:460
3399336994127,8cyclictest12350irq/32-eth000:00:520
3399338994021,4cyclictest171rcu_preempt23:44:302
3399337994035,2cyclictest0-21swapper/121:00:011
339933799403,30cyclictest3597846-21ssh00:30:441
3399337994031,6cyclictest421-21dbus-daemon23:55:571
3399337994031,5cyclictest3452874-21mailstats21:30:531
339933799402,28cyclictest624-21lldpd21:45:461
3399336994033,4cyclictest3523992-21munin-run23:00:010
3399336994032,5cyclictest1606395-21cups-browsed21:00:010
3399336994030,4cyclictest0-21swapper/020:40:470
3399336994027,7cyclictest12450irq/33-eth023:55:500
339933899393,28cyclictest3477044-21ssh22:00:512
3399338993932,4cyclictest3406827-21ls19:35:012
3399338993931,5cyclictest3554090-21sed23:35:562
3399338993930,6cyclictest261-21systemd-journal23:15:012
3399338993929,6cyclictest3582092-21systemctl00:10:552
3399338993928,7cyclictest420-21avahi-daemon22:41:512
3399338993927,9cyclictest3600665-21sh00:35:012
339933899392,33cyclictest3480910-21ssh22:05:442
339933899392,33cyclictest3480910-21ssh22:05:442
3399337993928,7cyclictest261-21systemd-journal21:57:261
339933799392,26cyclictest3577963-21ssh00:05:571
339933699392,31cyclictest3493093-21ssh22:20:490
3399336993913,19cyclictest3553642-21ssh23:35:500
339933899383,22cyclictest3541550-21ssh23:20:512
3399338993830,5cyclictest0-21swapper/220:40:012
3399338993827,7cyclictest3572424-21cupsd00:25:392
3399338993827,7cyclictest3572424-21cupsd00:25:392
3399337993831,5cyclictest3399335-21cyclictest20:55:011
3399337993830,3cyclictest0-21swapper/122:00:461
3399337993829,5cyclictest0-21swapper/122:50:001
3399337993829,3cyclictest0-21swapper/123:30:011
3399336993832,4cyclictest3568384-21munin-plugin-st23:55:010
3399336993832,4cyclictest3568384-21munin-plugin-st23:55:010
3399336993831,4cyclictest3552334-21latency_hist23:35:020
3399336993828,7cyclictest3529334-21sh23:05:450
3399336993827,8cyclictest3581569-21latency00:10:410
3399336993823,12cyclictest3459450-21sendmail21:40:010
3399336993817,4cyclictest0-21swapper/022:30:570
3399336993810,21cyclictest3511973-21kworker/u8:10+xprtiod23:10:450
339933899373,29cyclictest3460842-21ssh21:40:502
3399338993729,5cyclictest421-21dbus-daemon21:50:012
3399338993729,5cyclictest3418811-21cron20:15:012
3399338993726,7cyclictest3517128-21needreboot22:50:422
3399337993731,3cyclictest0-21swapper/121:20:011
3399337993727,7cyclictest3566197-21unixbench_multi23:50:561
3399337993727,7cyclictest3566197-21unixbench_multi23:50:561
3399336993731,4cyclictest3536761-21ssh23:15:360
3399336993729,5cyclictest3448327-21kernelversion21:25:400
3399338993629,3cyclictest0-21swapper/222:40:012
3399338993628,4cyclictest0-21swapper/219:20:422
3399338993627,5cyclictest3409826-21cron19:45:002
3399338993622,11cyclictest0-21swapper/223:45:522
339933799362,28cyclictest3537561-21ssh23:15:531
339933799361,30cyclictest3557680-21sh23:40:501
339933699365,10cyclictest151ktimers/023:20:480
339933699363,26cyclictest3525380-21ssh23:00:500
3399336993629,5cyclictest3598475-21ssh00:30:590
3399336993628,4cyclictest0-21swapper/000:06:020
339933899358,10cyclictest36-21ksoftirqd/223:55:502
3399338993529,4cyclictest3399335-21cyclictest23:05:562
3399338993529,3cyclictest0-21swapper/223:15:362
3399338993526,6cyclictest3443379-21/usr/sbin/munin21:20:012
3399337993527,5cyclictest3420338-21idleruntime-cro20:20:011
3399337993526,5cyclictest3411852-21fschecks_time19:50:381
3399337993526,5cyclictest3411852-21fschecks_time19:50:381
3399336993529,3cyclictest1056-21mta-sts-daemon23:50:020
3399336993527,4cyclictest1054-21mta-sts-daemon21:35:000
3399336993521,7cyclictest12350irq/32-eth021:40:460
3399336993521,4cyclictest151ktimers/000:20:590
3399336993521,11cyclictest3528076-21kworker/u8:13-xprtiod23:40:290
339933899345,22cyclictest3501140-21ssh22:30:472
339933899342,7cyclictest36-21ksoftirqd/221:30:452
3399338993425,6cyclictest0-21swapper/220:05:002
3399338993425,5cyclictest420-21avahi-daemon20:50:382
339933899341,29cyclictest3577809-21sh00:05:512
339933799345,16cyclictest3590029-21ssh00:20:561
3399337993426,5cyclictest3529699-21rm23:05:541
3399337993426,4cyclictest0-21swapper/123:35:451
3399337993426,4cyclictest0-21swapper/100:10:361
3399336993424,7cyclictest3430190-21ntp_kernel_pll_20:50:460
339933899339,7cyclictest36-21ksoftirqd/219:15:462
339933899337,16cyclictest3444742-21ssh21:20:502
3399338993325,5cyclictest3568385-21idleruntime-cro23:55:012
3399338993325,5cyclictest3568385-21idleruntime-cro23:55:012
3399338993324,5cyclictest3435319-21cron21:10:002
339933899332,25cyclictest3525264-21ssh23:00:462
3399338993318,3cyclictest171rcu_preempt20:45:412
3399337993327,3cyclictest0-21swapper/122:35:011
3399337993325,5cyclictest3499910-21cat22:30:021
3399337993325,5cyclictest3485411-21timerandwakeup22:10:571
3399337993325,5cyclictest3409859-21systemd-run19:45:011
339933699335,23cyclictest3444716-21ssh21:20:480
3399336993327,4cyclictest3496800-21latency22:25:400
3399336993327,4cyclictest3480382-21aten_r2power_vo22:05:350
3399336993327,4cyclictest3480382-21aten_r2power_vo22:05:350
3399336993327,4cyclictest151ktimers/021:55:430
3399336993325,5cyclictest3439371-21cron21:15:000
3399336993325,4cyclictest1051-21mta-sts-daemon21:46:360
339933699331,27cyclictest3476958-21ntpq22:00:480
3399336993311,17cyclictest659-21snmpd22:11:480
3399338993224,5cyclictest3449203-21unixbench_multi21:25:572
3399338993223,5cyclictest0-21swapper/223:35:012
339933899322,23cyclictest3436661-21sh21:10:472
3399338993214,12cyclictest3488713-21/usr/sbin/munin22:15:432
3399337993225,3cyclictest0-21swapper/123:20:501
3399337993225,3cyclictest0-21swapper/122:20:451
3399337993224,5cyclictest3401584-21missed_timers19:15:431
339933799322,24cyclictest3573923-21ssh00:00:531
3399336993227,2cyclictest0-21swapper/023:25:490
339933699321,27cyclictest3505143-21ssh22:35:470
339933899317,5cyclictest36-21ksoftirqd/219:50:472
339933899317,5cyclictest36-21ksoftirqd/219:50:472
3399338993125,4cyclictest341rcuc/220:30:012
3399338993115,10cyclictest441-21gpm23:27:072
339933799316,14cyclictest1051-21mta-sts-daemon21:05:431
339933799313,23cyclictest3533329-21sh23:10:441
3399337993123,5cyclictest3480598-21if_eth022:05:391
3399337993123,5cyclictest3480598-21if_eth022:05:391
3399337993123,5cyclictest3416201-21aten_r2power_po20:05:371
3399337993122,6cyclictest439-21systemd-logind19:35:011
339933799312,22cyclictest3509188-21ssh22:40:461
3399336993122,5cyclictest151ktimers/020:15:440
339933699312,24cyclictest3509172-21ssh22:40:450
3399336993117,7cyclictest3409828-21cron19:45:000
339933899303,8cyclictest36-21ksoftirqd/222:45:482
3399338993022,5cyclictest439-21systemd-logind20:10:022
3399338993022,5cyclictest3414325-21awk20:00:002
3399338993021,3cyclictest171rcu_preempt19:10:492
3399337993023,4cyclictest3523992-21munin-run23:00:011
3399337993022,5cyclictest3421861-21systemd-run20:25:011
3399337993022,5cyclictest3408364-21systemd-run19:40:011
3399337993021,6cyclictest3410356-21fschecks_count19:45:381
3399337993021,6cyclictest3410356-21fschecks_count19:45:381
3399337993021,5cyclictest1051-21mta-sts-daemon20:45:521
3399337993019,7cyclictest0-21swapper/121:20:471
3399336993024,4cyclictest3426328-21idleruntime-cro20:40:010
3399336993022,5cyclictest3604662-21cron00:40:000
3399336993021,6cyclictest1606395-21cups-browsed21:52:290
3399336993015,11cyclictest0-21swapper/000:30:010
3399336993015,11cyclictest0-21swapper/000:30:010
339933899297,7cyclictest351ktimers/220:45:012
339933899295,17cyclictest428-21mta-sts-daemon19:35:432
3399338992921,4cyclictest351ktimers/221:00:492
3399338992917,3cyclictest171rcu_preempt19:10:022
3399338992914,12cyclictest3410734-21ntp_kernel_pll_19:45:472
3399338992914,12cyclictest3410734-21ntp_kernel_pll_19:45:472
339933799294,8cyclictest171rcu_preempt20:10:441
3399337992924,2cyclictest0-21swapper/120:00:001
3399337992923,3cyclictest0-21swapper/120:40:011
3399337992921,5cyclictest1054-21mta-sts-daemon20:03:401
3399336992920,7cyclictest3516961-21kernelversion22:50:390
3399336992920,6cyclictest3418702-21unixbench_multi20:10:570
3399336992918,8cyclictest3389752-21kworker/u8:6-xprtiod19:55:000
3399336992918,8cyclictest3389752-21kworker/u8:6-xprtiod19:55:000
3399336992918,8cyclictest0-21swapper/019:25:370
3399336992911,11cyclictest1051-21mta-sts-daemon19:55:530
339933899288,5cyclictest171rcu_preempt19:25:202
3399338992822,4cyclictest341rcuc/220:20:382
3399338992821,4cyclictest351ktimers/220:35:012
3399337992821,3cyclictest0-21swapper/123:49:431
3399337992820,5cyclictest3423236-21unixbench_multi20:25:561
339933699282,22cyclictest421-21dbus-daemon22:19:270
3399336992820,5cyclictest1051-21mta-sts-daemon19:37:040
3399336992820,3cyclictest0-21swapper/020:25:470
3399336992815,9cyclictest0-21swapper/021:10:010
339933899276,11cyclictest36-21ksoftirqd/220:15:442
3399337992722,2cyclictest0-21swapper/119:30:021
3399337992720,4cyclictest518-21rs:main1
3399337992717,8cyclictest3400841-21sh19:15:011
3399336992719,5cyclictest1606395-21cups-browsed19:11:100
3399336992716,9cyclictest3433821-21cron21:05:000
339933799262,3cyclictest171rcu_preempt19:20:431
339933699263,16cyclictest428-21mta-sts-daemon20:00:510
3399336992619,4cyclictest3429207-21tail20:45:580
3399336992519,4cyclictest3389751-21kworker/u8:5-xprtiod19:30:500
339933699232,15cyclictest3410707-21ntpq19:45:450
339933699232,15cyclictest3410707-21ntpq19:45:450
3399336992319,2cyclictest0-21swapper/020:20:490
339933799225,3cyclictest28-21ksoftirqd/120:30:511
3399336992217,2cyclictest0-21swapper/020:30:380
339933799214,3cyclictest171rcu_preempt20:40:511
339933699211,17cyclictest3416807-21ntpq20:05:540
339933999172,8cyclictest0-21swapper/321:20:453
339933999163,7cyclictest0-21swapper/300:30:483
339933999161,7cyclictest0-21swapper/323:40:513
339933699162,12cyclictest0-21swapper/019:10:010
339933999152,7cyclictest0-21swapper/323:15:463
339933999152,7cyclictest0-21swapper/322:45:443
339933999152,7cyclictest0-21swapper/321:40:493
339933999151,8cyclictest0-21swapper/300:20:553
339933999143,6cyclictest0-21swapper/300:00:003
339933999142,8cyclictest0-21swapper/323:30:443
339933999142,8cyclictest0-21swapper/322:40:473
339933999142,6cyclictest0-21swapper/321:10:443
339933999142,5cyclictest0-21swapper/320:05:503
339933999141,8cyclictest0-21swapper/321:45:433
339933999141,8cyclictest0-21swapper/300:25:363
339933999141,8cyclictest0-21swapper/300:25:363
339933999141,7cyclictest0-21swapper/322:10:503
339933999141,7cyclictest0-21swapper/321:50:483
339933999141,6cyclictest0-21swapper/320:30:453
339933999141,5cyclictest0-21swapper/321:00:453
339933999141,5cyclictest0-21swapper/320:50:453
339933999141,5cyclictest0-21swapper/320:00:473
339933999133,7cyclictest0-21swapper/323:00:473
339933999133,6cyclictest0-21swapper/322:55:593
339933999133,5cyclictest0-21swapper/323:05:373
339933999132,7cyclictest0-21swapper/323:25:403
339933999132,6cyclictest0-21swapper/323:45:403
339933999132,6cyclictest0-21swapper/322:50:433
339933999131,8cyclictest0-21swapper/323:10:573
339933999131,8cyclictest0-21swapper/322:30:563
339933999131,7cyclictest0-21swapper/323:20:433
339933999131,7cyclictest0-21swapper/322:16:003
339933999131,7cyclictest0-21swapper/321:35:353
339933999131,7cyclictest0-21swapper/300:10:353
339933999131,7cyclictest0-21swapper/300:05:573
339933999131,7cyclictest0-21swapper/300:00:383
339933999131,6cyclictest0-21swapper/320:20:463
339933999131,3cyclictest0-21swapper/300:35:433
339933999122,6cyclictest0-21swapper/320:45:013
339933999121,7cyclictest0-21swapper/322:35:343
339933999121,7cyclictest0-21swapper/322:00:463
339933999121,7cyclictest0-21swapper/321:55:453
339933999121,7cyclictest0-21swapper/321:15:393
339933999121,6cyclictest0-21swapper/319:15:533
339933999121,5cyclictest0-21swapper/322:10:033
339933999121,5cyclictest0-21swapper/322:10:033
339933999121,5cyclictest0-21swapper/320:25:453
339933999121,5cyclictest0-21swapper/319:50:453
339933999121,5cyclictest0-21swapper/319:50:453
339933999121,5cyclictest0-21swapper/319:45:453
339933999121,5cyclictest0-21swapper/319:45:453
339933999112,6cyclictest0-21swapper/321:30:353
339933999112,5cyclictest0-21swapper/323:35:343
339933999112,5cyclictest0-21swapper/322:25:383
339933999112,5cyclictest0-21swapper/321:25:193
339933999112,5cyclictest0-21swapper/320:50:013
339933999112,5cyclictest0-21swapper/319:35:543
339933999111,7cyclictest0-21swapper/323:50:333
339933999111,7cyclictest0-21swapper/323:50:333
339933999111,6cyclictest0-21swapper/322:20:403
339933999111,6cyclictest0-21swapper/321:05:383
339933999111,6cyclictest0-21swapper/320:55:503
339933999111,6cyclictest0-21swapper/300:15:033
339933999101,6cyclictest0-21swapper/320:10:393
339933999101,6cyclictest0-21swapper/319:40:373
339933999101,5cyclictest0-21swapper/319:10:463
33993399993,4cyclictest0-21swapper/319:20:333
33993399991,5cyclictest0-21swapper/320:15:353
33993399991,5cyclictest0-21swapper/319:55:413
33993399991,5cyclictest0-21swapper/319:30:433
33993399991,4cyclictest0-21swapper/319:10:033
33993399991,3cyclictest0-21swapper/320:35:333
33993399990,5cyclictest0-21swapper/319:25:403
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional