You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 09:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Fri Aug 07, 2026 00:43:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2314637995442,7cyclictest151ktimers/000:30:550
2314637995139,9cyclictest0-21swapper/000:30:020
2314639994824,5cyclictest171rcu_preempt22:25:542
2314639994824,5cyclictest171rcu_preempt22:25:542
2314637994833,4cyclictest2387474-21systemd-run22:10:010
2314639994736,8cyclictest2366005-21sh21:35:022
2314637994637,6cyclictest2326624-21cut19:50:000
2314637994637,6cyclictest2326624-21cut19:50:000
2314638994538,3cyclictest0-21swapper/100:05:471
2314637994535,5cyclictest151ktimers/019:40:520
2314639994434,7cyclictest2396623-21cron22:25:012
2314638994436,5cyclictest2347652-21latency_hist21:00:021
2314638994423,13cyclictest271ktimers/122:25:541
2314638994423,13cyclictest271ktimers/122:25:541
2314637994436,5cyclictest2438033-21unixbench_singl23:31:040
2314637994436,5cyclictest2438033-21unixbench_singl23:31:040
2314639994335,5cyclictest2427699-21aten_r2power_cu23:15:442
2314639994334,6cyclictest2356781-21idleruntime-cro21:20:012
2314639994334,6cyclictest2335614-21munin-plugin-st20:20:012
2314638994335,5cyclictest2430444-21latency_hist23:20:031
2314638994335,5cyclictest0-21swapper/122:40:011
2314637994335,5cyclictest2434772-21switchtime23:26:040
2314637994335,5cyclictest2434772-21switchtime23:26:040
231463799432,34cyclictest665-21snmpd22:10:520
231463799432,34cyclictest611083-21cups-browsed21:10:520
2314637994310,25cyclictest2416168-21ssh22:55:540
2314637994232,7cyclictest151ktimers/020:11:010
2314637994134,3cyclictest0-21swapper/022:35:570
2314637994127,9cyclictest151ktimers/020:50:540
2314639994030,6cyclictest2433868-21/usr/sbin/munin23:25:442
2314639994030,6cyclictest2433868-21/usr/sbin/munin23:25:442
2314638994033,4cyclictest1-21systemd22:15:021
2314638994032,5cyclictest439-21systemd-logind20:45:011
2314638994030,7cyclictest2350611-21diskmemload00:00:011
2314638994030,7cyclictest2350611-21diskmemload00:00:011
2314637994024,4cyclictest0-21swapper/022:15:510
2314639993933,3cyclictest0-21swapper/200:40:012
2314639993930,5cyclictest611083-21cups-browsed23:57:182
2314639993930,5cyclictest611083-21cups-browsed23:57:182
2314639993929,7cyclictest2418028-21idleruntime-cro23:00:002
2314638993931,5cyclictest2466563-21awk00:20:011
2314638993931,5cyclictest2410475-21timerwakeupswit22:46:071
2314638993931,5cyclictest2410475-21timerwakeupswit22:46:071
2314638993931,5cyclictest2357981-21awk21:21:021
2314637993932,4cyclictest2378251-21latency_hist21:55:020
2314637993931,5cyclictest2384419-21systemd-run22:05:020
2314637993930,6cyclictest151ktimers/020:45:540
2314637993929,6cyclictest2354438-21fschecks_count21:15:440
2314637993927,9cyclictest2397661-21missed_timers22:25:500
2314637993927,9cyclictest2397661-21missed_timers22:25:500
2314639993831,4cyclictest2433267-21ssh23:25:012
2314639993831,4cyclictest2433267-21ssh23:25:012
231463999382,33cyclictest2407030-21ssh22:40:572
2314638993829,6cyclictest2477146-21threads00:36:091
2314637993832,4cyclictest2411935-21ssh22:50:020
2314637993832,4cyclictest2411935-21ssh22:50:020
2314637993831,4cyclictest0-21swapper/021:21:080
2314637993827,7cyclictest421-21dbus-daemon21:39:430
2314637993827,7cyclictest421-21dbus-daemon21:39:430
2314638993731,3cyclictest0-21swapper/121:10:011
2314637993730,5cyclictest2421085-21cron23:05:010
2314639993630,3cyclictest0-21swapper/200:15:022
2314639993629,4cyclictest2424149-21grep23:10:012
2314639993628,5cyclictest2457697-21latency_hist00:05:022
2314639993625,8cyclictest2367722-21sed21:36:042
2314639993625,8cyclictest2367722-21sed21:36:042
231463899363,26cyclictest665-21snmpd22:55:531
2314638993628,5cyclictest2408890-21systemd-run22:45:011
2314638993628,5cyclictest2378977-21fschecks_time21:55:471
2314638993626,6cyclictest1-21systemd21:26:031
2314638993625,8cyclictest2355184-21switchtime21:16:031
2314637993628,5cyclictest12650irq/32-eth000:01:020
231463999353,25cyclictest2437397-21ssh23:30:502
231463999353,25cyclictest2437397-21ssh23:30:502
2314639993525,6cyclictest2385504-21ntp_kernel_err22:05:512
2314638993527,5cyclictest2328104-21cron19:55:001
2314638993527,5cyclictest1002-21mta-sts-daemon20:30:571
2314638993525,7cyclictest2461175-21cpu00:10:421
2314637993529,3cyclictest0-21swapper/022:25:010
2314637993528,3cyclictest0-21swapper/023:10:010
2314637993527,5cyclictest0-21swapper/000:05:530
2314637993525,7cyclictest2443875-21switchtime23:41:040
2314637993523,8cyclictest2453072-21timerwakeupswit23:56:080
2314637993523,8cyclictest2453072-21timerwakeupswit23:56:080
2314637993519,10cyclictest12650irq/32-eth023:35:580
2314637993518,10cyclictest0-21swapper/000:35:570
2314639993427,4cyclictest2427212-21idleruntime-cro23:15:012
2314639993426,5cyclictest2466550-21cut00:20:012
2314639993426,4cyclictest0-21swapper/200:33:022
2314639993422,8cyclictest2446306-21latency23:45:482
2314639993422,7cyclictest2350611-21diskmemload21:10:532
231463999342,23cyclictest2376441-21ssh21:51:022
2314638993427,4cyclictest2331074-21unixbench_singl20:01:061
2314638993427,4cyclictest1001-21mta-sts-daemon23:25:551
2314638993427,4cyclictest1001-21mta-sts-daemon23:25:551
2314638993427,3cyclictest0-21swapper/123:50:011
2314638993425,6cyclictest487-21rs:main1
2314638993423,8cyclictest2473536-21latency00:30:501
231463799346,23cyclictest12650irq/32-eth000:20:530
2314637993424,8cyclictest2427232-21ssh23:15:020
2314637993422,8cyclictest12650irq/32-eth022:30:560
2314637993422,8cyclictest12650irq/32-eth022:30:560
2314639993328,3cyclictest0-21swapper/223:45:022
2314639993325,5cyclictest2467275-21/usr/sbin/munin00:20:452
2314639993325,5cyclictest2404458-21grep22:36:042
2314639993325,5cyclictest2393585-21sendmail22:20:012
2314639993325,5cyclictest2383080-21unixbench_singl22:01:042
2314639993325,5cyclictest2370141-21needreboot21:40:512
2314639993325,4cyclictest0-21swapper/223:03:142
2314639993322,9cyclictest2470027-21sh00:25:422
2314639993317,12cyclictest2373547-21mailstats21:46:022
231463899334,24cyclictest566-21gmain21:00:541
2314638993328,2cyclictest0-21swapper/121:15:011
2314638993327,3cyclictest0-21swapper/100:05:021
2314638993325,5cyclictest2425894-21unixbench_singl23:11:041
231463899332,22cyclictest2370355-21ssh21:41:001
2314637993326,5cyclictest2363663-21sort21:30:450
2314637993326,4cyclictest2349054-21unixbench_singl21:01:040
2314637993324,6cyclictest1-21systemd23:16:020
2314637993322,6cyclictest2379371-21sh21:55:540
2314637993319,11cyclictest2328104-21cron19:55:000
2314637993316,10cyclictest1000-21mta-sts-daemon22:40:580
231463799331,23cyclictest2431557-21ssh23:21:040
231463799331,23cyclictest2431557-21ssh23:21:040
2314639993224,5cyclictest0-21swapper/222:31:052
2314639993224,5cyclictest0-21swapper/222:31:052
2314639993222,6cyclictest439-21systemd-logind20:10:022
231463999322,23cyclictest2458774-21ssh00:05:592
2314639993218,3cyclictest171rcu_preempt19:15:012
2314639993210,11cyclictest428-21mta-sts-daemon19:50:522
2314638993221,6cyclictest271ktimers/100:20:551
2314637993225,4cyclictest2332632-21systemd-run20:10:020
2314637993224,5cyclictest420-21avahi-daemon19:26:000
2314637993222,8cyclictest2315144-21fschecks_count19:10:440
2314637993221,4cyclictest2347692-21latency_hist21:00:020
2314637993220,9cyclictest0-21swapper/000:20:010
2314637993217,11cyclictest151ktimers/019:35:570
2314639993123,5cyclictest2448884-21telnet23:50:432
2314639993123,5cyclictest2412502-21aten_r2power_po22:50:442
2314639993123,5cyclictest2412502-21aten_r2power_po22:50:442
2314639993123,5cyclictest2409699-21sh22:45:482
2314639993123,5cyclictest2409699-21sh22:45:482
2314639993123,4cyclictest2363015-21latency_hist21:30:032
2314639993120,8cyclictest2389061-21timerwakeupswit22:11:042
2314639993118,3cyclictest171rcu_preempt20:21:042
2314638993126,2cyclictest0-21swapper/121:40:011
2314638993126,2cyclictest0-21swapper/121:40:011
2314638993126,2cyclictest0-21swapper/119:10:521
2314638993124,4cyclictest2442330-21ssh23:40:001
2314638993123,5cyclictest2387607-21wc22:10:021
2314638993123,5cyclictest2322196-21latency_hist19:35:021
2314638993122,7cyclictest2345352-21missed_timers20:50:471
231463899312,21cyclictest2391670-21ssh22:15:541
231463799313,21cyclictest2461706-21ssh00:10:550
231463799312,27cyclictest0-21swapper/019:55:520
2314637993117,11cyclictest2413353-21sh22:51:040
2314637993117,11cyclictest2413353-21sh22:51:040
231463799311,26cyclictest2449939-21grep23:51:060
2314639993024,3cyclictest0-21swapper/223:40:012
2314639993023,4cyclictest2350538-21head21:06:072
2314639993022,5cyclictest2319160-21systemd-run19:25:012
231463899303,21cyclictest2413190-21ssh22:50:591
231463899303,21cyclictest2413190-21ssh22:50:591
2314638993023,4cyclictest0-21swapper/123:30:511
2314638993023,4cyclictest0-21swapper/123:30:511
2314638993022,5cyclictest421-21dbus-daemon22:00:491
2314637993023,3cyclictest151ktimers/021:25:550
2314637993016,10cyclictest151ktimers/020:00:530
2314637993016,10cyclictest12650irq/32-eth021:45:580
2314639992924,2cyclictest0-21swapper/220:05:002
2314639992922,4cyclictest2379978-21unixbench_singl21:56:072
2314639992922,4cyclictest0-21swapper/221:00:012
2314639992922,3cyclictest0-21swapper/219:45:582
2314639992922,3cyclictest0-21swapper/219:45:582
2314639992920,6cyclictest2344619-21munin-plugin-st20:50:012
2314639992920,6cyclictest2322061-21sed19:31:042
2314638992924,2cyclictest0-21swapper/123:00:431
2314638992923,3cyclictest1001-21mta-sts-daemon23:09:221
2314638992922,4cyclictest2402736-21munin-run22:35:001
2314638992922,4cyclictest2402736-21munin-run22:35:001
2314638992922,4cyclictest1000-21mta-sts-daemon23:20:471
2314638992922,4cyclictest1000-21mta-sts-daemon23:20:471
231463899292,22cyclictest2470949-21awk00:26:051
2314638992921,5cyclictest2375615-21/usr/sbin/munin21:50:461
2314638992918,8cyclictest2448954-21aten2.4-expect23:50:441
2314637992918,8cyclictest2370737-21grep21:41:060
2314639992824,2cyclictest0-21swapper/220:25:512
2314639992820,5cyclictest2340102-21cron20:35:012
2314639992820,5cyclictest1002-21mta-sts-daemon19:40:502
2314639992819,6cyclictest2329597-21cron20:00:012
2314639992818,8cyclictest2319836-21latency19:25:482
2314638992822,3cyclictest0-21swapper/122:25:021
2314638992821,4cyclictest2334914-21needreboot20:15:491
2314638992820,5cyclictest420-21avahi-daemon20:20:511
2314637992821,4cyclictest2446847-21switchtime23:46:030
2314637992821,4cyclictest2322140-21sh19:35:010
2314637992814,10cyclictest151ktimers/020:30:550
2314639992722,2cyclictest0-21swapper/219:40:012
2314639992719,5cyclictest2334025-21unixbench_singl20:11:102
231463899272,19cyclictest2326149-21ps19:45:581
231463899272,19cyclictest2326149-21ps19:45:581
2314638992720,4cyclictest1000-21mta-sts-daemon19:15:551
2314638992719,5cyclictest265-21systemd-journal20:40:011
2314638992719,5cyclictest2364662-21head21:31:051
2314638992719,5cyclictest2325124-21latency_hist19:45:011
2314637992718,7cyclictest958-21sendmail-mta21:05:580
2314637992714,9cyclictest421-21dbus-daemon20:40:010
2314639992620,3cyclictest0-21swapper/221:21:022
2314639992617,3cyclictest0-21swapper/221:04:122
231463899262,18cyclictest611083-21cups-browsed19:25:561
2314638992619,4cyclictest2373654-21ssh21:46:051
2314638992619,4cyclictest2338546-21unixbench_singl20:26:021
2314637992622,2cyclictest0-21swapper/020:20:010
2314637992620,3cyclictest0-21swapper/019:10:010
2314639992517,5cyclictest2345996-21unixbench_multi20:51:022
2314638992518,5cyclictest1003-21mta-sts-daemon19:21:001
2314638992516,5cyclictest421-21dbus-daemon20:10:021
2314637992519,4cyclictest1837581-21kworker/u8:0+nfsiod19:20:480
2314637992517,5cyclictest2342150-21fschecks_count20:40:430
2314639992417,4cyclictest1000-21mta-sts-daemon20:40:522
2314638992418,3cyclictest0-21swapper/120:11:071
2314638992417,5cyclictest2344475-21grep20:46:031
231463899241,18cyclictest2329000-21ntpq19:55:531
2314637992413,4cyclictest2338518-21unixbench_multi20:26:020
2314639992318,2cyclictest0-21swapper/219:10:022
2314639992317,3cyclictest0-21swapper/219:16:082
231463899233,13cyclictest443-21gpm19:35:571
231463999221,15cyclictest2341131-21ps20:36:002
231463799221,10cyclictest2336963-21unixbench-2d20:21:040
231463799211,15cyclictest2317006-21ntpq19:15:560
231464099186,6cyclictest0-21swapper/322:10:593
231464099181,7cyclictest0-21swapper/322:05:543
231464099181,6cyclictest0-21swapper/320:35:553
2314638991812,3cyclictest0-21swapper/119:10:011
231464099173,8cyclictest0-21swapper/300:15:513
231464099172,8cyclictest0-21swapper/323:05:533
231464099162,8cyclictest0-21swapper/323:51:023
231464099162,8cyclictest0-21swapper/323:25:143
231464099162,8cyclictest0-21swapper/323:25:143
231464099156,5cyclictest0-21swapper/322:35:493
231464099153,7cyclictest0-21swapper/321:26:013
231464099152,8cyclictest0-21swapper/323:35:513
231464099152,8cyclictest0-21swapper/322:45:573
231464099152,8cyclictest0-21swapper/322:45:573
231464099152,8cyclictest0-21swapper/322:15:513
231464099152,8cyclictest0-21swapper/321:51:003
231464099152,8cyclictest0-21swapper/319:50:013
231464099152,8cyclictest0-21swapper/319:50:013
231464099152,7cyclictest0-21swapper/322:30:463
231464099152,7cyclictest0-21swapper/322:30:463
231464099151,8cyclictest0-21swapper/300:31:023
231464099151,7cyclictest0-21swapper/319:55:583
231464099142,7cyclictest0-21swapper/323:45:513
231464099142,7cyclictest0-21swapper/323:15:493
231464099142,7cyclictest0-21swapper/322:40:563
231464099142,7cyclictest0-21swapper/322:00:403
231464099142,7cyclictest0-21swapper/321:00:013
231464099142,7cyclictest0-21swapper/300:10:573
231464099142,5cyclictest0-21swapper/319:30:513
231464099141,8cyclictest0-21swapper/322:25:563
231464099141,8cyclictest0-21swapper/322:25:563
231464099141,7cyclictest0-21swapper/300:20:443
231464099141,6cyclictest0-21swapper/321:00:533
231464099141,6cyclictest0-21swapper/320:30:523
231464099141,6cyclictest0-21swapper/320:25:493
231464099141,6cyclictest0-21swapper/319:20:593
231464099136,5cyclictest0-21swapper/320:50:463
231464099136,4cyclictest0-21swapper/323:55:543
231464099136,4cyclictest0-21swapper/323:55:543
231464099136,4cyclictest0-21swapper/323:40:553
231464099136,4cyclictest0-21swapper/322:20:553
231464099136,4cyclictest0-21swapper/321:55:443
231464099133,6cyclictest0-21swapper/319:30:013
231464099133,5cyclictest0-21swapper/323:10:423
231464099133,5cyclictest0-21swapper/321:35:473
231464099133,5cyclictest0-21swapper/321:35:473
231464099132,7cyclictest0-21swapper/321:35:033
231464099132,7cyclictest0-21swapper/321:21:073
231464099132,7cyclictest0-21swapper/321:15:453
231464099132,6cyclictest0-21swapper/323:00:423
231464099132,6cyclictest0-21swapper/322:50:443
231464099132,6cyclictest0-21swapper/322:50:443
231464099131,7cyclictest0-21swapper/321:45:523
231464099131,6cyclictest0-21swapper/320:20:523
231464099127,3cyclictest0-21swapper/320:11:103
231464099126,4cyclictest0-21swapper/320:46:003
231464099126,4cyclictest0-21swapper/300:25:463
231464099125,4cyclictest0-21swapper/320:45:013
231464099124,5cyclictest0-21swapper/320:00:553
231464099122,6cyclictest0-21swapper/321:40:463
231464099122,6cyclictest0-21swapper/321:10:013
231464099121,7cyclictest0-21swapper/323:20:443
231464099121,7cyclictest0-21swapper/323:20:443
231464099121,7cyclictest0-21swapper/322:55:533
231464099121,7cyclictest0-21swapper/321:10:533
231464099121,7cyclictest0-21swapper/300:05:583
231464099116,3cyclictest0-21swapper/320:15:503
231464099116,3cyclictest0-21swapper/320:05:573
231464099116,3cyclictest0-21swapper/319:50:433
231464099116,3cyclictest0-21swapper/319:35:563
231464099116,3cyclictest0-21swapper/319:20:013
231464099115,3cyclictest0-21swapper/323:30:303
231464099115,3cyclictest0-21swapper/323:30:303
231464099111,7cyclictest0-21swapper/319:10:023
231464099111,6cyclictest0-21swapper/300:35:443
231464099105,3cyclictest0-21swapper/319:40:463
231464099105,3cyclictest0-21swapper/319:10:463
231464099101,7cyclictest0-21swapper/300:00:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional