You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 12:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sat Jul 12, 2025 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3260899250,2cyclictest21456-21sendmail-msp21:00:003
3260299250,24cyclictest0-21swapper/121:55:421
32595992523,1cyclictest10135-21perf19:30:000
32604992422,1cyclictest17509-21turbostat.cron23:05:002
32608992323,0cyclictest0-21swapper/300:15:413
32608992320,2cyclictest313-21systemd-journal20:10:013
3260899230,1cyclictest16163-21apt-get21:55:413
3260899230,1cyclictest0-21swapper/323:20:003
32604992320,2cyclictest22381-21basename23:15:002
3260499230,1cyclictest17337-21latency_hist19:45:002
32602992323,0cyclictest0-21swapper/121:45:411
32602992321,1cyclictest0-21swapper/119:33:171
3260299230,2cyclictest22050-21awk00:20:001
32595992322,0cyclictest13020-21apt-get22:55:420
3259599230,2cyclictest29590-21idleruntime-cro23:30:000
3259599230,22cyclictest612-21gdbus20:55:000
3259599230,22cyclictest5118-21apt22:40:000
3260899220,21cyclictest16856-21cron00:10:003
3260899220,21cyclictest0-21swapper/300:35:003
32604992220,1cyclictest0-21swapper/221:25:002
3260499220,21cyclictest4957-21idleruntime-cro19:20:002
3260499220,21cyclictest388-21apt-get19:10:402
3260499220,21cyclictest0-21swapper/223:50:392
3260499220,1cyclictest609-21polkitd19:40:002
3260499220,1cyclictest3463-21idleruntime-cro21:30:002
3260499220,1cyclictest25815-21apt-get22:15:432
32602992222,0cyclictest0-21swapper/119:15:401
3260299220,21cyclictest29700-21apt-get20:10:401
3260299220,21cyclictest12063-21awk00:00:001
3260299220,21cyclictest0-21swapper/122:40:401
3260299220,21cyclictest0-21swapper/122:20:421
3260299220,21cyclictest0-21swapper/121:40:441
3260299220,21cyclictest0-21swapper/120:40:411
3260299220,21cyclictest0-21swapper/120:30:391
3260299220,21cyclictest0-21swapper/120:05:421
3260299220,21cyclictest0-21swapper/119:50:391
32595992222,0cyclictest0-21swapper/020:15:420
32595992220,1cyclictest15477-21perf21:55:000
3259599220,21cyclictest31067-21perf21:20:010
3259599220,21cyclictest22376-21apt-get00:20:410
3259599220,21cyclictest14220-21munin-run20:45:000
3259599220,21cyclictest0-21swapper/022:40:400
3259599220,21cyclictest0-21swapper/000:25:390
3259599220,21cyclictest0-21swapper/000:05:390
3259599220,1cyclictest28655-21perf21:14:590
3259599220,1cyclictest24792-21munin-run23:20:000
3259599220,1cyclictest14483-21munin-run00:05:010
3259599220,1cyclictest0-21swapper/000:19:590
32608992120,1cyclictest30194-21apt-get23:30:033
32608992119,1cyclictest593-21dbus-daemon21:45:013
32608992119,1cyclictest19976-21perf23:10:003
32608992119,1cyclictest0-21swapper/322:00:403
32608992118,2cyclictest1773-21cron20:20:003
3260899210,21cyclictest0-21swapper/323:50:403
3260899210,1cyclictest12845-21apt-get19:35:403
3260899210,1cyclictest12535-21turbostat.cron19:34:593
3260899210,1cyclictest0-21swapper/319:10:393
3260899210,0cyclictest24916-21apt-get20:00:433
3260899210,0cyclictest0-21swapper/323:20:393
3260899210,0cyclictest0-21swapper/322:45:013
32604992121,0cyclictest19301-21apt-get20:55:402
32604992120,1cyclictest0-21swapper/221:45:412
3260499210,2cyclictest612-21gdbus23:30:012
3260499210,21cyclictest0-21swapper/200:15:412
3260499210,20cyclictest0-21swapper/223:55:442
3260499210,20cyclictest0-21swapper/221:50:392
3260499210,20cyclictest0-21swapper/219:45:402
3260499210,0cyclictest14773-21apt-get00:05:392
3260499210,0cyclictest12379-21apt-get00:00:382
3260499210,0cyclictest0-21swapper/200:35:412
32602992121,0cyclictest0-21swapper/120:00:431
32602992120,1cyclictest25087-21apt-get23:20:391
32602992120,1cyclictest22670-21apt-get23:15:401
32602992120,1cyclictest0-21swapper/119:25:031
32602992119,1cyclictest19011-21munin-run20:55:001
32602992119,1cyclictest0-21swapper/119:25:001
3260299210,20cyclictest0-21swapper/122:05:391
3260299210,20cyclictest0-21swapper/119:10:391
3260299210,20cyclictest0-21swapper/100:20:411
3260299210,1cyclictest0-21swapper/121:30:401
3260299210,0cyclictest6161-21apt-get21:35:401
3260299210,0cyclictest6161-21apt-get21:35:401
3260299210,0cyclictest0-21swapper/123:10:431
32595992120,0cyclictest30622-21apt-get22:25:400
32595992119,1cyclictest10685-21munin-run21:45:000
32595992119,1cyclictest0-21swapper/022:00:410
3259599210,21cyclictest0-21swapper/019:35:420
3259599210,20cyclictest0-21swapper/022:10:430
3259599210,20cyclictest0-21swapper/021:25:390
3259599210,20cyclictest0-21swapper/000:30:410
3259599210,0cyclictest0-21swapper/019:30:380
32608992020,0cyclictest0-21swapper/323:45:013
32608992020,0cyclictest0-21swapper/322:50:403
32608992020,0cyclictest0-21swapper/322:15:433
32608992020,0cyclictest0-21swapper/321:45:413
32608992020,0cyclictest0-21swapper/320:25:413
32608992020,0cyclictest0-21swapper/319:45:403
32608992020,0cyclictest0-21swapper/300:00:383
32608992017,2cyclictest7177-21wc23:50:013
3260899200,0cyclictest0-21swapper/322:55:413
3260899200,0cyclictest0-21swapper/321:50:403
3260899200,0cyclictest0-21swapper/300:10:423
32604992020,0cyclictest0-21swapper/223:40:392
32604992020,0cyclictest0-21swapper/223:15:402
32604992020,0cyclictest0-21swapper/222:45:382
32604992020,0cyclictest0-21swapper/222:40:402
32604992020,0cyclictest0-21swapper/222:25:412
32604992020,0cyclictest0-21swapper/220:00:432
32604992020,0cyclictest0-21swapper/219:20:382
32604992019,1cyclictest30277-21apt-get23:30:432
3260499200,20cyclictest0-21swapper/220:20:402
3260499200,1cyclictest15114-21wc23:00:002
3260499200,0cyclictest32097-21apt-get20:15:422
3260499200,0cyclictest0-21swapper/221:10:402
3260499200,0cyclictest0-21swapper/221:05:402
3260499200,0cyclictest0-21swapper/220:40:422
32602992020,0cyclictest27480-21apt-get23:25:391
32602992020,0cyclictest0-21swapper/123:00:421
32602992020,0cyclictest0-21swapper/122:10:441
32602992020,0cyclictest0-21swapper/121:20:411
32602992020,0cyclictest0-21swapper/121:10:401
32602992019,0cyclictest0-21swapper/120:35:401
32602992018,1cyclictest4781-21sh23:45:001
32602992017,2cyclictest19804-21perf19:50:011
3260299200,0cyclictest13369-21apt-get21:50:391
3260299200,0cyclictest10622-21apt-get22:50:401
3260299200,0cyclictest0-21swapper/121:15:391
32595992020,0cyclictest574-21apt-get22:30:400
32595992020,0cyclictest0-21swapper/023:50:400
32595992020,0cyclictest0-21swapper/023:30:430
32595992020,0cyclictest0-21swapper/023:20:390
32595992020,0cyclictest0-21swapper/022:05:390
32595992019,1cyclictest0-21swapper/019:55:390
32595992016,3cyclictest5828-21cron21:35:000
3259599200,19cyclictest0-21swapper/019:15:400
3259599200,0cyclictest0-21swapper/023:45:380
3259599200,0cyclictest0-21swapper/022:20:410
3259599200,0cyclictest0-21swapper/020:25:410
3259599200,0cyclictest0-21swapper/000:10:420
32608991919,0cyclictest3762-21apt-get21:30:393
32608991919,0cyclictest32679-21apt-get23:35:413
32608991919,0cyclictest0-21swapper/322:30:403
32608991919,0cyclictest0-21swapper/322:20:423
32608991919,0cyclictest0-21swapper/321:15:393
32608991919,0cyclictest0-21swapper/319:15:403
32608991919,0cyclictest0-21swapper/300:35:413
32608991918,1cyclictest17358-21munin-run19:45:003
32608991917,1cyclictest2115-21apt-get20:20:403
32604991919,0cyclictest0-21swapper/221:00:412
32602991919,0cyclictest0-21swapper/123:45:391
32602991919,0cyclictest0-21swapper/123:35:401
32602991919,0cyclictest0-21swapper/122:30:401
32602991919,0cyclictest0-21swapper/121:25:391
32602991919,0cyclictest0-21swapper/120:45:391
32602991919,0cyclictest0-21swapper/100:25:391
32602991919,0cyclictest0-21swapper/100:05:001
32595991919,0cyclictest9696-21apt-get20:35:390
32595991919,0cyclictest0-21swapper/023:55:440
32595991919,0cyclictest0-21swapper/023:40:400
32595991919,0cyclictest0-21swapper/020:55:400
32595991919,0cyclictest0-21swapper/020:30:390
32595991919,0cyclictest0-21swapper/019:50:390
32595991919,0cyclictest0-21swapper/019:20:380
32595991917,1cyclictest593-21dbus-daemon20:05:000
32595991917,1cyclictest10975-21apt-get21:45:410
32595991917,1cyclictest0-21swapper/023:10:000
32608991818,0cyclictest31373-21apt-get21:20:413
32608991818,0cyclictest0-21swapper/323:25:393
32608991818,0cyclictest0-21swapper/320:50:393
32608991818,0cyclictest0-21swapper/320:10:413
32608991817,0cyclictest24164-21apt-get21:05:403
32604991818,0cyclictest24768-21apt-get00:25:402
32604991818,0cyclictest18577-21apt-get22:00:402
32604991818,0cyclictest0-21swapper/223:45:392
32604991818,0cyclictest0-21swapper/220:50:392
32604991818,0cyclictest0-21swapper/220:40:002
32604991818,0cyclictest0-21swapper/220:05:422
32604991816,1cyclictest17480-21apt-get00:10:022
32602991818,0cyclictest0-21swapper/120:20:401
32602991815,2cyclictest6582-21latency_hist20:30:001
32595991818,0cyclictest0-21swapper/022:45:390
32595991818,0cyclictest0-21swapper/020:20:410
32608991717,0cyclictest9880-21apt-get23:55:443
32608991717,0cyclictest6906-21apt-get20:30:393
32608991717,0cyclictest0-21swapper/322:25:403
32608991717,0cyclictest0-21swapper/321:00:423
32608991717,0cyclictest0-21swapper/319:50:393
32608991717,0cyclictest0-21swapper/319:20:393
32608991717,0cyclictest0-21swapper/300:25:393
32608991715,1cyclictest583-21avahi-daemon21:14:593
32608991715,1cyclictest313-21systemd-journal20:50:003
32604991717,0cyclictest0-21swapper/222:35:432
32604991717,0cyclictest0-21swapper/221:55:412
32604991717,0cyclictest0-21swapper/219:30:382
32604991715,1cyclictest0-21swapper/219:55:012
32602991717,0cyclictest0-21swapper/122:00:401
32602991717,0cyclictest0-21swapper/120:55:401
32602991717,0cyclictest0-21swapper/100:05:391
32595991717,0cyclictest0-21swapper/023:35:410
32595991717,0cyclictest0-21swapper/023:10:420
32595991717,0cyclictest0-21swapper/021:55:420
32595991717,0cyclictest0-21swapper/021:35:400
32595991717,0cyclictest0-21swapper/021:35:400
32595991717,0cyclictest0-21swapper/021:00:410
32608991616,0cyclictest20279-21apt-get23:10:423
32608991616,0cyclictest0-21swapper/322:35:443
32604991616,0cyclictest0-21swapper/223:35:402
32604991616,0cyclictest0-21swapper/222:05:402
32604991616,0cyclictest0-21swapper/221:15:392
32604991615,1cyclictest0-21swapper/220:45:402
32604991614,1cyclictest0-21swapper/219:55:472
32604991614,1cyclictest0-21swapper/200:20:512
3260499160,6cyclictest0-21swapper/220:30:392
3260499160,1cyclictest25555-21latency_hist22:15:002
3260499160,15cyclictest8551-21/usr/sbin/munin21:40:512
3260499160,15cyclictest313-21systemd-journal23:10:002
3260499160,15cyclictest0-21swapper/219:25:032
3260499160,13cyclictest583-21avahi-daemon23:25:012
32602991616,0cyclictest0-21swapper/123:30:031
32602991616,0cyclictest0-21swapper/122:55:411
32602991616,0cyclictest0-21swapper/100:10:421
32595991616,0cyclictest0-21swapper/019:10:390
32595991615,1cyclictest16573-21cron20:50:000
32595991614,1cyclictest17658-21chrt19:45:400
3259599160,16cyclictest0-21swapper/019:40:370
3259599160,15cyclictest29455-21munin-run20:10:000
32608991515,0cyclictest5431-21apt-get22:40:413
32608991515,0cyclictest12122-21apt-get20:40:423
32608991514,1cyclictest0-21swapper/319:55:393
3260899150,14cyclictest22072-21ntp_states22:05:453
32604991515,0cyclictest0-21swapper/221:30:392
32604991513,1cyclictest0-21swapper/222:35:002
3260499150,14cyclictest0-21swapper/220:25:462
3260499150,0cyclictest0-21swapper/220:10:402
32602991515,0cyclictest0-21swapper/119:55:401
32602991515,0cyclictest0-21swapper/119:40:371
32602991515,0cyclictest0-21swapper/100:30:411
3259599151,13cyclictest0-21swapper/023:05:000
32608991414,0cyclictest0-21swapper/320:35:473
3260899140,14cyclictest0-21swapper/323:02:073
3260899140,14cyclictest0-21swapper/322:10:063
3260899140,14cyclictest0-21swapper/321:35:053
3260899140,14cyclictest0-21swapper/321:35:053
3260899140,14cyclictest0-21swapper/321:25:103
3260899140,14cyclictest0-21swapper/300:20:283
3260899140,0cyclictest0-21swapper/319:28:043
32604991414,0cyclictest0-21swapper/221:36:032
32604991414,0cyclictest0-21swapper/221:36:022
3260499140,14cyclictest0-21swapper/222:50:292
3260499140,14cyclictest0-21swapper/222:21:022
3260499140,14cyclictest0-21swapper/200:32:292
32602991414,0cyclictest0-21swapper/123:50:401
32602991414,0cyclictest0-21swapper/123:05:151
32602991414,0cyclictest0-21swapper/122:25:581
32602991414,0cyclictest0-21swapper/122:15:071
3260299140,14cyclictest0-21swapper/121:05:071
3260299140,0cyclictest0-21swapper/122:35:331
3260299140,0cyclictest0-21swapper/119:35:001
32595991414,0cyclictest0-21swapper/022:50:140
32595991414,0cyclictest0-21swapper/022:15:030
32595991414,0cyclictest0-21swapper/000:35:430
3259599140,14cyclictest0-21swapper/021:20:340
3259599140,14cyclictest0-21swapper/021:05:590
3259599140,14cyclictest0-21swapper/020:10:350
32602991313,0cyclictest0-21swapper/122:45:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional