You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 02:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Nov 04, 2025 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1401999260,24cyclictest313-21systemd-journal10:00:000
1403399250,0cyclictest0-21swapper/309:50:363
1403399250,0cyclictest0-21swapper/307:45:363
1403399242,21cyclictest6385-21awk10:15:003
1403399240,0cyclictest0-21swapper/312:00:373
1403399240,0cyclictest0-21swapper/310:30:353
14019992422,1cyclictest13-21ksoftirqd/009:30:000
14033992316,6cyclictest3964-21cron10:10:003
1403399230,22cyclictest12862-21awk11:35:003
1403399230,22cyclictest10578-21dpkg08:10:013
1402899230,22cyclictest7461-21cron09:10:002
1402899230,22cyclictest313-21systemd-journal08:35:002
1402899230,22cyclictest2669-21latency_hist09:00:012
1402899230,22cyclictest10443-21apt12:35:382
14024992321,1cyclictest29425-21apt-get09:55:361
14024992321,1cyclictest27297-21wc12:04:591
1402499230,22cyclictest3240-21cron11:15:001
1402499230,22cyclictest0-21swapper/109:10:371
1401999230,22cyclictest593-21dbus-daemon09:15:000
1401999230,22cyclictest0-21swapper/008:35:000
14033992222,0cyclictest28814-21apt-get07:40:363
14033992220,1cyclictest2163-21sendmail-msp12:20:013
14033992220,1cyclictest19523-21munin-plugin-st09:34:593
14033992220,1cyclictest19523-21munin-plugin-st09:34:593
14033992220,1cyclictest0-21swapper/308:55:013
1403399220,21cyclictest27841-21dump-pmu-power08:44:593
1403399220,21cyclictest0-21swapper/312:20:403
1403399220,21cyclictest0-21swapper/312:10:393
1403399220,21cyclictest0-21swapper/311:55:393
1403399220,21cyclictest0-21swapper/308:55:373
1403399220,1cyclictest5974-21apt-get11:20:373
14028992220,1cyclictest27304-21ls12:05:002
14028992220,1cyclictest24318-21dump-pmu-power09:45:012
14028992220,1cyclictest0-21swapper/211:25:372
1402899220,21cyclictest7797-21apt-get09:10:382
1402899220,21cyclictest6382-21munin-run10:14:592
1402899220,21cyclictest609-21polkitd09:40:012
1402899220,21cyclictest15740-21apt-get08:20:382
1402899220,21cyclictest0-21swapper/211:40:002
1402899220,1cyclictest28721-21apt-get11:00:392
1402899220,1cyclictest1146-21apt-get11:10:392
1402499220,21cyclictest30299-21perf08:50:011
1402499220,21cyclictest25438-21sendmail-msp08:40:001
1402499220,21cyclictest16382-21cron10:35:011
1402499220,21cyclictest0-21swapper/111:20:381
1402499220,21cyclictest0-21swapper/108:10:361
1402499220,21cyclictest0-21swapper/108:05:361
1402499220,21cyclictest0-21swapper/107:35:361
1402499220,21cyclictest0-21swapper/107:15:011
1402499220,0cyclictest0-21swapper/110:00:371
14019992222,0cyclictest0-21swapper/011:10:400
14019992221,0cyclictest3554-21apt-get11:15:390
14019992220,1cyclictest12319-21munin-run09:20:000
14019992220,1cyclictest0-21swapper/008:35:390
1401999220,21cyclictest598-21rs:main0
1401999220,21cyclictest18768-21idleruntime-cro07:20:010
1401999220,21cyclictest14685-21cron09:25:000
1401999220,21cyclictest0-21swapper/012:20:410
1401999220,21cyclictest0-21swapper/011:30:370
1401999220,1cyclictest0-21swapper/011:59:590
1403399212,18cyclictest0-21swapper/311:50:383
14033992120,1cyclictest21212-21munin-run10:45:003
14033992119,2cyclictest0-21swapper/312:25:363
14033992118,3cyclictest0-21swapper/309:15:383
14033992118,2cyclictest17843-21munin-run08:25:003
1403399210,20cyclictest6989-21apt-get10:15:023
1403399210,20cyclictest0-21swapper/312:34:593
1403399210,20cyclictest0-21swapper/311:45:383
1403399210,20cyclictest0-21swapper/310:35:383
1403399210,20cyclictest0-21swapper/309:55:383
1403399210,20cyclictest0-21swapper/308:00:373
1403399210,1cyclictest2299-21ntpq07:50:443
1403399210,1cyclictest0-21swapper/311:05:003
1403399210,1cyclictest0-21swapper/309:35:393
1403399210,0cyclictest0-21swapper/310:20:393
1403399210,0cyclictest0-21swapper/310:20:393
1403399210,0cyclictest0-21swapper/308:45:393
14028992121,0cyclictest0-21swapper/211:45:382
14028992121,0cyclictest0-21swapper/211:15:382
14028992120,0cyclictest0-21swapper/211:40:382
14028992119,1cyclictest0-21swapper/209:35:002
14028992119,1cyclictest0-21swapper/209:35:002
1402899210,21cyclictest0-21swapper/208:25:342
1402899210,20cyclictest11891-21apt-get10:25:392
1402899210,20cyclictest0-21swapper/211:55:392
1402899210,0cyclictest0-21swapper/212:15:372
14024992120,1cyclictest1244-21apt-get07:50:381
14024992120,1cyclictest0-21swapper/109:30:371
14024992120,1cyclictest0-21swapper/109:30:371
14024992120,0cyclictest0-21swapper/107:30:371
14024992118,1cyclictest0-21swapper/109:40:381
1402499210,20cyclictest18791-21idleruntime-cro10:40:011
1402499210,20cyclictest10444-21apt-get12:35:381
1402499210,20cyclictest0-21swapper/109:20:391
1402499210,1cyclictest12876-21awk11:35:001
1402499210,0cyclictest0-21swapper/111:25:371
14019992121,0cyclictest0-21swapper/008:45:390
14019992119,1cyclictest593-21dbus-daemon09:00:000
14019992119,1cyclictest10610-21munin-plugin-st08:10:010
14019992118,2cyclictest13-21ksoftirqd/008:25:000
1401999210,21cyclictest0-21swapper/011:20:380
1401999210,21cyclictest0-21swapper/008:15:360
1401999210,1cyclictest313-21systemd-journal10:55:000
1401999210,0cyclictest0-21swapper/010:10:370
14033992020,0cyclictest13307-21apt-get08:15:363
14033992016,3cyclictest12615-21apt-get09:20:383
1403399200,1cyclictest19067-21/usr/sbin/munin07:20:403
1403399200,1cyclictest11993-21cat12:35:473
1403399200,1cyclictest0-21swapper/309:40:473
1403399200,19cyclictest0-21swapper/311:25:373
1403399200,19cyclictest0-21swapper/311:10:473
1403399200,0cyclictest26293-21apt-get07:35:353
1403399200,0cyclictest0-21swapper/309:05:383
1403399200,0cyclictest0-21swapper/307:55:363
14028992020,0cyclictest7069-21apt-get10:15:372
14028992020,0cyclictest31174-21apt-get11:05:402
14028992020,0cyclictest0-21swapper/210:00:382
14028992020,0cyclictest0-21swapper/209:50:362
14028992020,0cyclictest0-21swapper/209:00:392
14028992019,1cyclictest10769-21apt-get11:30:372
14028992019,0cyclictest23329-21apt-get08:35:392
1402899200,20cyclictest0-21swapper/207:50:382
1402899200,0cyclictest0-21swapper/212:30:372
1402899200,0cyclictest0-21swapper/212:05:402
1402899200,0cyclictest0-21swapper/210:05:372
1402899200,0cyclictest0-21swapper/208:15:352
1402899200,0cyclictest0-21swapper/207:20:352
14024992020,0cyclictest0-21swapper/112:15:371
14024992020,0cyclictest0-21swapper/111:55:391
14024992020,0cyclictest0-21swapper/111:15:391
14024992020,0cyclictest0-21swapper/110:55:381
14024992020,0cyclictest0-21swapper/108:50:371
14024992020,0cyclictest0-21swapper/108:25:341
14024992020,0cyclictest0-21swapper/108:20:391
14024992020,0cyclictest0-21swapper/107:45:361
14024992019,1cyclictest7648-21apt-get12:30:371
14024992019,0cyclictest0-21swapper/109:35:401
1402499200,0cyclictest0-21swapper/112:20:021
1402499200,0cyclictest0-21swapper/107:55:361
1402499200,0cyclictest0-21swapper/107:15:381
14019992020,0cyclictest8368-21apt-get11:25:370
14019992020,0cyclictest31267-21apt-get07:45:360
14019992020,0cyclictest0-21swapper/012:25:370
14019992020,0cyclictest0-21swapper/012:00:370
14019992020,0cyclictest0-21swapper/010:05:380
14019992019,1cyclictest0-21swapper/012:30:370
14019992018,1cyclictest12569-21idleruntime-cro12:40:000
1401999200,0cyclictest0-21swapper/010:40:360
1401999200,0cyclictest0-21swapper/010:15:020
1403399193,15cyclictest16183-21grep09:25:443
14033991919,0cyclictest0-21swapper/310:45:363
14033991919,0cyclictest0-21swapper/307:30:363
14033991919,0cyclictest0-21swapper/307:15:393
14033991918,1cyclictest31897-21apt-get10:00:383
14033991917,1cyclictest12692-21chrt08:11:223
14033991916,2cyclictest5679-21awk11:20:003
14033991916,2cyclictest29693-21awk12:10:013
1403399190,1cyclictest21763-21cstates07:25:383
1403399190,1cyclictest19327-21irqstats08:25:373
1403399190,18cyclictest24503-21grep10:50:443
1403399190,18cyclictest0-21swapper/311:10:013
1403399190,18cyclictest0-21swapper/309:45:383
1403399190,0cyclictest0-21swapper/308:30:363
14028991919,0cyclictest8468-21apt-get08:05:372
14028991919,0cyclictest0-21swapper/212:20:422
14028991919,0cyclictest0-21swapper/211:20:382
14028991918,0cyclictest5250-21apt-get12:25:372
14028991918,0cyclictest16681-21apt-get07:15:392
14024991919,0cyclictest0-21swapper/108:40:371
14024991919,0cyclictest0-21swapper/108:15:351
14024991918,1cyclictest1883-21apt-get10:05:371
14024991917,1cyclictest0-21swapper/108:05:001
14019991919,0cyclictest0-21swapper/012:10:390
14019991919,0cyclictest0-21swapper/009:00:380
14019991916,2cyclictest21920-21grep09:40:010
14033991818,0cyclictest0-21swapper/310:25:383
14033991816,1cyclictest15299-21awk11:40:003
14033991815,3cyclictest0-21swapper/309:00:393
1403399180,17cyclictest0-21swapper/307:10:443
1403399180,0cyclictest0-21swapper/310:55:393
14028991818,0cyclictest28142-21apt-get08:45:392
14028991818,0cyclictest0-21swapper/210:50:412
14028991818,0cyclictest0-21swapper/210:45:362
14028991818,0cyclictest0-21swapper/208:40:372
14024991818,0cyclictest0-21swapper/111:35:381
14024991818,0cyclictest0-21swapper/111:00:401
14024991818,0cyclictest0-21swapper/110:20:401
14024991818,0cyclictest0-21swapper/110:20:401
14024991817,0cyclictest3002-21apt-get09:00:381
14024991816,1cyclictest7352-21munin-run12:30:001
14024991816,1cyclictest21505-21apt-get10:45:351
14024991816,1cyclictest0-21swapper/107:24:591
14019991818,0cyclictest17992-21apt-get11:45:370
14019991818,0cyclictest0-21swapper/012:05:390
14019991818,0cyclictest0-21swapper/010:45:350
14019991818,0cyclictest0-21swapper/010:00:380
14019991818,0cyclictest0-21swapper/008:00:360
14019991818,0cyclictest0-21swapper/007:50:380
14019991818,0cyclictest0-21swapper/007:30:370
14019991817,1cyclictest0-21swapper/010:35:380
1403399170,16cyclictest0-21swapper/311:40:443
1403399170,16cyclictest0-21swapper/309:10:383
14028991717,0cyclictest16695-21apt-get10:35:392
14028991715,1cyclictest0-21swapper/209:15:382
14024991717,0cyclictest27601-21apt-get12:05:381
14024991717,0cyclictest27022-21apt-get09:50:361
14024991717,0cyclictest0-21swapper/110:50:411
14024991717,0cyclictest0-21swapper/110:15:371
14024991717,0cyclictest0-21swapper/109:15:371
14024991715,1cyclictest15595-21apt-get11:40:381
14019991717,0cyclictest0-21swapper/011:05:410
14019991717,0cyclictest0-21swapper/008:40:370
14019991717,0cyclictest0-21swapper/007:55:350
14019991717,0cyclictest0-21swapper/007:35:360
14019991717,0cyclictest0-21swapper/007:10:360
14028991616,0cyclictest10914-21apt-get08:10:372
14028991616,0cyclictest0-21swapper/212:10:382
14028991616,0cyclictest0-21swapper/209:45:372
14028991614,1cyclictest5700-21taskset07:59:472
14028991614,1cyclictest14282-21apt-get07:10:372
1402899160,2cyclictest16618-21sed09:25:452
14024991616,0cyclictest0-21swapper/111:45:371
14024991616,0cyclictest0-21swapper/110:25:391
14024991616,0cyclictest0-21swapper/109:25:371
14024991616,0cyclictest0-21swapper/108:30:371
14024991616,0cyclictest0-21swapper/107:40:361
14024991616,0cyclictest0-21swapper/107:25:371
14024991615,0cyclictest19112-21apt-get10:40:371
14024991614,1cyclictest29999-21apt-get12:10:381
1402499160,6cyclictest0-21swapper/111:05:411
14019991616,0cyclictest0-21swapper/009:50:350
14019991616,0cyclictest0-21swapper/007:20:370
1401999160,15cyclictest28789-21cat07:40:360
14028991515,0cyclictest23895-21apt-get07:30:362
14028991515,0cyclictest0-21swapper/210:40:372
14028991515,0cyclictest0-21swapper/207:35:372
14028991513,2cyclictest28-21ksoftirqd/211:50:422
14028991513,2cyclictest14718-21/usr/sbin/munin10:30:382
1402899150,14cyclictest27205-21memory10:55:432
14024991513,1cyclictest6359-21dump-pmu-power10:15:001
1402499150,14cyclictest20390-21apt-get11:50:381
14019991515,0cyclictest0-21swapper/011:00:400
14019991515,0cyclictest0-21swapper/010:55:390
14019991515,0cyclictest0-21swapper/009:40:380
1401999150,1cyclictest129102sleep008:14:110
1401999150,14cyclictest0-21swapper/010:20:490
1401999150,14cyclictest0-21swapper/010:20:490
1401999150,14cyclictest0-21swapper/007:25:390
14033991414,0cyclictest0-21swapper/308:38:433
14028991414,0cyclictest9496-21apt-get10:20:402
14028991414,0cyclictest9496-21apt-get10:20:392
1402899140,14cyclictest0-21swapper/209:21:182
1402899140,14cyclictest0-21swapper/208:00:062
1402899140,14cyclictest0-21swapper/207:45:252
1402899140,0cyclictest0-21swapper/207:43:012
1402499140,14cyclictest0-21swapper/109:45:291
1402499140,14cyclictest0-21swapper/109:05:431
14019991414,0cyclictest0-21swapper/012:15:420
14019991414,0cyclictest0-21swapper/011:40:370
14019991414,0cyclictest0-21swapper/010:30:190
14019991414,0cyclictest0-21swapper/008:50:360
1401999140,7cyclictest0-21swapper/011:50:370
1401999140,14cyclictest0-21swapper/011:37:030
1401999140,14cyclictest0-21swapper/009:45:490
1401999140,14cyclictest0-21swapper/009:30:280
1401999140,14cyclictest0-21swapper/009:30:280
1401999140,14cyclictest0-21swapper/009:05:100
1401999140,0cyclictest0-21swapper/008:25:360
14028991313,0cyclictest0-21swapper/208:50:372
14028991310,1cyclictest21463-21/usr/sbin/munin07:25:422
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional