You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 14:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Tue May 13, 2025 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14583992719,8cyclictest0-21swapper/221:20:332
14583992519,1cyclictest593-21dbus-daemon00:15:002
14584992321,1cyclictest12228-21apt-get00:35:333
14584992321,1cyclictest0-21swapper/300:35:013
1458499230,22cyclictest511-21latency_hist20:54:593
14583992321,1cyclictest0-21swapper/219:45:002
14583992320,2cyclictest16647-21awk22:34:592
1458399230,3cyclictest21456-21dump-pmu-power22:44:592
1458399230,22cyclictest6756-21awk00:25:002
1458399230,22cyclictest313-21systemd-journal23:15:002
1458399230,22cyclictest29414-21latency_hist21:55:002
14582992321,1cyclictest0-21swapper/123:30:361
1458299230,22cyclictest21447-21sh22:45:001
1458299230,22cyclictest0-21swapper/121:15:311
1458299230,1cyclictest21005-21awk20:30:001
1458299230,1cyclictest0-21swapper/119:55:321
14578992321,1cyclictest24643-21munin-run21:45:000
14578992321,1cyclictest10459-21apt-get21:15:310
14584992222,0cyclictest0-21swapper/323:20:023
14584992220,1cyclictest17425-21turbostat21:30:003
1458499220,21cyclictest593-21dbus-daemon21:10:003
1458499220,21cyclictest23853-21apt22:49:593
1458499220,21cyclictest23462-21apt-get23:55:003
1458499220,21cyclictest19381-21sendmail-msp19:20:003
1458499220,21cyclictest0-21swapper/320:25:353
1458499220,21cyclictest0-21swapper/319:30:343
1458499220,21cyclictest0-21swapper/300:00:003
14583992222,0cyclictest0-21swapper/200:25:342
14583992221,0cyclictest22532-21apt-get21:40:322
14583992220,1cyclictest6222-21latency_hist23:20:002
14583992220,1cyclictest28778-21awk00:05:002
14583992220,1cyclictest11420-21perf23:29:592
14583992219,2cyclictest1035-21apt-get23:10:002
1458399220,21cyclictest25816-21basename20:39:592
1458399220,21cyclictest24154-21apt-get22:50:342
1458399220,21cyclictest0-21swapper/200:05:022
1458399220,1cyclictest9022-21munin-run22:20:002
1458399220,1cyclictest12624-21sendmail21:20:012
1458399220,1cyclictest11783-21apt-get23:30:352
14582992220,1cyclictest6249-21awk23:19:591
14582992220,1cyclictest19669-21apt-get19:20:331
1458299220,21cyclictest7050-21apt-get00:25:361
1458299220,21cyclictest5383-21apt21:05:001
1458299220,21cyclictest4196-21sh22:10:011
1458299220,21cyclictest3711-21cron19:55:001
1458299220,21cyclictest29413-21turbostat.cron21:55:001
14578992220,1cyclictest23679-21apt-get20:35:330
14578992220,1cyclictest0-21swapper/020:55:330
1457899220,21cyclictest32127-21apt-get22:00:330
1457899220,21cyclictest17731-21apt-get21:30:340
1457899220,21cyclictest0-21swapper/023:20:020
1457899220,21cyclictest0-21swapper/020:40:340
1457899220,0cyclictest1631-21apt-get19:50:320
14584992121,0cyclictest0-21swapper/322:50:333
14584992120,0cyclictest22059-21apt-get19:25:323
14584992119,1cyclictest16313-21awk23:40:003
1458499210,20cyclictest9710-21apt-get22:20:343
1458499210,20cyclictest0-21swapper/322:05:303
1458499210,20cyclictest0-21swapper/320:00:323
1458499210,1cyclictest29389-21apt-get00:05:023
1458499210,1cyclictest15040-21perf21:25:013
1458499210,0cyclictest0-21swapper/321:55:333
14583992121,0cyclictest0-21swapper/222:25:342
14583992121,0cyclictest0-21swapper/221:45:322
14583992119,1cyclictest1322-21dump-pmu-power19:50:012
14583992119,1cyclictest0-21swapper/222:10:012
1458399210,20cyclictest0-21swapper/223:00:332
1458399210,20cyclictest0-21swapper/221:35:332
1458399210,1cyclictest16194-21sendmail-msp20:20:002
1458399210,1cyclictest11231-21apt-get20:10:302
1458399210,0cyclictest0-21swapper/219:50:322
14582992121,0cyclictest0-21swapper/123:40:351
14582992121,0cyclictest0-21swapper/122:10:341
14582992121,0cyclictest0-21swapper/121:55:321
14582992119,1cyclictest26546-21apt-get22:55:331
14582992119,1cyclictest14266-21awk22:30:001
1458299210,20cyclictest0-21swapper/123:05:331
1458299210,20cyclictest0-21swapper/122:15:301
1458299210,20cyclictest0-21swapper/121:05:341
1458299210,20cyclictest0-21swapper/119:10:311
1458299210,1cyclictest0-21swapper/122:40:001
14578992121,0cyclictest5675-21apt-get21:05:330
14578992121,0cyclictest0-21swapper/022:15:310
14578992121,0cyclictest0-21swapper/019:20:320
14578992120,1cyclictest593-21dbus-daemon21:40:000
14578992120,0cyclictest6448-21apt-get20:00:320
14578992119,1cyclictest593-21dbus-daemon21:54:590
1457899210,20cyclictest24926-21apt-get21:45:330
1457899210,20cyclictest14858-21apt-get19:10:310
1457899210,20cyclictest0-21swapper/020:25:350
1457899210,0cyclictest0-21swapper/020:45:340
1457899210,0cyclictest0-21swapper/000:10:330
14584992020,0cyclictest0-21swapper/322:30:333
14584992020,0cyclictest0-21swapper/321:40:313
14584992020,0cyclictest0-21swapper/321:10:313
14584992020,0cyclictest0-21swapper/320:45:333
14584992020,0cyclictest0-21swapper/320:40:343
14584992020,0cyclictest0-21swapper/319:45:333
14584992018,1cyclictest11408-21sh23:30:003
14584992018,1cyclictest0-21swapper/320:25:003
14584992016,3cyclictest12638-21perf21:20:013
1458499200,20cyclictest0-21swapper/321:50:343
1458499200,0cyclictest0-21swapper/320:30:293
1458499200,0cyclictest0-21swapper/319:55:323
1458499200,0cyclictest0-21swapper/300:20:343
1458499200,0cyclictest0-21swapper/300:10:333
14583992020,0cyclictest0-21swapper/223:55:012
14583992020,0cyclictest0-21swapper/222:10:352
14583992020,0cyclictest0-21swapper/219:35:352
14583992020,0cyclictest0-21swapper/219:10:312
14583992020,0cyclictest0-21swapper/200:15:342
14583992019,0cyclictest852-21apt-get20:55:332
14583992019,0cyclictest6237-21apt-get23:20:022
1458399200,1cyclictest0-21swapper/220:50:002
1458399200,19cyclictest0-21swapper/223:35:352
1458399200,0cyclictest0-21swapper/221:30:332
14582992020,0cyclictest9041-21perf23:25:001
14582992020,0cyclictest28489-21apt-get20:45:341
14582992020,0cyclictest0-21swapper/123:35:341
14582992020,0cyclictest0-21swapper/123:25:351
14582992020,0cyclictest0-21swapper/122:30:331
14582992020,0cyclictest0-21swapper/120:50:351
14582992020,0cyclictest0-21swapper/120:00:321
1458299200,20cyclictest0-21swapper/121:45:321
1458299200,19cyclictest598-21rs:main1
1458299200,0cyclictest0-21swapper/100:35:341
14578992020,0cyclictest21385-21apt-get23:50:360
14578992020,0cyclictest19369-21apt-get22:40:340
14578992020,0cyclictest15031-21turbostat.cron21:25:010
14578992020,0cyclictest0-21swapper/022:40:000
14578992020,0cyclictest0-21swapper/019:40:310
14578992019,1cyclictest29713-21apt-get21:55:320
14578992018,1cyclictest0-21swapper/023:30:000
1457899200,20cyclictest0-21swapper/019:45:320
1457899200,20cyclictest0-21swapper/000:25:350
1457899200,20cyclictest0-21swapper/000:05:330
1457899200,0cyclictest0-21swapper/020:50:350
14584991919,0cyclictest0-21swapper/323:40:343
14584991919,0cyclictest0-21swapper/323:15:363
14584991919,0cyclictest0-21swapper/322:55:333
14584991919,0cyclictest0-21swapper/322:40:343
14584991919,0cyclictest0-21swapper/322:10:353
14584991919,0cyclictest0-21swapper/320:35:323
14584991919,0cyclictest0-21swapper/320:05:343
14583991919,0cyclictest0-21swapper/223:50:362
14583991919,0cyclictest0-21swapper/221:25:312
14583991917,1cyclictest21780-21munin-run19:25:002
14583991917,1cyclictest17251-21apt-get19:15:332
14582991919,0cyclictest24172-21apt-get23:55:331
14582991919,0cyclictest0-21swapper/122:20:001
14582991919,0cyclictest0-21swapper/120:55:331
14582991919,0cyclictest0-21swapper/120:05:341
14582991919,0cyclictest0-21swapper/119:30:331
1458299190,0cyclictest0-21swapper/121:30:341
14578991919,0cyclictest0-21swapper/023:55:010
14578991919,0cyclictest0-21swapper/022:50:340
14578991919,0cyclictest0-21swapper/022:45:330
14578991919,0cyclictest0-21swapper/022:05:300
14578991919,0cyclictest0-21swapper/021:10:300
14578991919,0cyclictest0-21swapper/019:15:330
14578991919,0cyclictest0-21swapper/000:00:320
14578991917,1cyclictest9836-21apt-get00:30:340
14578991917,1cyclictest10936-21cron20:10:000
1457899190,17cyclictest0-21swapper/022:25:340
14584991818,0cyclictest26675-21apt-get00:00:333
14584991818,0cyclictest0-21swapper/323:30:363
14584991818,0cyclictest0-21swapper/322:35:333
14584991818,0cyclictest0-21swapper/322:25:343
14584991818,0cyclictest0-21swapper/320:15:333
14584991818,0cyclictest0-21swapper/319:20:323
14584991818,0cyclictest0-21swapper/300:25:363
14584991817,1cyclictest26854-21apt-get19:35:353
14584991817,0cyclictest0-21swapper/323:10:333
14583991818,0cyclictest11933-21perf00:35:002
14583991818,0cyclictest0-21swapper/222:00:342
14583991818,0cyclictest0-21swapper/221:55:332
14583991818,0cyclictest0-21swapper/221:08:342
14583991817,1cyclictest16598-21apt-get23:40:352
14583991816,1cyclictest0-21swapper/219:30:342
1458399180,17cyclictest0-21swapper/220:00:322
14582991818,0cyclictest16500-21apt-get20:20:311
14582991818,0cyclictest0-21swapper/123:10:331
14582991818,0cyclictest0-21swapper/122:50:341
14582991818,0cyclictest0-21swapper/121:40:321
14582991818,0cyclictest0-21swapper/121:10:301
14582991818,0cyclictest0-21swapper/119:15:331
14582991818,0cyclictest0-21swapper/100:15:021
14582991818,0cyclictest0-21swapper/100:10:311
14582991817,0cyclictest0-21swapper/121:35:331
1458299180,16cyclictest1686-21apache200:32:041
1458299180,0cyclictest26099-21apt-get20:40:341
14578991818,0cyclictest0-21swapper/023:45:320
14578991818,0cyclictest0-21swapper/019:25:320
14578991817,1cyclictest3282-21apt-get21:00:340
14578991817,1cyclictest15330-21apt-get21:25:320
14578991816,1cyclictest0-21swapper/020:15:010
14584991717,0cyclictest18990-21apt-get23:45:313
14584991717,0cyclictest0-21swapper/321:36:563
14584991717,0cyclictest0-21swapper/319:50:313
14584991717,0cyclictest0-21swapper/319:40:313
14584991715,1cyclictest4370-21basename00:20:003
14583991717,0cyclictest30881-21apt-get20:50:352
14583991717,0cyclictest0-21swapper/223:45:322
14583991717,0cyclictest0-21swapper/221:00:342
14583991717,0cyclictest0-21swapper/220:40:332
14583991717,0cyclictest0-21swapper/220:30:292
14583991715,1cyclictest6120-21cron20:00:002
14582991717,0cyclictest0-21swapper/123:50:351
14582991717,0cyclictest0-21swapper/122:45:331
1458299170,16cyclictest0-21swapper/100:20:341
14578991717,0cyclictest4657-21apt-get00:20:350
14578991717,0cyclictest0-21swapper/023:35:350
14578991717,0cyclictest0-21swapper/019:55:320
14578991715,1cyclictest21281-21apt-get20:30:290
14578991715,1cyclictest18681-21cron23:45:010
14578991715,1cyclictest11872-21munin-run22:25:000
14584991616,0cyclictest0-21swapper/322:15:313
14584991616,0cyclictest0-21swapper/321:00:353
14584991616,0cyclictest0-21swapper/320:13:353
14584991614,1cyclictest0-21swapper/319:10:303
1458499160,16cyclictest0-21swapper/322:00:383
14583991616,0cyclictest0-21swapper/220:20:322
14583991614,1cyclictest14297-21cron00:40:002
14582991616,0cyclictest28960-21apt-get23:00:331
14582991616,0cyclictest0-21swapper/120:15:021
14582991616,0cyclictest0-21swapper/119:35:351
14582991616,0cyclictest0-21swapper/100:00:331
14582991613,2cyclictest31350-21cron19:45:011
14578991616,0cyclictest0-21swapper/023:30:360
14578991616,0cyclictest0-21swapper/023:15:360
14578991613,2cyclictest13385-21perf20:15:000
14584991515,0cyclictest0-21swapper/321:30:343
14584991513,1cyclictest313-21systemd-journal23:05:003
14582991513,1cyclictest0-21swapper/120:35:411
1458299150,12cyclictest766-21snmpd21:20:001
14578991515,0cyclictest0-21swapper/023:00:330
14578991515,0cyclictest0-21swapper/000:15:340
1457899150,6cyclictest0-21swapper/000:35:340
14584991414,0cyclictest0-21swapper/323:05:053
1458499140,14cyclictest0-21swapper/321:46:063
14583991414,0cyclictest8068-21apt-get21:10:312
14583991414,0cyclictest0-21swapper/222:45:122
14583991414,0cyclictest0-21swapper/220:25:342
14583991414,0cyclictest0-21swapper/219:26:512
1458399140,14cyclictest0-21swapper/222:36:032
1458399140,14cyclictest0-21swapper/222:21:022
1458399140,14cyclictest0-21swapper/220:05:392
14582991414,0cyclictest0-21swapper/119:26:551
1458299140,14cyclictest0-21swapper/123:45:201
1458299140,14cyclictest0-21swapper/122:00:151
1458299140,14cyclictest0-21swapper/120:10:081
1458299140,0cyclictest0-21swapper/121:25:111
1458299140,0cyclictest0-21swapper/120:31:251
1457899145,8cyclictest578-21irqbalance23:10:480
14578991414,0cyclictest0-21swapper/022:30:010
14578991414,0cyclictest0-21swapper/022:10:030
14578991414,0cyclictest0-21swapper/019:35:250
1457899140,14cyclictest0-21swapper/022:55:160
1457899140,14cyclictest0-21swapper/020:20:550
1457899140,14cyclictest0-21swapper/019:31:100
1457899140,0cyclictest0-21swapper/023:05:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional