You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-10 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Thu Jul 10, 2025 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3134099250,24cyclictest0-21swapper/000:15:000
3134299240,23cyclictest0-21swapper/222:15:322
3134299240,22cyclictest313-21systemd-journal19:35:002
31341992423,1cyclictest20132-21apt-get21:00:291
3134099240,23cyclictest25934-21turbostat.cron23:25:000
3134399230,2cyclictest5608-21grep23:50:003
3134399230,22cyclictest20110-21munin-run00:20:003
3134399230,22cyclictest10641-21munin-plugin-st00:00:003
3134399230,22cyclictest0-21swapper/321:30:003
3134399230,22cyclictest0-21swapper/321:00:293
31342992321,1cyclictest1520-21cron22:35:002
31342992320,2cyclictest3322-21apt-get20:25:292
3134299230,2cyclictest583-21avahi-daemon22:45:002
31341992323,0cyclictest0-21swapper/122:30:311
3134199230,1cyclictest18638-21apt-get23:10:311
31343992220,1cyclictest15967-21munin-run23:05:003
3134399220,21cyclictest29100-21perf22:25:003
3134399220,21cyclictest0-21swapper/322:50:343
3134399220,21cyclictest0-21swapper/320:25:293
3134399220,1cyclictest19780-21apt-get22:05:293
31342992221,1cyclictest17977-21apt-get00:15:302
31342992221,1cyclictest0-21swapper/219:40:292
31342992220,1cyclictest5619-21idleruntime-cro23:50:002
31342992220,1cyclictest20575-21perf19:55:002
31342992220,1cyclictest20575-21perf19:55:002
31342992220,1cyclictest0-21swapper/221:15:002
31342992220,1cyclictest0-21swapper/200:40:002
3134299220,21cyclictest13565-21munin-run23:00:002
3134299220,21cyclictest0-21swapper/222:20:332
3134299220,21cyclictest0-21swapper/220:50:002
3134299220,1cyclictest3959-21munin-run22:40:012
3134199221,20cyclictest1345-21turbostat19:15:011
3134199221,1cyclictest10424-21munin-run00:00:001
3134199220,21cyclictest14973-21apt-get21:55:311
3134199220,21cyclictest0-21swapper/121:30:331
3134199220,21cyclictest0-21swapper/121:15:011
3134199220,21cyclictest0-21swapper/120:20:301
31340992221,1cyclictest10795-21apt-get00:00:310
31340992220,2cyclictest26990-21apt-get22:20:330
3134099220,21cyclictest3497-21apt-get23:45:280
3134099220,21cyclictest1828-21apt-get22:35:300
3134099220,21cyclictest0-21swapper/019:35:300
31343992121,0cyclictest0-21swapper/320:55:303
31343992121,0cyclictest0-21swapper/320:45:303
31343992120,0cyclictest8116-21apt-get20:35:313
31343992119,1cyclictest31534-21latency_hist22:30:003
31343992118,2cyclictest20724-21dpkg00:20:023
3134399210,20cyclictest31037-21apt-get23:35:333
3134399210,20cyclictest16074-21apt-get19:45:303
3134399210,20cyclictest0-21swapper/320:10:293
3134399210,1cyclictest24628-21idleruntime-cro21:10:003
3134399210,1cyclictest18344-21dump-pmu-power23:10:003
3134399210,1cyclictest0-21swapper/320:15:303
31342992121,0cyclictest16244-21apt-get23:05:302
31342992119,1cyclictest593-21dbus-daemon20:20:002
31342992119,1cyclictest10241-21perf20:40:002
3134299210,21cyclictest24625-21perf21:10:002
3134299210,20cyclictest21035-21apt-get23:15:332
3134299210,20cyclictest1087-21apt-get23:40:292
3134299210,20cyclictest0-21swapper/220:50:292
3134299210,1cyclictest7990-21idleruntime-cro23:55:002
3134299210,1cyclictest0-21swapper/223:20:022
3134299210,0cyclictest28463-21apt-get20:10:292
3134299210,0cyclictest10533-21apt-get20:40:332
3134299210,0cyclictest0-21swapper/223:10:302
3134299210,0cyclictest0-21swapper/219:10:312
31341992121,0cyclictest7783-21apt-get21:40:281
31341992121,0cyclictest0-21swapper/123:05:301
31341992119,1cyclictest6341-21dump-pmu-power22:44:591
3134199210,21cyclictest0-21swapper/121:25:301
3134199210,21cyclictest0-21swapper/119:40:291
3134199210,20cyclictest0-21swapper/119:20:271
31340992121,0cyclictest6466-21apt-get19:25:290
31340992121,0cyclictest0-21swapper/023:15:330
31340992121,0cyclictest0-21swapper/020:20:310
31340992120,1cyclictest25582-21apt-get00:30:310
31340992120,1cyclictest17110-21awk22:00:000
31340992120,0cyclictest0-21swapper/019:45:310
31340992119,1cyclictest10375-21systemctl00:00:010
3134099210,21cyclictest0-21swapper/000:25:300
3134099210,1cyclictest0-21swapper/023:30:280
3134099210,0cyclictest5717-21apt-get20:30:280
31343992020,0cyclictest0-21swapper/323:50:293
31343992020,0cyclictest0-21swapper/323:25:313
31343992020,0cyclictest0-21swapper/323:20:023
31343992020,0cyclictest0-21swapper/300:10:273
31343992018,1cyclictest0-21swapper/320:35:003
31343992018,1cyclictest0-21swapper/319:15:003
31343992017,2cyclictest25395-21awk20:05:013
31343992017,2cyclictest14677-21idleruntime-cro21:54:593
3134399200,19cyclictest0-21swapper/322:15:323
3134399200,0cyclictest0-21swapper/323:30:283
3134399200,0cyclictest0-21swapper/323:15:323
3134399200,0cyclictest0-21swapper/321:40:283
31342992020,0cyclictest0-21swapper/222:50:332
31342992020,0cyclictest0-21swapper/221:55:322
31342992020,0cyclictest0-21swapper/221:45:302
31342992020,0cyclictest0-21swapper/221:20:322
31342992020,0cyclictest0-21swapper/200:30:312
31342992018,1cyclictest754-21apt23:39:592
31342992018,1cyclictest0-21swapper/223:30:282
3134299200,20cyclictest0-21swapper/223:00:302
3134299200,20cyclictest0-21swapper/222:05:292
3134299200,0cyclictest0-21swapper/221:50:322
3134299200,0cyclictest0-21swapper/221:00:292
3134299200,0cyclictest0-21swapper/219:20:282
31341992020,0cyclictest9041-21apt-get22:50:331
31341992020,0cyclictest0-21swapper/123:00:301
31341992020,0cyclictest0-21swapper/122:25:311
31341992020,0cyclictest0-21swapper/121:20:321
31341992020,0cyclictest0-21swapper/119:30:291
31341992020,0cyclictest0-21swapper/100:25:301
31341992019,1cyclictest28635-21apt-get23:30:281
31341992019,1cyclictest10177-21apt-get21:45:281
31341992019,0cyclictest0-21swapper/120:30:291
31341992018,1cyclictest612-21gdbus23:45:001
31341992018,1cyclictest27657-21cron00:35:001
3134199200,0cyclictest0-21swapper/123:35:331
3134199200,0cyclictest0-21swapper/120:55:291
31340992020,0cyclictest0-21swapper/023:10:300
31340992020,0cyclictest0-21swapper/022:05:290
31340992020,0cyclictest0-21swapper/021:40:290
31340992020,0cyclictest0-21swapper/021:30:320
31340992020,0cyclictest0-21swapper/021:25:000
31340992020,0cyclictest0-21swapper/020:55:300
31340992020,0cyclictest0-21swapper/019:50:320
31340992020,0cyclictest0-21swapper/019:50:320
31340992020,0cyclictest0-21swapper/000:20:290
3134099200,19cyclictest0-21swapper/022:50:330
3134099200,0cyclictest0-21swapper/023:50:300
3134099200,0cyclictest0-21swapper/022:55:290
31343991919,0cyclictest0-21swapper/322:55:293
31343991919,0cyclictest0-21swapper/320:05:313
31343991919,0cyclictest0-21swapper/319:50:313
31343991919,0cyclictest0-21swapper/319:50:313
31343991919,0cyclictest0-21swapper/300:00:303
31343991918,1cyclictest11260-21apt-get19:35:293
31343991918,0cyclictest0-21swapper/320:40:323
31343991917,1cyclictest7479-21turbostat21:40:003
31342991919,0cyclictest0-21swapper/200:10:272
31342991919,0cyclictest0-21swapper/200:00:302
31342991917,1cyclictest19826-21grep21:00:002
3134299190,1cyclictest15268-21cron00:10:002
31341991919,0cyclictest0-21swapper/123:20:291
31341991919,0cyclictest0-21swapper/122:20:331
31341991919,0cyclictest0-21swapper/120:40:321
31341991919,0cyclictest0-21swapper/120:35:311
31341991917,1cyclictest30550-21apt20:14:591
31340991919,0cyclictest4071-21apt-get19:20:280
31340991919,0cyclictest26242-21apt-get23:25:300
31340991919,0cyclictest0-21swapper/023:35:330
31340991919,0cyclictest0-21swapper/022:10:310
31340991919,0cyclictest0-21swapper/021:35:300
31340991919,0cyclictest0-21swapper/021:10:290
31340991919,0cyclictest0-21swapper/020:50:290
31340991919,0cyclictest0-21swapper/020:00:250
31340991918,1cyclictest30113-21idleruntime-cro00:40:000
31340991918,1cyclictest13678-21apt-get19:40:290
31343991818,0cyclictest0-21swapper/322:30:303
31343991818,0cyclictest0-21swapper/319:40:293
31343991818,0cyclictest0-21swapper/300:35:313
31343991817,1cyclictest27413-21apt-get21:15:013
31343991817,1cyclictest17387-21apt-get22:00:303
31343991817,1cyclictest0-21swapper/300:30:313
31343991816,1cyclictest22175-21apt-get22:10:303
3134399180,0cyclictest8862-21apt-get19:30:283
31342991818,0cyclictest523-21apt-get21:25:292
31342991818,0cyclictest0-21swapper/222:45:282
31342991818,0cyclictest0-21swapper/219:25:292
31342991818,0cyclictest0-21swapper/200:20:302
31342991817,1cyclictest23274-21apt-get20:00:262
31342991817,0cyclictest0-21swapper/219:35:292
31342991816,1cyclictest30223-21sendmail-msp21:20:002
31341991818,0cyclictest0-21swapper/121:05:291
31341991818,0cyclictest0-21swapper/100:20:021
31341991818,0cyclictest0-21swapper/100:00:301
31341991816,1cyclictest15289-21grep00:10:001
31340991818,0cyclictest0-21swapper/021:05:290
31340991818,0cyclictest0-21swapper/000:15:300
31340991816,1cyclictest31671-21apt-get19:10:310
31340991816,1cyclictest31547-21turbostat22:30:000
31340991816,1cyclictest15297-21perf00:10:000
3134099180,17cyclictest0-21swapper/023:05:310
3134099180,0cyclictest0-21swapper/020:25:300
31343991717,0cyclictest5077-21perf21:35:003
31343991717,0cyclictest0-21swapper/323:40:303
31343991717,0cyclictest0-21swapper/323:10:303
31343991717,0cyclictest0-21swapper/321:10:283
31343991717,0cyclictest0-21swapper/319:25:293
31343991717,0cyclictest0-21swapper/300:05:333
31343991715,1cyclictest0-21swapper/320:00:003
31342991717,0cyclictest0-21swapper/223:25:312
31342991717,0cyclictest0-21swapper/219:15:322
3134299170,7cyclictest0-21swapper/220:20:302
31341991717,0cyclictest0-21swapper/123:50:301
31341991717,0cyclictest0-21swapper/122:45:281
31341991717,0cyclictest0-21swapper/120:45:301
31341991717,0cyclictest0-21swapper/119:45:321
31341991717,0cyclictest0-21swapper/100:15:301
31341991716,0cyclictest11435-21apt-get22:55:291
31341991716,0cyclictest0-21swapper/122:10:301
31341991715,1cyclictest25322-21apt-get21:10:291
31340991717,0cyclictest20865-21apt-get19:55:320
31340991717,0cyclictest0-21swapper/023:40:280
31340991717,0cyclictest0-21swapper/020:10:280
31343991616,0cyclictest915-21apt-get20:20:293
31343991616,0cyclictest0-21swapper/322:40:303
31343991616,0cyclictest0-21swapper/321:45:293
31343991616,0cyclictest0-21swapper/300:25:303
31343991614,1cyclictest6183-21munin-run19:25:003
31342991616,0cyclictest0-21swapper/223:55:302
31342991616,0cyclictest0-21swapper/221:35:312
3134299160,2cyclictest29771-21ls22:25:312
31341991616,0cyclictest15321-21apt-get20:50:291
31341991616,0cyclictest0-21swapper/121:35:291
31341991614,1cyclictest29023-21ntp_states00:35:351
3134199160,15cyclictest0-21swapper/120:25:341
31340991616,0cyclictest0-21swapper/023:00:300
31340991616,0cyclictest0-21swapper/022:45:280
31340991616,0cyclictest0-21swapper/021:50:320
31340991616,0cyclictest0-21swapper/021:25:290
31340991616,0cyclictest0-21swapper/020:40:320
31340991614,1cyclictest0-21swapper/022:30:400
31340991614,1cyclictest0-21swapper/021:00:340
3134399156,8cyclictest6637-21apt-get22:45:283
3134399151,5cyclictest0-21swapper/319:15:313
31343991513,1cyclictest0-21swapper/320:50:283
3134399150,5cyclictest30530-21apt-get21:20:323
31342991515,0cyclictest0-21swapper/222:10:302
31342991514,1cyclictest26070-21apt-get20:05:302
3134299150,14cyclictest21286-21df_inode19:55:342
31341991515,0cyclictest0-21swapper/123:45:281
31341991515,0cyclictest0-21swapper/123:25:311
31341991515,0cyclictest0-21swapper/119:50:321
31341991515,0cyclictest0-21swapper/119:50:321
31341991514,0cyclictest0-21swapper/122:00:291
31341991513,1cyclictest11751-21fschecks_time19:35:311
3134199150,14cyclictest0-21swapper/122:35:301
31340991515,0cyclictest0-21swapper/021:15:300
31340991514,1cyclictest0-21swapper/020:45:290
31343991414,0cyclictest0-21swapper/321:55:253
3134399140,0cyclictest0-21swapper/322:38:453
31342991414,0cyclictest0-21swapper/222:00:292
31342991414,0cyclictest0-21swapper/221:30:322
31342991414,0cyclictest0-21swapper/220:30:282
3134299140,13cyclictest0-21swapper/219:45:372
3134299140,0cyclictest0-21swapper/200:27:082
31341991414,0cyclictest0-21swapper/123:16:071
31341991414,0cyclictest0-21swapper/120:05:021
31341991414,0cyclictest0-21swapper/119:55:321
3134199140,14cyclictest0-21swapper/121:50:091
3134199140,14cyclictest0-21swapper/100:10:021
3134199140,0cyclictest0-21swapper/122:16:161
3134199140,0cyclictest0-21swapper/122:05:321
3134199140,0cyclictest0-21swapper/119:19:251
31340991414,0cyclictest0-21swapper/022:15:330
31340991414,0cyclictest0-21swapper/021:45:130
31340991414,0cyclictest0-21swapper/020:15:170
31340991414,0cyclictest0-21swapper/019:30:280
31340991413,1cyclictest5214-21needreboot22:40:350
3134099140,14cyclictest0-21swapper/022:00:350
3134099140,14cyclictest0-21swapper/019:18:500
3134099140,0cyclictest0-21swapper/020:35:250
31341991313,0cyclictest0-21swapper/119:25:541
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional