You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 05:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Mon Nov 03, 2025 00:44:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22263992725,1cyclictest22-21ksoftirqd/121:40:011
2226399250,1cyclictest151rcu_preempt23:34:591
22265992421,2cyclictest11808-21latency_hist23:15:013
22263992422,1cyclictest22-21ksoftirqd/123:19:591
2226599230,22cyclictest32168-21perf22:49:593
2226599230,22cyclictest27043-21munin-run23:45:003
2226499230,22cyclictest5310-21apt-get21:55:012
22263992321,1cyclictest6398-21latency_hist20:50:011
2226399230,22cyclictest4663-21perf19:40:001
2226399230,22cyclictest24312-21cron20:19:591
2226299230,22cyclictest2277-21idleruntime-cro00:00:000
2226299230,22cyclictest2135-21cron22:55:000
2226299230,22cyclictest17022-21grep23:25:000
2226299230,22cyclictest1492-21sh20:40:010
22265992220,1cyclictest1831-21apt-get20:40:283
22265992220,1cyclictest0-21swapper/322:20:013
22265992219,2cyclictest7123-21munin-run00:10:003
2226599220,21cyclictest313-21systemd-journal23:25:003
2226599220,21cyclictest15011-21apt-get20:00:283
2226599220,21cyclictest0-21swapper/323:35:343
2226599220,21cyclictest0-21swapper/322:40:293
2226599220,21cyclictest0-21swapper/321:10:293
2226599220,21cyclictest0-21swapper/319:15:283
2226599220,1cyclictest6007-21apt-get21:55:313
22264992221,0cyclictest10821-21apt-get22:05:302
22264992220,1cyclictest612-21gdbus22:34:592
2226499220,21cyclictest9044-21/usr/sbin/munin20:55:292
2226499220,21cyclictest7402-21apt-get19:45:292
2226499220,21cyclictest27717-21apt-get23:45:322
2226499220,21cyclictest27717-21apt-get23:45:322
2226499220,1cyclictest17717-21tr22:20:012
2226499220,1cyclictest17704-21apt-get23:25:322
2226499220,1cyclictest10532-21awk22:05:002
22263992220,1cyclictest21158-21apt-get21:20:301
22263992220,1cyclictest16694-21cron00:29:591
22263992220,1cyclictest15315-21latency_hist22:14:591
22263992220,1cyclictest11245-21idleruntime-cro21:00:001
2226399220,22cyclictest0-21swapper/120:00:291
2226399220,21cyclictest19524-21munin-run20:10:001
2226399220,1cyclictest0-21swapper/123:55:011
2226399220,1cyclictest0-21swapper/122:50:301
22262992221,1cyclictest0-21swapper/021:20:300
22262992221,1cyclictest0-21swapper/019:50:020
22262992220,1cyclictest4710-21latency_hist00:05:000
2226299220,22cyclictest0-21swapper/021:45:000
2226299220,21cyclictest24925-21idleruntime-cro22:35:000
2226299220,21cyclictest17113-21turbostat20:05:000
2226299220,21cyclictest16044-21grep21:10:000
2226299220,21cyclictest0-21swapper/022:35:340
2226299220,21cyclictest0-21swapper/021:15:320
2226299220,21cyclictest0-21swapper/021:10:290
2226299220,1cyclictest0-21swapper/021:30:290
2226299220,1cyclictest0-21swapper/019:30:000
22265992121,0cyclictest18740-21apt-get21:15:323
22265992120,1cyclictest396-21sh21:44:593
22265992119,1cyclictest7028-21munin-plugin-st23:05:013
2226599210,21cyclictest0-21swapper/323:25:323
2226599210,21cyclictest0-21swapper/321:00:303
2226599210,20cyclictest0-21swapper/320:55:293
2226599210,1cyclictest30482-21perf21:40:003
2226599210,0cyclictest30144-21apt-get19:25:293
2226599210,0cyclictest0-21swapper/320:05:283
22264992121,0cyclictest30050-21apt-get22:45:302
22264992121,0cyclictest14201-21apt-get23:20:012
22264992121,0cyclictest0-21swapper/221:10:292
22264992120,1cyclictest593-21dbus-daemon22:55:002
22264992120,0cyclictest28358-21apt-get21:35:322
22264992119,1cyclictest593-21dbus-daemon00:00:012
2226499210,21cyclictest12096-21apt-get23:15:292
2226499210,20cyclictest0-21swapper/220:25:322
2226499210,20cyclictest0-21swapper/219:40:272
2226499210,20cyclictest0-21swapper/219:39:592
2226499210,1cyclictest0-21swapper/221:35:012
2226499210,0cyclictest0-21swapper/220:30:282
22263992121,0cyclictest0-21swapper/123:20:001
22263992120,1cyclictest0-21swapper/121:25:281
22263992118,2cyclictest24642-21cut23:40:001
2226399210,21cyclictest0-21swapper/123:55:321
2226399210,21cyclictest0-21swapper/120:35:291
2226399210,21cyclictest0-21swapper/119:10:291
2226399210,20cyclictest0-21swapper/100:00:311
2226399210,1cyclictest11933-21munin-run00:19:591
2226399210,0cyclictest0-21swapper/123:00:351
2226399210,0cyclictest0-21swapper/120:25:321
22262992121,0cyclictest0-21swapper/000:35:290
22262992121,0cyclictest0-21swapper/000:35:290
22262992121,0cyclictest0-21swapper/000:15:320
22262992120,1cyclictest12925-21perf22:10:000
22262992120,1cyclictest0-21swapper/022:55:310
22262992120,1cyclictest0-21swapper/021:45:310
22262992120,0cyclictest4226-21apt-get20:45:280
22262992119,1cyclictest7083-21cron19:45:000
22262992119,1cyclictest50092chrt00:05:280
22262992119,1cyclictest17709-21cron22:20:010
22262992118,3cyclictest0-21swapper/019:20:300
2226299210,21cyclictest0-21swapper/020:40:280
2226299210,21cyclictest0-21swapper/019:45:290
2226299210,0cyclictest0-21swapper/023:25:320
2226299210,0cyclictest0-21swapper/021:35:320
2226299210,0cyclictest0-21swapper/019:15:270
22265992020,0cyclictest5005-21apt-get19:40:283
22265992020,0cyclictest22538-21apt-get19:10:293
22265992020,0cyclictest0-21swapper/322:10:293
22265992020,0cyclictest0-21swapper/320:15:313
22265992018,1cyclictest737-21apt-get21:45:303
2226599200,20cyclictest0-21swapper/300:15:333
2226599200,0cyclictest32515-21apt-get23:55:323
2226599200,0cyclictest0-21swapper/320:35:003
2226599200,0cyclictest0-21swapper/320:10:293
2226599200,0cyclictest0-21swapper/319:30:303
22264992020,0cyclictest7402-21apt-get00:10:292
22264992020,0cyclictest25328-21apt-get19:15:282
22264992020,0cyclictest17405-21apt-get20:05:282
22264992020,0cyclictest0-21swapper/223:10:312
22264992020,0cyclictest0-21swapper/222:20:282
22264992020,0cyclictest0-21swapper/221:15:312
22264992020,0cyclictest0-21swapper/220:40:282
22264992020,0cyclictest0-21swapper/200:30:292
22264992018,1cyclictest4686-21cron00:05:012
22264992018,1cyclictest19407-21apt-get00:35:292
22264992018,1cyclictest19407-21apt-get00:35:282
2226499200,20cyclictest0-21swapper/200:20:312
2226499200,0cyclictest9802-21apt-get00:15:322
2226499200,0cyclictest4915-21apt-get23:00:352
2226499200,0cyclictest0-21swapper/219:25:282
22263992020,0cyclictest0-21swapper/122:00:301
22263992020,0cyclictest0-21swapper/121:45:311
22263992020,0cyclictest0-21swapper/119:45:291
22263992019,1cyclictest22540-21turbostat22:30:011
22263992019,1cyclictest22-21ksoftirqd/121:35:011
22263992019,0cyclictest0-21swapper/123:25:311
22263992018,1cyclictest9399-21latency_hist23:10:011
22263992018,1cyclictest21545-21munin-run00:40:001
22263992018,1cyclictest21545-21munin-run00:40:001
2226399200,20cyclictest0-21swapper/121:40:281
2226399200,20cyclictest0-21swapper/119:30:291
2226399200,1cyclictest0-21swapper/123:45:001
2226399200,0cyclictest0-21swapper/119:40:281
2226399200,0cyclictest0-21swapper/100:20:301
22262992020,0cyclictest20101-21apt-get23:30:300
22262992020,0cyclictest0-21swapper/021:55:300
22262992020,0cyclictest0-21swapper/021:25:290
22262992019,1cyclictest0-21swapper/023:45:020
22262992019,1cyclictest0-21swapper/023:45:020
22265991919,0cyclictest32446-21apt-get22:50:293
22265991918,1cyclictest7304-21apt-get23:05:303
22265991918,1cyclictest27028-21apt-get20:25:323
22265991918,0cyclictest0-21swapper/323:15:293
22265991917,1cyclictest21533-21sendmail-msp00:39:593
22265991917,1cyclictest21533-21sendmail-msp00:39:593
22265991917,1cyclictest14712-21idleruntime-cro20:00:003
22264991919,0cyclictest0-21swapper/222:35:342
22264991919,0cyclictest0-21swapper/220:35:292
22264991918,1cyclictest27753-21apt-get19:20:322
22264991918,0cyclictest9515-21apt-get19:50:022
22264991918,0cyclictest2439-21apt-get22:55:322
22264991918,0cyclictest0-21swapper/221:45:312
22263991919,0cyclictest0-21swapper/119:50:021
22263991919,0cyclictest0-21swapper/100:10:281
22263991917,1cyclictest4541-21apt23:00:001
22263991917,1cyclictest11821-21munin-run23:15:001
22263991917,1cyclictest0-21swapper/123:45:021
22263991917,1cyclictest0-21swapper/123:45:021
22262991919,0cyclictest0-21swapper/022:25:300
22262991919,0cyclictest0-21swapper/021:50:320
22262991919,0cyclictest0-21swapper/020:55:280
22262991919,0cyclictest0-21swapper/019:55:300
22262991918,1cyclictest0-21swapper/000:10:280
22262991918,0cyclictest0-21swapper/000:25:310
22262991917,1cyclictest14337-21munin-run00:25:000
22265991818,0cyclictest23560-21apt-get21:25:283
22265991818,0cyclictest19125-21perf00:35:003
22265991818,0cyclictest0-21swapper/322:05:303
22265991818,0cyclictest0-21swapper/321:50:323
22265991818,0cyclictest0-21swapper/300:00:313
22265991817,0cyclictest10196-21apt-get19:50:293
22265991816,1cyclictest14617-21apt-get00:25:323
22264991818,0cyclictest11545-21apt-get21:00:302
22264991818,0cyclictest0-21swapper/223:05:302
22264991818,0cyclictest0-21swapper/221:20:292
22264991818,0cyclictest0-21swapper/220:50:012
22264991818,0cyclictest0-21swapper/220:00:292
22264991818,0cyclictest0-21swapper/200:25:312
22264991816,1cyclictest5005-21apt-get00:05:292
22263991818,0cyclictest29860-21perf19:25:001
22263991818,0cyclictest0-21swapper/120:10:291
22263991818,0cyclictest0-21swapper/119:25:281
22263991818,0cyclictest0-21swapper/119:15:011
22263991817,1cyclictest25241-21apt-get22:35:341
2226399180,18cyclictest0-21swapper/122:30:301
22262991818,0cyclictest0-21swapper/023:05:290
22262991818,0cyclictest0-21swapper/022:20:290
22262991818,0cyclictest0-21swapper/022:00:290
22262991818,0cyclictest0-21swapper/021:00:290
22262991818,0cyclictest0-21swapper/019:35:290
22262991817,1cyclictest24926-21apt-get23:40:320
22262991817,1cyclictest0-21swapper/023:00:340
22265991717,0cyclictest0-21swapper/323:50:323
22265991717,0cyclictest0-21swapper/322:30:303
22265991717,0cyclictest0-21swapper/320:45:293
22265991717,0cyclictest0-21swapper/319:45:283
22265991717,0cyclictest0-21swapper/300:10:293
22264991717,0cyclictest12595-21apt-get19:55:302
22264991717,0cyclictest0-21swapper/223:30:292
22264991717,0cyclictest0-21swapper/221:25:282
2226499170,17cyclictest0-21swapper/223:40:312
22263991717,0cyclictest0-21swapper/122:05:291
22263991717,0cyclictest0-21swapper/100:05:301
2226399170,16cyclictest16339-21apt-get21:10:281
22262991717,0cyclictest0-21swapper/023:35:330
22262991717,0cyclictest0-21swapper/022:45:300
22262991717,0cyclictest0-21swapper/020:30:290
22262991717,0cyclictest0-21swapper/020:25:330
22262991717,0cyclictest0-21swapper/000:30:300
22265991616,0cyclictest0-21swapper/323:30:303
22265991616,0cyclictest0-21swapper/322:25:303
22265991616,0cyclictest0-21swapper/321:20:303
22265991616,0cyclictest0-21swapper/319:20:323
22265991614,1cyclictest8785-21perf20:55:003
22265991614,1cyclictest27898-21expr23:45:313
22265991614,1cyclictest27898-21expr23:45:313
22265991614,1cyclictest0-21swapper/320:35:283
2226599160,2cyclictest16089-21cat21:10:003
22264991616,0cyclictest0-21swapper/222:25:292
22264991616,0cyclictest0-21swapper/221:50:332
22264991615,1cyclictest22510-21apt-get23:35:322
2226499160,12cyclictest766-21snmpd22:10:082
2226399164,11cyclictest22-21ksoftirqd/122:20:001
22263991616,0cyclictest0-21swapper/121:50:331
22263991616,0cyclictest0-21swapper/121:15:321
22263991616,0cyclictest0-21swapper/120:50:311
2226399160,7cyclictest0-21swapper/121:00:291
2226399160,15cyclictest211rcuc/120:35:011
22262991616,0cyclictest0-21swapper/023:15:290
22262991616,0cyclictest0-21swapper/023:10:300
22262991616,0cyclictest0-21swapper/019:10:280
2226299160,15cyclictest0-21swapper/022:40:320
22265991515,0cyclictest0-21swapper/300:20:313
22265991514,1cyclictest596-21in:imuxsock23:00:003
22264991513,1cyclictest28278-21head22:40:322
2226499150,14cyclictest22205-21apt-get20:15:302
2226499150,14cyclictest0-21swapper/219:10:282
22263991515,0cyclictest0-21swapper/120:40:291
22263991515,0cyclictest0-21swapper/119:55:301
22263991515,0cyclictest0-21swapper/100:30:301
2226399150,14cyclictest0-21swapper/122:20:471
22262991515,0cyclictest0-21swapper/020:50:310
22262991515,0cyclictest0-21swapper/020:20:290
22265991414,0cyclictest0-21swapper/322:35:333
22265991411,2cyclictest3155-21gltestperf19:35:313
2226599140,14cyclictest0-21swapper/322:20:173
2226599140,14cyclictest0-21swapper/321:30:123
2226599140,14cyclictest0-21swapper/320:22:483
2226599140,0cyclictest0-21swapper/322:02:383
22264991414,0cyclictest0-21swapper/220:20:222
2226499140,14cyclictest0-21swapper/223:52:382
2226499140,14cyclictest0-21swapper/220:50:222
2226499140,14cyclictest0-21swapper/220:10:062
2226499140,14cyclictest0-21swapper/219:30:372
22263991414,0cyclictest0-21swapper/122:48:371
22263991414,0cyclictest0-21swapper/121:09:441
2226399140,14cyclictest0-21swapper/122:40:321
22262991414,0cyclictest0-21swapper/023:50:320
22262991414,0cyclictest0-21swapper/020:10:010
22262991413,0cyclictest0-21swapper/020:05:270
2226299140,14cyclictest0-21swapper/020:15:330
2226299140,14cyclictest0-21swapper/019:30:380
2226299140,0cyclictest0-21swapper/022:10:280
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional