You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-22 - 13:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Wed Oct 22, 2025 00:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
624499270,26cyclictest0-21swapper/021:00:010
6244992621,4cyclictest31687-21apt-get20:04:270
624499260,25cyclictest0-21swapper/023:49:290
6246992423,1cyclictest18104-21apt-get22:59:272
6245992422,1cyclictest29494-21apt-get00:29:281
6244992420,4cyclictest0-21swapper/000:29:290
6250992321,1cyclictest12186-21apt-get23:54:253
6250992321,1cyclictest12186-21apt-get23:54:253
6250992320,2cyclictest31407-21munin-run21:10:003
625099230,23cyclictest0-21swapper/322:54:273
625099230,22cyclictest18612-21awk19:35:003
624699230,23cyclictest0-21swapper/223:29:272
624699230,22cyclictest27880-21turbostat22:10:002
624699230,22cyclictest10513-21apt22:44:282
6245992323,0cyclictest0-21swapper/123:29:281
6245992323,0cyclictest0-21swapper/122:59:271
6245992320,2cyclictest23487-21munin-plugin-st19:45:001
624599230,22cyclictest27872-21awk22:09:591
624599230,22cyclictest23059-21sh21:59:591
6244992320,2cyclictest20523-21chrt23:04:260
6244992320,2cyclictest19930-21awk22:59:590
624499230,2cyclictest18275-21sh20:40:000
624499230,22cyclictest27175-21perf23:15:010
624499230,22cyclictest16538-21date00:00:000
624499230,22cyclictest11412-21awk19:20:000
624499230,22cyclictest0-21swapper/020:50:010
6250992222,0cyclictest0-21swapper/322:59:263
6250992220,1cyclictest6802-21cut23:40:003
6250992220,1cyclictest301852sleep323:24:273
6250992220,1cyclictest1393-21perf21:15:003
6250992220,1cyclictest0-21swapper/322:15:003
6250992220,1cyclictest0-21swapper/320:24:273
625099220,21cyclictest8277-21perf20:20:003
625099220,21cyclictest26107-21munin-run00:20:003
625099220,21cyclictest25474-21sh22:05:003
625099220,21cyclictest16423-21grep00:00:013
6246992220,1cyclictest29265-21apt-get19:59:262
6246992220,1cyclictest0-21swapper/223:34:592
6246992219,2cyclictest18225-21wc21:50:002
624699220,21cyclictest23556-21munin-run20:50:012
624699220,21cyclictest23067-21cron22:00:002
624699220,21cyclictest16403-21perf00:00:012
624699220,21cyclictest15706-21apt-get22:54:272
624699220,21cyclictest11693-21apt-get20:29:282
624699220,1cyclictest5694-21apt-get22:34:282
624699220,1cyclictest31414-21awk21:10:002
624699220,1cyclictest13429-21sendmail-msp21:40:012
6245992221,0cyclictest10514-21apt-get22:44:281
6245992220,1cyclictest24744-21cron23:10:001
624599220,21cyclictest9793-21apt-get23:49:291
624599220,21cyclictest26545-21apt-get20:59:281
624599220,21cyclictest23562-21perf20:50:011
624599220,21cyclictest0-21swapper/123:54:271
624599220,21cyclictest0-21swapper/123:54:271
624599220,21cyclictest0-21swapper/123:24:261
624599220,21cyclictest0-21swapper/122:34:291
624599220,21cyclictest0-21swapper/121:09:591
624599220,21cyclictest0-21swapper/120:29:271
6244992221,1cyclictest7393-21apt-get23:44:280
6244992220,1cyclictest26491-21apt-get19:50:020
6244992220,1cyclictest0-21swapper/020:19:590
6244992219,2cyclictest25502-21perf22:04:590
6244992219,2cyclictest21883-21apt-get00:14:270
624499220,21cyclictest23481-21turbostat19:45:000
624499220,21cyclictest0-21swapper/022:49:280
624499220,1cyclictest593-21dbus-daemon21:04:590
624499220,1cyclictest0-21swapper/019:15:000
6250992121,0cyclictest4067-21apt-get20:14:253
6250992121,0cyclictest0-21swapper/319:59:253
6250992120,0cyclictest0-21swapper/300:29:293
6250992119,1cyclictest0-21swapper/319:24:263
6250992118,2cyclictest28898-21apt00:25:003
625099210,20cyclictest9793-21apt-get23:49:273
625099210,20cyclictest27769-21apt-get23:19:313
625099210,20cyclictest0-21swapper/322:09:273
625099210,1cyclictest0-21swapper/322:49:593
625099210,0cyclictest9928-21sendmail-msp22:39:593
625099210,0cyclictest0-21swapper/321:59:293
6246992120,1cyclictest0-21swapper/221:14:242
6246992120,0cyclictest28959-21apt-get21:04:272
6246992120,0cyclictest0-21swapper/221:24:232
6246992120,0cyclictest0-21swapper/200:20:022
6246992119,1cyclictest29580-21munin-run23:20:012
6246992119,1cyclictest18593-21sh19:34:592
6246992119,1cyclictest0-21swapper/223:50:002
6246992119,1cyclictest0-21swapper/219:30:002
624699210,3cyclictest881-21apt-get22:24:272
624699210,21cyclictest0-21swapper/223:09:272
624699210,21cyclictest0-21swapper/200:29:282
624699210,20cyclictest0-21swapper/221:19:262
624699210,20cyclictest0-21swapper/220:14:252
624699210,20cyclictest0-21swapper/219:39:252
624699210,0cyclictest9598-21apt-get19:19:272
624699210,0cyclictest0-21swapper/223:54:272
624699210,0cyclictest0-21swapper/223:54:272
6245992120,0cyclictest0-21swapper/121:14:231
6245992120,0cyclictest0-21swapper/119:59:261
6245992119,1cyclictest4363-21latency_hist23:35:001
6245992119,1cyclictest25369-21apt-get23:14:281
6245992119,1cyclictest18595-21awk19:35:001
6245992119,1cyclictest16204-21perf19:29:591
6245992119,1cyclictest0-21swapper/122:04:591
6245992119,1cyclictest0-21swapper/100:20:011
624599210,20cyclictest0-21swapper/123:19:301
624599210,20cyclictest0-21swapper/122:14:251
624599210,20cyclictest0-21swapper/100:00:011
6244992121,0cyclictest31998-21apt-get21:14:240
6244992120,0cyclictest26090-21apt-get22:09:270
6244992118,2cyclictest2703-21munin-plugin-st22:25:010
6244992118,2cyclictest18620-21idleruntime-cro19:35:000
624499210,21cyclictest8098-21apt-get22:39:290
624499210,21cyclictest0-21swapper/023:19:310
624499210,21cyclictest0-21swapper/021:49:290
624499210,20cyclictest0-21swapper/021:20:000
624499210,0cyclictest0-21swapper/020:14:240
624499210,0cyclictest0-21swapper/019:39:270
624499210,0cyclictest0-21swapper/000:39:300
6250992020,0cyclictest30891-21apt-get22:19:323
6250992020,0cyclictest0-21swapper/323:34:283
6250992020,0cyclictest0-21swapper/320:45:013
6250992020,0cyclictest0-21swapper/300:39:293
6250992019,0cyclictest0-21swapper/321:44:243
6250992018,1cyclictest14420-21apt-get19:29:323
6250992017,2cyclictest23663-21ls00:15:003
625099200,0cyclictest0-21swapper/322:44:273
625099200,0cyclictest0-21swapper/320:59:273
625099200,0cyclictest0-21swapper/320:54:263
625099200,0cyclictest0-21swapper/300:04:273
6246992020,0cyclictest0-21swapper/200:09:282
6246992019,0cyclictest6820-21apt-get21:29:292
624699200,0cyclictest17090-21apt-get00:04:272
624699200,0cyclictest1669-21apt-get20:09:272
624699200,0cyclictest0-21swapper/222:14:262
624699200,0cyclictest0-21swapper/220:24:272
6245992020,0cyclictest0-21swapper/120:04:281
6245992018,1cyclictest5875-21munin-run20:14:591
6245992018,1cyclictest18221-21turbostat21:50:001
624599200,20cyclictest0-21swapper/120:24:261
624599200,0cyclictest0-21swapper/120:09:271
624599200,0cyclictest0-21swapper/100:04:271
6244992020,0cyclictest9291-21apt-get20:24:270
6244992020,0cyclictest26088-21apt-get00:20:020
6244992020,0cyclictest0-21swapper/022:44:280
6244992020,0cyclictest0-21swapper/019:59:260
6244992018,1cyclictest0-21swapper/022:30:010
6244992015,0cyclictest0-21swapper/020:09:260
624499200,19cyclictest0-21swapper/020:44:270
624499200,0cyclictest0-21swapper/021:59:280
6250991919,0cyclictest19488-21apt-get00:09:283
6250991919,0cyclictest0-21swapper/320:44:273
6250991919,0cyclictest0-21swapper/319:49:243
6250991919,0cyclictest0-21swapper/319:44:253
6250991918,1cyclictest22970-21apt-get23:09:263
625099190,0cyclictest0-21swapper/321:39:263
6246991919,0cyclictest3288-21apt-get22:29:262
6246991919,0cyclictest0-21swapper/223:39:282
6246991919,0cyclictest0-21swapper/223:14:292
6246991919,0cyclictest0-21swapper/219:14:242
6246991918,1cyclictest0-21swapper/220:59:282
6246991917,1cyclictest31888-21apt-get00:34:282
6246991917,1cyclictest23695-21apt-get22:04:272
624699190,0cyclictest0-21swapper/219:50:022
6245991919,0cyclictest19488-21apt-get00:09:281
6245991919,0cyclictest0-21swapper/123:39:291
6245991919,0cyclictest0-21swapper/122:29:271
6245991919,0cyclictest0-21swapper/121:29:301
6245991919,0cyclictest0-21swapper/121:24:241
6245991919,0cyclictest0-21swapper/121:04:271
6245991919,0cyclictest0-21swapper/119:39:271
6245991919,0cyclictest0-21swapper/119:14:241
6245991918,0cyclictest14089-21apt-get20:34:251
6244991919,0cyclictest0-21swapper/023:39:290
6244991919,0cyclictest0-21swapper/021:54:270
6244991919,0cyclictest0-21swapper/021:44:250
6244991919,0cyclictest0-21swapper/021:34:280
624499190,18cyclictest0-21swapper/000:04:260
6250991818,0cyclictest0-21swapper/321:19:283
6250991818,0cyclictest0-21swapper/319:54:253
6250991817,1cyclictest6800-21apt-get19:14:243
6250991816,1cyclictest7510-21munin-run22:35:013
6250991816,1cyclictest0-21swapper/321:25:003
6246991818,0cyclictest21680-21apt-get19:44:262
6246991818,0cyclictest0-21swapper/220:39:262
6246991818,0cyclictest0-21swapper/220:34:252
6246991818,0cyclictest0-21swapper/219:24:262
6246991817,1cyclictest26103-21sendmail-msp00:20:002
6246991817,1cyclictest1877-21apt-get00:39:292
6246991817,1cyclictest0-21swapper/222:39:282
6245991818,0cyclictest16487-21apt-get20:39:271
6245991818,0cyclictest0-21swapper/121:39:281
6245991818,0cyclictest0-21swapper/119:50:021
6245991817,0cyclictest0-21swapper/120:44:281
6245991816,1cyclictest26088-21apt-get00:19:591
6244991818,0cyclictest0-21swapper/022:34:290
6244991818,0cyclictest0-21swapper/021:24:240
6244991816,1cyclictest25163-21ntp_kernel_err20:54:330
6244991816,1cyclictest13812-21perf19:25:010
6244991815,2cyclictest32708-21sendmail-msp22:20:000
6244991815,2cyclictest17478-21cron22:54:590
624499180,0cyclictest0-21swapper/000:34:290
6250991717,0cyclictest0-21swapper/321:34:273
6250991717,0cyclictest0-21swapper/319:19:273
6246991717,0cyclictest0-21swapper/223:44:272
6246991717,0cyclictest0-21swapper/223:24:262
6246991717,0cyclictest0-21swapper/222:49:282
6246991717,0cyclictest0-21swapper/220:44:272
6246991717,0cyclictest0-21swapper/219:47:322
6246991716,1cyclictest0-21swapper/200:15:002
624699170,16cyclictest30895-21sshd22:19:312
6245991717,0cyclictest0-21swapper/123:44:281
6245991717,0cyclictest0-21swapper/122:49:281
6245991717,0cyclictest0-21swapper/119:24:251
6245991716,0cyclictest0-21swapper/122:39:291
6245991716,0cyclictest0-21swapper/100:34:281
6245991715,1cyclictest23686-21munin-run00:14:591
6244991717,0cyclictest0-21swapper/023:29:280
6244991717,0cyclictest0-21swapper/020:34:260
6244991715,1cyclictest2569-21apt-get23:34:280
6250991616,0cyclictest0-21swapper/323:44:273
6250991616,0cyclictest0-21swapper/323:04:263
6250991616,0cyclictest0-21swapper/320:29:283
625099160,15cyclictest0-21swapper/322:24:333
6246991614,1cyclictest18827-21apt-get21:54:282
6246991614,1cyclictest14031-21apt-get21:44:262
6246991611,4cyclictest31687-21apt-get20:04:282
6245991616,0cyclictest0-21swapper/121:19:261
6245991616,0cyclictest0-21swapper/100:39:301
6244991616,0cyclictest0-21swapper/023:09:260
6244991616,0cyclictest0-21swapper/020:29:280
6244991614,1cyclictest13489-21gzip23:54:340
6244991614,1cyclictest13489-21gzip23:54:330
6244991614,1cyclictest0-21swapper/019:49:270
6250991515,0cyclictest0-21swapper/321:54:273
6250991515,0cyclictest0-21swapper/320:04:283
6250991515,0cyclictest0-21swapper/319:39:253
6250991515,0cyclictest0-21swapper/300:34:293
6250991513,1cyclictest0-21swapper/320:40:003
6246991514,1cyclictest20519-21apt-get23:04:272
624699150,1cyclictest0-21swapper/221:34:292
624599150,1cyclictest0-21swapper/122:19:401
624599150,10cyclictest241662sleep119:49:271
6244991515,0cyclictest0-21swapper/000:09:260
6244991513,1cyclictest24454-21tr00:19:280
6250991414,0cyclictest0-21swapper/320:31:323
625099140,14cyclictest0-21swapper/323:26:503
625099140,14cyclictest0-21swapper/321:26:283
625099140,14cyclictest0-21swapper/321:01:503
625099140,0cyclictest0-21swapper/323:10:013
625099140,0cyclictest0-21swapper/322:25:243
625099140,0cyclictest0-21swapper/321:45:023
625099140,0cyclictest0-21swapper/320:08:293
624699140,14cyclictest0-21swapper/220:51:212
624699140,0cyclictest0-21swapper/220:15:142
6245991414,0cyclictest0-21swapper/123:00:331
6245991414,0cyclictest0-21swapper/121:40:081
6245991414,0cyclictest0-21swapper/121:30:121
6245991414,0cyclictest0-21swapper/120:15:151
624599140,14cyclictest0-21swapper/122:53:191
624599140,0cyclictest0-21swapper/120:50:191
6244991414,0cyclictest0-21swapper/021:35:100
6244991414,0cyclictest0-21swapper/021:26:040
624499140,14cyclictest0-21swapper/022:10:380
624499140,14cyclictest0-21swapper/019:25:530
624499140,0cyclictest0-21swapper/023:22:050
624499140,0cyclictest0-21swapper/021:05:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional