You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 02:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TI
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot6s.osadl.org (updated Mon Nov 24, 2025 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4173103998333,10cyclictest0-21swapper/210:25:282
417309999833,36cyclictest16484-21sh10:45:220
417309999832,72cyclictest535-21tail09:10:290
417309999822,37cyclictest4174341-21munin-node07:15:050
417309999791,47cyclictest0-21swapper/007:55:190
4173103997536,5cyclictest28922-21munin-plugin-st12:00:002
4173103997524,10cyclictest0-21swapper/209:40:302
417309999751,35cyclictest0-21swapper/009:55:220
4173103997337,6cyclictest4192889-21grep09:00:282
4173103997230,10cyclictest4192107-21sendmail08:59:592
417309999711,59cyclictest4178987-21/usr/sbin/munin07:40:260
4173103996827,6cyclictest0-21swapper/208:20:192
417309999681,51cyclictest0-21swapper/008:15:130
4173103996628,10cyclictest0-21swapper/210:15:242
4173103996627,10cyclictest0-21swapper/210:25:012
417309999661,53cyclictest0-21swapper/012:15:140
4173103996527,10cyclictest0-21swapper/209:50:222
417309999651,58cyclictest17967-21apt10:55:100
417309999651,49cyclictest0-21swapper/011:40:140
417309999641,50cyclictest0-21swapper/011:50:160
417309999631,55cyclictest4175865-21users07:20:270
417309999631,55cyclictest22245-21latency_hist11:20:020
417309999631,49cyclictest0-21swapper/008:30:110
417309999631,47cyclictest0-21swapper/009:05:140
417309999631,47cyclictest0-21swapper/009:05:140
417309999621,59cyclictest0-21swapper/008:00:270
417309999621,59cyclictest0-21swapper/008:00:270
417309999621,48cyclictest0-21swapper/009:40:140
417309999612,32cyclictest28541-21irqstats11:55:250
417309999611,52cyclictest0-21swapper/012:20:120
417309999611,48cyclictest0-21swapper/012:10:150
4173099996112,42cyclictest19978-21/usr/sbin/munin11:05:240
4173099996112,42cyclictest19978-21/usr/sbin/munin11:05:230
417310299608,21cyclictest0-21swapper/111:20:011
417310299608,21cyclictest0-21swapper/111:20:011
417309999601,50cyclictest0-21swapper/008:55:110
417309999601,47cyclictest131ktimers/008:25:120
417309999592,49cyclictest3434-21apt-get09:30:140
417309999592,36cyclictest35108-21diskstats12:35:190
4173099995920,31cyclictest9125-21tail10:00:300
417309999591,52cyclictest34367-21dropbear12:30:200
417309999591,49cyclictest0-21swapper/009:15:120
417309999591,44cyclictest0-21swapper/008:40:140
417309999591,30cyclictest29202-21df_inode12:00:160
417309999581,51cyclictest0-21swapper/009:35:250
417309999581,51cyclictest0-21swapper/009:35:240
417309999581,49cyclictest9701-21/usr/sbin/munin10:05:240
417309999581,41cyclictest0-21swapper/010:50:130
4173099995718,33cyclictest3119-21tail09:25:300
417309999571,50cyclictest4173345-21/usr/sbin/munin07:10:190
417309999571,49cyclictest4190936-21/usr/sbin/munin08:50:240
417309999571,49cyclictest4190936-21/usr/sbin/munin08:50:230
417309999571,49cyclictest24425-21/usr/sbin/munin11:30:260
4173102995640,11cyclictest18755-21apt-key11:00:021
417309999561,52cyclictest1819-21/usr/sbin/munin09:20:180
417309999561,52cyclictest1819-21/usr/sbin/munin09:20:170
417309999561,52cyclictest10900-21sshd10:11:220
417309999561,50cyclictest4180710-21uname07:50:270
417309999561,48cyclictest4190080-21/usr/sbin/munin08:45:260
417309999561,48cyclictest4183657-21wc08:10:000
417309999561,48cyclictest27328-21latency_hist11:50:000
417309999561,48cyclictest23203-21diskstats11:25:190
417309999561,30cyclictest4188154-21diskstats08:35:190
417309999561,30cyclictest4188154-21diskstats08:35:190
417309999552,45cyclictest33360-21apt-get12:25:140
4173099995519,28cyclictest0-21swapper/007:30:200
417309999551,49cyclictest23013-21latency_hist11:25:020
417309999551,49cyclictest23013-21latency_hist11:25:010
417309999551,48cyclictest12242-21irqstats10:20:230
417309999551,47cyclictest4184093-21/usr/sbin/munin08:10:250
417309999551,46cyclictest4192321-21apt-get09:00:110
417309999543,44cyclictest11686-21/usr/sbin/munin10:15:260
417309999541,47cyclictest4179908-21/usr/sbin/munin07:45:260
417309999541,47cyclictest13457-21apt-get10:30:060
417309999541,46cyclictest21050-21/usr/sbin/munin11:10:300
417309999541,43cyclictest12820-21apt-get10:25:130
417309999531,49cyclictest19219-21/usr/sbin/munin11:00:250
417309999531,26cyclictest0-21swapper/007:30:000
417310299527,16cyclictest0-21swapper/109:05:011
417309999522,43cyclictest30228-21/usr/sbin/munin12:05:230
417309999521,48cyclictest4186037-21/usr/sbin/munin08:20:300
417309999521,42cyclictest14589-21apt-get10:35:130
417309999511,47cyclictest15543-21ssh10:40:200
417309999511,47cyclictest15543-21ssh10:40:200
4173099995114,32cyclictest4178363-21grep07:35:290
4173103995036,10cyclictest0-21swapper/212:35:002
4173102995037,10cyclictest231ktimers/110:30:051
417309999503,43cyclictest24901-21apt-get11:35:150
417309999501,46cyclictest7241-21/usr/sbin/munin09:50:260
417309999481,44cyclictest6275-21/usr/sbin/munin09:45:240
417310299472,8cyclictest0-21swapper/110:10:021
4173102994719,11cyclictest4193158-21/usr/sbin/munin09:05:201
4173102994719,11cyclictest4193158-21/usr/sbin/munin09:05:201
4173103994630,9cyclictest0-21swapper/207:10:212
4173103994528,12cyclictest0-21swapper/210:45:302
4173102994530,8cyclictest1539-21sendmail09:20:011
4173102994527,11cyclictest231ktimers/108:00:111
4173102994527,11cyclictest231ktimers/108:00:111
4173103994431,9cyclictest4189937-21sshd08:45:222
4173103994332,7cyclictest14-21rcu_preempt08:30:012
417310399432,34cyclictest4177770-21/usr/sbin/munin07:35:182
4173102994336,5cyclictest0-21swapper/107:55:191
4173103994231,6cyclictest21125-21grep11:10:302
4173103994231,6cyclictest0-21swapper/211:35:282
4173103994229,6cyclictest0-21swapper/212:10:282
4173103994225,10cyclictest0-21swapper/209:10:292
4173102994233,5cyclictest0-21swapper/111:30:281
4173102994218,14cyclictest4173094-21cyclictest08:55:011
417309999427,27cyclictest0-21swapper/007:10:010
4173102994129,7cyclictest0-21swapper/111:05:301
4173102994129,7cyclictest0-21swapper/111:05:301
4173102994128,6cyclictest0-21swapper/112:15:271
417310299404,25cyclictest0-21swapper/111:15:001
4173102994033,4cyclictest4183485-21munin-plugin-st08:09:591
4173102994029,7cyclictest9365-21grep10:05:011
4173102994023,9cyclictest0-21swapper/107:25:201
4173103993929,5cyclictest0-21swapper/212:00:262
4173103993924,10cyclictest0-21swapper/210:00:022
417310399391,3cyclictest0-21swapper/210:05:012
417310299399,25cyclictest0-21swapper/108:15:261
4173102993932,3cyclictest0-21swapper/109:40:271
4173102993927,7cyclictest0-21swapper/109:25:291
4173102993924,8cyclictest0-21swapper/109:45:231
4173102993924,8cyclictest0-21swapper/108:50:231
4173102993924,8cyclictest0-21swapper/108:50:231
4173102993924,11cyclictest0-21swapper/107:20:251
4173102993915,11cyclictest22-21rcuc/108:35:011
417310299391,28cyclictest0-21swapper/108:20:171
4173103993827,5cyclictest0-21swapper/212:25:302
4173103993822,10cyclictest0-21swapper/212:05:202
417310299389,12cyclictest34196-21/usr/sbin/munin12:30:191
417310299388,25cyclictest0-21swapper/111:00:191
417310299386,15cyclictest22-21rcuc/110:15:021
4173102993829,5cyclictest0-21swapper/109:50:181
4173102993828,7cyclictest231ktimers/111:45:281
4173102993828,6cyclictest231ktimers/111:50:271
4173102993825,9cyclictest0-21swapper/112:25:211
4173102993825,7cyclictest231ktimers/112:00:221
4173102993819,12cyclictest22-21rcuc/110:45:241
417310399377,2cyclictest0-21swapper/211:25:012
417310399377,2cyclictest0-21swapper/211:25:012
417310399377,12cyclictest4174111-21meminfo07:15:022
4173103993729,4cyclictest168450irq/134-8000000.ethernet-tx210:36:022
4173103993726,6cyclictest2334-21grep09:20:282
4173103993726,6cyclictest2334-21grep09:20:282
4173103993724,5cyclictest0-21swapper/208:10:012
417310399371,5cyclictest0-21swapper/208:40:012
417310399371,5cyclictest0-21swapper/208:40:012
417310399371,2cyclictest0-21swapper/208:05:012
417310399371,2cyclictest0-21swapper/208:05:012
4173102993728,5cyclictest0-21swapper/108:30:011
4173102993727,7cyclictest231ktimers/109:35:151
4173102993727,7cyclictest231ktimers/109:35:151
4173102993725,7cyclictest0-21swapper/107:45:281
4173102993717,11cyclictest17127-21/usr/sbin/munin10:50:181
4173102993715,10cyclictest4173094-21cyclictest08:45:151
417310299371,29cyclictest1657-21lldpd07:50:261
4173102993710,3cyclictest33160-21munin-plugin-st12:25:001
4173103993627,5cyclictest4189572-21cron08:44:592
4173103993627,5cyclictest1265-21rs:main2
4173103993625,7cyclictest0-21swapper/208:30:282
4173103993625,5cyclictest0-21swapper/208:10:192
417310399361,29cyclictest0-21swapper/209:45:212
4173102993627,5cyclictest231ktimers/112:15:011
4173102993627,4cyclictest4177525-21grep07:30:281
4173102993626,6cyclictest10804-21grep10:10:311
4173102993626,5cyclictest0-21swapper/110:20:161
4173102993625,8cyclictest0-21swapper/110:00:001
4173102993625,7cyclictest4178974-21phc2sys-jitter07:40:271
4173102993622,10cyclictest4173094-21cyclictest12:35:141
4173102993610,11cyclictest22-21rcuc/108:45:011
4173103993525,7cyclictest24060-21/usr/sbin/munin11:30:292
4173103993524,6cyclictest0-21swapper/212:35:302
4173103993524,6cyclictest0-21swapper/207:20:212
4173103993523,5cyclictest0-21swapper/210:45:012
4173103993523,5cyclictest0-21swapper/210:45:012
4173103993522,7cyclictest0-21swapper/209:05:302
4173103993522,7cyclictest0-21swapper/209:05:302
417310399351,28cyclictest0-21swapper/208:50:172
417310399351,28cyclictest0-21swapper/208:50:172
4173102993527,6cyclictest231ktimers/111:55:241
4173102993526,5cyclictest0-21swapper/109:10:301
4173102993525,7cyclictest231ktimers/111:40:131
4173102993524,8cyclictest231ktimers/107:35:121
4173102993522,8cyclictest22-21rcuc/109:20:121
4173102993522,8cyclictest22-21rcuc/109:20:111
4173102993518,9cyclictest0-21swapper/111:30:001
4173103993425,4cyclictest0-21swapper/211:20:002
4173103993424,6cyclictest4809-21grep09:35:292
4173103993424,6cyclictest4809-21grep09:35:292
4173103993424,6cyclictest4176382-21kernelversion07:25:252
4173103993424,6cyclictest168450irq/134-8000000.ethernet-tx210:55:282
4173103993424,5cyclictest168450irq/134-8000000.ethernet-tx211:50:282
4173102993426,5cyclictest29975-21/usr/sbin/munin12:05:231
4173102993424,6cyclictest22-21rcuc/110:40:261
4173102993424,6cyclictest22-21rcuc/110:40:261
4173102993424,5cyclictest22-21rcuc/111:40:011
4173103993323,7cyclictest0-21swapper/211:25:292
4173103993323,6cyclictest0-21swapper/207:55:282
4173103993317,9cyclictest0-21swapper/210:10:232
417310399331,2cyclictest0-21swapper/210:35:002
417310399331,27cyclictest0-21swapper/209:30:532
4173102993323,5cyclictest22-21rcuc/107:10:281
4173103993223,6cyclictest47-21kauditd09:29:592
4173103993221,8cyclictest32583-21df12:20:182
4173103993220,7cyclictest17666-21grep10:50:282
4173102993222,6cyclictest231ktimers/111:15:191
4173103993122,6cyclictest4180372-21df07:50:172
4173103993122,5cyclictest0-21swapper/211:05:292
4173103993122,5cyclictest0-21swapper/211:05:292
4173103993121,6cyclictest0-21swapper/207:50:002
4173103993119,9cyclictest4185264-21sendmail08:20:002
4173103993116,11cyclictest0-21swapper/211:49:592
4173102993124,4cyclictest231ktimers/108:35:291
4173102993124,4cyclictest231ktimers/108:35:291
4173102993121,5cyclictest231ktimers/109:35:011
4173102993121,5cyclictest0-21swapper/107:10:011
4173102993120,6cyclictest13464-21http10:30:001
4173103993022,5cyclictest4177459-21unixbench_multi07:30:292
4173102993023,4cyclictest0-21swapper/108:10:161
4173102993021,6cyclictest22-21rcuc/107:15:161
4173103992917,6cyclictest0-21swapper/207:40:212
4173102992917,9cyclictest15247-21sendmail-msp10:40:001
4173103992820,4cyclictest0-21swapper/209:15:192
4173103992819,4cyclictest0-21swapper/212:15:282
4173103992817,8cyclictest0-21swapper/211:00:272
4173103992717,7cyclictest9977-21grep10:05:292
4173103992212,5cyclictest311ktimers/207:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional