You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 22:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Wed Nov 26, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
162299260,25cyclictest3994-21awk07:15:001
1631992423,1cyclictest2067-21apt-get07:12:313
1631992422,2cyclictest22105-21perf11:10:013
162299240,23cyclictest0-21swapper/110:10:021
162199240,23cyclictest13437-21awk08:40:000
1627992320,2cyclictest21775-21idleruntime-cro07:49:592
162799230,22cyclictest0-21swapper/210:45:002
1622992321,1cyclictest313-21systemd-journal12:40:001
1622992320,2cyclictest9683-21chrt07:27:311
162299230,2cyclictest30968-21apt12:35:001
162299230,22cyclictest12417-21basename10:50:001
162299230,1cyclictest313-21systemd-journal11:40:001
162299230,0cyclictest0-21swapper/111:10:011
162299230,0cyclictest0-21swapper/108:32:321
1621992321,1cyclictest9992-21perf10:44:590
1621992321,1cyclictest6969-21apt-get11:47:320
162199230,22cyclictest4065-21cron11:40:000
162199230,0cyclictest0-21swapper/007:37:310
1631992222,0cyclictest9066-21apt-get08:32:323
1631992220,2cyclictest12418-21dump-pmu-power10:50:003
163199220,21cyclictest10005-21grep10:45:003
163199220,21cyclictest0-21swapper/311:22:313
163199220,21cyclictest0-21swapper/309:37:303
163199220,21cyclictest0-21swapper/309:15:013
163199220,1cyclictest20983-21cut12:15:003
163199220,1cyclictest16243-21/usr/sbin/munin08:47:313
1627992220,1cyclictest8580-21cron08:30:002
1627992220,1cyclictest30282-21sh09:15:002
1627992220,1cyclictest0-21swapper/212:20:122
162799220,21cyclictest5387-21apt-get10:35:012
162799220,21cyclictest362-21apt10:27:322
162799220,21cyclictest14033-21munin-run07:35:002
162799220,21cyclictest0-21swapper/210:17:332
162799220,21cyclictest0-21swapper/209:07:312
162799220,21cyclictest0-21swapper/208:57:322
162799220,21cyclictest0-21swapper/208:07:312
162799220,1cyclictest6822-21munin-run07:20:002
1622992219,2cyclictest0-21swapper/109:37:311
162299221,20cyclictest26637-21apt-get12:27:331
162299220,21cyclictest8040-21apt-get10:42:331
162299220,21cyclictest363-21apt-get10:27:321
162299220,21cyclictest0-21swapper/110:52:311
162299220,1cyclictest20424-21apt-get10:02:311
1621992219,2cyclictest593-21dbus-daemon10:25:010
162199220,22cyclictest0-21swapper/007:40:020
162199220,21cyclictest31850-21apt-get08:12:300
162199220,21cyclictest0-21swapper/011:42:310
162199220,21cyclictest0-21swapper/011:07:320
162199220,1cyclictest8343-21apt-get09:37:310
162199220,1cyclictest3818-21munin-run08:19:590
1631992121,0cyclictest0-21swapper/307:22:303
1631992120,0cyclictest0-21swapper/308:12:323
1631992119,1cyclictest3547-21apt09:27:323
163199210,21cyclictest19848-21apt-get07:47:333
163199210,21cyclictest0-21swapper/312:20:113
163199210,21cyclictest0-21swapper/309:12:343
163199210,20cyclictest21123-21apt-get08:57:323
163199210,20cyclictest0-21swapper/312:32:323
163199210,20cyclictest0-21swapper/311:47:323
163199210,1cyclictest6814-21turbostat07:20:003
163199210,1cyclictest0-21swapper/312:00:003
1627992121,0cyclictest0-21swapper/212:02:302
1627992120,0cyclictest0-21swapper/211:32:312
1627992120,0cyclictest0-21swapper/207:27:322
162799210,20cyclictest0-21swapper/210:07:312
162799210,1cyclictest0-21swapper/211:40:002
162799210,0cyclictest0-21swapper/209:27:312
1622992121,0cyclictest22242-21apt-get07:52:331
1622992121,0cyclictest19848-21apt-get07:47:331
1622992121,0cyclictest0-21swapper/112:22:331
1622992120,1cyclictest6662-21apt-get08:27:301
1622992120,1cyclictest0-21swapper/111:32:311
1622992120,1cyclictest0-21swapper/108:12:311
1622992119,1cyclictest30292-21munin-run09:14:591
1622992119,1cyclictest16529-21turbostat.cron07:40:001
162299210,20cyclictest17753-21apt-get11:02:341
162299210,20cyclictest0-21swapper/112:02:311
162299210,20cyclictest0-21swapper/108:17:321
162299210,20cyclictest0-21swapper/107:15:011
162299210,1cyclictest26605-21sendmail-msp08:00:011
162299210,1cyclictest0-21swapper/109:57:351
162299210,0cyclictest31133-21apt-get09:17:311
162299210,0cyclictest0-21swapper/110:32:311
162299210,0cyclictest0-21swapper/109:27:321
162299210,0cyclictest0-21swapper/109:02:321
1621992121,0cyclictest7279-21apt-get07:22:290
1621992121,0cyclictest24240-21apt-get12:22:330
1621992121,0cyclictest0-21swapper/012:07:330
1621992121,0cyclictest0-21swapper/011:27:320
1621992121,0cyclictest0-21swapper/010:27:320
1621992121,0cyclictest0-21swapper/009:32:300
1621992120,0cyclictest4627-21apt-get07:15:000
1621992119,1cyclictest28588-21munin-plugin-st12:30:000
162199210,20cyclictest0-21swapper/012:17:320
162199210,1cyclictest15553-21apt-get09:52:310
162199210,0cyclictest0-21swapper/011:22:320
162199210,0cyclictest0-21swapper/010:07:310
1631992020,0cyclictest28017-21apt-get10:17:343
1631992020,0cyclictest0-21swapper/312:27:333
1631992020,0cyclictest0-21swapper/311:42:313
1631992020,0cyclictest0-21swapper/310:02:313
1631992020,0cyclictest0-21swapper/309:22:313
1631992020,0cyclictest0-21swapper/309:22:313
1631992020,0cyclictest0-21swapper/308:17:323
1631992020,0cyclictest0-21swapper/307:32:323
1631992019,0cyclictest25380-21apt-get10:10:023
163199200,20cyclictest0-21swapper/312:37:333
163199200,20cyclictest0-21swapper/311:12:343
163199200,20cyclictest0-21swapper/308:37:323
163199200,0cyclictest0-21swapper/311:27:313
163199200,0cyclictest0-21swapper/310:57:323
163199200,0cyclictest0-21swapper/310:30:003
163199200,0cyclictest0-21swapper/309:52:313
1627992020,0cyclictest0-21swapper/212:12:332
1627992020,0cyclictest0-21swapper/211:12:332
1627992020,0cyclictest0-21swapper/209:47:322
1627992020,0cyclictest0-21swapper/208:17:322
1627992020,0cyclictest0-21swapper/208:12:322
1627992019,1cyclictest0-21swapper/212:40:002
1627992018,1cyclictest26593-21basename08:00:012
1627992018,1cyclictest19682-21cron11:05:002
1627992018,1cyclictest1-21systemd08:05:002
162799200,20cyclictest0-21swapper/209:57:352
162799200,19cyclictest0-21swapper/210:47:322
162799200,0cyclictest0-21swapper/211:17:322
162799200,0cyclictest0-21swapper/210:52:322
162799200,0cyclictest0-21swapper/210:32:312
1622992020,0cyclictest0-21swapper/110:37:311
1622992020,0cyclictest0-21swapper/110:17:341
1622992019,1cyclictest15278-21apt-get10:57:311
1622992019,0cyclictest22820-21apt-get10:07:301
1622992018,1cyclictest13159-21apt-get09:47:321
162299200,20cyclictest0-21swapper/111:47:321
162299200,20cyclictest0-21swapper/108:22:311
1621992020,0cyclictest0-21swapper/010:10:020
1621992020,0cyclictest0-21swapper/008:57:310
1621992020,0cyclictest0-21swapper/007:47:330
1621992019,1cyclictest29036-21apt-get12:32:300
1621992018,1cyclictest14013-21dump-pmu-power07:35:000
1621992017,2cyclictest313-21systemd-journal09:04:590
162199200,0cyclictest0-21swapper/011:52:320
163199198,11cyclictest34-21ksoftirqd/307:42:323
1631991919,0cyclictest23540-21apt-get09:02:323
1631991919,0cyclictest18011-21apt-get09:57:353
1631991919,0cyclictest0-21swapper/310:52:323
1631991919,0cyclictest0-21swapper/310:37:313
1631991919,0cyclictest0-21swapper/310:32:313
1631991919,0cyclictest0-21swapper/310:07:313
1631991919,0cyclictest0-21swapper/308:00:003
1631991918,1cyclictest30432-21apt-get10:22:323
1631991917,2cyclictest0-21swapper/307:27:323
1627991919,0cyclictest14472-21apt-get07:37:312
1627991919,0cyclictest0-21swapper/211:47:322
1627991919,0cyclictest0-21swapper/211:07:312
1627991919,0cyclictest0-21swapper/210:12:322
1627991919,0cyclictest0-21swapper/210:02:312
1622991919,0cyclictest11462-21apt-get08:37:321
1622991919,0cyclictest0-21swapper/112:12:331
1622991919,0cyclictest0-21swapper/111:52:311
1622991919,0cyclictest0-21swapper/107:42:331
1622991918,1cyclictest6517-21awk11:45:001
1622991917,1cyclictest9237-21munin-run07:25:001
1622991917,1cyclictest12714-21munin-run09:45:001
162299190,18cyclictest0-21swapper/109:07:311
1621991919,0cyclictest0-21swapper/010:47:320
1621991919,0cyclictest0-21swapper/009:17:320
1621991919,0cyclictest0-21swapper/008:52:330
1621991918,1cyclictest0-21swapper/007:27:320
1621991917,1cyclictest17306-21munin-run11:00:000
1631991818,0cyclictest0-21swapper/312:02:313
1631991818,0cyclictest0-21swapper/311:02:343
1631991818,0cyclictest0-21swapper/308:07:303
1631991818,0cyclictest0-21swapper/307:37:313
1631991817,1cyclictest0-21swapper/308:22:313
1631991817,0cyclictest32156-21apt-get11:32:313
1631991816,1cyclictest4126-21sendmail-msp11:40:003
1627991818,0cyclictest0-21swapper/211:57:342
1627991818,0cyclictest0-21swapper/207:52:332
162799180,17cyclictest594-21wpa_supplicant12:30:362
1622991818,0cyclictest21427-21apt-get12:17:321
1622991818,0cyclictest0-21swapper/108:02:311
1622991816,2cyclictest0-21swapper/109:22:301
1622991816,2cyclictest0-21swapper/109:22:291
1621991816,1cyclictest29975-21latency_hist10:20:000
162199180,18cyclictest28308-21/usr/sbin/munin09:12:340
1631991717,0cyclictest0-21swapper/311:17:323
1631991717,0cyclictest0-21swapper/308:27:313
1631991716,1cyclictest9367-21apt-get11:52:323
1631991716,1cyclictest0-21swapper/309:07:313
1627991717,0cyclictest5941-21apt-get09:32:292
1627991717,0cyclictest0-21swapper/212:27:332
1627991717,0cyclictest0-21swapper/212:17:322
1627991717,0cyclictest0-21swapper/209:52:312
1627991717,0cyclictest0-21swapper/209:02:322
1627991717,0cyclictest0-21swapper/208:47:312
1627991717,0cyclictest0-21swapper/208:37:312
1627991717,0cyclictest0-21swapper/208:32:322
1627991717,0cyclictest0-21swapper/207:12:312
1627991716,0cyclictest0-21swapper/208:42:302
162799170,16cyclictest583-21avahi-daemon10:22:382
1622991717,0cyclictest0-21swapper/109:52:311
1622991717,0cyclictest0-21swapper/108:52:331
1622991716,0cyclictest29453-21apt-get08:07:311
162299170,0cyclictest0-21swapper/112:07:321
1621991717,0cyclictest0-21swapper/012:37:310
1621991717,0cyclictest0-21swapper/009:57:340
1621991717,0cyclictest0-21swapper/008:42:300
1621991717,0cyclictest0-21swapper/007:12:300
1621991715,1cyclictest4785-21awk10:35:000
162199170,0cyclictest0-21swapper/008:27:310
1631991615,0cyclictest0-21swapper/312:07:323
1627991615,0cyclictest0-21swapper/209:22:302
1627991615,0cyclictest0-21swapper/209:22:302
1627991614,1cyclictest9211-21cron07:25:002
1627991614,1cyclictest593-21dbus-daemon07:44:592
162799160,15cyclictest20414-21unixbench_singl08:52:422
1622991616,0cyclictest24941-21apt-get11:17:321
1622991616,0cyclictest0-21swapper/111:12:341
1622991616,0cyclictest0-21swapper/108:42:301
1622991615,0cyclictest0-21swapper/109:32:301
1622991613,2cyclictest0-21swapper/110:22:321
1621991616,0cyclictest0-21swapper/011:02:340
1621991616,0cyclictest0-21swapper/008:47:310
1621991616,0cyclictest0-21swapper/008:02:310
162199160,15cyclictest6238-21grep10:37:330
162199160,15cyclictest3548-21apt-get09:27:320
1631991515,0cyclictest0-21swapper/312:17:313
1631991515,0cyclictest0-21swapper/309:47:323
1631991514,1cyclictest27057-21apt-get08:02:313
1631991511,4cyclictest12678-21cron09:44:593
1627991515,0cyclictest0-21swapper/209:37:312
162799150,14cyclictest17258-21readlink11:00:002
162299150,1cyclictest0-21swapper/108:57:401
1621991515,0cyclictest0-21swapper/012:05:010
1621991515,0cyclictest0-21swapper/011:17:320
1621991515,0cyclictest0-21swapper/010:02:310
1621991515,0cyclictest0-21swapper/007:57:290
1621991514,1cyclictest612-21gdbus09:50:000
1621991513,1cyclictest0-21swapper/007:55:010
162199150,1cyclictest20747-21users12:12:430
162199150,15cyclictest0-21swapper/008:22:310
163199140,14cyclictest0-21swapper/309:30:333
163199140,14cyclictest0-21swapper/308:53:223
163199140,0cyclictest0-21swapper/307:52:163
1627991414,0cyclictest0-21swapper/211:44:192
1627991413,0cyclictest0-21swapper/209:17:322
162799140,14cyclictest0-21swapper/212:06:372
162799140,14cyclictest0-21swapper/211:25:342
162799140,0cyclictest0-21swapper/211:50:332
162799140,0cyclictest0-21swapper/209:40:162
1622991414,0cyclictest0-21swapper/111:27:321
1622991414,0cyclictest0-21swapper/108:47:311
1622991414,0cyclictest0-21swapper/107:34:101
162299140,0cyclictest0-21swapper/111:57:021
162299140,0cyclictest0-21swapper/111:20:461
162199141,9cyclictest61550irq/126-eno109:40:070
1621991414,0cyclictest0-21swapper/011:12:430
162199140,14cyclictest0-21swapper/009:06:440
162199140,14cyclictest0-21swapper/008:05:010
162199140,0cyclictest0-21swapper/011:55:520
162199140,0cyclictest0-21swapper/011:31:030
162199140,0cyclictest0-21swapper/010:51:060
162199140,0cyclictest0-21swapper/009:20:380
162199140,0cyclictest0-21swapper/009:20:380
162199140,0cyclictest0-21swapper/008:34:370
163199130,0cyclictest0-21swapper/308:40:093
1627991313,0cyclictest0-21swapper/211:20:292
162799130,13cyclictest0-21swapper/208:20:542
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional