You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 22:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Sat Jun 28, 2025 12:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3035099240,1cyclictest13481-21apt-get07:44:312
30345992422,1cyclictest22959-21cron10:15:001
30345992421,2cyclictest313-21systemd-journal10:04:591
30354992321,1cyclictest0-21swapper/308:00:003
3035499230,0cyclictest0-21swapper/309:14:323
30350992319,3cyclictest3122-21sendmail-msp07:20:002
3035099231,21cyclictest10350-21grep07:35:002
3035099230,22cyclictest26118-21apt-get09:15:002
3035099230,22cyclictest0-21swapper/212:39:342
3034599230,22cyclictest30185-21awk10:30:001
30342992321,1cyclictest11736-21dump-pmu-power08:45:000
3034299230,22cyclictest30173-21latency_hist10:30:010
3034299230,22cyclictest0-21swapper/010:14:320
30354992220,1cyclictest7762-21apt-get11:55:023
30354992220,1cyclictest2901-21apt-get11:49:333
30354992219,2cyclictest7950-21awk07:30:003
3035499220,21cyclictest7854-21munin-run10:50:003
3035499220,21cyclictest27345-21apt-get12:39:343
3035499220,21cyclictest15044-21perf11:05:013
3035499220,21cyclictest0-21swapper/312:04:343
3035499220,1cyclictest593-21dbus-daemon12:20:003
3035499220,1cyclictest23308-21apt-get11:24:373
3035499220,0cyclictest0-21swapper/307:39:333
30350992220,1cyclictest18187-21idleruntime-cro10:04:592
3035099220,21cyclictest26739-21apt-get09:15:032
3035099220,21cyclictest23722-21turbostat.cron09:10:002
3035099220,21cyclictest22988-21perf10:15:002
3035099220,21cyclictest0-21swapper/209:25:002
3034599220,21cyclictest9197-21apt-get09:49:351
3034599220,21cyclictest25084-21latency_hist11:25:001
3034599220,21cyclictest0-21swapper/112:15:001
3034599220,21cyclictest0-21swapper/110:44:361
3034599220,21cyclictest0-21swapper/107:44:311
3034599220,1cyclictest5078-21apt-get08:34:341
3034599220,1cyclictest313-21systemd-journal11:00:001
30342992222,0cyclictest0-21swapper/009:59:340
30342992220,1cyclictest7851-21perf10:50:000
30342992219,2cyclictest596-21in:imuxsock10:04:590
30342992219,2cyclictest21962-21munin-plugin-st12:25:000
30342992219,2cyclictest19271-21cut07:54:370
30342992219,2cyclictest12627-21sendmail11:00:000
3034299220,21cyclictest6798-21apt-get09:44:310
3034299220,21cyclictest30499-21apt-get11:39:350
3034299220,21cyclictest27490-21perf11:30:000
3034299220,21cyclictest0-21swapper/012:39:340
3034299220,21cyclictest0-21swapper/011:59:310
3034299220,21cyclictest0-21swapper/011:25:010
3034299220,21cyclictest0-21swapper/011:14:330
3034299220,21cyclictest0-21swapper/009:54:340
3034299220,1cyclictest593-21dbus-daemon07:39:590
3034299220,1cyclictest5300-21apt-get11:54:340
30354992119,1cyclictest3131-21awk07:20:003
30354992119,1cyclictest21939-21apt12:25:003
3035499210,21cyclictest15777-21dump-pmu-power10:00:003
3035499210,20cyclictest0-21swapper/311:14:333
3035499210,20cyclictest0-21swapper/311:09:373
3035499210,1cyclictest2291-21latency_hist11:45:003
3035499210,1cyclictest0-21swapper/310:44:353
3035499210,0cyclictest0-21swapper/309:39:313
30350992121,0cyclictest0-21swapper/211:24:372
30350992120,1cyclictest29888-21perf11:35:002
30350992120,0cyclictest22554-21apt-get12:29:342
30350992120,0cyclictest15706-21apt-get11:09:362
30350992119,1cyclictest9341-21munin-run08:40:002
30350992119,1cyclictest11590-21apt-get09:54:352
30350992118,2cyclictest12354-21grep12:05:002
3035099210,0cyclictest0-21swapper/209:59:352
3035099210,0cyclictest0-21swapper/208:29:312
3035099210,0cyclictest0-21swapper/208:14:312
3035099210,0cyclictest0-21swapper/207:49:312
30345992121,0cyclictest0-21swapper/109:44:321
30345992119,2cyclictest29892-21wc11:35:011
30345992119,1cyclictest16517-21turbostat08:55:001
3034599210,21cyclictest12957-21apt-get12:09:361
3034599210,21cyclictest0-21swapper/111:39:351
3034599210,21cyclictest0-21swapper/111:29:351
3034599210,21cyclictest0-21swapper/109:15:031
3034599210,20cyclictest3108-21perf07:20:001
3034599210,20cyclictest11069-21apt-get07:39:331
3034599210,20cyclictest0-21swapper/111:14:331
3034599210,0cyclictest20892-21apt-get11:19:371
3034599210,0cyclictest0-21swapper/111:09:361
30342992120,1cyclictest28101-21apt-get11:34:370
30342992119,1cyclictest14730-21sh12:10:000
3034299210,20cyclictest8449-21apt-get10:54:340
3034299210,20cyclictest0-21swapper/010:24:330
3034299210,20cyclictest0-21swapper/008:04:330
3034299210,1cyclictest25370-21perf10:20:000
3034299210,0cyclictest0-21swapper/010:09:340
3034299210,0cyclictest0-21swapper/008:59:320
3034299210,0cyclictest0-21swapper/007:34:330
30354992020,0cyclictest0-21swapper/311:54:343
30354992020,0cyclictest0-21swapper/310:04:333
30354992020,0cyclictest0-21swapper/309:29:323
30354992020,0cyclictest0-21swapper/308:54:363
30354992018,1cyclictest4475-21munin-run08:30:003
30354992018,1cyclictest17649-21cron07:50:003
3035499200,20cyclictest0-21swapper/312:29:343
3035499200,0cyclictest19538-21apt-get09:04:333
3035499200,0cyclictest0-21swapper/311:39:353
3035499200,0cyclictest0-21swapper/309:24:333
3035499200,0cyclictest0-21swapper/307:54:333
3035499200,0cyclictest0-21swapper/307:14:333
30350992020,0cyclictest0-21swapper/212:24:352
30350992020,0cyclictest0-21swapper/212:19:332
30350992020,0cyclictest0-21swapper/211:49:342
30350992020,0cyclictest0-21swapper/211:19:372
30350992020,0cyclictest0-21swapper/211:04:332
30350992020,0cyclictest0-21swapper/210:59:342
30350992020,0cyclictest0-21swapper/209:49:362
30350992020,0cyclictest0-21swapper/208:59:322
30350992019,0cyclictest32689-21apt-get08:24:332
30350992018,1cyclictest18271-21apt-get07:54:332
30350992018,1cyclictest14138-21munin-run08:50:002
3035099200,20cyclictest0-21swapper/207:24:302
3035099200,19cyclictest0-21swapper/207:29:352
3035099200,0cyclictest0-21swapper/211:39:352
3035099200,0cyclictest0-21swapper/211:29:352
3035099200,0cyclictest0-21swapper/210:54:342
3035099200,0cyclictest0-21swapper/208:54:372
30345992020,0cyclictest5995-21apt-get10:49:311
30345992020,0cyclictest0-21swapper/110:54:341
30345992019,0cyclictest24328-21apt-get09:14:321
30345992018,1cyclictest593-21dbus-daemon12:29:591
30345992018,1cyclictest2298-21basename11:45:001
3034599200,20cyclictest0-21swapper/108:44:341
3034599200,19cyclictest32090-21lockfile-create08:20:001
3034599200,0cyclictest0-21swapper/111:04:331
30342992020,0cyclictest27865-21apt-get08:14:310
30342992020,0cyclictest13257-21apt-get11:04:330
30342992020,0cyclictest0-21swapper/011:19:370
30342992020,0cyclictest0-21swapper/008:54:360
30342992018,1cyclictest26713-21cron12:35:000
30342992010,9cyclictest11609-21ntp_states12:04:390
3034299200,20cyclictest0-21swapper/009:14:320
30354991919,0cyclictest0-21swapper/312:34:363
30354991919,0cyclictest0-21swapper/312:09:363
30354991919,0cyclictest0-21swapper/311:19:373
30354991919,0cyclictest0-21swapper/309:19:333
30354991918,1cyclictest8568-21apt-get07:34:323
30354991918,0cyclictest0-21swapper/311:34:373
3035499190,0cyclictest0-21swapper/308:44:333
30350991919,0cyclictest24954-21apt-get12:34:362
30350991919,0cyclictest0-21swapper/211:54:342
30350991919,0cyclictest0-21swapper/210:34:342
30350991919,0cyclictest0-21swapper/209:35:002
30350991919,0cyclictest0-21swapper/209:29:322
30350991918,1cyclictest30917-21apt-get07:14:322
3035099190,1cyclictest0-21swapper/209:39:312
30345991919,0cyclictest0-21swapper/112:34:361
30345991919,0cyclictest0-21swapper/111:55:021
30345991919,0cyclictest0-21swapper/111:49:321
30345991919,0cyclictest0-21swapper/108:59:321
30345991919,0cyclictest0-21swapper/108:29:311
30345991919,0cyclictest0-21swapper/107:14:321
30345991917,1cyclictest2984-21perf10:40:001
30342991919,0cyclictest1931-21apt-get09:34:330
30342991919,0cyclictest0-21swapper/011:09:360
30342991919,0cyclictest0-21swapper/008:34:350
30342991919,0cyclictest0-21swapper/007:29:350
3034299190,18cyclictest612-21gdbus07:45:000
30354991818,0cyclictest27796-21perf10:25:003
30354991818,0cyclictest0-21swapper/310:59:343
30354991818,0cyclictest0-21swapper/310:39:333
30354991818,0cyclictest0-21swapper/310:19:363
30354991818,0cyclictest0-21swapper/308:24:333
30354991818,0cyclictest0-21swapper/308:19:323
30354991818,0cyclictest0-21swapper/308:09:363
30354991817,1cyclictest0-21swapper/308:34:353
30354991817,0cyclictest0-21swapper/309:54:343
30354991816,1cyclictest25699-21apt-get11:29:353
30354991816,1cyclictest0-21swapper/308:05:003
30350991818,0cyclictest3603-21apt-get10:44:352
30350991818,0cyclictest0-21swapper/210:49:312
30350991818,0cyclictest0-21swapper/207:59:312
30350991817,0cyclictest0-21swapper/210:09:332
30345991817,0cyclictest30798-21apt-get10:34:341
30345991817,0cyclictest18793-21apt-get10:09:341
3034599180,18cyclictest0-21swapper/108:14:311
3034599180,17cyclictest0-21swapper/107:49:311
30342991818,0cyclictest0-21swapper/009:49:360
30342991818,0cyclictest0-21swapper/007:24:290
30342991817,1cyclictest0-21swapper/010:44:350
30342991816,1cyclictest3107-21latency_hist07:20:000
3034299180,17cyclictest0-21swapper/010:34:350
30354991717,0cyclictest0-21swapper/312:14:323
30354991717,0cyclictest0-21swapper/310:34:343
30354991717,0cyclictest0-21swapper/309:34:323
30354991717,0cyclictest0-21swapper/308:49:323
3035499170,16cyclictest0-21swapper/309:49:363
30350991717,0cyclictest18490-21apt-get11:14:332
30350991716,0cyclictest0-21swapper/209:44:312
3035099170,16cyclictest0-21swapper/208:20:002
30345991717,0cyclictest0-21swapper/109:59:341
30345991717,0cyclictest0-21swapper/107:24:301
30345991716,1cyclictest25478-21apt-get08:09:361
3034599170,1cyclictest13369-21turbostat09:55:001
30342991717,0cyclictest0-21swapper/012:14:320
30342991717,0cyclictest0-21swapper/009:15:030
30342991717,0cyclictest0-21swapper/008:39:340
30342991715,1cyclictest32086-21awk08:20:000
30354991616,0cyclictest0-21swapper/308:59:323
30354991616,0cyclictest0-21swapper/308:39:343
30354991615,0cyclictest21185-21apt-get10:14:333
30354991614,1cyclictest0-21swapper/309:44:333
30354991614,1cyclictest0-21swapper/307:45:003
30350991616,0cyclictest23586-21apt-get10:19:352
30350991616,0cyclictest0-21swapper/212:09:362
30350991616,0cyclictest0-21swapper/209:04:322
30350991614,1cyclictest2992-21sendmail-msp10:40:002
30345991616,0cyclictest12335-21apt-get08:49:321
30345991616,0cyclictest0-21swapper/112:19:331
30345991616,0cyclictest0-21swapper/111:54:341
30345991616,0cyclictest0-21swapper/109:34:321
30345991615,1cyclictest25997-21apt-get10:24:331
30345991615,1cyclictest0-21swapper/107:59:311
30345991615,0cyclictest0-21swapper/110:19:361
30342991616,0cyclictest2672-21apt-get08:29:310
30342991616,0cyclictest0-21swapper/008:09:360
30342991614,1cyclictest195052sleep009:04:310
30354991515,0cyclictest0-21swapper/310:09:343
30354991513,1cyclictest30161-21sh10:30:003
3035499150,1cyclictest0-21swapper/310:54:443
3035499150,14cyclictest0-21swapper/307:20:393
30350991514,1cyclictest0-21swapper/208:34:342
30350991512,2cyclictest11702-21cron08:45:002
30345991515,0cyclictest0-21swapper/109:04:321
30345991515,0cyclictest0-21swapper/108:40:001
30345991513,1cyclictest12359-21munin-run12:05:001
3034299155,9cyclictest781-21lldpd12:28:380
3034299153,12cyclictest766-21snmpd09:21:420
30342991513,1cyclictest2299-21cat11:45:000
30342991513,1cyclictest0-21swapper/011:50:000
3035499140,14cyclictest0-21swapper/308:10:083
3035499140,0cyclictest0-21swapper/309:05:203
30350991414,0cyclictest7137-21apt-get11:55:022
30350991414,0cyclictest0-21swapper/212:10:372
30350991414,0cyclictest0-21swapper/210:20:132
3035099140,14cyclictest0-21swapper/208:05:092
3035099140,14cyclictest0-21swapper/207:36:152
30345991414,0cyclictest0-21swapper/112:35:201
3034599140,14cyclictest0-21swapper/112:21:051
3034599140,14cyclictest0-21swapper/109:26:081
3034599140,14cyclictest0-21swapper/109:21:111
3034599140,14cyclictest0-21swapper/107:50:141
3034599140,0cyclictest0-21swapper/109:35:141
3034599140,0cyclictest0-21swapper/107:30:281
3034599140,0cyclictest0-21swapper/107:27:551
30342991414,0cyclictest0-21swapper/010:35:170
30342991414,0cyclictest0-21swapper/008:46:390
3034299140,14cyclictest0-21swapper/009:35:210
3034299140,14cyclictest0-21swapper/009:05:020
3034299140,14cyclictest0-21swapper/007:56:510
3034299140,14cyclictest0-21swapper/007:45:160
3034299140,14cyclictest0-21swapper/007:10:270
3034299140,0cyclictest0-21swapper/012:15:130
3034299140,0cyclictest0-21swapper/009:25:020
3034299140,0cyclictest0-21swapper/008:20:220
30350991313,0cyclictest0-21swapper/208:00:152
3035099130,0cyclictest0-21swapper/210:25:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional