You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 16:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Tue Aug 26, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2364699250,2cyclictest25449-21idleruntime-cro09:25:002
2364899240,23cyclictest17600-21latency_hist11:19:593
23646992413,4cyclictest29608-21nvmesmart_nvme010:39:422
2364299240,23cyclictest16125-21apt-get10:14:321
2364299240,1cyclictest7887-21apt-get12:09:381
2364899230,22cyclictest10641-21idleruntime-cro08:55:003
2364899230,22cyclictest10397-21turbostat.cron11:04:593
2364699230,0cyclictest0-21swapper/210:34:352
23642992321,1cyclictest20347-21munin-run10:20:001
2364299230,22cyclictest21338-21apt-get10:24:341
23641992323,0cyclictest0-21swapper/012:29:320
2364199230,2cyclictest12516-21dump-pmu-power12:15:000
23648992220,1cyclictest27419-21munin-run08:25:003
23648992220,1cyclictest0-21swapper/308:15:003
2364899221,0cyclictest0-21swapper/308:35:033
2364899220,21cyclictest5717-21latency_hist07:40:013
2364899220,21cyclictest17347-21munin-run12:25:003
2364899220,21cyclictest0-21swapper/308:04:323
2364899220,1cyclictest8734-21apt-get07:49:333
2364899220,1cyclictest313-21systemd-journal10:55:003
2364899220,1cyclictest28411-21dump-pmu-power07:20:003
2364899220,1cyclictest14950-21awk12:20:003
23646992221,0cyclictest14056-21apt-get09:04:322
23646992220,1cyclictest20613-21apt-get11:29:332
2364699220,21cyclictest6080-21apt-get09:54:382
2364699220,21cyclictest593-21dbus-daemon08:39:592
2364699220,21cyclictest31428-21apt-get07:29:362
2364699220,21cyclictest23123-21perf10:25:002
2364699220,21cyclictest20344-21munin-plugin-st10:20:002
2364699220,21cyclictest15999-21apt-get08:04:322
2364699220,21cyclictest0-21swapper/208:29:362
2364699220,1cyclictest3921-21apt-get07:39:352
2364699220,1cyclictest0-21swapper/211:00:002
23642992222,0cyclictest0-21swapper/108:55:011
23642992220,1cyclictest5720-21perf07:40:001
23642992220,1cyclictest32728-21cron10:45:001
23642992220,1cyclictest24826-21munin-run11:35:001
23642992220,1cyclictest24826-21munin-run11:35:001
23642992220,1cyclictest24563-21munin-plugin-st12:40:001
23642992220,1cyclictest22586-21grep08:15:011
23642992220,1cyclictest20653-21munin-plugin-st09:15:011
2364299220,21cyclictest29031-21apt-get07:24:361
2364299220,21cyclictest28478-21apt-get09:34:361
2364299220,21cyclictest0-21swapper/111:59:361
2364299220,21cyclictest0-21swapper/109:29:371
2364299220,21cyclictest0-21swapper/109:24:331
2364299220,21cyclictest0-21swapper/109:04:331
2364299220,1cyclictest8179-21cron08:49:591
2364299220,1cyclictest5132-21apt-get10:55:001
2364299220,1cyclictest2474-21idleruntime-cro11:54:591
2364299220,1cyclictest0-21swapper/109:20:011
2364299220,1cyclictest0-21swapper/108:35:001
23641992221,1cyclictest28409-21apt-get08:29:350
23641992220,1cyclictest20159-21cron08:10:010
2364199220,21cyclictest23122-21grep10:24:590
2364199220,21cyclictest1177-21apt-get08:39:350
2364199220,21cyclictest0-21swapper/007:14:370
2364199220,1cyclictest7933-21latency_hist11:00:010
23648992121,0cyclictest0-21swapper/310:24:343
23648992121,0cyclictest0-21swapper/308:19:333
23648992121,0cyclictest0-21swapper/307:14:363
23648992119,1cyclictest598-21rs:main3
23648992118,2cyclictest593-21dbus-daemon10:19:593
2364899210,20cyclictest26079-21apt-get09:29:383
2364899210,20cyclictest19716-21idleruntime-cro12:30:003
2364899210,20cyclictest0-21swapper/312:09:373
2364899210,20cyclictest0-21swapper/311:49:333
2364899210,20cyclictest0-21swapper/309:49:323
2364899210,20cyclictest0-21swapper/308:29:343
2364899210,20cyclictest0-21swapper/307:29:353
2364899210,1cyclictest0-21swapper/310:09:323
2364899210,0cyclictest0-21swapper/307:59:333
2364699210,20cyclictest31995-21apt-get11:49:412
2364699210,20cyclictest30961-21apt-get10:44:332
2364699210,20cyclictest30824-21munin-run07:25:012
2364699210,20cyclictest0-21swapper/211:34:362
2364699210,20cyclictest0-21swapper/211:34:352
2364699210,20cyclictest0-21swapper/208:44:322
2364699210,1cyclictest24218-21apt-get07:14:362
2364699210,1cyclictest15508-21awk10:10:002
2364699210,0cyclictest31267-21apt-get09:39:332
2364699210,0cyclictest0-21swapper/212:04:332
2364699210,0cyclictest0-21swapper/211:35:012
2364699210,0cyclictest0-21swapper/207:44:362
23642992120,1cyclictest13730-21apt-get10:09:331
23642992120,0cyclictest0-21swapper/111:39:341
23642992120,0cyclictest0-21swapper/107:49:331
23642992119,1cyclictest30356-21munin-run10:40:001
23642992119,1cyclictest0-21swapper/112:29:331
2364299210,21cyclictest0-21swapper/110:04:341
2364299210,21cyclictest0-21swapper/107:29:361
2364299210,20cyclictest10727-21apt-get12:14:341
2364299210,20cyclictest0-21swapper/107:14:361
2364299210,1cyclictest5458-21latency_hist09:50:001
2364299210,0cyclictest0-21swapper/111:19:341
23641992121,0cyclictest0-21swapper/011:54:350
23641992121,0cyclictest0-21swapper/008:40:010
23641992121,0cyclictest0-21swapper/007:44:350
23641992120,1cyclictest0-21swapper/009:20:000
23641992119,1cyclictest18229-21cron09:10:000
2364199210,21cyclictest0-21swapper/009:49:330
2364199210,20cyclictest0-21swapper/008:04:330
2364199210,0cyclictest0-21swapper/011:24:360
2364199210,0cyclictest0-21swapper/011:14:340
2364199210,0cyclictest0-21swapper/009:29:370
2364199210,0cyclictest0-21swapper/007:59:340
23648992020,0cyclictest0-21swapper/310:44:343
23648992020,0cyclictest0-21swapper/309:24:333
23648992020,0cyclictest0-21swapper/308:44:333
23648992019,1cyclictest1400-21apt-get07:34:323
23648992019,1cyclictest11184-21apt-get07:54:323
23648992018,1cyclictest0-21swapper/310:55:013
23648992018,1cyclictest0-21swapper/308:35:013
2364899200,20cyclictest0-21swapper/311:24:373
2364899200,0cyclictest18852-21apt-get09:14:343
2364899200,0cyclictest10940-21apt-get10:04:343
2364899200,0cyclictest0-21swapper/309:59:333
2364899200,0cyclictest0-21swapper/308:49:333
23646992020,0cyclictest26613-21apt-get07:19:332
23646992020,0cyclictest0-21swapper/209:34:352
23646992020,0cyclictest0-21swapper/209:29:382
23646992018,1cyclictest20633-21dump-pmu-power09:15:002
23646992017,2cyclictest19736-21wc12:30:012
2364699200,20cyclictest0-21swapper/208:59:332
2364699200,19cyclictest23053-21perf09:20:012
2364699200,0cyclictest0-21swapper/211:09:342
2364699200,0cyclictest0-21swapper/210:49:332
2364699200,0cyclictest0-21swapper/207:54:312
23642992020,0cyclictest2978-21apt-get08:40:011
23642992020,0cyclictest0-21swapper/110:29:341
23642992020,0cyclictest0-21swapper/109:39:331
23642992020,0cyclictest0-21swapper/108:54:331
23642992020,0cyclictest0-21swapper/107:54:311
23642992020,0cyclictest0-21swapper/107:34:321
2364299200,0cyclictest0-21swapper/109:59:341
23641992020,0cyclictest937-21apt-get10:49:330
23641992020,0cyclictest30805-21apt-get08:34:340
23641992020,0cyclictest23018-21apt-get11:34:350
23641992020,0cyclictest23018-21apt-get11:34:350
23641992020,0cyclictest0-21swapper/012:24:330
23641992020,0cyclictest0-21swapper/011:44:340
23641992020,0cyclictest0-21swapper/010:29:340
23641992020,0cyclictest0-21swapper/010:14:320
23641992020,0cyclictest0-21swapper/009:04:340
23641992020,0cyclictest0-21swapper/007:24:350
23641992020,0cyclictest0-21swapper/007:19:340
23641992019,1cyclictest0-21swapper/008:49:320
23641992018,1cyclictest12718-21apt-get10:05:020
2364199200,20cyclictest0-21swapper/007:49:330
2364199200,0cyclictest0-21swapper/011:29:320
23648991919,0cyclictest29031-21apt-get07:24:353
23648991919,0cyclictest0-21swapper/312:39:343
23648991919,0cyclictest0-21swapper/311:54:343
23648991919,0cyclictest0-21swapper/310:50:013
23648991919,0cyclictest0-21swapper/308:55:013
23648991918,1cyclictest16125-21apt-get10:14:323
23646991919,0cyclictest8469-21apt-get09:59:332
23646991919,0cyclictest0-21swapper/211:59:352
23646991919,0cyclictest0-21swapper/211:54:342
23646991919,0cyclictest0-21swapper/211:19:352
23646991919,0cyclictest0-21swapper/210:54:392
23646991919,0cyclictest0-21swapper/208:54:332
23646991919,0cyclictest0-21swapper/208:14:322
23646991919,0cyclictest0-21swapper/207:34:332
23646991918,0cyclictest0-21swapper/208:24:342
23646991917,1cyclictest18223-21apt-get11:24:372
23646991917,1cyclictest17886-21cron10:14:592
23642991919,0cyclictest0-21swapper/111:09:351
23642991918,1cyclictest5488-21apt-get12:04:341
23641991919,0cyclictest0-21swapper/011:09:350
23641991919,0cyclictest0-21swapper/010:44:330
23641991919,0cyclictest0-21swapper/010:04:350
23641991919,0cyclictest0-21swapper/008:14:310
23641991917,1cyclictest5735-21latency_hist07:40:000
23648991818,0cyclictest23746-21apt-get10:29:343
23648991818,0cyclictest0-21swapper/309:44:353
23648991818,0cyclictest0-21swapper/309:19:343
23648991817,1cyclictest34-21ksoftirqd/311:35:023
2364899180,18cyclictest0-21swapper/311:34:423
2364899180,18cyclictest0-21swapper/311:34:423
23646991818,0cyclictest0-21swapper/211:04:592
23646991817,1cyclictest15544-21apt-get12:24:332
23642991818,0cyclictest5132-21apt-get10:55:011
23642991818,0cyclictest0-21swapper/108:04:321
23642991816,2cyclictest22099-21cron12:35:001
23641991818,0cyclictest18529-21apt-get10:19:360
23641991818,0cyclictest0-21swapper/011:04:340
23641991818,0cyclictest0-21swapper/007:54:320
23641991817,1cyclictest0-21swapper/010:54:390
23641991816,1cyclictest13128-21apt-get12:19:330
23641991816,1cyclictest0-21swapper/011:19:590
23648991717,0cyclictest0-21swapper/312:04:593
23648991717,0cyclictest0-21swapper/309:34:363
23648991716,1cyclictest6338-21apt-get07:44:353
2364899170,15cyclictest0-21swapper/312:10:003
2364699174,12cyclictest23523-21munin-node08:46:472
23646991717,0cyclictest0-21swapper/208:34:352
2364699170,14cyclictest0-21swapper/209:43:402
23642991717,0cyclictest0-21swapper/111:44:331
23642991717,0cyclictest0-21swapper/108:19:341
23642991717,0cyclictest0-21swapper/107:19:341
23642991715,1cyclictest15204-21munin-plugin-st11:15:001
2364199174,12cyclictest23190-21apt-get08:19:330
23641991717,0cyclictest8469-21apt-get09:59:340
23641991717,0cyclictest0-21swapper/012:39:330
23641991717,0cyclictest0-21swapper/009:14:340
23641991716,0cyclictest24814-21apt-get11:35:010
23641991715,1cyclictest3070-21apt-get11:59:360
23641991715,1cyclictest27944-21grep10:35:000
23648991614,1cyclictest30040-21perf11:45:013
23648991614,1cyclictest22033-21sed11:29:403
23648991613,2cyclictest4891-21sendmail-msp11:59:593
23646991616,0cyclictest20345-21apt-get12:34:382
23646991616,0cyclictest0-21swapper/210:04:332
23646991616,0cyclictest0-21swapper/209:49:332
23646991616,0cyclictest0-21swapper/208:09:332
23646991614,1cyclictest18244-21perf09:10:002
23642991616,0cyclictest0-21swapper/111:04:341
23642991616,0cyclictest0-21swapper/110:49:321
23642991616,0cyclictest0-21swapper/109:44:341
23641991616,0cyclictest0-21swapper/010:39:360
23641991614,1cyclictest3057-21sh09:45:000
23648991514,1cyclictest0-21swapper/309:35:023
2364899150,1cyclictest0-21swapper/311:09:353
23646991515,0cyclictest0-21swapper/212:09:372
23646991515,0cyclictest0-21swapper/208:19:342
23646991513,1cyclictest23365-21sed12:39:362
23646991513,1cyclictest0-21swapper/207:59:322
2364699150,1cyclictest0-21swapper/211:14:382
23642991515,0cyclictest0-21swapper/109:54:381
23642991515,0cyclictest0-21swapper/107:44:351
23642991513,1cyclictest15388-21perf07:59:591
2364299150,14cyclictest22388-21apt11:30:001
23641991515,0cyclictest8793-21apt-get08:54:340
23641991515,0cyclictest0-21swapper/009:54:390
23641991513,1cyclictest22143-21munin-run12:35:000
2364899140,14cyclictest0-21swapper/311:10:583
2364899140,14cyclictest0-21swapper/310:30:043
2364899140,14cyclictest0-21swapper/309:05:523
2364899140,14cyclictest0-21swapper/309:00:403
2364899140,0cyclictest0-21swapper/312:30:313
2364899140,0cyclictest0-21swapper/309:50:093
2364899140,0cyclictest0-21swapper/308:05:273
2364699140,14cyclictest0-21swapper/211:41:262
2364699140,14cyclictest0-21swapper/210:25:382
2364699140,14cyclictest0-21swapper/207:47:572
2364699140,0cyclictest0-21swapper/212:15:262
2364699140,0cyclictest0-21swapper/212:10:172
23642991414,0cyclictest25617-21apt-get08:24:341
23642991414,0cyclictest0-21swapper/111:20:111
23642991414,0cyclictest0-21swapper/110:32:321
23642991414,0cyclictest0-21swapper/108:06:101
2364299140,14cyclictest0-21swapper/112:20:061
2364299140,14cyclictest0-21swapper/109:05:431
2364299140,0cyclictest0-21swapper/111:45:201
2364299140,0cyclictest0-21swapper/108:35:151
23641991414,0cyclictest0-21swapper/012:05:450
23641991414,0cyclictest0-21swapper/009:35:070
23641991414,0cyclictest0-21swapper/008:22:390
2364199140,14cyclictest0-21swapper/009:30:190
2364199140,14cyclictest0-21swapper/009:21:050
2364199140,0cyclictest0-21swapper/008:56:100
2364199140,0cyclictest0-21swapper/007:31:400
2364199140,0cyclictest0-21swapper/007:25:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional