You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-15 - 22:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Mon Sep 15, 2025 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18980992624,1cyclictest17248-21awk09:19:590
1899099252,2cyclictest593-21dbus-daemon12:20:013
1898699250,24cyclictest0-21swapper/211:56:172
18980992525,0cyclictest29716-21latency_hist10:50:000
1899099240,23cyclictest5153-21awk08:55:003
1899099240,23cyclictest17258-21idleruntime-cro09:20:003
18986992422,1cyclictest0-21swapper/207:15:012
18986992419,4cyclictest0-21swapper/207:56:172
1898699240,23cyclictest15649-21apt-get12:36:172
18990992320,2cyclictest7379-21perf11:10:003
1899099230,22cyclictest14901-21sh10:20:003
1899099230,22cyclictest1452-21wc07:39:593
18986992323,0cyclictest22858-21apt-get10:36:172
18986992322,1cyclictest12536-21apt-get11:21:162
18986992321,1cyclictest32763-21cron08:44:592
1898699230,0cyclictest10112-21apt-get11:16:132
18983992321,1cyclictest17250-21munin-run09:19:591
1898399230,22cyclictest0-21swapper/107:31:171
18980992322,1cyclictest0-21swapper/011:31:160
18980992320,2cyclictest19662-21munin-run09:24:590
1898099231,21cyclictest0-21swapper/010:06:170
18990992220,1cyclictest32036-21sh09:50:003
18990992220,1cyclictest20002-21apt-get09:26:173
18990992219,2cyclictest7646-21idleruntime-cro09:00:003
18990992217,2cyclictest0-21swapper/310:31:183
1899099220,21cyclictest323-21perf08:44:593
1899099220,21cyclictest28309-21apt-get08:36:153
1899099220,21cyclictest27668-21apt-get10:46:133
1899099220,21cyclictest21712-21apt-get07:16:163
1899099220,21cyclictest10056-21apt-get10:11:173
1899099220,21cyclictest0-21swapper/312:36:173
1899099220,21cyclictest0-21swapper/308:46:163
18986992222,0cyclictest7986-21apt-get09:01:172
18986992220,1cyclictest20105-21perf10:30:002
18986992218,3cyclictest2873-21apt-get11:01:172
1898699220,21cyclictest7722-21apt-get11:11:192
1898699220,21cyclictest28962-21latency_hist07:30:002
1898699220,21cyclictest27967-21munin-run08:34:592
1898699220,21cyclictest0-21swapper/212:21:172
1898699220,1cyclictest32114-21apt-get10:55:002
1898699220,1cyclictest0-21swapper/210:14:592
18983992222,0cyclictest0-21swapper/110:50:001
1898399220,21cyclictest21120-21apt-get08:21:171
1898399220,21cyclictest0-21swapper/111:41:161
1898399220,21cyclictest0-21swapper/111:21:171
1898399220,21cyclictest0-21swapper/109:41:161
1898399220,21cyclictest0-21swapper/109:41:161
1898399220,1cyclictest4464-21ls10:00:011
1898399220,1cyclictest0-21swapper/108:00:011
1898399220,0cyclictest15518-21apt-get12:35:021
18980992222,0cyclictest0-21swapper/008:11:130
18980992221,1cyclictest0-21swapper/011:21:180
1898099221,1cyclictest0-21swapper/007:16:150
1898099220,21cyclictest0-21swapper/012:21:180
1898099220,1cyclictest0-21swapper/008:46:160
1898099220,0cyclictest0-21swapper/009:06:160
18990992121,0cyclictest401-21apt-get10:56:163
18990992121,0cyclictest0-21swapper/309:31:183
18990992121,0cyclictest0-21swapper/309:11:143
18990992121,0cyclictest0-21swapper/307:21:193
18990992120,1cyclictest24950-21apt-get11:46:163
18990992119,1cyclictest31355-21sh07:34:593
1899099210,20cyclictest25911-21apt-get08:31:143
1899099210,0cyclictest0-21swapper/308:11:143
18986992120,0cyclictest0-21swapper/208:46:162
18986992116,4cyclictest29667-21munin-run09:45:002
18986992116,4cyclictest29667-21munin-run09:45:002
1898699210,20cyclictest22552-21apt-get11:41:162
1898699210,20cyclictest0-21swapper/209:06:172
1898699210,20cyclictest0-21swapper/207:51:182
1898699210,0cyclictest7660-21apt-get10:06:182
1898699210,0cyclictest0-21swapper/212:06:172
1898699210,0cyclictest0-21swapper/211:46:172
1898699210,0cyclictest0-21swapper/210:21:172
1898699210,0cyclictest0-21swapper/210:16:142
1898699210,0cyclictest0-21swapper/207:16:152
18983992121,0cyclictest211rcuc/108:46:161
18983992121,0cyclictest0-21swapper/107:51:191
18983992120,1cyclictest4944-21apt-get12:11:161
18983992120,1cyclictest23515-21apt-get08:26:161
18983992120,1cyclictest0-21swapper/111:31:151
18983992119,1cyclictest593-21dbus-daemon10:25:001
18983992119,1cyclictest2044-21turbostat09:55:011
1898399210,21cyclictest211rcuc/107:16:141
1898399210,20cyclictest0-21swapper/112:31:121
1898399210,20cyclictest0-21swapper/110:15:011
1898399210,20cyclictest0-21swapper/109:30:011
1898399210,0cyclictest0-21swapper/107:36:171
18980992121,0cyclictest0-21swapper/010:11:160
18980992121,0cyclictest0-21swapper/008:36:150
18980992118,2cyclictest0-21swapper/009:30:000
1898099210,21cyclictest0-21swapper/009:30:020
1898099210,20cyclictest0-21swapper/010:56:160
1898099210,20cyclictest0-21swapper/009:56:150
1898099210,20cyclictest0-21swapper/007:31:180
1898099210,0cyclictest0-21swapper/007:51:180
18990992020,0cyclictest0-21swapper/312:06:203
18990992020,0cyclictest0-21swapper/311:36:173
18990992020,0cyclictest0-21swapper/309:41:153
18990992020,0cyclictest0-21swapper/309:41:153
18990992020,0cyclictest0-21swapper/307:51:173
18990992018,1cyclictest20770-21munin-run08:20:013
18990992018,1cyclictest14584-21apt-get11:25:023
1899099200,20cyclictest0-21swapper/311:11:193
1899099200,0cyclictest0-21swapper/310:01:173
18986992020,0cyclictest18691-21apt-get08:16:162
18986992020,0cyclictest0-21swapper/211:51:172
18986992020,0cyclictest0-21swapper/208:21:162
18986992018,1cyclictest14823-21perf09:15:002
18986992017,2cyclictest0-21swapper/209:46:162
18986992014,5cyclictest29713-21perf10:49:592
18986992014,5cyclictest0-21swapper/212:20:002
1898699200,20cyclictest0-21swapper/210:31:192
1898699200,20cyclictest0-21swapper/209:26:162
1898699200,0cyclictest0-21swapper/207:31:172
18983992020,0cyclictest25194-21apt-get09:36:161
18983992020,0cyclictest24950-21apt-get11:46:171
18983992020,0cyclictest10381-21apt-get09:06:171
18983992020,0cyclictest0-21swapper/110:26:171
18983992020,0cyclictest0-21swapper/110:01:161
18983992020,0cyclictest0-21swapper/108:51:171
18983992019,1cyclictest0-21swapper/110:05:011
18983992018,1cyclictest0-21swapper/112:10:001
1898399200,0cyclictest0-21swapper/111:01:171
18980992020,0cyclictest0-21swapper/012:16:170
18980992020,0cyclictest0-21swapper/010:16:150
18980992020,0cyclictest0-21swapper/008:21:170
18980992018,2cyclictest31647-21savelog07:35:190
18980992018,1cyclictest7137-21latency_hist10:04:590
18980992018,1cyclictest23302-21grep10:36:190
18980992017,2cyclictest404-21if_enp1s012:01:180
18980992017,2cyclictest27232-21ls09:40:010
18980992017,2cyclictest0-21swapper/008:40:010
1898099200,0cyclictest0-21swapper/011:41:160
1898099200,0cyclictest0-21swapper/009:51:170
1898099200,0cyclictest0-21swapper/008:26:170
1898099200,0cyclictest0-21swapper/008:01:160
1898099200,0cyclictest0-21swapper/007:11:140
18990991919,0cyclictest19319-21apt-get07:11:153
18990991919,0cyclictest0-21swapper/312:11:163
18990991919,0cyclictest0-21swapper/311:51:163
18990991919,0cyclictest0-21swapper/310:05:023
18990991918,1cyclictest17725-21apt-get11:31:153
18986991919,0cyclictest4237-21apt-get07:46:152
18986991919,0cyclictest0-21swapper/211:26:182
18986991919,0cyclictest0-21swapper/209:56:152
18986991919,0cyclictest0-21swapper/208:51:172
18986991918,0cyclictest0-21swapper/209:51:182
18986991915,3cyclictest17597-21apt-get09:21:172
1898699190,18cyclictest0-21swapper/212:11:192
18983991919,0cyclictest7635-21perf09:00:011
18983991919,0cyclictest11504-21apt-get08:01:161
18983991919,0cyclictest0-21swapper/112:16:161
18983991919,0cyclictest0-21swapper/111:11:181
18983991919,0cyclictest0-21swapper/110:56:171
18983991919,0cyclictest0-21swapper/108:11:131
18983991918,1cyclictest20463-21apt-get10:31:191
18983991918,0cyclictest30010-21apt-get09:46:171
18983991918,0cyclictest0-21swapper/110:11:161
18980991919,0cyclictest0-21swapper/012:31:130
18980991919,0cyclictest0-21swapper/011:36:180
18980991919,0cyclictest0-21swapper/010:41:170
18980991919,0cyclictest0-21swapper/009:46:170
18980991919,0cyclictest0-21swapper/007:46:150
18980991918,1cyclictest26175-21cron07:25:010
18980991917,1cyclictest6282-21runrttasks08:56:180
18990991818,0cyclictest9815-21apt-get12:21:183
18990991818,0cyclictest30071-21apt-get10:51:173
18990991818,0cyclictest0-21swapper/311:16:143
18990991818,0cyclictest0-21swapper/310:36:163
18990991818,0cyclictest0-21swapper/309:21:173
18990991818,0cyclictest0-21swapper/308:21:183
18990991818,0cyclictest0-21swapper/307:56:173
18990991817,1cyclictest25277-21apt-get10:41:163
18990991817,0cyclictest0-21swapper/312:31:123
18986991818,0cyclictest0-21swapper/211:31:552
18986991816,1cyclictest0-21swapper/211:36:172
18986991816,1cyclictest0-21swapper/210:05:002
18986991813,4cyclictest1776-21cron12:05:012
1898699180,1cyclictest1840-21apt-get07:41:142
18983991818,0cyclictest27348-21apt-get11:51:171
18983991818,0cyclictest19319-21apt-get07:11:141
18983991818,0cyclictest0-21swapper/112:21:171
18983991818,0cyclictest0-21swapper/108:36:151
18983991816,1cyclictest310-21dump-pmu-power08:45:001
18983991816,1cyclictest26196-21dump-pmu-power07:25:001
18980991818,0cyclictest0-21swapper/012:06:160
18980991818,0cyclictest0-21swapper/008:51:160
18980991818,0cyclictest0-21swapper/008:06:140
18980991817,1cyclictest0-21swapper/011:46:240
18980991817,1cyclictest0-21swapper/008:31:210
18980991817,1cyclictest0-21swapper/007:41:190
18980991817,0cyclictest11214-21cat08:00:000
18980991817,0cyclictest0-21swapper/011:56:160
18980991816,2cyclictest26898-21acpi07:26:160
18980991816,2cyclictest0-21swapper/011:11:180
18980991816,1cyclictest19344-21hddtemp_smartct08:16:200
18980991812,6cyclictest0-21swapper/011:51:200
1898099180,17cyclictest27610-21apt-get09:41:160
1898099180,17cyclictest27610-21apt-get09:41:160
18990991717,0cyclictest0-21swapper/311:41:163
18990991717,0cyclictest0-21swapper/309:51:163
18990991717,0cyclictest0-21swapper/307:46:143
18990991716,1cyclictest29747-21apt-get11:56:173
18990991716,1cyclictest0-21swapper/308:06:153
18990991715,1cyclictest0-21swapper/308:26:173
1898699171,8cyclictest0-21swapper/208:01:172
18986991717,0cyclictest5513-21apt-get08:56:162
18986991717,0cyclictest0-21swapper/208:06:152
18986991716,0cyclictest16289-21apt-get08:11:122
18986991716,0cyclictest0-21swapper/209:30:012
18986991715,1cyclictest14883-21awk12:34:592
1898699170,16cyclictest0-21swapper/208:26:162
18983991717,0cyclictest0-21swapper/109:24:591
18983991716,0cyclictest0-21swapper/110:41:171
18983991715,1cyclictest13893-21apt-get08:06:141
18980991717,0cyclictest0-21swapper/011:25:020
18980991717,0cyclictest0-21swapper/009:01:160
18980991716,1cyclictest15661-21taskset10:21:170
18980991716,1cyclictest0-21swapper/012:30:000
18980991716,1cyclictest0-21swapper/011:06:210
18980991714,2cyclictest0-21swapper/010:31:200
18980991711,5cyclictest0-21swapper/011:01:150
18990991616,0cyclictest0-21swapper/312:30:003
18990991616,0cyclictest0-21swapper/311:21:183
18990991616,0cyclictest0-21swapper/310:20:023
18990991616,0cyclictest0-21swapper/307:41:143
1899099160,15cyclictest19072-21grep10:26:223
18986991616,0cyclictest0-21swapper/207:36:162
18983991616,0cyclictest0-21swapper/112:01:151
18983991616,0cyclictest0-21swapper/111:36:181
18983991616,0cyclictest0-21swapper/107:46:151
18983991614,2cyclictest151rcu_preempt07:28:261
18983991614,1cyclictest0-21swapper/109:29:591
18983991611,0cyclictest151rcu_preempt10:39:451
18980991616,0cyclictest0-21swapper/012:35:020
18980991616,0cyclictest0-21swapper/011:16:130
18980991616,0cyclictest0-21swapper/010:51:240
18990991515,0cyclictest0-21swapper/308:01:173
18990991515,0cyclictest0-21swapper/307:26:173
18990991513,1cyclictest1809-21grep12:05:013
1899099150,0cyclictest0-21swapper/309:56:153
1898699157,7cyclictest578-21irqbalance10:44:162
18986991513,2cyclictest0-21swapper/208:37:512
18986991513,1cyclictest2492-21turbostat.cron11:00:002
18986991513,1cyclictest12508-21perf12:29:592
1898699150,1cyclictest17197-21cron09:20:002
1898699150,15cyclictest0-21swapper/211:06:172
1898699150,14cyclictest0-21swapper/207:21:262
18983991514,1cyclictest0-21swapper/111:25:011
18983991511,3cyclictest2582-21mii-tool07:41:181
18980991515,0cyclictest0-21swapper/010:26:180
18990991414,0cyclictest0-21swapper/309:35:453
1899099140,14cyclictest0-21swapper/311:01:463
1899099140,14cyclictest0-21swapper/309:05:033
1899099140,0cyclictest0-21swapper/309:00:463
1898699140,0cyclictest0-21swapper/209:36:042
18983991412,2cyclictest0-21swapper/110:51:251
1898399140,14cyclictest0-21swapper/109:00:561
1898399140,14cyclictest0-21swapper/108:30:151
1898399140,14cyclictest0-21swapper/108:16:241
1898399140,0cyclictest0-21swapper/112:26:001
1898399140,0cyclictest0-21swapper/111:15:461
1898399140,0cyclictest0-21swapper/111:07:551
1898099140,14cyclictest0-21swapper/012:10:360
1898099140,14cyclictest0-21swapper/009:14:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional