You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 12:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Dec 01, 2025 00:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20384210275,9sleep30-21swapper/319:07:213
2038229683,9sleep10-21swapper/119:07:201
1989228264,13sleep20-21swapper/219:05:282
2017928167,9sleep00-21swapper/019:05:290
327622620,0sleep30-21swapper/300:00:003
169182510,0sleep00-21swapper/022:40:350
20680992727,0cyclictest0-21swapper/321:41:133
2068099262,1cyclictest0-21swapper/322:45:333
2067899260,1cyclictest0-21swapper/200:15:102
2067899260,1cyclictest0-21swapper/200:15:102
2067499252,21cyclictest2667-21ssh00:25:131
2068099235,12cyclictest0-21swapper/321:55:163
2068099231,21cyclictest0-21swapper/322:18:293
2067899230,23cyclictest0-21swapper/223:58:482
2068099211,20cyclictest13996-21sh21:31:503
20678992120,1cyclictest0-21swapper/223:50:502
2067899210,20cyclictest25220-21aten2_r7power_p21:40:142
2068099200,19cyclictest0-21swapper/323:44:053
2068099190,19cyclictest0-21swapper/323:24:053
2067899190,19cyclictest0-21swapper/222:26:072
2067499195,13cyclictest13953-21kworker/u8:121:14:381
20680991817,1cyclictest0-21swapper/322:20:093
20680991816,1cyclictest0-21swapper/321:11:293
2068099180,18cyclictest0-21swapper/322:27:103
2068099180,18cyclictest0-21swapper/300:20:573
2068099170,2cyclictest0-21swapper/323:54:393
2067899170,17cyclictest0-21swapper/221:25:152
2067499173,2cyclictest14340-21apt-get21:55:161
2066999175,6cyclictest2238-21munin-run00:25:000
20669991716,1cyclictest0-21swapper/022:50:160
2066999170,17cyclictest0-21swapper/021:47:500
2068099163,12cyclictest11562-21apt-get21:05:163
2068099163,12cyclictest11562-21apt-get21:05:153
20680991614,1cyclictest9431-21users22:35:303
20680991614,1cyclictest9431-21users22:35:293
2067899169,7cyclictest13624-21sh23:01:132
20678991613,2cyclictest28774-21apache_accesses22:05:132
20678991613,2cyclictest0-21swapper/221:19:122
2067499169,6cyclictest7722-21munin-node21:51:581
2067499164,6cyclictest0-21swapper/120:40:151
2066999160,10cyclictest0-21swapper/021:20:130
20680991511,3cyclictest12078-21cat23:00:203
20678991514,1cyclictest0-21swapper/222:31:212
2067899150,15cyclictest0-21swapper/222:00:232
20674991511,0cyclictest0-21swapper/100:15:091
20674991511,0cyclictest0-21swapper/100:15:091
2066999150,12cyclictest0-21swapper/023:13:540
159232150,0chrt0-21swapper/221:33:412
20680991413,1cyclictest0-21swapper/321:36:493
20680991411,2cyclictest25861-21ssh23:32:203
2068099140,14cyclictest0-21swapper/300:11:023
2067899149,3cyclictest13673-21diskmemload23:46:572
2067899142,10cyclictest49-21kcompactd023:15:132
20678991412,2cyclictest32-21ksoftirqd/223:21:312
20678991411,2cyclictest19757-21grep22:20:232
20678991411,2cyclictest17565-21latency_hist22:20:002
20678991411,2cyclictest0-21swapper/221:46:492
20678991410,3cyclictest23064-21grep23:30:202
20678991410,3cyclictest1419-21apache200:32:442
20674991412,1cyclictest251ksoftirqd/120:09:591
20674991411,2cyclictest21549-21sed20:15:271
20669991411,2cyclictest5302-21cat19:45:020
2066999140,13cyclictest4409-21cut22:55:180
2068099139,3cyclictest22041-21grep21:15:183
2068099134,4cyclictest224-21systemd-journal20:40:023
2068099133,7cyclictest18107-21apt-get23:05:153
20680991313,0cyclictest0-21swapper/321:25:293
20680991310,2cyclictest4897-21grep20:50:183
20680991310,2cyclictest3241-21cron22:55:003
20680991310,2cyclictest30238-21sed20:35:193
20680991310,2cyclictest29191-21ssh21:20:153
2068099130,12cyclictest9811-21tail00:30:203
2067899139,3cyclictest19219-21ls21:35:222
2067899138,4cyclictest32-21ksoftirqd/219:36:212
2067899138,2cyclictest13673-21diskmemload23:05:152
2067899133,10cyclictest24358-21kworker/2:000:05:312
20678991311,2cyclictest0-21swapper/221:55:012
20678991310,2cyclictest8899-21grep21:50:282
20678991310,2cyclictest3354-21cut00:25:172
20678991310,2cyclictest31030-21grep22:50:302
20678991310,2cyclictest26491-21ls23:10:232
2067899130,12cyclictest19447-21cron20:14:592
2067499139,3cyclictest28295-21sh22:05:001
2067499139,3cyclictest13169-21grep21:05:281
2067499139,3cyclictest13169-21grep21:05:281
2067499139,3cyclictest11642-21wc19:55:251
2067499139,2cyclictest25832-21grep19:20:171
2067499131,10cyclictest3160-21basename22:10:001
20674991310,2cyclictest30110-21sensors_fan21:20:221
20674991310,2cyclictest15006-21grep00:10:231
2067499130,11cyclictest0-21swapper/121:33:021
2066999139,3cyclictest20024-21cron00:15:010
2066999138,4cyclictest8970-21apt-get00:30:150
2066999131,6cyclictest27285-21munin-run00:20:010
2066999131,6cyclictest27285-21munin-run00:20:000
2066999131,5cyclictest13673-21diskmemload21:50:120
20669991311,2cyclictest16190-21apache223:37:350
20669991310,2cyclictest8346-21tr23:20:220
20669991310,2cyclictest28280-21idleruntime-cro22:05:000
20669991310,2cyclictest27880-21cron21:20:000
20669991310,2cyclictest26081-21tr23:55:150
20669991310,2cyclictest22938-21wc23:30:200
20669991310,2cyclictest20188-21cut20:15:150
20669991310,2cyclictest17806-21wc22:20:000
20669991310,2cyclictest16225-21wc20:05:210
20669991310,2cyclictest13149-21grep20:00:180
20669991310,2cyclictest11699-21grep23:00:190
2066999130,1cyclictest0-21swapper/021:33:020
2068099129,2cyclictest6243-21taskset22:56:013
2068099129,2cyclictest6128-21chrt23:19:333
2068099129,2cyclictest3598-21cat20:45:273
2068099129,2cyclictest3460-21sed00:25:173
2068099129,2cyclictest30190-21awk19:30:013
2068099129,2cyclictest25948-21tr19:20:173
2068099129,2cyclictest22378-21grep00:15:233
2068099129,2cyclictest22378-21grep00:15:233
2068099129,2cyclictest20697-21sed20:15:193
2068099129,2cyclictest15467-21chrt20:05:163
2068099129,2cyclictest11747-21grep19:55:263
2068099128,2cyclictest25617-21ssh23:55:043
2068099125,1cyclictest23502-21ntp_states20:20:233
2068099124,7cyclictest11283-21cron21:05:013
2067899129,2cyclictest871-21grep20:40:232
2067899129,2cyclictest5184-21wc19:45:002
2067899129,2cyclictest29190-21cron20:35:002
2067899129,2cyclictest27342-21sed19:20:292
2067899129,2cyclictest21142-21grep19:10:162
2067899129,2cyclictest21142-21grep19:10:162
2067899129,2cyclictest15459-21ls21:10:232
2067899129,2cyclictest127142sleep222:38:332
2067899129,2cyclictest127142sleep222:38:332
2067899129,2cyclictest12255-21munin-run19:59:592
2067899128,3cyclictest15637-21tr00:35:172
2067899124,4cyclictest32323-21chrt19:32:122
20678991212,0cyclictest0-21swapper/222:40:512
2067899120,1cyclictest0-21swapper/223:41:432
2067499129,2cyclictest9858-21cron19:54:591
2067499129,2cyclictest7495-21grep20:55:181
2067499129,2cyclictest4959-21tail23:40:171
2067499129,2cyclictest3677-21cron23:40:001
2067499129,2cyclictest2575-21df20:45:161
2067499129,2cyclictest20003-21grep23:05:281
2067499129,2cyclictest18141-21awk23:50:001
2067499129,2cyclictest16503-21cat22:40:271
2067499129,2cyclictest13277-21grep20:00:171
2067499128,3cyclictest4410-21sort22:55:181
2067499128,3cyclictest1725-21wc00:00:201
2067499127,5cyclictest95972sleep100:07:091
2067499120,2cyclictest0-21swapper/122:35:071
2067499120,2cyclictest0-21swapper/122:35:071
2067499120,1cyclictest0-21swapper/119:40:151
2066999129,2cyclictest9329-21ssh22:35:290
2066999129,2cyclictest9329-21ssh22:35:280
2066999129,2cyclictest8981-21grep21:00:010
2066999129,2cyclictest5662-21wc20:50:230
2066999129,2cyclictest4943-21grep23:40:180
2066999129,2cyclictest27640-21df20:30:170
2066999129,2cyclictest27361-21sed22:25:280
2066999129,2cyclictest24873-21wc23:10:010
2066999129,2cyclictest14909-21sh21:55:180
2066999129,2cyclictest1077-21grep22:30:190
2066999126,2cyclictest224-21systemd-journal23:19:590
2066999124,4cyclictest224-21systemd-journal19:49:590
2066999121,6cyclictest21184-21perf00:40:000
2068099119,1cyclictest170052sleep323:26:063
2068099118,2cyclictest7221-21chrt22:12:023
2068099118,2cyclictest7184-21chrt19:46:173
2068099118,2cyclictest699-21rm23:37:073
2068099118,2cyclictest6922-21ssh21:50:113
2068099118,2cyclictest6004-21ssh21:49:303
2068099118,2cyclictest28727-21ssh22:05:123
2068099118,2cyclictest26401-21ntpq23:10:213
2068099118,2cyclictest18265-21ssh00:36:373
2068099118,2cyclictest18111-21head20:10:193
2068099118,2cyclictest15691-21ssh23:47:413
2068099118,2cyclictest15119-21cat22:40:153
2068099118,2cyclictest11440-21ssh00:08:543
2068099117,3cyclictest7821-21basename19:50:163
20680991110,1cyclictest0-21swapper/322:02:303
2068099110,10cyclictest29132-21seq20:34:173
2068099110,10cyclictest23675-21expr19:15:173
2067899118,2cyclictest9884-21grep21:00:172
2067899118,2cyclictest8404-21cut19:50:192
2067899118,2cyclictest6076-21ssh22:55:512
2067899118,2cyclictest5922-21ssh22:10:472
2067899118,2cyclictest31015-21ssh23:35:262
2067899118,2cyclictest2979-21ssh00:00:512
2067899118,2cyclictest26289-21sh22:47:422
2067899118,2cyclictest20531-21grep20:15:182
2067899118,2cyclictest17579-21sh00:12:392
2067899116,4cyclictest2987-21cat20:45:202
2067899116,4cyclictest24099-21cat19:15:192
2067899110,10cyclictest23307-21kworker/2:221:23:412
2067499119,2cyclictest25-21ksoftirqd/121:42:111
2067499118,2cyclictest8754-21sh00:30:031
2067499118,2cyclictest6170-21ssh21:26:181
2067499118,2cyclictest32317-21ssh22:51:431
2067499118,2cyclictest32219-21chrt19:30:551
2067499118,2cyclictest28524-21ssh22:26:291
2067499118,2cyclictest28321-21grep19:25:131
2067499118,2cyclictest27501-21wc23:55:261
2067499118,2cyclictest27387-21ssh23:10:451
2067499118,2cyclictest25-21ksoftirqd/121:20:011
2067499118,2cyclictest24735-21cut20:25:151
2067499118,2cyclictest23954-21sed20:20:281
2067499118,2cyclictest22355-21wc19:10:281
2067499118,2cyclictest22355-21wc19:10:271
2067499118,2cyclictest20683-21sh23:50:341
2067499118,2cyclictest1838-21head23:15:271
2067499118,2cyclictest18290-21grep22:20:141
2067499118,2cyclictest18260-21ssh23:27:181
2067499118,2cyclictest17869-21apache_volume21:35:151
2067499118,2cyclictest1758-21sh22:30:231
2067499118,2cyclictest1711-21grep21:45:271
2067499118,2cyclictest14987-21ssh23:02:321
2067499118,2cyclictest12648-21grep22:15:281
2067499117,3cyclictest5517-21grep19:45:131
2067499117,2cyclictest27783-21grep00:20:121
2067499110,10cyclictest20291-21chrt00:39:011
2066999119,1cyclictest30402-21wc22:05:270
2066999119,1cyclictest28663-21cut19:25:150
2066999118,2cyclictest8862-21grep19:50:230
2066999118,2cyclictest827-21grep19:35:150
2066999118,2cyclictest7530-21ssh21:27:330
2066999118,2cyclictest7119-21ssh00:04:580
2066999118,2cyclictest6679-21ssh00:27:530
2066999118,2cyclictest5976-21ssh22:10:490
2066999118,2cyclictest4170-21wc20:50:010
2066999118,2cyclictest32740-21cut19:35:010
2066999118,2cyclictest23352-21ssh23:53:060
2066999118,2cyclictest22789-21grep22:45:180
2066999118,2cyclictest21557-21sort19:10:190
2066999118,2cyclictest21557-21sort19:10:180
2066999118,2cyclictest19960-21ssh21:35:370
2066999118,2cyclictest19005-21cut22:20:160
2066999118,2cyclictest17624-21ssh23:49:310
2066999118,2cyclictest15057-21cut23:25:170
2066999118,2cyclictest14429-21tr21:10:170
2066999118,2cyclictest11839-21cut21:05:180
2066999118,2cyclictest11839-21cut21:05:170
2066999117,2cyclictest27906-21latency_hist19:25:010
2066999115,2cyclictest13-21ksoftirqd/019:59:590
2068099108,1cyclictest324852sleep322:30:003
2068099108,1cyclictest30762-21expr19:30:153
2068099108,1cyclictest3062-21aten2_r7power_e19:40:153
2068099108,1cyclictest21317-21expr19:10:183
2068099108,1cyclictest21317-21expr19:10:173
2068099107,2cyclictest7914-21tr20:55:223
2068099100,9cyclictest12886-21expr20:00:153
2068099100,10cyclictest0-21swapper/319:35:293
2067899108,1cyclictest72152sleep219:46:392
2067899108,1cyclictest6783-21apache_volume20:55:142
2067899108,1cyclictest4747-21expr20:50:162
2067899108,1cyclictest312342sleep220:35:472
2067899108,1cyclictest289482sleep200:20:192
2067899108,1cyclictest15878-21cat20:05:182
2067899107,2cyclictest25340-21grep20:25:192
2067899107,2cyclictest14048-21ssh23:25:002
2067899101,6cyclictest28339-21grep19:25:132
2067899100,4cyclictest0-21swapper/221:05:152
2067899100,4cyclictest0-21swapper/221:05:152
2067499108,1cyclictest6262-21taskset20:51:561
2067499108,1cyclictest4025-21ls22:10:141
2067499108,1cyclictest29992-21sed20:35:191
2067499108,1cyclictest17633-21expr20:10:171
2067499108,1cyclictest11079-21chrt21:02:271
2067499107,2cyclictest7767-21cut23:20:181
2067499107,2cyclictest24037-21grep19:15:181
2067499107,2cyclictest21743-21ssh22:45:011
2067499107,2cyclictest21582-21cat23:30:011
2067499100,9cyclictest270532sleep120:30:051
2066999108,1cyclictest9786-21grep21:00:160
2066999108,1cyclictest253702sleep019:19:440
2066999108,1cyclictest24859-21grep20:25:160
2066999108,1cyclictest191992sleep020:11:540
2066999107,2cyclictest8939-21ssh00:06:300
2066999107,2cyclictest32263-21egrep20:40:160
2066999107,2cyclictest30847-21cut20:35:260
2066999107,2cyclictest25889-21grep21:40:170
206809998,1cyclictest26559-21seq20:26:443
206789997,1cyclictest22374-21wget20:20:152
206789997,1cyclictest12892-21cat20:00:152
206749996,2cyclictest2614-21chrt19:38:481
206699997,1cyclictest22325-21cat20:20:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional