You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-15 - 17:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Oct 15, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10544210879,10sleep10-21swapper/107:08:451
1045229784,8sleep30-21swapper/307:07:353
1035228169,8sleep00-21swapper/007:06:150
1033428164,13sleep20-21swapper/207:06:012
1072899320,32cyclictest0-21swapper/111:59:081
1072899320,0cyclictest0-21swapper/110:15:481
10733992320,2cyclictest20501-21latency_hist12:05:002
89682200,2chrt8962-21sh11:30:003
10737992019,1cyclictest0-21swapper/310:35:133
1072899200,1cyclictest0-21swapper/110:35:141
1073399193,10cyclictest224-21systemd-journal09:20:002
1073399190,3cyclictest0-21swapper/211:28:282
1072899194,6cyclictest0-21swapper/107:50:171
1073799180,2cyclictest0-21swapper/310:15:143
1073399183,1cyclictest3737-21diskmemload11:02:372
1073399180,17cyclictest0-21swapper/212:36:482
1072399186,6cyclictest18082-21munin-run07:25:000
10733991716,0cyclictest0-21swapper/210:18:352
10728991716,1cyclictest0-21swapper/112:06:211
1073399167,4cyclictest493-21systemd-logind09:01:392
1072899169,6cyclictest7706-21apache209:11:041
10728991611,3cyclictest9524-21wc09:40:321
1073799157,4cyclictest11580-21wc09:15:293
10737991511,1cyclictest25665-21apt-get09:55:143
10733991514,1cyclictest0-21swapper/212:29:192
10733991513,1cyclictest0-21swapper/208:40:252
10733991512,2cyclictest0-21swapper/211:23:042
1072899151,1cyclictest0-21swapper/108:25:171
1072399153,9cyclictest7730-21cups-browsed08:40:000
10723991514,1cyclictest0-21swapper/011:57:580
10737991411,2cyclictest10735-21fgrep11:30:243
10737991411,2cyclictest10735-21fgrep11:30:243
10737991410,2cyclictest18963-21grep07:25:173
1073799140,1cyclictest0-21swapper/310:44:313
1073399145,8cyclictest31414-21runrttasks09:32:312
10733991412,1cyclictest0-21swapper/210:35:132
10733991411,2cyclictest0-21swapper/208:35:142
10733991410,3cyclictest4272-21head08:00:282
10733991410,3cyclictest27423-21grep09:55:262
1072899148,4cyclictest21032-21grep09:50:251
1072899144,6cyclictest7720-21apache211:47:391
10728991414,0cyclictest0-21swapper/112:38:161
10728991414,0cyclictest0-21swapper/110:41:001
10728991410,3cyclictest3285-21cat12:15:211
10728991410,3cyclictest23022-21turbostat11:15:001
10728991410,2cyclictest4331-21wc10:30:281
10728991410,2cyclictest1686-21grep09:05:111
1072899140,1cyclictest0-21swapper/109:35:421
1072399146,2cyclictest19283-21awk08:35:000
10723991411,2cyclictest25372-21awk10:50:000
10723991410,3cyclictest22751-21cut09:25:180
1072399140,12cyclictest23341-21cat08:40:260
1073799139,3cyclictest18289-21cat12:25:313
1073799139,2cyclictest31930-21tr09:00:193
1073799133,2cyclictest224-21systemd-journal11:50:003
1073799132,2cyclictest224-21systemd-journal12:39:413
10737991310,2cyclictest5068-21cat11:00:003
10737991310,2cyclictest4194-21grep09:10:143
10737991310,2cyclictest30292-21grep11:20:183
10737991310,2cyclictest30083-21head09:30:293
10737991310,2cyclictest24901-21cut08:45:183
10737991310,2cyclictest23369-21cat08:40:263
10737991310,2cyclictest2303-21cut12:15:153
10737991310,2cyclictest14701-21tr07:15:213
10737991310,2cyclictest14118-21grep08:20:283
10737991310,2cyclictest12114-21latency_hist08:20:003
10737991310,2cyclictest10416-21cat12:20:263
1073399139,3cyclictest24553-21wc11:15:212
1073399137,5cyclictest5880-21grep08:05:162
1073399137,5cyclictest32-21ksoftirqd/211:35:172
1073399133,6cyclictest0-21swapper/211:05:222
10733991312,1cyclictest0-21swapper/207:20:152
10733991310,2cyclictest30869-21sort07:50:202
10733991310,2cyclictest20790-21cut07:30:182
10733991310,2cyclictest17763-21grep09:20:262
10733991310,2cyclictest17321-21cat10:15:002
1073399130,13cyclictest0-21swapper/211:52:282
1073399130,12cyclictest3737-21diskmemload12:06:342
1072899139,3cyclictest25772-21grep07:40:151
1072899139,3cyclictest25772-21grep07:40:151
1072899139,3cyclictest11072-21sed09:15:221
1072899135,4cyclictest11470-21cron10:10:001
1072899132,10cyclictest6866-21ntp_states11:00:221
10728991312,1cyclictest0-21swapper/109:48:461
10728991310,2cyclictest31353-21tail11:20:281
10728991310,2cyclictest30530-21wc10:25:211
10728991310,2cyclictest29536-21sed09:30:241
10728991310,2cyclictest24311-21grep07:35:211
10728991310,2cyclictest22064-21cut09:25:141
10728991310,2cyclictest17440-21grep08:30:141
1072399139,2cyclictest15513-21grep08:25:210
1072399136,7cyclictest114962sleep010:36:450
1072399135,2cyclictest224-21systemd-journal07:55:000
1072399134,7cyclictest25657-21apache_volume09:55:130
1072399134,6cyclictest16569-21apt-get12:25:190
1072399131,7cyclictest0-21swapper/010:40:110
1072399131,12cyclictest0-21swapper/010:55:160
10723991310,2cyclictest31118-21cat10:55:000
10723991310,2cyclictest28462-21grep07:45:150
10723991310,2cyclictest1911-21sed07:55:270
10723991310,2cyclictest19086-21grep07:25:190
10723991310,2cyclictest16308-21ssh09:20:150
10723991310,2cyclictest15668-21cron07:20:000
10723991310,2cyclictest12623-21grep07:10:370
10723991310,2cyclictest12103-21cron08:20:000
10723991310,2cyclictest11072-21grep11:30:280
10723991310,2cyclictest11072-21grep11:30:270
1073799129,2cyclictest9558-21sed11:55:273
1073799129,2cyclictest7903-21grep09:40:143
1073799129,2cyclictest6847-21grep10:05:173
1073799129,2cyclictest32392-21grep10:00:163
1073799129,2cyclictest22351-21ssh11:40:233
1073799129,2cyclictest16846-21sed11:35:283
1073799129,2cyclictest16828-21tail09:20:173
1073799129,2cyclictest14469-21tr12:00:173
1073799128,2cyclictest31198-21wc10:55:003
1073799128,2cyclictest12437-21wc11:05:203
1073799126,5cyclictest27034-21rm12:30:583
1073799124,4cyclictest17262-21munin-run10:15:003
1073799120,4cyclictest224-21systemd-journal09:10:003
1073399129,2cyclictest6693-21cut10:05:172
1073399129,2cyclictest25884-21chrt07:40:152
1073399129,2cyclictest25884-21chrt07:40:152
1073399129,2cyclictest25346-21cron10:50:002
1073399129,2cyclictest19259-21sh08:35:002
1073399129,2cyclictest15495-21grep09:45:252
1073399129,2cyclictest10108-21gltestperf11:30:182
1073399129,2cyclictest10108-21gltestperf11:30:182
1073399129,2cyclictest0-21swapper/209:10:172
1073399128,3cyclictest1756-21tr09:05:112
10733991210,2cyclictest0-21swapper/209:25:262
10733991210,1cyclictest30064-21cut10:25:182
1073399120,10cyclictest0-21swapper/211:55:162
1073399120,0cyclictest0-21swapper/209:42:472
1072899129,2cyclictest8325-21tail08:10:201
1072899129,2cyclictest319-21cut10:00:181
1072899129,2cyclictest29252-21fgrep12:10:221
1072899129,2cyclictest26932-21chrt10:50:231
1072899129,2cyclictest26326-21tail12:30:261
1072899129,2cyclictest2392-21cat08:00:001
1072899129,2cyclictest23610-21head08:40:271
1072899129,2cyclictest18139-21cat07:25:001
1072899129,2cyclictest17725-21ssh11:36:331
1072899129,2cyclictest13934-21ssh12:00:021
1072899129,2cyclictest13028-21sort08:20:181
1072899128,2cyclictest19821-21wc07:25:271
1072899128,2cyclictest10242-21grep08:15:151
1072899127,3cyclictest26726-21wc11:45:011
1072899121,9cyclictest22755-21munin-node11:52:251
10728991212,0cyclictest0-21swapper/109:24:551
1072899120,4cyclictest224-21systemd-journal10:15:001
1072399129,2cyclictest9233-21grep08:10:290
1072399129,2cyclictest9029-21grep12:20:150
1072399129,2cyclictest870-21grep10:00:210
1072399129,2cyclictest7305-21perf08:10:000
1072399129,2cyclictest3808-21grep08:00:230
1072399129,2cyclictest324-21cat11:50:000
1072399129,2cyclictest30101-21ssh09:30:290
1072399129,2cyclictest29313-21ssh12:10:230
1072399129,2cyclictest27645-21grep08:50:300
1072399129,2cyclictest25805-21head12:30:220
1072399129,2cyclictest25331-21wc07:40:010
1072399129,2cyclictest23843-21grep10:20:210
1072399129,2cyclictest2384-21wc10:30:000
1072399129,2cyclictest21859-21tr07:30:250
1072399129,2cyclictest14019-21grep08:20:280
1072399129,2cyclictest12679-21grep10:10:190
1072399129,2cyclictest10525-21grep09:15:190
1072399128,2cyclictest17637-21grep10:15:170
1073799119,1cyclictest19164-21date11:10:283
1073799118,2cyclictest6004-21ssh11:53:543
1073799118,2cyclictest29495-21cut10:25:163
1073799118,2cyclictest28621-21sed12:10:173
1073799118,2cyclictest28328-21ls08:50:383
1073799118,2cyclictest27065-21chrt11:17:263
1073799118,2cyclictest26923-21taskset10:22:173
1073799118,2cyclictest23837-21sed09:25:283
1073799118,2cyclictest21639-21ls08:40:003
1073799118,2cyclictest1983-21tr09:35:163
1073799118,2cyclictest17844-21tune2fs08:30:163
1073799118,2cyclictest14431-21grep09:45:153
1073799115,2cyclictest224-21systemd-journal07:10:383
1073799114,2cyclictest224-21systemd-journal08:07:423
1073799110,10cyclictest3376-21head08:00:183
1073799110,10cyclictest29846-21diskstats08:55:173
1073399118,2cyclictest5624-21ssh12:17:062
1073399118,2cyclictest32386-21grep10:00:162
1073399118,2cyclictest31032-21chrt08:55:442
1073399118,2cyclictest30550-21ssh10:54:062
1073399118,2cyclictest29325-21ssh12:10:232
1073399118,2cyclictest28893-21head11:45:272
1073399118,2cyclictest2590-21basename10:30:122
1073399118,2cyclictest25295-21grep10:20:322
1073399118,2cyclictest24195-21grep07:35:202
1073399118,2cyclictest23990-21ssh09:53:292
1073399118,2cyclictest21547-21sh11:13:152
1073399118,2cyclictest1994-21grep09:35:162
1073399118,2cyclictest13944-21expr07:15:162
1073399118,2cyclictest10999-21grep07:10:142
1073399118,2cyclictest10777-21sed08:15:172
1073399118,2cyclictest10300-21grep12:20:232
1073399118,1cyclictest19398-21sh10:44:062
1073399117,3cyclictest25456-21uname12:30:192
1073399110,1cyclictest0-21swapper/211:40:432
1072899118,2cyclictest5455-21tr08:05:141
1072899118,2cyclictest31992-21ssh10:55:181
1072899118,2cyclictest29638-21chrt07:45:511
1072899118,2cyclictest26630-21cat09:55:191
1072899118,2cyclictest25876-21wc08:45:281
1072899118,2cyclictest23966-21ssh11:15:171
1072899118,2cyclictest23531-21rm10:20:161
1072899118,2cyclictest21751-21ntpq10:45:211
1072899118,2cyclictest16320-21ssh12:25:081
1072899118,2cyclictest15342-21ssh12:24:311
1072899118,2cyclictest12670-21head07:10:371
1072899118,2cyclictest11348-21ssh11:05:111
1072899117,3cyclictest2953-21basename11:25:141
1072899117,2cyclictest28225-21find08:50:351
1072899116,3cyclictest9430-21grep11:30:151
1072899116,3cyclictest9430-21grep11:30:151
1072399118,2cyclictest9741-21ssh09:40:350
1072399118,2cyclictest8165-21ssh10:05:450
1072399118,2cyclictest6719-21ssh12:18:370
1072399118,2cyclictest6375-21chrt09:38:570
1072399118,2cyclictest5304-21rm11:00:000
1072399118,2cyclictest4385-21grep09:10:150
1072399118,2cyclictest3298-21ssh11:51:070
1072399118,2cyclictest31937-21cut09:00:190
1072399118,2cyclictest30967-21sed08:55:280
1072399118,2cyclictest30233-21grep11:20:180
1072399118,2cyclictest24636-21chrt08:45:170
1072399118,2cyclictest24103-21ssh11:41:570
1072399118,2cyclictest2326-21head09:05:150
1072399118,2cyclictest2216-21ssh12:35:390
1072399118,2cyclictest21606-21grep12:05:170
1072399118,2cyclictest20639-21head09:50:220
1072399118,2cyclictest19935-21ssh11:11:130
1072399118,2cyclictest16199-21ssh12:00:360
1072399118,2cyclictest16183-21rm09:46:130
1072399118,2cyclictest12885-21sensors11:05:250
1072399115,1cyclictest224-21systemd-journal11:40:000
1072399110,1cyclictest0-21swapper/011:15:270
1073799108,1cyclictest8327-21expr08:10:203
1073799108,1cyclictest6099-21expr11:00:153
1073799108,1cyclictest557-21expr07:55:153
1073799108,1cyclictest32057-21taskset07:51:273
1073799108,1cyclictest27866-21apt07:45:113
1073799108,1cyclictest25491-21apt-get07:40:123
1073799108,1cyclictest25491-21apt-get07:40:123
1073799108,1cyclictest24902sleep310:30:063
1073799108,1cyclictest21557-21cat07:30:213
1073799108,1cyclictest178702sleep307:22:373
1073799108,1cyclictest16596-21taskset08:26:263
1073799107,2cyclictest23574-21sh07:35:153
1073799107,2cyclictest20982-21expr10:45:153
1073799107,2cyclictest20339-21expr09:50:203
1073799100,6cyclictest0-21swapper/312:05:033
1073399108,1cyclictest910-21head07:55:172
1073399108,1cyclictest18545-21grep07:25:152
1073399108,1cyclictest16668-21taskset08:27:222
1073399108,1cyclictest14644-21date08:25:002
1073399107,2cyclictest7862-21grep08:10:172
1073399107,2cyclictest31485-21sh10:55:152
1073399107,2cyclictest268492sleep208:50:152
1073399107,2cyclictest25450-21ntpq08:45:222
1072899108,1cyclictest3926-21tr08:00:241
1072899108,1cyclictest31128-21chrt08:56:591
1072899108,1cyclictest154692sleep107:17:341
1072899107,2cyclictest31960-21tr09:00:191
1072899107,2cyclictest20967-21tr07:30:181
1072399108,1cyclictest27228-21vmstat07:40:280
1072399108,1cyclictest27228-21vmstat07:40:270
1072399107,2cyclictest4321-21cat10:30:280
1072399107,2cyclictest3180-21tr11:25:150
107339996,2cyclictest28145-21grep07:45:132
107289996,2cyclictest19564-21tr08:35:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional