You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 00:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Nov 26, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16602210982,9sleep30-21swapper/307:09:313
16605210692,9sleep10-21swapper/107:09:321
1640028169,8sleep00-21swapper/007:06:540
1652527867,6sleep20-21swapper/207:08:322
1672899290,2cyclictest0-21swapper/011:06:090
1673599260,3cyclictest0-21swapper/210:28:052
1673799240,1cyclictest0-21swapper/310:06:133
1672899244,2cyclictest141rcu_preempt10:12:280
16735992322,1cyclictest0-21swapper/211:23:052
16731992320,2cyclictest10779-21cstates11:50:161
16737992219,2cyclictest252782sleep310:18:233
16737992219,2cyclictest15068-21cpuspeed_turbos10:10:163
1673599221,2cyclictest10219-21diskmemload10:30:572
16731992219,2cyclictest7632-21grep12:35:161
16731992219,2cyclictest6318-21sort12:10:251
16737992118,2cyclictest5931-21needreboot11:45:203
16737992118,2cyclictest21874-21vmstat10:40:313
16737992118,2cyclictest11770-21pmu-power11:50:233
16731992119,1cyclictest500-21in:imuxsock11:29:261
16731992119,1cyclictest27917-21switchtime10:45:261
1672899210,20cyclictest0-21swapper/012:33:090
16735992018,2cyclictest0-21swapper/210:48:102
1673599200,1cyclictest0-21swapper/209:42:122
16731992018,1cyclictest26203-21cstates12:25:141
1672899201,16cyclictest0-21swapper/011:15:130
1672899200,19cyclictest0-21swapper/010:17:010
1673599192,17cyclictest0-21swapper/212:25:322
16731991917,1cyclictest6535-21tr10:55:141
16731991916,2cyclictest25-21ksoftirqd/108:25:001
16731991916,2cyclictest25134-21missed_timers11:35:201
16731991915,3cyclictest19766-21grep09:45:231
1672899190,1cyclictest0-21swapper/011:31:510
16737991816,1cyclictest472-21avahi-daemon10:26:043
16737991815,2cyclictest25830-21df_abs11:10:163
1673599187,8cyclictest0-21swapper/210:10:182
1673599180,2cyclictest0-21swapper/212:34:292
1673599180,1cyclictest0-21swapper/211:55:142
1673599180,15cyclictest0-21swapper/211:31:512
16731991816,1cyclictest7744-21snmpd09:19:001
16731991815,2cyclictest1074-21needreboot12:30:191
1672899180,16cyclictest0-21swapper/011:20:130
1673799172,2cyclictest0-21swapper/312:29:133
16737991714,2cyclictest26691-21cat10:45:153
1673599172,15cyclictest0-21swapper/212:35:042
1673599170,17cyclictest0-21swapper/211:50:062
16731991715,1cyclictest7994-21idleruntime10:30:191
16731991715,1cyclictest496-21dbus-daemon10:05:221
16731991710,3cyclictest6344-21cat11:45:261
1673799160,2cyclictest0-21swapper/309:46:353
16735991616,0cyclictest0-21swapper/211:48:482
1673599160,16cyclictest0-21swapper/211:35:382
16731991614,1cyclictest472-21avahi-daemon09:22:571
16731991614,1cyclictest220452sleep111:06:371
16731991613,2cyclictest60142sleep109:33:571
16731991613,2cyclictest25-21ksoftirqd/110:00:231
16731991613,2cyclictest12937-21aten2_r7power_e11:00:161
16728991615,1cyclictest0-21swapper/010:06:520
1672899160,15cyclictest0-21swapper/012:39:040
1673799150,8cyclictest224-21systemd-journal10:54:173
1673599153,2cyclictest0-21swapper/211:13:382
1673599153,11cyclictest0-21swapper/210:36:522
16735991514,1cyclictest0-21swapper/211:17:572
16735991513,2cyclictest62050irq/128-enp2s0-10:00:192
1673599150,15cyclictest0-21swapper/211:41:532
1673599150,15cyclictest0-21swapper/209:35:542
1673199155,9cyclictest30788-21apt-get08:45:141
16728991514,1cyclictest0-21swapper/009:39:090
16728991511,3cyclictest5477-21/usr/sbin/munin09:00:190
1672899151,13cyclictest24794-21cut11:35:170
1672899150,14cyclictest0-21swapper/009:23:150
1672899150,14cyclictest0-21swapper/008:00:140
1673599143,6cyclictest1291-21runrttasks11:02:112
16735991413,1cyclictest0-21swapper/209:59:332
1673599140,1cyclictest0-21swapper/210:17:012
1673599140,11cyclictest0-21swapper/207:30:132
16731991411,2cyclictest21291-21awk10:40:251
16731991410,3cyclictest30105-21sed08:40:301
16731991410,3cyclictest18442-21cat07:10:271
1672899147,6cyclictest2858-21munin-run08:55:000
1672899143,3cyclictest224-21systemd-journal07:13:320
16728991413,1cyclictest0-21swapper/011:47:100
1673799139,3cyclictest6712-21grep09:35:003
1673799139,3cyclictest23505-21uname09:20:193
1673799139,3cyclictest12536-21wc11:00:003
1673799139,2cyclictest7292-21grep07:55:163
16737991311,1cyclictest5940-21grep12:10:203
1673799131,11cyclictest32060-21sensors11:40:243
16737991310,2cyclictest5001-21cat11:20:003
16737991310,2cyclictest30307-21grep07:35:173
16737991310,2cyclictest28952-21tail09:25:203
16737991310,2cyclictest25920-21wc07:25:203
16737991310,2cyclictest22089-21head08:25:213
16737991310,2cyclictest16944-21cut11:55:163
16737991310,2cyclictest12930-21tr09:40:223
1673599135,7cyclictest500-21in:imuxsock12:23:472
1673599131,12cyclictest0-21swapper/209:16:442
16735991311,1cyclictest0-21swapper/209:12:382
16735991310,2cyclictest4806-21sed08:55:272
16735991310,2cyclictest31998-21grep08:45:222
16735991310,2cyclictest26703-21cron07:30:012
16735991310,2cyclictest12810-21ssh12:15:222
1673599130,2cyclictest0-21swapper/212:13:332
1673599130,12cyclictest27816-21grep10:20:222
1673199136,5cyclictest10219-21diskmemload09:25:261
16731991310,2cyclictest2759-21sh10:00:001
16731991310,2cyclictest25-21ksoftirqd/112:00:001
16731991310,2cyclictest11151-21grep09:10:171
16731991310,2cyclictest10764-21grep08:00:271
1673199130,2cyclictest0-21swapper/111:31:081
1673199130,11cyclictest1291-21runrttasks09:53:381
1672899139,3cyclictest24270-21cron07:25:000
1672899131,1cyclictest141rcu_preempt09:27:360
1673799129,2cyclictest8819-21sh10:05:003
1673799129,2cyclictest6725-21grep11:20:253
1673799129,2cyclictest28832-21head10:20:483
1673799129,2cyclictest25814-21ls11:35:263
1673799129,2cyclictest25028-21sed11:10:003
1673799129,2cyclictest23928-21wc12:00:173
1673799129,2cyclictest19384-21grep11:30:253
1673799129,2cyclictest16653-21latency_hist09:15:003
1673799129,2cyclictest1002-21ssh12:30:193
1673599129,2cyclictest9328-21tr10:05:132
1673599129,2cyclictest7352-21sort07:55:172
1673599129,2cyclictest4035-21turbostat.cron07:50:012
1673599129,2cyclictest4035-21turbostat.cron07:50:002
1673599129,2cyclictest26903-21uname08:35:192
1673599129,2cyclictest23279-21uname07:20:222
1673599128,4cyclictest29759-21cat09:25:292
1673599128,3cyclictest29504-21grep08:40:222
16735991211,1cyclictest0-21swapper/212:09:442
16735991211,1cyclictest0-21swapper/210:43:532
16735991210,1cyclictest6649-21chrt09:00:202
16735991210,1cyclictest16974-21head08:15:172
1673599120,1cyclictest0-21swapper/210:52:142
1673199129,2cyclictest7806-21tr07:55:201
1673199129,2cyclictest476-21grep10:50:151
1673199129,2cyclictest32591-21ssh10:25:001
1673199129,2cyclictest3216-21grep08:55:141
1673199129,2cyclictest31329-21ssh11:40:191
1673199129,2cyclictest19466-21sh10:40:001
1673199129,2cyclictest13472-21sed09:40:251
1673199125,4cyclictest29978-21/usr/sbin/munin12:05:211
1673199122,8cyclictest7621-21idleruntime-cro09:05:001
1672899129,2cyclictest29122-21grep08:40:190
1672899129,2cyclictest28421-21tr07:30:220
1672899129,2cyclictest28212-21grep10:45:270
1672899129,2cyclictest21195-21grep07:15:290
1672899129,2cyclictest10382-21cat09:10:000
1672899128,3cyclictest10485-21ssh10:32:070
1672899124,2cyclictest224-21systemd-journal08:49:130
1672899122,4cyclictest224-21systemd-journal10:03:480
1672899120,6cyclictest0-21swapper/011:28:500
1672899120,1cyclictest0-21swapper/011:11:330
1672899120,11cyclictest4881-21taskset10:53:420
1672899120,11cyclictest0-21swapper/010:40:150
1673799118,2cyclictest7253-21ssh09:35:123
1673799118,2cyclictest5213-21cron09:00:003
1673799118,2cyclictest4372-21wget07:50:183
1673799118,2cyclictest32178-21ssh09:55:423
1673799118,2cyclictest30505-21expr12:05:173
1673799118,2cyclictest27895-21cron08:40:003
1673799118,2cyclictest25329-21ntpq09:50:213
1673799118,2cyclictest24376-21cat07:25:003
1673799118,2cyclictest23849-21df08:30:163
1673799118,2cyclictest20269-21head07:15:193
1673799118,2cyclictest19774-21sh12:20:173
1673799118,2cyclictest19382-21head08:20:183
1673799118,2cyclictest1778-21cat08:50:193
1673799118,2cyclictest17670-21tail09:15:183
1673799118,2cyclictest15684-21chrt08:11:563
1673799118,2cyclictest15637-21rm12:17:253
1673799118,2cyclictest14840-21ssh11:00:333
1673799118,2cyclictest14507-21cut10:35:173
1673799118,2cyclictest13040-21sh10:34:473
1673799117,2cyclictest27939-21grep07:30:173
1673799111,9cyclictest0-21swapper/312:35:023
1673599119,2cyclictest9859-21taskset10:57:072
1673599119,2cyclictest4304-21dpkg07:50:142
1673599119,2cyclictest27156-21sh09:52:072
1673599119,2cyclictest20687-21seq08:24:472
1673599119,1cyclictest9323-21expr08:00:162
1673599119,1cyclictest26462sleep208:52:372
1673599119,1cyclictest26189-21ssh12:01:362
1673599119,1cyclictest24103-21cat09:20:262
1673599119,1cyclictest23753-21expr08:30:162
1673599119,1cyclictest14551-21grep08:10:162
1673599119,1cyclictest12806-21processes11:25:232
1673599119,1cyclictest12645-21ls08:05:222
1673599118,2cyclictest3176-21grep09:30:352
1673599118,2cyclictest17210-21tr07:10:162
1673599111,10cyclictest0-21swapper/209:46:092
1673199119,1cyclictest26026-21cut08:35:141
1673199118,2cyclictest7806-21rm11:21:111
1673199118,2cyclictest364-21ssh11:15:221
1673199118,2cyclictest2609-21grep07:45:171
1673199118,2cyclictest2609-21grep07:45:171
1673199118,2cyclictest25111-21chrt12:24:421
1673199118,2cyclictest24981-21ssh10:18:011
1673199118,2cyclictest24107-21sed08:30:171
1673199118,2cyclictest23623-21df12:00:151
1673199118,2cyclictest20318-21head07:15:201
1673199118,2cyclictest15526-21sort10:10:181
1673199118,2cyclictest14576-21tail08:10:171
1673199118,2cyclictest1244-21grep07:40:271
1673199118,2cyclictest12354-21grep12:15:191
1673199118,2cyclictest12155-21grep08:05:181
1673199118,2cyclictest11793-21ssh09:39:131
1673199118,2cyclictest10213-21dump-pmu-power09:10:001
1673199113,5cyclictest224-21systemd-journal07:40:001
1673199113,2cyclictest224-21systemd-journal07:33:371
16731991110,1cyclictest0-21swapper/110:29:111
1673199110,10cyclictest27291-21ssh11:10:351
1672899119,1cyclictest41932sleep010:27:130
1672899119,1cyclictest3864-21cut08:55:170
1672899119,1cyclictest26049-21tr08:35:140
1672899119,1cyclictest10670-21grep11:50:160
1672899118,2cyclictest9023-21sh10:56:090
1672899118,2cyclictest6033-21head07:50:590
1672899118,2cyclictest4966-21rm12:10:150
1672899118,2cyclictest31940-21basename07:40:120
1672899118,2cyclictest31901-21ssh11:40:230
1672899118,2cyclictest31713-21cat12:05:280
1672899118,2cyclictest30054-21ssh09:55:130
1672899118,2cyclictest26962-21sh10:20:090
1672899118,2cyclictest23827-21expr08:30:150
1672899118,2cyclictest18751-21ssh11:55:310
1672899118,2cyclictest18719-21ssh12:20:040
1672899118,2cyclictest18612-21apache_volume08:20:140
1672899118,2cyclictest18586-21ssh09:45:160
1672899118,2cyclictest16896-21cat09:15:120
1672899118,2cyclictest166122sleep009:43:010
1672899118,2cyclictest16397-21wc08:15:140
1672899118,2cyclictest14851-21taskset11:00:340
1672899118,2cyclictest11819-21cat08:05:160
1672899118,2cyclictest11418-21sh12:15:100
1672899117,3cyclictest26666-21rm09:51:150
1672899110,4cyclictest0-21swapper/009:10:150
1672899110,10cyclictest30632-21chrt12:28:100
1672899110,10cyclictest13681-21apt10:35:140
1673799108,1cyclictest37892sleep307:47:033
1673799108,1cyclictest37892sleep307:47:033
1673799108,1cyclictest30978-21wc08:45:153
1673799108,1cyclictest28361-21dpkg08:40:023
1673799108,1cyclictest18481-21cat07:10:273
1673799108,1cyclictest12124-21cut08:05:173
1673799107,2cyclictest9213-21tr08:00:153
1673799107,2cyclictest32695-21head07:40:183
1673799107,2cyclictest16473-21tr08:15:143
1673799107,2cyclictest11224-21ssh11:25:063
1673799100,9cyclictest8338-21cut09:05:143
1673799100,9cyclictest7084-21cat09:00:243
1673599109,1cyclictest9254-21ntpq09:05:202
1673599109,1cyclictest15702sleep207:44:022
1673599108,1cyclictest18978-21ssh11:05:022
1673599107,2cyclictest22208-21grep08:25:232
1673599101,6cyclictest31679-21idleruntime-cro07:40:012
1673599100,9cyclictest19763-21cat07:15:182
1673199108,1cyclictest4738-21expr07:50:391
1673199107,2cyclictest797-21tr08:50:141
1673199107,2cyclictest24571-21taskset07:25:121
1673199107,2cyclictest22447-21tr07:20:161
1673199105,3cyclictest20989-21apt-get08:25:131
1673199100,5cyclictest224-21systemd-journal08:20:001
1672899108,1cyclictest6642-21chrt07:55:100
1672899108,1cyclictest25935-21readlink07:25:200
1672899108,1cyclictest2133-21grep09:30:240
1672899107,2cyclictest4164-21sed07:50:000
1672899107,2cyclictest4164-21sed07:50:000
1672899107,2cyclictest29795-21tr07:35:140
1672899107,2cyclictest22880-21chrt08:27:270
1672899102,4cyclictest23038-21apt-get12:00:130
167289980,4cyclictest224-21systemd-journal08:10:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional