You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-21 - 08:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Jun 21, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
935929878,16sleep00-21swapper/019:08:110
940429363,9sleep30-21swapper/319:08:463
930129351,38sleep20-21swapper/219:07:252
731528364,15sleep10-21swapper/119:05:111
245802440,0sleep10-21swapper/121:22:421
959999270,20cyclictest225-21systemd-journal21:15:012
959499250,2cyclictest0-21swapper/122:18:441
959999240,24cyclictest0-21swapper/223:01:552
959199240,22cyclictest0-21swapper/023:40:000
960199230,1cyclictest0-21swapper/300:21:563
9594992221,1cyclictest0-21swapper/123:33:351
959499220,2cyclictest2327-21diskmemload21:18:541
9601992118,2cyclictest0-21swapper/300:26:183
960199210,1cyclictest0-21swapper/321:26:033
9601992018,1cyclictest7025-21kworker/3:022:23:123
960199200,3cyclictest0-21swapper/300:14:043
9599992020,0cyclictest0-21swapper/221:43:282
959999200,20cyclictest0-21swapper/222:34:392
959499200,3cyclictest2327-21diskmemload22:30:441
9591992019,1cyclictest0-21swapper/022:06:470
960199190,1cyclictest0-21swapper/323:07:473
960199190,18cyclictest0-21swapper/323:56:183
959499190,2cyclictest0-21swapper/121:57:071
959499190,16cyclictest0-21swapper/100:26:181
959199197,7cyclictest22659-21perf20:44:590
9601991817,1cyclictest7025-21kworker/3:022:04:023
960199180,17cyclictest0-21swapper/322:05:143
960199180,14cyclictest0-21swapper/322:30:143
960199180,0cyclictest0-21swapper/300:31:323
959999184,13cyclictest55750irq/128-enp2s0-22:15:192
959999180,3cyclictest0-21swapper/223:36:062
9594991818,0cyclictest0-21swapper/119:25:131
9594991817,1cyclictest0-21swapper/123:02:041
9594991812,6cyclictest38150irq/131-snd_hda22:10:021
960199172,10cyclictest13685-21ssh21:55:143
960199171,15cyclictest0-21swapper/323:42:313
959999175,5cyclictest31160-21apt-config19:55:002
959999170,16cyclictest13940-21kworker/u8:022:53:592
959999170,16cyclictest0-21swapper/223:25:292
959999170,16cyclictest0-21swapper/222:07:182
9594991716,1cyclictest0-21swapper/100:07:411
959499171,14cyclictest0-21swapper/122:00:131
9594991711,3cyclictest225-21systemd-journal21:15:011
959499170,4cyclictest2327-21diskmemload21:52:331
959499170,3cyclictest13195-21latency19:15:191
959499170,3cyclictest0-21swapper/122:43:471
959499170,2cyclictest0-21swapper/122:52:561
959499170,17cyclictest0-21swapper/123:19:351
959199175,6cyclictest4509-21awk21:50:000
959199175,6cyclictest32380-21munin-run21:05:000
960199165,5cyclictest23855-21grep00:15:253
960199162,2cyclictest0-21swapper/323:31:423
960199161,14cyclictest0-21swapper/321:10:153
960199160,2cyclictest0-21swapper/323:20:493
960199160,16cyclictest0-21swapper/322:52:563
959999160,16cyclictest0-21swapper/220:25:242
959999160,15cyclictest0-21swapper/200:28:082
959999160,14cyclictest0-21swapper/222:24:572
9594991614,1cyclictest0-21swapper/100:15:141
959499160,4cyclictest2327-21diskmemload22:26:521
959499160,15cyclictest0-21swapper/119:55:141
959199165,3cyclictest225-21systemd-journal00:30:000
959199163,10cyclictest2327-21diskmemload23:25:190
9591991613,2cyclictest13-21ksoftirqd/021:30:010
9601991512,2cyclictest0-21swapper/322:26:593
959999150,13cyclictest55750irq/128-enp2s0-21:25:142
959499152,1cyclictest2327-21diskmemload21:25:391
9594991514,1cyclictest0-21swapper/100:10:431
959499150,14cyclictest0-21swapper/120:30:131
959199152,3cyclictest2327-21diskmemload23:10:140
9591991511,3cyclictest14060-21grep23:50:270
960199145,6cyclictest2590-21apt-get23:25:143
9601991413,1cyclictest0-21swapper/322:13:393
9601991412,1cyclictest7025-21kworker/3:021:48:323
960199140,7cyclictest0-21swapper/323:49:223
959999145,8cyclictest32-21ksoftirqd/221:35:172
959999144,4cyclictest0-21swapper/200:34:592
959999143,7cyclictest1142-21runrttasks22:58:022
959999142,9cyclictest28462-21fschecks_count23:40:162
9599991412,1cyclictest0-21swapper/222:11:172
959999140,14cyclictest0-21swapper/223:50:462
959499149,5cyclictest38150irq/131-snd_hda20:10:161
959499149,4cyclictest6022-21cat22:10:001
959499144,2cyclictest225-21systemd-journal23:25:011
959499142,2cyclictest0-21swapper/121:33:441
9594991412,1cyclictest25-21ksoftirqd/123:08:021
959499141,13cyclictest0-21swapper/123:41:101
959499140,1cyclictest0-21swapper/100:30:311
959199143,6cyclictest28595-21awk23:59:590
959199140,1cyclictest0-21swapper/022:53:410
960199131,7cyclictest2327-21diskmemload22:15:143
9601991310,2cyclictest970-21sed21:05:163
9601991310,2cyclictest970-21sed21:05:153
9601991310,2cyclictest27265-21cat23:39:593
9601991310,2cyclictest26768-21sed23:00:213
960199130,1cyclictest0-21swapper/321:34:403
960199130,12cyclictest5489-21grep20:05:253
960199130,10cyclictest0-21swapper/300:37:153
959999134,5cyclictest32-21ksoftirqd/223:06:272
959999133,8cyclictest22022-21/usr/sbin/munin22:00:142
9599991312,1cyclictest29974-21ssh23:21:312
9599991310,2cyclictest29373-21tr21:45:152
9599991310,2cyclictest27534-21latency_hist21:25:012
9599991310,2cyclictest23330-21sed19:35:272
9599991310,2cyclictest18874-21grep23:15:182
9599991310,2cyclictest18254-21rm00:12:242
959999130,12cyclictest13299-21ssh21:15:282
959499130,13cyclictest0-21swapper/100:24:301
959199139,3cyclictest6298-21uname00:05:190
959199139,3cyclictest32081-21fgrep00:20:230
959199139,3cyclictest24823-21grep19:40:150
959199139,3cyclictest23755-21sendmail22:39:590
959199135,7cyclictest15501-21munin-run20:30:000
959199135,3cyclictest31452-21apt-get19:55:140
959199134,7cyclictest13-21ksoftirqd/020:55:120
959199133,5cyclictest14874-21/usr/sbin/munin21:55:190
9591991310,2cyclictest5973-21head21:30:260
9591991310,2cyclictest28510-21grep23:20:260
9591991310,2cyclictest19824-21grep20:35:260
959199130,12cyclictest0-21swapper/000:35:140
960199129,2cyclictest944-21cut22:45:153
960199129,2cyclictest8839-21tr20:15:153
960199129,2cyclictest7589-21ssh21:50:443
960199129,2cyclictest7358-21ssh00:05:413
960199129,2cyclictest32497-21sed19:55:203
960199129,2cyclictest26804-21expr19:45:123
960199129,2cyclictest26279-21ssh22:40:283
960199129,2cyclictest23411-21ssh22:58:523
960199129,2cyclictest12268-21grep19:15:143
960199129,2cyclictest10803-21cut23:10:163
960199126,5cyclictest32370-21sh21:05:003
960199122,4cyclictest13571-21pmu-power23:50:223
960199120,11cyclictest21193-21cut20:40:163
959999129,2cyclictest7378-21cut20:10:222
959999129,2cyclictest4550-21grep20:05:162
959999129,2cyclictest29992-21cron00:20:002
959999129,2cyclictest28781-21cat19:49:592
959999129,2cyclictest26217-21cron19:45:012
959999129,2cyclictest21448-21sh19:35:012
959999128,3cyclictest6837-21ssh21:31:062
959999128,3cyclictest23969-21cut00:35:162
959999121,8cyclictest2327-21diskmemload22:44:422
959499129,2cyclictest6468-21ssh22:48:441
959499129,2cyclictest3646-21cat23:25:181
959499129,2cyclictest25771-21ssh22:20:561
959499129,2cyclictest24591-21sed20:45:271
959499129,2cyclictest19252-21ssh23:35:081
959499129,2cyclictest12629-21sed20:20:261
959499129,2cyclictest10544-21grep23:10:141
959499129,1cyclictest30361-21ntpq21:45:211
959499128,3cyclictest28205-21ssh00:37:431
959499128,3cyclictest15917-21ssh23:51:481
959499128,2cyclictest9606-21ssh23:48:251
959499128,1cyclictest26081-21taskset19:43:201
9594991210,1cyclictest4895-21grep20:05:191
959499120,1cyclictest0-21swapper/123:56:181
959499120,11cyclictest22775-21cat20:44:591
959499120,10cyclictest13915-21rt-features21:35:231
959199129,2cyclictest7383-21cut20:10:220
959199129,2cyclictest26087-21grep20:50:160
959199129,2cyclictest20213-21head23:35:160
959199129,2cyclictest1679-21tr20:00:120
959199129,2cyclictest13224-21sed21:15:260
959199129,2cyclictest12848-21sort19:15:160
959199129,2cyclictest11273-21grep23:30:130
959199128,3cyclictest30916-21wc00:00:260
959199125,1cyclictest10789-21perf21:15:010
960199119,1cyclictest30241-21ssh00:00:193
960199119,1cyclictest29470-21grep19:50:173
960199119,1cyclictest26848-21cat20:50:213
960199119,1cyclictest19488-21ssh21:20:053
960199119,1cyclictest18395-21sh23:15:163
960199119,1cyclictest169342sleep319:25:083
960199119,1cyclictest12463-21sed20:20:253
960199118,2cyclictest7876-21sed20:10:263
960199118,2cyclictest25808-21grep19:40:253
960199118,2cyclictest22903-21apache_volume20:45:133
960199118,2cyclictest21797-21cut19:35:143
960199118,2cyclictest21052-21tr21:40:153
960199118,2cyclictest1606-21cpu20:00:133
960199118,2cyclictest15790-21ssh22:35:093
960199118,2cyclictest12421-21cat21:15:203
960199118,2cyclictest10574-21grep19:10:173
960199118,2cyclictest10574-21grep19:10:163
960199111,4cyclictest12343-21/usr/sbin/munin21:35:153
960199110,1cyclictest0-21swapper/320:35:133
960199110,10cyclictest17608-21taskset20:31:493
960199110,10cyclictest15203-21grep19:20:163
959999119,1cyclictest205332sleep220:40:112
959999118,2cyclictest8012-21ssh00:06:112
959999118,2cyclictest5123-21ssh21:50:102
959999118,2cyclictest3567-21ssh00:23:072
959999118,2cyclictest32442-21ssh22:45:042
959999118,2cyclictest32311-21sed22:25:172
959999118,2cyclictest2745-21wc20:00:202
959999118,2cyclictest20330-21apt-config23:55:012
959999118,2cyclictest1608-21python21:05:232
959999118,2cyclictest1608-21python21:05:222
959999118,2cyclictest15961-21ssh21:55:402
959999118,2cyclictest11762-21ssh23:50:002
959999117,3cyclictest30262-21ssh00:00:202
959999117,3cyclictest20449-21wc20:40:002
959999117,3cyclictest17877-21sleep20:35:002
959999117,3cyclictest14829-21tr19:20:132
959999117,3cyclictest11923-21sh19:15:002
959999117,3cyclictest11923-21sh19:14:592
959999113,6cyclictest17189-21ssh22:35:202
959999110,11cyclictest0-21swapper/223:11:432
959999110,10cyclictest18128-21ssh23:34:252
959499119,1cyclictest9870-21hostname19:10:141
959499119,1cyclictest9870-21hostname19:10:141
959499119,1cyclictest915-21tail21:05:161
959499119,1cyclictest915-21tail21:05:151
959499119,1cyclictest29686-21seq20:56:291
959499119,1cyclictest229512sleep121:40:331
959499119,1cyclictest22148-21expr19:35:141
959499119,1cyclictest16931-21ssh22:55:091
959499119,1cyclictest16194-21tr22:35:141
959499118,2cyclictest29943-21tr00:00:171
959499118,2cyclictest29588-21head19:50:171
959499118,2cyclictest26905-21cat20:50:251
959499118,2cyclictest26852-21expr19:45:131
959499118,2cyclictest19236-21grep20:35:201
959499118,2cyclictest1678-21grep20:00:121
959499117,1cyclictest15078-21df_abs19:20:131
959499112,4cyclictest0-21swapper/121:05:001
959499111,1cyclictest9391-21hddtemp_smartct20:15:171
959199118,2cyclictest847-21awk23:05:000
959199118,2cyclictest8381-21head22:10:280
959199118,2cyclictest4927-21cat23:45:190
959199118,2cyclictest31453-21ssh22:25:110
959199118,2cyclictest25054-21http20:50:000
959199118,2cyclictest24888-21ssh00:16:020
959199118,2cyclictest24391-21cut22:40:140
959199118,2cyclictest23088-21cat22:00:190
959199118,2cyclictest22869-21ssh21:40:300
959199118,2cyclictest21299-21ssh22:19:020
959199118,2cyclictest21056-21sed19:30:270
959199118,2cyclictest1974-21ssh23:05:150
959199118,2cyclictest19279-21ssh21:39:150
959199118,2cyclictest19074-21tail23:15:180
959199118,2cyclictest18827-21ssh00:31:560
959199118,2cyclictest18662-21ssh00:12:420
959199118,2cyclictest17859-21head22:55:180
959199118,2cyclictest11922-21cron19:15:000
959199118,2cyclictest11922-21cron19:14:590
959199118,2cyclictest10598-21chrt20:18:290
959199118,2cyclictest10458-21ssh22:31:090
959199117,3cyclictest5565-21tr21:50:160
959199117,2cyclictest32702-21grep22:45:130
959199117,1cyclictest25466-21ssh22:20:420
959199113,6cyclictest0-21swapper/021:20:140
960199108,1cyclictest213192sleep319:33:253
960199107,2cyclictest28612-21sed20:55:173
960199107,2cyclictest14969-21cat20:25:253
959999108,1cyclictest32605-21ntp_states19:55:222
959999108,1cyclictest28230-21expr20:55:142
959999108,1cyclictest12398-21sshd20:20:232
959999108,1cyclictest12296-21hostname19:15:142
959999107,2cyclictest27278-21chrt20:51:012
959999107,2cyclictest18157-21awk19:25:212
959999106,3cyclictest32-21ksoftirqd/220:15:192
959999103,3cyclictest225-21systemd-journal20:50:002
959499108,1cyclictest13781-21expr20:25:151
959499107,2cyclictest19532-21grep19:30:141
959199108,1cyclictest28866-21apt-get19:50:130
959199108,1cyclictest27529-21cat19:45:180
959199108,1cyclictest15057-21expr19:20:140
959199108,1cyclictest130702sleep020:24:310
959199108,1cyclictest0-21swapper/019:25:160
959199107,2cyclictest5023-21tr20:05:210
959199107,2cyclictest21846-21basename19:35:140
959199107,2cyclictest16476-21head20:30:140
959199101,5cyclictest225-21systemd-journal21:10:000
959199101,5cyclictest225-21systemd-journal21:10:000
95999996,2cyclictest30271-21hostname21:00:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional