You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 13:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Dec 06, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
508929977,16sleep10-21swapper/119:05:181
618229884,9sleep30-21swapper/319:07:213
604728167,9sleep00-21swapper/019:05:370
425927867,6sleep20-21swapper/219:05:112
647499270,1cyclictest14705-21cat00:30:182
194322260,0sleep00-21swapper/022:31:060
647499231,3cyclictest0-21swapper/221:25:162
646599230,1cyclictest0-21swapper/021:13:460
647499220,22cyclictest0-21swapper/222:19:592
647499220,1cyclictest0-21swapper/221:50:142
647199220,1cyclictest0-21swapper/100:10:181
6478992119,1cyclictest3327-21/usr/sbin/munin21:35:223
6474992119,1cyclictest32-21ksoftirqd/200:37:362
6465992119,2cyclictest8622-21kworker/u8:222:24:230
6478992017,2cyclictest25979-21/usr/sbin/munin23:25:243
647499204,5cyclictest32164-21/usr/sbin/munin22:40:272
6474992017,3cyclictest37850irq/131-snd_hda21:35:562
6471992017,2cyclictest5081-21/usr/sbin/munin22:45:131
6465992019,1cyclictest13-21ksoftirqd/000:36:340
6465992016,3cyclictest13-21ksoftirqd/021:45:560
646599200,1cyclictest0-21swapper/021:28:030
177132200,0sleep10-21swapper/123:42:271
647199198,9cyclictest10672-21ssh21:40:111
6465991917,1cyclictest13-21ksoftirqd/021:38:460
646599190,18cyclictest0-21swapper/022:46:310
647499180,18cyclictest31532-21diskmemload23:33:062
647499180,13cyclictest0-21swapper/222:45:122
646599182,1cyclictest141rcu_preempt22:26:030
6465991816,2cyclictest13-21ksoftirqd/023:05:010
646599180,1cyclictest141rcu_preempt00:06:460
647899170,14cyclictest6711-21apt-get19:10:143
646599175,7cyclictest26053-21ssh23:25:170
646599171,0cyclictest141rcu_preempt00:20:380
646599170,14cyclictest0-21swapper/000:14:340
647499169,3cyclictest291irq_work/200:18:022
647499162,7cyclictest0-21swapper/221:45:172
6474991615,1cyclictest0-21swapper/223:00:222
6474991615,1cyclictest0-21swapper/221:10:132
647499160,3cyclictest0-21swapper/219:25:122
647499160,1cyclictest0-21swapper/223:14:392
647499160,16cyclictest0-21swapper/200:11:102
647499160,15cyclictest0-21swapper/223:54:222
647199161,14cyclictest19428-21ssh23:20:161
647199160,15cyclictest0-21swapper/100:21:271
646599162,13cyclictest12788-21apt-get23:15:160
6465991614,1cyclictest13-21ksoftirqd/021:15:130
6465991613,2cyclictest13-21ksoftirqd/023:55:190
646599160,1cyclictest0-21swapper/023:46:090
647899156,4cyclictest224-21systemd-journal00:09:583
6478991510,3cyclictest22768-21ls19:40:253
647499159,5cyclictest27215-21tail23:25:232
647499152,1cyclictest224-21systemd-journal23:25:002
6474991512,3cyclictest0-21swapper/222:23:122
6474991511,3cyclictest13795-21apt-config19:25:002
647499150,3cyclictest31532-21diskmemload21:42:412
647499150,2cyclictest0-21swapper/222:37:332
6471991513,1cyclictest31929-21/usr/sbin/munin21:10:201
646599153,7cyclictest19286-21df22:55:170
646599152,10cyclictest15368-21/usr/sbin/munin00:30:230
646599150,3cyclictest0-21swapper/019:25:120
646599150,1cyclictest141rcu_preempt22:06:120
646599150,1cyclictest0-21swapper/021:54:490
647899143,2cyclictest224-21systemd-journal00:11:113
6478991410,3cyclictest25278-21tune2fs20:55:173
6478991410,3cyclictest19143-21grep20:40:273
6478991410,2cyclictest7014-21grep22:45:263
6478991410,2cyclictest6975-21sort22:00:003
647899140,1cyclictest0-21swapper/323:19:513
6474991413,1cyclictest26834-21smartctl19:50:182
6474991413,1cyclictest26834-21smartctl19:50:172
6474991411,3cyclictest18944-21kworker/u8:121:24:472
647499140,5cyclictest32-21ksoftirqd/222:25:192
647199143,4cyclictest28498-21ssh21:30:131
6471991410,3cyclictest10885-21sed00:25:311
646599149,1cyclictest13-21ksoftirqd/019:15:010
647899139,3cyclictest32017-21grep23:05:153
647899139,3cyclictest32017-21grep23:05:143
647899139,3cyclictest24450-21cron00:40:003
647899139,3cyclictest15081-21sh00:05:003
647899139,3cyclictest1502-21head20:05:153
647899137,2cyclictest400-21grep23:30:163
6478991310,2cyclictest7013-21sed23:10:213
6478991310,2cyclictest321-21ssh21:10:173
6478991310,2cyclictest26095-21head21:50:163
6478991310,2cyclictest25049-21wc23:00:003
6478991310,2cyclictest19075-21ssh21:45:203
6478991310,2cyclictest13377-21wc22:50:223
647899130,12cyclictest29113-21tr00:15:153
647899130,12cyclictest11959-21grep20:25:303
647499139,3cyclictest7581-21cat21:15:202
647499135,8cyclictest0-21swapper/223:06:252
647499135,8cyclictest0-21swapper/223:06:252
6474991312,1cyclictest3848-21kworker/u8:020:02:242
647499131,1cyclictest31532-21diskmemload21:59:142
6474991311,2cyclictest311rcuc/200:10:002
6474991311,1cyclictest32-21ksoftirqd/220:20:002
647499130,8cyclictest0-21swapper/223:40:142
647499130,3cyclictest0-21swapper/222:00:252
647499130,0cyclictest0-21swapper/222:12:372
647199139,3cyclictest9179-21grep22:00:261
647199139,3cyclictest8684-21head20:20:171
647199139,3cyclictest32463-21grep20:00:271
647199139,3cyclictest29952-21grep22:15:201
647199139,3cyclictest27299-21cron23:49:591
647199135,4cyclictest224-21systemd-journal19:15:381
647199132,10cyclictest1794-21sensors23:30:231
6471991310,2cyclictest4660-21cron22:45:011
6471991310,2cyclictest4308-21grep22:20:211
6471991310,2cyclictest31684-21/usr/sbin/munin23:05:221
6471991310,2cyclictest31684-21/usr/sbin/munin23:05:221
6471991310,2cyclictest26336-21cat21:50:181
6471991310,2cyclictest22231-21cat21:25:241
6471991310,2cyclictest18633-21cat19:35:001
647199130,12cyclictest27654-21cat23:25:291
647199130,12cyclictest18294-21basename22:55:011
646599137,5cyclictest0-21swapper/000:26:130
646599131,11cyclictest31900-21cut20:00:210
646599130,7cyclictest0-21swapper/022:00:170
646599130,7cyclictest0-21swapper/000:02:570
646599130,5cyclictest0-21swapper/023:14:400
647899129,2cyclictest3304-21df00:20:143
647899129,2cyclictest30221-21cat21:30:263
647899129,2cyclictest21393-21wc20:45:233
647899129,2cyclictest20771-21ssh23:45:143
647899128,3cyclictest8096-21cut23:35:203
647899128,3cyclictest27094-21wc19:50:203
647899128,3cyclictest27094-21wc19:50:193
647899128,2cyclictest15333-21wc23:40:233
647899120,1cyclictest0-21swapper/322:06:593
647499129,3cyclictest183572sleep219:32:282
647499129,2cyclictest6415-21ssh23:57:572
647499129,2cyclictest13443-21cut20:30:192
647499129,1cyclictest22809-21ssh23:45:312
647499128,4cyclictest188702sleep222:55:152
647499128,3cyclictest6668-21taskset19:10:062
647499124,4cyclictest500-21in:imuxsock00:27:412
6474991211,1cyclictest32-21ksoftirqd/222:34:252
647199129,2cyclictest7530-21sort19:10:181
647199129,2cyclictest7009-21head23:10:201
647199129,2cyclictest5555-21tr20:15:141
647199129,2cyclictest3688-21tr21:35:151
647199129,2cyclictest31025-21grep21:05:271
647199129,2cyclictest29026-21cut00:15:151
647199129,2cyclictest27796-21grep21:00:191
647199129,2cyclictest25766-21sh19:50:001
647199129,2cyclictest22192-21tr22:10:171
647199129,2cyclictest19061-21tr22:55:161
647199129,2cyclictest18711-21wc20:40:241
647199129,2cyclictest12421-21apt20:29:591
647199128,3cyclictest14508-21cat23:15:281
6471991210,1cyclictest12945-21grep20:30:181
647199120,8cyclictest0-21swapper/119:50:131
647199120,8cyclictest0-21swapper/119:50:131
647199120,1cyclictest0-21swapper/100:30:151
647199120,11cyclictest20544-21head20:45:141
646599129,2cyclictest9575-21grep20:20:270
646599129,2cyclictest3435-21tr20:10:190
646599129,2cyclictest29198-21apache_volume22:15:150
646599129,2cyclictest24573-21ssh23:23:560
646599129,2cyclictest19616-21head19:35:170
646599129,2cyclictest18979-21ssh21:23:350
646599129,2cyclictest18187-21head20:40:180
646599129,2cyclictest11144-21expr21:40:140
646599128,4cyclictest161rcuc/019:55:150
646599128,3cyclictest27631-21date19:50:270
646599128,3cyclictest27631-21date19:50:260
646599126,6cyclictest31546-21kworker/u8:123:52:140
646599123,6cyclictest29855-21uname00:15:210
646599122,7cyclictest31684-21/usr/sbin/munin23:05:210
646599122,7cyclictest31684-21/usr/sbin/munin23:05:200
646599121,2cyclictest181irq_work/021:31:560
6465991210,2cyclictest13-21ksoftirqd/021:58:390
646599120,9cyclictest0-21swapper/022:13:050
646599120,1cyclictest141rcu_preempt23:30:040
646599120,11cyclictest24333-21grep22:35:170
646599120,11cyclictest16539-21ssh22:53:110
647899118,2cyclictest8056-21basename20:20:143
647899118,2cyclictest6404-21ssh23:57:563
647899118,2cyclictest5217-21taskset20:14:423
647899118,2cyclictest3817-21grep22:20:183
647899118,2cyclictest31600-21sh20:00:183
647899118,2cyclictest3151-21ssh22:43:303
647899118,2cyclictest30528-21grep22:15:253
647899118,2cyclictest30156-21ssh23:50:533
647899118,2cyclictest29902-21df21:05:163
647899118,2cyclictest26523-21grep23:00:193
647899118,2cyclictest23857-21expr22:35:153
647899118,2cyclictest20770-21ssh21:25:033
647899118,2cyclictest19835-21ssh21:24:213
647899118,2cyclictest17048-21cut22:30:153
647899118,2cyclictest16982-21taskset20:37:463
647899118,2cyclictest16787-21ssh00:31:363
647899118,2cyclictest15747-21grep19:25:263
647899118,2cyclictest15019-21sh21:42:413
647899118,2cyclictest12479-21ssh00:27:473
647899118,2cyclictest10164-21ssh22:01:143
647899118,2cyclictest10090-21ssh22:25:003
647899117,3cyclictest28902-21df19:55:143
647899117,3cyclictest23894-21grep19:45:163
647899116,4cyclictest13824-21ntp_offset20:30:233
647899110,10cyclictest28558-21ssh22:14:543
647499119,2cyclictest21196-21acpi19:40:112
647499119,2cyclictest12388-21expr22:50:162
647499119,1cyclictest9510-21expr19:15:142
647499119,1cyclictest3524-21expr20:10:182
647499119,1cyclictest30608-21sed21:05:222
647499119,1cyclictest26482sleep220:07:142
647499119,1cyclictest19901-21aten2_r7power_p20:45:112
647499119,1cyclictest19168-21expr19:35:142
647499119,1cyclictest17509-21aten2_r7power_e20:40:152
647499119,1cyclictest10428-21aten2_r7power_v20:25:162
647499119,1cyclictest10077-21ntpq00:00:222
647499119,1cyclictest100202sleep220:24:572
647499118,2cyclictest7201-21ssh23:35:112
647499118,2cyclictest3657-21wc00:20:162
647499118,2cyclictest28293-21wc21:00:232
647499118,2cyclictest25384-21sed20:55:182
647499113,2cyclictest141rcu_preempt22:05:342
647499110,1cyclictest141rcu_preempt19:50:002
647499110,11cyclictest12783-21ssh23:15:142
647199118,2cyclictest7479-21ssh23:35:181
647199118,2cyclictest4242-21cat20:10:231
647199118,2cyclictest3836-21cat23:55:281
647199118,2cyclictest3010-21seq21:56:191
647199118,2cyclictest28574-21head23:50:161
647199118,2cyclictest25893-21wc20:55:221
647199118,2cyclictest24361-21cron20:55:001
647199118,2cyclictest23439-21ssh22:35:071
647199118,2cyclictest19839-21ssh22:31:351
647199118,2cyclictest19548-21grep19:35:171
647199118,2cyclictest18458-21ssh21:45:151
647199118,2cyclictest15795-21grep00:05:151
647199118,2cyclictest15753-21cut22:05:201
647199118,2cyclictest15310-21grep19:25:201
647199118,2cyclictest11479-21rm21:18:191
647199118,2cyclictest11195-21head22:25:241
647199110,10cyclictest26084-21ssh23:00:171
647199110,10cyclictest23048-21sshd19:40:421
646599119,1cyclictest760-21seq20:05:110
646599119,1cyclictest264792sleep020:56:400
646599119,1cyclictest254032sleep019:45:280
646599119,1cyclictest230722sleep019:41:000
646599119,1cyclictest22933-21cut20:50:180
646599119,1cyclictest21756-21taskset20:47:040
646599119,1cyclictest14150-21cut23:40:150
646599119,1cyclictest13284-21grep20:30:200
646599119,1cyclictest12969-21wc19:20:260
646599118,2cyclictest7645-21expr23:35:180
646599111,6cyclictest30607-21apt-get22:40:170
646599110,10cyclictest7532-21taskset20:18:470
646599110,10cyclictest16463-21cut19:30:140
647899108,1cyclictest11017-21taskset19:17:483
647899107,2cyclictest7293-21taskset20:15:463
647899107,2cyclictest6635-21grep21:15:153
647899107,2cyclictest22056-21ssh23:21:143
647899107,2cyclictest18067-21grep19:30:263
647899107,2cyclictest12029-21chrt19:20:153
647499109,1cyclictest28419-21sh21:30:112
647499108,2cyclictest15514-21ls20:35:142
647499108,1cyclictest30513-21cron20:00:002
647499108,1cyclictest22672-21expr20:50:162
647199108,1cyclictest9635-21ssh00:00:181
647199108,1cyclictest29343-21cut19:55:181
647199108,1cyclictest20433-21basename00:35:211
647199108,1cyclictest13653-21ssh21:20:091
647199107,2cyclictest2563-21seq20:06:081
647199107,2cyclictest15543-21df20:35:151
647199107,2cyclictest11685-21grep19:20:141
646599109,1cyclictest16603-21sed20:35:250
646599108,2cyclictest27847-21wc21:00:200
646599108,1cyclictest9795-21tail19:15:160
646599108,1cyclictest11044-21chrt20:25:190
646599107,2cyclictest30446-21sort21:05:200
64789997,1cyclictest27970-21sed21:00:203
64789996,2cyclictest22947-21grep20:50:183
64789996,2cyclictest19093-21grep19:35:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional