You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 18:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Dec 07, 2025 12:44:01)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22465210779,9sleep30-21swapper/307:06:053
2273229986,9sleep10-21swapper/107:09:331
2276129163,6sleep00-21swapper/007:09:520
2246427868,6sleep20-21swapper/207:06:042
2286199230,3cyclictest141rcu_preempt10:52:461
2286399220,2cyclictest0-21swapper/207:55:152
2286699210,1cyclictest0-21swapper/310:30:333
2286699210,1cyclictest0-21swapper/310:30:333
2286399211,19cyclictest0-21swapper/209:44:442
2286399204,15cyclictest15378-21gltestperf12:05:182
22863991817,1cyclictest0-21swapper/210:29:372
22863991816,2cyclictest30895-21kworker/2:209:48:492
22863991816,2cyclictest12237-21kworker/2:110:03:492
2286399181,16cyclictest0-21swapper/211:24:462
2286399180,1cyclictest0-21swapper/212:25:512
2286199187,8cyclictest15490-21diskmemload12:30:131
2286199180,1cyclictest0-21swapper/110:20:151
2286199180,1cyclictest0-21swapper/110:12:301
2285699183,9cyclictest21238-21idleruntime-cro12:09:590
2285699180,2cyclictest141rcu_preempt12:32:580
22863991716,1cyclictest20760-21kworker/2:011:55:092
22863991716,1cyclictest0-21swapper/208:05:142
2286399171,16cyclictest5316-21kworker/2:207:10:312
2286399167,9cyclictest30896-21kworker/2:008:54:112
2286399166,10cyclictest30896-21kworker/2:008:49:182
2286399164,12cyclictest30896-21kworker/2:009:01:272
22863991611,3cyclictest0-21swapper/211:10:012
22863991610,5cyclictest218022sleep210:30:242
22863991610,5cyclictest218022sleep210:30:242
22861991613,2cyclictest25-21ksoftirqd/111:20:001
2285699163,6cyclictest26248-21basename09:45:000
2285699160,12cyclictest0-21swapper/009:20:470
2286699155,7cyclictest24320-21apt-get08:20:183
2286699151,1cyclictest141rcu_preempt12:18:373
2286399154,11cyclictest16813-21kworker/2:208:14:392
22863991514,1cyclictest20760-21kworker/2:011:49:292
22863991510,5cyclictest10432-21kworker/2:010:55:312
22863991510,3cyclictest37850irq/131-snd_hda11:29:202
2286399150,1cyclictest0-21swapper/207:45:132
22861991515,0cyclictest0-21swapper/112:22:391
22861991510,3cyclictest24453-21sleep11:24:591
22866991410,3cyclictest18295-21cut11:20:003
2286699140,12cyclictest25693-21sed11:25:193
2286399148,6cyclictest17485-21kworker/2:211:04:312
22863991413,1cyclictest0-21swapper/210:45:412
22863991410,3cyclictest27465-21head08:25:212
22863991410,3cyclictest1873-21tail07:30:312
2286399140,1cyclictest0-21swapper/210:07:472
2286399140,1cyclictest0-21swapper/209:55:152
2286399140,1cyclictest0-21swapper/208:55:242
2286399140,13cyclictest0-21swapper/212:37:092
2286199143,4cyclictest15160-21df_abs09:35:161
2286199143,4cyclictest0-21swapper/111:50:001
22861991411,2cyclictest24792-21grep07:10:271
2286199141,11cyclictest10841-21runrttasks12:25:131
22861991410,2cyclictest20016-21sort10:55:221
2285699144,1cyclictest10997-21apt-get09:00:140
2286699139,3cyclictest8616-21grep09:55:243
2286699139,3cyclictest20346-21grep11:20:283
2286699139,3cyclictest1053-21grep10:15:133
2286699139,2cyclictest12404-21grep07:55:163
22866991312,1cyclictest0-21swapper/312:34:213
22866991311,1cyclictest0-21swapper/311:59:093
22866991311,1cyclictest0-21swapper/311:05:173
22866991310,2cyclictest8703-21tr08:55:143
22866991310,2cyclictest7020-21grep11:35:353
22866991310,2cyclictest24322-21grep07:10:223
22866991310,2cyclictest14219-21wc10:50:303
2286399139,3cyclictest31509-21cut08:35:152
2286399139,3cyclictest31509-21cut08:35:142
2286399138,4cyclictest24552-21sh09:15:582
2286399133,6cyclictest7493-21apt-get12:00:162
2286399133,6cyclictest7493-21apt-get12:00:162
2286399132,0cyclictest0-21swapper/212:24:172
2286399131,8cyclictest0-21swapper/207:50:132
2286399131,1cyclictest3620-21ssh09:51:272
2286399131,1cyclictest29767-21taskset10:36:512
22863991311,2cyclictest32-21ksoftirqd/212:30:192
22863991310,2cyclictest7491-21cat10:20:002
22863991310,2cyclictest29920-21tr09:20:232
22863991310,2cyclictest27759-21grep10:10:172
22863991310,2cyclictest21692-21basename12:10:122
22863991310,2cyclictest11987-21sed11:40:152
2286199139,3cyclictest14882-21grep12:05:151
2286199139,2cyclictest28032-21tr09:45:241
2286199138,3cyclictest28743-21grep09:20:151
2286199131,8cyclictest7648-21sshd11:36:131
22861991310,2cyclictest28304-21cat12:15:011
22861991310,2cyclictest24027-21latency_hist08:20:001
22861991310,2cyclictest21668-21sed10:30:231
22861991310,2cyclictest21668-21sed10:30:231
22861991310,2cyclictest18836-21sed08:05:261
2286199130,8cyclictest4746-21latency_hist11:35:001
2286199130,12cyclictest0-21swapper/110:25:171
2285699139,3cyclictest24578-21cron10:59:590
2285699139,2cyclictest9510-21cron07:50:000
2285699136,3cyclictest224-21systemd-journal12:29:480
2285699135,4cyclictest0-21swapper/012:15:270
2285699132,3cyclictest0-21swapper/009:35:160
2285699131,3cyclictest224-21systemd-journal09:25:130
22856991310,2cyclictest28417-21grep10:35:280
22856991310,2cyclictest23519-21sed08:15:270
22856991310,2cyclictest23376-21chrt09:15:200
22856991310,2cyclictest15928-21ssh10:25:310
22856991310,2cyclictest15284-21wc10:00:260
22856991310,2cyclictest14692-21wc09:06:070
2286699129,2cyclictest8627-21sort12:00:213
2286699129,2cyclictest8627-21sort12:00:213
2286699129,2cyclictest6410-21grep08:50:163
2286699129,2cyclictest5885-21cron08:50:003
2286699129,2cyclictest5719-21grep07:40:183
2286699129,2cyclictest32251-21uname11:30:223
2286699129,2cyclictest25915-21cut11:50:173
2286699129,2cyclictest2302-21cron07:35:003
2286699129,2cyclictest19633-21cut08:10:153
2286699129,2cyclictest1528-21basename08:40:163
2286699129,2cyclictest12782-21ssh11:40:213
2286699129,2cyclictest10296-21tail09:30:293
2286699128,3cyclictest6186-21cron10:44:593
2286699128,3cyclictest28010-21cron09:20:003
2286699128,3cyclictest27042-21grep10:10:143
2286699128,3cyclictest20796-21sed10:55:293
2286699128,2cyclictest27756-21wc07:20:013
2286699124,3cyclictest15490-21diskmemload11:45:213
2286699120,2cyclictest0-21swapper/312:25:023
2286699120,11cyclictest8503-21cat07:45:193
2286399129,2cyclictest6193-21grep07:40:222
2286399129,2cyclictest3623-21cat08:45:002
2286399129,2cyclictest28577-21grep07:20:182
2286399128,1cyclictest16671-21taskset08:02:282
2286399127,4cyclictest14118-21ssh10:50:302
2286399124,4cyclictest224-21systemd-journal07:20:012
2286399122,3cyclictest0-21swapper/209:28:352
2286199129,2cyclictest30587-21idleruntime-cro11:30:001
2286199129,2cyclictest2160-21ssh10:40:291
2286199129,2cyclictest21300-21head10:05:211
2286199129,2cyclictest15595-21cat09:10:011
2286199129,2cyclictest13845-21ssh12:04:441
2286199129,2cyclictest13845-21ssh12:04:431
2286199128,3cyclictest4997-21basename07:40:141
2286199128,3cyclictest21823-21cat09:15:011
2286199128,2cyclictest19563-21grep08:10:141
2286199127,3cyclictest2648-21grep07:35:141
2286199123,1cyclictest0-21swapper/109:19:101
2285699129,2cyclictest6727-21seq11:35:330
2285699129,2cyclictest3938-21grep12:20:170
2285699129,2cyclictest27486-21grep12:35:170
2285699129,2cyclictest21465-21sh10:30:210
2285699129,2cyclictest21465-21sh10:30:210
2285699129,2cyclictest2011-21sort08:40:180
2285699129,2cyclictest15993-21grep08:00:220
2285699129,2cyclictest14564-21wc10:50:340
2285699129,2cyclictest13899-21awk11:15:280
2285699128,3cyclictest4190-21taskset11:57:070
2285699128,2cyclictest5911-21perf08:50:000
2285699128,2cyclictest25794-21grep11:00:180
2285699127,3cyclictest12113-21cut11:40:150
2285699126,6cyclictest0-21swapper/011:45:330
2285699124,4cyclictest224-21systemd-journal09:54:180
2285699120,2cyclictest0-21swapper/008:20:180
286382110,1sleep30-21swapper/311:02:233
2286699119,1cyclictest255922sleep310:09:013
2286699119,1cyclictest15490-21diskmemload10:35:143
2286699118,2cyclictest9446-21ssh10:46:313
2286699118,2cyclictest7798-21taskset11:10:533
2286699118,2cyclictest3460-21grep12:20:143
2286699118,2cyclictest3319-21ssh09:51:083
2286699118,2cyclictest29537-21ssh09:46:343
2286699118,2cyclictest27411-21grep08:25:213
2286699118,2cyclictest23856-21ssh12:10:383
2286699118,2cyclictest23561-21grep08:15:273
2286699118,2cyclictest21277-21tr09:40:133
2286699118,2cyclictest17467-21ssh12:06:193
2286699118,2cyclictest17185-21ssh09:36:103
2286699118,2cyclictest16162-21ssh10:01:023
2286699118,2cyclictest15133-21sed09:06:123
2286699118,2cyclictest12584-21ssh10:23:463
2286699117,3cyclictest16254-21cat09:10:153
2286699110,10cyclictest3640-21cut09:25:243
2286699110,0cyclictest0-21swapper/309:20:163
2286399118,2cyclictest9134-21ssh09:30:192
2286399118,2cyclictest6992-21grep11:35:352
2286399118,2cyclictest6207-21munin-run10:45:002
2286399118,2cyclictest31245-21ssh12:16:172
2286399118,2cyclictest2204-21sh11:07:012
2286399118,2cyclictest19866-21sh09:12:502
2286399118,2cyclictest1608-21ssh11:31:412
2286399118,2cyclictest14645-21sshd09:35:142
2286399118,2cyclictest13686-21tr09:06:002
2286399118,2cyclictest12567-21grep11:15:172
2286399118,1cyclictest11162-21sh10:22:122
2286399113,7cyclictest26413-21cron08:25:002
2286199118,2cyclictest8133-21taskset08:52:391
2286199118,2cyclictest8106-21sed09:55:191
2286199118,2cyclictest8085-21sort07:45:161
2286199118,2cyclictest7926-21ssh11:11:031
2286199118,2cyclictest777-21grep11:05:281
2286199118,2cyclictest6663-21ssh10:19:241
2286199118,2cyclictest5070-21ssh11:58:071
2286199118,2cyclictest32163-21grep08:35:181
2286199118,2cyclictest32163-21grep08:35:181
2286199118,2cyclictest31939-21grep07:25:261
2286199118,2cyclictest30569-21ssh12:37:041
2286199118,2cyclictest30241-21ssh10:37:241
2286199118,2cyclictest29018-21ssh11:51:531
2286199118,2cyclictest27130-21sed07:15:281
2286199118,2cyclictest26606-21wc08:25:001
2286199118,2cyclictest21189-21cat09:40:131
2286199118,2cyclictest1815-21sed07:30:311
2286199118,2cyclictest16256-21expr08:00:261
2286199118,2cyclictest1607-21sed09:50:181
2286199118,2cyclictest15427-21cat10:00:271
2286199118,2cyclictest14526-21ssh11:41:341
2286199118,2cyclictest12437-21cut07:55:161
2286199118,2cyclictest11939-21ssh09:32:171
2286199117,3cyclictest25897-21ssh11:00:181
2286199115,2cyclictest224-21systemd-journal08:35:001
2286199110,4cyclictest0-21swapper/108:45:131
2286199110,10cyclictest1618-21ssh12:19:071
2286199110,10cyclictest10578-21chrt08:58:341
2285699118,2cyclictest5850-21ssh10:18:370
2285699118,2cyclictest5493-21grep11:10:150
2285699118,2cyclictest5044-21cut07:40:140
2285699118,2cyclictest4184-21sed07:35:260
2285699118,2cyclictest3666-21ssh10:42:080
2285699118,2cyclictest31865-21grep07:25:260
2285699118,2cyclictest30490-21ssh09:47:370
2285699118,2cyclictest29245-21ssh11:28:260
2285699118,2cyclictest28603-21ssh10:10:250
2285699118,2cyclictest25497-21ssh11:50:050
2285699118,2cyclictest22283-21ssh12:10:150
2285699118,2cyclictest21623-21ssh09:14:530
2285699118,2cyclictest20668-21sshd10:05:180
2285699118,2cyclictest19724-21wc11:20:210
2285699118,2cyclictest17226-21grep08:05:140
2285699118,2cyclictest13277-21cat10:00:000
2285699118,2cyclictest1324-21ssh11:31:210
2285699118,2cyclictest10555-21grep07:50:200
2285699118,2cyclictest10536-21ssh09:30:390
2285699118,2cyclictest10485-21ssh12:01:330
2285699118,2cyclictest10485-21ssh12:01:330
2285699118,2cyclictest10386-21ssh10:21:040
2286699108,1cyclictest9754-21basename07:50:103
2286699108,1cyclictest422-21cat08:35:423
2286699108,1cyclictest422-21cat08:35:423
2286699108,1cyclictest31122-21if_enp2s007:25:183
2286699108,1cyclictest2535-21taskset07:35:033
2286699108,1cyclictest13919-21seq10:25:013
2286699108,1cyclictest10945-21taskset09:00:033
2286699107,2cyclictest29089-21cut08:30:133
2286699107,2cyclictest28730-21if_enp0s31f607:20:193
2286699107,2cyclictest26891-21grep12:35:153
2286699107,2cyclictest18100-21cut08:05:193
2286699106,3cyclictest15316-21df_abs08:00:143
2286399108,1cyclictest3320-21gltestperf07:35:172
2286399107,2cyclictest31885-21grep07:25:262
2286399107,2cyclictest29468-21ls08:30:152
2286399107,2cyclictest223842sleep208:15:162
2286399104,3cyclictest15490-21diskmemload11:51:182
2286199108,1cyclictest6542-21ssh10:45:101
2286199108,1cyclictest296642sleep107:21:061
2286199108,1cyclictest115702sleep107:51:031
2286199107,2cyclictest2918-21grep09:25:181
2286199101,3cyclictest0-21swapper/109:00:141
2285699108,1cyclictest522-21expr07:30:200
2285699108,1cyclictest31777-21expr11:05:160
2285699108,1cyclictest29261-21taskset07:20:250
2285699108,1cyclictest27281-21chrt07:16:160
2285699108,1cyclictest20788-21grep08:10:230
2285699108,1cyclictest121182sleep007:55:020
2285699108,1cyclictest1001-21taskset08:38:460
2285699108,1cyclictest1001-21taskset08:38:460
2285699107,2cyclictest29385-21expr08:30:150
2285699107,2cyclictest28036-21basename08:25:260
2285699100,9cyclictest9567-21ssh10:46:480
2285699100,9cyclictest23235-21grep07:10:150
2285699100,6cyclictest224-21systemd-journal09:00:000
228619997,1cyclictest2992-21cat08:40:271
228619997,1cyclictest26838-21egrep08:25:181
228569997,1cyclictest6289-21aten2_r7power_v08:50:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional