You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 14:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Nov 16, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30138210276,9sleep30-21swapper/307:05:243
2896029681,11sleep10-21swapper/107:05:091
129252880,0sleep20-21swapper/211:10:142
3095628668,13sleep20-21swapper/207:08:202
3102628266,10sleep00-21swapper/007:09:140
98082570,0sleep10-21swapper/109:49:541
34322570,0sleep20-21swapper/209:18:342
3116499241,3cyclictest25-21ksoftirqd/112:34:011
3116499240,4cyclictest241rcuc/112:10:191
3116399240,2cyclictest0-21swapper/010:49:550
31164992320,3cyclictest25-21ksoftirqd/109:53:591
3116499230,22cyclictest0-21swapper/111:51:311
31170992120,1cyclictest0-21swapper/210:40:152
3116499218,3cyclictest25-21ksoftirqd/112:09:041
3116499210,21cyclictest0-21swapper/112:29:431
3116499210,16cyclictest25-21ksoftirqd/109:25:521
3117099200,20cyclictest0-21swapper/211:55:542
31164992010,9cyclictest24666-21diskmemload11:28:141
3116499200,1cyclictest0-21swapper/111:39:101
3117099180,0cyclictest0-21swapper/212:36:392
3116399180,17cyclictest0-21swapper/012:32:530
3117699170,14cyclictest0-21swapper/311:05:123
3116499174,6cyclictest24666-21diskmemload12:00:561
3116499172,2cyclictest0-21swapper/111:13:481
31164991716,1cyclictest0-21swapper/111:31:081
31164991715,1cyclictest2749-21apt-get08:25:021
3117699160,1cyclictest0-21swapper/311:31:063
31164991614,2cyclictest0-21swapper/111:18:011
31164991612,3cyclictest25-21ksoftirqd/110:35:351
3116499160,1cyclictest141rcu_preempt10:32:541
3116499160,1cyclictest0-21swapper/110:27:171
31176991512,1cyclictest0-21swapper/309:23:253
3117699150,15cyclictest0-21swapper/308:41:393
3117699150,15cyclictest0-21swapper/308:41:393
31170991511,4cyclictest0-21swapper/211:44:052
31170991510,3cyclictest5796-21cut08:30:182
31170991510,2cyclictest37850irq/131-snd_hda12:08:042
3116499154,11cyclictest0-21swapper/111:05:121
31164991514,1cyclictest0-21swapper/110:58:151
31164991512,2cyclictest241rcuc/109:37:401
31164991512,2cyclictest241rcuc/108:45:221
3116499150,1cyclictest0-21swapper/108:40:381
3116499150,1cyclictest0-21swapper/108:40:381
3116499150,14cyclictest0-21swapper/110:52:591
3116399150,2cyclictest0-21swapper/010:34:250
3116399150,13cyclictest0-21swapper/012:28:470
3117699143,10cyclictest0-21swapper/309:10:153
31176991412,1cyclictest0-21swapper/309:16:353
31176991410,3cyclictest30827-21uname09:40:223
3117699140,1cyclictest0-21swapper/309:33:443
3117699140,13cyclictest0-21swapper/310:00:003
3117099142,11cyclictest30430-21sh11:23:402
31170991410,3cyclictest16828-21tr09:55:162
3117099140,2cyclictest141rcu_preempt12:00:092
3116499149,2cyclictest3534-21cron07:20:001
3116499148,5cyclictest3207-21apache211:45:501
3116499147,2cyclictest25-21ksoftirqd/109:41:491
3116499146,4cyclictest1291-21runrttasks10:04:241
3116499146,4cyclictest1291-21runrttasks10:04:241
31164991412,2cyclictest241rcuc/109:15:211
31164991411,2cyclictest19969-21tr11:40:251
31164991410,3cyclictest7190-21grep12:35:311
3116499140,1cyclictest0-21swapper/109:21:251
3116399141,7cyclictest10916-21munin-run09:25:000
31163991410,3cyclictest8690-21awk11:05:280
31163991410,2cyclictest24699-21cat11:45:000
161812140,0chrt0-21swapper/110:45:401
3117699139,3cyclictest685-21sed07:10:263
3117699139,3cyclictest32079-21systemctl11:00:003
3117699139,2cyclictest26470-21grep08:05:203
31176991310,2cyclictest32133-21cat08:15:263
31176991310,2cyclictest28738-21sort08:10:193
31176991310,2cyclictest17095-21cat10:20:273
31176991310,2cyclictest11379-21ssh09:25:163
3117699130,3cyclictest0-21swapper/310:30:173
3117699130,2cyclictest0-21swapper/312:36:113
3117699130,2cyclictest0-21swapper/311:29:113
3117099139,4cyclictest37850irq/131-snd_hda09:51:392
3117099139,3cyclictest12715-21wc07:35:292
3117099139,3cyclictest10043-21ssh12:17:332
31170991310,2cyclictest12534-21sh08:45:002
31170991310,2cyclictest12534-21sh08:45:002
3117099130,13cyclictest0-21swapper/210:55:472
3117099130,12cyclictest13713-21tr11:35:342
3116499139,3cyclictest23288-21cron09:35:001
3116499139,3cyclictest10133-21cron08:40:001
3116499133,9cyclictest15109-21chrt12:20:251
3116499133,9cyclictest0-21swapper/111:20:161
3116499133,10cyclictest0-21swapper/110:07:061
31164991310,2cyclictest5941-21grep08:30:181
31164991310,2cyclictest25603-21wc08:05:001
3116499130,12cyclictest0-21swapper/110:41:211
3116399139,2cyclictest13796-21tr07:40:180
3116399139,2cyclictest12931-21apt-get11:10:140
31163991311,1cyclictest0-21swapper/009:31:510
31163991310,2cyclictest8327-21grep11:55:270
31163991310,2cyclictest4001-21head08:25:190
31163991310,2cyclictest2701-21ssh10:10:000
31163991310,2cyclictest19707-21basename09:00:000
3116399130,12cyclictest0-21swapper/012:35:170
206732130,1sleep30-21swapper/312:04:213
3117699129,2cyclictest549-21sed11:50:183
3117699129,2cyclictest5418-21grep08:30:003
3117699129,2cyclictest2744-21sh08:25:013
3117699129,2cyclictest22684-21cut12:05:183
3117699129,2cyclictest22253-21forks10:25:183
3117699129,2cyclictest21633-21tr12:25:183
3117699129,2cyclictest20233-21head07:50:273
3117699129,2cyclictest16104-21ls09:55:013
3117699129,2cyclictest15861-21grep10:45:273
3117699129,2cyclictest10501-21grep07:30:293
3117699129,2cyclictest10056-21head10:15:203
3117699128,3cyclictest5307-21wc07:20:263
31176991212,0cyclictest0-21swapper/309:47:213
3117099129,2cyclictest9533-21head07:30:192
3117099129,2cyclictest7123-21tail07:25:182
3117099129,2cyclictest667-21grep11:25:212
3117099129,2cyclictest27230-21wc08:05:292
3117099129,2cyclictest24432-21cut09:35:202
3117099129,2cyclictest23037-21systemctl08:00:002
3117099129,2cyclictest19726-21sed07:50:222
3117099129,2cyclictest17337-21grep10:20:282
3117099129,2cyclictest16061-21cut07:45:162
3117099129,2cyclictest1468-21rm10:35:012
3117099128,3cyclictest574-21grep07:10:252
3117099128,3cyclictest31011-21cat09:15:022
3117099127,4cyclictest26409-21chrt10:02:102
3117099127,4cyclictest26409-21chrt10:02:092
3117099125,6cyclictest7744-21snmpd11:30:442
3117099122,9cyclictest2393-21kworker/2:208:25:272
31170991212,0cyclictest6250-21kworker/u8:011:05:282
31170991211,1cyclictest0-21swapper/212:24:202
31170991210,1cyclictest24666-21diskmemload09:41:562
3117099120,9cyclictest28165-21grep08:10:162
3117099120,3cyclictest13530-21ssh10:18:182
3117099120,1cyclictest0-21swapper/210:06:082
3117099120,12cyclictest0-21swapper/209:05:292
3116499129,2cyclictest9532-21tail07:30:191
3116499129,2cyclictest19251-21tail07:50:171
3116499129,2cyclictest16675-21grep07:45:181
3116499129,1cyclictest0-21swapper/111:57:241
3116499128,3cyclictest31973-21wc08:15:221
3116499126,4cyclictest25-21ksoftirqd/110:15:371
3116499126,4cyclictest25-21ksoftirqd/108:51:271
3116499126,4cyclictest25-21ksoftirqd/107:56:171
3116499126,4cyclictest25-21ksoftirqd/107:37:331
3116499126,4cyclictest25-21ksoftirqd/107:22:201
3116499123,9cyclictest0-21swapper/109:11:341
31164991211,1cyclictest0-21swapper/109:56:591
3116499120,1cyclictest0-21swapper/110:23:041
3116399129,2cyclictest8593-21cat07:30:000
3116399129,2cyclictest4590-21head07:20:180
3116399129,2cyclictest32030-21grep07:10:160
3116399129,2cyclictest30088-21df10:05:180
3116399129,2cyclictest27667-21ssh11:20:280
3116399129,2cyclictest27360-21sed08:05:300
3116399129,2cyclictest25333-21taskset10:00:570
3116399129,2cyclictest25333-21taskset10:00:570
3116399129,2cyclictest22827-21tr09:05:190
3116399129,2cyclictest22322-21grep12:05:170
3116399129,2cyclictest21087-21grep11:15:280
3116399129,2cyclictest16903-21cat07:45:200
3116399129,2cyclictest14688-21rm12:20:210
3116399129,2cyclictest12723-21grep09:25:400
3116399128,3cyclictest13-21ksoftirqd/009:16:000
3116399128,3cyclictest0-21swapper/009:45:140
31163991210,2cyclictest13-21ksoftirqd/012:02:180
3116399120,1cyclictest0-21swapper/011:30:150
3117699118,2cyclictest855-21rm12:31:483
3117699118,2cyclictest8285-21grep08:35:173
3117699118,2cyclictest7830-21cat07:25:273
3117699118,2cyclictest6689-21ssh12:15:223
3117699118,2cyclictest4784-21seq11:03:183
3117699118,2cyclictest3939-21rm10:35:273
3117699118,2cyclictest3830-21sed10:10:193
3117699118,2cyclictest30611-21uniq12:10:183
3117699118,2cyclictest29173-21perf09:40:003
3117699118,2cyclictest27607-21ssh10:03:283
3117699118,2cyclictest27607-21ssh10:03:283
3117699118,2cyclictest26066-21tr11:20:173
3117699118,2cyclictest25000-21ssh11:45:133
3117699118,2cyclictest24418-21seq09:07:103
3117699118,2cyclictest23424-21cut08:00:183
3117699118,2cyclictest21689-21head07:55:193
3117699118,2cyclictest21104-21expr10:50:173
3117699118,2cyclictest19616-21rm11:40:223
3117699118,2cyclictest19426-21taskset08:57:083
3117699118,2cyclictest19338-21rm11:15:153
3117699118,2cyclictest16282-21taskset11:37:553
3117699118,2cyclictest16201-21load08:50:203
3117699118,2cyclictest14024-21ls08:45:233
3117699118,2cyclictest13918-21expr07:40:183
3117699118,2cyclictest13656-21tr12:20:143
3117699118,2cyclictest13426-21sh11:59:553
3117699118,2cyclictest13225-21expr11:10:163
3117099119,2cyclictest32-21ksoftirqd/210:10:162
3117099119,1cyclictest21861-21tr10:25:162
3117099118,2cyclictest8575-21ssh09:22:152
3117099118,2cyclictest6714-21ssh09:46:262
3117099118,2cyclictest32673-21expr11:50:172
3117099118,2cyclictest32078-21ssh12:30:482
3117099118,2cyclictest3103-21sed07:15:412
3117099118,2cyclictest25880-21taskset11:45:182
3117099118,2cyclictest2476-21sh11:01:052
3117099118,2cyclictest21601-21aten2_r7power_c12:25:182
3117099118,2cyclictest20919-21tr10:50:172
3117099118,2cyclictest19010-21sh10:48:472
3117099118,2cyclictest18570-21cut09:30:182
3117099118,2cyclictest17594-21tr08:55:142
3117099118,2cyclictest13710-21cat08:45:192
3117099118,2cyclictest12774-21ntp_offset09:25:412
3117099117,3cyclictest14276-21tr07:40:212
3117099113,3cyclictest0-21swapper/209:00:182
3117099112,6cyclictest0-21swapper/210:35:142
3116499118,2cyclictest9735-21ssh12:17:181
3116499118,2cyclictest7360-21cut07:25:201
3116499118,2cyclictest3759-21grep10:10:191
3116499118,2cyclictest29942-21taskset08:12:161
3116499118,2cyclictest19449-21taskset08:57:251
3116499117,3cyclictest32523-21ssh11:00:131
3116499117,2cyclictest24092-21wc09:05:301
3116499116,2cyclictest25-21ksoftirqd/107:43:541
3116499115,4cyclictest25-21ksoftirqd/109:01:251
3116499113,4cyclictest224-21systemd-journal08:25:001
3116399119,1cyclictest9552-21grep10:40:300
3116399118,2cyclictest6755-21grep08:30:240
3116399118,2cyclictest464-21ssh09:42:040
3116399118,2cyclictest350-21wc08:20:000
3116399118,2cyclictest32627-21wget11:00:150
3116399118,2cyclictest32573-21tr11:25:190
3116399118,2cyclictest28711-21sort08:10:200
3116399118,2cyclictest28353-21ssh09:39:050
3116399118,2cyclictest27412-21sh10:29:530
3116399118,2cyclictest27290-21ssh10:55:200
3116399118,2cyclictest25890-21tail09:10:190
3116399118,2cyclictest23751-21cat08:00:180
3116399118,2cyclictest21262-21cat09:00:210
3116399118,2cyclictest1655-21vmstat11:50:290
3116399118,2cyclictest1333-21grep08:20:180
3116399118,2cyclictest1325-21chrt07:15:070
3116399118,2cyclictest13010-21awk12:20:000
3116399118,2cyclictest12269-21cat07:35:230
3116399118,2cyclictest11948-21ssh10:16:240
3116399118,2cyclictest10641-21tr09:50:160
3116399117,3cyclictest4860-21ssh10:36:210
3117699108,1cyclictest9356-21ntpq10:40:233
3117699108,1cyclictest5621-21seq08:30:113
3117699108,1cyclictest20269-21wget09:00:153
3117699108,1cyclictest1761-21expr07:15:243
3117699108,1cyclictest16276-21expr07:45:163
3117699108,1cyclictest12869-21grep07:35:303
3117699107,2cyclictest29364-21sh10:05:013
3117099108,1cyclictest9181-21ntpq08:35:242
3117099108,1cyclictest4454-21grep07:20:172
3117099108,1cyclictest30167-21cat12:10:162
3117099108,1cyclictest19074-21sh11:15:022
3117099108,1cyclictest16086-21cat08:50:192
3117099107,2cyclictest32001-21grep08:15:232
3117099107,2cyclictest23531-21cut08:00:182
3117099101,1cyclictest0-21swapper/208:20:142
3116499101,5cyclictest25-21ksoftirqd/107:10:251
3116399108,1cyclictest8241-21cat08:35:170
3116399108,1cyclictest30310-21expr12:10:160
3116399108,1cyclictest24846-21ssh11:45:060
3116399108,1cyclictest22957-21taskset07:59:270
3116399108,1cyclictest15409-21expr08:50:150
3116399108,1cyclictest15012-21cat08:50:000
3116399108,1cyclictest12945-21cat11:35:290
3116399108,1cyclictest10811-21taskset07:34:020
3116399107,2cyclictest20372-21ssh10:50:010
3116399107,2cyclictest19733-21cut07:50:220
3116399107,2cyclictest16701-21ssh10:20:220
3116399107,2cyclictest16425-21ssh09:55:080
311649994,4cyclictest25-21ksoftirqd/108:05:161
311639997,1cyclictest10780-21expr08:40:340
311639997,1cyclictest10780-21expr08:40:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional