You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 23:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu May 14, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1235521240,0sleep10-21swapper/110:09:581
1662929582,8sleep00-21swapper/007:09:460
1659429279,9sleep20-21swapper/207:09:212
1635829265,9sleep30-21swapper/307:06:173
1662328764,12sleep10-21swapper/107:09:441
76742440,0sleep30-21swapper/309:27:493
307092260,1sleep0141rcu_preempt10:40:110
1673699260,25cyclictest0-21swapper/212:02:372
1673299261,2cyclictest0-21swapper/109:25:181
1673299260,23cyclictest0-21swapper/110:35:331
1673699250,3cyclictest0-21swapper/211:20:152
1673699250,22cyclictest0-21swapper/210:55:102
1673999235,16cyclictest18382-21awk12:10:003
293822220,2sleep229378-21ssh09:40:262
1673999220,21cyclictest7115-21kworker/3:011:31:043
16736992221,1cyclictest0-21swapper/211:03:292
1673699220,1cyclictest0-21swapper/211:12:172
1673699220,1cyclictest0-21swapper/211:12:162
1673999210,19cyclictest7115-21kworker/3:011:27:173
1673699210,20cyclictest0-21swapper/210:42:142
1673299212,19cyclictest23954-21kworker/1:112:01:121
1673299212,18cyclictest12578-21kworker/1:110:10:551
16730992121,0cyclictest0-21swapper/010:02:000
16739992018,2cyclictest0-21swapper/311:36:193
1673699200,20cyclictest0-21swapper/210:32:492
1673699200,19cyclictest0-21swapper/209:51:482
1673299201,19cyclictest20956-21kworker/1:011:34:031
1673299201,17cyclictest0-21swapper/111:20:111
16739991916,1cyclictest0-21swapper/312:23:283
1673699190,2cyclictest0-21swapper/210:06:052
1673699190,1cyclictest0-21swapper/212:15:292
16732991917,1cyclictest15389-21kworker/1:111:17:031
16732991916,2cyclictest25-21ksoftirqd/110:25:001
16732991913,6cyclictest0-21swapper/109:58:391
1673099190,18cyclictest0-21swapper/011:37:350
16732991817,1cyclictest0-21swapper/111:46:121
1673299180,1cyclictest0-21swapper/107:15:121
1673999172,9cyclictest26228-21cron07:30:003
1673699170,17cyclictest0-21swapper/210:16:502
1673299172,1cyclictest9536-21diskmemload10:57:221
1673299170,2cyclictest0-21swapper/112:25:241
1673099173,10cyclictest16939-21apache_processe11:30:110
16730991717,0cyclictest0-21swapper/012:24:400
16730991716,1cyclictest0-21swapper/012:16:400
16730991713,3cyclictest21265-21sh10:34:310
202742160,1chrt0-21swapper/209:15:262
1673999160,1cyclictest0-21swapper/309:52:113
1673999160,16cyclictest0-21swapper/310:43:533
16736991613,2cyclictest32-21ksoftirqd/212:06:172
16736991610,4cyclictest11116-21ssh12:25:012
1673699160,16cyclictest0-21swapper/211:37:532
1673299161,1cyclictest141rcu_preempt10:00:071
1673299161,14cyclictest4822-21latency_hist08:59:591
1673099161,10cyclictest0-21swapper/008:05:130
1673099160,1cyclictest0-21swapper/009:47:240
16739991514,1cyclictest39-21ksoftirqd/312:31:473
16739991511,2cyclictest23185-21sed10:15:253
1673999151,12cyclictest0-21swapper/312:18:453
1673999150,1cyclictest0-21swapper/311:22:213
1673699152,13cyclictest9536-21diskmemload12:14:262
1673699150,4cyclictest0-21swapper/210:46:222
1673699150,15cyclictest0-21swapper/212:32:492
1673699150,15cyclictest0-21swapper/209:31:372
16732991512,2cyclictest0-21swapper/111:12:041
16732991512,2cyclictest0-21swapper/111:12:031
1673299150,1cyclictest0-21swapper/109:34:201
16730991510,3cyclictest20478-21munin-plugin-st08:25:000
1673099150,2cyclictest0-21swapper/011:04:040
1673099150,14cyclictest0-21swapper/011:50:060
1673099150,0cyclictest0-21swapper/009:36:380
1673999149,3cyclictest14913-21sh11:09:333
1673999142,6cyclictest0-21swapper/309:35:233
16739991413,1cyclictest12920-21ssh10:10:133
16739991413,1cyclictest0-21swapper/310:33:523
16739991410,3cyclictest9919-21tr11:45:143
16739991410,2cyclictest4231-21ssh09:45:173
1673699148,5cyclictest25799-21sed11:55:002
1673699144,4cyclictest8127-21cron08:00:012
16736991410,2cyclictest31394-21grep07:40:142
1673699140,12cyclictest0-21swapper/211:07:212
16732991410,3cyclictest18210-21cat08:20:001
16732991410,3cyclictest11547-21head08:05:171
16732991410,2cyclictest19071-21idleruntime-cro07:15:001
1673099149,3cyclictest9399-21head11:25:150
1673099149,3cyclictest18769-21sleep09:35:010
16730991410,3cyclictest25742-21wc11:15:210
1673099140,14cyclictest0-21swapper/012:04:080
1673999139,3cyclictest28775-21ls09:40:193
1673999139,2cyclictest21985-21grep07:20:143
1673999135,3cyclictest7179-21munin-run11:05:013
16739991310,2cyclictest29984-21ls10:00:213
16739991310,2cyclictest24040-21rm11:15:003
16739991310,2cyclictest13641-21wc10:29:593
16739991310,2cyclictest11546-21tail08:05:173
1673999130,2cyclictest0-21swapper/309:56:313
1673999130,2cyclictest0-21swapper/309:32:133
1673999130,12cyclictest0-21swapper/308:20:123
1673999130,11cyclictest23466-21tr10:55:153
1673699139,3cyclictest9252-21wc11:45:012
1673699139,3cyclictest26759-21cut09:20:122
1673699139,3cyclictest21892-21grep08:25:222
1673699139,3cyclictest13511-21grep08:10:122
1673699139,2cyclictest12798-21ssh12:25:172
1673699131,11cyclictest19658-21ssh10:52:292
16736991310,2cyclictest4345-21grep08:55:272
16736991310,2cyclictest27723-21wc11:55:222
16736991310,2cyclictest24842-21head07:25:162
16736991310,2cyclictest10518-21cron08:05:012
1673699130,13cyclictest0-21swapper/212:37:512
1673699130,13cyclictest0-21swapper/212:37:502
1673299139,3cyclictest22848-21grep10:55:001
1673299138,4cyclictest26299-21chrt09:39:251
1673299132,9cyclictest26238-21awk07:30:001
1673299131,10cyclictest0-21swapper/108:20:121
16732991310,2cyclictest946-21ls07:45:011
16732991310,2cyclictest3177-21grep11:40:261
16732991310,2cyclictest29018-21ps09:40:221
16732991310,2cyclictest23433-21grep07:20:251
1673299130,13cyclictest0-21swapper/112:38:531
1673299130,13cyclictest0-21swapper/112:38:531
1673099139,3cyclictest9172-21cat11:45:010
1673099139,3cyclictest19691-21head12:10:160
1673099138,3cyclictest23354-21grep07:20:250
1673099134,8cyclictest5236-21munin-run10:24:590
1673099132,8cyclictest30291-21tr08:45:120
1673099131,6cyclictest8142-21munin-run08:00:010
16730991310,2cyclictest6769-21grep10:45:130
16730991310,2cyclictest27384-21grep09:20:150
16730991310,2cyclictest27258-21grep07:30:160
16730991310,2cyclictest27258-21grep07:30:160
16730991310,2cyclictest26838-21grep11:55:160
16730991310,2cyclictest26653-21chrt10:18:070
16730991310,2cyclictest15058-21head08:10:250
1673999129,2cyclictest2429-21cut12:00:153
1673999129,2cyclictest21032-21grep07:15:263
1673999129,2cyclictest10923-21grep09:10:193
1673999123,4cyclictest225-21systemd-journal07:40:013
1673999122,9cyclictest1142-21runrttasks09:21:073
1673999122,5cyclictest0-21swapper/311:55:133
1673999120,0cyclictest0-21swapper/309:18:593
1673699129,2cyclictest4052-21grep09:25:232
1673699129,2cyclictest32508-21cat08:49:592
1673699129,2cyclictest31083-21tr10:20:222
1673699129,2cyclictest1833-21grep07:45:142
1673699129,2cyclictest18100-21perf08:20:002
1673699128,3cyclictest25752-21cut11:15:212
1673699128,2cyclictest23370-21tr08:30:132
1673699122,3cyclictest28526-21/usr/sbin/munin10:00:132
1673699121,8cyclictest9521-21latency_hist09:10:012
1673699120,2cyclictest0-21swapper/209:36:322
1673299129,2cyclictest7109-21cut07:55:201
1673299129,2cyclictest5292-21tr09:00:161
1673299129,2cyclictest27694-21ssh12:15:141
1673299129,2cyclictest20724-21cut12:30:151
1673299129,2cyclictest14880-21grep10:30:161
1673299129,2cyclictest13684-21ssh11:08:351
1673299127,5cyclictest38150irq/131-snd_hda11:35:531
1673299127,4cyclictest7746-21vmstat10:25:271
1673299126,2cyclictest225-21systemd-journal08:14:591
1673299124,4cyclictest225-21systemd-journal09:10:001
1673299120,12cyclictest0-21swapper/112:23:291
1673299120,11cyclictest0-21swapper/110:47:361
1673299120,0cyclictest0-21swapper/109:54:351
1673099129,2cyclictest7814-21grep11:05:130
1673099129,2cyclictest4565-21ssh10:05:110
1673099129,2cyclictest3896-21apache207:10:220
1673099129,2cyclictest31984-21head07:40:170
1673099129,2cyclictest30501-21sed07:35:240
1673099129,2cyclictest29133-21grep12:35:180
1673099129,2cyclictest29133-21grep12:35:170
1673099129,2cyclictest25739-21grep07:25:250
1673099129,2cyclictest24810-21sed08:30:250
1673099129,2cyclictest23197-21head10:35:170
1673099129,2cyclictest22814-21chrt11:14:130
1673099129,2cyclictest22814-21chrt11:14:130
1673099129,2cyclictest21391-21cut08:25:160
1673099122,4cyclictest0-21swapper/011:47:170
16730991211,1cyclictest32673-21ssh11:20:120
1673099120,2cyclictest0-21swapper/009:05:140
103362120,1sleep2311rcuc/211:45:162
1673999118,2cyclictest8510-21wc10:45:263
1673999118,2cyclictest6802-21ssh10:05:363
1673999118,2cyclictest6515-21sed07:55:143
1673999118,2cyclictest5714-21grep09:00:183
1673999118,2cyclictest31218-21ssh12:36:113
1673999118,2cyclictest31218-21ssh12:36:103
1673999118,2cyclictest30732-21ssh11:18:543
1673999118,2cyclictest28125-21grep07:30:253
1673999118,2cyclictest28125-21grep07:30:243
1673999118,2cyclictest2772-21egrep08:55:143
1673999118,2cyclictest27148-21awk08:35:263
1673999118,2cyclictest2623-21wc07:45:223
1673999118,2cyclictest22464-21aten2_r7power_e10:35:143
1673999118,2cyclictest19790-21ssh11:11:503
1673999118,2cyclictest18768-21ssh12:10:083
1673999118,2cyclictest17853-21chrt11:50:113
1673999118,2cyclictest17849-21chrt08:17:093
1673999118,2cyclictest15284-21ssh10:50:133
1673999118,2cyclictest13436-21ssh12:25:253
1673999118,2cyclictest1251-21ssh10:21:563
1673999114,5cyclictest477-21rs:main3
1673999110,10cyclictest22755-21taskset08:28:583
1673699119,1cyclictest17794-21sed11:30:162
1673699118,2cyclictest8852-21ssh10:26:202
1673699118,2cyclictest8392-21ssh11:25:032
1673699118,2cyclictest8202-21hddtemp_smartct09:05:162
1673699118,2cyclictest6969-21taskset09:02:532
1673699118,2cyclictest4605-21wc07:50:192
1673699118,2cyclictest3472-21ssh09:45:132
1673699118,2cyclictest32629-21apt08:50:122
1673699118,2cyclictest28373-21grep08:40:162
1673699118,2cyclictest26208-21grep08:35:172
1673699118,2cyclictest23460-21ssh09:56:222
1673699118,2cyclictest20278-21uname07:15:192
1673699118,2cyclictest20191-21chrt08:21:282
1673699118,2cyclictest17254-21grep07:10:152
1673699118,2cyclictest13735-21head10:10:172
1673699110,11cyclictest0-21swapper/210:36:152
1673299118,2cyclictest881-21sh09:24:021
1673299118,2cyclictest8414-21apt-get08:00:111
1673299118,2cyclictest757-21uname11:00:221
1673299118,2cyclictest4817-21ssh10:43:541
1673299118,2cyclictest3724-21grep07:50:141
1673299118,2cyclictest30512-21grep08:45:131
1673299118,2cyclictest30099-21tr07:35:201
1673299118,2cyclictest28136-21tr08:40:151
1673299118,2cyclictest28132-21tail11:55:261
1673299118,2cyclictest24321-21ssh10:16:171
1673299118,2cyclictest20809-21ssh09:15:471
1673299118,2cyclictest20661-21ssh11:51:011
1673299118,2cyclictest20339-21ssh12:10:211
1673299118,2cyclictest15044-21ssh09:12:481
1673299117,3cyclictest405-21tr08:50:141
1673299117,1cyclictest0-21swapper/108:30:121
1673299116,4cyclictest21385-21fschecks_count08:25:161
1673099118,2cyclictest9960-21ssh09:10:100
1673099118,2cyclictest6424-21grep10:25:170
1673099118,2cyclictest5086-21ssh09:25:460
1673099118,2cyclictest4237-21cat08:55:270
1673099118,2cyclictest29562-21grep09:40:270
1673099118,2cyclictest2917-21grep07:45:230
1673099118,2cyclictest24488-21ssh09:18:410
1673099118,2cyclictest22307-21ssh12:30:420
1673099118,2cyclictest18078-21sh08:20:000
1673099118,2cyclictest14828-21ssh10:50:030
1673099118,2cyclictest13029-21ssh10:10:130
1673099118,2cyclictest12474-21ssh09:50:140
1673099118,2cyclictest12455-21grep12:25:150
1673099117,3cyclictest29533-21sed08:40:260
1673099117,3cyclictest12389-21grep12:05:250
1673099115,2cyclictest225-21systemd-journal08:05:000
1673099113,4cyclictest225-21systemd-journal08:54:550
1673099110,8cyclictest20421-21aten2_r7power_v09:55:140
1673999108,1cyclictest8389-21cat08:00:103
1673999108,1cyclictest4277-21cut07:50:163
1673999108,1cyclictest30361-21grep08:45:143
1673999108,1cyclictest188142sleep307:11:493
1673999108,1cyclictest15416-21chrt08:14:153
1673999107,2cyclictest8346-21uname09:05:183
1673999107,2cyclictest32719-21basename08:50:143
1673999107,2cyclictest28295-21expr08:40:153
1673999107,2cyclictest23691-21dnf08:30:143
1673699108,1cyclictest30886-21chrt07:39:052
1673699108,1cyclictest21788-21cut07:20:132
1673699107,2cyclictest27097-21cut07:30:162
1673699107,2cyclictest27097-21cut07:30:152
1673699107,2cyclictest12903-21awk08:10:002
1673299108,1cyclictest12721-21ssh12:05:351
1673299107,2cyclictest8615-21ssh11:25:111
1673299107,2cyclictest6408-21ssh09:46:061
1673299107,2cyclictest26781-21tr07:30:141
1673299107,2cyclictest26781-21tr07:30:141
1673299107,2cyclictest1432-21tr07:45:121
1673299102,3cyclictest26549-21sort08:35:191
1673099108,1cyclictest3566-21cat07:50:110
1673099107,2cyclictest23095-21ssh10:55:010
1673099107,2cyclictest20304-21expr07:15:180
1673099102,5cyclictest1142-21runrttasks09:00:170
167399997,1cyclictest32389-21awk07:40:203
167309997,1cyclictest26474-21cat08:35:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional