You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 09:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Aug 27, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2554129482,8sleep00-21swapper/019:07:010
2567529077,9sleep20-21swapper/219:08:482
2546728666,16sleep30-21swapper/319:06:033
2564428265,13sleep10-21swapper/119:08:231
252862670,0sleep20-21swapper/223:09:462
70642590,0sleep00-21swapper/023:55:560
239252530,0sleep20-21swapper/223:27:442
25866992925,3cyclictest0-21swapper/222:56:232
2586699260,24cyclictest0-21swapper/221:51:412
2586699260,24cyclictest0-21swapper/221:32:172
2586699250,23cyclictest0-21swapper/222:30:442
25866992221,1cyclictest0-21swapper/221:19:302
2586699220,21cyclictest0-21swapper/222:42:232
2586699220,19cyclictest0-21swapper/222:07:432
2586299220,1cyclictest0-21swapper/121:27:211
2586699210,20cyclictest0-21swapper/200:09:302
2586299210,1cyclictest0-21swapper/122:10:071
2586099212,1cyclictest141rcu_preempt00:29:540
2586899200,19cyclictest0-21swapper/322:53:213
2586699200,1cyclictest0-21swapper/222:50:342
2586699200,19cyclictest0-21swapper/223:01:342
189142200,0sleep20-21swapper/222:45:452
2586899191,5cyclictest141rcu_preempt00:37:203
2586899190,4cyclictest0-21swapper/320:45:123
25866991918,1cyclictest0-21swapper/221:55:382
25866991918,1cyclictest0-21swapper/221:55:382
2586699190,18cyclictest0-21swapper/222:23:332
2586699190,18cyclictest0-21swapper/222:01:472
2586699190,18cyclictest0-21swapper/222:01:462
2586699190,17cyclictest0-21swapper/223:32:432
2586299194,1cyclictest18807-21diskmemload22:44:061
2586299190,1cyclictest0-21swapper/122:25:191
2586899180,18cyclictest0-21swapper/300:24:553
2586699180,3cyclictest0-21swapper/200:13:152
2586699180,18cyclictest0-21swapper/223:41:322
2586299180,18cyclictest0-21swapper/123:48:351
2586299180,17cyclictest0-21swapper/123:52:481
2586899170,13cyclictest0-21swapper/300:30:113
25866991715,1cyclictest0-21swapper/200:00:132
2586699170,17cyclictest0-21swapper/221:47:532
2586699170,16cyclictest0-21swapper/222:27:302
2586699170,0cyclictest0-21swapper/223:46:282
2586699161,7cyclictest0-21swapper/200:30:102
25866991614,1cyclictest0-21swapper/223:24:372
25866991613,2cyclictest5932-21ssh00:15:112
2586699160,16cyclictest0-21swapper/221:44:462
2586299164,2cyclictest141rcu_preempt00:00:341
2586299163,9cyclictest225-21systemd-journal19:59:591
2586299160,15cyclictest0-21swapper/119:10:201
122242160,0sleep00-21swapper/000:18:480
2586699150,2cyclictest0-21swapper/223:57:562
2586299159,5cyclictest38150irq/131-snd_hda21:15:191
25862991514,1cyclictest0-21swapper/121:20:141
2586299151,13cyclictest22220-21munin-run22:10:001
25862991511,2cyclictest10452-21wc23:00:191
2586299150,0cyclictest0-21swapper/121:55:361
2586299150,0cyclictest0-21swapper/121:55:351
2586299150,0cyclictest0-21swapper/100:13:021
2586099154,7cyclictest537-21snmpd23:24:030
2586099150,1cyclictest0-21swapper/023:07:220
2586099150,13cyclictest0-21swapper/022:06:140
25868991410,3cyclictest12151-21grep20:55:163
2586899140,1cyclictest0-21swapper/323:19:433
2586899140,0cyclictest0-21swapper/321:49:293
2586699148,4cyclictest32-21ksoftirqd/200:35:152
25866991412,2cyclictest32-21ksoftirqd/221:35:072
2586699141,1cyclictest10057-21ssh21:23:542
25866991411,2cyclictest30263-21wc23:50:262
25866991411,2cyclictest17784-21cat19:59:592
25866991410,3cyclictest20007-21cron20:05:002
25866991410,2cyclictest25121-21seq20:15:132
2586699140,1cyclictest0-21swapper/223:39:092
2586299148,5cyclictest20551-21grep20:05:161
25862991413,1cyclictest0-21swapper/100:40:001
25862991412,2cyclictest38150irq/131-snd_hda21:44:401
25862991411,2cyclictest4791-21ssh23:35:211
25862991410,3cyclictest24700-21grep22:30:171
25862991410,3cyclictest1090-21processes20:30:211
2586299140,7cyclictest0-21swapper/120:13:411
2586299140,7cyclictest0-21swapper/120:13:411
2586299140,1cyclictest0-21swapper/122:58:511
2586099148,4cyclictest0-21swapper/023:30:150
2586099147,6cyclictest18805-21munin-run21:10:000
25860991412,2cyclictest0-21swapper/000:30:560
25860991411,2cyclictest0-21swapper/021:51:400
2586899139,3cyclictest6272-21tr19:35:143
2586899139,3cyclictest28818-21grep19:15:133
25868991310,2cyclictest28650-21tr23:30:213
25868991310,2cyclictest18358-21cron23:25:003
25868991310,2cyclictest11436-21cut19:45:173
2586699139,3cyclictest27974-21sed23:10:252
2586699139,3cyclictest11708-21cp20:55:002
2586699138,3cyclictest28941-21tr19:15:142
25866991310,2cyclictest9059-21cron20:50:002
25866991310,2cyclictest2728-21wc19:25:262
25866991310,2cyclictest22279-21wc00:25:002
25866991310,2cyclictest18204-21wc21:05:272
2586699130,12cyclictest0-21swapper/222:16:522
2586299138,4cyclictest16967-21sh00:20:571
2586299131,8cyclictest27185-21kworker/1:120:25:131
25862991313,0cyclictest0-21swapper/123:11:451
2586299131,2cyclictest141rcu_preempt23:17:371
2586299131,1cyclictest24510-21ssh21:32:081
25862991311,1cyclictest10438-21/usr/sbin/munin23:20:221
25862991310,2cyclictest7663-21sh22:40:011
25862991310,2cyclictest2786-21cron23:35:001
25862991310,2cyclictest2262-21grep20:35:141
25862991310,2cyclictest2202-21cat19:25:191
25862991310,2cyclictest14700-21cat21:45:251
2586299130,9cyclictest0-21swapper/119:15:021
2586299130,1cyclictest0-21swapper/123:06:051
2586299130,12cyclictest19517-21cron23:45:001
2586099136,6cyclictest6716-21perf20:45:000
25860991312,1cyclictest18807-21diskmemload21:34:440
25860991312,1cyclictest0-21swapper/019:20:100
25860991310,2cyclictest8807-21grep22:40:140
2586099130,6cyclictest8094-21cron19:40:000
2586099130,0cyclictest0-21swapper/022:02:160
2586099130,0cyclictest0-21swapper/022:02:160
2586899129,2cyclictest6778-21ssh00:15:163
2586899129,2cyclictest5763-21ssh23:55:213
2586899129,2cyclictest3345-21cron19:30:003
2586899129,2cyclictest3117-21cron21:20:003
2586899129,2cyclictest30570-21grep21:55:173
2586899129,2cyclictest30570-21grep21:55:173
2586899129,2cyclictest29267-21perf21:55:003
2586899129,2cyclictest28273-21sed20:20:183
2586899129,2cyclictest28208-21grep19:15:003
2586899129,2cyclictest24655-21grep22:30:173
2586899129,2cyclictest21610-21cut00:05:173
2586899129,2cyclictest14901-21basename22:25:013
2586899129,2cyclictest14567-21grep21:00:153
2586899129,2cyclictest11981-21expr23:40:143
2586899129,2cyclictest11470-21sh21:25:003
2586899128,3cyclictest5674-21cut20:40:183
2586899128,3cyclictest26577-21rm23:48:563
2586899128,3cyclictest1786-21cat23:15:003
2586899127,4cyclictest18807-21diskmemload23:27:433
2586899120,11cyclictest8616-21grep19:40:133
2586699129,2cyclictest5304-21tail20:40:162
2586699129,2cyclictest22938-21tr20:10:142
2586699129,2cyclictest22938-21tr20:10:142
2586699129,2cyclictest14545-21tr21:00:132
2586699129,2cyclictest12770-21grep21:25:182
2586699129,2cyclictest1206-21wc22:35:212
2586699129,2cyclictest10033-21tail19:40:332
2586699128,3cyclictest25330-21apache200:25:132
2586699124,4cyclictest225-21systemd-journal21:00:002
2586699122,2cyclictest56550irq/124-enp0s3119:30:142
2586299129,2cyclictest9163-21grep19:40:241
2586299129,2cyclictest9075-21basename20:50:001
2586299129,2cyclictest6881-21wc20:45:011
2586299129,2cyclictest6842-21chrt23:55:451
2586299129,2cyclictest6575-21cat00:35:001
2586299129,2cyclictest6499-21cron22:20:001
2586299129,2cyclictest26288-21ssh00:07:291
2586299129,2cyclictest23340-21sed21:50:261
2586299129,2cyclictest14254-21tr19:50:201
2586299127,4cyclictest22520-21ssh22:48:321
2586299126,2cyclictest225-21systemd-journal20:05:001
25862991210,1cyclictest5127-21cat19:30:271
25862991210,1cyclictest0-21swapper/119:35:141
2586299120,11cyclictest16496-21cat21:05:001
2586299120,11cyclictest10558-21chrt00:17:281
2586099129,2cyclictest32377-21cut20:30:140
2586099129,2cyclictest26375-21tr19:10:130
2586099129,2cyclictest20938-21cat21:50:000
2586099129,2cyclictest16589-21cut22:45:110
2586099129,2cyclictest12505-21head23:40:150
2586099129,2cyclictest10989-21awk23:00:250
2586099124,4cyclictest2241-21/usr/sbin/munin22:55:210
25860991211,1cyclictest0-21swapper/022:30:440
2586099120,10cyclictest0-21swapper/020:25:150
2586099120,0cyclictest0-21swapper/022:22:280
2586899119,1cyclictest7116-21grep22:00:193
2586899119,1cyclictest7116-21grep22:00:193
2586899118,2cyclictest8583-21ssh21:41:583
2586899118,2cyclictest8234-21ssh23:37:433
2586899118,2cyclictest4299-21ssh22:37:233
2586899118,2cyclictest398-21sed22:15:273
2586899118,2cyclictest3901-21grep20:35:263
2586899118,2cyclictest32473-21tr20:30:143
2586899118,2cyclictest28691-21aten2_r7power_c21:35:123
2586899118,2cyclictest2636-21wc22:55:263
2586899118,2cyclictest25882-21ssh22:11:203
2586899118,2cyclictest24286-21ssh00:25:243
2586899118,2cyclictest22695-21ssh21:30:423
2586899118,2cyclictest21905-21ssh23:07:073
2586899118,2cyclictest2173-21ssh00:12:273
2586899118,2cyclictest19137-21ssh21:10:043
2586899118,2cyclictest18352-21ssh22:26:113
2586899118,2cyclictest17885-21wc22:45:193
2586899118,2cyclictest15771-21grep19:55:153
2586899118,2cyclictest14825-21sed00:00:283
2586899118,2cyclictest14356-21grep19:50:213
2586899118,2cyclictest12182-21ssh23:01:083
2586899118,2cyclictest11297-21chrt20:53:183
2586899118,2cyclictest10611-21ssh22:40:463
2586899116,2cyclictest18447-21grep20:00:133
2586899114,3cyclictest0-21swapper/320:15:133
25868991110,1cyclictest39-21ksoftirqd/322:05:463
2586899110,10cyclictest32149-21ssh23:51:453
2586699118,2cyclictest7581-21cat19:35:262
2586699118,2cyclictest6783-21sh23:17:322
2586699118,2cyclictest493-21grep20:30:162
2586699118,2cyclictest24563-21wc22:10:262
2586699118,2cyclictest22666-21sh21:11:272
2586699118,2cyclictest13878-21grep19:50:172
2586699117,4cyclictest0-21swapper/219:10:492
2586699117,3cyclictest491-21grep19:20:242
25866991110,1cyclictest0-21swapper/220:25:122
2586299118,2cyclictest5710-21rm22:00:051
2586299118,2cyclictest5710-21rm22:00:051
2586299118,2cyclictest29755-21cat21:35:181
2586299118,2cyclictest25329-21expr22:50:151
2586299118,2cyclictest22941-21tr00:25:151
2586299118,2cyclictest22164-21rm21:11:031
2586299118,2cyclictest11310-21taskset20:53:261
2586299118,2cyclictest10722-21sh22:21:461
2586299110,0cyclictest0-21swapper/120:18:041
2586099119,1cyclictest26123-21chrt23:10:120
2586099119,1cyclictest21065-21sort21:10:240
2586099119,1cyclictest13263-21ntpq21:25:200
2586099118,2cyclictest6056-21taskset21:20:450
2586099118,2cyclictest4687-21basename21:40:140
2586099118,2cyclictest3379-21sh22:36:410
2586099118,2cyclictest28871-21taskset19:15:140
2586099118,2cyclictest27219-21ssh00:08:120
2586099118,2cyclictest26288-21ls20:15:190
2586099118,2cyclictest2538-21expr20:35:150
2586099118,2cyclictest24051-21ssh23:46:580
2586099118,2cyclictest18675-21grep20:00:150
2586099118,2cyclictest16516-21sed22:25:190
2586099118,2cyclictest16300-21grep00:20:290
2586099118,2cyclictest15257-21perf19:55:000
2586099118,2cyclictest14041-21turbostat.cron21:00:000
2586099118,2cyclictest13721-21ssh00:00:180
25860991110,1cyclictest0-21swapper/022:16:310
2586099110,10cyclictest1160-21ssh00:11:390
2586899108,1cyclictest30128-21cut20:25:143
2586899108,1cyclictest11718-21ssh21:25:023
2586899107,2cyclictest22895-21egrep20:10:143
2586899107,2cyclictest22895-21egrep20:10:143
2586899107,2cyclictest21552-21grep20:05:223
2586899107,2cyclictest18005-21grep21:05:233
2586899100,9cyclictest5504-21taskset19:32:183
2586699108,1cyclictest12259-21sed19:45:262
2586699107,2cyclictest20910-21sed20:05:172
2586699100,9cyclictest292542sleep220:20:402
2586299108,1cyclictest27645-21wc20:20:151
2586299107,2cyclictest31489-21cat19:20:111
2586299107,2cyclictest20492-21ssh23:25:231
2586299107,2cyclictest12381-21grep19:45:271
2586299107,2cyclictest12060-21cut20:55:151
2586299100,9cyclictest186622sleep121:08:231
2586099109,1cyclictest0-21swapper/023:28:310
2586099108,2cyclictest3961-21cat19:30:160
2586099108,1cyclictest93012sleep020:50:110
2586099108,1cyclictest8416-21wget19:40:140
2586099108,1cyclictest7342-21expr20:45:140
2586099108,1cyclictest6803-21ssh00:35:060
2586099108,1cyclictest3186-21ssh23:35:070
2586099108,1cyclictest288902sleep020:20:260
2586099108,1cyclictest28468-21ssh21:35:030
2586099108,1cyclictest28301-21sh23:50:060
2586099108,1cyclictest226082sleep022:10:090
2586099108,1cyclictest22427-21chrt20:09:560
2586099108,1cyclictest16729-21grep19:55:220
2586099108,1cyclictest1298-21grep19:25:140
2586099107,2cyclictest29748-21ssh21:55:120
2586099107,2cyclictest29748-21ssh21:55:110
2586099107,2cyclictest25086-21ssh22:50:150
2586099107,2cyclictest23902-21ls20:10:210
2586099107,2cyclictest23902-21ls20:10:210
2586099107,2cyclictest14592-21tr21:00:150
2586099107,2cyclictest10884-21aten2_r7power_v19:45:130
2586099103,2cyclictest28779-21grep21:15:190
2586099102,4cyclictest1796-21sshd23:14:590
258689997,1cyclictest837-21taskset19:23:283
258669996,2cyclictest2227-21wget20:35:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional