You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 12:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Sep 11, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
265629773,8sleep00-21swapper/019:07:590
257029369,8sleep20-21swapper/219:06:522
261828067,9sleep30-21swapper/319:07:293
263727868,6sleep10-21swapper/119:07:441
166472290,1chrt16648-21ssh22:00:212
290699270,27cyclictest0-21swapper/200:23:512
2906992624,1cyclictest0-21swapper/223:40:112
290699260,26cyclictest0-21swapper/222:30:232
290699260,26cyclictest0-21swapper/221:14:502
290699260,24cyclictest0-21swapper/223:59:112
291099250,2cyclictest0-21swapper/321:43:243
2906992422,1cyclictest0-21swapper/221:25:202
290699240,24cyclictest0-21swapper/223:10:552
291099230,22cyclictest0-21swapper/322:54:113
291099230,21cyclictest0-21swapper/322:45:033
290499235,10cyclictest0-21swapper/121:00:111
290699227,13cyclictest0-21swapper/223:53:262
2906992218,3cyclictest27934-21diskmemload00:15:142
290499222,19cyclictest0-21swapper/122:25:131
2906992118,3cyclictest27934-21diskmemload21:53:162
2906992115,6cyclictest0-21swapper/222:25:012
290699210,20cyclictest0-21swapper/219:35:132
290699207,11cyclictest0-21swapper/200:34:292
2906992018,1cyclictest1872-21sh23:09:352
2906992016,1cyclictest55650irq/127-enp2s0-21:15:152
290499200,19cyclictest0-21swapper/121:10:121
290099200,20cyclictest15306-21ssh00:15:120
305032190,0sleep00-21swapper/022:47:280
2910991919,0cyclictest0-21swapper/322:13:133
2910991919,0cyclictest0-21swapper/321:54:103
290699198,3cyclictest55850irq/129-enp2s0-23:17:362
290699192,17cyclictest0-21swapper/222:51:582
290699190,18cyclictest0-21swapper/221:34:332
291099183,6cyclictest0-21swapper/320:10:133
2910991817,1cyclictest0-21swapper/323:06:323
290699187,10cyclictest0-21swapper/200:35:222
290699186,9cyclictest0-21swapper/200:01:402
290699184,14cyclictest0-21swapper/221:44:102
2906991817,1cyclictest0-21swapper/200:11:542
2906991817,1cyclictest0-21swapper/200:11:542
2906991817,0cyclictest27934-21diskmemload23:23:322
2906991816,1cyclictest27934-21diskmemload23:30:162
2906991816,1cyclictest27934-21diskmemload22:20:112
2906991815,2cyclictest27934-21diskmemload22:05:402
2906991813,1cyclictest0-21swapper/200:05:162
2906991810,5cyclictest0-21swapper/222:47:542
2904991813,4cyclictest0-21swapper/123:40:161
290499180,1cyclictest0-21swapper/122:20:211
291099171,3cyclictest0-21swapper/322:37:213
291099171,3cyclictest0-21swapper/322:37:213
290699170,15cyclictest0-21swapper/222:16:262
2904991711,2cyclictest11031-21head19:25:161
290099175,6cyclictest0-21swapper/021:57:400
290099174,7cyclictest13494-21munin-run21:40:000
2900991716,1cyclictest0-21swapper/020:10:130
291099168,3cyclictest141rcu_preempt23:58:533
2910991615,1cyclictest0-21swapper/322:34:043
290699169,7cyclictest0-21swapper/221:38:582
290699165,11cyclictest0-21swapper/221:59:092
2906991614,1cyclictest27934-21diskmemload22:57:452
2906991610,6cyclictest0-21swapper/223:37:322
290699160,15cyclictest0-21swapper/222:11:552
290099163,7cyclictest7602-21apt-get00:30:010
290099162,11cyclictest31026-21apt-get21:50:130
290099162,10cyclictest20824-21ssh23:40:130
72992150,0chrt0-21swapper/023:31:280
2910991512,2cyclictest0-21swapper/322:41:023
291099150,12cyclictest0-21swapper/300:00:143
290699158,7cyclictest0-21swapper/200:25:112
290699157,7cyclictest0-21swapper/220:05:142
290699153,1cyclictest0-21swapper/219:25:132
290699152,2cyclictest0-21swapper/223:25:362
2906991511,1cyclictest27934-21diskmemload22:43:002
290499150,15cyclictest0-21swapper/123:47:321
290499150,15cyclictest0-21swapper/123:23:281
290499150,13cyclictest0-21swapper/100:11:351
290499150,13cyclictest0-21swapper/100:11:351
2900991511,2cyclictest0-21swapper/021:41:100
291099142,8cyclictest13616-21/usr/sbin/munin23:35:203
2910991410,3cyclictest19390-21grep20:50:183
2910991410,2cyclictest5431-21cat19:15:013
290699149,4cyclictest0-21swapper/223:03:312
290699148,6cyclictest0-21swapper/221:20:322
290699144,10cyclictest0-21swapper/221:45:512
290699143,10cyclictest16516-21grep20:45:152
290699143,10cyclictest16516-21grep20:45:152
2906991413,1cyclictest0-21swapper/220:40:132
2906991412,1cyclictest1841-21latency_hist20:15:002
2906991412,1cyclictest0-21swapper/222:36:042
2906991412,1cyclictest0-21swapper/222:36:032
290499143,7cyclictest20961-21runrttasks22:41:331
290499143,6cyclictest12870-21/usr/sbin/munin19:30:141
2904991413,1cyclictest0-21swapper/100:01:261
2904991411,2cyclictest3730-21sh21:52:531
2904991411,2cyclictest0-21swapper/123:08:161
2904991410,3cyclictest15666-21tr21:40:231
290499140,1cyclictest0-21swapper/122:12:251
290499140,13cyclictest30499-21munin-run21:50:001
290099146,7cyclictest3888-21munin-run21:15:010
2900991411,2cyclictest24601-21basename19:55:000
291099139,3cyclictest4193-21ssh23:30:173
291099139,2cyclictest7168-21wc21:35:223
291099137,5cyclictest29628-21apt-get20:05:133
2910991313,0cyclictest0-21swapper/321:24:073
2910991313,0cyclictest0-21swapper/300:11:213
2910991313,0cyclictest0-21swapper/300:11:213
2910991310,2cyclictest4063-21cron20:20:003
2910991310,2cyclictest28905-21sed23:25:163
2910991310,2cyclictest24176-21ssh22:05:153
2910991310,2cyclictest14348-21wc19:30:243
2910991310,2cyclictest11203-21cron20:34:593
290699139,2cyclictest24027-21/usr/sbin/munin21:00:152
290699137,5cyclictest21042-21acpi20:55:122
290699135,6cyclictest27934-21diskmemload23:47:302
290699135,5cyclictest0-21swapper/220:00:132
2906991311,1cyclictest20802-21perf20:55:002
290499139,3cyclictest27796-21grep20:00:161
290499139,2cyclictest24053-21grep22:05:151
290499139,2cyclictest18666-21grep23:00:141
2904991310,2cyclictest9660-21grep19:20:241
2904991310,2cyclictest32395-21sh00:25:151
2904991310,2cyclictest27464-21grep21:05:271
2904991310,2cyclictest21296-21ls19:45:191
2904991310,2cyclictest19787-21sh19:45:001
2904991310,2cyclictest13499-21ls21:40:001
2904991310,2cyclictest13070-21cut23:35:171
290499130,1cyclictest27934-21diskmemload00:16:311
290099135,7cyclictest27003-21lockfile-create20:00:000
290099135,7cyclictest12438-21apt-get19:30:000
290099134,8cyclictest3135-21apt-get19:10:130
2900991312,1cyclictest0-21swapper/021:05:140
2900991310,2cyclictest11552-21cut23:15:120
2900991310,2cyclictest10322-21head20:30:210
290099130,12cyclictest31471-21sed21:30:270
291099129,2cyclictest8408-21ssh00:30:143
291099129,2cyclictest8126-21grep22:15:123
291099129,2cyclictest6966-21cut20:25:143
291099129,2cyclictest6577-21wc19:15:193
291099129,2cyclictest4823-21sed23:10:263
291099129,2cyclictest361-21sort00:25:163
291099129,2cyclictest27398-21wc21:05:263
291099129,2cyclictest23995-21grep00:20:153
291099129,2cyclictest23158-21sh21:00:003
291099129,2cyclictest16002-21tail19:35:173
291099129,2cyclictest12189-21sed22:55:293
291099129,2cyclictest11299-21cut23:15:113
291099128,3cyclictest17934-21sed20:45:263
291099128,3cyclictest17934-21sed20:45:263
291099124,5cyclictest5372-21apt-get23:50:153
291099124,2cyclictest10418-21taskset21:56:373
291099120,12cyclictest0-21swapper/323:46:263
290699129,2cyclictest7220-21tail19:15:252
290699129,2cyclictest13114-21unixbench_multi20:35:262
290699128,3cyclictest26060-21grep21:05:152
290699128,3cyclictest18385-21grep19:40:162
290699123,2cyclictest225-21systemd-journal19:30:122
290499129,2cyclictest31643-21ssh00:05:191
290499129,2cyclictest26623-21grep22:45:181
290499129,2cyclictest2224-21grep22:30:221
290499129,2cyclictest15528-21grep19:35:141
290499129,2cyclictest14050-21tr23:55:151
290499129,2cyclictest12958-21grep21:20:161
290499129,2cyclictest1217-21sh23:28:151
290499129,2cyclictest10929-21uniq22:55:181
290499128,3cyclictest18359-21sh20:50:001
290499128,3cyclictest18359-21sh20:50:001
290499127,4cyclictest9328-21awk22:15:201
290499127,2cyclictest8494-21ssh23:13:131
290499127,2cyclictest6224-21ssh22:52:191
290499124,5cyclictest6481-21awk20:24:591
290499120,1cyclictest141rcu_preempt21:19:161
290099129,2cyclictest7805-21grep00:10:200
290099129,2cyclictest7805-21grep00:10:200
290099129,2cyclictest21199-21cut21:25:130
290099129,2cyclictest20417-21cat23:40:000
290099129,2cyclictest18572-21cat22:40:190
290099129,2cyclictest18358-21grep22:20:260
290099129,2cyclictest16504-21grep20:45:150
290099129,2cyclictest16504-21grep20:45:140
290099129,2cyclictest16180-21grep22:00:180
290099129,2cyclictest16086-21wc22:19:590
290099129,2cyclictest13673-21cat20:40:000
290099128,3cyclictest22198-21grep20:55:200
290099128,3cyclictest18061-21turbostat.cron23:00:010
290099128,2cyclictest14123-21grep20:40:140
290099124,1cyclictest13-21ksoftirqd/022:10:530
290099120,12cyclictest0-21swapper/023:54:000
291099118,2cyclictest4212-21ssh00:08:253
291099118,2cyclictest31723-21ssh21:11:253
291099118,2cyclictest29440-21ssh22:27:303
291099118,2cyclictest29237-21ssh21:30:023
291099118,2cyclictest25136-21ssh23:42:143
291099118,2cyclictest22607-21ssh23:21:093
291099118,2cyclictest20860-21ssh21:25:003
291099118,2cyclictest19514-21ssh22:21:193
291099118,2cyclictest19298-21ssh00:35:573
291099118,2cyclictest18328-21ssh23:00:033
291099118,2cyclictest17373-21ssh00:15:303
291099118,2cyclictest14494-21grep20:40:163
291099118,2cyclictest11533-21sed19:25:203
291099117,2cyclictest18776-21grep19:40:193
291099110,10cyclictest11071-21ssh21:19:073
291099110,10cyclictest0-21swapper/321:46:453
290699119,1cyclictest0-21swapper/219:20:112
290699118,2cyclictest7017-21grep20:25:142
290699118,2cyclictest4079-21apt20:20:002
290699118,2cyclictest3414-21tr19:10:152
290699118,2cyclictest20822-21grep19:45:152
290499118,2cyclictest9679-21tr00:30:211
290499118,2cyclictest9135-21ssh22:35:141
290499118,2cyclictest9135-21ssh22:35:131
290499118,2cyclictest6179-21tail23:50:191
290499118,2cyclictest3578-21ssh21:33:581
290499118,2cyclictest31417-21taskset20:06:081
290499118,2cyclictest23986-21cut00:20:151
290499118,2cyclictest18978-21ssh00:35:411
290499118,2cyclictest16176-21tail22:00:181
290499118,2cyclictest12254-21cut20:35:171
290499117,3cyclictest4159-21ssh23:30:161
290499117,3cyclictest3838-21tail19:10:171
290499112,1cyclictest27934-21diskmemload21:26:581
290099118,2cyclictest875-21ssh23:27:590
290099118,2cyclictest8050-21rm23:12:520
290099118,2cyclictest6255-21cat21:15:280
290099118,2cyclictest3186-21head20:15:230
290099118,2cyclictest31324-21grep00:05:170
290099118,2cyclictest30253-21sh23:45:160
290099118,2cyclictest28050-21ssh21:48:050
290099118,2cyclictest27972-21grep20:00:170
290099118,2cyclictest25828-21rm22:05:310
290099118,2cyclictest24915-21grep22:25:140
290099118,2cyclictest24534-21ssh00:00:450
290099118,2cyclictest23437-21chrt00:20:010
290099118,2cyclictest20491-21grep23:20:150
290099118,2cyclictest19878-21tail23:00:210
290099118,2cyclictest1886-21grep22:30:190
290099118,2cyclictest18162-21ssh23:57:210
290099118,2cyclictest17864-21cat00:35:210
290099118,2cyclictest17676-21apt19:40:120
290099118,2cyclictest15626-21ssh21:21:080
290099118,2cyclictest15504-21grep19:35:130
290099118,2cyclictest15013-21ls19:35:000
290099118,2cyclictest11916-21chrt00:31:430
290099117,3cyclictest9705-21wc22:55:010
290099111,9cyclictest0-21swapper/023:06:080
290099110,10cyclictest11407-21ssh22:35:400
290099110,10cyclictest11407-21ssh22:35:400
291099108,1cyclictest99862sleep319:24:203
291099108,1cyclictest266602sleep319:56:163
291099108,1cyclictest24444-21chrt19:53:173
291099108,1cyclictest16001-21cut22:00:173
291099107,2cyclictest5995-21tail20:20:253
291099107,2cyclictest27385-21grep20:00:153
291099107,2cyclictest23465-21cut21:00:133
291099107,2cyclictest20330-21expr19:45:143
290699108,1cyclictest63322sleep220:23:252
290699108,1cyclictest26644-21chrt19:56:042
290699107,2cyclictest9337-21tr20:30:132
290699107,2cyclictest22709-21cut19:50:142
290499108,1cyclictest8733-21taskset20:28:481
290499108,1cyclictest39072sleep120:18:031
290499108,1cyclictest22817-21taskset20:55:431
290499107,2cyclictest9732-21grep20:30:161
290499107,2cyclictest6885-21ssh21:55:111
290499107,2cyclictest6670-21ntpq19:15:191
290499107,2cyclictest309-21sed20:10:171
290499107,2cyclictest26365-21cat19:55:241
290499107,2cyclictest23395-21irqstats19:50:181
290499107,2cyclictest19010-21cat20:50:161
290499107,2cyclictest13972-21grep20:40:141
290499107,2cyclictest13638-21ssh23:15:431
290099108,1cyclictest6123-21sh19:15:150
290099108,1cyclictest23591-21grep21:00:130
290099108,1cyclictest20469-21cat19:45:140
290099108,1cyclictest19085-21expr20:50:150
290099107,2cyclictest7526-21smartctl20:25:170
290099107,2cyclictest5514-21ntp_states20:20:190
290099107,2cyclictest30380-21tail20:05:180
290099107,2cyclictest10068-21awk19:25:010
29109997,1cyclictest13355-21chrt20:37:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional