You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-05 - 03:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue May 05, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
263412175153,15sleep10-21swapper/119:06:091
265712141107,24sleep30-21swapper/319:09:083
247412138103,24sleep20-21swapper/219:05:092
263252136102,23sleep00-21swapper/019:05:560
263192790,3sleep00-21swapper/023:19:080
251042690,2sleep30-21swapper/300:05:303
288962670,1sleep00-21swapper/000:09:160
199722650,1sleep20-21swapper/223:14:412
137322620,1sleep20-21swapper/222:20:432
187612570,2sleep20-21swapper/220:05:122
179042540,2sleep10-21swapper/122:24:551
9942510,8sleep122-21ksoftirqd/122:35:231
311582500,1sleep30-21swapper/321:45:403
2691399500,32cyclictest4866-21rm22:15:051
2691399490,3cyclictest8179-21ssh21:29:341
71932480,1sleep00-21swapper/023:52:380
137352480,1sleep00-21swapper/000:21:230
300952470,1sleep10-21swapper/119:15:191
307082460,1sleep30-21swapper/321:21:213
2691399465,25cyclictest762-21seq20:35:581
57092440,1sleep10-21swapper/123:03:481
2691399449,20cyclictest20026-21sh21:37:581
26913994310,16cyclictest26448-21sh22:30:221
137712420,2chrt0-21swapper/223:33:512
117672420,2chrt0-21swapper/222:44:192
2691399419,18cyclictest26567-21sh21:42:491
2691399419,18cyclictest21520-21fschecks_count21:15:151
2691399406,20cyclictest23763-21sh00:29:481
2691399405,22cyclictest27302-21sh00:07:401
2691399401,26cyclictest30008-21ls21:45:171
26913994011,15cyclictest23437-21sh22:52:311
26913994010,17cyclictest22457-21sh00:04:321
2691399395,21cyclictest13432-21sh00:21:061
2691399382,29cyclictest30066-21ssh00:10:121
2691599362,12cyclictest9057-21ssh21:30:123
2691399360,5cyclictest636-21runrttasks21:05:371
2691399360,5cyclictest636-21runrttasks21:05:371
94342350,2chrt0-21swapper/323:54:553
2691599353,29cyclictest1637-21ntpq23:00:213
26915993512,20cyclictest22361-21expr23:40:133
2691599350,8cyclictest4047-21uptime21:25:253
26913993511,21cyclictest19157-21tune2fs20:05:161
2691399350,3cyclictest24288-21ssh00:30:061
2691599340,7cyclictest2123-21/usr/sbin/munin20:40:183
2691399349,22cyclictest26464-21seq23:19:161
2691399349,21cyclictest26478-21cat20:20:241
2691399348,23cyclictest4117-21load23:50:181
2691599332,27cyclictest21637-21cron23:40:003
26915993313,17cyclictest11240-21ntpq21:55:203
26915993311,19cyclictest16775-21netstat21:35:193
2691399332,28cyclictest29064-21grep21:20:201
26913993311,19cyclictest4869-21cat20:45:241
26913993310,21cyclictest24924-21ls22:05:221
26913993310,20cyclictest14437-21expr19:55:131
2691399330,6cyclictest12709-21ssh23:32:461
2691399330,4cyclictest8390-21ssh22:40:541
2691399330,3cyclictest1272-21ssh23:24:461
2691399330,3cyclictest1272-21ssh23:24:451
26915993213,16cyclictest17606-21sh00:25:073
26915993210,5cyclictest23128-21expr20:15:133
26915993210,5cyclictest23128-21expr20:15:133
26915993210,20cyclictest27223-21date20:25:013
26915993210,19cyclictest9933-21basename21:00:103
2691399329,20cyclictest5992-21expr19:35:141
2691399328,22cyclictest21086-21rm22:02:541
2691399328,21cyclictest31461-21cut20:35:011
2691399326,23cyclictest5330-21sed19:35:011
2691399322,27cyclictest1899-21sed20:40:151
2691399320,5cyclictest5006-21idleruntime00:15:181
2691399320,4cyclictest0-21swapper/121:55:141
2691399320,3cyclictest3781-21ssh22:14:261
2691399320,3cyclictest24516-21rm22:29:501
2691399320,3cyclictest18083-21sh23:36:251
2691399320,3cyclictest11400-21sed21:00:261
2691399320,2cyclictest640-21snmpd21:10:341
2691399320,2cyclictest123452chrt23:08:461
2691399320,29cyclictest13034-21grep22:45:131
2691599318,6cyclictest15520-21cat19:55:243
2691599312,7cyclictest6160-21apt-get22:40:093
26915993111,18cyclictest23500-21ntp_kernel_pll_21:40:203
2691599310,8cyclictest24876-21ntp_states22:05:223
2691599310,8cyclictest21540-21ssh22:03:213
2691399319,19cyclictest10309-21df19:45:141
26913993110,18cyclictest16748-21df20:00:151
26913993110,18cyclictest14115-21cat23:10:121
2691399310,5cyclictest24478-21ssh23:41:061
2691399310,4cyclictest9822-21kernelversion21:30:181
2691399310,3cyclictest6343-21ssh21:51:341
2691399310,3cyclictest6343-21ssh21:51:341
255132310,4sleep225511-21ssh22:30:142
2691599309,5cyclictest27175-21cat22:55:183
2691599309,4cyclictest14547-21expr21:10:123
2691599308,6cyclictest17400-21seq23:12:063
2691599305,6cyclictest26092-21sed22:55:003
2691599303,4cyclictest0-21swapper/319:40:113
26915993012,15cyclictest14492-21sh00:22:113
26915993010,6cyclictest7794-21expr23:05:173
2691399308,19cyclictest8046-21expr19:40:121
2691399305,22cyclictest6126-1kworker/u9:223:25:051
26913993010,17cyclictest28909-21expr23:45:131
2691399300,5cyclictest4869-21ssh00:39:421
2691399300,4cyclictest28026-21unixbench_singl22:55:261
37962290,2chrt0-21swapper/000:14:530
326252290,9sleep228-21ksoftirqd/220:35:212
2691599298,7cyclictest3375-21cat23:25:253
2691599298,4cyclictest4146-21sh22:14:483
2691599297,5cyclictest8878-21seq23:30:143
2691599297,5cyclictest25309-21cat22:30:113
2691599296,7cyclictest31391-21swap00:10:233
2691599296,5cyclictest12587-21grep19:50:173
2691399298,18cyclictest21217-21expr20:10:121
26913992911,15cyclictest27874-21ntpq19:10:201
2691399290,6cyclictest16276-40mandb00:00:001
26912992912,10cyclictest52750irq/139-enp3s0-22:35:100
133912290,1sleep20-21swapper/222:45:142
2691599287,6cyclictest18170-21cat22:24:593
2691599286,5cyclictest4711-21grep20:45:233
2691599286,5cyclictest13367-21sed21:05:273
2691599286,5cyclictest13367-21sed21:05:263
2691599281,9cyclictest20076-21sed22:25:263
2691599281,7cyclictest30970-21unixbench_singl20:30:253
26915992811,15cyclictest13554-21ssh23:57:193
2691599280,9cyclictest21523-21sed23:15:163
2691399285,20cyclictest25056-21latency_hist20:20:011
2691399285,20cyclictest25056-21latency_hist20:20:001
26913992813,13cyclictest12554-21fschecks_time19:50:151
2691399280,4cyclictest9285-21users20:55:251
2691599279,6cyclictest18305-21ls20:05:003
2691599278,5cyclictest8599-21cat20:55:193
2691599272,10cyclictest7977-21seq21:53:123
2691599272,10cyclictest7977-21seq21:53:113
2691599271,8cyclictest28196-21find19:10:233
2691599271,8cyclictest21124-21ssh21:15:113
2691599271,8cyclictest14237-21/usr/sbin/munin22:45:233
2691599271,7cyclictest4300-21/usr/sbin/munin00:15:243
2691599271,7cyclictest28805-21chrt23:45:123
26915992711,13cyclictest10876-21ntp_kernel_pll_19:45:193
26915992710,3cyclictest5738-21ssh22:40:003
2691599270,8cyclictest29252-21sed23:20:253
2691599270,8cyclictest29252-21sed23:20:253
2691399274,6cyclictest31776-21expr19:20:131
2691399271,5cyclictest3067-21cron19:30:001
116892270,2sleep20-21swapper/223:55:262
2691599269,3cyclictest27975-21if_err_enp3s020:25:173
2691599265,7cyclictest3903-21sed19:30:163
2691599264,5cyclictest31965-21grep19:20:173
2691599264,5cyclictest2498-21expr19:25:243
2691599261,8cyclictest5374-21/usr/sbin/munin22:15:143
2691599261,7cyclictest7559-21cron19:40:003
2691599261,7cyclictest318-21cat00:35:233
2691599261,7cyclictest24882-21fschecks_time00:30:153
26915992612,12cyclictest29683-21fschecks_count19:15:153
26915992610,3cyclictest18577-21ssh00:00:363
26915992610,13cyclictest19423-21kernelversion20:05:183
2691599260,8cyclictest6641-21grep20:50:233
2691399261,7cyclictest7034-21grep20:50:261
2691399261,5cyclictest27429-21grep20:25:111
2691299261,24cyclictest0-21swapper/019:12:410
2691599252,7cyclictest21735-21grep20:10:193
26915992510,13cyclictest310-21sensors_temp20:35:223
26914992422,1cyclictest0-21swapper/219:12:342
26914992414,7cyclictest6002-21df19:35:132
228062240,1sleep20-21swapper/221:40:142
313272230,3sleep031309-21ssh22:10:170
2691499238,6cyclictest29486-21cat23:45:172
2691499232,14cyclictest3112-21sh23:50:032
26914992313,7cyclictest24266-21cat00:05:222
26912992314,6cyclictest15583-21sed23:10:240
26912992311,8cyclictest560-21dbus-daemon23:45:230
26914992214,6cyclictest16709-21cat21:35:192
26914992214,5cyclictest52950irq/141-enp3s0-22:25:502
26914992212,8cyclictest9943-21seq19:45:022
26914992212,6cyclictest6464-21grep22:15:232
26912992212,7cyclictest27402-21cron21:19:590
26914992110,9cyclictest278-21systemd-journal00:40:002
2691299213,12cyclictest28057-21apt-get21:20:130
26912992114,5cyclictest26766-21sort22:55:160
26912992113,6cyclictest15174-21sshd19:55:190
2691499202,13cyclictest27662-21ssh23:20:122
2691499202,13cyclictest27662-21ssh23:20:122
2691499201,7cyclictest9682-21hddtemp_smartct21:30:162
26914992013,5cyclictest28-21ksoftirqd/223:00:002
26914992013,4cyclictest17876-21tr00:25:142
26914992012,6cyclictest25657-21tune2fs20:20:152
26914992011,7cyclictest17571-21python320:00:242
26914992011,6cyclictest4386-21missed_timers21:50:192
26914992011,6cyclictest4386-21missed_timers21:50:192
2691499200,10cyclictest3085-21/usr/sbin/munin23:25:232
2691299209,3cyclictest0-21swapper/000:30:220
2691299203,6cyclictest11735-21apt-get22:20:120
26912992014,4cyclictest9726-21sh19:45:010
26912992014,4cyclictest25871-21sh22:54:590
26912992014,4cyclictest16404-21sh21:59:570
26912992012,6cyclictest5484-21seq23:03:350
26912992012,6cyclictest28487-21seq22:08:250
26912992012,6cyclictest19131-21date00:25:240
26912992011,7cyclictest2903-21seq20:40:400
2691299201,11cyclictest16233-21munin-run00:00:000
219062200,1sleep20-21swapper/220:10:212
2691499192,12cyclictest2376-21ssh21:25:132
2691499192,12cyclictest1360-21apt-get20:40:122
26914991912,5cyclictest9677-21run-parts21:00:002
26914991912,5cyclictest14093-21sed21:10:012
26914991912,5cyclictest14093-21sed21:10:012
26914991912,4cyclictest13967-21diskmemload23:01:272
26914991911,6cyclictest10173-21expr21:00:142
26914991911,5cyclictest7124-21sed20:50:262
26914991910,7cyclictest8213-21tune2fs19:40:142
26914991910,6cyclictest560-21dbus-daemon21:25:002
26914991910,6cyclictest15134-21ntp_kernel_err19:55:202
2691499190,5cyclictest0-21swapper/200:32:092
2691499190,16cyclictest16607-21grep22:00:012
2691299199,7cyclictest636-21runrttasks00:35:190
2691299198,8cyclictest19829-1kworker/u9:019:30:120
2691299192,12cyclictest75-21kswapd021:50:110
2691299192,12cyclictest75-21kswapd021:50:100
2691299192,11cyclictest5644-21apt-get20:50:130
26912991912,5cyclictest32322-21sed20:35:170
26912991911,6cyclictest20252-21awk20:05:240
26912991911,5cyclictest4345-21grep20:45:190
26912991910,7cyclictest25755-21which22:30:160
26912991910,6cyclictest8667-21awk20:55:200
26912991910,6cyclictest29221-21cron20:30:000
2691499182,11cyclictest14377-21apt-get21:10:122
26914991812,4cyclictest3898-21sh22:14:332
26914991812,4cyclictest20817-21sh00:02:522
26914991812,4cyclictest16899-21sh23:35:242
26914991811,5cyclictest30332-21expr00:10:132
26914991811,5cyclictest28-21ksoftirqd/223:41:052
26914991811,5cyclictest22343-21sed23:15:232
26914991811,5cyclictest21496-21diskstats21:15:152
26914991811,5cyclictest11824-21ls00:20:152
2691499181,12cyclictest23804-21apt-get22:05:122
2691499181,11cyclictest13967-21diskmemload22:00:132
26914991810,6cyclictest30099-21cat19:15:182
26914991810,5cyclictest28-21ksoftirqd/221:45:242
2691299188,8cyclictest25821-21if_err_enp3s020:20:160
2691299182,12cyclictest5384-21ntp_states00:15:210
2691299182,10cyclictest52750irq/139-enp3s0-23:40:110
2691299181,8cyclictest1948-21/usr/sbin/munin23:25:230
26912991813,3cyclictest7268-21sh23:05:090
26912991813,3cyclictest18992-21sh21:36:540
26912991813,3cyclictest16388-21/usr/sbin/munin20:00:230
26912991812,4cyclictest21657-21systemctl23:40:000
26912991812,4cyclictest10551-21sh22:43:050
26912991812,3cyclictest17082-21cpuspeed_turbos22:00:130
26912991811,5cyclictest28925-21tr23:20:220
26912991811,5cyclictest28925-21tr23:20:220
26912991811,5cyclictest278-21systemd-journal19:30:000
26912991811,5cyclictest22124-21sed20:10:230
2691299181,15cyclictest13043-21ntpq19:50:220
26912991811,5cyclictest11312-21date21:00:240
26912991811,5cyclictest10139-21sed23:30:250
26912991811,4cyclictest2810-21fschecks_time21:25:160
26912991811,4cyclictest13967-21diskmemload21:42:070
26912991810,6cyclictest560-21dbus-daemon22:15:010
26912991810,6cyclictest24792-21seq20:18:260
26912991810,6cyclictest24792-21seq20:18:250
26912991810,6cyclictest17285-21expr00:00:150
26912991810,6cyclictest13315-21tune2fs22:45:150
26912991810,6cyclictest12-21ksoftirqd/021:45:540
26912991810,5cyclictest32051-21idleruntime19:20:180
26912991810,5cyclictest29483-21apt-get20:30:110
26912991810,5cyclictest29367-21sort19:15:130
200512180,10sleep228-21ksoftirqd/222:50:172
2691499179,6cyclictest1408-21cat19:25:132
2691499179,5cyclictest3516-21expr19:30:132
2691499172,3cyclictest141rcu_preempt00:15:152
2691499171,11cyclictest7368-21apt-get23:05:142
26914991710,5cyclictest28094-21load20:25:182
26914991710,5cyclictest23237-21expr20:15:142
26914991710,5cyclictest23237-21expr20:15:142
26914991710,5cyclictest12404-21expr19:50:142
2691299178,6cyclictest5247-21gpgv19:35:010
26912991712,3cyclictest13967-21diskmemload21:10:070
26912991712,3cyclictest10767-21users21:30:260
26912991711,4cyclictest13967-21diskmemload22:26:140
2691499169,5cyclictest3808-21ls20:45:142
2691499168,6cyclictest32629-21python319:20:242
2691499161,6cyclictest1103-21sed22:35:232
26914991610,4cyclictest29949-21gltestperf20:30:152
2691299169,5cyclictest12609-21head21:05:190
2691299169,5cyclictest12609-21head21:05:180
2691299169,5cyclictest10172-21expr19:45:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional