You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 12:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Apr 30, 2026 00:45:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
91702175100,24sleep20-21swapper/219:06:202
9251217499,25sleep10-21swapper/119:07:231
9173216095,54sleep00-21swapper/019:06:220
92282142106,25sleep30-21swapper/319:07:053
104272660,3sleep210410-21ssh22:10:172
233942650,5sleep00-21swapper/021:53:460
296362600,2sleep00-21swapper/021:34:320
126512580,2sleep00-21swapper/022:40:050
78912570,2sleep10-21swapper/121:41:351
63932570,1sleep00-21swapper/021:16:340
281932550,2sleep30-21swapper/321:05:253
265322550,2sleep00-21swapper/021:56:430
238072550,2sleep10-21swapper/123:16:341
9728995211,25cyclictest19899-21cat19:30:252
9727995212,23cyclictest11768-21sh22:39:261
9727995112,22cyclictest19652-21sh21:26:121
9727995013,19cyclictest32672-21sh23:50:321
71262500,3sleep10-21swapper/123:57:201
63292490,1sleep06328-21ntp_kernel_pll_20:15:190
152202460,1sleep00-21swapper/020:35:250
106172460,11sleep334-21ksoftirqd/321:20:143
972899458,21cyclictest5073-21sh22:32:522
972899440,8cyclictest18384-21ssh00:06:432
972899440,7cyclictest14964-21ssh21:23:252
972899438,22cyclictest11037-21sh00:28:192
972899420,7cyclictest22775-21ssh22:48:362
147492420,3sleep39724-21cyclictest23:09:223
972899416,21cyclictest30965-21sh23:22:042
202842410,1sleep10-21swapper/100:35:441
66702400,1sleep30-21swapper/322:07:253
142872400,3chrt0-21swapper/000:30:210
972899390,6cyclictest19142-21expr23:40:132
972999384,15cyclictest13383-21cut22:40:153
972999380,9cyclictest151802rm21:47:123
9728993813,22cyclictest10499-21memory20:25:192
9728993813,22cyclictest10499-21memory20:25:192
146832380,7sleep3341ksoftirqd/300:30:243
972899370,7cyclictest3392-21ssh22:59:492
972899370,6cyclictest31008-21ssh00:17:332
972899370,5cyclictest21997-21grep20:50:262
9728993613,20cyclictest29672-21cat21:10:162
9728993610,15cyclictest28826-21diskmemload22:51:362
972899360,8cyclictest9323-21ssh21:43:022
972899360,7cyclictest32568-21ssh22:29:372
972899360,5cyclictest24100-21unixbench_singl20:55:262
9727993612,21cyclictest12268-21grep19:15:141
9728993513,19cyclictest24520-21sh00:11:112
9728993513,19cyclictest24520-21sh00:11:112
9728993512,20cyclictest11809-21sh23:33:332
9728993511,21cyclictest12163-21expr20:30:132
972899350,9cyclictest7057-21ssh23:01:452
972899350,6cyclictest12141-21ssh23:06:412
972899350,5cyclictest25846-21taskset22:23:022
972999342,12cyclictest29274-21apt-get19:55:113
972899349,22cyclictest32063-21timerwakeupswit21:35:252
972899346,25cyclictest26170-21sed21:00:242
972899342,26cyclictest16231-21apt-get20:40:122
972899341,23cyclictest28826-21diskmemload23:53:372
972899340,7cyclictest2349-21ssh22:03:182
972899340,7cyclictest2349-21ssh22:03:182
972899340,6cyclictest5545-21ssh23:28:572
972899340,4cyclictest16550-21ssh00:32:092
972899340,3cyclictest7527-21ssh22:08:182
972899340,3cyclictest18591-21cat21:50:142
9727993412,19cyclictest2629-21grep22:30:261
972999331,15cyclictest4887-21ssh21:40:043
972899339,21cyclictest19568-21rm23:12:292
972899339,21cyclictest15107-21find20:35:242
972899336,22cyclictest52950irq/141-enp3s0-21:30:262
972899330,7cyclictest26648-21ssh21:56:502
972899330,6cyclictest9277-21ssh23:59:292
972899330,6cyclictest27981-21needreboot19:50:182
972899330,4cyclictest3573-21ssh00:20:572
9727993310,8cyclictest1880-21cat20:05:181
9727993310,6cyclictest5912-21expr20:15:141
9727993310,21cyclictest26933-21awk21:05:131
318122330,4sleep3560-21dbus-daemon21:35:233
972899327,22cyclictest0-21swapper/222:19:472
972899326,23cyclictest12117-21expr19:15:132
972899321,28cyclictest21178-21head19:35:162
972899321,28cyclictest21178-21head19:35:162
9728993211,18cyclictest6991-21seq20:15:592
9728993210,19cyclictest8240-21sh22:35:522
972899320,6cyclictest14531-21sh00:03:012
972899320,5cyclictest25983-21ssh23:45:292
972899320,5cyclictest17045-21ssh21:25:062
972899320,4cyclictest5464-21taskset21:15:382
972899320,4cyclictest15167-21ssh22:41:052
972799329,6cyclictest12647-21sed21:45:201
972799328,7cyclictest26694-21pluginstate21:05:001
972799328,5cyclictest28275-21basename23:20:161
972799322,27cyclictest4597-21grep23:00:161
9727993214,16cyclictest13159-21diskstats22:40:141
9727993211,18cyclictest27894-21memory19:50:161
972899319,18cyclictest14307-21sh23:35:522
972899316,22cyclictest30072-21ls19:55:182
972899312,9cyclictest17046-21cat19:25:172
972899311,11cyclictest31439-21taskset20:00:122
972899310,3cyclictest23928-21ssh23:16:412
972899310,3cyclictest15442-21python319:20:212
972899310,3cyclictest14161-21ssh21:46:102
972899310,27cyclictest20718-21taskset00:36:112
972799318,4cyclictest19166-21expr23:40:151
9727993112,16cyclictest24540-21sh22:21:431
9727993112,16cyclictest14949-21sh22:14:021
972899300,8cyclictest26073-21gzip19:45:222
972899300,3cyclictest8246-21irqstats20:20:182
9727993012,15cyclictest8141-21sh22:08:551
310802300,2sleep30-21swapper/322:00:193
310802300,2sleep30-21swapper/322:00:193
972899291,9cyclictest9769-21apt-get19:10:102
972899291,8cyclictest18714-21/usr/sbin/munin20:45:152
972899291,7cyclictest2660-21/usr/sbin/munin20:05:262
972799299,5cyclictest22-21ksoftirqd/120:50:011
972799295,6cyclictest6872-21seq23:30:051
972799292,8cyclictest220462sleep100:10:111
972799292,8cyclictest220462sleep100:10:111
9727992912,15cyclictest25925-21sh21:56:061
9727992912,15cyclictest1298-21/usr/sbin/munin00:20:111
9727992912,14cyclictest30253-21sh22:54:251
9727992911,16cyclictest11702-21sh00:29:001
9727992911,15cyclictest16164-21sh23:37:461
9727992910,5cyclictest22-21ksoftirqd/122:25:381
972899280,8cyclictest27362-21tune2fs21:05:172
972799289,3cyclictest4090-21sh21:39:431
972799287,5cyclictest30785-21vmstat19:55:251
972799282,4cyclictest0-21swapper/119:40:131
972799281,8cyclictest15429-21sort23:10:011
9727992812,14cyclictest16673-21sh00:32:171
9727992810,15cyclictest5727-21sh23:29:091
248062280,2chrt0-21swapper/022:50:120
972899271,6cyclictest3296-21/usr/sbin/munin20:10:132
972899270,8cyclictest23183-21expr19:40:142
972799279,6cyclictest16191-21basename19:25:001
972799279,3cyclictest15295-21sh21:23:461
972799278,3cyclictest11623-21cron20:30:001
972799278,3cyclictest11623-21cron20:30:001
972799277,4cyclictest17858-21unixbench_multi19:25:251
972799277,3cyclictest0-21swapper/121:13:351
972799274,5cyclictest13388-21vmstat20:30:251
972799272,5cyclictest25074-21apt-get23:45:041
972799271,8cyclictest9795-21rm21:20:001
972799271,7cyclictest9068-21head20:20:251
972799271,7cyclictest19978-21unixbench_multi19:30:261
972799271,7cyclictest19346-21tail21:50:201
972799271,7cyclictest16939-21head23:10:211
9727992711,14cyclictest18405-21sh22:17:181
9727992710,3cyclictest29928-21sh21:34:501
9727992710,3cyclictest18073-21sh00:06:241
972799270,9cyclictest21688-21ls20:50:241
972799267,3cyclictest32615-21rm22:01:051
972799267,3cyclictest32615-21rm22:01:051
972799261,8cyclictest23798-21/usr/sbin/munin20:55:231
972799261,8cyclictest2060-21rm22:58:291
972799261,8cyclictest16597-21grep20:40:161
972799261,8cyclictest11542-21find00:00:241
972799261,7cyclictest25668-21tr19:45:171
972799261,1cyclictest0-21swapper/119:14:471
972799260,9cyclictest19372-21ssh22:45:081
972799260,8cyclictest15098-21gunzip20:35:231
972699261,24cyclictest0-21swapper/019:14:130
972699260,24cyclictest0-21swapper/019:15:100
97142260,1sleep00-21swapper/023:31:260
972999252,12cyclictest25336-21cut23:45:013
972999251,22cyclictest0-21swapper/319:15:073
972999251,12cyclictest8317-21sh23:30:233
972799259,3cyclictest20875-21cstates19:35:131
972799259,3cyclictest20875-21cstates19:35:131
972799255,5cyclictest31525-21grep20:00:131
972799251,9cyclictest3634-21diskstats20:10:161
972799250,8cyclictest28854-21ssh00:15:221
972999246,4cyclictest63282ntp_kernel_pll_20:15:193
9729992422,1cyclictest0-21swapper/319:14:243
9729992417,5cyclictest3046-21ssh00:20:263
972999233,15cyclictest29281-21ssh21:10:133
972999232,16cyclictest17005-21ssh21:25:043
9726992312,9cyclictest560-21dbus-daemon21:10:000
169412230,2chrt0-21swapper/022:15:490
972999222,13cyclictest20505-21apt-get20:50:123
9729992215,4cyclictest25884-21sh22:50:223
972699228,12cyclictest1958-21rm23:52:160
972699221,5cyclictest0-21swapper/022:04:550
972699221,5cyclictest0-21swapper/022:04:550
9726992214,6cyclictest12-21ksoftirqd/000:00:000
9726992213,7cyclictest30099-21ls19:55:180
972999212,14cyclictest515-21irqbalance00:00:113
9729992114,5cyclictest636-21runrttasks19:30:123
9729992113,6cyclictest278-21systemd-journal23:59:553
9729992112,6cyclictest10077-21diskstats20:25:143
9729992112,6cyclictest10077-21diskstats20:25:143
9726992116,4cyclictest12-21ksoftirqd/023:40:000
9726992115,5cyclictest12-21ksoftirqd/023:45:000
9726992113,6cyclictest30854-21ls21:10:260
9726992113,6cyclictest16186-21diskstats23:10:140
119992210,1sleep30-21swapper/322:39:403
972999207,8cyclictest636-21runrttasks21:50:133
972999202,13cyclictest27135-21apt-get19:50:113
972999201,7cyclictest27818-21munin-run22:25:003
9729992012,5cyclictest25682-21awk21:00:193
9729992011,6cyclictest5126-21ls23:00:213
9729992011,6cyclictest11237-21grep22:10:253
972699202,7cyclictest10852-21hddtemp_smartct21:20:150
972699202,13cyclictest7596-21apt-get20:20:120
9726992013,5cyclictest17939-21idleruntime-cro21:50:000
9726992012,6cyclictest26674-21cron21:05:000
9726992012,6cyclictest21463-21memory19:35:180
9726992012,6cyclictest21463-21memory19:35:180
9726992012,6cyclictest1731-21sed20:05:170
9726992012,6cyclictest11943-21seq22:39:380
9726992012,6cyclictest11809-21ls20:30:010
9726992012,6cyclictest11809-21ls20:30:010
9726992012,5cyclictest636-21runrttasks22:33:360
9726992012,5cyclictest12-21ksoftirqd/023:20:120
9726992011,7cyclictest26675-21seq19:47:300
972999199,7cyclictest560-21dbus-daemon20:05:013
972999199,7cyclictest12826-21sed00:30:013
972999192,12cyclictest640-21snmpd20:40:133
9729991913,4cyclictest5790-21sh21:15:573
9729991912,4cyclictest28826-21diskmemload00:10:133
9729991912,4cyclictest28826-21diskmemload00:10:133
9729991911,6cyclictest2222-21which22:30:233
9729991911,5cyclictest16738-21cat19:25:143
972699197,3cyclictest52750irq/139-enp3s0-00:00:120
972699194,4cyclictest23621-1kworker/u9:222:49:360
972699192,8cyclictest21565-21ssh22:20:130
9726991913,4cyclictest7584-21sh23:02:180
9726991913,4cyclictest3490-21sh21:39:070
9726991912,5cyclictest20227-21seq19:32:170
9726991912,4cyclictest25965-21sh23:45:280
9726991911,6cyclictest22203-21grep00:10:120
9726991911,6cyclictest22203-21grep00:10:120
9726991911,6cyclictest18638-21cat20:45:140
972999184,3cyclictest28826-21diskmemload22:30:003
972999183,12cyclictest25494-21ssh21:30:253
972999182,12cyclictest75-21kswapd023:20:113
972999182,11cyclictest13818-21apt-get19:20:043
972999182,11cyclictest1153-21apt-get20:05:123
972999181,7cyclictest26520-21ssh23:46:033
9729991812,4cyclictest21618-21idleruntime-cro00:10:003
9729991812,3cyclictest30500-21sh00:17:033
9729991811,5cyclictest9031-21awk20:20:253
972999181,15cyclictest31194-21df_inode23:50:133
9729991811,5cyclictest22854-21expr20:55:133
9729991811,4cyclictest28826-21diskmemload23:10:133
9729991810,6cyclictest3623-21basename23:00:003
972699188,3cyclictest313262ssh22:55:170
972699183,11cyclictest640-21snmpd22:10:120
972699182,11cyclictest22864-21apt-get19:40:130
972699181,8cyclictest3185-21ssh23:26:340
9726991813,3cyclictest4271-21sh00:21:400
9726991813,3cyclictest18355-21sh00:06:420
9726991813,3cyclictest13013-21sh23:07:360
9726991813,3cyclictest13002-21sendmail_mailtr20:30:210
9726991812,4cyclictest30118-21sh00:16:400
9726991812,4cyclictest20596-21sh00:36:030
9726991812,4cyclictest19844-21sh21:26:240
9726991811,5cyclictest20712-21expr20:50:140
972699181,13cyclictest0-21swapper/022:09:590
9726991810,6cyclictest5102-21cat21:40:120
9726991810,6cyclictest31715-21expr20:00:120
9726991810,6cyclictest16829-21grep19:25:160
9726991810,6cyclictest16604-21tune2fs20:40:160
264672180,1chrt0-21swapper/323:19:173
972999179,6cyclictest5001-21seq20:12:543
972999179,6cyclictest24218-21cat22:50:003
972999179,6cyclictest20429-21wc20:50:013
972999178,6cyclictest24737-21cron19:45:003
9729991712,3cyclictest20679-21/usr/sbin/munin19:35:203
9729991712,3cyclictest20679-21/usr/sbin/munin19:35:203
9729991711,4cyclictest28826-21diskmemload23:35:223
9729991711,3cyclictest28826-21diskmemload23:27:423
9729991711,3cyclictest28826-21diskmemload00:36:353
9729991710,5cyclictest34-21ksoftirqd/321:56:553
9729991710,5cyclictest14239-21expr20:35:153
9729991710,5cyclictest12233-21grep20:30:143
972699179,6cyclictest23596-21pluginstate20:55:220
972699179,6cyclictest15042-21memory19:20:170
972699177,7cyclictest22299-21rm23:15:020
9726991712,3cyclictest8510-21sh00:25:460
9726991712,3cyclictest4391-21sendmail_mailst20:10:250
9726991712,3cyclictest28258-21sh22:25:160
169962170,9sleep334-21ksoftirqd/322:15:523
972999168,6cyclictest25794-21grep19:45:183
9726991610,4cyclictest28348-21sed19:50:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional