You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 08:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Sep 02, 2026 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
183822178103,27sleep00-21swapper/019:07:060
184472162112,28sleep30-21swapper/319:07:573
183692162107,27sleep20-21swapper/219:06:562
186062161101,30sleep10-21swapper/119:10:001
241392620,2sleep10-21swapper/122:57:271
312342580,3sleep118877-21cyclictest21:51:151
155962580,7sleep122-21ksoftirqd/122:50:491
155962580,7sleep122-21ksoftirqd/122:50:491
145602560,1sleep20-21swapper/222:03:192
232592540,2sleep20-21swapper/223:44:402
222162540,4sleep118877-21cyclictest21:21:011
222162540,4sleep118877-21cyclictest21:21:001
128902510,2sleep20-21swapper/220:10:152
1888299479,23cyclictest32198-21sh23:26:483
287502460,3sleep2271rcuc/200:11:372
318812450,1sleep10-21swapper/123:03:211
18881994414,16cyclictest11234-21sh22:48:142
1888199441,24cyclictest5597-21rm00:19:012
1888199441,24cyclictest5597-21rm00:19:002
1888199441,24cyclictest10953-21sh00:22:322
244612430,3sleep00-21swapper/022:57:460
1888199431,24cyclictest11664-21ssh23:11:472
1888199431,23cyclictest13368-21sh23:36:412
1888199430,25cyclictest14796-21smartctl22:50:242
1888199430,25cyclictest14796-21smartctl22:50:242
227102420,1sleep30-21swapper/300:30:323
1888299421,6cyclictest14690-21sh22:03:263
18881994213,15cyclictest29928-21sh23:01:272
1888199421,29cyclictest0-21swapper/222:57:152
1888199421,22cyclictest7662-21ssh22:21:212
1888199420,26cyclictest22545-21ssh21:21:192
1888199420,26cyclictest22545-21ssh21:21:192
1888199411,27cyclictest24910-21sh23:21:212
1888199411,25cyclictest2457-21sh23:29:252
1888199411,24cyclictest6512-21ssh23:31:402
1888199411,24cyclictest5724-21diskmemload23:57:062
1888199411,24cyclictest5724-21diskmemload21:33:262
1888199411,24cyclictest19343-21ssh22:06:172
1888199411,24cyclictest19343-21ssh22:06:162
1888199411,23cyclictest22819-21ssh22:32:522
1888199411,21cyclictest3865-21ssh23:05:462
1888199410,25cyclictest19028-21rm21:42:402
1888199410,25cyclictest16342-21ssh21:17:002
1888199410,25cyclictest1564-21ssh22:17:062
1888199410,25cyclictest1455-21ssh23:51:412
1888199410,21cyclictest27545-21ssh22:35:482
1888199401,22cyclictest5724-21diskmemload23:19:582
1888199400,25cyclictest30499-21rm00:36:292
1888199400,23cyclictest27787-21ssh21:49:352
1888199400,23cyclictest22049-21ssh00:06:442
1888199400,23cyclictest18493-21ssh00:04:572
1888199400,22cyclictest5557-21sh21:56:112
1888199400,22cyclictest17807-21sh00:27:242
1888299395,2cyclictest0-21swapper/322:42:103
1888199391,24cyclictest28555-21ssh21:25:322
1888199391,23cyclictest11261-21ssh21:36:472
1888199391,22cyclictest5724-21diskmemload21:11:092
1888199391,22cyclictest5724-21diskmemload00:30:482
1888199390,25cyclictest15529-21ssh22:27:232
1888199390,24cyclictest476-21awk20:55:222
1888199390,23cyclictest27707-21rm22:12:482
1888199390,23cyclictest1590-21ssh22:40:272
1888199390,20cyclictest0-21swapper/223:49:112
153592390,4chrt0-21swapper/021:40:130
303662380,2sleep30-21swapper/322:38:353
80192370,3chrt34-21ksoftirqd/322:45:253
1888299374,26cyclictest702-21apt-get21:30:043
1888299372,31cyclictest8155-21runrttasks21:35:103
1888299371,6cyclictest25260-21ssh21:47:053
1888299370,7cyclictest6933-21ssh00:20:053
1888199370,21cyclictest27332-21chrt20:43:572
1888299368,25cyclictest9400-21sed23:10:183
1888299361,9cyclictest1207-21sh23:05:003
1888199360,22cyclictest31959-21ssh21:51:562
1888199360,21cyclictest29345-21taskset20:47:212
244902350,3chrt0-21swapper/021:46:190
1888299351,12cyclictest30419-21sh22:15:033
18882993511,21cyclictest11902-21sed20:05:243
1888299350,7cyclictest9716-21ssh22:23:233
1888299350,7cyclictest25651-21ssh23:22:063
1888299350,5cyclictest5724-21diskmemload00:29:183
1888299350,5cyclictest13321-21ssh22:25:263
1888299350,3cyclictest26649-21/usr/sbin/munin21:25:133
271602340,2chrt0-21swapper/123:46:471
1888299348,23cyclictest26349-21sed20:40:203
1888299348,22cyclictest14599-21sed20:15:013
1888299347,24cyclictest21943-21cat20:30:193
1888299347,22cyclictest17920-21processes22:05:233
1888299347,22cyclictest17920-21processes22:05:223
1888299346,25cyclictest14757-21grep21:15:293
1888299346,24cyclictest20967-21taskset21:20:213
1888299346,24cyclictest20967-21taskset21:20:213
1888299341,6cyclictest20773-21ssh23:42:123
1888299341,30cyclictest29401-21/usr/sbin/munin00:35:273
18882993411,20cyclictest6951-21awk23:55:253
1888299340,4cyclictest32677-21sh00:15:133
1888299340,4cyclictest32677-21sh00:15:133
312262330,2chrt0-21swapper/100:37:121
1888299338,22cyclictest27789-21sed19:30:003
1888299331,28cyclictest2653-21sed21:00:213
18882993311,19cyclictest4436-21fschecks_count19:50:153
1888299330,5cyclictest14949-21cstates20:15:133
1888199331,29cyclictest2896-21ls19:45:212
1888199331,20cyclictest29556-21chrt19:31:552
1888199330,19cyclictest8354-21/usr/sbin/munin20:00:162
1888299328,21cyclictest5974-21kworker/u8:023:38:523
1888299320,4cyclictest22699-21grep19:15:253
1888199320,18cyclictest0-21swapper/220:09:392
1888299319,19cyclictest5724-21diskmemload22:10:113
1888299317,21cyclictest27732-21awk20:45:153
1888299313,4cyclictest1039-21awk19:40:233
1888299313,25cyclictest0-21swapper/320:35:113
1888299312,3cyclictest5724-21diskmemload23:30:133
1888299312,26cyclictest31943-21cron19:40:003
1888299311,3cyclictest624-21gdbus20:25:243
1888299311,11cyclictest21310-21chrt00:06:013
18882993110,19cyclictest1846-21sh21:54:153
1888299310,4cyclictest24467-21sh22:57:483
1888299310,2cyclictest0-21swapper/300:00:213
1888199315,23cyclictest6913-21cat19:55:212
1888199311,16cyclictest0-21swapper/219:51:542
1888199310,17cyclictest0-21swapper/220:15:452
1888299307,20cyclictest23511-21cpuspeed_turbos19:20:133
1888299306,4cyclictest15360-21/usr/sbin/munin21:40:133
1888299305,6cyclictest560-21dbus-daemon20:00:013
1888299301,9cyclictest25004-21munin-run22:35:003
1888299301,8cyclictest1585-21ssh23:05:013
1888299301,5cyclictest9561-21latency_hist22:00:003
1888299301,3cyclictest5724-21diskmemload21:10:233
1888299301,26cyclictest28690-21wc19:30:203
1888299301,11cyclictest1340-21taskset23:51:353
18882993011,16cyclictest5724-21diskmemload22:50:263
18882993011,16cyclictest5724-21diskmemload22:50:253
1888299300,6cyclictest5724-21diskmemload00:11:493
1888199301,28cyclictest0-21swapper/219:24:222
1888199301,18cyclictest22887-21runrttasks20:32:022
193132290,2chrt0-21swapper/122:06:151
193132290,2chrt0-21swapper/122:06:151
1888299291,7cyclictest2366-21fschecks_time19:45:153
1888299291,7cyclictest20931-21sed19:15:013
1888299291,6cyclictest15350-21/usr/sbin/munin23:15:213
1888299290,8cyclictest25241-21rm23:45:223
1888199291,16cyclictest1595-21taskset19:43:552
18881992911,15cyclictest21272-21apt19:15:122
1888299286,1cyclictest0-21swapper/321:00:013
1888299281,7cyclictest4208-21tune2fs21:05:143
1888299281,7cyclictest17482-21gltestperf20:20:163
1888299281,4cyclictest9371-21pmu-power20:00:223
1888299281,4cyclictest30349-21sed20:50:153
1888199288,17cyclictest30628-21sed20:50:192
1888199281,14cyclictest0-21swapper/221:00:172
1888199281,13cyclictest17713-21memory20:20:192
18879992823,3cyclictest23411-21/usr/sbin/munin19:20:170
1888199271,23cyclictest5263-21unixbench_multi21:05:242
1888199271,23cyclictest21091-21latency_hist19:15:012
1887999271,24cyclictest0-21swapper/019:29:130
18881992610,14cyclictest23993-21if_enp3s020:35:162
1888199260,23cyclictest21009-21idleruntime-cro20:30:012
1888099260,24cyclictest0-21swapper/119:29:131
1888099260,24cyclictest0-21swapper/119:20:081
243732250,9sleep122-21ksoftirqd/122:34:241
1888199251,21cyclictest25971-21cstates19:25:132
1888199250,22cyclictest30435-21tune2fs19:35:152
18880992523,1cyclictest0-21swapper/120:06:421
18880992523,1cyclictest0-21swapper/119:32:051
18880992522,2cyclictest0-21swapper/119:58:001
1888099249,9cyclictest21881-21ssh21:45:131
18880992422,1cyclictest0-21swapper/120:13:371
18880992422,1cyclictest0-21swapper/119:54:051
1888099242,15cyclictest12184-21ssh00:00:121
1888099241,11cyclictest66050lldpd23:29:511
18879992422,1cyclictest0-21swapper/020:14:200
18879992422,1cyclictest0-21swapper/019:41:430
18879992422,1cyclictest0-21swapper/019:31:380
18879992414,7cyclictest20259-21python320:25:240
1887999240,1cyclictest0-21swapper/019:39:390
70912230,9sleep122-21ksoftirqd/122:20:481
18880992312,8cyclictest17218-21apt-get23:40:121
18879992321,1cyclictest0-21swapper/019:19:530
18879992313,7cyclictest29947-21cron22:15:000
1887999231,13cyclictest22579-21grep00:30:260
18880992220,1cyclictest0-21swapper/119:19:191
1888099221,10cyclictest81372sh23:56:331
1887999222,6cyclictest0-21swapper/023:58:000
18879992213,7cyclictest7902-21seq19:57:330
219562210,4sleep121945-21ntp_states00:30:221
18880992115,4cyclictest6236-21dump-pmu-power22:45:001
18880992115,4cyclictest22-21ksoftirqd/122:05:001
18880992115,4cyclictest22-21ksoftirqd/100:15:001
18880992113,6cyclictest22-21ksoftirqd/121:45:011
18880992113,6cyclictest17286-21seq21:17:541
1888099211,15cyclictest30420-21ssh22:15:031
1887999215,10cyclictest52950irq/141-enp3s0-22:40:100
1887999212,6cyclictest6135-21apt-get21:10:110
18879992119,1cyclictest0-21swapper/019:53:480
18879992114,4cyclictest19890-21grep22:30:240
1888099207,11cyclictest3989-21ssh21:55:201
1888099202,13cyclictest7085-21ssh00:20:131
1888099201,5cyclictest11104-21irqstats23:35:171
18880992012,6cyclictest31954-21apt-get20:55:121
18880992012,6cyclictest15198-21pmu-power00:25:211
18880992011,7cyclictest9816-21awk20:00:261
18880992011,7cyclictest31670-21seq20:54:471
18880992010,7cyclictest28905-21cat20:45:251
1887999208,10cyclictest0-21swapper/021:15:180
1887999207,11cyclictest18288-21ls22:30:020
1887999201,5cyclictest0-21swapper/021:58:330
18879992012,6cyclictest5071-21ls21:05:230
18879992012,6cyclictest3368-21grep23:05:250
18879992011,6cyclictest30172-21grep23:25:210
18879992010,6cyclictest52950irq/141-enp3s0-23:40:110
18879992010,3cyclictest95302ssh00:21:090
221912190,2chrt0-21swapper/020:30:230
1888099199,7cyclictest52750irq/139-enp3s0-23:05:121
1888099199,7cyclictest23288-21ssh23:20:201
1888099192,12cyclictest21242-21apt-get20:30:151
18880991913,4cyclictest674-21netstat19:40:201
18880991913,4cyclictest28954-21sh21:25:551
18880991913,3cyclictest20070-21sh00:05:221
18880991912,5cyclictest20568-21seq23:18:501
18880991912,5cyclictest16421-21seq20:16:511
18880991911,6cyclictest30456-21cut19:35:151
18880991911,6cyclictest19059-21expr19:10:141
18880991910,6cyclictest9712-21head23:10:221
18880991910,5cyclictest0-21swapper/121:30:131
1887999199,6cyclictest52950irq/141-enp3s0-21:35:110
1887999198,9cyclictest11451-21rm23:11:340
1887999192,13cyclictest16741-21ssh22:05:120
1887999192,13cyclictest16741-21ssh22:05:120
1887999192,12cyclictest640-21snmpd00:35:130
1887999192,12cyclictest4994-21apt-get22:20:130
1887999192,12cyclictest10368-21ssh23:35:120
1887999191,8cyclictest19890-21apt-get21:20:110
1887999191,8cyclictest19890-21apt-get21:20:110
18879991914,3cyclictest28768-21cpuspeed_turbos21:50:130
18879991913,4cyclictest672-21sh22:16:120
18879991913,4cyclictest3822-21sh21:31:100
18879991913,4cyclictest32286-21df_inode20:55:140
18879991913,4cyclictest31486-21sh22:39:410
18879991913,4cyclictest29729-21sh21:26:400
18879991913,4cyclictest15190-21sh00:01:420
18879991912,5cyclictest28412-21cat20:45:190
18879991912,5cyclictest2258-21seq21:00:160
18879991911,6cyclictest612-21entropy00:15:160
18879991911,6cyclictest612-21entropy00:15:160
18879991911,6cyclictest560-21dbus-daemon22:00:230
18879991911,6cyclictest27179-21seq20:41:580
18879991910,7cyclictest23522-21expr20:35:130
1888099189,6cyclictest3043-21timerwakeupswit21:00:251
1888099188,7cyclictest1-21systemd22:10:241
1888099188,5cyclictest22-21ksoftirqd/120:25:121
1888099186,11cyclictest6558-21fschecks_count21:10:141
1888099182,10cyclictest16959-21chrt20:20:131
1888099181,8cyclictest8163-21ssh21:35:111
1888099181,8cyclictest6481-21ssh23:31:381
18880991812,4cyclictest8585-21sh22:45:391
18880991812,4cyclictest27433-21dump-pmu-power20:45:001
18880991811,4cyclictest32053-21processes23:50:231
18880991810,6cyclictest4079-21expr21:05:121
18880991810,6cyclictest3924-21sed19:50:001
18880991810,6cyclictest12299-21tune2fs22:25:161
1888099180,6cyclictest26981-21timerwakeupswit22:35:251
1887999189,7cyclictest15470-21cat23:15:130
1887999189,7cyclictest12596-21cat20:10:010
1887999182,11cyclictest14794-21apt-get20:15:130
1887999181,8cyclictest13815-21ssh22:50:150
1887999181,8cyclictest13815-21ssh22:50:140
18879991813,3cyclictest16733-21sh00:26:200
18879991812,4cyclictest24570-21sh23:45:160
18879991812,4cyclictest19119-21cpuspeed_turbos00:05:140
18879991811,5cyclictest31233-21df23:50:140
18879991811,5cyclictest23280-21ntpq23:20:210
1887999181,12cyclictest3495-21apt-get23:30:130
1887999181,12cyclictest16956-21apt-get20:20:130
18879991810,6cyclictest7538-21ls22:45:200
18879991810,6cyclictest3591-21seq19:47:050
1888099172,9cyclictest32650-21apt-get00:15:141
1888099172,9cyclictest32650-21apt-get00:15:131
1887999171,6cyclictest26186-21df00:10:150
1887999171,11cyclictest29815-21apt-get20:50:120
18879991710,5cyclictest27117-21find23:00:090
18879991710,5cyclictest19114-21expr19:10:140
1888099167,7cyclictest23446-21cat20:35:121
1887999169,5cyclictest8359-21cat20:00:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional