You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-12 - 06:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Aug 12, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1239222097,99sleep00-21swapper/019:07:550
1119221096,94sleep20-21swapper/219:06:212
319172143106,26sleep30-21swapper/319:05:053
12792143108,25sleep10-21swapper/119:08:241
20242720,2sleep00-21swapper/022:07:270
289712710,2sleep228956-21ssh21:15:162
156032700,3sleep228-21ksoftirqd/223:30:122
114872700,9sleep122-21ksoftirqd/100:39:341
4782620,3sleep20-21swapper/222:05:572
9412590,2sleep30-21swapper/323:19:143
149502580,2sleep00-21swapper/023:53:380
96712550,4sleep2271rcuc/219:25:262
276072540,1sleep10-21swapper/100:26:411
163332520,1sleep00-21swapper/022:41:580
92702510,1sleep30-21swapper/322:36:373
170192510,2sleep20-21swapper/221:30:212
190442500,2sleep10-21swapper/119:50:001
299202490,2sleep334-21ksoftirqd/323:40:203
299202490,2sleep334-21ksoftirqd/323:40:203
175072470,2sleep20-21swapper/222:43:092
50262460,2sleep30-21swapper/319:15:243
16682460,2sleep20-21swapper/223:19:572
63242450,2sleep20-21swapper/220:35:162
39622450,2sleep00-21swapper/000:33:480
230052450,1sleep00-21swapper/021:10:250
29752440,1sleep00-21swapper/021:20:160
293102440,1sleep00-21swapper/000:28:250
262272440,1sleep30-21swapper/321:37:093
130232430,1sleep20-21swapper/219:35:172
34862420,1sleep30-21swapper/323:20:223
235012420,2chrt0-21swapper/222:23:472
277882410,1sleep10-21swapper/100:02:581
271622400,1sleep00-21swapper/000:02:200
14442400,2sleep10-21swapper/123:19:441
132502390,2sleep10-21swapper/100:15:461
168099358,18cyclictest9671-21rm22:37:001
168099336,10cyclictest24226-21ssh22:24:311
168099335,18cyclictest28632-21ssh22:27:111
168099329,20cyclictest24372-21latency_hist00:25:001
168099328,8cyclictest32399-21ssh00:05:521
168099327,8cyclictest9995-21chrt00:14:131
168099326,8cyclictest21206-21diskmemload22:45:481
168099317,8cyclictest10982-21ssh21:25:271
168099317,10cyclictest25649-21rm23:13:131
168099316,12cyclictest7327-21ssh23:23:531
168099315,11cyclictest28005-21ssh23:39:251
168099314,9cyclictest4421-21ssh00:34:141
168099314,10cyclictest21206-21diskmemload22:31:031
154262310,3chrt0-21swapper/300:17:563
168099304,9cyclictest21206-21diskmemload21:53:521
168099304,8cyclictest31873-21sh22:53:571
168099304,10cyclictest29914-21sh22:04:401
311332290,4sleep131135-21fschecks_count22:05:151
168099296,8cyclictest21206-21diskmemload22:43:131
168099295,8cyclictest0-21swapper/121:20:351
168099295,15cyclictest0-21swapper/123:48:181
168099294,9cyclictest29336-21cut21:15:191
168099294,9cyclictest19012-21ssh21:55:281
168099294,8cyclictest20268-21ssh21:32:531
168099294,8cyclictest16268-21ssh23:54:561
168099294,10cyclictest6742-21latency19:20:191
33182280,1sleep00-21swapper/022:56:040
33182280,1sleep00-21swapper/022:56:030
313462280,5sleep30-21swapper/321:40:333
168099287,8cyclictest5868-21ssh21:45:451
168099285,8cyclictest1620-21ssh23:43:441
168099285,8cyclictest1620-21ssh23:43:431
168099284,9cyclictest21206-21diskmemload22:56:371
168099284,9cyclictest21206-21diskmemload22:56:361
168099282,16cyclictest0-21swapper/121:44:021
168099282,10cyclictest20362-21ssh23:09:391
1680992811,14cyclictest11123-21ls20:45:181
225972270,3sleep00-21swapper/021:35:010
168099275,8cyclictest0-21swapper/123:28:381
168099266,6cyclictest8059-21ssh22:11:481
168099266,6cyclictest8059-21ssh22:11:471
168099265,8cyclictest25580-21sh21:12:391
168099264,7cyclictest21206-21diskmemload23:56:421
168099262,8cyclictest25379-21ssh21:36:181
1680992623,2cyclictest0-21swapper/119:15:001
1679992624,1cyclictest0-21swapper/019:14:070
167999261,21cyclictest5924-21cron20:35:000
168199251,23cyclictest0-21swapper/219:13:562
168099254,8cyclictest21206-21diskmemload23:05:001
168099252,9cyclictest17831-21ssh23:30:561
1680992512,10cyclictest19175-21tr19:50:111
103622250,2sleep20-21swapper/200:38:262
38132240,3sleep20-21swapper/200:09:452
194112240,0sleep0131rcuc/021:05:150
168299248,12cyclictest25726-21kworker/u8:100:10:133
168299246,12cyclictest21206-21diskmemload23:05:143
1682992419,4cyclictest34-21ksoftirqd/322:25:003
168299241,1cyclictest0-21swapper/319:14:163
1681992422,1cyclictest0-21swapper/219:15:372
168099243,6cyclictest32174-21apt-get20:20:121
168099242,8cyclictest648-21ntpd20:58:161
1680992415,4cyclictest23397-21apt-get20:00:161
168099241,14cyclictest0-21swapper/122:15:191
167999242,15cyclictest21166-21ssh23:10:130
1679992414,7cyclictest16644-21ntpq23:30:200
167999241,10cyclictest9422sh23:19:130
168299232,17cyclictest1536-21kworker/u8:122:45:113
1682992314,6cyclictest13443-21users22:15:273
1681992317,4cyclictest28-21ksoftirqd/223:10:002
168099239,11cyclictest3186-21ls20:25:251
168099235,13cyclictest0-21swapper/119:40:121
1680992310,4cyclictest7166-21sed20:35:251
1679992321,1cyclictest0-21swapper/019:15:040
116282230,3sleep20-21swapper/223:26:272
98792220,2sleep20-21swapper/200:14:072
168299227,4cyclictest141rcu_preempt22:10:123
168299227,4cyclictest141rcu_preempt22:10:123
168299222,15cyclictest3310-21kworker/u8:200:00:143
1682992216,4cyclictest34-21ksoftirqd/323:40:003
1682992212,7cyclictest27926-21cron21:15:003
1682992210,9cyclictest34-21ksoftirqd/321:55:123
168199229,12cyclictest28-21ksoftirqd/219:25:002
1681992216,4cyclictest27608-21sh00:26:412
1681992214,6cyclictest7911-21ls22:35:252
168099229,10cyclictest19066-21dump-pmu-power21:05:001
168099228,11cyclictest5338-21cat20:30:251
168099222,7cyclictest0-21swapper/119:30:211
1680992214,4cyclictest8165-21apt-get19:25:121
1680992212,8cyclictest28268-21cat20:10:191
1680992212,7cyclictest4983-21mailstats19:15:231
1680992212,7cyclictest26691-21python320:05:301
1680992210,9cyclictest9706-21cat20:40:231
1680992210,9cyclictest21706-21gltestperf19:55:171
1680992210,9cyclictest13166-21grep19:35:181
167999221,9cyclictest23202grep20:25:160
167999220,8cyclictest66050lldpd22:38:010
145982220,1sleep00-21swapper/021:52:460
168299219,7cyclictest5705-21cpu00:35:133
168299212,14cyclictest25986-21ssh00:25:243
1682992110,8cyclictest7718-21ssh23:00:103
1682992110,10cyclictest16999-21ssh22:42:383
168199219,10cyclictest0-21swapper/223:45:182
1681992115,4cyclictest8347-21sh23:24:542
1681992113,6cyclictest21545-21diskstats23:10:162
1681992113,6cyclictest16125-21ls19:40:242
1681992110,5cyclictest13399-21seq21:51:342
1680992111,7cyclictest20742-21grep21:05:281
1680992111,7cyclictest13607-21awk20:50:221
167999212,12cyclictest636-21runrttasks20:15:130
1679992112,7cyclictest3882-21users23:20:250
180992200,1sleep00-21swapper/019:45:220
168299209,7cyclictest53050irq/142-enp3s0-21:20:143
168299202,13cyclictest31895-21apt-get22:30:043
168299202,12cyclictest943-21apt-get22:55:113
168299202,12cyclictest943-21apt-get22:55:113
1682992014,4cyclictest10117-21sh21:50:003
1682992012,6cyclictest20100-21sed21:05:213
1682992012,6cyclictest11552-21ls23:50:253
1682992012,6cyclictest10901-21cut20:45:163
1682992011,7cyclictest15478-21cut20:55:193
1682992010,7cyclictest560-21dbus-daemon20:30:003
168299200,8cyclictest18676-21kernelversion00:20:183
168199209,7cyclictest52950irq/141-enp3s0-00:20:122
168199209,7cyclictest52950irq/141-enp3s0-00:00:132
168199202,12cyclictest19755-21apt-get22:45:122
1681992015,3cyclictest9429-21sh21:49:192
1681992014,4cyclictest28-21ksoftirqd/200:00:002
1681992013,5cyclictest21206-21diskmemload21:44:112
1681992012,6cyclictest560-21dbus-daemon23:05:012
1681992012,6cyclictest4301-21unixbench-2d21:20:292
1681992012,6cyclictest4238-21expr20:30:142
1681992012,6cyclictest25767-21idleruntime22:25:142
1681992012,6cyclictest23857-21perl20:00:202
1681992011,7cyclictest11814-21cat22:15:142
1681992011,7cyclictest10595-21sort20:45:142
1681992011,6cyclictest18058-21ntp_kernel_err21:00:212
168099207,10cyclictest29877-21dpkg20:15:011
167999203,7cyclictest560-21dbus-daemon00:15:000
167999203,15cyclictest141rcu_preempt21:40:130
1679992012,6cyclictest5856-21sed22:10:180
1679992012,6cyclictest5856-21sed22:10:180
1679992012,6cyclictest32573-21perl22:30:170
1679992012,6cyclictest29193-21cat21:15:180
1679992012,6cyclictest18924-21ssh21:31:340
1679992012,6cyclictest15853-21ls23:05:270
1679992012,6cyclictest11897-21seq19:31:220
1679992011,7cyclictest21180-21ssh23:58:080
1679992011,7cyclictest18442-21perl00:20:170
308912190,2chrt0-21swapper/322:05:133
168299198,8cyclictest34-21ksoftirqd/322:50:033
168299198,7cyclictest53050irq/142-enp3s0-22:25:103
168299196,11cyclictest34-21ksoftirqd/323:15:003
168299191,9cyclictest724-21ssh21:18:313
168299191,6cyclictest0-21swapper/321:30:263
1682991913,4cyclictest20513-21sh23:57:283
1682991912,5cyclictest30251-21ls22:05:003
1682991912,5cyclictest27461-21sed20:10:013
1682991912,4cyclictest11626-21sh23:26:283
1682991912,4cyclictest10280-21turbostat20:45:003
1682991911,6cyclictest8144-21basename20:40:003
1682991911,6cyclictest31194-21sed00:05:223
1682991911,6cyclictest16153-21sed19:40:253
168199198,9cyclictest0-21swapper/222:13:402
168199198,9cyclictest0-21swapper/222:13:402
168199197,10cyclictest24580-21gltestperf22:00:172
168199192,12cyclictest17254-21ssh21:55:122
1681991914,3cyclictest9456-21processes20:40:202
1681991914,3cyclictest14001-21sh00:16:312
1681991914,3cyclictest11461-21sensors_temp19:30:232
1681991914,3cyclictest1085-21sh00:30:532
1681991911,6cyclictest307-21basename20:20:182
1681991911,6cyclictest10985-21date21:25:272
1681991910,6cyclictest20235-21grep19:50:202
1681991910,6cyclictest20074-21memory21:05:212
167999199,7cyclictest11292-21memory20:45:200
167999192,12cyclictest5674-21ssh00:35:130
167999192,11cyclictest636-21runrttasks21:45:120
1679991913,4cyclictest689-21sh23:42:480
1679991913,4cyclictest689-21sh23:42:480
1679991913,4cyclictest15435-21sh21:29:480
1679991912,5cyclictest29501-21seq20:14:350
1679991912,5cyclictest27468-21seq22:26:010
1679991911,6cyclictest22968-21hostname23:35:170
1679991911,6cyclictest19818-21cat19:50:160
1679991911,6cyclictest17759-21ls21:00:180
1679991911,6cyclictest13203-21load19:35:190
1679991910,7cyclictest16190-21date19:40:250
1679991910,6cyclictest24371-21pmu-power20:00:260
1679991910,5cyclictest18439-21apt-get22:20:120
276632181,14sleep30-21swapper/320:10:143
168299186,9cyclictest34-21ksoftirqd/323:30:123
168299186,11cyclictest34-21ksoftirqd/321:25:223
168299186,10cyclictest34-21ksoftirqd/319:25:253
168299183,2cyclictest141rcu_preempt00:34:313
1682991812,4cyclictest8095-21sh23:48:303
1682991812,4cyclictest21487-21expr19:55:153
168299181,14cyclictest278-21systemd-journal20:35:003
168299181,13cyclictest24317-21ntp_states20:00:263
168199189,7cyclictest1369-21uptime22:30:272
168199182,12cyclictest21231-21apt-get19:55:142
168199182,11cyclictest967-21ssh22:55:122
168199182,11cyclictest967-21ssh22:55:122
168199182,11cyclictest22318-21apt-get23:35:122
168199181,7cyclictest11672-21grep23:50:252
1681991813,3cyclictest31124-21sh23:40:482
1681991813,3cyclictest31124-21sh23:40:472
1681991812,4cyclictest29560-21latency_hist20:15:012
1681991812,4cyclictest26928-21ls22:50:142
1681991812,4cyclictest26026-21sh21:13:052
1681991811,5cyclictest23943-21awk21:35:222
1681991810,6cyclictest15271-21fw_packets20:55:172
167999189,6cyclictest278-21systemd-journal23:30:000
167999182,12cyclictest30277-21apt-get00:05:150
167999181,8cyclictest18314-21ntp_states21:55:210
1679991812,4cyclictest6030-21sh23:46:230
1679991811,5cyclictest9309-21ntpq20:40:180
1679991811,5cyclictest8723-21ntpq23:00:180
1679991810,6cyclictest26279-21readlink22:50:010
1679991810,6cyclictest25971-21perl20:05:230
1679991810,6cyclictest25473-21cat22:00:250
1679991810,6cyclictest21156-21cat22:45:240
1679991810,6cyclictest14817-21grep20:55:140
1679991810,6cyclictest14615-21date20:55:010
1679991810,6cyclictest11995-21expr22:15:150
1679991810,5cyclictest8717-21sed19:25:160
125832180,2sleep00-21swapper/000:15:250
69722170,2sleep00-21swapper/020:35:220
168299179,6cyclictest482-21ls20:20:203
168299177,6cyclictest0-21swapper/321:50:113
168299176,8cyclictest278-21systemd-journal20:20:013
1682991711,4cyclictest11602-21sed19:30:243
1682991710,5cyclictest12876-21cut19:35:163
168199179,6cyclictest26283-21ls20:05:262
168199179,6cyclictest12799-21expr20:50:132
168199178,7cyclictest3120-21cat20:25:252
1681991710,5cyclictest17420-21fw_packets19:45:152
167999178,6cyclictest7126-21awk19:20:230
1679991712,3cyclictest21206-21/usr/sbin/munin19:55:210
168299169,5cyclictest17104-21cut19:45:133
1682991611,3cyclictest6003-21/usr/sbin/munin19:20:243
168299161,10cyclictest17177-21apt-get21:00:143
168199169,5cyclictest31812-21seq20:19:052
167999169,5cyclictest1061-21seq20:20:250
168299159,4cyclictest19726-21cat19:50:153
168299157,6cyclictest13510-21grep20:50:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional