You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 00:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Mon Mar 23, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
102872147135,7sleep30-21swapper/307:09:173
102762141104,26sleep20-21swapper/207:09:072
100912141104,26sleep10-21swapper/107:06:421
1067421350,3sleep20-21swapper/209:19:402
101452135100,24sleep00-21swapper/007:07:260
14082690,2sleep00-21swapper/011:13:400
280032650,2sleep30-21swapper/311:09:323
180362570,1sleep30-21swapper/309:49:103
247842550,2sleep30-21swapper/311:55:123
170952550,9sleep334-21ksoftirqd/312:13:223
316362530,1sleep10-21swapper/111:11:281
217982530,1sleep10-21swapper/108:50:141
71192510,4sleep00-21swapper/009:16:050
325112500,1sleep20-21swapper/212:25:012
252932490,1chrt0-21swapper/309:30:133
146542470,2sleep10-21swapper/111:23:311
203902450,1sleep20-21swapper/209:25:422
87472440,2sleep20-21swapper/211:19:192
14432430,2sleep10-21swapper/109:12:091
243542410,2sleep00-21swapper/009:29:460
87692370,1sleep30-21swapper/310:05:313
237782370,2sleep00-21swapper/010:41:080
165752350,2sleep30-21swapper/311:00:183
135782340,1sleep10-21swapper/110:58:191
107572330,4sleep3331rcuc/312:32:243
310992300,4sleep3331rcuc/309:58:503
1082199301,27cyclictest10365-21apt-get07:10:031
10821992920,3cyclictest12302-21ssh12:33:571
183092280,2sleep20-21swapper/207:25:202
10823992811,10cyclictest2505-21ssh10:00:553
10823992811,10cyclictest2505-21ssh10:00:553
81382270,2sleep10-21swapper/109:17:071
10823992610,10cyclictest8103-21ssh09:17:053
10822992613,7cyclictest7054-21apt-get10:05:152
275962250,2sleep00-21swapper/012:20:240
26862250,3sleep122-21ksoftirqd/110:25:251
10823992523,1cyclictest0-21swapper/307:10:053
10823992517,6cyclictest23474-21ssh11:54:333
10823992514,7cyclictest25854-21apt-get12:20:103
10821992516,4cyclictest11278-1kworker/u9:112:36:141
10820992517,6cyclictest31158-21ssh09:10:170
75242240,2sleep20-21swapper/208:15:192
213602240,2sleep00-21swapper/011:52:240
115932240,2sleep00-21swapper/012:09:340
10822992422,1cyclictest0-21swapper/207:10:082
1082199241,11cyclictest174932sh09:24:341
226702230,2sleep00-21swapper/011:29:450
172752230,6sleep3331rcuc/308:40:013
162592230,2sleep00-21swapper/009:47:220
1082399232,6cyclictest34-21ksoftirqd/311:30:443
1082399232,6cyclictest34-21ksoftirqd/311:30:433
10823992316,5cyclictest560-21dbus-daemon09:55:003
1082399231,5cyclictest16122-21apt-get11:25:033
1082399231,10cyclictest22012rm10:50:083
1082299233,7cyclictest31314-21ssh12:00:042
1082199237,9cyclictest22-21ksoftirqd/109:28:411
10821992313,7cyclictest27556-21users12:20:241
10820992313,8cyclictest4058-21sshd12:27:250
93032220,1sleep00-21swapper/012:30:570
6642220,2sleep30-21swapper/311:36:503
243242220,1sleep00-21swapper/011:05:450
1082399227,3cyclictest141rcu_preempt07:35:003
10823992218,3cyclictest52950irq/141-enp3s0-10:35:243
1082399221,5cyclictest34-21ksoftirqd/308:55:013
10823992214,5cyclictest6835-21memory09:40:183
10823992213,7cyclictest27797-21cut10:45:013
10823992213,7cyclictest12066-21which08:25:223
1082299226,3cyclictest141rcu_preempt10:52:392
1082299221,6cyclictest12722-21ntp_states09:20:212
10822992214,6cyclictest6952-21ssh11:41:252
10822992214,6cyclictest14325-21ssh11:23:122
10822992213,7cyclictest1132-21ntp_states08:00:232
10821992213,7cyclictest1980-21unixbench_singl10:00:261
10821992213,7cyclictest1980-21unixbench_singl10:00:251
10821992213,6cyclictest2663-21diskstats10:50:141
1082199221,18cyclictest52750irq/139-enp3s0-12:00:131
1082099222,17cyclictest6145-21tr09:40:130
107152220,4sleep3331rcuc/311:45:013
1082399216,6cyclictest34-21ksoftirqd/309:13:183
1082299217,12cyclictest16387-21ssh11:00:162
1082299215,3cyclictest47582ssh10:27:082
1082299214,4cyclictest29982-21diskmemload10:33:042
1082299211,4cyclictest28374-21diskstats10:45:132
10822992113,6cyclictest26522-21sed09:00:172
10822992112,7cyclictest8794-21apt-get08:20:112
10822992112,7cyclictest7025-21ntp_states09:40:212
10822992112,6cyclictest18304-21ntp_states08:40:202
10822992111,6cyclictest17726-21ssh11:50:152
10822992110,9cyclictest560-21dbus-daemon08:55:012
1082199215,14cyclictest19953-21turbostat10:15:001
1082199214,3cyclictest141rcu_preempt10:35:151
10821992115,4cyclictest22-21ksoftirqd/112:00:011
10821992114,4cyclictest13869-21kernelversion07:15:181
10821992113,6cyclictest22354-21ntp_kernel_pll_10:40:201
1082099216,3cyclictest143792rm12:10:370
1082099215,2cyclictest141rcu_preempt07:11:050
10820992115,4cyclictest7024-21/usr/sbin/munin10:05:210
10820992113,6cyclictest16104-21sed10:35:260
10820992112,7cyclictest27968-21/usr/sbin/munin10:45:130
10820992112,6cyclictest24611-21ls11:55:120
1082099211,16cyclictest53050irq/142-enp3s0-12:15:130
1082099211,16cyclictest53050irq/142-enp3s0-12:15:130
1082099210,5cyclictest0-21swapper/011:45:520
78472200,3sleep17863-21ssh11:42:181
1082399208,4cyclictest29982-21diskmemload11:47:453
1082399206,12cyclictest28035-21ls10:20:203
1082399206,11cyclictest9030-21cpu10:55:123
1082399204,6cyclictest34-21ksoftirqd/308:00:163
1082399202,7cyclictest28341-21apt-get07:50:113
1082399202,12cyclictest17520-21apt-get07:25:133
10823992015,3cyclictest5881-21tr12:05:113
10823992014,4cyclictest34-21ksoftirqd/312:20:003
10823992014,4cyclictest34-21ksoftirqd/312:19:593
10823992012,6cyclictest21067-21if_enp3s010:15:173
10823992011,6cyclictest879-21fw_packets12:25:153
10823992011,6cyclictest2156-21latency10:25:203
10823992011,6cyclictest2086-21sed10:50:013
10823992010,7cyclictest3590-21cat08:05:253
1082299209,8cyclictest7388-21sed12:05:242
1082299209,6cyclictest30922-21ssh11:35:192
1082299204,3cyclictest29982-21diskmemload10:56:382
1082299203,3cyclictest29982-21diskmemload09:39:172
1082299202,12cyclictest7056-21ssh12:30:112
10822992013,4cyclictest31359-21ntp_kernel_err07:55:202
10822992012,6cyclictest27089-21ls09:55:192
10822992012,5cyclictest28-21ksoftirqd/210:40:122
1082199208,4cyclictest29982-21diskmemload12:25:371
10821992016,2cyclictest0-21swapper/110:09:071
10821992015,4cyclictest22-21ksoftirqd/108:29:591
10821992014,4cyclictest25293-21sh11:30:381
10821992014,4cyclictest25293-21sh11:30:381
10821992013,4cyclictest14320-21cron10:35:001
10821992012,6cyclictest31202-21seq10:22:501
10821992012,6cyclictest20750-21ls12:15:261
10821992012,6cyclictest20750-21ls12:15:261
10821992012,6cyclictest16756-21unixbench-2d07:20:241
10821992012,6cyclictest15982-21memory08:35:191
10821992012,5cyclictest52750irq/139-enp3s0-11:16:491
10821992012,5cyclictest25131-21grep09:30:121
10821992012,5cyclictest13048-21seq08:30:071
10821992010,7cyclictest12312-21sensors_temp11:45:241
10821992010,6cyclictest31737-21apt-get09:35:121
1082099208,5cyclictest12-21ksoftirqd/011:39:550
1082099201,4cyclictest20365-21apt-get10:15:110
10820992013,4cyclictest29982-21diskmemload09:32:260
10820992012,6cyclictest17974-21basename09:24:590
10820992012,5cyclictest24344-21netstat11:30:200
10820992012,5cyclictest24344-21netstat11:30:200
10820992011,6cyclictest12293-21sed08:25:240
10820992010,7cyclictest278-21systemd-journal11:19:540
1082399199,7cyclictest6962-21df_inode08:15:143
10823991914,3cyclictest25685-21latency_hist09:00:013
10823991914,3cyclictest18985-21sh10:14:023
10823991913,3cyclictest31877-21tr09:35:123
10823991912,5cyclictest9389-21sed10:30:213
10823991911,6cyclictest10703-21basename08:24:593
10823991911,5cyclictest52950irq/141-enp3s0-12:01:203
10823991911,5cyclictest18103-21ls08:40:183
10823991910,6cyclictest31848-21cat07:55:253
10823991910,6cyclictest19272-21sed09:25:193
10823991910,5cyclictest52950irq/141-enp3s0-11:15:193
1082299193,3cyclictest29982-21diskmemload10:01:102
1082299193,3cyclictest29982-21diskmemload10:01:092
1082299193,3cyclictest29982-21diskmemload09:45:362
1082299192,3cyclictest29982-21diskmemload12:38:332
1082299191,5cyclictest32149-21apt-get12:25:022
10822991914,3cyclictest0-21swapper/211:10:112
10822991913,4cyclictest18950irq/143-ahci[0012:20:002
10822991913,4cyclictest18950irq/143-ahci[0012:20:002
10822991912,5cyclictest21917-21wc10:15:242
10822991911,5cyclictest52850irq/140-enp3s0-09:30:162
10822991910,7cyclictest560-21dbus-daemon11:05:242
10822991910,6cyclictest52850irq/140-enp3s0-09:50:132
10822991910,6cyclictest19725-21cat07:30:142
10821991914,3cyclictest28437-21sh11:09:591
10821991914,3cyclictest18545-21sh11:25:301
10821991914,3cyclictest17317-21unixbench_singl11:00:251
10821991913,4cyclictest6599-21sh09:40:161
10821991913,4cyclictest5586-21sh12:05:011
10821991913,4cyclictest30136-21sh10:45:471
10821991913,4cyclictest27219-21mailstats09:55:211
10821991913,4cyclictest20595-21open_files07:30:221
10821991912,5cyclictest778-21smartctl08:00:161
10821991912,5cyclictest18286-21ntpq08:40:201
10821991912,5cyclictest14084-21cat09:45:241
10821991911,6cyclictest29164-21ls07:50:181
10821991911,6cyclictest23106-21ls07:35:251
10821991911,6cyclictest20383-21open_inodes09:50:211
10821991911,6cyclictest19179-21cat11:50:271
10821991910,7cyclictest13857-21ls12:10:231
1082099199,6cyclictest53050irq/142-enp3s0-10:25:130
1082099192,14cyclictest16615-21needreboot11:00:190
10820991914,3cyclictest9885-21grep08:20:210
10820991914,3cyclictest32691-21/usr/sbin/munin10:00:220
10820991914,3cyclictest32691-21/usr/sbin/munin10:00:220
10820991913,4cyclictest12-21ksoftirqd/009:59:590
10820991911,5cyclictest29967-21cron09:10:000
10820991911,5cyclictest23025-21sed07:35:250
1082099191,15cyclictest15091-21ssh10:10:240
10820991910,7cyclictest5302-21ntp_states11:40:210
10820991910,6cyclictest550-21systemd-logind10:34:220
10820991910,6cyclictest31711-21acpi09:35:110
10823991813,3cyclictest28700-21sh11:10:023
10823991812,4cyclictest13382-21sh11:22:153
10823991811,5cyclictest6502-21date08:14:593
10823991811,5cyclictest28359-21expr09:05:143
10823991810,5cyclictest26466-21expr07:45:133
10823991810,5cyclictest21912-21grep07:35:143
10823991810,5cyclictest13486-21cat07:15:143
1082299182,11cyclictest13218-21apt-get07:15:132
1082299181,8cyclictest31126-21ssh09:10:172
10822991813,3cyclictest17178-21sh10:12:142
10822991812,4cyclictest25679-21sh11:55:212
10822991812,4cyclictest12978-21sed08:30:012
10822991811,5cyclictest16064-21python310:35:262
1082299181,15cyclictest15915-21cut07:20:152
10822991811,4cyclictest29982-21diskmemload11:30:212
10822991811,4cyclictest29982-21diskmemload11:30:212
10822991811,4cyclictest29982-21diskmemload11:25:132
10822991810,6cyclictest24840-21sed07:40:182
10822991810,5cyclictest5273-21sed08:10:182
10821991813,3cyclictest32636-21sh11:36:221
10821991812,3cyclictest29982-21diskmemload10:15:061
10821991811,5cyclictest8958-21cat08:20:131
10821991811,5cyclictest31366-21ntp_kernel_pll_07:55:201
10821991811,5cyclictest20841-21ls08:45:251
1082099189,6cyclictest53050irq/142-enp3s0-12:35:120
1082099182,11cyclictest19520-21apt-get08:45:130
10820991813,3cyclictest19261-21/usr/sbin/munin09:50:240
10820991812,4cyclictest12-21ksoftirqd/007:30:000
10820991812,3cyclictest7612-21sed08:15:220
10820991812,3cyclictest29982-21diskmemload12:01:200
10820991811,5cyclictest14137-21ntpq07:15:210
1082099181,12cyclictest27096-21apt-get10:20:130
10820991810,6cyclictest9112-21grep10:55:130
10820991810,6cyclictest31656-21grep07:55:230
10820991810,6cyclictest2739-21tune2fs10:50:150
10820991810,5cyclictest17808-21cut08:40:150
1082399176,3cyclictest29982-21diskmemload12:35:523
10823991712,3cyclictest25930-21/usr/sbin/munin09:00:143
10823991711,4cyclictest25227-21swap07:40:243
10823991711,4cyclictest12818-21sh09:20:223
10823991710,5cyclictest13964-21ntpq08:30:203
1082299179,6cyclictest12970-21fw_packets12:10:142
1082299173,4cyclictest28-21ksoftirqd/208:35:222
10822991712,3cyclictest23769-21/usr/sbin/munin08:55:122
10822991711,4cyclictest20306-21missed_timers08:45:192
10822991710,5cyclictest28-21ksoftirqd/210:23:102
1082199179,6cyclictest17682-21cat07:25:131
10821991712,3cyclictest28069-21/usr/sbin/munin09:05:121
10821991710,5cyclictest6780-21expr08:15:131
10821991710,5cyclictest4578-21grep08:10:111
10821991710,5cyclictest3179-21cat08:05:211
10821991710,5cyclictest27930-21seq07:48:091
10821991710,5cyclictest26589-21cat09:00:161
10821991710,5cyclictest24374-21idleruntime08:55:181
1082099179,6cyclictest25792-21sed09:00:000
1082099178,6cyclictest4491-21apt08:10:120
1082099175,7cyclictest53050irq/142-enp3s0-11:20:120
1082099172,11cyclictest2360-21apt-get08:05:140
10820991713,2cyclictest25930-21/usr/sbin/munin09:00:150
10820991711,4cyclictest28220-21sed07:50:010
10820991710,5cyclictest15706-21grep08:35:170
10820991710,5cyclictest12-21ksoftirqd/008:34:590
10820991710,5cyclictest1199-21head08:00:240
1082399169,5cyclictest10806-21gdbus07:20:063
10823991612,3cyclictest19872-21fschecks_count08:45:153
1082299166,7cyclictest27071-21netstat07:45:202
1082299161,6cyclictest28069-21/usr/sbin/munin09:05:132
1082299161,6cyclictest12485-21cat11:45:252
1082299161,13cyclictest4234-21cron08:10:002
10822991610,4cyclictest636-21runrttasks07:54:152
1082199161,13cyclictest25959-21munin-run07:45:001
1082099168,6cyclictest29075-21sed07:50:180
1082099163,8cyclictest636-21runrttasks07:30:120
10820991611,3cyclictest17271-21latency_hist07:24:590
10820991610,4cyclictest636-21runrttasks08:50:110
10822991510,3cyclictest21851-21cpuspeed_turbos07:35:142
1082299150,6cyclictest13620-21sed08:30:142
10820991511,2cyclictest24030-21/usr/sbin/munin07:40:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional