You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-14 - 04:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Jun 14, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13882169149,14sleep20-21swapper/219:07:052
1330214897,25sleep10-21swapper/119:06:241
14542144107,26sleep30-21swapper/319:07:543
15132137101,25sleep00-21swapper/019:08:400
304162710,2sleep10-21swapper/122:55:001
114202620,2sleep00-21swapper/023:54:140
27152590,14sleep30-21swapper/320:25:183
27662550,1sleep20-21swapper/222:34:102
190262550,8sleep012-21ksoftirqd/021:32:270
194112540,3sleep30-21swapper/321:05:143
168352540,2sleep20-21swapper/222:20:142
284512530,6sleep128453-21fschecks_count20:10:141
246542530,1sleep20-21swapper/222:25:232
198572500,1sleep30-21swapper/322:46:033
282382490,1sleep30-21swapper/322:04:383
192802460,2sleep00-21swapper/022:45:280
154672450,1sleep00-21swapper/000:20:340
75162440,1sleep20-21swapper/223:25:382
21612440,2sleep20-21swapper/219:10:122
189099417,16cyclictest5950-21rm00:38:432
142282380,1sleep30-21swapper/323:06:223
142282380,1sleep30-21swapper/323:06:223
189099378,12cyclictest16717-21rm23:08:572
189099378,12cyclictest16717-21rm23:08:562
35542350,2chrt0-21swapper/000:11:550
231822340,1sleep00-21swapper/000:02:330
189099348,9cyclictest21150-21diskmemload21:52:052
189099346,14cyclictest0-21swapper/200:24:402
189099345,13cyclictest28497-21kworker/u8:121:20:202
189099344,18cyclictest21115-21ls00:25:192
228192330,2chrt0-21swapper/123:13:291
189099337,16cyclictest1179-21ssh22:08:242
189099337,11cyclictest5575-21rm00:14:022
189099336,8cyclictest0-21swapper/221:57:592
189099336,11cyclictest21424-21sh23:36:302
189099336,10cyclictest5443-21grep22:35:242
189099334,9cyclictest0-21swapper/200:35:012
162662330,2chrt0-21swapper/321:30:173
189099329,9cyclictest1161-21sh22:56:292
189099329,8cyclictest21150-21diskmemload23:44:512
189099327,14cyclictest0-21swapper/223:03:192
189099327,11cyclictest11793-21sh00:18:382
189099326,9cyclictest15288-21ssh22:43:232
189099325,10cyclictest21150-21diskmemload21:10:452
189099324,19cyclictest0-21swapper/223:45:272
189099324,15cyclictest0-21swapper/222:50:012
189099318,11cyclictest29569-21sh22:54:102
189099317,10cyclictest21150-21diskmemload23:57:592
189099315,9cyclictest28950-21ssh23:18:032
189099315,9cyclictest21150-21diskmemload23:51:452
189099314,17cyclictest0-21swapper/200:30:592
134732310,2sleep30-21swapper/322:17:143
20392300,1sleep10-21swapper/123:21:451
189099307,9cyclictest26713-21sh21:38:422
189099306,9cyclictest0-21swapper/223:34:492
189099305,8cyclictest7801-21sh22:13:122
189099304,9cyclictest25918-21sh22:02:152
189099304,11cyclictest31084-21sh21:41:272
35582290,1sleep00-21swapper/021:45:200
189099298,8cyclictest11353-21sh21:26:152
189099295,10cyclictest0-21swapper/221:16:262
189099294,9cyclictest27074-21timerwakeupswit20:05:242
189099294,13cyclictest14391-21ssh22:18:102
189099293,10cyclictest21150-21diskmemload21:30:522
144142290,6sleep3141rcu_preempt22:42:303
189099287,7cyclictest4548-21ssh21:45:462
189099285,8cyclictest21150-21diskmemload23:10:132
189099282,9cyclictest28010-21ssh00:05:432
189099275,7cyclictest13950-21mailstats20:50:242
189099274,8cyclictest20818-21ls00:00:242
1889992719,6cyclictest803-21diskstats00:10:181
1888992713,9cyclictest640-21snmpd21:15:140
188899271,24cyclictest0-21swapper/019:11:070
68552260,2sleep10-21swapper/123:00:301
189099265,8cyclictest32321-21memory23:20:192
1890992610,13cyclictest12604-21cron19:35:012
188999261,24cyclictest0-21swapper/119:11:331
189199251,23cyclictest0-21swapper/319:11:463
1890992512,10cyclictest1868-21basename20:25:012
188999251,17cyclictest28399-21chrt21:40:121
306852240,3chrt0-21swapper/000:32:390
306852240,3chrt0-21swapper/000:32:380
188999241,14cyclictest27685-21cron21:15:011
188899241,9cyclictest27682rm22:34:090
271902230,2sleep10-21swapper/123:16:151
21512230,2sleep10-21swapper/122:57:281
1891992314,7cyclictest24518-21ntp_states20:00:203
189199231,11cyclictest18817-21latency_hist00:00:013
189199231,10cyclictest222772needreboot19:55:193
189199231,10cyclictest11792rm23:20:523
189099239,9cyclictest6241-21chrt19:20:122
189099235,12cyclictest0-21swapper/220:25:132
1890992311,9cyclictest7858-21unixbench_singl20:35:242
1890992311,9cyclictest12565-21cron20:49:592
1890992311,8cyclictest12842-21/usr/sbin/munin19:35:282
1889992318,4cyclictest7801-21ssh21:24:231
1889992311,3cyclictest21595-21sh23:36:411
188899231,7cyclictest21150-21diskmemload22:55:260
1888992317,4cyclictest28634-21ls22:05:000
1888992310,9cyclictest9063-21/usr/sbin/munin21:50:180
148002230,1sleep00-21swapper/022:18:350
116122230,1sleep30-21swapper/320:45:203
280292220,3sleep10-21swapper/123:41:301
278522221,7sleep327894-21ntpq00:30:223
278522221,7sleep327894-21ntpq00:30:223
189199222,14cyclictest15688-21ssh21:55:153
1891992214,6cyclictest20442-21ntp_states00:00:213
1891992213,6cyclictest8660-21users00:15:263
189199221,10cyclictest14649-21ntp_states00:20:213
1890992212,7cyclictest9100-21cut20:40:172
1890992212,7cyclictest29319-21find20:10:242
1890992211,8cyclictest17267-21cpuspeed_turbos19:45:142
1890992210,9cyclictest24838-21xargs20:00:232
1890992210,9cyclictest22433-21pmu-power19:55:222
1890992210,9cyclictest19476-21cstates19:50:152
1889992215,4cyclictest22415-21/usr/sbin/munin22:00:191
1889992210,9cyclictest3343-21ssh22:10:091
188899221,4cyclictest27697-21munin-run21:15:000
1888992212,7cyclictest278-21systemd-journal23:45:000
112382220,1sleep30-21swapper/321:50:533
1891992112,6cyclictest911-21pluginstate20:20:223
1891992112,6cyclictest1507-21tune2fs00:35:163
1891992111,7cyclictest30815-21cron23:20:003
1891992110,6cyclictest52850irq/140-enp3s0-22:20:193
189099219,9cyclictest19929-21sed21:05:182
189099219,9cyclictest15261-21expr20:55:132
189099219,8cyclictest0-21swapper/220:30:112
189099218,10cyclictest30527-21cstates20:15:122
1890992113,5cyclictest17633-21tr21:00:172
1890992111,7cyclictest8937-21grep19:25:142
1890992111,7cyclictest4542-21gltestperf19:15:152
1889992116,3cyclictest190182sleep100:00:011
1889992113,6cyclictest21040-21seq21:08:481
1889992113,6cyclictest18855-21seq21:03:111
188899212,13cyclictest32382-21apt-get20:20:140
188899212,13cyclictest31492-21ssh23:20:130
103102210,2chrt12-21ksoftirqd/020:43:330
189199209,7cyclictest12830-21apt-get20:50:133
1891992012,6cyclictest9257-21sed20:40:183
1891992012,6cyclictest819-21ls23:45:243
1891992012,6cyclictest7584-21ls19:20:243
1891992012,6cyclictest2903-21sed21:20:153
189099209,8cyclictest15025-21apt-get19:40:112
188999206,12cyclictest18147-21sh22:45:151
1889992014,4cyclictest6236-21apt19:20:101
1889992014,4cyclictest11798-21sh22:15:301
1889992013,5cyclictest30459-21expr22:30:171
1889992012,6cyclictest31447-21seq21:17:071
1889992012,6cyclictest16018-21awk19:40:211
1889992011,6cyclictest2315-21cat20:25:141
188899209,9cyclictest15872-21cstates21:55:160
188899209,8cyclictest560-21dbus-daemon00:00:000
188899208,9cyclictest18244-21dump-pmu-power23:35:000
188899203,13cyclictest28386-21apt-get21:40:110
188899202,12cyclictest8583-21apt-get19:25:110
1888992013,5cyclictest17238-21grep22:20:150
1888992013,4cyclictest28996-21/usr/sbin/munin22:05:260
1888992011,6cyclictest560-21dbus-daemon21:30:000
1888992010,8cyclictest23337-21systemctl19:59:590
1888992010,7cyclictest4001-21basename20:30:010
85492190,2sleep10-21swapper/121:49:551
302682190,2chrt0-21swapper/120:15:131
255292190,3chrt0-21swapper/323:40:133
189199195,5cyclictest34-21ksoftirqd/322:40:013
1891991914,3cyclictest6222-21proc_pri23:00:223
1891991914,3cyclictest11991-21sh21:26:563
1891991913,4cyclictest6370-21sh23:50:133
1891991912,5cyclictest27721-21awk00:05:273
1891991912,4cyclictest21150-21diskmemload21:45:013
1891991911,6cyclictest4572-21df20:30:123
1891991911,6cyclictest30047-21ntpq22:05:203
1891991911,6cyclictest19710-21ntpq23:35:213
189199191,16cyclictest18061-21sh23:34:493
1891991911,6cyclictest1721-21run-parts00:10:263
1891991911,6cyclictest10789-21cat19:30:143
1891991910,7cyclictest278-21systemd-journal19:30:003
1891991910,6cyclictest29408-21ntp_kernel_pll_21:40:193
1891991910,5cyclictest52850irq/140-enp3s0-23:10:103
188999199,8cyclictest13802-21cut20:50:231
188999199,3cyclictest141rcu_preempt00:00:151
188999198,7cyclictest52750irq/139-enp3s0-00:20:131
188999192,12cyclictest6008-21apt-get23:50:121
188999192,12cyclictest25736-21apt-get20:05:131
188999191,7cyclictest27428-21hddtemp_smartct00:30:151
188999191,7cyclictest27428-21hddtemp_smartct00:30:141
188999191,7cyclictest1419-21seq00:35:151
1889991914,3cyclictest30916-21sh22:05:361
1889991913,4cyclictest7007-21sh22:36:441
1889991913,4cyclictest10344-21sh21:50:241
1889991912,5cyclictest18780-21seq19:46:431
1889991912,5cyclictest10947-21expr22:40:141
1889991911,6cyclictest6814-21seq00:15:041
1889991911,6cyclictest23817-21expr20:00:111
1889991911,6cyclictest22388-21seq19:55:211
1889991911,6cyclictest20872-21cat00:25:161
1889991911,6cyclictest17304-21smartctl22:20:171
1889991911,6cyclictest15549-21expr21:30:131
1889991911,6cyclictest13023-21cat19:35:141
1889991911,6cyclictest12566-21cron20:50:001
1889991911,5cyclictest12086-21acpi23:30:111
188999191,13cyclictest15695-21sh21:55:141
1889991910,7cyclictest5620-21date20:30:241
1889991910,7cyclictest23118-21tr21:35:221
188899198,8cyclictest278-21systemd-journal21:24:590
188899198,8cyclictest14724-21ls20:54:590
188899193,12cyclictest3235-21apt-get22:10:020
188899193,10cyclictest640-21snmpd19:30:130
188899191,6cyclictest1086-21grep23:45:260
1888991914,3cyclictest5097-21processes19:15:200
1888991913,4cyclictest12-21ksoftirqd/000:10:010
1888991913,3cyclictest27615-21sh21:39:390
1888991913,3cyclictest1882-21missed_timers00:35:190
1888991912,5cyclictest5044-21awk23:00:110
1888991912,5cyclictest24164-21wc22:25:190
1888991912,5cyclictest14009-21tr19:35:230
1888991911,5cyclictest278-21systemd-journal22:40:010
188899191,15cyclictest25507-21ssh23:15:180
1888991911,5cyclictest21150-21diskmemload23:07:050
1888991911,5cyclictest21150-21diskmemload23:07:050
1888991910,6cyclictest6912-21cut19:20:170
1888991910,6cyclictest24806-21cat20:00:230
204852180,1sleep00-21swapper/023:11:040
189199189,6cyclictest28107-21cat21:15:133
189199189,6cyclictest25851-21grep20:05:143
189199187,7cyclictest52850irq/140-enp3s0-22:50:143
189199185,4cyclictest141rcu_preempt21:10:153
1891991813,3cyclictest24977-21uptime22:25:253
1891991813,3cyclictest2331-21sh22:57:403
1891991812,4cyclictest26089-21sh21:38:043
1891991811,5cyclictest15709-21date20:55:193
1891991810,6cyclictest8508-21cut20:40:013
1891991810,6cyclictest21095-21cat00:25:193
1891991810,6cyclictest17832-21cat21:00:183
1891991810,6cyclictest15170-21grep19:40:133
188999189,6cyclictest32725-21cat20:20:151
188999182,11cyclictest4072-21apt-get19:15:121
188999182,11cyclictest14965-21apt-get20:55:131
1889991813,3cyclictest8722-21sh21:25:071
1889991813,3cyclictest8273-21latency_hist20:40:001
1889991812,3cyclictest32256-21grep23:45:161
1889991811,5cyclictest19606-21ls19:50:151
1889991810,6cyclictest8934-21awk19:25:141
1889991810,6cyclictest5582-21basename23:25:111
1889991810,6cyclictest26279-21cat00:05:141
188899189,6cyclictest17163-21apt-get19:45:110
188899181,7cyclictest11640-21ssh22:40:200
188899181,5cyclictest19313-21hddtemp_smartct23:35:180
1888991812,3cyclictest21150-21diskmemload23:28:470
1888991811,5cyclictest26016-21expr20:05:140
1888991811,4cyclictest6761-21sh00:15:010
1888991810,6cyclictest5695-21awk20:30:250
1888991810,6cyclictest30069-21wc20:15:010
1888991810,4cyclictest21150-21diskmemload00:26:220
189199179,6cyclictest5176-21tr19:15:213
189199179,6cyclictest30781-21sed22:30:203
189199179,6cyclictest17626-21fw_packets19:45:173
189199178,6cyclictest30803-21basename20:15:163
1891991712,3cyclictest6115-21gltestperf23:25:163
1891991710,5cyclictest3325-21basename22:10:093
1891991710,5cyclictest13973-21tr19:35:233
188999179,6cyclictest10121-21seq20:41:081
188999171,11cyclictest23285-21ssh22:25:111
188999171,11cyclictest11625-21ssh23:05:121
188999171,11cyclictest11625-21ssh23:05:111
1889991710,5cyclictest10795-21cat19:30:131
188899179,6cyclictest20015-21latency19:50:200
188899178,7cyclictest32248-21sed20:20:010
1888991711,3cyclictest15531-21gltestperf19:40:160
1888991710,5cyclictest6369-21find20:35:110
1888991710,5cyclictest16536-21seq20:56:080
189199165,9cyclictest560-21dbus-daemon20:15:013
189199161,6cyclictest19278-21/usr/sbin/munin19:50:133
188899168,5cyclictest52950irq/141-enp3s0-22:50:220
1888991611,3cyclictest636-21runrttasks21:03:340
188899161,11cyclictest21153-21grep21:10:010
1888991510,3cyclictest10858-21cstates20:45:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional