You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-24 - 11:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jul 24, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2467029581,9sleep20-21swapper/219:06:472
2472028774,9sleep00-21swapper/019:07:270
2469128457,9sleep30-21swapper/319:07:043
2465928265,13sleep10-21swapper/119:06:381
2500899290,28cyclictest0-21swapper/223:12:032
103162290,0sleep00-21swapper/021:43:400
25008992827,1cyclictest0-21swapper/223:16:022
2500899280,27cyclictest0-21swapper/223:00:502
25008992717,9cyclictest17579-21diskmemload22:15:222
2500899267,14cyclictest0-21swapper/223:25:162
2500899265,20cyclictest0-21swapper/222:20:142
2500899234,18cyclictest0-21swapper/221:12:192
25008992315,6cyclictest31567-21grep00:10:252
2500899230,23cyclictest0-21swapper/222:02:102
2500899230,16cyclictest0-21swapper/200:35:142
2500899226,15cyclictest0-21swapper/221:17:402
25008992219,2cyclictest17579-21diskmemload23:42:192
2500299220,1cyclictest0-21swapper/023:55:130
2501099217,13cyclictest11264-21apache200:10:013
2500899217,11cyclictest0-21swapper/221:45:012
2500899216,15cyclictest0-21swapper/222:33:362
25008992020,0cyclictest0-21swapper/223:32:462
25008992019,1cyclictest55650irq/127-enp2s0-23:58:512
25008992013,6cyclictest0-21swapper/200:08:582
25008992012,7cyclictest17579-21diskmemload21:26:192
2500499201,17cyclictest24371-21cstates20:15:141
2500299208,6cyclictest9451-21perf19:45:000
2500299202,3cyclictest0-21swapper/022:05:590
2500899199,9cyclictest17579-21diskmemload23:21:192
2500899199,9cyclictest0-21swapper/221:54:482
25008991915,3cyclictest19381-21kworker/u8:122:48:512
25008991912,5cyclictest8866-21fschecks_count20:50:162
25004991919,0cyclictest0-21swapper/123:28:141
25010991818,0cyclictest0-21swapper/321:34:523
2501099180,2cyclictest0-21swapper/321:00:143
2500899185,13cyclictest0-21swapper/200:16:412
2500899183,12cyclictest0-21swapper/221:43:232
25008991817,1cyclictest0-21swapper/222:10:052
25008991816,2cyclictest0-21swapper/223:06:042
25008991814,3cyclictest30137-21pmu-power23:50:232
2500899181,16cyclictest29075-21turbostat.cron21:59:542
25008991810,8cyclictest0-21swapper/222:54:262
2500499182,4cyclictest225-21systemd-journal22:37:181
25004991816,1cyclictest0-21swapper/119:25:161
2500499180,4cyclictest0-21swapper/123:05:521
2500499180,18cyclictest0-21swapper/122:17:231
2500299188,3cyclictest13938-21sendmail-msp00:20:000
25002991812,5cyclictest0-21swapper/022:56:150
2501099173,9cyclictest10470-21/usr/sbin/munin23:00:203
25010991714,2cyclictest39-21ksoftirqd/300:17:393
2500899178,7cyclictest22339-21fw_packets20:10:162
25008991715,1cyclictest17579-21diskmemload22:26:492
25008991714,2cyclictest17579-21diskmemload22:39:422
25008991714,2cyclictest0-21swapper/200:02:452
25008991711,5cyclictest0-21swapper/200:22:512
25008991711,4cyclictest0-21swapper/222:55:162
2500899170,15cyclictest0-21swapper/223:45:182
2500499172,13cyclictest6686-21hddtemp_smartct20:45:211
25004991717,0cyclictest0-21swapper/121:29:031
2500299177,4cyclictest0-21swapper/021:15:270
2500299175,6cyclictest19089-21munin-run21:30:000
25002991716,1cyclictest0-21swapper/022:33:370
2500299170,1cyclictest0-21swapper/000:26:140
2501099167,9cyclictest0-21swapper/322:38:063
2500899162,10cyclictest17579-21diskmemload22:09:082
25008991615,1cyclictest0-21swapper/200:30:122
25008991614,1cyclictest17579-21diskmemload21:21:102
2500499161,14cyclictest21335-21ssh22:09:381
2500499160,16cyclictest1040-21ssh23:33:321
2500499160,15cyclictest0-21swapper/122:27:511
2500299164,4cyclictest21559-21apt-get00:05:160
2501099155,6cyclictest1142-21runrttasks22:00:043
25010991514,1cyclictest0-21swapper/323:22:553
25010991513,1cyclictest0-21swapper/322:14:523
25010991510,3cyclictest6813-21cron19:40:003
2501099150,14cyclictest0-21swapper/300:35:423
2500899158,6cyclictest0-21swapper/221:30:132
2500899156,9cyclictest0-21swapper/200:29:192
25008991512,1cyclictest17579-21diskmemload23:37:322
2500499150,13cyclictest0-21swapper/100:24:151
2500499150,0cyclictest0-21swapper/123:22:301
2500299152,2cyclictest0-21swapper/000:10:150
2500299150,1cyclictest0-21swapper/022:53:070
2500299150,14cyclictest0-21swapper/021:30:120
2501099144,3cyclictest28135-21systemd-cgroups00:08:583
2501099142,6cyclictest1-21systemd22:22:253
2501099142,10cyclictest1006-21apt-get22:55:133
25010991413,1cyclictest0-21swapper/322:28:303
25010991412,2cyclictest11188-21kworker/3:122:30:433
25010991412,1cyclictest0-21swapper/300:04:133
25010991411,2cyclictest27769-21cut23:30:143
25010991410,3cyclictest3051-21sed23:15:173
25010991410,2cyclictest20661-21grep23:45:133
2501099140,13cyclictest0-21swapper/323:14:443
2500899148,6cyclictest0-21swapper/220:40:002
2500899148,4cyclictest0-21swapper/219:35:192
2500899146,7cyclictest0-21swapper/219:10:252
2500899146,7cyclictest0-21swapper/219:10:252
2500899145,5cyclictest10340-21fschecks_time19:45:162
2500899144,6cyclictest1142-21runrttasks20:20:122
25008991410,2cyclictest14536-21grep19:55:152
25004991413,1cyclictest0-21swapper/121:33:551
25004991412,2cyclictest25-21ksoftirqd/121:40:461
2500499140,13cyclictest0-21swapper/123:19:091
2500299141,7cyclictest19018-21munin-run20:04:590
25002991410,3cyclictest23777-21tr22:10:220
25002991410,3cyclictest16335-21ls22:25:190
25002991410,2cyclictest13801-21grep21:00:170
2500299140,13cyclictest28465-21grep21:35:140
2501099139,3cyclictest5434-21tr23:55:163
2501099139,3cyclictest312-21sed20:30:253
2501099138,3cyclictest11978-21ssh21:25:173
2501099134,6cyclictest21155-21/usr/sbin/munin21:50:243
2501099133,3cyclictest225-21systemd-journal00:30:013
25010991312,1cyclictest0-21swapper/321:22:073
25010991310,2cyclictest8580-21wc19:40:223
25010991310,2cyclictest5311-21grep23:35:243
25010991310,2cyclictest32110-21cron19:25:003
25010991310,2cyclictest27367-21basename19:15:013
25010991310,2cyclictest27367-21basename19:15:003
2500899139,2cyclictest396-21sed20:30:242
2500899139,2cyclictest3801-21ls21:40:002
2500899138,4cyclictest0-21swapper/221:05:002
2500899136,7cyclictest0-21swapper/222:43:312
2500899136,6cyclictest0-21swapper/219:40:182
25008991310,2cyclictest7516-21grep20:45:282
25008991310,2cyclictest736-21grep19:25:182
2500499139,2cyclictest25-21ksoftirqd/123:44:071
2500499138,4cyclictest11246-21apache200:05:151
25004991311,2cyclictest38150irq/131-snd_hda00:17:481
25004991310,2cyclictest7471-21grep22:00:261
25004991310,2cyclictest4176-21awk23:55:001
25004991310,2cyclictest32123-21cat20:30:161
25004991310,2cyclictest31300-21cut00:30:131
25004991310,2cyclictest27335-21cron19:15:001
25004991310,2cyclictest27335-21cron19:15:001
25004991310,2cyclictest21889-21tr20:10:151
25004991310,2cyclictest20637-21grep23:45:131
25004991310,2cyclictest14150-21cut00:00:181
2500499130,13cyclictest0-21swapper/121:24:261
2500499130,0cyclictest0-21swapper/122:56:211
2500299139,3cyclictest23247-21grep20:10:260
2500299139,3cyclictest20999-21basename00:05:000
2500299139,3cyclictest17905-21ntp_kernel_pll_22:45:200
25002991310,2cyclictest8300-21grep20:50:140
25002991310,2cyclictest4647-21tr20:40:200
25002991310,2cyclictest27210-21ssh23:49:130
25002991310,2cyclictest20028-21tail20:05:170
25002991310,2cyclictest11370-21sort23:20:170
2500299130,1cyclictest0-21swapper/021:46:320
2501099129,2cyclictest4245-21head20:40:173
2501099129,2cyclictest30686-21sed23:50:283
2501099129,2cyclictest25326-21grep22:50:153
2501099129,2cyclictest25263-21tr20:15:203
2501099129,2cyclictest17406-21gltestperf22:45:173
2501099129,2cyclictest12120-21wc20:55:253
2501099128,3cyclictest29288-21sed19:15:263
2501099128,2cyclictest9970-21tr19:45:143
2501099121,10cyclictest0-21swapper/322:15:513
2501099120,11cyclictest0-21swapper/323:25:123
2501099120,11cyclictest0-21swapper/300:33:413
2500899127,4cyclictest4639-21tr20:40:202
2500899126,4cyclictest24378-21cstates20:15:142
2500899121,7cyclictest0-21swapper/220:05:122
2500499129,2cyclictest9360-21cat20:50:191
2500499129,2cyclictest6218-21cat19:35:251
2500499129,2cyclictest27165-21tr20:20:141
2500499129,2cyclictest26106-21uname22:50:191
2500499129,2cyclictest25759-21grep22:30:271
2500499129,2cyclictest21708-21sh21:11:321
2500499129,2cyclictest21483-21cut21:50:161
2500499129,2cyclictest12831-21basename21:00:011
2500499128,3cyclictest1671-21grep20:35:151
2500499122,1cyclictest141rcu_preempt23:37:021
25004991211,1cyclictest30685-21fschecks_time19:20:191
2500499120,10cyclictest0-21swapper/121:46:571
2500299129,2cyclictest840-21cron20:35:000
2500299129,2cyclictest32266-21chrt00:30:180
2500299129,2cyclictest3048-21grep19:30:160
2500299129,2cyclictest28706-21ssh23:30:200
2500299129,2cyclictest20291-21ssh21:10:280
2500299129,2cyclictest19851-21head23:25:160
2500299129,2cyclictest13737-21grep19:50:270
2500299129,2cyclictest13-21ksoftirqd/023:53:480
2500299129,2cyclictest1176-21ls19:25:220
2500299129,2cyclictest10457-21turbostat.cron21:25:000
2500299128,3cyclictest27336-21cron19:15:010
2500299128,3cyclictest27336-21cron19:15:000
2500299128,2cyclictest1859-21cat23:15:000
2500299126,5cyclictest11264-21apache223:18:380
2500299125,5cyclictest22629-21grep21:50:230
2500299123,5cyclictest225-21systemd-journal20:20:000
25002991210,2cyclictest13-21ksoftirqd/022:20:210
2500299120,6cyclictest30928-21perf20:30:000
290002110,0sleep10-21swapper/123:11:021
2501099118,2cyclictest9772-21ssh22:40:253
2501099118,2cyclictest27650-21grep20:20:203
2501099118,2cyclictest27000-21tr21:15:163
2501099118,2cyclictest22268-21uniq20:10:163
2501099118,2cyclictest20166-21ssh23:05:403
2501099118,2cyclictest19919-21grep20:05:173
2501099118,2cyclictest17987-21cat21:10:023
2501099118,2cyclictest16513-21ssh00:20:293
2501099118,2cyclictest16172-21taskset21:05:173
2501099118,2cyclictest16172-21taskset21:05:173
2501099118,2cyclictest15596-21cat19:55:213
2501099118,2cyclictest15207-21cat22:05:203
2501099118,2cyclictest15034-21ssh21:45:373
2501099118,2cyclictest1303-21ssh21:57:073
2501099118,2cyclictest12628-21diskstats23:40:173
2501099118,2cyclictest10117-21chrt20:51:503
2501099117,3cyclictest3224-21cron20:40:003
2501099117,3cyclictest14230-21sh19:54:593
2501099115,3cyclictest225-21systemd-journal20:28:253
2501099111,7cyclictest0-21swapper/321:40:173
2501099110,10cyclictest32653-21ssh21:37:223
2500899118,2cyclictest3999-21sed19:30:262
2500899118,2cyclictest18555-21sed20:00:272
2500899118,2cyclictest13936-21taskset19:51:192
2500899117,3cyclictest12838-21awk21:00:002
2500899114,7cyclictest0-21swapper/219:20:172
2500899114,6cyclictest29411-21grep20:25:162
2500899113,7cyclictest0-21swapper/221:05:162
2500899113,7cyclictest0-21swapper/221:05:152
2500899110,10cyclictest28289-21grep19:15:172
2500499119,1cyclictest23672-21grep00:25:181
2500499119,1cyclictest22629-21tr22:10:141
2500499118,2cyclictest9714-21tr23:00:161
2500499118,2cyclictest9143-21ssh22:40:181
2500499118,2cyclictest6694-21sh22:20:111
2500499118,2cyclictest4019-21sed19:30:251
2500499118,2cyclictest30761-21sh21:55:191
2500499118,2cyclictest29358-21ssh21:16:001
2500499118,2cyclictest20004-21head20:05:171
2500499118,2cyclictest18599-21grep20:00:271
2500499118,2cyclictest14542-21cat21:00:261
2500499118,2cyclictest12869-21ssh00:38:111
2500499118,2cyclictest0-21swapper/121:35:321
2500499117,3cyclictest30092-21basename00:10:131
2500499115,2cyclictest30921-21perf20:30:001
2500499110,10cyclictest0-21swapper/119:40:501
2500299119,1cyclictest30251-21sort21:55:170
2500299118,2cyclictest8650-21ntp_kernel_pll_00:35:210
2500299118,2cyclictest8420-21ssh22:01:060
2500299118,2cyclictest4464-21taskset23:35:160
2500299118,2cyclictest31976-21ssh22:35:050
2500299118,2cyclictest30706-21grep19:20:190
2500299118,2cyclictest30682-21ssh22:15:100
2500299118,2cyclictest21012-21ssh23:06:190
2500299118,2cyclictest18422-21ssh00:21:590
2500299118,2cyclictest15272-21sed19:55:180
2500299118,2cyclictest13287-21ssh23:40:220
2500299118,2cyclictest10552-21ssh22:40:450
2500299118,2cyclictest10233-21grep23:00:180
2500299116,2cyclictest452-21systemd-logind20:22:030
2500299116,2cyclictest225-21systemd-journal19:39:030
2501099107,2cyclictest5974-21cut20:45:163
2501099107,2cyclictest2548-21cat19:30:143
2501099107,2cyclictest2143-21wc19:30:003
2500499108,1cyclictest7247-21ssh23:55:541
2500499108,1cyclictest3837-21expr20:40:151
2500499108,1cyclictest173592sleep121:07:201
2500499108,1cyclictest173592sleep121:07:191
2500499108,1cyclictest14458-21taskset19:55:021
2500499108,1cyclictest12352-21tr19:50:151
2500499107,2cyclictest25-21ksoftirqd/119:20:001
2500499107,2cyclictest16906-21tr22:45:151
2500499100,9cyclictest10092-21expr19:45:151
2500299108,1cyclictest28792sleep020:35:370
2500299108,1cyclictest16332-21if_enp2s021:05:190
2500299108,1cyclictest16332-21if_enp2s021:05:190
2500299107,2cyclictest9981-21grep19:45:150
2500299107,2cyclictest6561-21cut20:45:200
2500299106,3cyclictest10894-21grep20:55:150
250109997,1cyclictest18880-21taskset20:03:313
250029997,1cyclictest29423-21chrt19:16:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional