You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-16 - 03:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Dec 16, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17975210388,10sleep00-21swapper/019:08:380
1762129076,9sleep20-21swapper/219:05:262
1612628060,3sleep10-21swapper/119:05:141
1796327966,9sleep30-21swapper/319:08:293
1817299250,1cyclictest0-21swapper/121:50:151
1817499221,19cyclictest13973-21apache_volume23:30:132
1817499200,2cyclictest0-21swapper/222:30:152
18172992018,1cyclictest0-21swapper/119:10:151
1817299181,16cyclictest0-21swapper/122:20:141
1817899174,7cyclictest455-21dbus-daemon23:40:003
18168991713,4cyclictest13-21ksoftirqd/000:08:030
1816899170,12cyclictest0-21swapper/020:05:140
18172991615,1cyclictest0-21swapper/122:25:191
18172991615,1cyclictest0-21swapper/122:00:011
18168991614,1cyclictest4732-21awk22:05:010
18168991614,1cyclictest0-21swapper/021:10:150
1817899159,4cyclictest31199-21awk20:45:003
1816899156,8cyclictest0-21swapper/022:20:130
18168991513,2cyclictest0-21swapper/023:30:130
1816899150,2cyclictest0-21swapper/020:50:140
1817899149,3cyclictest31555-21tail00:05:183
18178991410,3cyclictest17273-21grep20:15:143
1817899140,13cyclictest12122-21munin-run22:20:003
1817499142,1cyclictest0-21swapper/223:20:132
18174991410,3cyclictest29473-21grep21:45:262
1817299146,4cyclictest225-21systemd-journal00:05:011
18172991411,2cyclictest8752-21sort23:20:001
18172991410,3cyclictest17591-21grep21:20:271
18172991410,2cyclictest17379-21tr22:30:161
18172991410,2cyclictest13081-21cron21:15:001
1817299140,12cyclictest0-21swapper/120:30:141
1816899147,6cyclictest20464-21munin-run21:30:000
1816899142,5cyclictest225-21systemd-journal22:32:040
1816899142,4cyclictest225-21systemd-journal20:00:000
1816899140,12cyclictest14940-21tr20:10:170
1817899139,3cyclictest21367-21sort21:30:173
1817899139,3cyclictest13728-21grep00:35:173
18178991310,2cyclictest720-21grep20:45:263
1817899130,9cyclictest0-21swapper/321:55:153
1817899130,8cyclictest0-21swapper/323:25:153
1817499138,2cyclictest475-21in:imuxsock19:25:072
18174991310,2cyclictest5361-21grep23:10:262
18174991310,2cyclictest32127-21grep20:45:172
18174991310,2cyclictest29758-21grep20:40:172
18174991310,2cyclictest22335-21sed21:30:262
18174991310,2cyclictest2129-21grep23:05:182
1817499130,12cyclictest14512-21apt22:25:002
1817299139,2cyclictest14572-21sort23:30:171
1817299132,7cyclictest4952-21sh19:50:001
18172991310,2cyclictest3508-21grep00:15:151
18172991310,2cyclictest21245-21tr19:15:161
18172991310,2cyclictest19345-21sort23:40:181
18172991310,2cyclictest14576-21grep00:35:261
1816899139,3cyclictest22629-21head22:40:170
1816899139,3cyclictest1207-21wc21:55:210
18168991310,2cyclictest622-21sed23:00:260
18168991310,2cyclictest25509-21sed22:45:230
18168991310,2cyclictest10278-21grep20:00:150
1817899129,2cyclictest7335-21munin-plugin-st19:55:003
1817899129,2cyclictest3094-21grep20:50:263
1817899129,2cyclictest30564-21sh00:05:013
1817899129,2cyclictest26429-21basename20:35:003
1817899129,2cyclictest2614-21wc19:45:003
1817899129,2cyclictest24035-21sort19:20:143
1817899129,2cyclictest23562-21grep20:25:283
1817899129,2cyclictest2034-21grep23:05:183
1817899129,2cyclictest19150-21sed19:10:183
1817899129,2cyclictest13080-21cron21:15:003
1817899129,2cyclictest12715-21grep20:05:153
1817899129,2cyclictest10824-21wc21:10:003
1817899129,2cyclictest10808-21head20:00:173
1817899128,3cyclictest6953-21head21:00:183
1817899128,3cyclictest32128-21grep19:35:263
18178991210,1cyclictest22699-21grep19:15:343
1817499129,2cyclictest5826-21grep19:50:192
1817499129,2cyclictest26951-21grep22:50:142
1817499129,2cyclictest26412-21perf22:50:002
1817499129,2cyclictest2621-21grep20:50:222
1817499129,2cyclictest22692-21head22:40:172
1817499129,2cyclictest20293-21head22:35:172
1817499129,2cyclictest19862-21grep20:20:152
1817499129,2cyclictest13475-21basename21:15:172
1817499129,2cyclictest12679-21cron00:35:002
1817499123,4cyclictest225-21systemd-journal22:15:012
1817299129,2cyclictest8883-21cut00:25:181
1817299129,2cyclictest7811-21sed21:00:271
1817299129,2cyclictest6094-21head22:05:201
1817299129,2cyclictest602-21sed23:00:261
1817299129,2cyclictest28844-21head19:30:171
1817299129,2cyclictest2322-21sh22:00:001
1817299129,2cyclictest2119-21head23:05:181
1817299129,2cyclictest16994-21sort23:35:181
1817299129,2cyclictest14119-21grep20:05:301
1817299128,3cyclictest10827-21cut20:00:161
1817299124,4cyclictest225-21systemd-journal00:15:001
1816899129,2cyclictest28343-21grep20:35:270
1816899129,2cyclictest28175-21apt00:00:000
1816899129,2cyclictest24283-21sed21:35:210
1816899129,2cyclictest24283-21sed21:35:210
1816899129,2cyclictest20498-21cron19:15:000
1816899129,2cyclictest18859-21head20:15:260
1816899129,2cyclictest16428-21tail22:25:260
1816899129,2cyclictest12203-21grep00:30:240
1816899129,2cyclictest12106-21grep23:25:180
1816899129,2cyclictest10822-21sort22:15:180
1816899128,3cyclictest5538-21cron00:20:000
1816899128,3cyclictest309-21cat19:40:000
1816899128,3cyclictest16682-21tail21:20:190
1816899128,3cyclictest0-21swapper/000:10:130
1816899126,5cyclictest3621-21gltestperf19:45:160
1816899121,2cyclictest225-21systemd-journal21:49:070
18168991211,1cyclictest0-21swapper/020:30:140
18168991210,1cyclictest32345-21idleruntime-cro21:55:000
1817899118,2cyclictest6066-21tr00:20:153
1817899118,2cyclictest3728-21nfs4_client22:00:203
1817899118,2cyclictest30182-21grep22:55:243
1817899118,2cyclictest29563-21cat19:30:243
1817899118,2cyclictest27742-21sed23:55:273
1817899118,2cyclictest27475-21grep19:25:273
1817899118,2cyclictest26891-21grep22:50:143
1817899118,2cyclictest26493-21cat21:40:213
1817899118,2cyclictest21955-21wget22:40:143
1817899118,2cyclictest21839-21grep22:40:003
1817899118,2cyclictest19154-21expr23:40:153
1817899118,2cyclictest15524-21grep20:10:193
1817899117,3cyclictest5854-21cron23:15:003
1817899117,3cyclictest25795-21cron23:55:003
1817899117,2cyclictest23690-21sort21:35:163
1817899117,2cyclictest23690-21sort21:35:163
1817899117,2cyclictest15483-21head22:25:173
1817899110,10cyclictest11288-21date23:25:003
1817499119,1cyclictest6463-21sort00:20:182
1817499119,1cyclictest18186-21grep21:25:002
1817499119,1cyclictest15041-21grep20:10:172
1817499118,2cyclictest3212-21grep22:00:172
1817499118,2cyclictest32042-21cat19:35:262
1817499118,2cyclictest30560-21cron00:05:012
1817499118,2cyclictest29640-21cat19:30:262
1817499118,2cyclictest27702-21cat20:35:202
1817499118,2cyclictest25215-21sleep21:40:002
1817499118,2cyclictest25215-21sleep21:40:002
1817499118,2cyclictest23982-21grep23:50:152
1817499118,2cyclictest22775-21tail23:45:262
1817499118,2cyclictest21666-21grep19:15:182
1817499118,2cyclictest18051-21grep23:35:272
1817499118,2cyclictest17553-21cat20:15:152
1817499118,2cyclictest1586-21cut19:40:192
1817499118,2cyclictest15051-21tr22:25:142
1817499118,2cyclictest13279-21sshd23:27:262
1817499118,2cyclictest13235-21cat20:05:182
1817499118,2cyclictest1293-21tr00:10:152
1817499118,2cyclictest11785-21taskset22:15:582
1817499118,2cyclictest11089-21tr21:10:162
1817299118,2cyclictest7741-21taskset00:22:351
1817299118,2cyclictest6332-21grep19:50:221
1817299118,2cyclictest3504-21cron20:55:001
1817299118,2cyclictest30687-21tail22:55:291
1817299118,2cyclictest28334-21tail20:35:271
1817299118,2cyclictest28144-21tr21:45:151
1817299118,2cyclictest26596-21cut21:40:221
1817299118,2cyclictest2241-21taskset19:40:441
1817299118,2cyclictest21360-21grep23:45:151
1817299118,2cyclictest21288-21grep20:20:271
1817299118,2cyclictest19804-21tr22:35:151
1817299118,2cyclictest19095-21sed21:25:181
1817299118,2cyclictest17939-21sed20:15:171
1817299118,2cyclictest13485-21cron23:30:001
1817299117,3cyclictest31593-21tr20:45:141
1817299117,2cyclictest26483-21head19:25:181
1817299117,2cyclictest26294-21tr23:55:151
1817299110,11cyclictest0-21swapper/123:25:001
1817299110,10cyclictest21949-21cut22:40:141
1816899118,2cyclictest8856-21cut00:25:170
1816899118,2cyclictest5883-21idleruntime-cro21:00:000
1816899118,2cyclictest31248-21date23:00:000
1816899118,2cyclictest30561-21cron00:05:010
1816899118,2cyclictest29827-21grep19:30:260
1816899118,2cyclictest27594-21dump-pmu-power21:45:000
1816899118,2cyclictest25873-21wc23:55:010
1816899118,2cyclictest23645-21grep20:25:280
1816899118,2cyclictest2081-21grep19:40:260
1816899118,2cyclictest18883-21grep23:40:160
1816899117,3cyclictest29801-21grep20:40:180
1816899117,2cyclictest7611-21grep22:10:160
1816899114,2cyclictest23665-21tr19:20:120
1816899112,3cyclictest452-21systemd-logind23:39:040
1816899111,8cyclictest0-21swapper/021:00:140
1816899111,6cyclictest0-21swapper/020:45:140
1817899108,1cyclictest9334-21unixbench_singl19:55:263
1817899108,1cyclictest4247-21cat20:55:153
1817899108,1cyclictest32735-21gzip23:00:213
1817899108,1cyclictest27043-21cat20:35:163
1817899108,1cyclictest21694-21taskset23:45:163
1817899108,1cyclictest18982-21chrt22:31:353
1817899108,1cyclictest13998-21cut23:30:143
1817899108,1cyclictest13335-21uname22:20:183
1817899108,1cyclictest1030-21basename00:10:123
1817899107,2cyclictest8518-21cat00:25:163
1817899107,2cyclictest8160-21chrt23:16:263
1817899107,2cyclictest4776-21taskset19:47:483
1817899107,2cyclictest3716-21tr00:15:153
1817899107,2cyclictest24548-21cut22:45:153
1817899107,2cyclictest19787-21ls20:20:153
1817899107,2cyclictest18699-21ls21:25:163
1817899107,2cyclictest16726-21head21:20:193
1817899107,2cyclictest14086-21head21:15:193
1817899107,2cyclictest11993-21cat00:30:233
1817899100,9cyclictest28619-21tail21:45:173
1817499108,2cyclictest7956-21expr19:55:152
1817499108,1cyclictest8077-21taskset21:02:302
1817499108,1cyclictest4994-21apt-get22:05:142
1817499108,1cyclictest32595-21apache_accesses21:55:122
1817499108,1cyclictest31257-21expr00:05:162
1817499108,1cyclictest311rcuc/220:55:282
1817499108,1cyclictest3115-21grep19:45:132
1817499108,1cyclictest30923-21perl21:50:172
1817499108,1cyclictest308652sleep222:56:382
1817499108,1cyclictest27896-21taskset23:56:332
1817499108,1cyclictest26149-21grep21:40:182
1817499108,1cyclictest24140-21grep19:20:152
1817499108,1cyclictest197252sleep219:10:242
1817499107,2cyclictest19009-21grep23:40:162
1817499107,2cyclictest12977-21basename00:35:132
1817499105,4cyclictest7422-21cat23:15:212
1817299108,1cyclictest3870-21basename20:55:151
1817299108,1cyclictest32675-21which00:05:561
1817299108,1cyclictest30731-21expr19:35:151
1817299108,1cyclictest25413-21ntpq22:45:211
1817299108,1cyclictest23744-21grep21:35:161
1817299108,1cyclictest23744-21grep21:35:161
1817299108,1cyclictest216342sleep121:30:181
1817299108,1cyclictest15991-21taskset20:10:241
1817299108,1cyclictest15143-21taskset21:16:051
1817299108,1cyclictest10840-21expr00:30:131
1817299108,1cyclictest105852sleep121:08:311
1817299107,2cyclictest7837-21tr22:10:171
1817299107,2cyclictest7667-21grep19:55:141
1817299107,2cyclictest5294-21wc23:10:251
1817299107,2cyclictest27312-21cut22:50:161
1817299107,2cyclictest23934-21grep19:20:131
1817299107,2cyclictest10630-21grep22:15:171
1817299105,4cyclictest30120-21sed20:40:191
1817299102,6cyclictest23036-21tail20:25:221
1816899108,1cyclictest5524-21chrt23:10:460
1816899108,1cyclictest229432sleep023:46:370
1816899108,1cyclictest225492sleep021:31:380
1816899108,1cyclictest21472-21chrt22:38:050
1816899108,1cyclictest14353-21sed21:15:220
1816899107,2cyclictest6324-21grep19:50:220
1816899107,2cyclictest5976-21tr00:20:160
1816899107,2cyclictest26945-21grep22:50:130
1816899107,2cyclictest0-21swapper/021:05:150
1816899105,4cyclictest20960-21/usr/sbin/munin19:15:200
1816899103,7cyclictest0-21swapper/023:05:150
1816899101,8cyclictest0-21swapper/000:35:160
1816899101,2cyclictest13-21ksoftirqd/020:20:000
1816899100,6cyclictest13-21ksoftirqd/022:08:360
181789997,1cyclictest4993-21apt22:05:143
181789997,1cyclictest31293-21cat21:50:203
181789995,3cyclictest9731-21cron22:15:013
181749998,1cyclictest9738-21cat00:25:262
181749998,1cyclictest24340-21aten2_r7power_c20:30:142
181749997,2cyclictest21943-21taskset20:25:132
181749997,1cyclictest9059-21expr21:05:152
181749997,1cyclictest3855-21expr00:15:162
181749997,1cyclictest19166-21basename21:25:192
181749996,2cyclictest32176-21grep23:00:172
181749996,2cyclictest10700-21uniq20:00:172
181729997,1cyclictest25021-21grep23:50:231
181689995,3cyclictest7468-21tr23:15:230
181689991,5cyclictest13-21ksoftirqd/023:21:060
181689990,6cyclictest13-21ksoftirqd/019:25:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional