You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 07:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Dec 11, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2051210187,9sleep10-21swapper/119:09:041
1655521010,0sleep00-21swapper/022:09:570
182029582,9sleep30-21swapper/319:06:053
206728968,6sleep20-21swapper/219:09:172
183728367,9sleep00-21swapper/019:06:190
67752760,0sleep30-21swapper/322:47:193
2230993029,1cyclictest0-21swapper/300:10:493
2230992726,1cyclictest0-21swapper/300:31:493
230602250,0sleep10-21swapper/122:14:151
2230992322,1cyclictest0-21swapper/321:12:303
222599230,1cyclictest0-21swapper/122:42:461
222599230,1cyclictest0-21swapper/122:00:151
2230992221,1cyclictest24455-21ssh21:52:093
2230992221,1cyclictest0-21swapper/321:18:293
2225992222,0cyclictest0-21swapper/121:11:101
223099207,12cyclictest2631-21/usr/sbin/munin22:00:133
223099200,20cyclictest0-21swapper/323:47:243
222599200,17cyclictest0-21swapper/123:24:161
2230991919,0cyclictest0-21swapper/321:22:503
222599190,19cyclictest0-21swapper/122:08:411
223099171,15cyclictest0-21swapper/323:14:403
222599170,11cyclictest0-21swapper/122:45:151
222599170,0cyclictest0-21swapper/123:38:241
222399174,7cyclictest3625-21awk20:20:000
222399170,1cyclictest0-21swapper/021:42:590
2230991613,2cyclictest39-21ksoftirqd/323:09:403
222799160,1cyclictest0-21swapper/221:54:422
2223991615,1cyclictest0-21swapper/021:37:560
2223991614,1cyclictest13-21ksoftirqd/000:05:200
222799159,5cyclictest4471-21grep21:15:272
222799150,14cyclictest0-21swapper/223:50:152
222599150,1cyclictest0-21swapper/123:29:451
2223991513,1cyclictest13-21ksoftirqd/000:02:030
2223991513,1cyclictest0-21swapper/023:32:560
223099145,8cyclictest1291-21runrttasks21:32:453
222799147,3cyclictest1291-21runrttasks22:39:212
222799143,10cyclictest0-21swapper/200:38:482
222799141,4cyclictest17483-21df21:25:162
2227991414,0cyclictest0-21swapper/223:36:252
2227991413,1cyclictest0-21swapper/221:45:202
2227991412,2cyclictest0-21swapper/223:58:572
222799140,0cyclictest0-21swapper/222:00:142
222599142,2cyclictest11674-21grep21:20:271
2225991410,3cyclictest5525-21grep20:20:251
222399148,2cyclictest18543-21latency_hist00:30:010
222399144,9cyclictest12078-21perf23:14:590
222399140,2cyclictest0-21swapper/023:51:150
222399140,13cyclictest17885-21grep22:10:180
223099139,2cyclictest1607-21cut20:15:143
2230991310,2cyclictest7207-21sed21:40:183
2230991310,2cyclictest6561-21ls23:55:253
2230991310,2cyclictest31185-21cat22:20:003
2230991310,2cyclictest27722-21cat20:00:213
2230991310,2cyclictest18996-21sh22:10:283
2230991310,2cyclictest18686-21sort00:05:013
2230991310,2cyclictest16910-21grep23:40:193
2230991310,2cyclictest14831-21grep22:30:213
223099130,12cyclictest2409-21wc00:15:283
223099130,12cyclictest0-21swapper/321:48:303
222799138,5cyclictest18503-21apache223:44:382
222799131,1cyclictest141rcu_preempt21:10:172
2227991311,2cyclictest13630-21ssh23:15:202
2227991310,2cyclictest8565-21cron20:30:002
2227991310,2cyclictest29875-21grep20:05:172
2227991310,2cyclictest17699-21grep19:40:192
222799130,2cyclictest0-21swapper/200:03:072
222799130,12cyclictest1186-21cron20:15:002
222799130,0cyclictest0-21swapper/223:13:552
222599139,2cyclictest18666-21tr20:50:151
222599132,2cyclictest28033-21unixbench_singl00:10:311
2225991310,2cyclictest6550-21grep21:40:161
2225991310,2cyclictest26868-21grep00:35:241
222599130,2cyclictest0-21swapper/122:34:121
222599130,13cyclictest0-21swapper/123:10:191
222399139,3cyclictest26579-21wc00:35:220
222399136,4cyclictest9466-21latency_hist22:05:000
222399132,2cyclictest141rcu_preempt20:40:130
2223991311,2cyclictest4141-21ls19:10:250
2223991311,1cyclictest0-21swapper/022:25:140
2223991310,2cyclictest4665-21grep22:45:270
2223991310,2cyclictest21536-21grep19:50:000
2223991310,2cyclictest16126-21grep20:45:150
222399130,2cyclictest0-21swapper/023:45:070
222399130,1cyclictest0-21swapper/021:58:310
222399130,12cyclictest8912-21cut20:30:140
222399130,12cyclictest0-21swapper/019:15:130
223099129,2cyclictest5658-21grep19:15:153
223099129,2cyclictest3416-21grep23:30:263
223099129,2cyclictest3145-21df19:10:163
223099129,2cyclictest28560-21sed22:40:153
223099129,2cyclictest27315-21grep00:35:283
223099129,2cyclictest24581-21mailstats21:00:243
223099129,2cyclictest20269-21cat00:05:233
223099129,2cyclictest17127-21tr20:45:203
223099129,2cyclictest15295-21grep19:35:163
223099128,3cyclictest26427-21turbostat.cron23:25:003
223099126,3cyclictest493-21systemd-logind19:34:213
2230991212,0cyclictest0-21swapper/323:01:503
223099120,11cyclictest12784-21tr23:15:153
223099120,11cyclictest11870-21sort20:35:173
222799129,3cyclictest32-21ksoftirqd/219:15:122
222799129,2cyclictest8183-21grep19:20:182
222799129,2cyclictest3631-21head22:45:182
222799129,2cyclictest30307-21chrt00:12:292
222799129,2cyclictest27675-21cron21:10:002
222799129,2cyclictest23938-21cut23:00:142
222799129,2cyclictest22033-21grep19:50:122
222799129,2cyclictest1972-21ls00:15:242
222799129,2cyclictest16074-21cat19:35:252
222799129,2cyclictest11058-21tr21:20:232
222799128,3cyclictest11924-21grep22:50:282
222799125,4cyclictest25062-21/usr/sbin/munin00:35:002
222799121,1cyclictest29493-21ssh22:18:332
2227991210,1cyclictest2179-21apt22:00:012
222799120,12cyclictest0-21swapper/223:32:372
222799120,11cyclictest0-21swapper/200:20:252
222599129,2cyclictest910-21grep22:20:251
222599129,2cyclictest7168-21awk19:20:001
222599129,2cyclictest32738-21head20:10:221
222599129,2cyclictest3255-21cut19:10:161
222599129,2cyclictest3182-21grep21:15:161
222599129,2cyclictest27295-21latency_hist22:40:011
222599129,2cyclictest24888-21grep22:15:161
222599129,2cyclictest24424-21tr21:30:171
222599129,2cyclictest23867-21awk23:00:131
222599129,2cyclictest23478-21grep19:50:251
222599129,2cyclictest22180-21wc21:50:251
222599129,2cyclictest17206-21grep20:45:211
222599129,2cyclictest16242-21grep23:40:151
222599129,2cyclictest12867-21sort19:30:181
222599128,3cyclictest18298-21ssh22:55:251
222599127,2cyclictest0-21swapper/119:55:141
222599120,12cyclictest0-21swapper/121:28:421
222599120,0cyclictest0-21swapper/100:20:131
222399129,2cyclictest7592-21tr20:25:230
222399129,2cyclictest5584-21grep20:20:250
222399129,2cyclictest29852-21wc21:10:250
222399129,2cyclictest27705-21awk21:10:010
222399129,2cyclictest26066-21grep19:55:270
222399129,2cyclictest23850-21cut23:00:130
222399129,2cyclictest21522-21cut21:50:190
222399129,2cyclictest13279-21tail19:30:200
222399129,2cyclictest1213-21awk20:15:010
222399128,3cyclictest10198-21ssh23:35:220
222399125,6cyclictest19140-21munin-run19:44:590
222399125,5cyclictest18913-21apt-config00:30:000
2223991210,1cyclictest32554-21wc23:05:270
2223991210,1cyclictest19481-21wc20:50:200
2223991210,1cyclictest14575-21wc21:45:170
222399120,0cyclictest0-21swapper/022:57:260
223099119,1cyclictest24626-21grep19:55:163
223099118,2cyclictest955-21ssh21:35:313
223099118,2cyclictest5592-21grep20:20:253
223099118,2cyclictest32254-21sort22:20:163
223099118,2cyclictest31663-21wc20:10:143
223099118,2cyclictest29798-21ssh23:26:223
223099118,2cyclictest29796-21head21:55:283
223099118,2cyclictest29207-21grep20:05:153
223099118,2cyclictest24955-21ssh22:37:493
223099118,2cyclictest2413-21ssh23:52:583
223099118,2cyclictest19123-21cron19:44:593
223099118,2cyclictest17407-21expr21:25:163
223099118,2cyclictest16938-21cut22:55:153
223099118,2cyclictest14358-21ssh22:07:513
223099118,2cyclictest13864-21taskset00:25:223
223099118,2cyclictest13324-21ssh22:51:533
223099118,2cyclictest11448-21ssh23:35:583
223099118,2cyclictest10546-21ssh00:22:243
223099118,2cyclictest10526-21grep19:25:183
223099117,3cyclictest19623-21tr19:45:173
223099112,7cyclictest22927-21perf21:00:003
223099112,4cyclictest224-21systemd-journal21:05:233
2230991111,0cyclictest0-21swapper/320:25:153
223099110,10cyclictest14104-21grep20:40:153
223099110,10cyclictest0-21swapper/322:26:003
222799119,1cyclictest4338-21grep19:10:272
222799119,1cyclictest22937-21idleruntime-cro21:00:002
222799119,1cyclictest134502sleep222:30:112
222799119,1cyclictest12005-21tail20:35:172
222799118,2cyclictest4625-21ssh21:39:002
222799118,2cyclictest3582-21cron20:20:012
222799118,2cyclictest24633-21tr19:55:152
222799118,2cyclictest11322-21rm22:28:232
222799118,2cyclictest10216-21grep22:05:172
222799115,6cyclictest37850irq/131-snd_hda22:57:482
222799115,6cyclictest0-21swapper/221:30:492
222799111,10cyclictest27043-21kworker/2:122:41:252
222799110,7cyclictest0-21swapper/221:43:002
222799110,1cyclictest0-21swapper/200:29:502
222799110,11cyclictest0-21swapper/222:20:142
222799110,10cyclictest20642-21taskset00:05:282
222599119,1cyclictest14945-21grep21:45:201
222599118,2cyclictest9220-21rm22:26:221
222599118,2cyclictest3155-21ssh00:16:121
222599118,2cyclictest30694-21tr23:50:171
222599118,2cyclictest29307-21grep21:55:221
222599118,2cyclictest27010-21tr20:00:171
222599118,2cyclictest25654-21hostname21:05:131
222599118,2cyclictest25171-21grep23:45:251
222599118,2cyclictest19314-21tr00:05:181
222599118,2cyclictest12545-21ssh00:00:151
222599118,2cyclictest12387-21ssh23:15:081
222599118,2cyclictest11499-21grep20:35:161
222599117,3cyclictest32653-21ssh21:35:221
222599113,5cyclictest12898-21cpuspeed_turbos00:25:171
222599110,9cyclictest0-21swapper/123:05:041
222599110,10cyclictest5534-21sed23:55:151
222599110,10cyclictest17567-21grep19:40:191
222399119,1cyclictest6790-21grep23:55:270
222399119,1cyclictest21998-21sed20:55:240
222399119,1cyclictest169772sleep021:25:130
222399118,2cyclictest3913-21ssh00:17:040
222399118,2cyclictest31383-21ssh22:20:050
222399118,2cyclictest29293-21sh22:40:220
222399118,2cyclictest28794-21tail23:25:280
222399118,2cyclictest279222sleep020:00:240
222399118,2cyclictest24252-21wc21:00:200
222399118,2cyclictest24063-21ssh21:30:120
222399118,2cyclictest19983-21sh23:42:090
222399118,2cyclictest19884-21ssh23:20:150
222399118,2cyclictest16682-21ssh22:31:390
222399118,2cyclictest10656-21grep21:20:180
222399117,3cyclictest9849-21grep19:25:160
222399114,6cyclictest16711-21sendmail19:39:590
222399111,9cyclictest26064-21apache_volume00:10:130
222399111,4cyclictest141rcu_preempt22:15:210
223099108,1cyclictest9414-21df20:30:163
223099108,1cyclictest93962sleep319:23:133
223099108,1cyclictest21859-21aten2_r7power_p19:50:123
223099108,1cyclictest19810-21seq20:50:243
222799108,2cyclictest5031-21ntpq20:20:202
222799108,2cyclictest32420-21ssh23:05:252
222799108,2cyclictest16105-21basename20:45:142
222799108,1cyclictest28271-21grep23:25:232
222799108,1cyclictest18312-21tr22:10:212
222799107,2cyclictest19998-21grep19:45:192
222799107,2cyclictest19184-21grep20:50:192
222799107,2cyclictest14378-21head20:40:172
222799106,2cyclictest25212-21taskset21:03:562
222799105,2cyclictest224-21systemd-journal23:20:232
222799100,9cyclictest0-21swapper/223:46:062
222799100,9cyclictest0-21swapper/219:30:272
222799100,3cyclictest0-21swapper/219:25:142
222599108,1cyclictest29699-21sed20:05:161
222599108,1cyclictest23190-21taskset21:00:121
222599108,1cyclictest14502sleep123:30:051
222599108,1cyclictest11570-21chrt19:25:381
222599107,2cyclictest8162-21head20:25:291
222599107,2cyclictest3144-21tail20:15:281
222599107,2cyclictest23574-21ssh00:33:121
222599107,2cyclictest22486-21grep20:55:281
222599107,2cyclictest15316-21sed20:40:251
222599107,2cyclictest11972-21ssh22:50:291
222599101,1cyclictest0-21swapper/120:30:181
222399109,0cyclictest0-21swapper/020:35:150
222399108,1cyclictest6704-21ssh00:20:040
222399107,2cyclictest2731-21cut21:15:150
222399107,2cyclictest22152-21cat19:50:140
222399107,2cyclictest21782-21wc22:35:200
222399100,9cyclictest8899-21sed19:20:260
222399100,9cyclictest0-21swapper/023:15:150
100152100,0chrt0-21swapper/022:50:130
22279998,1cyclictest27047-21cut20:00:172
22279998,1cyclictest10006-21open_files20:30:212
22259997,1cyclictest7535-21hostname19:20:151
22259997,1cyclictest21177-21chrt19:45:571
22259995,3cyclictest16261-21grep19:35:251
22239996,2cyclictest29957-21basename20:05:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional