You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-10 - 06:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Jul 10, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2917529867,10sleep20-21swapper/219:07:082
2906829167,8sleep00-21swapper/019:05:460
2916628771,10sleep30-21swapper/319:07:013
2909428164,13sleep10-21swapper/119:06:071
2948399270,26cyclictest0-21swapper/022:46:280
2948999260,25cyclictest0-21swapper/223:27:202
2948999250,21cyclictest0-21swapper/222:55:592
2948799240,1cyclictest0-21swapper/121:47:071
29483992424,0cyclictest0-21swapper/022:53:050
2948999230,23cyclictest0-21swapper/222:26:282
2948999230,22cyclictest0-21swapper/222:19:152
29493992219,2cyclictest1258-21/usr/sbin/munin00:30:223
2948999221,5cyclictest21080-21apache221:18:152
29483992219,2cyclictest7008-21/usr/sbin/munin21:20:210
29493992119,1cyclictest28062-21/usr/sbin/munin22:50:163
2948999210,21cyclictest0-21swapper/200:07:262
29487992119,1cyclictest21997-21/usr/sbin/munin23:25:141
29487992019,0cyclictest0-21swapper/122:15:421
2948799200,19cyclictest0-21swapper/121:37:511
2948399200,1cyclictest0-21swapper/022:55:210
2948999192,4cyclictest0-21swapper/223:42:012
2948999190,15cyclictest0-21swapper/222:20:172
29487991916,2cyclictest27073-21/usr/sbin/munin22:30:141
29487991915,2cyclictest0-21swapper/123:06:281
29487991915,2cyclictest0-21swapper/123:06:281
29483991917,1cyclictest25404-21/usr/sbin/munin00:25:250
2948399190,1cyclictest0-21swapper/022:32:490
2948399190,18cyclictest0-21swapper/021:38:200
165472190,0sleep10-21swapper/122:23:451
29489991817,1cyclictest0-21swapper/223:53:042
2948999180,17cyclictest0-21swapper/222:11:512
2948799184,13cyclictest22067-21diskmemload22:57:011
29487991816,1cyclictest0-21swapper/123:15:161
2948799180,1cyclictest0-21swapper/121:54:511
2948399181,1cyclictest141rcu_preempt23:43:590
29493991715,1cyclictest17047-21/usr/sbin/munin00:20:183
2948999171,15cyclictest25460-21kworker/u8:123:34:572
2948999170,2cyclictest0-21swapper/221:38:232
2948799172,14cyclictest7999-21apt-get20:40:111
29487991716,1cyclictest0-21swapper/121:17:231
29487991715,1cyclictest0-21swapper/100:20:251
2948799170,1cyclictest0-21swapper/123:43:361
2948799170,15cyclictest0-21swapper/123:56:471
2948399176,6cyclictest24332-21munin-run21:50:010
29493991613,2cyclictest2028-21/usr/sbin/munin22:15:243
2948999162,14cyclictest0-21swapper/222:45:482
2948999160,15cyclictest0-21swapper/200:00:522
2948999160,14cyclictest0-21swapper/200:37:462
2948799162,14cyclictest22067-21diskmemload22:41:181
2948799160,0cyclictest0-21swapper/122:27:451
2948799160,0cyclictest0-21swapper/121:33:001
29483991611,3cyclictest29549-21sh23:10:140
2948399160,1cyclictest0-21swapper/022:39:210
29493991513,1cyclictest24782-21/usr/sbin/munin21:50:153
29489991514,1cyclictest0-21swapper/221:27:432
2948999150,2cyclictest0-21swapper/223:57:282
2948999150,15cyclictest0-21swapper/222:53:052
2948999150,0cyclictest0-21swapper/222:42:062
2948799158,6cyclictest17808-21grep00:00:311
29487991514,1cyclictest0-21swapper/121:22:421
29487991510,3cyclictest325-21grep23:50:221
2948399157,6cyclictest13-21ksoftirqd/023:27:270
2948399157,2cyclictest22067-21diskmemload00:20:000
2948399150,1cyclictest0-21swapper/021:52:160
2949399147,5cyclictest23923-21apt-get21:30:123
2949399142,8cyclictest30987-21diskmemload21:15:173
29493991411,2cyclictest32624-21/usr/sbin/munin00:10:173
2949399140,12cyclictest0-21swapper/322:59:343
2948999149,3cyclictest26661-21head20:10:162
29489991413,0cyclictest0-21swapper/221:10:122
29489991410,3cyclictest8999-21sed21:40:182
29489991410,3cyclictest2743-21cron20:30:002
29489991410,3cyclictest19934-21grep19:55:202
2948999140,4cyclictest0-21swapper/222:05:252
2948999140,1cyclictest0-21swapper/223:01:132
2948999140,1cyclictest0-21swapper/200:25:162
2948999140,14cyclictest0-21swapper/223:08:522
2948999140,14cyclictest0-21swapper/223:08:512
2948999140,14cyclictest0-21swapper/200:19:522
2948999140,12cyclictest0-21swapper/200:34:272
2948799141,11cyclictest22067-21diskmemload21:25:321
29487991410,2cyclictest17717-21cat00:40:001
2948799140,8cyclictest0-21swapper/122:10:151
2948799140,0cyclictest0-21swapper/123:20:341
2948399147,6cyclictest19616-21cron21:05:000
29483991410,3cyclictest20982-21cat20:00:000
29483991410,2cyclictest17292-21wc19:50:190
2948399140,9cyclictest0-21swapper/022:00:000
2948399140,3cyclictest0-21swapper/023:33:590
2949399139,3cyclictest1332-21grep19:15:263
2949399132,5cyclictest8028-21grep20:40:133
29493991310,2cyclictest23454-21grep23:25:223
29493991310,2cyclictest21619-21df23:05:173
29493991310,2cyclictest21619-21df23:05:173
29493991310,2cyclictest19420-21grep22:05:273
29493991310,2cyclictest19323-21sort22:45:003
29493991310,2cyclictest16136-21grep23:40:233
2948999139,3cyclictest8925-21turbostat19:34:592
2948999139,2cyclictest14880-21grep20:54:592
2948999136,3cyclictest30695-21apt-get20:20:122
2948999134,3cyclictest225-21systemd-journal21:34:322
29489991310,2cyclictest9458-21grep19:35:152
29489991310,2cyclictest29792sleep221:55:372
29489991310,2cyclictest19007-21grep00:20:282
29489991310,2cyclictest16705-21cut21:45:152
29489991310,2cyclictest16059-21cron19:50:012
29489991310,2cyclictest13700-21basename19:45:012
2948799139,3cyclictest8324-21ssh00:15:001
2948799139,3cyclictest23615-21basename20:05:131
2948799139,2cyclictest19160-21grep21:00:261
2948799137,5cyclictest10419-21ssh22:39:401
29487991311,1cyclictest0-21swapper/100:09:521
29487991310,2cyclictest4772-21munin-plugin-st23:15:001
29487991310,2cyclictest28809-21tune2fs22:50:161
29487991310,2cyclictest15640-21grep19:45:271
2948799130,3cyclictest0-21swapper/123:47:421
2948799130,11cyclictest0-21swapper/121:14:181
2948399139,3cyclictest5501-21wc19:25:210
2948399139,3cyclictest5501-21wc19:25:200
2948399139,3cyclictest23676-21cron00:05:000
2948399139,2cyclictest9649-21grep20:40:250
2948399139,2cyclictest29784-21grep19:10:150
2948399138,3cyclictest14856-21wc23:20:180
2948399131,6cyclictest30968-21ssh23:49:590
29483991311,1cyclictest15145-21ssh21:25:030
29483991310,2cyclictest32196-21wc00:10:000
29483991310,2cyclictest21444-21tr20:00:150
29483991310,2cyclictest16174-21cat19:50:010
29483991310,2cyclictest15839-21grep20:55:160
2948399130,3cyclictest0-21swapper/022:22:080
2948399130,10cyclictest0-21swapper/021:30:290
2948399130,0cyclictest0-21swapper/000:31:550
2949399129,2cyclictest7135-21grep23:35:183
2949399129,2cyclictest4434-21wc20:30:263
2949399129,2cyclictest27163-21ssh00:25:303
2949399129,2cyclictest26726-21tail22:10:163
2949399129,2cyclictest2503-21sh21:36:013
2949399129,2cyclictest24550-21grep00:05:163
2949399129,2cyclictest18424-21grep21:45:273
2949399129,2cyclictest16919-21sh22:05:003
2949399129,2cyclictest16079-21cut00:00:163
2949399129,2cyclictest12443-21cut23:00:143
2949399128,3cyclictest6552-21awk19:30:003
2949399128,3cyclictest6552-21awk19:30:003
2949399128,3cyclictest3738-21grep19:20:263
2949399128,3cyclictest16635-21cat00:20:003
2949399128,3cyclictest15945-21basename21:45:013
2949399124,8cyclictest0-21swapper/323:34:563
2949399122,6cyclictest0-21swapper/323:45:243
2949399120,9cyclictest0-21swapper/322:30:473
2949399120,4cyclictest225-21systemd-journal22:27:553
2948999129,2cyclictest6543-21cron19:30:002
2948999129,2cyclictest6543-21cron19:30:002
2948999129,2cyclictest22047-21cron21:10:002
2948999129,2cyclictest14197-21sh23:40:012
2948799129,2cyclictest916-21tr20:25:151
2948799129,2cyclictest8564-21cron22:00:001
2948799129,2cyclictest7092-21grep23:35:171
2948799129,2cyclictest6850-21tr19:30:131
2948799129,2cyclictest16929-21munin-plugin-st22:05:001
2948799129,2cyclictest13669-21tr23:00:201
2948799128,3cyclictest394-21head19:15:161
2948799128,3cyclictest31603-21cut23:30:191
2948799128,3cyclictest16930-21grep19:50:161
2948799123,6cyclictest30770-21aten2_r7power_p20:20:131
2948799120,11cyclictest13691-21sh19:45:011
2948799120,11cyclictest11074-21grep20:45:161
2948399129,2cyclictest7550-21grep19:30:160
2948399129,2cyclictest4065-21sed22:15:320
2948399129,2cyclictest2298-21grep20:25:270
2948399122,9cyclictest23638-21wget20:05:120
2948399120,2cyclictest0-21swapper/021:15:100
2949399118,2cyclictest8556-21ssh22:38:143
2949399118,2cyclictest5225-21ssh23:15:123
2949399118,2cyclictest4446-21ssh21:56:463
2949399118,2cyclictest330-21ssh23:11:363
2949399118,2cyclictest25327-21chrt20:06:033
2949399118,2cyclictest25152-21ssh21:10:543
2949399118,2cyclictest23059-21ssh22:46:263
2949399118,2cyclictest22410-21ssh21:29:223
2949399118,2cyclictest16980-21grep19:50:163
2949399118,2cyclictest15997-21taskset19:49:173
2949399118,2cyclictest14476-21head23:20:173
2949399118,2cyclictest13860-21ssh23:58:553
2949399118,2cyclictest13375-21taskset19:41:043
2949399118,2cyclictest12337-21taskset20:48:213
2949399118,2cyclictest11402-21ssh00:35:273
2949399117,3cyclictest7404-21ssh21:20:163
2949399117,2cyclictest10652-21cut22:20:173
2949399116,1cyclictest39-21ksoftirqd/323:50:293
2949399114,5cyclictest39-21ksoftirqd/319:35:153
2949399110,10cyclictest22873-21sed20:00:283
2948999118,2cyclictest9415-21ls20:40:252
2948999118,2cyclictest7774-21ssh21:20:182
2948999118,2cyclictest6978-21ssh23:15:282
2948999118,2cyclictest4886-21tail22:35:262
2948999118,2cyclictest3850-21taskset19:20:572
2948999118,2cyclictest3643-21sort20:30:172
2948999118,2cyclictest30896-21sh23:10:232
2948999118,2cyclictest29807-21cat20:15:242
2948999118,2cyclictest27461-21cstates22:30:162
2948999118,2cyclictest26609-21cat21:50:272
2948999118,2cyclictest22937-21chrt20:00:532
2948999118,2cyclictest20321-21ssh23:24:062
2948999118,2cyclictest11060-21sed22:00:272
2948999111,10cyclictest0-21swapper/200:10:122
2948799119,1cyclictest20172-21grep21:05:161
2948799118,2cyclictest9541-21tr21:40:231
2948799118,2cyclictest5574-21cat20:35:111
2948799118,2cyclictest4513-21cut19:25:151
2948799118,2cyclictest4513-21cut19:25:151
2948799118,2cyclictest4172-21ssh00:31:051
2948799118,2cyclictest31135-21cat19:10:261
2948799118,2cyclictest29911-21sed20:15:251
2948799118,2cyclictest28665-21sh00:26:271
2948799118,2cyclictest21545-21wc22:45:271
2948799118,2cyclictest21002-21cat20:00:011
2948799118,2cyclictest10117-21grep00:15:221
2948799117,3cyclictest21230-21grep20:00:141
2948799116,2cyclictest273-21systemd-udevd19:24:041
2948799114,3cyclictest18001-21df_abs22:05:151
2948399118,2cyclictest8228-21ssh23:55:190
2948399118,2cyclictest6866-21ssh23:35:160
2948399118,2cyclictest4700-21head20:30:270
2948399118,2cyclictest32740-21cut19:15:160
2948399118,2cyclictest30618-21ssh22:12:360
2948399118,2cyclictest29395-21ssh21:14:150
2948399118,2cyclictest27879-21taskset20:13:190
2948399118,2cyclictest23173-21ssh23:05:350
2948399118,2cyclictest23173-21ssh23:05:350
2948399118,2cyclictest20539-21sort21:05:180
2948399118,2cyclictest19696-21ssh22:05:380
2948399118,2cyclictest19472-21ssh23:04:250
2948399118,2cyclictest15474-21ssh00:38:230
2948399118,2cyclictest14065-21wc20:50:220
2948399118,2cyclictest14046-21ssh22:41:000
2948399118,2cyclictest12915-21ssh21:42:370
2948399118,2cyclictest12128-21chrt20:45:390
2948399118,2cyclictest11125-21chrt19:37:550
2948399117,3cyclictest6756-21head23:15:250
2948399116,3cyclictest627-21wc23:50:250
2948399114,6cyclictest17130-21ssh00:20:130
2948399110,9cyclictest0-21swapper/022:00:130
2948399110,1cyclictest0-21swapper/022:26:420
2949399108,1cyclictest31716-21wc20:20:193
2949399108,1cyclictest30650-21sed19:10:193
2949399108,1cyclictest30122-21taskset20:16:333
2949399108,1cyclictest19902-21cut21:05:153
2949399108,1cyclictest126992sleep320:50:023
2949399107,2cyclictest7915-21grep19:30:193
2949399107,2cyclictest736-21grep20:25:133
2949399107,2cyclictest6306-21grep20:35:163
2949399107,2cyclictest18179-21grep21:00:163
2949399107,2cyclictest15818-21grep20:55:163
2948999108,1cyclictest32184-21cut19:15:132
2948999108,1cyclictest30377-21grep19:10:172
2948999108,1cyclictest25224-21ssh23:45:292
2948999108,1cyclictest23949-21expr20:05:142
2948999108,1cyclictest17581-21wget21:00:142
2948999108,1cyclictest11373-21cat20:45:182
2948999107,2cyclictest6317-21head20:35:162
2948999107,2cyclictest16455-21aten2_r7power_v19:50:132
2948999100,9cyclictest15542-21expr20:55:142
2948799108,1cyclictest9965-21sed19:35:171
2948799108,1cyclictest26331-21expr20:10:141
2948799108,1cyclictest16135-21memory20:55:191
2948799107,2cyclictest4976-21taskset20:33:151
2948399108,1cyclictest7823-21cat20:40:000
2948399108,1cyclictest3918-21taskset19:21:470
2948399107,2cyclictest32470-21ssh00:10:040
2948399107,2cyclictest32303-21sed20:20:250
2948399107,2cyclictest29048-21cut20:15:160
294939997,1cyclictest25904-21bc20:10:103
294939997,1cyclictest18893-21egrep19:55:123
294879997,1cyclictest12948-21cat20:50:141
294839996,2cyclictest12643-21sort19:40:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional