You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 05:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Aug 28, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10895210050,45sleep10-21swapper/119:05:551
1136929178,9sleep00-21swapper/019:07:310
1151528764,8sleep20-21swapper/219:09:252
1136228370,9sleep30-21swapper/319:07:253
48032650,0sleep00-21swapper/021:29:350
1163499260,17cyclictest13-21ksoftirqd/023:02:060
11646992322,1cyclictest0-21swapper/222:18:362
1164099230,4cyclictest24782-21head21:40:281
11634992321,1cyclictest32418-21munin-run00:20:010
1164699220,22cyclictest0-21swapper/222:59:142
11634992217,4cyclictest19950-21seq22:35:310
1163499220,1cyclictest0-21swapper/000:21:140
11640992113,5cyclictest25776-21ssh21:22:101
1164099211,18cyclictest25365-21aten2_r7power_v22:20:161
1164999200,7cyclictest0-21swapper/322:10:133
1164099200,1cyclictest0-21swapper/121:18:131
1163499204,13cyclictest0-21swapper/021:35:140
1163499203,16cyclictest0-21swapper/023:45:040
11634992019,1cyclictest161rcuc/021:30:120
11634992018,2cyclictest0-21swapper/021:56:550
1163499200,18cyclictest0-21swapper/022:59:000
1164699190,14cyclictest0-21swapper/223:46:062
1164099190,18cyclictest0-21swapper/122:13:231
1164099190,18cyclictest0-21swapper/100:30:111
1163499197,9cyclictest0-21swapper/000:05:150
1163499197,9cyclictest0-21swapper/000:05:150
1163499196,7cyclictest21147-21munin-run19:30:000
1163499195,9cyclictest4223-21diskmemload00:25:150
1163499195,8cyclictest10420-21perf20:15:000
1163499194,3cyclictest13-21ksoftirqd/021:24:420
11634991918,1cyclictest0-21swapper/023:40:050
1163499191,6cyclictest0-21swapper/021:45:050
11634991916,2cyclictest4223-21diskmemload22:24:060
11634991913,5cyclictest0-21swapper/023:50:140
1163499190,19cyclictest0-21swapper/022:09:190
256792180,1chrt0-21swapper/300:15:203
1164999183,13cyclictest0-21swapper/323:08:403
1163499181,4cyclictest13-21ksoftirqd/021:18:030
11634991813,4cyclictest4223-21diskmemload21:53:040
1163499181,17cyclictest13-21ksoftirqd/022:31:490
1163499181,17cyclictest13-21ksoftirqd/022:31:490
1163499181,14cyclictest13-21ksoftirqd/023:27:210
1164999171,11cyclictest0-21swapper/320:05:133
1164699170,0cyclictest0-21swapper/223:24:422
1164099176,10cyclictest29848-21kworker/u8:023:39:581
11640991715,2cyclictest0-21swapper/122:28:591
11634991715,2cyclictest0-21swapper/022:53:520
11634991715,2cyclictest0-21swapper/022:13:040
11634991714,2cyclictest4223-21diskmemload23:55:190
11634991713,1cyclictest3277-21systemctl20:00:000
1164999169,2cyclictest25823-21apt-get22:40:013
1164699161,5cyclictest225-21systemd-journal22:22:222
11646991614,1cyclictest4223-21diskmemload00:09:422
11646991614,1cyclictest4223-21diskmemload00:09:412
11646991611,4cyclictest15851-21apache200:19:482
1164699160,15cyclictest0-21swapper/221:45:582
1164699160,13cyclictest0-21swapper/200:36:412
1163499163,7cyclictest1994-21awk22:45:010
1163499160,1cyclictest0-21swapper/022:28:420
1164999154,5cyclictest0-21swapper/319:30:133
11646991511,3cyclictest25183-21open_inodes19:35:202
1164699150,15cyclictest0-21swapper/222:36:112
1164099152,12cyclictest0-21swapper/122:40:271
1164099150,1cyclictest0-21swapper/123:43:441
1164099150,15cyclictest0-21swapper/123:32:241
1164099150,14cyclictest0-21swapper/121:32:231
1163499152,13cyclictest4223-21diskmemload23:30:520
11634991513,1cyclictest4223-21diskmemload22:45:160
11634991513,1cyclictest4216-21cron21:10:000
11634991512,2cyclictest17522-21ssh00:10:220
11634991512,2cyclictest17522-21ssh00:10:210
1163499150,14cyclictest0-21swapper/023:07:570
1164999149,3cyclictest27091-21grep22:20:273
1164999141,5cyclictest0-21swapper/323:55:383
11649991411,2cyclictest24018-21munin-plugin-st00:15:013
11649991411,2cyclictest24018-21munin-plugin-st00:15:013
11649991411,2cyclictest15555-21cat00:10:013
11649991410,3cyclictest2395-21wc22:25:243
1164699141,8cyclictest0-21swapper/223:55:142
11646991411,2cyclictest27150-21grep19:40:172
11646991411,2cyclictest22391-21apt-key21:40:002
1164699140,1cyclictest0-21swapper/223:01:462
1164099149,4cyclictest23513-21ls19:35:001
1164099145,5cyclictest31004-21kernelversion21:25:191
11640991412,2cyclictest25-21ksoftirqd/122:15:211
11640991410,3cyclictest38150irq/131-snd_hda21:46:181
1164099140,1cyclictest0-21swapper/122:47:111
1163499147,7cyclictest0-21swapper/000:01:280
11634991410,3cyclictest13-21ksoftirqd/023:13:020
1163499140,1cyclictest0-21swapper/023:15:040
1164999139,3cyclictest15060-21head21:35:183
1164999139,2cyclictest697-21grep00:20:153
11649991310,2cyclictest5005-21tr21:10:143
11649991310,2cyclictest31773-21grep21:45:173
11649991310,2cyclictest28404-21cat23:20:003
11649991310,2cyclictest17445-21cut22:15:163
11649991310,2cyclictest13985-21sed21:15:183
11649991310,2cyclictest13746-21sort20:20:173
1164699139,3cyclictest14511-21grep21:35:152
1164699139,3cyclictest11411-21df22:50:162
1164699139,2cyclictest15367-21uname19:15:202
1164699138,3cyclictest7677-21sh00:24:272
1164699136,3cyclictest15444-21apache_volume21:55:122
11646991310,2cyclictest8959-21grep21:50:272
11646991310,2cyclictest32001-21grep21:00:012
11646991310,2cyclictest28856-21grep19:45:122
11646991310,2cyclictest16758-21cut19:20:122
11646991310,2cyclictest15073-21uniq23:50:162
11646991310,2cyclictest13413-21grep21:15:152
1164699130,12cyclictest0-21swapper/222:42:042
1164699130,12cyclictest0-21swapper/221:28:182
1164099139,4cyclictest38150irq/131-snd_hda21:55:211
1164099139,3cyclictest24262-21grep19:35:141
1164099139,3cyclictest21288-21grep20:35:211
1164099139,1cyclictest225-21systemd-journal00:29:551
1164099138,2cyclictest23569-21cat20:40:191
1164099138,2cyclictest23569-21cat20:40:181
11640991312,1cyclictest0-21swapper/121:53:191
11640991310,2cyclictest6086-21ls21:10:221
11640991310,2cyclictest25-21ksoftirqd/123:20:161
1164099130,1cyclictest0-21swapper/100:09:421
1164099130,1cyclictest0-21swapper/100:09:411
1164099130,12cyclictest0-21swapper/100:39:441
1163499139,3cyclictest21630-21tr19:30:140
1163499139,3cyclictest17028-21tr22:15:140
1163499134,8cyclictest0-21swapper/020:05:250
1163499132,6cyclictest225-21systemd-journal22:05:000
1163499131,6cyclictest4302-21munin-run23:25:000
11634991312,1cyclictest4223-21diskmemload23:36:250
11634991311,1cyclictest21835-21grep20:35:280
11634991310,2cyclictest5760-21wc20:05:000
11634991310,2cyclictest20032-21sort20:35:000
11634991310,2cyclictest17047-21sed20:25:250
11634991310,2cyclictest12363-21grep20:15:250
1164999129,2cyclictest9072-21cut00:25:153
1164999129,2cyclictest28162-21cut20:50:163
1164999129,2cyclictest2795-21grep19:55:253
1164999129,2cyclictest27289-21cut22:40:163
1164999129,2cyclictest23444-21tail21:20:293
1164999129,2cyclictest19044-21grep20:30:223
1164999129,2cyclictest1839-21grep22:05:283
1164999129,2cyclictest17126-21sed20:25:253
1164999129,2cyclictest14803-21df19:15:163
1164999129,2cyclictest12766-21grep23:10:153
1164999128,3cyclictest3569-21basename20:00:133
1164999128,3cyclictest32249-21wc19:50:183
1164999128,3cyclictest25131-21awk00:35:013
1164999128,3cyclictest16236-21grep23:50:263
1164999124,4cyclictest225-21systemd-journal21:41:033
1164999122,4cyclictest516-21gdbus21:05:233
1164999121,3cyclictest225-21systemd-journal23:25:003
1164999120,11cyclictest0-21swapper/321:53:413
1164699129,2cyclictest4217-21cron21:10:012
1164699129,2cyclictest32649-21df21:00:142
1164699129,2cyclictest3209-21grep22:25:312
1164699129,2cyclictest2360-21chrt23:42:372
1164699129,2cyclictest22373-21grep23:15:282
1164699129,2cyclictest22329-21grep23:35:172
1164699129,2cyclictest1896-21sed19:55:162
1164699129,2cyclictest17555-21munin-run20:29:592
1164699128,3cyclictest18025-21uniq00:30:162
1164699120,11cyclictest5985-21grep21:30:132
1164699120,11cyclictest0-21swapper/222:49:562
1164099129,2cyclictest28175-21sed20:50:151
1164099129,2cyclictest26995-21grep19:40:161
1164099129,2cyclictest25187-21grep00:15:161
1164099129,2cyclictest22150-21cron23:55:001
1164099129,2cyclictest22010-21sh21:40:001
1164099129,2cyclictest20420-21sed22:55:201
1164099128,3cyclictest32048-21cat19:50:171
1164099127,4cyclictest25962-21chrt22:00:291
11640991210,1cyclictest422-21grep21:00:171
11640991210,1cyclictest21725-21ssh23:15:221
11640991210,1cyclictest19279-21grep19:25:141
1163499129,2cyclictest31919-21head19:50:150
1163499129,2cyclictest30469-21grep19:45:260
1163499129,2cyclictest14811-21taskset19:15:160
1163499128,3cyclictest25792-21head20:45:170
1163499128,3cyclictest23887-21wc21:40:200
1163499126,4cyclictest0-21swapper/000:38:270
1163499124,4cyclictest225-21systemd-journal21:05:010
11634991211,1cyclictest0-21swapper/000:30:130
1163499120,11cyclictest29494-21sh20:55:000
1164999118,2cyclictest8800-21wc00:05:213
1164999118,2cyclictest7108-21grep21:30:213
1164999118,2cyclictest6277-21wc23:45:153
1164999118,2cyclictest5429-21sed23:25:143
1164999118,2cyclictest32617-21expr21:00:143
1164999118,2cyclictest30563-21ssh23:01:093
1164999118,2cyclictest3027-21ssh22:45:163
1164999118,2cyclictest29846-21ssh21:25:103
1164999118,2cyclictest29835-21ssh23:40:063
1164999118,2cyclictest29369-21grep19:45:143
1164999118,2cyclictest28058-21ssh00:35:463
1164999118,2cyclictest26553-21tr19:40:143
1164999118,2cyclictest26150-21tr20:45:213
1164999118,2cyclictest23614-21ssh22:00:043
1164999118,2cyclictest21632-21grep23:35:133
1164999118,2cyclictest20807-21cut20:35:183
1164999118,2cyclictest20312-21cut22:55:193
1164999118,2cyclictest20260-21grep19:25:213
1164999118,2cyclictest17971-21fgrep19:20:203
1164999118,2cyclictest15446-21ssh21:55:123
1164999118,2cyclictest13251-21ssh23:30:143
1164999118,2cyclictest12377-21ssh22:31:083
1164999118,2cyclictest12377-21ssh22:31:073
1164999118,2cyclictest10679-21ssh22:50:063
1164999116,2cyclictest31877-21munin-run21:00:013
1164999110,10cyclictest31171-21ssh00:00:033
1164699118,2cyclictest9833-21expr22:30:152
1164699118,2cyclictest9833-21expr22:30:142
1164699118,2cyclictest5382-21entropy21:10:162
1164699118,2cyclictest5244-21sed20:00:262
1164699118,2cyclictest32315-21expr00:00:202
1164699118,2cyclictest29458-21taskset20:54:382
1164699118,2cyclictest25423-21expr20:45:152
1164699118,2cyclictest23587-21wc19:35:002
1164699118,2cyclictest22438-21cat21:20:192
1164699118,2cyclictest20470-21tr20:35:152
1164699118,2cyclictest18256-21ssh23:13:452
1164699118,2cyclictest14913-21sort23:30:272
1164699118,2cyclictest1344-21rm22:05:242
1164699118,2cyclictest13323-21tr20:20:152
1164699118,2cyclictest12646-21ssh00:26:492
1164699118,2cyclictest11926-21ssh22:11:462
1164699118,2cyclictest10617-21ssh23:28:232
1164699118,2cyclictest10159-21taskset20:12:042
1164699110,1cyclictest0-21swapper/222:00:432
1164099119,1cyclictest0-21swapper/123:07:241
1164099118,2cyclictest9514-21ssh22:30:131
1164099118,2cyclictest9514-21ssh22:30:121
1164099118,2cyclictest8388-21ls00:25:011
1164099118,2cyclictest6488-21cat23:25:241
1164099118,2cyclictest30030-21grep20:55:151
1164099118,2cyclictest29945-21ssh23:00:381
1164099118,2cyclictest28924-21grep19:45:141
1164099118,2cyclictest2855-21grep19:55:251
1164099118,2cyclictest2848-21grep21:05:181
1164099118,2cyclictest24323-21ssh22:38:551
1164099118,2cyclictest17879-21ssh23:13:271
1164099118,2cyclictest16156-21head20:25:161
1164099118,2cyclictest11019-21basename22:50:151
1164099110,8cyclictest0-21swapper/120:20:131
1164099110,4cyclictest22761-21tr23:55:151
1164099110,3cyclictest0-21swapper/100:00:011
1164099110,1cyclictest0-21swapper/123:45:211
1164099110,11cyclictest0-21swapper/122:08:041
1163499118,2cyclictest5972-21grep21:10:200
1163499118,2cyclictest32041-21wc21:00:010
1163499118,2cyclictest26518-21cut19:40:140
1163499118,2cyclictest24276-21tr19:35:140
1163499118,2cyclictest23809-21awk20:40:220
1163499118,2cyclictest23809-21awk20:40:220
1163499118,2cyclictest13777-21tail20:20:170
1163499115,5cyclictest16622-21/usr/sbin/munin19:20:210
1163499112,8cyclictest11884-21apt-get19:10:120
1164999108,1cyclictest12110-21expr19:10:153
1164999107,2cyclictest12799-21sendmail20:20:003
1164999101,4cyclictest0-21swapper/320:10:133
1164699108,1cyclictest22967-21expr20:40:152
1164699108,1cyclictest22967-21expr20:40:142
1164699108,1cyclictest15775-21cat00:10:022
1164699108,1cyclictest15775-21cat00:10:012
1164699108,1cyclictest13297-21expr19:10:262
1164699108,1cyclictest11277-21cut20:15:162
1164699108,1cyclictest1089-21cut19:55:002
1164699107,2cyclictest7594-21sed20:05:302
1164699107,2cyclictest3524-21sh23:05:102
1164699107,2cyclictest18800-21cat20:30:192
1164099108,1cyclictest7738-21taskset20:06:281
1164099108,1cyclictest18623-21taskset19:23:201
1164099108,1cyclictest11882-21apt19:10:131
1164099108,1cyclictest10621-21chrt20:15:021
1164099107,2cyclictest5488-21taskset20:03:051
1164099107,2cyclictest19884-21ssh00:11:471
1164099107,2cyclictest19884-21ssh00:11:461
1164099100,9cyclictest269862sleep120:48:211
116499996,2cyclictest25391-21tr19:35:223
116499996,2cyclictest22547-21chrt20:40:093
116499996,2cyclictest22547-21chrt20:40:093
116469997,1cyclictest19386-21expr19:25:142
116409997,1cyclictest8867-21diskstats20:10:161
116409994,4cyclictest15132-21cat19:15:181
116409986,1cyclictest18240-21cat20:30:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional