You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 20:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jun 26, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
56729275,13sleep20-21swapper/207:06:352
81629165,10sleep30-21swapper/307:09:493
46329077,9sleep00-21swapper/007:05:320
66328568,13sleep10-21swapper/107:07:501
92299280,27cyclictest0-21swapper/211:00:082
92599260,1cyclictest0-21swapper/311:23:113
92299260,26cyclictest0-21swapper/210:19:592
925992524,1cyclictest0-21swapper/310:41:543
92599242,20cyclictest24716-21grep10:25:253
91799230,1cyclictest0-21swapper/110:12:001
91399230,22cyclictest0-21swapper/009:00:120
92299220,2cyclictest0-21swapper/211:40:112
922992117,3cyclictest0-21swapper/209:55:182
92299210,1cyclictest0-21swapper/212:11:292
91799212,1cyclictest21254-21wc12:00:201
92599200,0cyclictest0-21swapper/312:11:413
922992017,2cyclictest32-21ksoftirqd/209:40:582
922991913,5cyclictest0-21swapper/211:10:242
92299191,17cyclictest28001-21ssh12:05:042
92299190,19cyclictest0-21swapper/211:52:292
92299190,17cyclictest0-21swapper/209:30:152
91799190,1cyclictest0-21swapper/107:25:181
91399190,15cyclictest0-21swapper/009:30:130
925991817,1cyclictest0-21swapper/312:02:223
92599180,15cyclictest0-21swapper/308:00:123
92299182,15cyclictest0-21swapper/212:20:132
92299181,1cyclictest0-21swapper/208:25:132
92299180,1cyclictest0-21swapper/210:52:202
92299180,18cyclictest0-21swapper/212:28:522
92299180,18cyclictest0-21swapper/209:53:192
92299180,18cyclictest0-21swapper/209:49:122
92299180,16cyclictest0-21swapper/210:42:472
917991816,1cyclictest13368-21cstates07:35:131
92299172,15cyclictest0-21swapper/211:25:122
92299172,15cyclictest0-21swapper/211:25:122
92299170,17cyclictest0-21swapper/209:28:012
91799172,10cyclictest9729-21latency_hist11:35:011
91399176,10cyclictest0-21swapper/010:45:140
91399170,1cyclictest0-21swapper/010:56:490
92299164,1cyclictest141rcu_preempt11:56:412
92299160,16cyclictest0-21swapper/211:45:052
917991615,1cyclictest26630-21diskmemload10:22:041
91799160,1cyclictest0-21swapper/110:58:441
91799160,0cyclictest0-21swapper/112:18:151
925991515,0cyclictest0-21swapper/310:08:443
925991515,0cyclictest0-21swapper/310:08:443
92299156,3cyclictest0-21swapper/208:10:002
92299151,3cyclictest21426-21kworker/u8:311:39:402
92299150,13cyclictest0-21swapper/211:22:152
91799158,6cyclictest16209-21grep10:40:161
91799155,9cyclictest38150irq/131-snd_hda09:15:141
91399158,6cyclictest12817-21awk07:35:000
91399157,7cyclictest31804-21runrttasks12:25:500
91399154,10cyclictest3074-21ssh09:15:130
91399150,2cyclictest0-21swapper/010:24:330
925991413,1cyclictest0-21swapper/309:34:313
925991411,2cyclictest32332-21wc10:30:173
925991411,2cyclictest30316-21wc09:50:193
925991411,2cyclictest19790-21sh09:44:363
925991411,2cyclictest10434-21turbostat07:29:593
925991410,3cyclictest27279-21ssh09:10:143
925991410,3cyclictest18183-21ls11:40:003
925991410,2cyclictest429-21tail10:50:163
92299141,9cyclictest14966-21/usr/sbin/munin10:00:222
922991410,3cyclictest5402-21tr12:30:142
91799148,5cyclictest14894-21ls10:40:011
917991410,3cyclictest0-21swapper/111:06:371
91799140,13cyclictest0-21swapper/109:43:491
91399141,7cyclictest4909-21basename08:25:010
91399141,12cyclictest26630-21diskmemload11:47:030
913991410,3cyclictest15179-21awk07:40:000
92599139,3cyclictest30169-21grep12:05:273
92599139,3cyclictest26630-21diskmemload10:04:023
92599139,3cyclictest2106-21tr11:30:163
92599133,4cyclictest225-21systemd-journal09:39:253
925991310,2cyclictest31000-21sed10:10:173
925991310,2cyclictest29554-21tr12:25:143
925991310,2cyclictest29092-21wc08:05:233
925991310,2cyclictest1414-21tr07:10:273
925991310,2cyclictest11694-21tr09:20:133
925991310,2cyclictest1130-21cut08:15:163
925991310,2cyclictest10791-21grep07:30:153
92599130,11cyclictest0-21swapper/311:42:223
92299139,3cyclictest24249-21awk09:05:002
92299139,3cyclictest20458-21ssh12:00:152
92299138,4cyclictest18452-21expr07:45:152
92299133,8cyclictest31149-21ssh10:30:102
922991310,2cyclictest9728-21wc07:25:232
922991310,2cyclictest3378-21grep07:15:002
922991310,2cyclictest2648-21grep11:30:182
92299130,1cyclictest141rcu_preempt10:21:042
92299130,13cyclictest0-21swapper/212:17:182
92299130,12cyclictest15431-21grep12:35:262
92299130,0cyclictest0-21swapper/210:06:372
92299130,0cyclictest0-21swapper/210:06:362
91799139,3cyclictest2686-21ssh11:50:001
91799139,3cyclictest26156-21tr11:25:141
91799139,3cyclictest26156-21tr11:25:131
91799139,3cyclictest12185-21tail09:20:151
91799132,11cyclictest23786-21ssh11:42:491
917991313,0cyclictest0-21swapper/112:08:021
917991310,2cyclictest5386-21ssh09:55:161
917991310,2cyclictest22341-21head12:20:221
917991310,2cyclictest18766-21grep07:45:181
917991310,2cyclictest18167-21grep08:50:211
917991310,2cyclictest13944-21ls10:20:001
91799130,12cyclictest23133-21cut10:25:131
91799130,12cyclictest0-21swapper/112:36:361
91799130,12cyclictest0-21swapper/109:11:271
913991313,0cyclictest0-21swapper/009:45:130
913991311,2cyclictest13-21ksoftirqd/010:02:170
913991310,2cyclictest645-21grep08:15:140
913991310,2cyclictest6370-21awk12:30:180
913991310,2cyclictest26001-21sed08:00:140
913991310,2cyclictest13921-21sed09:40:210
91399130,12cyclictest28875-21cat12:25:010
92599129,2cyclictest7041-21tail12:30:253
92599129,2cyclictest31194-21grep08:10:163
92599129,2cyclictest21689-21wc09:00:003
92599128,3cyclictest8289-21cut08:30:173
92599128,3cyclictest27125-21ssh11:45:133
92599128,3cyclictest11409-21cat08:35:253
92599126,3cyclictest13461-21ssh12:35:123
92599125,4cyclictest0-21swapper/309:45:123
92599120,12cyclictest0-21swapper/311:15:343
92599120,12cyclictest0-21swapper/311:09:453
92299129,2cyclictest31180-21grep08:10:162
92299129,2cyclictest20958-21df07:50:142
92299129,2cyclictest16386-21grep07:40:162
92299129,2cyclictest11664-21tr09:20:132
92299129,2cyclictest1160-21grep08:15:162
92299129,2cyclictest10389-21grep08:35:152
92299128,3cyclictest13348-21tr07:35:132
92299125,6cyclictest21780-21apache210:38:252
92299121,6cyclictest25217-21cpuspeed_turbos11:05:142
92299120,12cyclictest0-21swapper/210:56:292
92299120,11cyclictest4299-21cat09:15:202
91799129,2cyclictest9190-21tail08:30:271
91799129,2cyclictest5305-21grep11:50:381
91799129,2cyclictest29366-21tr09:30:201
91799129,2cyclictest27938-21tr08:05:141
91799129,2cyclictest21511-21cron09:00:001
91799129,2cyclictest17771-21cron07:45:001
91799129,2cyclictest1544-21grep10:50:261
91799129,2cyclictest15370-21tr08:45:171
91799128,3cyclictest3972-21ls08:20:221
91799128,1cyclictest13363-21ssh10:00:081
91799121,6cyclictest1277-21cstates11:10:141
917991210,1cyclictest20839-21apt-get09:45:131
917991210,1cyclictest17184-21df11:00:161
91799120,1cyclictest0-21swapper/109:25:131
91799120,11cyclictest9287-21tr11:15:141
91799120,11cyclictest4430-21tr09:35:121
91399129,2cyclictest9984-21sed07:25:240
91399129,2cyclictest30999-21grep10:10:160
91399129,2cyclictest2354-21sshd11:10:220
91399129,2cyclictest19342-21grep11:40:140
91399129,2cyclictest19237-21ls07:45:230
91399129,2cyclictest17408-21grep11:00:170
91399129,2cyclictest13882-21ls12:15:210
91399129,2cyclictest12943-21cron10:00:010
91399129,2cyclictest11754-21tr11:55:130
91399128,3cyclictest9191-21ssh11:15:120
91399128,3cyclictest19991-21tr09:25:140
91399128,3cyclictest12574-21cat09:20:180
91399125,6cyclictest31496-21ssh11:28:170
91399125,6cyclictest31496-21ssh11:28:160
91399123,4cyclictest26630-21diskmemload12:10:220
91399120,12cyclictest0-21swapper/010:09:260
91399120,12cyclictest0-21swapper/010:09:260
91399120,11cyclictest30693-21sleep10:29:590
91399120,10cyclictest0-21swapper/010:40:140
230152120,0chrt0-21swapper/107:55:031
92599119,1cyclictest3873-21grep09:15:163
92599118,2cyclictest7904-21sh09:56:023
92599118,2cyclictest7555-21cat10:35:153
92599118,2cyclictest7466-21ssh11:52:143
92599118,2cyclictest6234-21wc07:20:153
92599118,2cyclictest29686-21ssh11:26:503
92599118,2cyclictest29686-21ssh11:26:503
92599118,2cyclictest26188-21ssh10:45:463
92599118,2cyclictest25206-21tail09:05:153
92599118,2cyclictest23084-21cut07:55:143
92599118,2cyclictest20951-21sh12:20:133
92599118,2cyclictest20659-21tr07:50:133
92599118,2cyclictest18934-21taskset08:52:253
92599118,2cyclictest18692-21sort07:45:173
92599118,2cyclictest17449-21taskset07:40:573
92599118,2cyclictest1655-21grep11:10:183
92599118,2cyclictest16508-21basename11:00:103
92599118,2cyclictest14919-21tr08:45:153
92599118,2cyclictest13291-21tail12:15:163
92599118,2cyclictest13023-21grep08:40:163
92599118,2cyclictest11202-21ssh10:17:523
92599117,3cyclictest9633-21wc10:55:203
92599110,3cyclictest0-21swapper/310:20:143
92599110,10cyclictest11253-21ssh11:55:023
92299118,2cyclictest9629-21cron08:35:012
92299118,2cyclictest9313-21cut11:15:142
92299118,2cyclictest7808-21chrt07:22:062
92299118,2cyclictest5692-21sleep07:19:592
92299118,2cyclictest3524-21grep08:20:172
92299118,2cyclictest31455-21ssh09:12:152
92299118,2cyclictest25877-21cut09:05:212
92299118,2cyclictest24042-21grep10:45:162
92299118,2cyclictest23332-21cut07:55:162
92299118,2cyclictest23013-21egrep10:25:132
92299118,2cyclictest21212-21taskset08:56:132
92299118,2cyclictest18723-21head08:50:272
92299118,2cyclictest1601-21ssh10:11:532
92299118,2cyclictest14281-21taskset08:43:382
92299111,8cyclictest4732-21df_abs09:35:132
91799118,2cyclictest362-21grep08:15:131
91799118,2cyclictest31149-21ssh10:30:101
91799118,2cyclictest26000-21df08:00:141
91799118,2cyclictest23484-21grep09:00:241
91799118,2cyclictest21667-21wc07:50:201
91799118,2cyclictest20474-21ssh11:21:101
91799118,2cyclictest19299-21cron12:00:011
91799118,2cyclictest12377-21vmstat11:35:271
91799118,2cyclictest12074-21taskset07:30:241
91799118,2cyclictest11061-21ntpq08:35:211
91799116,4cyclictest5782-21cut12:30:161
91799114,6cyclictest4648-21grep07:15:231
91799114,5cyclictest7254-21cron08:30:001
91799110,1cyclictest0-21swapper/110:07:501
91799110,1cyclictest0-21swapper/110:07:491
91399118,2cyclictest9625-21cron08:35:000
91399118,2cyclictest9406-21ssh11:53:460
91399118,2cyclictest8171-21ssh10:15:290
91399118,2cyclictest7251-21cron08:30:000
91399118,2cyclictest613-21sed10:30:240
91399118,2cyclictest5277-21taskset09:35:160
91399118,2cyclictest30652-21ssh09:11:380
91399118,2cyclictest30006-21ssh09:50:160
91399118,2cyclictest2985-21ssh10:51:280
91399118,2cyclictest29098-21head12:05:170
91399118,2cyclictest24973-21expr09:05:140
91399118,2cyclictest23845-21head07:55:180
91399118,2cyclictest21631-21head07:50:200
91399118,2cyclictest21473-21taskset08:59:330
91399118,2cyclictest20992-21ssh11:21:350
91399118,2cyclictest19087-21ssh12:38:180
91399118,2cyclictest14894-21grep08:45:150
91399118,2cyclictest12079-21cat11:35:250
91399118,2cyclictest11956-21taskset08:39:290
91399118,2cyclictest10721-21ssh10:36:440
92599108,1cyclictest5518-21expr08:25:143
92599108,1cyclictest54462sleep307:17:303
92599107,2cyclictest22129-21cut09:00:143
92599107,2cyclictest20238-21ssh09:25:163
92599107,2cyclictest13797-21tail07:35:163
92299108,1cyclictest25465-21cut08:00:132
92299108,1cyclictest15524-21cat08:45:182
92299108,1cyclictest11713-21seq07:30:222
91799108,1cyclictest6382-21expr07:20:141
91799108,1cyclictest3217-21chrt07:14:281
91799108,1cyclictest29617-21ls09:50:141
91799107,2cyclictest7347-21ssh12:11:221
91799107,2cyclictest31184-21head08:10:161
91799107,2cyclictest28337-21ssh10:47:271
91799107,2cyclictest14048-21seq08:40:381
91799102,4cyclictest225-21systemd-journal12:25:001
91399108,1cyclictest29287-21cat08:05:260
91399108,1cyclictest21886-21vmstat12:00:260
91399108,1cyclictest17329-21cut08:50:140
91399107,2cyclictest30399-21basename08:10:120
91399107,2cyclictest25330-21expr11:05:140
91399107,2cyclictest12916-21grep08:40:160
91399106,3cyclictest17338-21grep07:40:260
91399100,9cyclictest2160-21cat11:30:150
9259996,2cyclictest3030-21grep08:20:133
9179997,1cyclictest26586-21chrt09:09:521
9139997,1cyclictest35042sleep007:15:060
9139997,1cyclictest1413-21grep07:10:280
9139996,2cyclictest6486-21grep07:20:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional