You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 00:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Apr 11, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3191129986,9sleep00-21swapper/007:08:230
2990028968,16sleep10-21swapper/107:05:021
3178428745,38sleep20-21swapper/207:06:432
3169828370,9sleep30-21swapper/307:05:363
142052450,0sleep30-21swapper/308:45:213
3212999310,18cyclictest0-21swapper/309:57:073
3212999290,28cyclictest0-21swapper/311:25:533
3212999280,21cyclictest0-21swapper/312:35:143
32118992827,1cyclictest0-21swapper/009:30:080
3212999271,25cyclictest24657-21diskmemload09:29:543
3212999271,24cyclictest0-21swapper/310:28:483
3212999262,23cyclictest24657-21diskmemload09:44:243
3212999257,17cyclictest24657-21diskmemload09:51:223
3212999250,24cyclictest1858-21ssh11:30:183
3212999250,20cyclictest24657-21diskmemload09:22:383
32118992512,5cyclictest13-21ksoftirqd/012:10:250
3212999240,22cyclictest0-21swapper/309:47:593
3212599245,9cyclictest4072-21turbostat.cron10:15:002
3212599240,23cyclictest0-21swapper/211:00:102
3212399240,22cyclictest0-21swapper/111:41:511
32118992423,1cyclictest13-21ksoftirqd/010:52:530
32118992422,1cyclictest0-21swapper/009:26:430
3212999234,13cyclictest24657-21diskmemload09:30:133
3212999230,22cyclictest0-21swapper/309:16:023
3212999230,21cyclictest0-21swapper/311:06:303
3212999230,19cyclictest24657-21diskmemload11:58:343
3212999230,18cyclictest0-21swapper/311:00:143
3212999230,17cyclictest0-21swapper/310:56:123
3212599230,1cyclictest0-21swapper/209:30:522
32123992321,1cyclictest12977-21apt-get10:20:131
3212999222,19cyclictest0-21swapper/312:00:133
3212999222,19cyclictest0-21swapper/312:00:123
3212999220,21cyclictest0-21swapper/310:31:223
3212999220,19cyclictest0-21swapper/310:45:143
3212999220,16cyclictest0-21swapper/310:22:463
32118992220,1cyclictest0-21swapper/010:10:140
3211899220,3cyclictest0-21swapper/009:11:380
3211899220,17cyclictest0-21swapper/010:28:030
214122220,0sleep10-21swapper/111:03:361
107432220,4sleep339-21ksoftirqd/310:37:493
3212999216,14cyclictest26481-21ssh11:45:143
3212999214,14cyclictest24657-21diskmemload12:12:193
32129992113,7cyclictest0-21swapper/312:32:593
3212999211,19cyclictest24657-21diskmemload10:40:413
3212999211,17cyclictest0-21swapper/310:03:283
3212999210,20cyclictest0-21swapper/312:28:123
3212599210,21cyclictest0-21swapper/211:10:092
32118992119,2cyclictest13-21ksoftirqd/011:14:220
3211899211,7cyclictest13-21ksoftirqd/011:50:080
32118992113,3cyclictest13-21ksoftirqd/012:37:240
3211899210,1cyclictest0-21swapper/012:27:590
3211899210,19cyclictest0-21swapper/009:48:560
3212999203,16cyclictest10211-21sendmail09:39:593
3212999200,9cyclictest0-21swapper/309:14:553
3212999200,18cyclictest0-21swapper/312:05:173
3212999200,16cyclictest24657-21diskmemload12:15:373
3212599200,20cyclictest0-21swapper/211:18:372
3212599200,1cyclictest141rcu_preempt11:48:122
32123992020,0cyclictest0-21swapper/110:56:141
3212399200,20cyclictest0-21swapper/111:19:411
3211899201,4cyclictest26117-21ssh11:45:130
3211899200,2cyclictest0-21swapper/011:33:080
3212999194,14cyclictest24657-21diskmemload11:53:343
3212999193,14cyclictest0-21swapper/311:23:253
3212999190,18cyclictest24657-21diskmemload11:41:213
3212999190,18cyclictest0-21swapper/312:20:313
3212999190,18cyclictest0-21swapper/310:05:133
3212599195,1cyclictest141rcu_preempt09:48:082
32125991919,0cyclictest0-21swapper/210:05:542
3212599190,1cyclictest0-21swapper/210:33:232
3212399197,3cyclictest6288-21switchtime12:30:241
3212399190,2cyclictest0-21swapper/110:39:001
3212399190,0cyclictest0-21swapper/107:15:031
3211899199,2cyclictest13-21ksoftirqd/009:22:390
3211899191,5cyclictest13-21ksoftirqd/009:38:580
3212999181,16cyclictest55650irq/127-enp2s0-10:19:523
3212999180,18cyclictest0-21swapper/311:15:203
3212999180,17cyclictest0-21swapper/310:10:243
3212999180,16cyclictest0-21swapper/308:50:273
3212999180,16cyclictest0-21swapper/307:35:203
3212999180,13cyclictest0-21swapper/311:10:033
3212599184,0cyclictest141rcu_preempt11:33:152
32125991818,0cyclictest0-21swapper/211:57:542
3212399180,18cyclictest0-21swapper/109:52:081
3212399180,17cyclictest0-21swapper/112:10:101
3212399180,0cyclictest0-21swapper/109:38:191
32118991816,1cyclictest161rcuc/010:38:090
32118991816,1cyclictest0-21swapper/010:09:130
3211899180,5cyclictest0-21swapper/012:02:570
3211899180,5cyclictest0-21swapper/012:02:560
3212999170,17cyclictest0-21swapper/311:38:183
3212999170,16cyclictest24657-21diskmemload10:54:583
3212999170,15cyclictest0-21swapper/308:10:133
3212999170,14cyclictest0-21swapper/308:55:143
3212599179,5cyclictest27802-21apache211:53:362
3212599171,15cyclictest0-21swapper/211:25:002
3212599170,2cyclictest0-21swapper/212:33:592
3212599170,2cyclictest0-21swapper/210:43:092
3212599170,16cyclictest0-21swapper/212:06:272
3212599170,12cyclictest141rcu_preempt12:13:312
32123991716,1cyclictest0-21swapper/111:47:001
32123991716,1cyclictest0-21swapper/110:13:011
3211899173,7cyclictest28511-21munin-run08:09:590
32118991714,2cyclictest13-21ksoftirqd/010:56:270
32118991713,2cyclictest13-21ksoftirqd/011:42:490
3211899170,3cyclictest0-21swapper/011:01:250
3211899170,1cyclictest0-21swapper/012:23:240
3212599165,1cyclictest141rcu_preempt11:26:412
3212599163,7cyclictest225-21systemd-journal07:20:012
32125991615,1cyclictest291irq_work/212:21:092
3212599160,1cyclictest0-21swapper/210:35:112
3212599160,16cyclictest0-21swapper/209:25:442
3212399160,1cyclictest0-21swapper/111:08:301
3212399160,15cyclictest0-21swapper/109:49:591
3212399160,0cyclictest0-21swapper/111:10:501
3211899167,3cyclictest13-21ksoftirqd/007:54:590
3211899164,6cyclictest8005-21basename08:35:000
3211899163,1cyclictest141rcu_preempt07:35:190
3211899161,7cyclictest13-21ksoftirqd/011:28:350
32118991612,4cyclictest13-21ksoftirqd/011:20:200
32118991611,2cyclictest141rcu_preempt11:15:170
32118991610,5cyclictest13-21ksoftirqd/012:33:430
3211899160,1cyclictest0-21swapper/010:18:580
3211899160,15cyclictest0-21swapper/012:08:230
3211899160,13cyclictest0-21swapper/008:52:280
3212999159,5cyclictest9054-21taskset07:25:323
3212999150,14cyclictest0-21swapper/308:20:153
3212999150,13cyclictest0-21swapper/307:15:013
3212999150,12cyclictest0-21swapper/309:00:133
3212599152,1cyclictest1734-21taskset07:11:062
32125991512,2cyclictest32-21ksoftirqd/212:19:182
3212599150,1cyclictest141rcu_preempt12:02:482
3212599150,1cyclictest141rcu_preempt12:02:472
3212599150,14cyclictest0-21swapper/210:29:582
32123991515,0cyclictest0-21swapper/112:38:101
3211899152,11cyclictest13-21ksoftirqd/010:41:520
3211899151,2cyclictest141rcu_preempt09:57:100
3211899151,1cyclictest141rcu_preempt10:48:340
3211899150,6cyclictest0-21swapper/010:33:550
3211899150,1cyclictest0-21swapper/007:30:280
16732150,0chrt0-21swapper/109:34:521
3212999148,5cyclictest3559-21wc07:15:193
3212999148,5cyclictest16313-21chrt07:42:023
3212999148,5cyclictest11740-21taskset07:34:343
3212999147,6cyclictest12002-21cat08:40:233
32129991410,3cyclictest26576-21tr08:05:163
3212999140,13cyclictest3878-21expr08:25:133
3212999140,12cyclictest0-21swapper/309:05:133
3212999140,11cyclictest0-21swapper/307:20:113
3212599145,2cyclictest141rcu_preempt10:15:142
3212599145,1cyclictest21202-21sh11:41:372
3212599142,7cyclictest141rcu_preempt09:12:042
3212599142,1cyclictest4349-21ssh10:53:202
3212599142,12cyclictest0-21swapper/210:45:592
3212599140,6cyclictest13540-21sh10:58:592
3212599140,1cyclictest0-21swapper/211:35:122
3212399142,11cyclictest24657-21diskmemload10:50:451
32123991410,3cyclictest24535-21grep10:45:271
3212399140,2cyclictest0-21swapper/111:27:331
3211899145,6cyclictest13-21ksoftirqd/009:20:000
32118991412,2cyclictest13-21ksoftirqd/009:43:250
3211899141,1cyclictest141rcu_preempt07:45:320
32118991410,4cyclictest13-21ksoftirqd/009:52:030
32118991410,3cyclictest4102-21wc07:15:270
32118991410,3cyclictest13-21ksoftirqd/011:36:470
3211899140,1cyclictest0-21swapper/010:00:330
3212999131,11cyclictest1-21systemd08:30:243
3212999131,11cyclictest0-21swapper/308:00:143
32129991310,2cyclictest31869-21grep08:15:183
3212599138,2cyclictest4527-21cat08:25:172
32125991312,1cyclictest0-21swapper/211:09:132
3212599131,1cyclictest141rcu_preempt08:55:142
32125991310,2cyclictest880-21cat08:20:002
32125991310,2cyclictest5697-21grep07:20:162
32125991310,2cyclictest5295-21wc09:55:272
32125991310,2cyclictest28510-21sh08:09:592
32125991310,2cyclictest21662-21cat09:00:252
32125991310,2cyclictest10326-21grep07:30:162
3212399138,2cyclictest13868-21grep10:00:261
32123991310,2cyclictest5339-21grep10:15:171
32123991310,2cyclictest24227-21cron11:24:591
3212399130,1cyclictest0-21swapper/109:11:591
3211899137,2cyclictest13-21ksoftirqd/012:15:150
3211899134,7cyclictest13-21ksoftirqd/011:55:470
3211899132,5cyclictest13-21ksoftirqd/011:05:020
32118991310,2cyclictest9854-21grep08:35:280
32118991310,2cyclictest14584-21wc08:45:260
3212999129,2cyclictest22342-21sed07:55:183
3212999127,4cyclictest19906-21grep07:50:183
3212999120,11cyclictest9572-21fgrep08:35:233
3212599129,2cyclictest7292-21cat08:30:272
3212599129,2cyclictest25199-21tr08:00:232
3212599129,2cyclictest24235-21sed09:05:262
3212599129,2cyclictest17516-21grep07:45:172
32125991212,0cyclictest0-21swapper/209:44:322
32125991211,1cyclictest32-21ksoftirqd/212:25:432
3212599120,11cyclictest29726-21cat08:10:192
3212599120,11cyclictest29587-21sed09:50:272
3212599120,11cyclictest0-21swapper/209:22:472
3212399129,2cyclictest9424-21perf07:30:011
3212399129,2cyclictest7734-21chrt09:19:051
3212399129,2cyclictest5684-21sed07:20:161
3212399129,2cyclictest1195-21grep08:20:161
3212399129,2cyclictest11522-21head09:40:171
3212399129,2cyclictest10478-21grep09:20:181
3212399128,3cyclictest29600-21ssh12:25:171
3212399128,3cyclictest24487-21taskset09:08:101
3212399124,6cyclictest11768-21/usr/sbin/munin08:40:201
3212399123,6cyclictest15393-21apt-get08:50:141
3212399121,8cyclictest38150irq/131-snd_hda08:30:141
3212399120,12cyclictest0-21swapper/110:34:131
3212399120,12cyclictest0-21swapper/109:59:261
3212399120,11cyclictest0-21swapper/112:00:101
3212399120,11cyclictest0-21swapper/112:00:091
3211899129,2cyclictest11961-21wc08:40:220
3212999111,5cyclictest0-21swapper/307:45:143
3212599118,2cyclictest22719-21cat07:55:222
3212599118,2cyclictest16391-21ssh12:36:322
3212599118,2cyclictest14472-21ssh10:00:512
32125991110,1cyclictest8276-21kworker/u8:210:20:172
3212399118,2cyclictest9642-21sshd07:30:081
3212399118,2cyclictest4724-21grep08:25:201
3212399118,2cyclictest3680-21cut11:50:241
3212399118,2cyclictest28806-21fgrep12:05:231
3212399118,2cyclictest28499-21latency_hist08:09:591
3212399118,2cyclictest2648-21rm11:30:271
3212399118,2cyclictest24675-21head08:00:181
3212399118,2cyclictest23015-21ssh10:25:261
3212399118,2cyclictest22414-21ssh12:20:371
3212399118,2cyclictest2071-21munin-setwarn07:15:001
3212399118,2cyclictest20302-21ssh10:05:121
3212399118,2cyclictest15035-21fschecks_time10:40:161
3212399118,2cyclictest14711-21sed08:45:271
3212399118,2cyclictest14148-21ssh12:15:431
3212399115,5cyclictest11646-21ssh11:55:201
3212399110,10cyclictest8561-21cat08:35:161
3212399110,0cyclictest0-21swapper/109:28:431
3211899119,1cyclictest1779-21sort08:20:190
3211899118,2cyclictest5540-21chrt08:29:150
3211899118,2cyclictest28837-21cut08:10:150
3211899118,2cyclictest25063-21grep08:00:220
3211899118,2cyclictest23174-21sed07:55:260
3211899118,2cyclictest22551-21cut09:05:130
3211899118,2cyclictest15230-21cut07:40:170
3211899118,2cyclictest14180-21ssh10:20:190
3211899117,2cyclictest13-21ksoftirqd/008:20:000
3211899117,1cyclictest6710-21taskset07:20:560
3211899112,4cyclictest0-21swapper/009:00:170
3211899110,1cyclictest0-21swapper/007:10:140
3212599108,2cyclictest4959-21ssh09:35:542
3212599108,2cyclictest3418-21ssh09:15:402
3212599108,1cyclictest9931-21grep08:35:272
3212599107,2cyclictest14732-21sed08:45:272
3212399108,1cyclictest15365-21irqstats07:40:181
3212399108,1cyclictest12133-21cut07:35:151
3212399107,2cyclictest9174-21ssh11:35:051
3212399107,2cyclictest31605-21cat08:15:151
3212399107,2cyclictest28775-21taskset08:10:141
3212399107,2cyclictest22017-21seq09:01:571
3212399107,2cyclictest20884-21grep07:50:281
3212399106,3cyclictest21794-21grep07:55:161
3212399106,3cyclictest18900-21tr08:55:211
3211899101,5cyclictest0-21swapper/007:25:160
321259998,1cyclictest7714-21expr07:25:162
321259998,1cyclictest16527-21ntpq08:50:212
321259997,2cyclictest19601-21expr07:50:162
321259997,2cyclictest11498-21if_enp0s31f608:40:182
321259997,1cyclictest1849-21grep08:20:192
321259997,1cyclictest14680-21grep07:40:142
321259997,1cyclictest12303-21cut07:35:162
321239997,1cyclictest17222-21expr07:45:161
321189990,1cyclictest141rcu_preempt08:55:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional