You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 18:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 10, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2831828865,19sleep20-21swapper/207:05:232
2894128568,13sleep10-21swapper/107:05:421
2913228161,16sleep30-21swapper/307:08:133
2714528169,8sleep00-21swapper/007:05:090
2936599250,24cyclictest0-21swapper/310:57:273
166582250,0sleep20-21swapper/207:50:142
2935999237,1cyclictest141rcu_preempt11:11:401
2936599220,2cyclictest141rcu_preempt12:28:323
2936599220,21cyclictest0-21swapper/310:47:193
29354992222,0cyclictest0-21swapper/010:21:520
2935999211,1cyclictest17171-21cpuspeed_turbos09:45:131
2935999211,19cyclictest11986-21latency_hist12:35:011
2935499210,14cyclictest0-21swapper/008:20:140
2936599200,1cyclictest0-21swapper/309:25:133
29354992013,6cyclictest63562sleep012:30:430
2935499200,1cyclictest0-21swapper/011:15:230
2936399190,19cyclictest0-21swapper/209:12:382
2935999192,2cyclictest22190-21diskmemload12:02:311
2935999192,2cyclictest22190-21diskmemload12:02:301
2935499190,1cyclictest0-21swapper/010:39:540
29365991811,6cyclictest11819-21apache210:10:543
2936599180,3cyclictest0-21swapper/312:23:073
2936399182,2cyclictest141rcu_preempt11:00:122
29363991817,1cyclictest0-21swapper/210:18:472
29363991817,1cyclictest0-21swapper/210:18:462
29359991816,2cyclictest0-21swapper/112:10:571
2935999180,16cyclictest22190-21diskmemload09:18:031
29365991715,1cyclictest32745-21apt-get11:30:133
2936599170,16cyclictest0-21swapper/309:10:123
2936399170,13cyclictest0-21swapper/211:45:042
2935999170,1cyclictest0-21swapper/110:49:421
2935499179,8cyclictest1035-21packagekitd11:13:340
2935499170,1cyclictest0-21swapper/010:19:510
2935499170,1cyclictest0-21swapper/010:19:500
2936599162,8cyclictest0-21swapper/312:00:173
2936599162,8cyclictest0-21swapper/312:00:163
2936599160,16cyclictest0-21swapper/310:40:373
2936399168,6cyclictest11825-21apache211:34:062
29359991613,2cyclictest38150irq/131-snd_hda10:40:591
2935999160,1cyclictest0-21swapper/110:59:571
2935999160,15cyclictest0-21swapper/112:23:041
2935499161,14cyclictest0-21swapper/012:17:280
2935499160,1cyclictest0-21swapper/012:00:190
2935499160,1cyclictest0-21swapper/012:00:180
2935499160,1cyclictest0-21swapper/010:06:560
2935499160,16cyclictest0-21swapper/011:45:380
253282160,1chrt25329-21/usr/sbin/munin09:50:122
29365991514,1cyclictest55850irq/129-enp2s0-12:19:033
29365991514,1cyclictest0-21swapper/310:26:183
2936599151,10cyclictest0-21swapper/311:53:083
2936599150,1cyclictest0-21swapper/310:54:413
2936599150,15cyclictest0-21swapper/311:15:343
29363991511,3cyclictest0-21swapper/211:12:492
2936399150,3cyclictest0-21swapper/209:56:012
2936399150,14cyclictest0-21swapper/210:07:272
2935999159,2cyclictest2373-21grep11:50:151
2935999158,6cyclictest38150irq/131-snd_hda12:16:321
2935999151,13cyclictest19102-21ssh11:02:331
29359991510,3cyclictest13370-21grep10:20:251
2935999150,13cyclictest23666-21cat09:30:001
2935499153,1cyclictest141rcu_preempt10:56:420
29354991514,1cyclictest0-21swapper/012:12:300
79062142,1sleep20-21swapper/208:40:112
29365991411,2cyclictest19234-21sh10:05:173
29365991410,3cyclictest0-21swapper/312:35:123
2936599140,13cyclictest9273-21cron10:00:003
2936399148,2cyclictest307722sleep212:07:002
29363991411,3cyclictest29345-21cyclictest10:43:052
29363991410,3cyclictest3897-21ls10:35:012
29363991410,3cyclictest30918-21turbostat11:10:002
2935999146,1cyclictest22190-21diskmemload10:39:331
2935999142,1cyclictest22190-21diskmemload11:28:191
29359991413,1cyclictest0-21swapper/111:41:111
2935999141,12cyclictest22190-21diskmemload09:20:111
2935499145,6cyclictest7612-21cron08:40:010
29354991413,1cyclictest0-21swapper/009:29:330
29354991413,1cyclictest0-21swapper/007:15:130
29354991412,2cyclictest13-21ksoftirqd/009:34:520
2935499140,1cyclictest0-21swapper/009:55:220
239392140,0sleep20-21swapper/211:44:042
184672140,2sleep20-21swapper/209:26:082
2936599139,3cyclictest3482-21sh10:15:143
2936599139,3cyclictest3482-21sh10:15:143
2936599139,3cyclictest24725-21wc08:05:193
2936599139,3cyclictest23396-21cron08:04:593
2936599139,3cyclictest19301-21grep09:00:253
2936599131,11cyclictest0-21swapper/309:34:383
29365991310,2cyclictest3814-21cron10:35:003
29365991310,2cyclictest32241-21grep11:10:163
29365991310,2cyclictest26264-21grep11:25:273
29365991310,2cyclictest16027-21grep08:55:163
29365991310,2cyclictest11673-21grep10:00:253
29365991310,2cyclictest1078-21cat09:35:193
29365991310,2cyclictest10361-21ssh12:14:493
29365991310,2cyclictest0-21swapper/310:24:573
2936599130,13cyclictest0-21swapper/309:23:563
2936399139,3cyclictest8980-21grep09:20:232
2936399139,3cyclictest19878-21wc09:05:012
2936399139,3cyclictest11596-21wc07:39:592
2936399139,2cyclictest7192-21tr07:30:132
2936399139,2cyclictest13202-21grep08:50:142
29363991311,1cyclictest0-21swapper/210:00:282
29363991310,2cyclictest18845-21grep09:45:242
29363991310,2cyclictest15793-21grep08:55:142
29363991310,2cyclictest11345-21wc11:55:202
2936399130,12cyclictest28553-21grep10:10:252
2935999139,2cyclictest21586-21wc08:00:151
29359991313,0cyclictest0-21swapper/109:35:051
29359991313,0cyclictest0-21swapper/109:10:121
29359991312,1cyclictest0-21swapper/109:30:271
29359991311,1cyclictest0-21swapper/110:07:131
29359991310,2cyclictest9181-21tail09:40:161
2935999130,2cyclictest0-21swapper/109:51:581
2935999130,1cyclictest0-21swapper/111:35:021
2935999130,13cyclictest0-21swapper/112:28:331
2935999130,13cyclictest0-21swapper/111:49:111
2935999130,12cyclictest0-21swapper/107:15:131
2935499139,3cyclictest627-21head09:15:260
2935499139,3cyclictest28569-21grep10:10:250
2935499139,3cyclictest0-21swapper/011:01:260
2935499131,6cyclictest6448-21awk07:29:590
29354991310,2cyclictest899-21grep08:25:150
29354991310,2cyclictest22295-21cat09:10:010
29354991310,2cyclictest20803-21basename10:25:170
29354991310,2cyclictest17759-21ssh11:40:130
2935499130,12cyclictest0-21swapper/012:05:120
2936599129,2cyclictest30321-21grep07:10:163
2936599129,2cyclictest26802-21grep08:10:173
2936599129,2cyclictest22556-21ssh11:05:003
2936599129,2cyclictest18096-21sed11:40:163
2936599129,2cyclictest17311-21head11:20:203
2936599129,2cyclictest15032-21sh08:55:013
2936599129,2cyclictest12286-21cron10:40:013
2936599129,2cyclictest10011-21cron08:45:003
2936599124,4cyclictest225-21systemd-journal09:44:553
2936599120,11cyclictest26555-21cat09:50:193
2936399129,2cyclictest2983-21tr12:10:132
2936399129,2cyclictest2966-21grep11:50:202
2936399129,2cyclictest28598-21sed12:25:172
2936399129,2cyclictest27521-21cat08:10:262
2936399129,2cyclictest24879-21sed09:30:182
2936399129,2cyclictest22316-21grep10:45:162
2936399129,2cyclictest21568-21cut08:00:142
2936399129,2cyclictest19926-21grep12:20:122
2936399129,2cyclictest15354-21cut07:45:202
2936399129,2cyclictest12019-21sed12:15:162
2936399128,3cyclictest31885-21grep09:15:162
2936399126,5cyclictest29856-21/usr/sbin/munin10:50:112
29363991212,0cyclictest0-21swapper/212:33:482
2935999129,2cyclictest3036-21cut07:20:191
2935999129,2cyclictest30268-21sort07:10:151
2935999129,2cyclictest28140-21cron08:15:001
2935999129,2cyclictest21374-21sh10:25:221
2935999129,2cyclictest16072-21grep08:55:161
2935999128,3cyclictest2231-21cut09:55:181
2935999127,3cyclictest17626-21grep07:50:191
2935999126,2cyclictest225-21systemd-journal08:19:591
2935999122,7cyclictest22714-21ssh11:05:011
2935999120,1cyclictest22190-21diskmemload11:16:351
2935999120,1cyclictest22190-21diskmemload10:02:511
2935499129,2cyclictest30121-21grep10:50:130
2935499129,2cyclictest27522-21sh12:25:000
2935499129,2cyclictest26304-21tr08:10:150
2935499129,2cyclictest22992-21cut09:10:120
2935499129,2cyclictest17160-21tr09:45:130
2935499129,2cyclictest16254-21turbostat.cron07:50:000
2935499129,2cyclictest1230-21grep09:35:180
2935499128,3cyclictest32373-21ssh11:29:590
2935499128,3cyclictest10555-21grep07:35:200
2935499121,3cyclictest225-21systemd-journal12:29:120
29354991210,1cyclictest29380-21cron10:50:010
2936599119,1cyclictest6864-21ssh12:31:053
2936599118,2cyclictest6114-21grep08:35:163
2936599118,2cyclictest29430-21cat08:15:203
2936599118,2cyclictest2881-21uname07:20:183
2936599118,2cyclictest27642-21rm11:45:253
2936599118,2cyclictest27494-21grep12:05:143
2936599118,2cyclictest21923-21taskset09:06:493
2936599118,2cyclictest20293-21ssh09:46:293
2936599118,2cyclictest1756-21ssh09:16:163
2936599118,2cyclictest17386-21seq11:01:143
2936599118,2cyclictest1576-21chrt07:19:063
2936599118,2cyclictest14465-21expr07:45:133
2936599118,2cyclictest13136-21systemctl07:40:233
2936599118,2cyclictest11960-21ssh11:36:173
2936599118,2cyclictest11429-21ssh11:55:203
2936599118,1cyclictest4677-21expr07:25:163
2936399119,1cyclictest20856-21sed09:05:162
2936399118,2cyclictest9043-21seq10:56:212
2936399118,2cyclictest6387-21ssh10:35:282
2936399118,2cyclictest6223-21grep08:35:172
2936399118,2cyclictest31228-21sed07:10:242
2936399118,2cyclictest30889-21ssh11:28:562
2936399118,2cyclictest30520-21cron08:19:592
2936399118,2cyclictest24405-21grep08:05:162
2936399118,2cyclictest23985-21ssh10:27:042
2936399118,2cyclictest21370-21ssh12:01:142
2936399118,2cyclictest21370-21ssh12:01:132
2936399118,2cyclictest20613-21sed07:55:272
2936399118,2cyclictest18863-21ssh11:21:092
2936399118,2cyclictest1504-21grep09:35:212
2936399118,2cyclictest13820-21sed12:35:202
2936399118,2cyclictest13541-21ssh10:20:302
2936399118,2cyclictest12617-21ssh11:17:522
2936399118,2cyclictest11635-21ssh11:36:052
2936399118,2cyclictest10933-21sort08:45:172
2935999118,2cyclictest8786-21sh10:18:251
2935999118,2cyclictest8786-21sh10:18:241
2935999118,2cyclictest7584-21ssh11:34:291
2935999118,2cyclictest6280-21cut08:35:171
2935999118,2cyclictest5271-21cut07:25:201
2935999118,2cyclictest30139-21grep10:50:141
2935999118,2cyclictest30114-21ssh12:06:301
2935999118,2cyclictest29682-21tr10:30:191
2935999118,2cyclictest2889-21sleep08:30:001
2935999118,2cyclictest26995-21cut10:10:131
2935999118,2cyclictest25350-21grep08:05:251
2935999118,2cyclictest16600-21cut11:20:151
2935999118,2cyclictest15730-21ssh11:58:241
2935999118,2cyclictest15324-21ssh12:36:051
2935999118,2cyclictest10007-21cron08:45:001
2935499118,2cyclictest5255-21basename08:34:590
2935499118,2cyclictest30062-21expr07:10:130
2935499118,2cyclictest28923-21forks10:30:140
2935499118,2cyclictest28113-21ssh09:51:000
2935499118,2cyclictest2746-21ssh11:30:450
2935499118,2cyclictest23340-21seq08:04:230
2935499118,2cyclictest22890-21ssh11:05:100
2935499118,2cyclictest2110-21grep11:50:130
2935499118,2cyclictest18094-21ssh11:20:330
2935499118,2cyclictest17909-21cat09:00:140
2935499118,2cyclictest16626-21fgrep08:55:210
2935499118,2cyclictest15154-21ssh11:38:480
2935499118,2cyclictest15033-21ssh09:43:580
2935499118,2cyclictest13904-21ssh10:40:170
2935499118,2cyclictest13652-21grep08:50:170
2935499118,2cyclictest13392-21ssh09:23:420
2935499118,2cyclictest12819-21ssh10:01:090
2935499118,2cyclictest12280-21ssh12:35:040
2935499117,2cyclictest18929-21grep07:55:140
2935499113,4cyclictest0-21swapper/011:55:110
2936599108,1cyclictest7456-21sed07:30:143
2936599108,1cyclictest3612-21cat08:30:143
2936599108,1cyclictest18858-21chrt07:55:083
2936599108,1cyclictest11183-21chrt07:37:003
2936599108,1cyclictest10765-21df08:45:153
2936599107,2cyclictest31624-21hostname08:20:183
2936599107,2cyclictest1002-21tr08:25:143
2936399108,1cyclictest716-21taskset08:25:032
2936399108,1cyclictest55122sleep207:25:222
2936399108,1cyclictest30923-21cut08:20:152
2936399108,1cyclictest13709-21chrt07:42:572
2936399107,2cyclictest8134-21ssh09:40:012
2936399107,2cyclictest3280-21cut08:30:132
2936399107,2cyclictest2669-21head07:20:162
2936399100,9cyclictest15702sleep207:19:012
2935999108,1cyclictest31951-21ntpq08:20:211
2935999108,1cyclictest19962-21cat07:55:201
2935999107,2cyclictest7931-21wc07:30:181
2935999107,2cyclictest5319-21cat08:35:001
2935999107,2cyclictest18401-21grep09:00:171
2935999107,2cyclictest12005-21grep08:45:261
2935999107,2cyclictest10992-21tail07:35:251
2935999100,9cyclictest13130-21cat08:50:141
2935499108,1cyclictest7385-21taskset07:30:130
2935499108,1cyclictest3768-21taskset07:21:280
2935499108,1cyclictest16562-21grep07:50:130
2935499107,2cyclictest30417-21taskset08:18:430
2935499107,2cyclictest25088-21ls08:05:230
2935499107,2cyclictest13781-21taskset07:43:540
2935499102,2cyclictest10038-21idleruntime-cro08:45:010
2935499100,9cyclictest10710-21expr08:45:140
293659997,1cyclictest16972-21expr07:50:153
293599997,1cyclictest22007-21chrt09:07:551
293599997,1cyclictest15267-21ntpq07:45:191
293599997,1cyclictest12075-21cat07:40:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional