You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 23:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Sep 02, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17634210274,9sleep00-21swapper/007:09:590
1729729178,9sleep20-21swapper/207:05:392
1743528566,15sleep10-21swapper/107:07:271
1747928269,9sleep30-21swapper/307:08:023
299952810,1sleep10-21swapper/112:34:291
1772399289,15cyclictest32-21ksoftirqd/208:50:212
17723992711,5cyclictest32-21ksoftirqd/212:12:032
17723992624,1cyclictest32-21ksoftirqd/211:54:212
17723992512,3cyclictest32-21ksoftirqd/211:26:532
1772399240,1cyclictest0-21swapper/209:48:422
17725992310,2cyclictest39-21ksoftirqd/309:23:003
1772599230,2cyclictest0-21swapper/309:05:223
17723992321,1cyclictest32-21ksoftirqd/210:31:442
1772599222,3cyclictest141rcu_preempt10:20:133
17725992220,1cyclictest39-21ksoftirqd/310:03:053
17725992216,4cyclictest39-21ksoftirqd/309:35:523
1772599220,2cyclictest0-21swapper/311:30:483
1772399220,1cyclictest0-21swapper/211:17:552
1771599220,2cyclictest0-21swapper/011:47:240
1771599220,1cyclictest0-21swapper/009:20:280
17725992118,2cyclictest39-21ksoftirqd/310:30:003
17725992112,1cyclictest39-21ksoftirqd/310:14:173
1772399214,2cyclictest141rcu_preempt10:55:272
17723992119,1cyclictest32-21ksoftirqd/210:29:012
17723992117,1cyclictest141rcu_preempt10:02:192
17723992113,2cyclictest32-21ksoftirqd/210:38:182
1772399211,2cyclictest141rcu_preempt10:53:072
17719992119,1cyclictest0-21swapper/109:25:201
1772399201,8cyclictest29278-21pmu-power09:20:222
1772399200,4cyclictest141rcu_preempt07:40:222
1772399200,1cyclictest141rcu_preempt08:35:022
1772399200,19cyclictest0-21swapper/212:25:132
17719992019,1cyclictest25-21ksoftirqd/111:19:301
17719992017,2cyclictest0-21swapper/110:50:141
1771999200,1cyclictest0-21swapper/112:28:081
1771999200,1cyclictest0-21swapper/112:20:131
1772599199,1cyclictest39-21ksoftirqd/311:42:073
1772399198,2cyclictest32-21ksoftirqd/209:10:042
1772399195,1cyclictest141rcu_preempt11:20:222
17723991916,1cyclictest32-21ksoftirqd/209:26:252
1772399190,1cyclictest0-21swapper/211:36:372
1771599192,16cyclictest0-21swapper/010:05:130
1772599187,1cyclictest141rcu_preempt11:55:213
1772599182,4cyclictest141rcu_preempt09:10:113
1772599180,3cyclictest141rcu_preempt10:31:443
1772399186,2cyclictest141rcu_preempt09:57:272
1772399182,2cyclictest141rcu_preempt12:06:292
17723991813,1cyclictest141rcu_preempt12:34:312
1771999182,2cyclictest141rcu_preempt11:56:531
1771999181,6cyclictest28290-21cpuspeed_turbos11:35:141
1771999181,14cyclictest10343-21diskmemload09:10:351
1771599188,5cyclictest11533-21perf08:05:000
1771599180,2cyclictest0-21swapper/011:09:390
1772599174,3cyclictest141rcu_preempt10:40:253
17725991715,2cyclictest39-21ksoftirqd/309:47:443
1772599171,1cyclictest141rcu_preempt09:56:363
1772399175,2cyclictest141rcu_preempt11:55:402
1772399174,3cyclictest141rcu_preempt08:00:252
1772399174,2cyclictest141rcu_preempt11:11:402
17723991714,1cyclictest32-21ksoftirqd/211:07:462
1772399170,17cyclictest0-21swapper/210:05:482
1772399170,15cyclictest32-21ksoftirqd/212:18:492
1771999171,15cyclictest0-21swapper/109:40:131
17719991710,4cyclictest2439-21grep11:00:161
1772399167,1cyclictest32-21ksoftirqd/210:17:092
1772399165,9cyclictest19619-21apt-get11:30:132
1772399165,1cyclictest141rcu_preempt09:37:322
1772399164,6cyclictest10343-21diskmemload11:45:002
1772399160,16cyclictest0-21swapper/210:14:092
17719991614,1cyclictest0-21swapper/111:40:151
17719991613,2cyclictest4075-21fschecks_count08:55:171
17719991612,3cyclictest0-21swapper/112:19:591
1771999160,15cyclictest0-21swapper/108:00:261
1771599160,0cyclictest0-21swapper/010:28:150
1772599155,3cyclictest141rcu_preempt11:17:093
1772599154,2cyclictest141rcu_preempt11:00:163
1772599154,2cyclictest141rcu_preempt09:17:413
1772599154,1cyclictest39-21ksoftirqd/312:21:463
1772599152,7cyclictest39-21ksoftirqd/308:45:003
1772599151,2cyclictest141rcu_preempt08:30:263
17725991512,1cyclictest39-21ksoftirqd/312:28:303
17725991511,3cyclictest39-21ksoftirqd/312:30:223
17723991513,1cyclictest311rcuc/211:00:202
1772399151,1cyclictest141rcu_preempt11:46:592
1772399150,1cyclictest141rcu_preempt10:44:552
1771999150,13cyclictest0-21swapper/109:31:121
1771599151,7cyclictest11844-21munin-run11:45:000
17715991511,3cyclictest13-21ksoftirqd/011:30:120
17715991510,3cyclictest14334-21ls12:05:180
1771599150,2cyclictest0-21swapper/010:48:040
1771599150,2cyclictest0-21swapper/010:48:040
1771599150,15cyclictest0-21swapper/010:54:130
1771599150,0cyclictest0-21swapper/011:38:080
1771599150,0cyclictest0-21swapper/010:17:560
1771599150,0cyclictest0-21swapper/007:30:140
1772599147,4cyclictest39-21ksoftirqd/310:49:183
1772599147,4cyclictest39-21ksoftirqd/310:49:183
1772599144,1cyclictest39-21ksoftirqd/307:41:203
1772599142,2cyclictest141rcu_preempt12:04:203
1772599142,1cyclictest141rcu_preempt11:20:313
1772599142,12cyclictest39-21ksoftirqd/311:47:483
17725991412,1cyclictest39-21ksoftirqd/311:37:123
17725991412,1cyclictest28591-21apt11:55:003
17725991411,2cyclictest39-21ksoftirqd/309:50:153
17725991410,3cyclictest3185-21cron08:55:013
17725991410,2cyclictest39-21ksoftirqd/310:39:293
1772399148,2cyclictest32-21ksoftirqd/209:31:502
1772399147,3cyclictest32-21ksoftirqd/212:04:532
1772399143,4cyclictest32-21ksoftirqd/210:46:112
1772399143,4cyclictest32-21ksoftirqd/210:46:112
17723991412,1cyclictest311rcuc/209:15:352
17723991411,2cyclictest15758-21grep08:10:262
17723991410,1cyclictest141rcu_preempt09:41:552
1772399140,1cyclictest10343-21diskmemload12:36:112
1772399140,1cyclictest0-21swapper/209:00:222
1771999148,5cyclictest27295-21basename07:30:001
17719991410,2cyclictest6628-21grep09:00:171
1771599143,10cyclictest10832-21apache_processe09:10:120
17715991411,2cyclictest10481-21ssh09:10:000
17715991410,2cyclictest27606-21awk08:35:230
315402130,3sleep3381rcuc/312:15:233
1772599139,3cyclictest8209-21ssh10:05:263
1772599136,4cyclictest39-21ksoftirqd/311:27:323
1772599136,1cyclictest0-21swapper/309:40:133
1772599135,4cyclictest39-21ksoftirqd/309:32:243
1772599134,7cyclictest39-21ksoftirqd/310:52:303
1772599134,1cyclictest141rcu_preempt11:07:593
1772599132,1cyclictest141rcu_preempt10:17:263
1772599131,1cyclictest141rcu_preempt08:05:253
17725991310,2cyclictest5117-21grep08:55:273
1772399137,2cyclictest32-21ksoftirqd/210:22:522
17723991310,2cyclictest5111-21grep08:55:272
1771999139,3cyclictest32420-21cut10:20:171
1771999132,7cyclictest0-21swapper/110:25:131
1771999131,3cyclictest0-21swapper/108:44:551
1771999131,2cyclictest0-21swapper/111:05:111
17719991312,1cyclictest0-21swapper/109:53:271
17719991310,2cyclictest3033-21grep07:45:181
17719991310,2cyclictest26259-21grep10:35:261
17719991310,2cyclictest26006-21cut10:55:161
17719991310,2cyclictest22035-21tr12:10:141
17719991310,2cyclictest20368-21grep11:10:271
17719991310,2cyclictest14901-21cut08:10:171
1771999130,12cyclictest10343-21diskmemload09:47:581
1771999130,12cyclictest10343-21diskmemload09:15:151
1771999130,10cyclictest0-21swapper/107:40:131
1771599139,2cyclictest8372-21cron10:45:000
1771599139,2cyclictest15320-21ls12:25:170
1771599139,2cyclictest1142-21runrttasks10:11:260
1771599136,2cyclictest21722-21apache_accesses12:10:110
1771599131,6cyclictest9146-21awk08:00:000
1771599131,0cyclictest0-21swapper/008:30:130
17715991310,2cyclictest812-21apt08:50:000
17715991310,2cyclictest20282-21grep11:10:270
17715991310,2cyclictest19009-21wc07:10:190
17715991310,2cyclictest19009-21wc07:10:190
17715991310,2cyclictest16536-21basename08:15:130
17715991310,2cyclictest1333-21cat11:00:000
1771599130,12cyclictest13819-21tr09:50:150
322022120,1sleep094-21kswapd010:40:000
301472120,0chrt0-21swapper/010:00:040
1772599129,3cyclictest0-21swapper/312:05:163
1772599129,2cyclictest24834-21cron07:25:003
1772599129,2cyclictest21446-21tr08:25:153
1772599129,2cyclictest20053-21cron07:15:003
1772599129,2cyclictest20053-21cron07:15:003
1772599128,4cyclictest25310-21apache211:14:513
1772599126,4cyclictest39-21ksoftirqd/309:00:163
1772599126,2cyclictest39-21ksoftirqd/312:10:173
1772599125,5cyclictest0-21swapper/310:55:263
1772399129,2cyclictest8453-21tr09:05:142
1772399129,2cyclictest3503-21tr07:45:232
1772399127,5cyclictest311rcuc/207:50:172
1772399127,4cyclictest32-21ksoftirqd/212:21:242
1772399127,4cyclictest32-21ksoftirqd/209:50:142
1772399127,4cyclictest32-21ksoftirqd/208:30:122
1772399127,4cyclictest32-21ksoftirqd/208:25:192
1772399124,7cyclictest20055-21cron07:15:002
1772399124,7cyclictest20055-21cron07:15:002
17723991210,2cyclictest32-21ksoftirqd/207:55:162
1772399120,1cyclictest141rcu_preempt07:15:162
1771999129,2cyclictest30154-21grep07:35:141
1771999129,2cyclictest29156-21grep09:20:201
1771999129,2cyclictest28829-21cat09:40:011
1771999129,2cyclictest25388-21grep08:30:261
1771999129,2cyclictest22437-21cron07:20:001
1771999129,2cyclictest20999-21ls11:30:221
1771999129,2cyclictest19113-21awk11:30:001
1771999129,2cyclictest17590-21sh10:30:241
1771999128,3cyclictest8826-21grep09:05:161
1771999128,3cyclictest6261-21grep07:50:261
1771999128,2cyclictest32587-21sort10:40:011
1771999128,1cyclictest15687-21ssh10:49:101
1771999128,1cyclictest15687-21ssh10:49:101
1771999127,4cyclictest19263-21ssh11:49:181
1771999120,8cyclictest0-21swapper/112:35:381
1771999120,11cyclictest1829-21grep08:50:161
1771599129,2cyclictest7476-21grep09:00:260
1771599129,2cyclictest2804-21grep08:50:260
1771599129,2cyclictest23817-21sed09:55:280
1771599129,2cyclictest16060-21grep10:30:140
1771599125,6cyclictest13047-21munin-run09:50:000
1772599119,1cyclictest10343-21diskmemload12:35:063
1772599118,2cyclictest5602-21ssh09:25:233
1772599118,2cyclictest3255-21sort07:45:203
1772599118,2cyclictest31402-21grep08:45:143
1772599118,2cyclictest26446-21basename08:35:153
1772599118,2cyclictest11504-21cron08:05:003
1772599117,2cyclictest21988-21tail07:15:263
1772599116,4cyclictest39-21ksoftirqd/308:20:193
1772599116,4cyclictest39-21ksoftirqd/307:55:153
1772599115,4cyclictest39-21ksoftirqd/308:10:163
1772399118,2cyclictest32077-21cut08:45:182
1772399116,4cyclictest32-21ksoftirqd/208:20:222
1772399116,4cyclictest32-21ksoftirqd/208:15:152
1772399116,4cyclictest32-21ksoftirqd/207:20:252
1772399114,6cyclictest25859-21perl07:25:162
1771999119,1cyclictest27590-21tr08:35:231
1771999118,2cyclictest8244-21head07:55:221
1771999118,2cyclictest72572sleep110:05:181
1771999118,2cyclictest5352-21tr12:00:141
1771999118,2cyclictest31775-21grep08:45:161
1771999118,2cyclictest30694-21wc10:00:141
1771999118,2cyclictest27378-21ssh09:58:051
1771999118,2cyclictest24064-21tail10:15:181
1771999114,6cyclictest20022-21needreboot08:20:201
1771999113,7cyclictest22701-21cron10:15:001
17719991110,1cyclictest0-21swapper/111:50:391
1771999110,11cyclictest0-21swapper/111:21:411
1771599119,1cyclictest51512sleep012:19:410
1771599118,2cyclictest903-21ssh11:18:560
1771599118,2cyclictest8259-21ssh12:20:350
1771599118,2cyclictest6611-21taskset07:53:260
1771599118,2cyclictest4814-21ssh12:00:020
1771599118,2cyclictest4493-21ssh11:20:240
1771599118,2cyclictest3204-21mailstats11:00:210
1771599118,2cyclictest31003-21ssh12:35:050
1771599118,2cyclictest29527-21grep09:40:140
1771599118,2cyclictest29247-21ssh11:55:140
1771599118,2cyclictest23008-21ssh11:50:410
1771599118,2cyclictest21508-21grep09:35:170
1771599118,2cyclictest17520-21ssh11:28:460
1771599118,2cyclictest16612-21ssh09:32:090
1771599118,2cyclictest15471-21find08:10:220
1771599118,2cyclictest1448-21ssh10:20:360
1771599118,2cyclictest11804-21basename08:05:140
1771599117,3cyclictest4012-21acpi09:25:120
1771599117,3cyclictest21033-21cron08:25:010
1771599110,1cyclictest0-21swapper/012:32:000
1771599110,10cyclictest23652-21ssh09:17:090
1772599108,1cyclictest4782-21egrep07:50:143
1772599108,1cyclictest319542sleep307:39:183
1772599107,2cyclictest27809-21expr07:30:153
1772599105,4cyclictest39-21ksoftirqd/308:15:193
1772599101,5cyclictest225-21systemd-journal07:30:003
1772399108,1cyclictest30294-21expr07:35:152
1772399108,1cyclictest27974-21cat07:30:152
1772399107,2cyclictest32-21ksoftirqd/208:40:142
1772399106,3cyclictest32-21ksoftirqd/208:05:162
1771999108,1cyclictest27821-21expr07:30:151
1771999108,1cyclictest198042sleep107:11:521
1771999108,1cyclictest198042sleep107:11:521
1771999108,1cyclictest15821-21ssh12:05:451
1771999108,1cyclictest11972-21wc08:05:141
1771999107,2cyclictest23477-21sed07:20:171
1771999107,2cyclictest22014-21head08:25:181
1771999107,2cyclictest18338-21taskset08:16:141
1771599108,1cyclictest686-21cat07:40:190
1771599108,1cyclictest30839-21grep07:35:190
1771599108,1cyclictest28352sleep007:45:160
1771599108,1cyclictest21550-21expr08:25:160
1771599107,2cyclictest5288-21chrt08:56:330
1771599107,2cyclictest24502-21taskset07:20:490
1771599107,2cyclictest22111-21chrt07:15:550
1771599106,3cyclictest25153-21basename07:25:130
177159997,1cyclictest290672sleep008:40:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional