You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 10:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Mar 10, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
361029278,9sleep00-21swapper/019:07:130
372128168,9sleep20-21swapper/219:08:412
229027958,16sleep30-21swapper/319:05:163
369027767,6sleep10-21swapper/119:08:151
92812630,0sleep20-21swapper/223:32:342
3903992928,1cyclictest0-21swapper/000:16:500
3911992624,1cyclictest32-21ksoftirqd/222:35:452
3911992517,2cyclictest32-21ksoftirqd/222:21:322
391199250,25cyclictest0-21swapper/223:55:532
390899240,1cyclictest0-21swapper/119:10:221
3908992323,0cyclictest0-21swapper/122:18:411
390899230,5cyclictest21982-21sed23:40:161
390899230,23cyclictest0-21swapper/122:25:231
3911992220,1cyclictest0-21swapper/220:20:142
3911992215,4cyclictest32-21ksoftirqd/222:14:262
391199220,22cyclictest0-21swapper/223:21:342
391199220,21cyclictest0-21swapper/223:44:462
391199220,20cyclictest0-21swapper/200:08:012
390399221,16cyclictest28024-21/usr/sbin/munin20:00:130
3914992119,1cyclictest0-21swapper/323:25:123
3911992117,3cyclictest32-21ksoftirqd/223:49:292
3908992121,0cyclictest0-21swapper/122:00:331
3908992120,1cyclictest0-21swapper/123:35:111
390899210,21cyclictest0-21swapper/123:52:171
390399218,6cyclictest26759-21perf21:05:000
391499200,1cyclictest537-21snmpd23:59:343
3908992020,0cyclictest0-21swapper/123:34:261
3911991914,2cyclictest32-21ksoftirqd/222:42:012
391199190,19cyclictest0-21swapper/200:32:062
391199190,19cyclictest0-21swapper/200:01:202
391199190,18cyclictest0-21swapper/222:54:282
390899191,16cyclictest0-21swapper/121:17:331
390899190,1cyclictest0-21swapper/100:04:201
390399195,7cyclictest5077-21basename20:20:000
210392190,0sleep10-21swapper/100:18:051
13462190,0sleep10-21swapper/121:31:231
391499181,13cyclictest0-21swapper/321:15:063
391199188,9cyclictest32-21ksoftirqd/223:25:122
391199183,4cyclictest32-21ksoftirqd/200:37:252
391199180,18cyclictest0-21swapper/223:15:042
390899181,17cyclictest0-21swapper/122:46:121
390899180,18cyclictest0-21swapper/123:23:451
390899180,15cyclictest0-21swapper/122:40:231
390399188,5cyclictest24675-21perf00:40:000
390399182,5cyclictest225-21systemd-journal00:35:000
3914991713,3cyclictest39-21ksoftirqd/321:55:153
391499170,17cyclictest0-21swapper/323:42:473
391199172,4cyclictest29128-21diskmemload21:51:532
3911991716,1cyclictest0-21swapper/222:46:092
3911991712,4cyclictest32-21ksoftirqd/223:52:402
3908991716,1cyclictest29128-21diskmemload23:03:471
3908991716,1cyclictest29128-21diskmemload23:03:471
390899170,16cyclictest0-21swapper/122:24:541
390899170,0cyclictest0-21swapper/122:34:291
390399179,4cyclictest25381-21sed00:20:210
390399175,6cyclictest11383-21munin-run23:15:010
390399174,6cyclictest8333-21perf22:15:000
390399170,2cyclictest0-21swapper/023:45:200
3911991615,1cyclictest32-21ksoftirqd/221:55:412
3911991615,1cyclictest0-21swapper/221:38:042
3911991614,2cyclictest32-21ksoftirqd/200:29:132
3911991614,2cyclictest32-21ksoftirqd/200:29:122
3911991614,1cyclictest32-21ksoftirqd/222:18:232
3911991613,2cyclictest32-21ksoftirqd/221:00:002
3911991612,2cyclictest32-21ksoftirqd/221:20:182
390399162,11cyclictest0-21swapper/000:13:410
390399160,1cyclictest0-21swapper/022:35:020
3914991511,3cyclictest31081-21grep21:10:233
3914991510,3cyclictest29101-21cron21:10:003
391499150,15cyclictest0-21swapper/322:06:173
391199153,7cyclictest32-21ksoftirqd/222:00:422
3911991514,1cyclictest32-21ksoftirqd/223:05:212
3911991514,1cyclictest32-21ksoftirqd/223:05:212
3911991514,1cyclictest311rcuc/223:37:452
391199151,1cyclictest141rcu_preempt23:12:592
3911991511,4cyclictest32-21ksoftirqd/222:33:152
3911991511,3cyclictest25840-21ls21:00:192
3911991511,1cyclictest32-21ksoftirqd/222:58:402
391199150,2cyclictest0-21swapper/221:27:082
3908991511,1cyclictest0-21swapper/121:42:451
390899150,1cyclictest0-21swapper/122:59:171
390899150,15cyclictest0-21swapper/122:14:411
390399157,7cyclictest1142-21runrttasks23:16:230
390399154,6cyclictest27173-21perf23:05:000
390399154,6cyclictest27173-21perf23:05:000
390399150,8cyclictest24339-21idleruntime-cro20:59:590
3914991411,2cyclictest10552-21grep19:20:243
391499141,11cyclictest20937-21sensors23:00:253
391499141,11cyclictest20937-21sensors23:00:243
3914991410,3cyclictest2436-21cron20:15:013
3914991410,2cyclictest3545-21awk22:30:273
391499140,12cyclictest0-21swapper/300:35:403
3911991412,1cyclictest32-21ksoftirqd/221:30:192
391199141,1cyclictest0-21swapper/221:16:572
3911991411,2cyclictest32-21ksoftirqd/200:18:452
3911991410,3cyclictest26309-21ssh22:25:162
390899149,3cyclictest15693-21wc22:00:011
390899142,2cyclictest7809-21apt-get00:30:011
390899141,2cyclictest141rcu_preempt21:36:461
390899140,1cyclictest0-21swapper/100:39:561
390399142,7cyclictest14580-21sendmail21:40:010
390399142,6cyclictest32511-21perf20:10:000
3903991410,3cyclictest29102-21cron21:09:590
3903991410,2cyclictest1059-21awk22:29:590
391499139,3cyclictest29576-21wc20:00:253
391499139,3cyclictest21625-21grep19:45:173
391499139,3cyclictest20057-21wc22:40:253
391499139,3cyclictest18774-21tr19:40:143
391499139,3cyclictest18110-21cut20:45:183
391499139,3cyclictest10773-21grep22:15:253
391499139,2cyclictest6161-21taskset20:20:183
391499132,6cyclictest16241-21ssh22:00:123
3914991313,0cyclictest0-21swapper/322:25:443
3914991310,2cyclictest508-21tail21:50:203
3914991310,2cyclictest4881-21head19:10:203
3914991310,2cyclictest23010-21tail21:25:163
3914991310,2cyclictest16177-21ls21:40:193
3914991310,2cyclictest14773-21apt-config20:39:593
3914991310,2cyclictest12708-21grep23:15:163
391499130,12cyclictest16355-21cron00:35:003
391199138,1cyclictest141rcu_preempt00:21:562
3911991310,2cyclictest27206-21ssh22:05:562
3911991310,2cyclictest24125-21grep21:45:192
3911991310,2cyclictest18722-21head20:45:232
390899139,3cyclictest10105-21cut20:30:131
390899139,2cyclictest31046-21tune2fs20:05:161
390899136,2cyclictest225-21systemd-journal20:00:001
390899131,8cyclictest225-21systemd-journal00:09:001
3908991311,2cyclictest0-21swapper/121:48:521
3908991310,2cyclictest25370-21head22:05:161
3908991310,2cyclictest24176-21cut00:20:131
3908991310,2cyclictest19234-21grep19:40:161
3908991310,2cyclictest10051-21basename22:35:141
390899130,4cyclictest225-21systemd-journal20:25:011
390899130,13cyclictest0-21swapper/121:23:321
390399138,3cyclictest17750-21tr20:45:160
390399135,5cyclictest18311-21apt-get22:40:120
3903991310,2cyclictest8433-21grep21:55:170
3903991310,2cyclictest5840-21grep21:15:150
3903991310,2cyclictest22218-21grep23:20:280
3903991310,2cyclictest21647-21sed19:45:180
3903991310,2cyclictest18194-21cron19:40:000
390399130,4cyclictest0-21swapper/023:07:340
390399130,4cyclictest0-21swapper/023:07:330
390399130,12cyclictest0-21swapper/023:51:320
391499129,2cyclictest625-21grep22:10:143
391499129,2cyclictest31560-21grep23:45:253
391499129,2cyclictest20618-21head20:50:183
391499129,2cyclictest17541-21grep00:15:283
391499129,2cyclictest13278-21grep23:35:123
391499129,2cyclictest11306-21grep22:35:223
391499129,2cyclictest10808-21grep20:30:173
391499128,3cyclictest6284-21ssh00:28:483
391499128,3cyclictest6284-21ssh00:28:483
391499128,3cyclictest2748-21tr22:50:143
391499128,3cyclictest14046-21ssh21:20:143
3914991210,1cyclictest0-21swapper/322:23:073
391499120,11cyclictest28499-21tail23:05:173
391499120,11cyclictest28499-21tail23:05:173
391199129,2cyclictest28092-21grep21:05:202
391199129,2cyclictest20108-21grep19:40:252
391199128,3cyclictest30517-21sed21:10:182
391199123,8cyclictest32-21ksoftirqd/223:02:052
391199123,8cyclictest32-21ksoftirqd/223:02:042
391199123,8cyclictest311rcuc/220:30:132
391199123,8cyclictest11455-21apache219:55:122
391199120,1cyclictest0-21swapper/220:40:122
391199120,11cyclictest12110-21sed19:25:162
390899129,2cyclictest30301-21grep23:45:151
390899129,2cyclictest28188-21df23:05:151
390899129,2cyclictest28188-21df23:05:151
390899129,2cyclictest14804-21head23:55:171
3908991211,1cyclictest0-21swapper/120:19:221
390899121,10cyclictest0-21swapper/121:50:131
3908991210,1cyclictest14708-21cat19:30:191
390899120,12cyclictest0-21swapper/100:25:181
390899120,12cyclictest0-21swapper/100:25:171
390399129,2cyclictest3817-21grep22:50:200
390399129,2cyclictest32579-21grep00:25:140
390399129,2cyclictest32579-21grep00:25:130
390399129,2cyclictest31691-21chrt21:30:170
390399129,2cyclictest29992-21pmu-power23:25:200
390399129,2cyclictest27809-21wc20:00:000
390399129,2cyclictest20405-21grep20:50:160
390399129,2cyclictest12475-21cron23:35:010
390399129,2cyclictest10560-21grep20:30:140
390399129,2cyclictest10151-21head19:20:210
390399128,3cyclictest6942-21sed20:20:250
390399128,3cyclictest4899-21ssh00:08:500
390399128,3cyclictest18111-21grep22:20:180
390399126,3cyclictest225-21systemd-journal19:15:000
391499118,2cyclictest9607-21taskset20:27:353
391499118,2cyclictest8204-21head19:15:263
391499118,2cyclictest7527-21sed21:35:173
391499118,2cyclictest6666-21ssh23:30:323
391499118,2cyclictest6143-21sh23:10:523
391499118,2cyclictest30632-21rm00:05:013
391499118,2cyclictest2937-21cut20:15:163
391499118,2cyclictest28257-21ssh00:03:293
391499118,2cyclictest27298-21df22:45:173
391499118,2cyclictest27207-21chrt21:46:523
391499118,2cyclictest25882-21grep00:20:243
391499118,2cyclictest24822-21grep21:00:123
391499118,2cyclictest23894-21taskset23:21:413
391499118,2cyclictest23527-21tr19:50:163
391499118,2cyclictest23527-21tr19:50:153
391499118,2cyclictest17019-21taskset20:42:333
391499118,2cyclictest16743-21cut19:35:153
391499118,2cyclictest14845-21ssh00:14:473
391499118,2cyclictest14695-21ssh22:56:483
391499118,2cyclictest13007-21awk19:25:263
391499118,2cyclictest11315-21taskset23:53:203
391499110,10cyclictest643-21ssh21:30:513
391199119,1cyclictest9464-21grep19:20:142
391199118,2cyclictest17756-21grep19:35:242
391199118,2cyclictest17670-21ssh21:40:552
391199118,2cyclictest15566-21taskset19:31:472
391199118,2cyclictest10673-21ssh00:11:302
391199117,3cyclictest32-21ksoftirqd/220:20:002
391199113,4cyclictest225-21systemd-journal20:10:002
391199110,2cyclictest5919-21taskset19:10:542
390899119,1cyclictest7028-21ssh00:10:021
390899119,1cyclictest13594-21grep20:35:221
390899118,2cyclictest22592-21who19:45:261
390899118,2cyclictest21248-21sensors20:50:241
390899118,2cyclictest16721-21grep20:40:251
390899117,3cyclictest1560-21ssh21:12:171
390899117,1cyclictest11806-21ssh23:15:071
390899110,1cyclictest0-21swapper/123:25:121
390899110,10cyclictest7984-21cut20:25:151
390899110,10cyclictest4903-21ssh22:50:411
390399118,2cyclictest31938-21ssh21:50:140
390399118,2cyclictest31587-21tail21:10:280
390399118,2cyclictest31254-21ssh22:09:080
390399118,2cyclictest23617-21ssh23:40:360
390399118,2cyclictest23066-21head21:25:160
390399118,2cyclictest2195-21ssh22:50:010
390399118,2cyclictest18683-21tr19:40:130
390399118,2cyclictest17630-21awk22:00:220
390399118,2cyclictest15570-21taskset19:31:500
390399118,2cyclictest14916-21kernelversion21:20:180
390399118,2cyclictest14570-21taskset20:39:380
390399118,2cyclictest1386-21chrt22:30:040
390399118,2cyclictest13804-21head23:35:150
390399118,2cyclictest10388-21cut22:15:220
390399117,2cyclictest6767-21grep19:15:140
390399112,6cyclictest2629-21apt-config20:15:010
390399112,2cyclictest0-21swapper/022:55:130
390399110,10cyclictest29846-21ssh00:04:450
390399110,10cyclictest19824-21ssh23:58:270
391499108,1cyclictest32209-21chrt20:06:343
391499108,1cyclictest25609-21apache_processe19:55:133
391499108,1cyclictest23043-21wc20:55:183
391499108,1cyclictest14146-21expr19:30:143
391199108,1cyclictest28648-21grep20:00:162
391199107,2cyclictest9419-21sed20:25:272
391199107,2cyclictest8125-21grep19:15:252
391199107,2cyclictest1601-21cut20:10:222
391199107,2cyclictest13105-21grep20:35:182
391199105,2cyclictest32-21ksoftirqd/219:50:132
391199105,2cyclictest32-21ksoftirqd/219:50:122
391199100,9cyclictest21264-21expr19:45:142
390899108,1cyclictest9305-21cat19:20:141
390899108,1cyclictest8597-21taskset19:19:431
390899108,1cyclictest28392-21ls20:00:151
390899108,1cyclictest24169-21taskset20:57:561
390899108,1cyclictest22551-21cat21:25:141
390899108,1cyclictest112452sleep119:25:011
390899107,2cyclictest743-21expr20:10:151
390899107,2cyclictest28138-21ntp_kernel_pll_21:05:211
390899107,2cyclictest25658-21ls21:00:171
390899107,2cyclictest17774-21tr20:45:161
390899106,3cyclictest5323-21timerandwakeup23:10:271
390899106,3cyclictest24616-21grep19:50:231
390899106,3cyclictest24616-21grep19:50:231
390399108,1cyclictest15656-21sed20:40:150
390399107,2cyclictest23548-21cstates19:50:160
390399107,2cyclictest23548-21cstates19:50:150
390399107,2cyclictest23190-21ssh21:45:040
390399107,2cyclictest15372-21expr21:40:140
390399100,9cyclictest8856-21sh20:25:230
39119991,1cyclictest0-21swapper/220:50:142
39089998,1cyclictest16431-21expr19:35:131
39039997,1cyclictest12586-21awk19:25:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional