You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 23:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Mar 25, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9637210563,38sleep00-21swapper/007:06:080
9696210350,48sleep20-21swapper/207:06:542
990328570,11sleep10-21swapper/107:09:341
218842830,0sleep10-21swapper/112:30:581
988327967,8sleep30-21swapper/307:09:203
1001399280,0cyclictest0-21swapper/009:46:510
1001399250,1cyclictest0-21swapper/010:48:440
1001899240,24cyclictest0-21swapper/109:29:101
73922220,1sleep232-21ksoftirqd/211:45:002
10027992219,1cyclictest0-21swapper/308:33:173
10018992221,1cyclictest0-21swapper/109:30:141
1002799210,19cyclictest25353-21kworker/3:208:40:073
1002799210,19cyclictest25353-21kworker/3:208:40:073
1002799210,19cyclictest25353-21kworker/3:208:35:153
1001899210,1cyclictest0-21swapper/112:18:071
10024992020,0cyclictest0-21swapper/207:30:122
10018992017,2cyclictest0-21swapper/109:44:201
10013992020,0cyclictest0-21swapper/010:35:100
1002799191,2cyclictest0-21swapper/309:15:073
1002799190,1cyclictest0-21swapper/310:33:163
1002799190,19cyclictest25353-21kworker/3:208:46:493
10018991917,2cyclictest0-21swapper/111:11:161
1001899190,3cyclictest0-21swapper/109:51:531
1001899190,19cyclictest0-21swapper/108:00:131
1001899190,18cyclictest0-21swapper/109:15:131
1002799181,16cyclictest11391-21ssh12:25:163
1002799180,4cyclictest0-21swapper/312:10:483
1002799180,18cyclictest0-21swapper/309:42:293
10018991814,2cyclictest0-21swapper/110:24:111
1001899180,18cyclictest0-21swapper/111:50:371
1002799173,1cyclictest141rcu_preempt09:11:483
10018991717,0cyclictest0-21swapper/110:06:171
10018991715,1cyclictest38150irq/131-snd_hda10:55:591
10018991714,1cyclictest3663-21diskmemload12:21:281
1001899170,17cyclictest0-21swapper/110:49:261
1001899170,16cyclictest0-21swapper/112:03:301
1002799162,13cyclictest0-21swapper/310:26:433
1002799160,1cyclictest0-21swapper/309:51:533
1001899163,6cyclictest38150irq/131-snd_hda09:55:591
1001899160,1cyclictest0-21swapper/111:25:191
1001899160,1cyclictest0-21swapper/109:47:161
1001899160,15cyclictest0-21swapper/110:33:101
1001899160,14cyclictest22702-21ssh09:20:241
290762150,0sleep30-21swapper/311:57:063
1002799150,4cyclictest0-21swapper/311:21:153
1002799150,2cyclictest0-21swapper/310:18:133
1002799150,1cyclictest0-21swapper/311:15:263
1002799150,15cyclictest0-21swapper/311:47:493
1002499152,10cyclictest0-21swapper/209:45:132
1002499150,3cyclictest0-21swapper/210:10:382
1001899154,5cyclictest0-21swapper/109:35:131
10018991513,1cyclictest25-21ksoftirqd/110:00:151
10018991510,3cyclictest29158-21wc07:45:251
1001899150,3cyclictest0-21swapper/110:17:021
1001899150,2cyclictest0-21swapper/112:10:291
1001399153,6cyclictest2156-21munin-run10:45:010
10013991511,3cyclictest32253-21tr10:05:130
1002799140,0cyclictest0-21swapper/310:00:463
1002499141,12cyclictest23003-21runrttasks11:53:582
10024991410,3cyclictest1215-21wc10:05:242
1002499140,13cyclictest0-21swapper/212:36:572
10018991410,3cyclictest12476-21wc12:25:261
10018991410,3cyclictest1169-21ssh12:38:251
10018991410,2cyclictest32623-21wc09:00:191
1001899140,2cyclictest141rcu_preempt11:37:281
1001899140,2cyclictest141rcu_preempt11:37:281
1001899140,1cyclictest0-21swapper/110:36:271
1001899140,13cyclictest27146-21df_inode10:40:151
1001899140,12cyclictest0-21swapper/108:25:151
1001399141,6cyclictest22102-21apt-get07:35:000
10013991410,3cyclictest28922-21gzip07:45:210
10013991410,3cyclictest28123-21grep12:15:250
1001399140,3cyclictest0-21swapper/009:50:460
1001399140,0cyclictest0-21swapper/009:21:590
1002799139,3cyclictest13740-21cat08:20:263
10027991312,1cyclictest0-21swapper/309:58:213
10027991310,2cyclictest18668-21grep10:35:163
10027991310,2cyclictest15046-21cat07:20:003
1002799130,12cyclictest3378-21wc12:40:003
1002499139,3cyclictest22223-21tune2fs11:15:162
1002499139,3cyclictest21061-21grep12:30:252
1002499139,2cyclictest9706-21tr12:05:132
1002499135,4cyclictest225-21systemd-journal12:14:552
10024991311,1cyclictest0-21swapper/209:53:312
10024991310,2cyclictest9681-21ssh11:45:262
10024991310,2cyclictest918-21ssh12:00:112
10024991310,2cyclictest23572-21head09:40:182
10024991310,2cyclictest19552-21cut08:35:132
10024991310,2cyclictest18378-21cut07:25:152
10024991310,2cyclictest15990-21wc08:25:242
10024991310,2cyclictest13601-21latency_hist09:35:002
10024991310,2cyclictest135222sleep210:32:022
10024991310,2cyclictest13000-21grep08:20:182
1002499130,1cyclictest0-21swapper/207:35:132
1002499130,12cyclictest27376-21wc12:15:192
1001899138,4cyclictest30247-21tr07:50:151
1001899138,4cyclictest16591-21ssh12:09:281
1001899138,3cyclictest10644-21wc08:15:181
1001899136,6cyclictest38150irq/131-snd_hda10:27:361
1001899134,5cyclictest225-21systemd-journal07:45:001
10018991310,3cyclictest27437-21apache211:22:081
10018991310,2cyclictest32613-21head11:40:151
10018991310,2cyclictest26906-21grep08:50:151
10018991310,2cyclictest23470-21grep11:15:261
10018991310,2cyclictest23350-21wc08:40:261
10018991310,2cyclictest23350-21wc08:40:251
10018991310,2cyclictest19368-21cron10:55:001
10018991310,2cyclictest15412-21taskset10:14:391
1001899130,1cyclictest141rcu_preempt07:30:001
1001399139,3cyclictest10789-21cut08:15:190
1001399138,3cyclictest19371-21cron10:55:000
10013991310,2cyclictest13946-21grep08:20:280
1001399130,12cyclictest8360-21cat08:10:190
1001399130,12cyclictest6608-21grep09:30:190
1001399130,12cyclictest1823-21sed10:25:170
1002799129,2cyclictest6027-21grep08:05:203
1002799129,2cyclictest5675-21cron09:50:003
1002799129,2cyclictest28737-21tr11:00:143
1002799129,2cyclictest22969-21grep09:20:273
1002799129,2cyclictest20553-21grep10:55:153
1002799129,2cyclictest10405-21tail08:15:163
1002799128,3cyclictest19223-21grep07:25:243
1002799126,2cyclictest225-21systemd-journal07:55:003
1002799123,4cyclictest225-21systemd-journal07:40:013
1002799121,5cyclictest0-21swapper/310:20:263
10027991212,0cyclictest0-21swapper/310:06:043
10027991210,1cyclictest7420-21cut11:25:193
1002799120,1cyclictest2686-21cstates12:20:153
1002799120,11cyclictest15086-21tr11:30:133
1002499129,2cyclictest31314-21dump-pmu-power09:00:002
1002499129,2cyclictest30411-21tail09:25:162
1002499129,2cyclictest25434-21grep10:00:252
1002499129,2cyclictest23020-21grep09:20:272
1002499129,2cyclictest17674-21grep10:15:202
1002499129,2cyclictest14646-21grep09:35:152
1002499123,6cyclictest537-21snmpd09:10:432
1002499122,7cyclictest31244-21grep11:20:182
10024991211,1cyclictest0-21swapper/211:03:072
10024991210,1cyclictest0-21swapper/211:32:392
1002499120,7cyclictest0-21swapper/210:45:512
1002499120,5cyclictest0-21swapper/210:44:562
1001899129,2cyclictest6016-21grep08:05:211
1001899129,2cyclictest5663-21wc11:05:181
1001899129,2cyclictest20129-21cut07:30:131
1001899129,2cyclictest13113-21tr07:15:151
1001899125,4cyclictest7948-21apt-get11:45:131
1001899120,9cyclictest0-21swapper/111:00:101
1001899120,1cyclictest0-21swapper/111:56:561
1001399129,2cyclictest4651-21cron08:05:000
1001399129,2cyclictest32649-21cat09:00:190
1001399129,2cyclictest27607-21head08:50:180
1001399129,2cyclictest24920-21wc07:40:000
1001399129,2cyclictest24403-21cat11:35:180
1001399129,2cyclictest24403-21cat11:35:180
1001399129,2cyclictest17143-21grep08:30:140
1001399129,2cyclictest1358-21sed11:40:240
1001399129,2cyclictest10453-21chrt11:08:260
1001399128,3cyclictest28721-21grep11:00:130
10013991210,1cyclictest0-21swapper/012:35:300
1001399120,1cyclictest141rcu_preempt10:21:250
1001399120,11cyclictest5218-21tr08:05:150
1001399120,11cyclictest17847-21basename12:10:140
1002799118,2cyclictest9927-21sed10:10:273
1002799118,2cyclictest7795-21sed08:10:153
1002799118,2cyclictest6373-21chrt11:44:123
1002799118,2cyclictest6064-21wc09:30:163
1002799118,2cyclictest4645-21sh11:05:133
1002799118,2cyclictest4086-21sh12:01:173
1002799118,2cyclictest32337-21tr09:00:173
1002799118,2cyclictest3213-21ssh09:28:393
1002799118,2cyclictest28746-21rm10:40:323
1002799118,2cyclictest28440-21ssh12:15:343
1002799118,2cyclictest23896-21ssh11:35:143
1002799118,2cyclictest23896-21ssh11:35:143
1002799118,2cyclictest22084-21ssh11:53:163
1002799118,2cyclictest20504-21ntp_states12:30:213
1002799118,2cyclictest19116-21sh11:13:353
1002799118,2cyclictest16567-21ssh09:35:513
1002799118,2cyclictest1632-21timerandwakeup07:55:253
1002799118,2cyclictest15990-21grep07:20:173
1002799118,2cyclictest11496-21wc07:10:193
1002799117,3cyclictest29317-21grep07:45:263
1002499119,1cyclictest13667-21sort11:10:162
1002499118,2cyclictest7848-21sh11:06:262
1002499118,2cyclictest6370-21tr11:25:132
1002499118,2cyclictest5715-21ssh10:27:342
1002499118,2cyclictest32342-21cut09:00:172
1002499118,2cyclictest32294-21sed07:55:012
1002499118,2cyclictest26991-21ssh10:20:292
1002499118,2cyclictest24795-21which11:35:212
1002499118,2cyclictest24795-21which11:35:212
1002499118,2cyclictest24743-21ssh11:55:092
1002499118,2cyclictest19679-21ssh10:55:032
1002499118,2cyclictest16638-21ssh09:55:242
1002499118,2cyclictest13368-21fschecks_time09:15:162
1002499118,2cyclictest13036-21ssh10:50:242
1002499118,2cyclictest11729-21taskset08:19:192
1002499118,2cyclictest11037-21tr12:25:152
1002499117,3cyclictest9406-21cron08:15:002
1002499110,10cyclictest2144-21ssh12:20:072
1001899119,1cyclictest0-21swapper/109:05:151
1001899118,2cyclictest4959-21grep09:10:171
1001899118,2cyclictest19674-21ssh11:32:381
1001899118,2cyclictest13006-21expr08:20:181
1001899118,2cyclictest12118-21tail07:10:251
1001899115,2cyclictest225-21systemd-journal08:00:001
1001399119,1cyclictest31877-21rm11:20:240
1001399119,1cyclictest25008-21tr11:55:140
1001399118,2cyclictest9852-21expr10:30:120
1001399118,2cyclictest9358-21ssh11:45:210
1001399118,2cyclictest6455-21expr11:25:130
1001399118,2cyclictest5591-21sh12:21:260
1001399118,2cyclictest4066-21ssh09:10:090
1001399118,2cyclictest30960-21ssh10:04:380
1001399118,2cyclictest30393-21expr07:50:150
1001399118,2cyclictest30091-21ssh09:25:140
1001399118,2cyclictest27469-21ssh09:42:490
1001399118,2cyclictest27315-21ssh12:35:000
1001399118,2cyclictest22781-21ssh10:18:470
1001399118,2cyclictest22595-21sort11:15:190
1001399118,2cyclictest22416-21ssh10:55:490
1001399118,2cyclictest20028-21ssh11:14:160
1001399118,2cyclictest19757-21expr08:35:140
1001399118,2cyclictest16695-21chrt09:35:580
1001399118,2cyclictest16463-21sed11:30:240
1001399118,2cyclictest1613-21sort12:00:150
1001399118,2cyclictest16002-21ssh09:55:190
1001399118,2cyclictest15954-21grep07:20:170
1001399118,2cyclictest14172-21ssh10:13:400
1001399118,2cyclictest12711-21ssh09:15:130
1001399118,2cyclictest12627-21ssh12:06:250
1001399118,2cyclictest12592-21ssh12:25:260
1001399118,2cyclictest1259-21cut07:55:190
1001399110,10cyclictest18078-21expr07:25:140
1002799108,2cyclictest3522-21sh10:45:163
1002799108,1cyclictest3616-21cat08:00:193
1002799108,1cyclictest15103-21cat08:25:163
1002799107,2cyclictest29979-21expr08:55:163
1002799107,2cyclictest25596-21expr07:40:143
1002799107,2cyclictest2332-21grep09:05:173
1002799107,2cyclictest20700-21head07:30:163
1002799107,2cyclictest11234-21sh10:50:053
1002799107,2cyclictest10985-21cut12:05:213
1002499108,1cyclictest4950-21grep08:05:142
1002499108,1cyclictest28700-21needreboot07:45:192
1002499108,1cyclictest23714-21chrt08:42:342
1002499108,1cyclictest23714-21chrt08:42:342
1002499108,1cyclictest18394-21taskset08:30:222
1002499108,1cyclictest17253-21chrt07:23:532
1002499108,1cyclictest12693-21ls07:15:012
1002499107,2cyclictest911-21grep07:55:162
1002499107,2cyclictest2760-21cat08:00:142
1002499107,2cyclictest1835-21cut09:05:162
1002499107,2cyclictest17883-21grep10:35:132
1002499100,6cyclictest225-21systemd-journal07:15:322
1001899108,1cyclictest24354-21taskset08:45:051
1001899108,1cyclictest19097-21chrt08:34:281
1001899108,1cyclictest17267-21chrt07:24:041
1001899107,2cyclictest30963-21taskset08:55:431
1001899100,9cyclictest21545-21perf08:40:001
1001399108,1cyclictest33842sleep009:06:360
1001399108,1cyclictest31221-21chrt08:59:030
1001399108,1cyclictest13160-21wc07:15:150
1001399107,2cyclictest25320-21tr07:40:130
1001399107,2cyclictest24429-21grep08:45:140
1001399107,2cyclictest16405-21ssh11:50:120
1001399107,2cyclictest15431-21grep08:25:190
1001399107,2cyclictest10658-21cpuspeed_turbos07:10:140
100279997,1cyclictest28685-21chrt08:51:573
100249997,1cyclictest262552sleep208:46:592
100249997,1cyclictest25120-21gpgv07:40:002
100249996,2cyclictest28449-21sed08:50:272
100189996,2cyclictest7593-21tr08:10:141
100189996,2cyclictest22403-21dpkg07:35:001
100139997,1cyclictest22939-21ntp_kernel_pll_08:40:200
100139997,1cyclictest22939-21ntp_kernel_pll_08:40:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional