You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-30 - 08:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 30, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2712129986,8sleep00-21swapper/019:08:150
2712129986,8sleep00-21swapper/019:08:140
2715829279,9sleep20-21swapper/219:08:452
2715829279,9sleep20-21swapper/219:08:442
2696028774,9sleep30-21swapper/319:06:093
2696028774,9sleep30-21swapper/319:06:083
2710228669,13sleep10-21swapper/119:08:001
2710228669,13sleep10-21swapper/119:07:591
199212560,1sleep232-21ksoftirqd/221:28:522
27336992619,7cyclictest25-21ksoftirqd/122:35:171
27336992619,5cyclictest25-21ksoftirqd/122:54:451
2734099250,22cyclictest0-21swapper/219:55:132
27345992422,2cyclictest39-21ksoftirqd/319:50:003
27336992423,1cyclictest0-21swapper/100:15:501
27345992321,1cyclictest39-21ksoftirqd/300:06:133
27336992316,7cyclictest25-21ksoftirqd/121:39:061
2733699228,14cyclictest241rcuc/122:17:171
2733699221,17cyclictest25-21ksoftirqd/119:35:161
2733699220,2cyclictest0-21swapper/121:57:101
2733699220,22cyclictest0-21swapper/123:01:501
307252210,0sleep00-21swapper/000:09:160
2734099211,1cyclictest0-21swapper/223:57:092
2734099210,1cyclictest0-21swapper/222:13:022
2733699210,2cyclictest0-21swapper/123:21:301
27345992016,2cyclictest39-21ksoftirqd/322:28:403
27345992015,4cyclictest39-21ksoftirqd/300:20:003
2734599201,16cyclictest0-21swapper/322:07:293
27340992018,2cyclictest94-21kswapd021:38:132
2733699206,12cyclictest25-21ksoftirqd/121:28:271
27336992019,1cyclictest0-21swapper/121:17:031
27336992018,2cyclictest0-21swapper/122:13:281
2734599195,2cyclictest141rcu_preempt23:35:143
2734099190,1cyclictest0-21swapper/222:58:122
2734099190,18cyclictest0-21swapper/222:20:122
2733699195,12cyclictest25-21ksoftirqd/122:45:411
2733699191,5cyclictest25-21ksoftirqd/123:17:291
2733699190,18cyclictest0-21swapper/123:31:371
2733699190,18cyclictest0-21swapper/122:57:391
2733699190,18cyclictest0-21swapper/121:34:521
2734599180,4cyclictest141rcu_preempt21:56:543
27340991817,1cyclictest0-21swapper/222:40:532
27340991817,1cyclictest0-21swapper/200:02:532
2734099180,1cyclictest0-21swapper/221:33:432
2734099180,1cyclictest0-21swapper/221:24:132
2734099180,18cyclictest0-21swapper/223:40:522
2734099180,18cyclictest0-21swapper/223:33:322
2734099180,18cyclictest0-21swapper/223:21:222
2734099180,18cyclictest0-21swapper/223:12:522
2734099180,18cyclictest0-21swapper/222:03:122
2734099180,18cyclictest0-21swapper/200:11:522
27336991817,1cyclictest25-21ksoftirqd/122:07:551
27336991816,1cyclictest20374-21diskmemload22:22:371
2733699180,3cyclictest25-21ksoftirqd/121:48:191
2733699180,1cyclictest0-21swapper/122:28:391
27335991817,1cyclictest0-21swapper/019:40:130
2733599180,1cyclictest0-21swapper/023:57:300
2733599180,1cyclictest0-21swapper/021:55:180
2734599174,3cyclictest141rcu_preempt22:20:233
2734599173,2cyclictest2310-21grep20:30:173
2734599172,3cyclictest141rcu_preempt22:10:193
27345991715,1cyclictest39-21ksoftirqd/300:27:213
2734599170,0cyclictest0-21swapper/322:34:233
2734099170,1cyclictest0-21swapper/223:02:332
2734099170,17cyclictest0-21swapper/200:08:122
2733699176,10cyclictest241rcuc/120:16:221
27336991716,1cyclictest0-21swapper/123:59:561
27336991715,2cyclictest25-21ksoftirqd/100:20:121
27336991715,1cyclictest20374-21diskmemload23:54:581
2733699171,4cyclictest25-21ksoftirqd/121:10:141
27336991713,3cyclictest25-21ksoftirqd/119:35:011
2733699170,3cyclictest25-21ksoftirqd/119:23:561
2733699170,3cyclictest0-21swapper/121:50:441
2733599179,7cyclictest7241-21perf19:35:010
2734599165,3cyclictest141rcu_preempt22:45:253
2734599163,3cyclictest225-21systemd-journal22:17:013
27345991615,1cyclictest39-21ksoftirqd/300:04:003
27345991614,2cyclictest39-21ksoftirqd/300:10:153
27345991614,1cyclictest39-21ksoftirqd/323:20:013
27345991612,4cyclictest141rcu_preempt22:39:393
2734599160,1cyclictest141rcu_preempt23:15:133
2734099162,9cyclictest4856-21munin-run19:30:012
27340991615,1cyclictest0-21swapper/222:38:362
27340991610,4cyclictest55650irq/127-enp2s0-22:48:012
2733699163,5cyclictest25-21ksoftirqd/123:39:561
2733699161,11cyclictest0-21swapper/122:01:341
2733599165,6cyclictest25192-21awk22:30:000
2733599160,0cyclictest0-21swapper/000:04:290
27345991514,1cyclictest39-21ksoftirqd/321:13:183
27345991514,1cyclictest0-21swapper/323:56:413
27345991513,1cyclictest55850irq/129-enp2s0-00:35:303
27345991513,1cyclictest0-21swapper/323:01:383
2734599151,2cyclictest141rcu_preempt23:47:303
27345991512,2cyclictest39-21ksoftirqd/323:30:233
27345991512,2cyclictest39-21ksoftirqd/323:10:273
2734599151,1cyclictest141rcu_preempt21:50:193
2734599151,13cyclictest0-21swapper/321:35:213
2734099157,6cyclictest0-21swapper/220:30:132
27340991512,2cyclictest32-21ksoftirqd/222:32:172
2734099150,2cyclictest0-21swapper/223:15:122
2733699156,9cyclictest25-21ksoftirqd/122:32:351
2733699155,9cyclictest20374-21diskmemload00:00:441
2733699154,5cyclictest25-21ksoftirqd/123:43:131
2733699153,6cyclictest25-21ksoftirqd/100:30:011
139012150,2sleep30-21swapper/322:41:483
2734599149,3cyclictest39-21ksoftirqd/323:50:143
27340991410,3cyclictest8358-21chmod22:20:012
2733599149,3cyclictest31572-21sed21:35:200
2733599143,2cyclictest0-21swapper/019:20:130
27335991410,3cyclictest11631-21tr23:00:160
27335991410,2cyclictest28282-21wc20:15:260
2733599140,12cyclictest19385-21grep22:45:170
2733599140,12cyclictest0-21swapper/023:17:160
2734599139,3cyclictest8961-21cat19:35:253
2734599139,3cyclictest7752-21tr22:00:133
2734599139,3cyclictest39-21ksoftirqd/322:54:583
2734599138,1cyclictest19616-21ssh21:47:503
2734599137,3cyclictest475-21in:imuxsock21:16:323
2734599133,7cyclictest39-21ksoftirqd/321:42:363
2734599132,5cyclictest225-21systemd-journal20:45:003
27345991312,1cyclictest0-21swapper/300:31:223
2734599131,1cyclictest141rcu_preempt23:41:383
27345991311,1cyclictest381rcuc/321:20:153
27345991310,2cyclictest16791-21sh19:55:003
2734099139,3cyclictest27765-21ls20:15:192
27340991310,2cyclictest8831-21cron00:35:002
27340991310,2cyclictest581-21cat00:30:012
27340991310,2cyclictest32-21ksoftirqd/221:19:322
27340991310,2cyclictest2497-21cat19:25:002
27340991310,2cyclictest23523-21cat20:05:252
27340991310,2cyclictest20910-21wc23:05:212
27340991310,2cyclictest17277-21grep00:20:162
27340991310,2cyclictest15574-21cron21:00:002
2734099130,1cyclictest0-21swapper/220:03:122
2734099130,1cyclictest0-21swapper/200:18:342
2734099130,12cyclictest0-21swapper/222:51:502
2734099130,12cyclictest0-21swapper/200:38:362
2734099130,0cyclictest0-21swapper/223:28:062
2733699139,3cyclictest0-21swapper/123:25:241
2733699135,7cyclictest0-21swapper/100:07:541
2733699132,11cyclictest25-21ksoftirqd/100:30:101
2733699131,2cyclictest8408-21apt20:45:001
27336991312,1cyclictest0-21swapper/121:44:341
27336991310,2cyclictest32350-21cut20:25:171
2733699130,4cyclictest0-21swapper/123:10:261
2733599139,3cyclictest3068-21wc20:30:260
2733599139,3cyclictest10519-21tr22:40:150
2733599135,7cyclictest4857-21awk19:30:000
2733599135,5cyclictest23838-21idleruntime-cro22:10:000
2733599132,10cyclictest29715-21cron21:35:000
2733599131,9cyclictest25027-21awk22:10:160
27335991310,2cyclictest8833-21sh00:35:000
27335991310,2cyclictest32518-21grep19:20:000
27335991310,2cyclictest19863-21sed21:05:280
27335991310,2cyclictest19863-21sed21:05:270
27335991310,2cyclictest1121-21dump-pmu-power22:35:010
27335991310,2cyclictest10087-21wc20:45:250
2733599130,12cyclictest10312-21head22:20:230
2733599130,0cyclictest0-21swapper/022:18:290
2734599129,2cyclictest18522-21grep00:20:263
2734599124,1cyclictest141rcu_preempt23:27:003
2734599123,2cyclictest537-21snmpd21:28:563
27345991210,2cyclictest39-21ksoftirqd/323:08:533
27345991210,2cyclictest39-21ksoftirqd/322:55:353
2734599120,11cyclictest4832-21cron19:30:013
2734099129,2cyclictest6789-21cat23:35:212
2734099129,2cyclictest30303-21munin-plugin-st23:49:592
2734099129,2cyclictest2981-21ssh21:56:472
2734099129,2cyclictest26036-21grep20:10:252
2734099129,2cyclictest18012-21rm22:25:162
2734099129,2cyclictest14849-21tr21:45:152
2734099129,2cyclictest1267-21ssh23:51:082
2734099129,2cyclictest12157-21taskset21:43:352
2734099124,3cyclictest18001-21munin-plugin-st21:05:012
2734099123,5cyclictest14381-21apache219:30:112
2734099120,1cyclictest0-21swapper/221:13:032
2733699129,2cyclictest4660-21tail20:35:191
2733699129,2cyclictest30297-21ssh23:49:591
2733699129,2cyclictest10213-21head00:35:181
2733699126,6cyclictest0-21swapper/122:40:151
2733699125,4cyclictest25-21ksoftirqd/100:13:551
2733699124,4cyclictest141rcu_preempt20:45:251
2733699122,6cyclictest25-21ksoftirqd/120:10:161
27336991211,1cyclictest20374-21diskmemload23:07:161
2733599129,2cyclictest8447-21sed22:00:160
2733599129,2cyclictest3680-21chrt00:11:550
2733599129,2cyclictest314442sleep023:50:150
2733599129,2cyclictest28588-21tr23:10:140
2733599129,2cyclictest24673-21tr20:10:140
2733599129,2cyclictest23787-21cut21:50:130
2733599129,2cyclictest19160-21cron20:00:000
2733599129,2cyclictest17263-21ssh21:45:580
2733599129,2cyclictest13698-21grep23:20:160
2733599129,2cyclictest12973-21grep19:45:150
2733599128,2cyclictest22264-21sed21:10:230
2733599123,6cyclictest0-21swapper/019:10:140
2733599123,6cyclictest0-21swapper/019:10:130
27335991211,1cyclictest0-21swapper/023:42:580
2733599120,1cyclictest0-21swapper/023:34:490
2733599120,12cyclictest0-21swapper/021:28:220
2734599119,1cyclictest24823-21expr20:10:153
2734599118,2cyclictest7430-21cat19:35:003
2734599118,2cyclictest5510-21grep20:35:283
2734599118,2cyclictest28638-21taskset20:16:393
2734599118,2cyclictest22032-21cut21:30:133
2734599114,3cyclictest225-21systemd-journal19:45:003
2734599111,4cyclictest225-21systemd-journal19:21:103
2734099118,2cyclictest8940-21tr20:45:142
2734099118,2cyclictest6271-21cat20:40:132
2734099118,2cyclictest4286-21cat20:35:172
2734099118,2cyclictest32563-21date19:20:012
2734099118,2cyclictest31812-21grep20:25:152
2734099118,2cyclictest28600-21wc19:10:172
2734099118,2cyclictest28600-21wc19:10:162
2734099118,2cyclictest26144-21ssh21:51:162
2734099118,2cyclictest21411-21ssh22:08:062
2734099118,2cyclictest20386-21basename21:10:002
2734099118,2cyclictest20386-21basename21:10:002
2734099118,2cyclictest15113-21expr19:50:172
2734099118,2cyclictest11801-21head20:50:172
2734099118,2cyclictest11482-21grep19:40:262
2733699119,1cyclictest6504-21awk21:20:221
2733699118,2cyclictest29440-21tr20:20:141
2733699118,2cyclictest22758-21grep20:05:181
2733699118,2cyclictest20541-21tail20:00:181
2733699118,2cyclictest18932-21grep21:05:171
2733699118,2cyclictest18932-21grep21:05:171
2733699118,2cyclictest15670-21wc19:50:211
2733699118,2cyclictest11719-21chrt19:41:451
2733699113,3cyclictest225-21systemd-journal19:26:101
2733599118,2cyclictest8411-21ssh21:40:380
2733599118,2cyclictest8171-21ssh21:21:130
2733599118,2cyclictest8042-21taskset23:35:460
2733599118,2cyclictest7727-21expr19:35:150
2733599118,2cyclictest3136-21cut22:55:170
2733599118,2cyclictest24463-21ssh23:45:270
2733599118,2cyclictest22806-21fgrep23:25:210
2733599118,2cyclictest20558-21grep23:05:190
2733599118,2cyclictest17522-21sed21:00:250
2733599118,2cyclictest17113-21rm00:20:150
2733599118,2cyclictest11297-21sed00:35:270
2733599118,2cyclictest0-21swapper/020:00:140
2733599117,3cyclictest26782-21sed00:25:250
2733599115,4cyclictest121092sleep020:50:200
2733599111,2cyclictest0-21swapper/020:35:150
2733599110,10cyclictest2415-21grep22:35:170
2734599109,1cyclictest14582-21cut20:55:183
2734599108,2cyclictest21232-21taskset20:00:543
2734599108,1cyclictest9389-21grep20:45:183
2734599108,1cyclictest30820-21sort19:15:193
2734599108,1cyclictest240352sleep320:08:393
2734599108,1cyclictest11279-21cut20:50:143
2734599105,3cyclictest19386-21/usr/sbin/munin21:05:223
2734599105,3cyclictest19386-21/usr/sbin/munin21:05:213
2734599104,2cyclictest225-21systemd-journal21:05:013
2734599100,2cyclictest0-21swapper/319:57:183
2734599100,10cyclictest31946-21expr20:25:153
2734099108,1cyclictest8631-21ntpq19:35:212
2734099108,1cyclictest291642sleep220:20:082
2733699108,1cyclictest32391-21dpkg19:20:011
2733699108,1cyclictest16018-21wc21:00:141
2733699107,2cyclictest28898-21cut19:10:201
2733699107,2cyclictest28898-21cut19:10:191
2733699107,2cyclictest18923-21chrt19:56:591
2733699107,2cyclictest1513-21basename20:30:111
2733699107,2cyclictest14120-21grep20:55:161
2733699103,4cyclictest25-21ksoftirqd/119:45:111
2733699102,5cyclictest0-21swapper/120:50:201
2733599108,1cyclictest32063-21chrt21:16:040
2733599108,1cyclictest302132sleep022:51:250
2733599108,1cyclictest22318-21expr20:05:150
2733599108,1cyclictest15521-21if_enp0s31f619:50:190
2733599107,2cyclictest30965-21taskset20:20:450
2733599107,2cyclictest13739-21expr20:55:130
2733599103,6cyclictest0-21swapper/000:15:150
2733599100,9cyclictest7120-21basename20:40:210
273459998,1cyclictest27899-21awk19:10:133
273459998,1cyclictest27899-21awk19:10:133
273459997,1cyclictest31037-21taskset20:21:403
273409996,2cyclictest12836-21cat19:45:142
273359996,2cyclictest32120-21grep20:25:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional