You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-30 - 17:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jun 30, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109329366,8sleep00-21swapper/007:07:020
119829266,9sleep20-21swapper/207:08:202
109928564,6sleep10-21swapper/107:07:071
63928369,9sleep30-21swapper/307:05:283
142799243,20cyclictest21584-21/usr/sbin/munin09:45:163
141999242,9cyclictest0-21swapper/008:30:120
142499230,22cyclictest0-21swapper/210:29:262
142199220,1cyclictest0-21swapper/109:34:111
142799210,0cyclictest0-21swapper/309:56:453
1424992121,0cyclictest0-21swapper/210:05:122
142499210,20cyclictest0-21swapper/209:27:502
142199212,18cyclictest0-21swapper/109:49:131
141999207,6cyclictest23168-21sendmail-msp11:59:590
141999200,16cyclictest0-21swapper/011:35:130
142499190,19cyclictest0-21swapper/211:33:312
1421991916,2cyclictest25-21ksoftirqd/111:06:021
142199190,3cyclictest0-21swapper/110:03:571
142199190,1cyclictest26853-21diskmemload11:20:071
309382180,0chrt0-21swapper/011:25:270
142799180,17cyclictest0-21swapper/310:42:463
1424991816,1cyclictest32-21ksoftirqd/209:36:252
142499180,2cyclictest0-21swapper/211:41:312
142199181,16cyclictest6362-21apt-get09:55:151
142199180,3cyclictest0-21swapper/109:52:331
142199180,18cyclictest0-21swapper/112:01:351
142199180,17cyclictest0-21swapper/110:44:091
142799173,13cyclictest0-21swapper/312:36:113
142499172,12cyclictest22290-21/usr/sbin/munin11:20:222
1424991716,1cyclictest55550irq/126-enp2s0-09:41:262
142499170,16cyclictest0-21swapper/210:53:372
1421991714,2cyclictest38150irq/131-snd_hda12:30:351
1421991713,2cyclictest0-21swapper/108:10:011
1421991713,2cyclictest0-21swapper/108:10:011
1421991712,4cyclictest25480-21cut12:20:141
142199170,1cyclictest0-21swapper/109:26:231
142199170,0cyclictest0-21swapper/112:27:221
142799165,7cyclictest31847-21/usr/sbin/munin10:10:193
142799160,2cyclictest0-21swapper/310:24:313
142799160,2cyclictest0-21swapper/310:18:093
142499165,4cyclictest225-21systemd-journal10:59:372
1424991615,1cyclictest0-21swapper/207:47:302
1424991614,1cyclictest0-21swapper/210:00:212
142499161,1cyclictest141rcu_preempt09:56:192
142499160,16cyclictest0-21swapper/209:51:022
142499160,15cyclictest0-21swapper/211:00:292
142199163,2cyclictest20265-21sh11:20:001
1421991614,2cyclictest0-21swapper/111:26:531
1421991614,1cyclictest0-21swapper/110:55:321
1421991613,2cyclictest0-21swapper/111:11:561
142199160,15cyclictest0-21swapper/109:23:101
1427991514,1cyclictest0-21swapper/310:56:473
142499154,1cyclictest141rcu_preempt08:00:162
142499153,11cyclictest0-21swapper/209:24:442
142499150,5cyclictest0-21swapper/209:19:192
142499150,3cyclictest0-21swapper/210:46:232
142499150,14cyclictest0-21swapper/211:10:282
142499150,14cyclictest0-21swapper/210:23:382
142199154,9cyclictest9198-21/usr/sbin/munin12:10:181
1421991515,0cyclictest0-21swapper/109:37:521
1421991513,2cyclictest25-21ksoftirqd/110:11:361
142199150,1cyclictest141rcu_preempt10:29:051
142199150,1cyclictest0-21swapper/110:39:501
142199150,15cyclictest0-21swapper/112:36:561
1424991414,0cyclictest0-21swapper/208:21:452
1424991412,1cyclictest32-21ksoftirqd/212:11:022
142499140,14cyclictest0-21swapper/211:49:542
142499140,14cyclictest0-21swapper/210:42:482
142499140,13cyclictest0-21swapper/212:16:132
142499140,12cyclictest0-21swapper/212:20:192
142199142,3cyclictest225-21systemd-journal11:45:271
1421991413,1cyclictest0-21swapper/111:02:271
1421991410,3cyclictest21001-21wc08:55:221
142199140,4cyclictest0-21swapper/110:49:091
142199140,13cyclictest17793-21chrt12:15:171
1419991411,2cyclictest30889-21grep09:50:170
141999140,14cyclictest0-21swapper/012:05:120
142799139,3cyclictest12158-21sed08:35:273
1427991312,1cyclictest18344-21kworker/3:011:20:133
142799131,11cyclictest1142-21runrttasks08:30:343
1427991310,2cyclictest4908-21sed11:10:203
1427991310,2cyclictest4009-21grep08:20:173
1427991310,2cyclictest21140-21sh09:45:003
1427991310,2cyclictest17714-21ssh11:55:493
142799130,5cyclictest0-21swapper/309:16:473
142499139,3cyclictest32613-21grep08:10:252
142499139,3cyclictest25179-21grep07:55:262
142499139,3cyclictest16323-21cut07:40:142
142499136,1cyclictest32-21ksoftirqd/209:46:372
142499133,6cyclictest0-21swapper/211:50:192
142499132,1cyclictest141rcu_preempt11:18:302
142499131,11cyclictest0-21swapper/211:09:242
1424991310,2cyclictest4136-21ssh09:34:552
1424991310,2cyclictest1088-21grep08:15:152
142499130,12cyclictest0-21swapper/211:29:262
142499130,10cyclictest0-21swapper/212:06:292
142199139,2cyclictest14855-21sh11:35:201
142199138,4cyclictest6925-21ssh09:16:391
1421991310,2cyclictest6384-21tail08:25:161
1421991310,2cyclictest4339-21grep10:50:261
1421991310,2cyclictest15522-21grep07:35:261
1421991310,2cyclictest15467-21tr08:45:141
1421991310,2cyclictest13409-21cut09:40:131
142199130,5cyclictest0-21swapper/110:18:411
142199130,13cyclictest0-21swapper/110:33:531
141999131,6cyclictest4286-21awk09:35:000
1419991310,2cyclictest8299-21grep07:20:270
1419991310,2cyclictest26286-21cat09:05:280
1419991310,2cyclictest20258-21sed11:00:180
1419991310,2cyclictest14493-21grep09:40:200
1419991310,2cyclictest13370-21cat11:15:170
141999130,7cyclictest30188-21perf11:45:000
141999130,12cyclictest10920-21tr12:30:220
141999130,12cyclictest10002-21cat07:25:190
141999130,12cyclictest0-21swapper/009:24:130
142799129,2cyclictest9472-21uname12:10:203
142799129,2cyclictest4331-21grep10:50:263
142799129,2cyclictest31640-21sh12:05:003
142799129,2cyclictest31369-21dump-pmu-power10:10:003
142799129,2cyclictest30911-21grep09:50:173
142799129,2cyclictest30625-21grep09:30:253
142799129,2cyclictest23286-21ls07:55:003
142799129,2cyclictest22960-21grep09:00:163
142799129,2cyclictest1701-21df12:25:163
142799129,2cyclictest14370-21grep07:35:153
142799120,1cyclictest0-21swapper/309:12:093
142799120,1cyclictest0-21swapper/308:10:113
142799120,10cyclictest18711-21ssh11:37:503
142499129,2cyclictest7751-21sh08:30:002
142499129,2cyclictest14948-21cron08:45:002
142499129,2cyclictest14548-21tail07:35:172
142499129,2cyclictest11380-21sed10:35:252
142499129,2cyclictest10135-21grep10:15:282
142499128,3cyclictest874-21grep10:10:232
142499128,3cyclictest17324-21cron08:50:002
142499120,2cyclictest0-21swapper/211:35:132
142499120,1cyclictest0-21swapper/211:55:142
142199129,2cyclictest28251-21cut09:10:181
142199129,2cyclictest26680-21grep08:00:151
142199129,2cyclictest23136-21ls12:00:001
142199129,2cyclictest23015-21ssh11:40:171
142199128,3cyclictest2514-21tail08:15:271
142199128,3cyclictest19308-21tail07:45:181
142199128,3cyclictest11035-21diskmemload11:52:121
1421991212,0cyclictest0-21swapper/112:09:151
141999129,2cyclictest8444-21grep12:10:140
141999129,2cyclictest6603-21cat08:25:180
141999129,2cyclictest5825-21sed11:10:270
141999129,2cyclictest5722-21cat07:15:260
141999129,2cyclictest564-21grep12:25:000
141999129,2cyclictest5628-21grep09:35:170
141999129,2cyclictest4049-21sed08:20:170
141999129,2cyclictest28244-21grep10:45:270
141999129,2cyclictest28146-21taskset08:05:000
141999129,2cyclictest26626-21fgrep10:25:240
141999129,2cyclictest25203-21grep07:55:260
141999129,2cyclictest22983-21cat10:05:010
141999129,2cyclictest22337-21sh11:20:220
141999129,2cyclictest21586-21grep08:55:280
141999129,2cyclictest21258-21basename07:50:150
141999129,2cyclictest1264-21grep10:30:140
141999129,2cyclictest10598-21taskset08:35:150
141999128,3cyclictest14462-21grep08:40:260
141999128,3cyclictest12245-21ssh10:55:220
141999125,4cyclictest26853-21diskmemload12:00:210
141999124,7cyclictest14307-21apt-get09:59:590
141999123,8cyclictest8682-21/usr/sbin/munin10:15:170
141999123,7cyclictest0-21swapper/011:45:130
142799119,1cyclictest6891-21grep11:50:143
142799119,1cyclictest24994-21grep09:05:163
142799119,1cyclictest16981-21grep07:40:173
142799119,1cyclictest15833-21head10:00:163
142799118,2cyclictest9887-21expr10:35:143
142799118,2cyclictest9153-21grep07:25:143
142799118,2cyclictest8394-21taskset07:20:343
142799118,2cyclictest5861-21grep07:15:263
142799118,2cyclictest4360-21ssh10:31:273
142799118,2cyclictest409-21grep12:05:163
142799118,2cyclictest3254-21ssh11:47:443
142799118,2cyclictest29074-21grep08:05:183
142799118,2cyclictest29074-21grep08:05:183
142799118,2cyclictest28233-21cron09:30:003
142799118,2cyclictest28055-21cron08:05:003
142799118,2cyclictest25858-21rm11:41:403
142799118,2cyclictest25707-21grep12:20:163
142799118,2cyclictest22161-21ssh12:18:073
142799118,2cyclictest19383-21ssh11:00:133
142799118,2cyclictest1903-21grep07:10:143
142799118,2cyclictest18878-21tr07:45:153
142799118,2cyclictest17924-21cat08:50:153
142799118,2cyclictest16309-21grep08:45:193
142799118,2cyclictest15516-21ssh12:33:273
142799118,2cyclictest14293-21chrt09:20:473
142799118,2cyclictest12833-21expr11:15:143
142799118,2cyclictest1145-21grep08:15:153
142799117,3cyclictest29672-21sed11:05:263
142799110,8cyclictest0-21swapper/311:25:023
142499118,2cyclictest8301-21grep08:30:142
142499118,2cyclictest5124-21ssh10:32:032
142499118,2cyclictest3300-21sed07:10:252
142499118,2cyclictest30467-21wc08:10:002
142499118,2cyclictest30467-21wc08:10:002
142499118,2cyclictest25462-21grep09:05:192
142499118,2cyclictest23567-21ssh12:00:082
142499118,2cyclictest23037-21cut09:00:152
142499118,2cyclictest18245-21ssh12:35:132
142499118,2cyclictest13562-21ssh12:31:572
142499118,2cyclictest12121-21awk08:35:272
142499118,2cyclictest11331-21grep07:30:002
142499110,2cyclictest0-21swapper/209:12:412
142499110,2cyclictest0-21swapper/208:50:142
142199118,2cyclictest8249-21tr08:30:141
142199118,2cyclictest6313-21cat11:30:211
142199118,2cyclictest4753-21sed07:15:161
142199118,2cyclictest25291-21grep09:05:171
142199118,2cyclictest25143-21cat10:05:241
142199118,2cyclictest16422-21cat07:40:151
142199118,2cyclictest13101-21tr08:40:151
142199110,10cyclictest19600-21taskset08:53:491
141999119,1cyclictest19087-21head12:35:170
141999119,1cyclictest18245-21expr10:40:140
141999119,1cyclictest17432-21grep12:15:140
141999118,2cyclictest4986-21chrt11:30:080
141999118,2cyclictest4402-21ssh12:26:260
141999118,2cyclictest4069-21wc10:50:250
141999118,2cyclictest3764-21sh07:15:000
141999118,2cyclictest31244-21ls08:10:130
141999118,2cyclictest28975-21cut08:05:170
141999118,2cyclictest28975-21cut08:05:170
141999118,2cyclictest28936-21seq11:05:190
141999118,2cyclictest2875-21ssh10:11:310
141999118,2cyclictest27531-21ssh09:10:150
141999118,2cyclictest25262-21ssh09:46:400
141999118,2cyclictest18789-21sh10:20:320
141999118,2cyclictest18670-21awk07:45:150
141999118,2cyclictest1163-21seq08:15:150
141999118,2cyclictest11519-21grep07:30:140
141999118,2cyclictest10058-21ssh09:19:050
142799107,2cyclictest6880-21grep08:25:213
142799107,2cyclictest5591-21expr11:30:163
142799107,2cyclictest32621-21ssh10:48:423
142799107,2cyclictest24907-21ssh10:25:073
142799107,2cyclictest24393-21cat07:55:173
142799107,2cyclictest20110-21tr08:55:163
142799107,2cyclictest13549-21tr08:40:173
142799107,2cyclictest10809-21ssh09:38:323
142499108,1cyclictest60342sleep207:16:052
142499108,1cyclictest21295-21wc08:55:242
142499107,2cyclictest5604-21ssh12:27:222
142499107,2cyclictest22648-21sshd07:50:262
142499102,3cyclictest32-21ksoftirqd/207:30:142
142199108,1cyclictest6769-21basename07:20:141
142199108,1cyclictest253922sleep107:56:391
142199108,1cyclictest24281-21taskset09:02:391
142199108,1cyclictest22963-21chrt07:50:581
142199108,1cyclictest10021-21sed07:25:191
142199107,2cyclictest4990-21head08:20:271
142199107,2cyclictest1917-21tr07:10:141
142199107,2cyclictest17937-21awk10:20:201
142199107,2cyclictest13189-21chrt07:30:261
142199106,3cyclictest31676-21cut08:10:151
142199100,9cyclictest11416-21ls08:35:191
141999108,1cyclictest23384-21ssh10:05:100
141999108,1cyclictest20781-21expr09:25:140
141999108,1cyclictest194572sleep008:51:590
141999108,1cyclictest16303-21hostname07:40:140
141999107,2cyclictest8522-21wc11:50:260
141999107,2cyclictest12507-21ssh10:36:110
141999100,9cyclictest15323-21wget08:45:140
14279997,1cyclictest11707-21grep07:30:143
14199997,1cyclictest22680-21expr09:00:140
14199997,1cyclictest13827-21apt-get07:35:130
14249985,2cyclictest8438-21taskset07:21:082
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional