You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 19:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Mar 21, 2026 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
394929481,9sleep00-21swapper/007:05:450
310729371,16sleep20-21swapper/207:05:202
398928263,15sleep30-21swapper/307:06:173
424528060,16sleep10-21swapper/107:09:361
177872480,0sleep30-21swapper/312:14:483
436999280,24cyclictest0-21swapper/209:51:032
4369992724,2cyclictest94-21kswapd009:35:022
436999260,26cyclictest0-21swapper/209:40:412
436999250,25cyclictest0-21swapper/210:01:182
436999250,23cyclictest0-21swapper/211:45:342
3552240,1sleep1241rcuc/112:04:081
286962240,0sleep30-21swapper/310:05:403
436999230,21cyclictest0-21swapper/209:16:012
436699230,22cyclictest0-21swapper/108:05:261
436699230,21cyclictest0-21swapper/111:35:141
437199220,0cyclictest141rcu_preempt09:16:393
437199220,0cyclictest0-21swapper/311:40:153
436999220,19cyclictest0-21swapper/212:00:062
4366992215,3cyclictest11040-21ssh12:29:511
4366992215,3cyclictest11040-21ssh12:29:501
4369992120,1cyclictest0-21swapper/210:38:512
436999200,18cyclictest0-21swapper/212:15:162
436299207,3cyclictest141rcu_preempt09:35:000
436299205,14cyclictest0-21swapper/011:05:150
436299200,3cyclictest0-21swapper/011:45:120
48812190,0sleep20-21swapper/212:25:252
48812190,0sleep20-21swapper/212:25:242
437199190,3cyclictest0-21swapper/311:38:513
436999190,1cyclictest0-21swapper/212:38:322
436999190,19cyclictest0-21swapper/211:58:012
4366991919,0cyclictest0-21swapper/110:00:381
436999180,18cyclictest0-21swapper/211:05:162
436999180,17cyclictest0-21swapper/209:33:162
436999180,13cyclictest0-21swapper/211:00:392
436699181,1cyclictest29781-21diskmemload11:13:441
436699180,18cyclictest0-21swapper/110:18:001
4371991715,1cyclictest39-21ksoftirqd/312:39:023
4371991715,1cyclictest39-21ksoftirqd/310:49:263
4371991714,2cyclictest0-21swapper/310:04:213
437199170,14cyclictest0-21swapper/312:31:263
4369991717,0cyclictest0-21swapper/210:58:072
436999170,17cyclictest0-21swapper/210:23:112
436999170,16cyclictest0-21swapper/209:58:172
4366991715,1cyclictest22071-21apt-get11:00:121
436699171,15cyclictest29781-21diskmemload09:33:581
436699171,15cyclictest29781-21diskmemload09:10:391
436699171,15cyclictest0-21swapper/108:45:141
436699170,2cyclictest0-21swapper/109:18:451
436699170,14cyclictest0-21swapper/110:05:131
437199163,9cyclictest39-21ksoftirqd/311:06:593
437199161,1cyclictest141rcu_preempt09:45:453
4371991610,5cyclictest39-21ksoftirqd/307:30:273
437199160,15cyclictest0-21swapper/308:35:143
4369991616,0cyclictest0-21swapper/209:24:102
4369991614,2cyclictest4356-21cyclictest07:10:262
436999160,16cyclictest0-21swapper/211:33:562
436999160,16cyclictest0-21swapper/210:45:292
436999160,16cyclictest0-21swapper/210:15:392
4366991614,2cyclictest0-21swapper/110:48:051
4366991612,2cyclictest19884-21taskset12:35:001
436699161,1cyclictest4740-21/usr/sbin/munin08:15:211
4366991611,4cyclictest25-21ksoftirqd/110:50:241
436699160,1cyclictest0-21swapper/109:58:071
436699160,1cyclictest0-21swapper/109:50:361
436299160,3cyclictest0-21swapper/011:39:180
437199150,0cyclictest0-21swapper/309:51:033
436999151,1cyclictest141rcu_preempt08:35:202
436999150,15cyclictest0-21swapper/211:43:192
4366991514,1cyclictest0-21swapper/111:40:531
436699150,1cyclictest0-21swapper/112:17:521
436699150,15cyclictest0-21swapper/109:28:511
436299155,5cyclictest0-21swapper/010:55:140
4362991514,1cyclictest0-21swapper/011:55:500
4362991510,3cyclictest24041-21sed11:00:250
276742150,0sleep30-21swapper/311:22:093
437199149,3cyclictest15541-21grep08:40:153
437199145,1cyclictest9014-21apt-get11:50:133
437199142,1cyclictest141rcu_preempt11:27:223
4371991411,2cyclictest5706-21grep10:30:263
4371991410,3cyclictest8190-21grep08:25:143
437199140,1cyclictest0-21swapper/311:56:473
436999146,7cyclictest23787-21apache210:06:132
436999144,6cyclictest14289-21apt-get07:30:032
436999140,12cyclictest17470-21grep07:35:142
436699142,1cyclictest23977-21ssh11:58:511
436699140,1cyclictest0-21swapper/111:47:141
436699140,13cyclictest26003-21cron07:55:001
4362991411,2cyclictest23995-21head09:25:180
437199139,3cyclictest8080-21head07:15:203
4371991313,0cyclictest0-21swapper/310:13:353
4371991310,2cyclictest537-21snmpd09:14:093
4371991310,2cyclictest4516-21grep12:25:203
4371991310,2cyclictest4516-21grep12:25:203
4371991310,2cyclictest32028-21cut09:30:163
4371991310,2cyclictest28406-21sleep08:00:013
4371991310,2cyclictest25607-21head07:50:283
4371991310,2cyclictest19725-21grep10:20:163
4371991310,2cyclictest18985-21grep08:45:203
4371991310,2cyclictest15583-21head09:20:163
4371991310,2cyclictest11152-21chrt07:20:553
436999139,3cyclictest5983-21cat09:15:002
436999139,3cyclictest18621-21head08:45:182
4369991311,1cyclictest24923-21sh09:45:152
4369991310,2cyclictest4218-21grep10:10:262
4369991310,2cyclictest4022-21head12:05:252
4369991310,2cyclictest30094-21sort08:00:262
4369991310,2cyclictest22653-21basename08:55:142
4369991310,2cyclictest20990-21grep08:50:182
436699139,4cyclictest0-21swapper/109:45:591
436699139,3cyclictest13844-21cron07:30:001
436699138,4cyclictest22970-21ssh10:40:381
4366991312,1cyclictest0-21swapper/110:30:391
4366991311,2cyclictest0-21swapper/111:24:571
4366991310,2cyclictest19350-21grep07:40:141
4366991310,2cyclictest17661-21cat07:35:161
4366991310,2cyclictest16932-21grep08:40:271
436699130,12cyclictest0-21swapper/108:25:161
436299139,3cyclictest3477-21grep08:15:120
436299139,3cyclictest10329-21wc07:20:190
436299138,3cyclictest11992-21wc12:10:250
436299136,6cyclictest7513-21perf11:30:000
436299135,2cyclictest374-21awk09:50:000
436299133,9cyclictest0-21swapper/008:45:020
436299132,3cyclictest225-21systemd-journal08:28:550
4362991311,2cyclictest13-21ksoftirqd/012:17:430
4362991311,1cyclictest29781-21diskmemload10:34:190
4362991310,2cyclictest8260-21grep09:15:250
4362991310,2cyclictest4664-21grep07:10:140
4362991310,2cyclictest30384-21wc10:45:190
4362991310,2cyclictest29445-21head08:00:180
4362991310,2cyclictest28412-21grep09:05:160
4362991310,2cyclictest21567-21sed10:40:170
4362991310,2cyclictest198-21jbd2/sda2-812:30:310
4362991310,2cyclictest19363-21tr07:40:140
4362991310,2cyclictest17418-21sh10:00:000
436299130,10cyclictest0-21swapper/010:35:350
437199129,2cyclictest7338-21tail11:10:163
437199129,2cyclictest6457-21sort08:20:163
437199129,2cyclictest4880-21tr07:10:143
437199129,2cyclictest3994-21cut08:15:153
437199129,2cyclictest2755-21latency_hist12:25:003
437199129,2cyclictest26797-21grep12:00:163
437199129,2cyclictest2343-21head11:45:203
437199129,2cyclictest18366-21tail07:35:233
437199128,3cyclictest22577-21ssh11:00:143
437199128,2cyclictest23683-21wc08:55:193
4371991212,0cyclictest0-21swapper/312:18:133
437199121,0cyclictest141rcu_preempt10:15:083
437199120,1cyclictest0-21swapper/309:25:293
436999129,2cyclictest6978-21sed10:50:262
436999129,2cyclictest3104-21cron08:15:012
436999129,2cyclictest21285-21sh10:40:142
436999129,2cyclictest15664-21grep11:15:172
436999129,2cyclictest13006-21wc07:25:222
436999129,2cyclictest12758-21cut12:30:202
436999128,3cyclictest5704-21wc08:20:002
436999128,2cyclictest21719-21tr07:45:152
436999124,4cyclictest225-21systemd-journal07:17:552
436999121,10cyclictest28466-21cut12:20:212
436999120,9cyclictest0-21swapper/212:10:022
436999120,1cyclictest0-21swapper/211:37:592
436699129,2cyclictest8864-21cut09:35:161
436699129,2cyclictest8453-21tr11:30:131
436699129,2cyclictest7896-21awk08:25:001
436699129,2cyclictest6712-21basename07:15:001
436699129,2cyclictest3141-21grep10:10:161
436699129,2cyclictest30971-21wc08:05:011
436699129,2cyclictest29033-21sh10:25:221
436699129,2cyclictest17105-21sed09:40:151
436699129,2cyclictest15063-21grep07:30:171
436699129,2cyclictest13004-21grep10:35:161
436699127,2cyclictest1187-21ssh11:06:111
436699126,4cyclictest22551-21ls09:25:011
4366991210,2cyclictest38150irq/131-snd_hda10:20:171
4366991210,2cyclictest14970-21grep10:55:231
436699120,1cyclictest0-21swapper/112:35:081
436699120,1cyclictest0-21swapper/111:18:281
436299129,2cyclictest4205-21grep10:10:250
436299129,2cyclictest2923-21head12:05:160
436299129,2cyclictest25893-21sort09:00:150
436299129,2cyclictest22009-21wc12:35:260
436299129,2cyclictest18246-21tr10:00:160
436299129,2cyclictest17303-21sort07:35:150
436299129,2cyclictest16550-21wc07:35:000
436299129,2cyclictest11882-21grep07:25:140
436299129,2cyclictest11310-21head08:30:150
436299128,2cyclictest8782-21tr09:35:170
436299128,2cyclictest17396-21cron08:45:000
436299126,2cyclictest732-21munin-run08:10:000
4362991211,1cyclictest0-21swapper/008:20:130
436299120,9cyclictest14899-21ssh11:15:130
437199119,1cyclictest39-21ksoftirqd/312:05:163
437199118,2cyclictest9973-21grep09:35:253
437199118,2cyclictest9011-21grep11:30:163
437199118,2cyclictest27555-21ssh10:25:053
437199118,2cyclictest2705-21head08:10:263
437199118,2cyclictest18430-21ssh09:40:323
437199118,2cyclictest17107-21ssh11:15:363
437199118,2cyclictest14675-21wc10:55:203
437199118,2cyclictest11345-21ssh10:53:443
437199118,2cyclictest11106-21ssh09:55:263
437199117,3cyclictest20969-21grep10:40:143
437199114,3cyclictest12190-21apache_volume10:35:113
436999118,2cyclictest4492-21ssh10:30:172
436999118,2cyclictest31529-21ssh10:26:492
436999118,2cyclictest28288-21cut09:05:172
436999118,2cyclictest23190-21ssh11:20:052
436999118,2cyclictest19858-21head07:40:162
436999118,2cyclictest11701-21ssh11:50:572
436999118,2cyclictest10851-21ssh11:12:182
436999118,2cyclictest10702-21run-parts07:20:212
436999116,1cyclictest23928-21ssh09:25:182
436999110,1cyclictest32059-21wget11:25:142
436699118,2cyclictest8756-21taskset07:15:551
436699118,2cyclictest6128-21ssh12:07:001
436699118,2cyclictest319212sleep111:25:121
436699118,2cyclictest28179-21if_enp2s012:20:191
436699118,2cyclictest26350-21tr07:55:141
436699118,2cyclictest23534-21taskset07:49:371
436699118,2cyclictest10969-21head11:50:251
436299118,2cyclictest9703-21ssh10:52:270
436299118,2cyclictest8245-21basename11:30:120
436299118,2cyclictest7722-21grep11:10:190
436299118,2cyclictest7576-21sort07:15:160
436299118,2cyclictest30549-21sort09:10:150
436299118,2cyclictest3025-21ssh12:25:020
436299118,2cyclictest3025-21ssh12:25:020
436299118,2cyclictest28289-21ssh12:00:400
436299118,2cyclictest27163-21cut10:05:170
436299118,2cyclictest26060-21cut11:40:160
436299118,2cyclictest25193-21sort11:20:260
436299118,2cyclictest25162-21ssh10:23:250
436299118,2cyclictest20837-21ssh09:23:420
436299118,2cyclictest16861-21grep09:40:140
436299118,2cyclictest13747-21ssh10:16:140
436299118,2cyclictest12732-21ssh11:51:430
436299118,2cyclictest1130-21wget08:10:130
436299118,2cyclictest1045-21ssh09:50:130
437199108,1cyclictest25922-21grep09:00:153
437199108,1cyclictest233402sleep307:47:033
437199108,1cyclictest220892sleep308:51:393
437199107,2cyclictest31726-21sort08:05:163
437199107,2cyclictest27967-21wc09:05:143
437199107,2cyclictest14091-21sort07:30:003
437199107,2cyclictest11269-21grep08:30:153
436999108,1cyclictest7608-21taskset08:21:302
436999108,1cyclictest31033-21cat08:05:112
436999108,1cyclictest27118-21cat07:55:182
436999108,1cyclictest25711-21expr09:00:132
436999108,1cyclictest247072sleep207:50:192
436999105,2cyclictest225-21systemd-journal08:32:552
436999100,9cyclictest9152-21cat08:25:212
436699108,1cyclictest9704-21ls07:20:141
436699108,1cyclictest22669-21aten2_r7power_e08:55:141
436699108,1cyclictest0-21swapper/112:10:051
436699107,2cyclictest25886-21taskset09:00:151
436699107,2cyclictest20975-21head08:50:171
436699107,2cyclictest10746-21grep08:30:131
436699106,3cyclictest13989-21grep08:35:201
436699106,3cyclictest1228-21tr08:10:141
436699100,9cyclictest27972-21cat09:05:151
436299108,1cyclictest20309-21aten2_r7power_e08:50:140
436299107,2cyclictest26880-21sort07:55:160
436299107,2cyclictest23949-21cut08:55:220
436299106,3cyclictest29191-21grep10:25:230
436299106,3cyclictest21610-21grep07:45:140
436299102,5cyclictest27888-21grep12:20:180
436299102,5cyclictest25560-21grep07:50:270
43719997,1cyclictest209682sleep307:42:033
43719992,3cyclictest29458-21gltestperf08:00:173
43699997,1cyclictest15672-21expr08:40:152
43629996,2cyclictest13109-21cat08:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional