You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-09 - 01:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Jul 08, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30749222098,97sleep00-21swapper/007:09:410
288702174100,25sleep10-21swapper/107:05:061
305382152130,15sleep20-21swapper/207:06:592
305542140103,25sleep30-21swapper/307:07:133
295872790,2sleep20-21swapper/210:05:182
294802640,2chrt0-21swapper/212:05:062
238822590,1sleep30-21swapper/310:00:393
93862570,1sleep00-21swapper/010:38:430
253352570,3sleep20-21swapper/209:14:552
194132560,2sleep30-21swapper/311:33:573
134502560,1sleep00-21swapper/011:05:170
24032550,3sleep00-21swapper/009:20:510
24032550,3sleep00-21swapper/009:20:500
114492550,2sleep3331rcuc/309:28:073
261502530,2sleep30-21swapper/312:25:463
261502530,2sleep30-21swapper/312:25:453
313812520,1sleep00-21swapper/011:18:490
74742510,2sleep10-21swapper/111:25:111
47942510,1sleep00-21swapper/010:59:230
193982480,2sleep30-21swapper/311:10:043
3104499450,5cyclictest4386-21ssh11:22:210
31044994410,18cyclictest18620-21diskmemload10:43:570
241612440,1sleep00-21swapper/010:00:560
22552420,3sleep334-21ksoftirqd/310:33:203
3104499369,16cyclictest18620-21diskmemload10:09:430
3104499360,6cyclictest24003-21ssh10:25:170
3104499360,4cyclictest31282-21grep12:05:250
134152360,1chrt0-21swapper/310:17:193
31044993511,21cyclictest8956-21date11:50:010
3104499350,6cyclictest6886-21sort11:00:170
3104499350,6cyclictest21642-21head11:35:140
3104499350,5cyclictest21322-21ssh10:47:100
31044993411,20cyclictest2704-21sed08:30:210
3104499340,5cyclictest2237-21tail07:15:250
3104499339,21cyclictest20503-21cat09:10:220
3104499332,26cyclictest0-21swapper/009:45:120
313802320,1sleep30-21swapper/309:42:453
3104499328,21cyclictest27117-21cut08:15:000
3104499328,21cyclictest23253-21gltestperf08:05:160
3104499328,20cyclictest6590-21grep07:25:260
3104499327,5cyclictest25238-21kworker/u8:109:54:120
3104499321,6cyclictest3989-21sshd12:34:160
3104499321,28cyclictest24929-21tr10:50:110
31044993210,20cyclictest14471-21readlink09:00:020
3104499320,6cyclictest9702-21sshd08:48:250
3104499320,4cyclictest21613-21ssh09:35:220
3104499318,20cyclictest10807-21grep07:35:260
3104499312,6cyclictest18620-21diskmemload09:40:120
3104499312,2cyclictest0-21swapper/012:15:100
3104499311,7cyclictest9437-21cstates11:50:130
3104499311,26cyclictest18620-21diskmemload11:30:220
31044993111,18cyclictest25796-21kworker/u8:109:15:220
31044993111,17cyclictest26079-21sh12:02:080
3104499310,6cyclictest8479-21ssh12:13:180
3104499310,5cyclictest636-21runrttasks12:25:180
3104499310,5cyclictest636-21runrttasks12:25:170
3104499310,5cyclictest32539-21ssh10:31:140
3104499310,4cyclictest9752-21apt10:15:090
3104499305,4cyclictest7623-21sh11:25:130
3104499302,3cyclictest0-21swapper/011:40:120
3104499300,4cyclictest30942-21users08:20:250
3104499299,17cyclictest17319-21/usr/sbin/munin12:20:150
3104499293,3cyclictest0-21swapper/007:10:110
3104499292,3cyclictest0-21swapper/007:45:130
3104499292,11cyclictest13821-21ssh09:30:130
3104499291,8cyclictest13747-21latency_hist07:44:590
3104499291,7cyclictest16821-21gltestperf07:50:150
3104499291,6cyclictest18039-21timerwakeupswit10:20:240
3104499291,3cyclictest0-21swapper/011:55:120
31044992912,15cyclictest6998-21sh09:25:120
31044992911,15cyclictest18620-21diskmemload09:55:560
3104499290,9cyclictest3559-21tune2fs10:10:170
3104499285,6cyclictest8988-21seq07:33:120
3104499281,8cyclictest5647-21grep12:35:150
3104499281,7cyclictest579-21grep08:25:240
3104499281,7cyclictest28671-21grep08:15:260
3104499281,7cyclictest19322-21needreboot07:55:210
3104499281,4cyclictest230192sleep011:12:100
31044992810,15cyclictest17136-21fschecks_time09:05:140
3104499280,6cyclictest12841-21if_enp3s008:55:170
31046992716,7cyclictest52850irq/140-enp3s0-10:10:142
3104599271,24cyclictest0-21swapper/107:15:051
3104599271,24cyclictest0-21swapper/107:14:181
3104499277,3cyclictest21225-21wc08:00:150
3104499274,6cyclictest6948-21mailstats08:40:220
3104499272,8cyclictest2978-21apt-get07:20:110
152142270,2sleep20-21swapper/212:18:192
65132260,2sleep10-21swapper/112:11:191
31047992623,2cyclictest0-21swapper/307:15:493
31045992610,8cyclictest141rcu_preempt12:06:071
3104499261,8cyclictest10463-21tune2fs08:50:160
3104499261,6cyclictest4418-21wc08:35:180
3104799251,23cyclictest0-21swapper/307:14:313
31045992515,6cyclictest52950irq/141-enp3s0-08:55:131
31045992512,8cyclictest52950irq/141-enp3s0-10:15:111
31047992414,7cyclictest578-21df09:20:143
31047992414,7cyclictest578-21df09:20:133
31046992422,1cyclictest0-21swapper/207:16:102
3104699241,22cyclictest0-21swapper/207:14:162
3104599248,14cyclictest141rcu_preempt10:02:491
31045992414,7cyclictest30592-21python308:20:231
93672232,4sleep228-21ksoftirqd/207:35:112
3104799233,6cyclictest18620-21diskmemload10:45:113
31047992317,4cyclictest5912-21sh11:00:023
31046992314,7cyclictest13144-21ntp_states08:55:222
3104599238,2cyclictest141rcu_preempt11:45:141
31045992316,5cyclictest22-21ksoftirqd/110:25:001
277022230,1sleep20-21swapper/209:40:142
242722230,20sleep124287-21rm11:36:521
3104799227,5cyclictest0-21swapper/312:30:253
3104699222,14cyclictest16522-21apt-get10:20:112
31046992216,3cyclictest12370-21grep10:40:222
31046992214,5cyclictest0-21swapper/211:30:112
3104599225,8cyclictest141rcu_preempt09:51:561
31045992212,7cyclictest21598-21pluginstate08:00:211
229802220,1sleep20-21swapper/211:12:092
61812210,3sleep10-21swapper/108:40:151
3104799219,3cyclictest7392sed10:55:263
31047992113,6cyclictest21549-21ls09:35:203
31047992112,7cyclictest15420-21ls09:00:203
31047992112,6cyclictest8113-21netstat07:30:193
31047992111,7cyclictest15161-21python307:45:243
31047992110,7cyclictest4997-21cut12:10:213
31047992110,5cyclictest383-21cat08:25:223
3104699212,13cyclictest7464-21sh11:25:122
3104699212,13cyclictest5930-21sh11:00:042
31046992115,4cyclictest3072-21switchtime09:45:252
31046992113,6cyclictest7113-21awk09:25:142
3104599218,3cyclictest22-21ksoftirqd/110:10:261
3104599213,3cyclictest141rcu_preempt09:45:121
3104599212,16cyclictest20700-21tr09:35:131
3104599211,7cyclictest141rcu_preempt12:00:001
3104599211,3cyclictest141rcu_preempt12:30:241
31045992113,6cyclictest11260-21cut10:40:121
31045992111,8cyclictest30223-21ssh12:29:531
31045992111,8cyclictest30223-21ssh12:29:521
31045992111,7cyclictest15897-21ls07:50:001
3104799202,14cyclictest25911-21ssh09:15:133
3104799202,13cyclictest15174-21apt-get09:55:123
31047992013,5cyclictest53050irq/142-enp3s0-10:50:123
31047992013,5cyclictest25455-21cstates08:10:143
31047992012,6cyclictest8558-21pluginstate11:25:223
31047992012,6cyclictest5606-21run-parts10:35:213
31047992012,6cyclictest27431-21basename11:39:593
31047992012,6cyclictest22176-21seq08:00:323
31047992012,6cyclictest17483-21ls09:05:173
31047992012,6cyclictest17234-21ntpq07:50:193
31047992011,6cyclictest11598-21cat12:40:013
3104799201,13cyclictest2420-21ssh11:45:113
3104699206,3cyclictest141rcu_preempt11:20:202
3104699206,10cyclictest19945-21kworker/u8:010:50:122
3104699203,12cyclictest75-21kswapd012:20:122
3104699202,13cyclictest9183-21ssh11:50:112
31046992013,4cyclictest2023-21cut08:30:132
31046992010,8cyclictest560-21dbus-daemon09:10:002
3104599209,9cyclictest22-21ksoftirqd/111:31:191
31045992012,6cyclictest4698-21entropy10:35:131
31045992012,6cyclictest29322-21seq10:29:451
31045992012,6cyclictest27413-21cut08:15:121
31045992012,6cyclictest19855-21seq09:58:231
31045992012,6cyclictest18356-21fw_packets10:45:141
31045992012,6cyclictest16851-21seq09:31:461
31045992012,6cyclictest13239-21forks11:05:151
31045992012,6cyclictest10444-21seq11:03:041
31045992011,7cyclictest6826-21awk12:35:231
31045992011,7cyclictest27318-21seq10:51:121
187542200,1sleep30-21swapper/307:55:153
139392200,2sleep10-21swapper/112:17:011
3104799199,6cyclictest53050irq/142-enp3s0-10:25:133
3104799192,12cyclictest19013-21apt-get09:10:103
3104799191,8cyclictest22853-21idleruntime-cro10:25:003
3104799191,7cyclictest18620-21diskmemload09:52:073
3104799191,4cyclictest0-21swapper/312:05:343
31047991913,4cyclictest7264-21sh10:12:563
31047991913,4cyclictest30196-21missed_timers08:20:193
31047991913,4cyclictest25714-21sh12:01:463
31047991913,4cyclictest1554-21sh09:45:063
31047991911,6cyclictest8779-21ls08:45:193
31047991911,6cyclictest5880-21cat08:40:133
31047991911,6cyclictest16070-21seq11:06:573
31047991911,6cyclictest12406-21ls08:55:133
31047991911,6cyclictest11048-21tune2fs12:15:143
31047991911,6cyclictest10294-21ntpq07:35:213
31047991911,5cyclictest14549-21irqstats09:30:193
31047991910,7cyclictest28998-21cat10:05:133
31047991910,7cyclictest1155-21cat11:20:133
31047991910,6cyclictest4424-21missed_timers08:35:183
31047991910,6cyclictest278-21systemd-journal08:19:593
3104699199,7cyclictest278-21systemd-journal11:44:592
3104699197,3cyclictest260392ssh12:25:402
3104699197,3cyclictest260392ssh12:25:402
3104699194,3cyclictest141rcu_preempt10:45:102
3104699193,11cyclictest31552-21apt-get10:55:122
31046991914,3cyclictest30741-21acpi12:30:102
31046991914,3cyclictest28538-21sh09:16:212
31046991913,4cyclictest300-21sh10:31:272
31046991912,5cyclictest21194-21grep09:35:182
31046991912,5cyclictest16549-21expr07:50:132
31046991912,4cyclictest26265-21sh11:38:512
31046991912,4cyclictest18620-21diskmemload12:14:172
31046991911,4cyclictest18620-21diskmemload12:04:462
3104699191,13cyclictest15187-21taskset09:55:122
31046991910,7cyclictest560-21dbus-daemon07:35:002
31046991910,6cyclictest20644-21apt08:00:112
3104599199,8cyclictest20521-21ssh12:21:521
3104599197,3cyclictest22-21ksoftirqd/109:40:131
3104599192,15cyclictest141rcu_preempt11:16:381
31045991914,3cyclictest1037-21cron08:29:591
31045991913,4cyclictest25128-21sh12:01:111
31045991913,4cyclictest10145-21cpuspeed_turbos08:50:141
31045991912,5cyclictest1194-21needreboot09:20:191
31045991912,5cyclictest1194-21needreboot09:20:181
31045991912,4cyclictest32394-21latency_hist09:20:011
31045991911,6cyclictest24379-21seq08:06:221
31045991911,6cyclictest20819-21tr11:10:221
31045991911,6cyclictest16309-21seq07:50:021
31045991911,6cyclictest12745-21ntp_kernel_err07:40:201
31045991911,6cyclictest12014-21seq09:28:401
31045991911,6cyclictest11271-21seq07:38:471
31045991910,7cyclictest7999-21cat07:30:181
31045991910,7cyclictest3407-21tune2fs07:20:151
31045991910,7cyclictest29034-21cat10:05:141
31045991910,6cyclictest278-21systemd-journal09:00:011
31045991910,5cyclictest52950irq/141-enp3s0-11:20:121
156832190,2chrt0-21swapper/210:19:352
3104799187,8cyclictest10221-21sh11:50:213
3104799181,6cyclictest18620-21diskmemload10:42:263
31047991813,3cyclictest6191-21rt-features07:25:243
31047991813,3cyclictest32404-21sh11:43:143
31047991813,3cyclictest23958-21swap08:05:223
31047991813,3cyclictest19606-21sh11:57:233
31047991813,3cyclictest17319-21/usr/sbin/munin12:20:133
31047991813,3cyclictest10064-21/usr/sbin/munin08:50:173
31047991811,5cyclictest3417-21grep07:20:153
31047991811,5cyclictest1738-21expr08:30:113
31047991810,6cyclictest27642-21cat11:15:243
31047991810,6cyclictest13101-21ls07:40:243
3104699188,7cyclictest9220-21cat09:50:172
3104699182,5cyclictest0-21swapper/211:45:222
3104699182,12cyclictest23304-21ssh10:25:122
3104699182,11cyclictest32656-21apt-get09:20:132
3104699182,11cyclictest32656-21apt-get09:20:122
3104699182,11cyclictest25134-21apt-get08:10:132
31046991813,3cyclictest26268-21apt11:15:112
31046991813,3cyclictest15029-21sh11:05:552
31046991812,4cyclictest19840-21sh11:57:382
31046991812,3cyclictest10625-21sed08:50:162
31046991811,5cyclictest24290-21awk08:05:252
31046991811,4cyclictest26724-21sh10:03:282
31046991810,6cyclictest6394-21tr12:35:212
31046991810,6cyclictest4571-21ntp_states08:35:212
31046991810,6cyclictest3036-21wc07:20:132
31046991810,6cyclictest15927-21cat09:00:252
3104599182,11cyclictest27878-21apt-get11:40:131
31045991812,4cyclictest8871-21sh08:45:221
31045991812,4cyclictest1973-21sh10:56:321
31045991811,5cyclictest3240-21seq08:33:081
31045991811,5cyclictest20132-21cut09:10:181
31045991811,5cyclictest18591-21seq09:09:511
31045991811,5cyclictest11603-21seq11:51:081
31045991810,6cyclictest26778-21seq08:12:051
31045991810,5cyclictest31344-21switchtime10:30:221
3104699179,5cyclictest322-21wc08:25:232
3104699172,9cyclictest51162sleep207:25:122
3104699172,10cyclictest7953-21apt-get08:45:132
3104699171,7cyclictest8789-21ssh10:38:082
31046991712,3cyclictest5788-21/usr/sbin/munin08:40:132
31046991711,3cyclictest15338-21timerwakeupswit09:30:252
31045991712,3cyclictest18645-21cstates07:55:131
3104699169,5cyclictest29977-21grep08:20:152
3104699162,8cyclictest272762sleep208:15:122
31046991611,3cyclictest19278-21missed_timers07:55:202
3104699161,11cyclictest12787-21ntp_states07:40:222
31046991610,4cyclictest15981-21latency_hist07:50:002
3104599169,5cyclictest5686-21cat07:25:181
31045991611,3cyclictest3628-21/usr/sbin/munin08:35:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional