You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-12 - 02:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Jul 11, 2026 12:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9449217097,25sleep30-21swapper/307:08:053
95492160107,29sleep10-21swapper/107:09:221
95872153102,28sleep20-21swapper/207:09:522
92652138100,27sleep00-21swapper/007:05:410
2868221030,1sleep20-21swapper/212:20:202
167852740,2sleep20-21swapper/210:35:252
67752720,8sleep122-21ksoftirqd/110:52:141
240682710,4sleep024072-21tail11:05:170
319662680,2chrt0-21swapper/209:35:182
319662680,2chrt0-21swapper/209:35:182
210332660,3sleep10-21swapper/110:15:181
268602640,2chrt0-21swapper/211:55:152
305852550,2sleep00-21swapper/012:21:340
287972540,1sleep10-21swapper/111:08:591
71962530,2sleep122-21ksoftirqd/109:40:261
228492510,1sleep00-21swapper/011:51:470
236792500,2sleep10-21swapper/109:29:391
36912490,1sleep10-21swapper/111:38:041
32032450,2chrt0-21swapper/210:01:382
7412430,1sleep30-21swapper/312:24:103
177512390,8sleep012-21ksoftirqd/010:36:120
178392380,7sleep1221ksoftirqd/109:49:091
81042350,17chrt0-21swapper/209:41:112
987499301,26cyclictest12979-21ntp_states07:15:231
987499301,26cyclictest12979-21ntp_states07:15:231
312682300,2sleep10-21swapper/110:22:311
111722290,3chrt0-21swapper/209:20:132
987399281,25cyclictest0-21swapper/007:16:290
987399281,25cyclictest0-21swapper/007:16:290
139702280,11sleep012-21ksoftirqd/009:45:270
273082270,3sleep334-21ksoftirqd/312:20:003
272682270,2sleep30-21swapper/309:31:293
217352270,3sleep221734-21awk07:35:232
987599261,23cyclictest0-21swapper/207:27:302
9874992615,7cyclictest25039-21apt-get07:45:151
270392260,1sleep20-21swapper/207:50:002
137222260,4sleep10-21swapper/112:09:551
987699251,23cyclictest0-21swapper/307:15:153
987699251,23cyclictest0-21swapper/307:15:143
9875992523,1cyclictest0-21swapper/207:14:592
987599250,23cyclictest0-21swapper/207:15:302
987599250,23cyclictest0-21swapper/207:15:292
9874992523,1cyclictest0-21swapper/107:30:461
253632250,16chrt0-21swapper/209:30:202
987699248,4cyclictest34-21ksoftirqd/310:00:013
9874992421,2cyclictest0-21swapper/107:22:511
9873992422,1cyclictest0-21swapper/007:22:510
9876992315,5cyclictest24578-21sh11:30:003
987599235,11cyclictest12385-21apt-get11:45:032
987599232,15cyclictest22048-21sh10:40:042
9875992316,5cyclictest18992-21ssh09:50:052
9875992314,7cyclictest24417-21python308:55:252
987499232,19cyclictest10569-21dump-pmu-power09:20:011
9874992313,7cyclictest52950irq/141-enp3s0-09:10:141
9873992315,6cyclictest9566-21ntp_states10:30:210
9873992315,6cyclictest23806-21ntpq07:40:190
97332230,10sleep228-21ksoftirqd/212:05:542
9876992214,5cyclictest53050irq/142-enp3s0-11:45:133
9876992213,6cyclictest25392-21awk07:45:183
9876992212,7cyclictest16508-21cat11:00:003
9876992212,6cyclictest1566-21cat10:00:233
987599221,6cyclictest0-21swapper/209:55:152
9875992213,6cyclictest12725-21ps11:20:222
9874992214,6cyclictest15269-21idleruntime08:35:161
9874992214,6cyclictest14118-21open_files11:45:221
104142220,3sleep20-21swapper/212:30:272
987699212,13cyclictest14259-21ssh12:10:133
9876992115,4cyclictest15787-21tune2fs10:35:163
9876992113,6cyclictest6613-21seq09:16:053
9876992113,5cyclictest53050irq/142-enp3s0-10:05:203
9876992113,5cyclictest17312-21basename09:24:593
987699211,18cyclictest17333-21sh09:48:383
9875992115,4cyclictest28-21ksoftirqd/207:24:592
9875992113,6cyclictest26339-21ntp_states09:00:222
9875992113,6cyclictest19147-21awk07:30:192
9875992113,5cyclictest52950irq/141-enp3s0-11:09:332
9875992113,5cyclictest2582-21grep12:25:172
9875992110,7cyclictest52950irq/141-enp3s0-11:40:282
987499219,10cyclictest11252-21gzip10:55:231
987499213,3cyclictest141rcu_preempt09:55:231
987499211,7cyclictest13762-21apt-get10:10:141
9874992114,5cyclictest30076-21ssh11:33:461
9874992113,6cyclictest1365-21pluginstate10:00:221
987399213,3cyclictest29446-21diskmemload12:05:120
987399211,6cyclictest0-21swapper/010:24:360
9873992114,5cyclictest377-21python308:00:230
9873992114,5cyclictest22383-21sed08:50:240
9873992113,6cyclictest31522-21sed08:00:000
9873992112,7cyclictest5199-21diskstats11:15:150
9873992110,7cyclictest52750irq/139-enp3s0-09:30:130
224272210,2sleep20-21swapper/210:15:382
202842210,2sleep10-21swapper/112:38:281
120532210,2sleep30-21swapper/311:44:413
9876992014,4cyclictest3394-21sh09:14:363
9876992014,4cyclictest21663-21processes07:35:233
9876992014,4cyclictest11502-21sh10:31:413
9876992013,5cyclictest20579-21ls11:50:183
9876992013,4cyclictest53050irq/142-enp3s0-10:50:123
9876992012,6cyclictest9457-21cat12:30:183
9876992012,6cyclictest8797-21expr08:20:183
9876992012,6cyclictest6570-21seq11:15:273
9876992012,6cyclictest19736-21ls08:45:203
9876992012,6cyclictest17258-21ntpq07:25:193
9876992012,6cyclictest1596-21seq09:36:283
9876992012,6cyclictest1596-21seq09:36:283
9876992012,6cyclictest14084-21ls10:10:143
9876992012,6cyclictest11960-21cut07:15:003
9876992011,6cyclictest24631-21date08:55:263
9876992010,7cyclictest22688-21tune2fs10:40:153
9876992010,6cyclictest53050irq/142-enp3s0-10:15:143
987599203,13cyclictest0-21swapper/211:15:142
987599202,12cyclictest7044-21ssh10:05:132
987599201,8cyclictest32561-21ssh11:11:002
987599201,7cyclictest3035-21ssh09:14:162
9875992013,5cyclictest21462-21entropy08:50:142
9875992012,6cyclictest23647-21ls07:40:182
9875992011,7cyclictest18689-21sed09:25:202
9875992011,7cyclictest15103-21sed12:10:192
987499209,7cyclictest52850irq/140-enp3s0-11:10:131
987499202,12cyclictest20636-21apt-get07:35:131
987499201,6cyclictest15265-21apt-get10:35:121
9874992013,5cyclictest560-21dbus-daemon08:20:001
9874992013,5cyclictest24349-21unixbench_singl07:40:251
9874992012,6cyclictest27846-21perl07:50:161
9874992012,6cyclictest2124-21cat12:25:131
9874992011,7cyclictest20158-21awk08:45:261
987399208,3cyclictest12-21ksoftirqd/009:57:080
987399205,4cyclictest66050lldpd08:27:120
987399203,2cyclictest29446-21diskmemload11:46:040
987399201,5cyclictest0-21swapper/012:36:450
987399201,5cyclictest0-21swapper/012:13:380
987399201,5cyclictest0-21swapper/009:35:300
987399201,5cyclictest0-21swapper/009:35:300
987399201,5cyclictest0-21swapper/009:13:250
9873992012,6cyclictest7097-21awk10:05:140
9873992012,5cyclictest8505-21sed12:30:010
9873992011,7cyclictest32328-21tail11:35:160
9873992010,7cyclictest14685-21apt-get08:35:110
9873992010,6cyclictest52750irq/139-enp3s0-09:20:120
987399200,5cyclictest0-21swapper/009:28:170
88372200,9sleep012-21ksoftirqd/010:54:170
215422200,1sleep30-21swapper/311:03:273
180182200,2sleep20-21swapper/212:36:132
9876991917,1cyclictest0-21swapper/307:22:123
9876991914,3cyclictest16774-21sh11:23:563
9876991913,4cyclictest6024-21sh10:27:593
9876991913,4cyclictest6024-21sh10:27:593
9876991913,4cyclictest4879-21sh12:26:323
9876991913,4cyclictest3867-21sh11:38:153
9876991913,4cyclictest30399-21sh11:34:063
9876991913,4cyclictest23756-21sh09:53:173
9876991913,4cyclictest19544-21sh09:25:303
9876991913,3cyclictest636-21runrttasks09:00:123
9876991912,5cyclictest5449-21sh09:40:043
9876991911,6cyclictest32542-21seq11:59:473
9876991911,6cyclictest29615-21date09:10:013
9876991911,6cyclictest29236-21ssh11:09:263
9876991911,6cyclictest1962-21which08:05:163
9876991911,6cyclictest17038-21ls08:40:133
9876991910,6cyclictest22682-21seq08:51:013
9876991910,6cyclictest17250-21/usr/sbin/munin12:35:263
9875991914,3cyclictest10272-21cpuspeed_turbos08:25:132
9875991912,5cyclictest9237-21cat08:20:242
9875991912,5cyclictest1665-21expr08:05:132
9875991912,5cyclictest12062-21seq10:55:452
9875991912,4cyclictest27369-21/usr/sbin/munin10:20:142
9875991911,6cyclictest14060-21seq10:34:152
9875991910,7cyclictest26486-21seq12:19:122
9875991910,6cyclictest9728-21date10:55:012
9875991910,5cyclictest52950irq/141-enp3s0-11:30:132
987599190,17cyclictest20035-21cron10:15:002
987499192,15cyclictest6512-21fschecks_count11:40:171
987499192,15cyclictest5372-21grep11:15:171
987499191,3cyclictest141rcu_preempt10:45:141
987499191,3cyclictest141rcu_preempt09:20:111
9874991913,4cyclictest8697-21interrupts08:20:161
9874991913,4cyclictest1812-21sh12:00:211
9874991912,5cyclictest9712-21date12:30:201
9874991912,5cyclictest24763-21awk08:55:261
9874991912,5cyclictest20235-21open_inodes09:50:221
9874991912,5cyclictest1612-21cat08:05:131
9874991912,5cyclictest14149-21seq08:31:301
987499191,16cyclictest8511-21apt-get10:30:131
9874991911,6cyclictest3823-21cat08:10:141
9874991911,6cyclictest27794-21expr09:05:131
9874991911,6cyclictest22351-21cat08:50:231
9874991911,6cyclictest22066-21seq10:40:051
9874991911,6cyclictest11699-21seq07:13:551
9874991910,6cyclictest27877-21grep11:55:251
9874991910,6cyclictest17291-21fw_packets11:00:161
987399194,3cyclictest29446-21diskmemload10:49:060
987399193,3cyclictest29446-21diskmemload11:12:400
987399193,12cyclictest13780-21sh10:10:140
987399193,11cyclictest5678-21apt-get08:15:130
987399192,4cyclictest141rcu_preempt10:40:030
987399191,6cyclictest0-21swapper/010:26:020
987399191,6cyclictest0-21swapper/010:26:020
987399191,5cyclictest0-21swapper/011:42:500
987399191,5cyclictest0-21swapper/011:00:180
9873991914,4cyclictest12-21ksoftirqd/009:44:590
987399191,3cyclictest29446-21diskmemload11:57:390
987399191,3cyclictest141rcu_preempt10:16:540
9873991913,4cyclictest2678-21sed08:05:250
9873991912,5cyclictest7068-21cron12:04:590
9873991911,6cyclictest17659-21sed08:40:180
987399191,16cyclictest10600-21ssh10:55:170
9873991911,6cyclictest10228-21sed12:30:260
987399190,6cyclictest0-21swapper/009:15:360
987399190,4cyclictest22645-21ssh12:15:260
987699181,5cyclictest10537-21diskstats08:25:153
9876991813,3cyclictest2750-21sh12:00:383
9876991812,4cyclictest29446-21diskmemload10:20:163
9876991812,4cyclictest2034-21sh10:49:133
9876991811,5cyclictest4370-21needreboot08:10:203
9876991811,5cyclictest30473-21awk07:55:233
9876991811,5cyclictest19104-21if_enp3s007:30:193
987699181,13cyclictest27343-21taskset07:50:023
9876991810,6cyclictest34-21ksoftirqd/311:10:363
9876991810,6cyclictest34-21ksoftirqd/308:30:203
9876991810,6cyclictest31952-21expr08:00:133
9876991810,6cyclictest24887-21grep07:45:013
9876991810,6cyclictest14852-21cat08:35:123
987599189,7cyclictest560-21dbus-daemon08:35:012
987599182,11cyclictest3542-21apt-get08:10:132
9875991813,3cyclictest2576-21latency10:25:192
9875991813,3cyclictest2576-21latency10:25:192
9875991812,4cyclictest333-21sh11:35:212
987599181,13cyclictest52950irq/141-enp3s0-11:00:122
987599181,12cyclictest28947-21apt-get10:45:122
987599181,12cyclictest18276-21apt-get11:25:132
9875991810,6cyclictest4540-21seq12:02:242
9875991810,6cyclictest30552-21cut07:55:222
9875991810,6cyclictest28698-21wc07:50:252
9875991810,6cyclictest28-21ksoftirqd/211:50:022
987499189,6cyclictest52850irq/140-enp3s0-10:25:131
987499189,6cyclictest52850irq/140-enp3s0-10:25:121
987499183,9cyclictest16843-21apt-get08:40:121
987499183,4cyclictest22-21ksoftirqd/108:26:051
987499183,3cyclictest141rcu_preempt09:35:221
987499183,3cyclictest141rcu_preempt09:35:211
987499182,12cyclictest8474-21grep10:05:261
987499181,8cyclictest29962-21ssh09:34:101
9874991813,3cyclictest32251-21hwlatdetect08:00:161
9874991813,3cyclictest15584-21python312:10:241
9874991812,3cyclictest24083-21sh11:29:311
9874991811,5cyclictest22368-21switchtime12:15:241
9874991811,4cyclictest13494-21sh11:20:391
9874991810,6cyclictest28738-21ntpq12:20:201
9874991810,6cyclictest26604-21cat09:00:241
987399189,7cyclictest28192-21sed07:50:190
987399181,8cyclictest32691-21ssh10:00:120
987399181,6cyclictest0-21swapper/009:51:010
9873991813,3cyclictest19799-21timerwakeupswit11:25:250
9873991812,4cyclictest18959-21/usr/sbin/munin08:45:230
9873991811,5cyclictest21379-21sed07:35:190
9873991811,5cyclictest17695-21sed07:25:240
9873991810,6cyclictest11534-21awk11:20:110
987399180,5cyclictest0-21swapper/011:30:340
243382180,3sleep10-21swapper/111:53:181
105472180,4sleep3331rcuc/312:06:443
987599179,6cyclictest32439-21cut08:00:182
987599177,7cyclictest7896-21cat08:20:002
987599173,8cyclictest18984-21apt-get08:45:142
987599172,10cyclictest12401-21apt-get09:45:152
9875991711,4cyclictest28-21ksoftirqd/208:40:002
9875991710,5cyclictest5335-21open_inodes09:15:232
9875991710,5cyclictest28138-21basename09:05:172
987499179,5cyclictest29711-21expr07:55:151
9874991710,5cyclictest16545-21cat07:25:121
987399179,6cyclictest4474-21awk08:10:220
9873991714,2cyclictest0-21swapper/007:10:000
9873991712,3cyclictest12517-21/usr/sbin/munin08:30:120
9873991710,5cyclictest8709-21cut08:20:180
9873991710,5cyclictest25519-21grep07:45:190
9873991710,5cyclictest23954-21latency08:55:190
9875991611,3cyclictest17607-21missed_timers08:40:182
9873991611,3cyclictest636-21runrttasks09:06:100
9873991611,3cyclictest18473-21/usr/sbin/munin07:30:120
987699158,5cyclictest6894-21sed08:15:243
987399158,5cyclictest25911-21grep09:00:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional