You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-03 - 23:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Aug 03, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
247182179102,25sleep10-21swapper/107:05:321
247922141104,27sleep30-21swapper/307:06:313
24828213899,26sleep20-21swapper/207:06:592
248262138102,25sleep00-21swapper/007:06:560
2537621360,2sleep20-21swapper/211:20:142
1717521240,2sleep20-21swapper/211:38:212
246692730,2sleep20-21swapper/211:44:112
246692730,2sleep20-21swapper/211:44:102
284872640,1sleep00-21swapper/010:34:210
22702630,2sleep10-21swapper/111:50:481
268022600,1sleep00-21swapper/011:20:330
308442580,9sleep334-21ksoftirqd/312:12:333
299522550,1sleep30-21swapper/309:46:163
164072530,1sleep20-21swapper/212:25:282
164072530,1sleep20-21swapper/212:25:272
162722510,2sleep216262-21ssh10:25:132
236482500,1sleep123649-21sh10:05:331
167862470,3sleep20-21swapper/212:01:482
308772450,1sleep30-21swapper/310:11:053
108162450,4sleep1211rcuc/109:32:261
25337993213,4cyclictest22-21ksoftirqd/109:49:281
142082320,1sleep00-21swapper/011:35:250
271972300,1sleep20-21swapper/212:34:432
2533799298,10cyclictest22-21ksoftirqd/111:05:171
25337992915,8cyclictest141rcu_preempt10:23:021
25337992911,4cyclictest22-21ksoftirqd/110:49:071
53362280,3sleep012-21ksoftirqd/007:35:570
25337992812,5cyclictest22-21ksoftirqd/112:26:231
25337992812,5cyclictest22-21ksoftirqd/112:26:221
2533799279,5cyclictest22-21ksoftirqd/111:47:541
2533799272,13cyclictest22-21ksoftirqd/112:12:471
25337992711,4cyclictest22-21ksoftirqd/110:59:501
25337992711,4cyclictest22-21ksoftirqd/110:59:491
188272270,2sleep30-21swapper/312:03:533
25339992616,6cyclictest11600-21sshd11:10:143
2533799262,16cyclictest20363-21rm12:05:131
2533799262,12cyclictest22-21ksoftirqd/109:23:091
2533799261,6cyclictest338-21apt-get09:25:121
25337992614,10cyclictest22-21ksoftirqd/107:49:591
25337992613,4cyclictest22-21ksoftirqd/109:50:221
2533799261,24cyclictest0-21swapper/107:13:321
25337992611,4cyclictest22-21ksoftirqd/111:34:041
25337992610,5cyclictest22-21ksoftirqd/111:42:021
25337992610,5cyclictest22-21ksoftirqd/111:42:021
2533699261,24cyclictest0-21swapper/007:13:360
117332260,2sleep00-21swapper/009:33:220
64572250,2sleep30-21swapper/310:17:223
25339992523,1cyclictest0-21swapper/307:14:463
2533899251,1cyclictest0-21swapper/207:13:152
2533799258,6cyclictest22-21ksoftirqd/110:30:401
2533799258,10cyclictest22-21ksoftirqd/109:40:101
25337992519,4cyclictest22-21ksoftirqd/112:00:011
25337992515,6cyclictest12089-21diskmemload12:35:141
25337992510,4cyclictest22-21ksoftirqd/110:37:001
183512250,1sleep30-21swapper/309:14:343
25338992416,5cyclictest22833-21sendmail08:20:012
25338992412,7cyclictest52850irq/140-enp3s0-12:20:122
2533899241,10cyclictest324032awk09:25:002
2533799248,4cyclictest22-21ksoftirqd/110:25:051
2533799248,3cyclictest141rcu_preempt09:55:191
2533799247,9cyclictest22-21ksoftirqd/111:28:491
25337992418,4cyclictest22-21ksoftirqd/111:15:001
25337992412,8cyclictest1290-21apt-get12:15:141
2533799241,12cyclictest22-21ksoftirqd/111:15:351
284752230,2sleep30-21swapper/311:46:163
278682231,19sleep228069-21gltestperf07:15:172
2533999231,10cyclictest320962ssh09:24:423
25338992317,4cyclictest28-21ksoftirqd/211:10:012
25338992310,11cyclictest28-21ksoftirqd/211:05:002
2533799238,9cyclictest22-21ksoftirqd/107:20:131
2533799238,4cyclictest22-21ksoftirqd/112:01:071
2533799237,4cyclictest22-21ksoftirqd/111:20:121
2533799236,4cyclictest22-21ksoftirqd/109:37:291
25337992317,4cyclictest22-21ksoftirqd/112:24:591
25337992317,4cyclictest22-21ksoftirqd/107:39:591
25337992312,4cyclictest22-21ksoftirqd/110:02:441
2533799231,10cyclictest22-21ksoftirqd/111:36:011
25337992310,3cyclictest141rcu_preempt12:30:121
235742230,2sleep00-21swapper/011:19:030
25339992211,5cyclictest32115-21expr07:25:143
2533999221,10cyclictest4453-21/usr/sbin/munin10:40:163
2533899222,9cyclictest258802grep08:25:162
2533799229,7cyclictest141rcu_preempt09:17:291
2533799229,3cyclictest22-21ksoftirqd/109:10:441
2533799229,2cyclictest141rcu_preempt10:51:391
2533799228,4cyclictest22-21ksoftirqd/110:11:161
2533799225,9cyclictest22-21ksoftirqd/108:25:191
25337992216,4cyclictest22-21ksoftirqd/108:35:001
25337992210,3cyclictest141rcu_preempt10:15:461
25336992212,7cyclictest2130-21kworker/u8:211:55:120
117222227,3sleep3141rcu_preempt09:57:053
326352210,2chrt0-21swapper/309:25:023
25339992115,4cyclictest34-21ksoftirqd/311:05:013
25339992115,4cyclictest18546-21sh08:10:003
25339992114,5cyclictest23851-21ls08:20:183
25339992113,6cyclictest28899-21ls07:15:263
25339992112,6cyclictest26846-21date12:10:013
25339992111,7cyclictest4021-21df_inode07:35:143
25339992111,7cyclictest32033-21df_inode11:25:143
25339992110,8cyclictest25458-21apt-get08:25:123
2533899213,6cyclictest278-21systemd-journal12:10:002
2533899212,12cyclictest31827-21apt-get07:25:132
25338992114,4cyclictest1769-21idleruntime-cro12:40:012
25338992113,6cyclictest28063-21fw_packets08:30:162
25338992112,6cyclictest13290-21ntp_states07:55:232
25338992112,6cyclictest13290-21ntp_states07:55:222
2533799219,4cyclictest22-21ksoftirqd/111:01:541
2533799217,2cyclictest141rcu_preempt10:42:281
25337992114,5cyclictest22-21ksoftirqd/108:00:001
25337992114,5cyclictest22-21ksoftirqd/108:00:001
25337992110,2cyclictest141rcu_preempt08:22:561
2533699218,10cyclictest8394-21cpu09:55:130
25336992114,4cyclictest636-21runrttasks09:20:120
25336992113,6cyclictest23861-21memory08:20:180
25336992113,6cyclictest20699-21df12:05:140
25336992112,7cyclictest1820-21fw_packets12:15:160
25336992112,6cyclictest52750irq/139-enp3s0-09:16:140
25336992112,6cyclictest1223-21ntp_states11:50:200
2533699211,18cyclictest10886-21seq11:09:440
173342210,1sleep20-21swapper/209:13:342
79872200,5sleep00-21swapper/009:55:010
2533999201,5cyclictest20992-21/usr/sbin/munin11:40:273
2533999201,5cyclictest20992-21/usr/sbin/munin11:40:263
25339992014,4cyclictest17098-21acpi10:50:113
25339992014,3cyclictest20838-21sh12:30:003
25339992014,3cyclictest20838-21sh12:30:003
25339992012,6cyclictest7788-21seq11:30:363
25339992012,5cyclictest12089-21diskmemload12:23:303
25339992011,7cyclictest21794-21tune2fs12:30:153
25339992011,6cyclictest19901-21run-parts08:10:223
2533899209,9cyclictest2539-21sh11:27:022
2533899209,8cyclictest1948-21df08:45:152
2533899209,7cyclictest52850irq/140-enp3s0-11:30:102
2533899209,7cyclictest29798-21cat10:35:172
2533899209,7cyclictest18360-21kworker/u8:210:30:122
2533899202,13cyclictest27217-21sh09:45:042
2533899202,13cyclictest11576-21apt-get11:10:142
2533899201,8cyclictest18742-21ssh11:15:162
25338992014,4cyclictest28-21ksoftirqd/210:05:002
25338992014,4cyclictest13264-21sh09:34:552
25338992012,6cyclictest10446-21ssh09:55:482
2533899201,13cyclictest31904-21taskset08:40:132
25338992010,8cyclictest32584-1kworker/u9:210:20:252
2533799208,2cyclictest141rcu_preempt08:10:001
2533799207,3cyclictest141rcu_preempt08:35:221
2533799204,2cyclictest141rcu_preempt08:50:001
25337992015,4cyclictest22-21ksoftirqd/108:05:001
25337992014,4cyclictest22-21ksoftirqd/109:00:001
25337992014,4cyclictest10957-21missed_timers09:05:211
25337992013,5cyclictest22-21ksoftirqd/107:55:001
25337992012,6cyclictest32290-21fw_packets07:25:161
2533699208,9cyclictest12-21ksoftirqd/008:50:240
2533699208,10cyclictest0-21swapper/010:55:010
2533699208,10cyclictest0-21swapper/010:55:000
2533699207,11cyclictest19535-21expr11:40:140
2533699207,11cyclictest19535-21expr11:40:130
25336992014,4cyclictest30283-21sh11:48:060
25336992014,4cyclictest30202-21gltestperf08:35:140
25336992014,4cyclictest18953-21sh09:38:540
25336992012,6cyclictest9606-21seq09:00:510
25336992012,6cyclictest7862-21cut09:00:010
25336992012,6cyclictest493-21seq10:13:110
25336992011,7cyclictest23464-21awk10:05:240
25336992011,6cyclictest13002-21tune2fs09:10:150
207232200,4sleep2271rcuc/209:15:212
169282200,1sleep00-21swapper/008:05:130
124872200,2sleep30-21swapper/311:35:083
110692200,2sleep30-21swapper/311:09:553
88542190,1sleep00-21swapper/010:43:480
2533999199,7cyclictest21270-21df_inode08:15:143
2533999198,7cyclictest31899-21apt-get08:40:123
2533999198,7cyclictest22877-21apt-get10:30:123
2533999192,11cyclictest23358-21apt-get10:55:043
2533999192,11cyclictest23358-21apt-get10:55:033
25339991914,3cyclictest9297-21threads09:00:243
25339991913,4cyclictest22830-21sh09:17:083
25339991912,5cyclictest7240-21cat11:55:153
25339991912,5cyclictest29711-21seq12:35:293
25339991911,6cyclictest5018-21sed08:50:233
25339991911,6cyclictest361-21expr11:50:123
25339991911,6cyclictest30632-21ntpq10:35:243
25339991911,6cyclictest30292-21if_enp3s007:20:193
25339991911,6cyclictest1863-21cat08:45:163
25339991911,6cyclictest12954-21idleruntime07:55:193
25339991911,6cyclictest12954-21idleruntime07:55:183
25339991911,5cyclictest12089-21diskmemload10:25:533
25339991910,6cyclictest53050irq/142-enp3s0-09:50:123
2533899197,3cyclictest141rcu_preempt10:40:192
25338991914,3cyclictest4629-21open_files07:35:232
25338991914,3cyclictest2816-21proc_pri09:50:212
25338991913,4cyclictest52850irq/140-enp3s0-11:55:412
25338991913,4cyclictest2592-21sendmail_mailtr12:15:232
25338991913,4cyclictest18535-21sh09:38:292
25338991913,3cyclictest29013-21sh11:46:482
2533899191,16cyclictest31582-21idleruntime-cro07:25:002
25338991911,6cyclictest23539-21perl08:20:152
25338991911,6cyclictest17989-21ls10:50:192
2533799197,2cyclictest141rcu_preempt08:40:201
2533799192,12cyclictest18803-21apt-get08:10:131
25337991915,3cyclictest211rcuc/109:00:011
25337991914,3cyclictest211rcuc/108:50:251
25337991910,6cyclictest28303-21sed07:15:201
25336991914,3cyclictest7605-21sh10:18:330
25336991914,3cyclictest30257-21sh10:35:210
25336991914,3cyclictest13945-21sh11:11:040
25336991914,3cyclictest11479-21sh10:20:410
25336991913,4cyclictest2375-21sh11:02:500
25336991913,4cyclictest16801-21hwlatdetect10:25:170
25336991913,4cyclictest15937-21sh12:00:570
25336991912,5cyclictest12079-21awk10:45:250
25336991911,6cyclictest377-21open_inodes08:40:210
25336991911,6cyclictest32140-21cat07:25:150
25336991911,6cyclictest25706-21expr08:25:140
25336991911,6cyclictest2363-21cat08:45:210
25336991911,6cyclictest17123-21apt10:50:110
25336991911,6cyclictest11961-21seq07:53:090
25336991911,5cyclictest5588-21sed11:29:590
25336991910,7cyclictest11895-21seq09:07:500
25336991910,6cyclictest15682-21sensors_temp08:00:240
2533699190,17cyclictest24295-21seq09:42:330
104482190,3sleep30-21swapper/310:20:203
2533999187,8cyclictest9017-21sh07:45:223
25339991813,3cyclictest3277-21sh12:15:423
25339991813,3cyclictest29870-21sh11:23:393
25339991813,3cyclictest19589-21sh10:03:143
25339991813,3cyclictest12718-21sh10:45:583
25339991812,4cyclictest27059-21sh10:08:583
25339991812,4cyclictest18997-21interrupts11:15:173
25339991812,4cyclictest11016-21sh09:32:393
25339991811,5cyclictest6199-21expr08:55:143
25339991811,5cyclictest6022-21cat07:40:103
25339991811,5cyclictest30167-21tune2fs08:35:143
25339991811,5cyclictest15487-21awk09:35:253
25339991810,6cyclictest21136-21basename09:40:173
2533899185,10cyclictest0-21swapper/209:40:192
2533899183,13cyclictest11033-21cut10:45:152
2533899181,4cyclictest0-21swapper/210:06:532
25338991813,3cyclictest5079-21sh10:16:002
25338991813,3cyclictest326-21sh12:14:312
25338991813,3cyclictest307-21/usr/sbin/munin09:25:122
25338991812,4cyclictest8625-21sed09:00:172
25338991812,4cyclictest30332-21sh10:10:322
25338991811,5cyclictest6878-21open_files07:40:192
25338991811,5cyclictest30031-21expr08:35:122
25338991810,5cyclictest6887-21awk08:55:212
25338991810,5cyclictest23833-21ssh10:55:042
25338991810,5cyclictest23833-21ssh10:55:032
25338991810,5cyclictest1940-21cut07:30:162
2533799189,6cyclictest1636-21grep07:30:151
2533799185,3cyclictest141rcu_preempt07:40:091
25337991811,5cyclictest21157-21expr08:15:131
25336991813,3cyclictest7651-21sh11:30:280
25336991813,3cyclictest32154-21sh12:37:590
25336991813,3cyclictest27575-21/usr/sbin/munin08:30:140
25336991813,3cyclictest19240-21fschecks_time08:10:160
25336991812,4cyclictest2769-21sh09:26:040
25336991811,5cyclictest17292-21seq12:26:220
25336991811,5cyclictest17292-21seq12:26:210
2533699181,14cyclictest21653-21diskstats12:30:140
25336991810,6cyclictest28986-21awk09:45:240
25336991810,6cyclictest2862-21ls07:30:260
25336991810,6cyclictest22813-21seq08:19:580
25336991810,6cyclictest15567-21tune2fs10:00:160
25336991810,6cyclictest12777-21grep07:55:180
25336991810,6cyclictest12777-21grep07:55:170
25336991810,6cyclictest10142-21date07:50:010
25336991810,5cyclictest7927-21cat12:20:110
25336991810,5cyclictest28264-21processes12:10:230
2533999179,6cyclictest1789-21expr07:30:143
2533999179,6cyclictest11149-21cat07:50:213
2533999178,6cyclictest15850-21timerwakeupswit08:00:263
25339991710,5cyclictest27815-21cat08:30:143
2533899179,6cyclictest8816-21grep07:45:192
2533899179,6cyclictest4616-21netstat08:50:202
2533899179,5cyclictest10930-21cat07:50:192
25338991712,3cyclictest18959-21cstates08:10:122
25338991711,4cyclictest12099-21idleruntime-cro09:10:002
25338991711,3cyclictest12089-21diskmemload11:50:092
2533699179,6cyclictest6281-21df07:40:120
2533699176,7cyclictest27523-21apt-get07:15:120
25339991610,4cyclictest10862-21kernelversion09:05:203
2533899161,4cyclictest0-21swapper/208:00:182
25338991611,3cyclictest16643-21/usr/sbin/munin08:05:242
2533699167,6cyclictest29824-21cpuspeed_turbos07:20:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional