You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 17:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun May 03, 2026 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
155852167116,44sleep20-21swapper/207:06:552
156732142106,25sleep30-21swapper/307:08:053
154872141105,25sleep10-21swapper/107:05:391
156652139101,26sleep00-21swapper/007:07:580
1738721240,2sleep30-21swapper/310:32:253
73542780,2sleep00-21swapper/010:49:440
26532640,2sleep10-21swapper/112:21:551
310972600,1sleep131096-21ssh11:06:351
115292570,3sleep011526-21ssh11:40:210
170052540,4sleep00-21swapper/010:32:030
166642520,2sleep00-21swapper/012:08:380
248732500,1sleep20-21swapper/209:25:222
219872500,1sleep30-21swapper/309:47:233
217722500,1sleep10-21swapper/109:23:251
82212490,2sleep00-21swapper/009:37:010
167752490,1chrt16776-21/usr/sbin/munin07:10:173
52322480,2sleep20-21swapper/211:11:252
39562480,2sleep20-21swapper/209:58:252
1609999430,22cyclictest0-21swapper/309:20:113
206642390,2chrt0-21swapper/308:35:043
16098993412,6cyclictest28310-21awk11:05:122
16098993411,6cyclictest1781-21date10:20:242
186572330,2chrt0-21swapper/211:45:242
1609899339,6cyclictest25992-21sed12:40:002
1609899331,29cyclictest52950irq/141-enp3s0-10:10:112
1609999322,14cyclictest23649-21/usr/sbin/munin07:25:233
1609899329,5cyclictest21346-21grep10:35:202
1609899328,4cyclictest2159-21wc07:50:242
89772310,2chrt0-21swapper/011:15:000
1609999315,9cyclictest14579-21rm11:42:443
1609899319,5cyclictest3312-21sed09:09:592
1609899319,5cyclictest10853-21sed11:15:242
1609899319,3cyclictest6558-21sort12:25:152
1609899317,5cyclictest20267-21apt-key08:35:012
1609899317,21cyclictest0-21swapper/211:55:462
1609899313,10cyclictest14821-21pmu-power10:30:222
1609899312,9cyclictest3252-21diskmemload11:00:132
1609899312,10cyclictest27104-21ssh10:40:132
1609899312,10cyclictest19055-21irqstats09:45:162
1609899311,9cyclictest15929-21ssh11:20:122
162942300,1sleep30-21swapper/312:32:083
1609999306,10cyclictest32441-21rm11:31:493
1609999304,10cyclictest4812-21latency_hist12:00:003
1609899308,5cyclictest26949-31updatedb.mlocat07:35:112
1609899307,4cyclictest32328-21cron10:20:002
1609899305,22cyclictest0-21swapper/210:45:152
1609899303,6cyclictest30053-21apt-get09:30:042
1609899303,6cyclictest17179-21tr09:20:132
1609899302,7cyclictest31848-21apt-get12:20:102
1609899302,6cyclictest26217-21ntp_states12:15:212
1609899301,9cyclictest4909-21/usr/sbin/munin09:35:112
1609899301,9cyclictest13991-21pmu-power12:30:202
1609899301,9cyclictest13807-21grep10:05:222
1609899301,8cyclictest4408-21grep11:35:162
16098993011,3cyclictest3252-21diskmemload11:42:112
16098993011,3cyclictest27571-21grep08:50:162
16098993010,3cyclictest25075-21cpuspeed_turbos08:45:132
16097993010,10cyclictest22-21ksoftirqd/111:39:371
1609999299,10cyclictest14033-21rm11:18:323
1609999295,11cyclictest20567-21rm11:46:573
1609999294,8cyclictest25709-21rm09:25:373
1609999294,10cyclictest7661-21sh09:12:423
1609999292,17cyclictest11198-21ssh12:28:473
1609899299,4cyclictest3252-21diskmemload09:10:282
1609899298,4cyclictest9357-21mailstats10:50:232
1609899298,4cyclictest0-21swapper/208:59:592
1609899296,6cyclictest30208-21memory11:30:182
1609899296,6cyclictest13787-21cron08:20:002
1609899292,8cyclictest20886-21fschecks_time07:20:162
1609899292,8cyclictest14850-21ssh10:55:122
1609899292,6cyclictest6357-21python312:00:222
1609899291,9cyclictest7086-21chrt10:25:132
1609899291,9cyclictest12342-21tail12:05:152
16098992911,3cyclictest25199-21cstates09:50:132
1609899290,9cyclictest17532-21sed07:10:252
1609799299,10cyclictest22-21ksoftirqd/112:38:161
16097992911,4cyclictest22-21ksoftirqd/110:07:371
1609799291,11cyclictest211rcuc/110:45:131
16097992910,4cyclictest22-21ksoftirqd/110:33:071
7152280,2chrt0-21swapper/011:56:100
1609999288,17cyclictest24280-21unixbench_multi08:40:243
1609999288,11cyclictest30163-21ssh11:05:393
1609999285,9cyclictest26656-21sh11:27:443
1609999285,9cyclictest26656-21sh11:27:433
1609999283,9cyclictest32365-21cat09:00:173
1609899288,4cyclictest26615-21perf07:35:002
1609899287,4cyclictest31893-21cat07:45:172
1609899285,4cyclictest0-21swapper/208:35:252
1609899281,9cyclictest3252-21diskmemload09:15:112
1609899281,9cyclictest25573-21taskset11:26:382
1609899281,9cyclictest25573-21taskset11:26:372
1609899281,8cyclictest24364-21ssh11:50:132
1609899281,7cyclictest3252-21diskmemload10:00:172
1609899281,7cyclictest18664-21cstates12:10:112
16098992810,4cyclictest3252-21diskmemload09:40:562
1609799289,3cyclictest141rcu_preempt12:01:241
1609799288,11cyclictest22-21ksoftirqd/110:41:491
1609799287,12cyclictest22-21ksoftirqd/111:58:131
1609799287,12cyclictest22-21ksoftirqd/109:38:461
16097992813,9cyclictest141rcu_preempt11:03:391
16097992811,4cyclictest22-21ksoftirqd/109:18:231
16097992810,5cyclictest22-21ksoftirqd/112:31:101
40032270,2sleep30-21swapper/310:22:213
1609999274,9cyclictest3252-21diskmemload11:03:213
1609999273,8cyclictest7742-21ssh11:37:363
1609999272,8cyclictest3252-21diskmemload09:42:463
1609999272,5cyclictest0-21swapper/309:39:103
1609999272,11cyclictest20627-21sh11:23:253
1609999271,9cyclictest0-21swapper/310:06:073
16097992716,4cyclictest22-21ksoftirqd/112:16:211
16097992711,4cyclictest22-21ksoftirqd/112:26:431
16097992711,4cyclictest22-21ksoftirqd/109:57:321
1609999264,8cyclictest32145-21rm10:19:503
1609999263,9cyclictest29174-21sh09:52:533
1609999263,9cyclictest25001-21apt-get08:45:143
1609999263,8cyclictest10718-21ssh10:27:263
1609999262,8cyclictest9167-21ssh10:01:533
1609999262,10cyclictest11550-21rm12:05:023
1609999261,16cyclictest19914-21ssh10:10:143
1609899268,4cyclictest6617-21mailstats08:00:232
1609899266,4cyclictest24012-21unixbench_multi07:25:262
1609799269,9cyclictest22-21ksoftirqd/110:25:121
1609799269,9cyclictest22-21ksoftirqd/109:44:491
1609799268,5cyclictest22-21ksoftirqd/110:10:281
1609799268,4cyclictest22-21ksoftirqd/110:16:251
1609799267,4cyclictest22-21ksoftirqd/111:51:351
1609799263,11cyclictest3252-21diskmemload11:45:161
16097992611,4cyclictest22-21ksoftirqd/109:31:451
16097992610,6cyclictest490-21rpcbind10:01:021
16096992624,1cyclictest0-21swapper/007:14:090
16096992618,6cyclictest18934-21irqstats07:15:180
16096992617,7cyclictest27286-21df07:35:130
1609999254,9cyclictest4500-21sh10:46:513
1609999254,7cyclictest6027-21rm11:12:123
1609999253,9cyclictest23651-21rm12:13:553
1609999253,8cyclictest2321-21ssh09:32:473
1609999253,8cyclictest20301-21ssh12:35:183
1609999253,18cyclictest5574-21grep12:00:163
1609999252,10cyclictest24974-21ssh10:38:203
1609999251,11cyclictest27290-21ssh12:15:503
1609899258,3cyclictest18268-21latency_hist08:30:012
1609899258,3cyclictest18268-21latency_hist08:30:012
16098992510,3cyclictest636-21runrttasks08:23:452
1609799258,3cyclictest22-21ksoftirqd/109:51:251
1609799255,7cyclictest22-21ksoftirqd/112:07:091
1609799255,7cyclictest22-21ksoftirqd/108:10:001
1609799255,11cyclictest22-21ksoftirqd/109:12:431
16097992518,5cyclictest22-21ksoftirqd/110:40:001
16097992518,5cyclictest22-21ksoftirqd/109:00:001
1609799251,1cyclictest0-21swapper/107:14:071
16097992510,4cyclictest22-21ksoftirqd/110:24:291
16097992510,2cyclictest141rcu_preempt12:11:451
1609699259,6cyclictest32197-21cat12:20:130
1609999242,8cyclictest3128-21taskset09:08:383
1609999242,8cyclictest27988-21ssh11:52:433
1609999242,13cyclictest5050-21taskset07:58:043
16099992414,7cyclictest28805-21users10:40:263
1609999241,14cyclictest1348-21ssh09:55:473
16097992418,4cyclictest22-21ksoftirqd/108:40:001
16097992417,5cyclictest22-21ksoftirqd/108:00:011
16097992410,3cyclictest141rcu_preempt11:30:131
1609999238,12cyclictest12987-21/usr/sbin/munin08:15:223
1609999235,15cyclictest2441-21head07:50:243
1609999232,8cyclictest15790-21taskset08:23:073
1609999231,9cyclictest3252-21diskmemload09:15:493
16099992312,8cyclictest16288-21cat08:25:153
16099992312,8cyclictest16288-21cat08:25:143
1609899231,8cyclictest18285-21apt-get07:15:122
1609799239,3cyclictest141rcu_preempt11:11:581
1609799238,4cyclictest22-21ksoftirqd/111:40:401
1609799238,4cyclictest22-21ksoftirqd/109:46:351
1609799236,4cyclictest22-21ksoftirqd/110:50:071
16097992310,4cyclictest22-21ksoftirqd/111:29:571
16097992310,4cyclictest22-21ksoftirqd/111:29:561
16097992310,2cyclictest141rcu_preempt11:22:491
1609799230,6cyclictest23233-21irqstats07:25:191
16096992312,8cyclictest4546-21memory09:10:170
16096992310,9cyclictest14773-21apt-get10:55:130
1609999228,11cyclictest8230-21/usr/sbin/munin10:50:123
1609999228,11cyclictest5662-21cstates08:00:133
1609999227,12cyclictest26238-21/usr/sbin/munin07:30:273
16099992215,5cyclictest3451-21sh12:22:433
16099992214,5cyclictest10107-21entropy08:10:143
16099992213,7cyclictest14981-21seq10:55:143
16099992213,6cyclictest31724-21fw_packets07:45:163
16099992213,4cyclictest18285-21apt-get07:15:133
16099992210,9cyclictest28638-21awk08:50:253
1609799229,2cyclictest141rcu_preempt09:29:051
1609799228,3cyclictest141rcu_preempt11:17:031
1609799226,9cyclictest141rcu_preempt07:30:221
1609799221,8cyclictest22-21ksoftirqd/110:55:591
16097992216,4cyclictest22-21ksoftirqd/108:55:011
16097992215,4cyclictest28835-21cron07:40:001
16096992214,6cyclictest10999-21ls08:10:230
16096992212,5cyclictest3252-21diskmemload09:25:140
223192210,2sleep00-21swapper/012:12:350
1609999218,10cyclictest8250-21/usr/sbin/munin08:05:173
16099992112,6cyclictest21596-21sensors_temp07:20:223
16097992115,4cyclictest22-21ksoftirqd/107:55:001
16097992114,5cyclictest22-21ksoftirqd/108:14:591
16097992114,5cyclictest22-21ksoftirqd/108:05:001
1609699216,5cyclictest1264-21missed_timers10:20:190
16096992116,4cyclictest12-21ksoftirqd/010:05:000
16096992114,4cyclictest30102-21needreboot08:55:190
16096992113,6cyclictest9487-21seq08:09:520
16096992113,6cyclictest1227-21date09:05:000
16096992112,6cyclictest22837-21sensors_temp11:00:230
16096992111,6cyclictest52750irq/139-enp3s0-11:35:200
16099992016,3cyclictest331rcuc/307:43:283
1609799209,3cyclictest22-21ksoftirqd/108:25:181
1609799209,3cyclictest22-21ksoftirqd/108:25:171
1609799204,10cyclictest141rcu_preempt07:20:201
1609799202,2cyclictest141rcu_preempt08:40:161
16097992015,4cyclictest22-21ksoftirqd/109:05:001
16097992014,5cyclictest22-21ksoftirqd/107:45:001
16097992014,4cyclictest22-21ksoftirqd/107:50:011
16097992013,5cyclictest22-21ksoftirqd/109:10:011
16097992012,6cyclictest19405-21seq07:15:241
16097992012,6cyclictest19066-21wc08:30:181
16097992010,2cyclictest141rcu_preempt08:24:591
16096992014,4cyclictest27847-21kernelversion10:40:190
16096992012,6cyclictest22933-21cat08:40:120
16096992012,6cyclictest12319-21cat09:40:160
256642190,2sleep00-21swapper/011:26:440
256642190,2sleep00-21swapper/011:26:440
1609999195,11cyclictest27042-21cpuspeed_turbos07:35:123
1609799191,5cyclictest0-21swapper/108:45:211
16097991913,4cyclictest22-21ksoftirqd/108:20:001
1609699192,7cyclictest25525-21/usr/sbin/munin12:15:140
16096991914,3cyclictest23107-21sh10:36:270
16096991914,3cyclictest16454-21sh10:07:300
16096991913,4cyclictest21186-21sh10:10:310
16096991913,4cyclictest1797-21sh11:33:350
16096991912,5cyclictest30478-21seq09:54:140
16096991912,5cyclictest22242-21seq07:24:070
16096991912,5cyclictest21414-21seq09:46:480
16096991911,6cyclictest7423-21cat08:05:010
16096991911,6cyclictest5633-21date10:00:010
16096991911,6cyclictest4087-21memory07:55:190
16096991911,6cyclictest278-21systemd-journal07:45:010
1609699191,15cyclictest17927-21hddtemp_smartct11:45:170
16096991910,7cyclictest26501-21awk08:45:270
16096991910,7cyclictest18116-21ntp_states09:20:200
129052190,2sleep00-21swapper/010:53:330
1609999188,8cyclictest29956-21/usr/sbin/munin08:55:183
16099991810,6cyclictest18550-21expr08:30:123
16098991811,5cyclictest9939-21expr08:10:132
16098991811,5cyclictest7946-21cat08:05:132
1609699189,6cyclictest15387-21timerwakeupswit08:20:250
1609699182,14cyclictest1906-21grep09:05:170
1609699182,12cyclictest19584-21apt-get12:35:140
16096991813,3cyclictest31779-21open_inodes09:30:220
16096991813,3cyclictest28276-21sh10:15:550
16096991813,3cyclictest23344-21missed_timers07:25:190
16096991813,3cyclictest13634-21latency12:30:170
16096991813,3cyclictest10291-21sh11:15:210
16096991812,4cyclictest9616-21sh12:03:190
16096991811,5cyclictest7099-21grep10:25:150
16096991811,4cyclictest3252-21diskmemload11:52:590
16096991811,4cyclictest3252-21diskmemload09:16:220
16096991810,6cyclictest16868-21sed11:20:190
16096991810,6cyclictest10567-21seq12:28:080
1609899177,9cyclictest31010-21idleruntime-cro07:45:012
1609899171,3cyclictest23538-21cut08:40:182
1609699179,6cyclictest27863-21cat08:50:170
1609699173,8cyclictest28248-21ssh11:05:130
1609699172,10cyclictest976-21apt-get07:50:130
16096991713,2cyclictest19854-21users08:30:240
16096991712,3cyclictest25055-21df_inode07:30:160
16096991710,5cyclictest31581-21expr07:45:130
16096991710,5cyclictest17678-21awk08:25:280
16096991710,5cyclictest17678-21awk08:25:270
1609899169,5cyclictest32274-21cat09:00:172
1609699169,5cyclictest22606-21readlink08:40:010
1609699168,6cyclictest560-21dbus-daemon08:19:590
16098991510,3cyclictest3304-21/usr/sbin/munin07:55:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional