You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 20:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Aug 02, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30840210161,35sleep00-21swapper/007:07:230
3091528560,9sleep30-21swapper/307:08:233
3094328370,9sleep20-21swapper/207:08:422
3082628063,13sleep10-21swapper/107:07:121
3112999260,25cyclictest0-21swapper/010:24:390
3113999250,25cyclictest0-21swapper/312:25:173
3113699250,24cyclictest0-21swapper/210:27:372
3112999252,22cyclictest0-21swapper/011:47:380
31129992423,1cyclictest0-21swapper/011:09:140
3112999240,23cyclictest0-21swapper/011:00:510
3113499225,16cyclictest29201-21ssh10:10:191
31134992220,1cyclictest12056-21aten2_r7power_c08:45:121
31139992118,2cyclictest39-21ksoftirqd/311:40:003
31139992118,2cyclictest39-21ksoftirqd/311:40:003
3113499210,20cyclictest23717-21diskmemload09:27:151
31129992120,1cyclictest0-21swapper/010:56:270
31134992018,1cyclictest17719-21awk12:00:001
3113499200,13cyclictest0-21swapper/110:35:181
3112999202,13cyclictest0-21swapper/007:30:130
220012200,0sleep30-21swapper/310:05:453
220012200,0sleep30-21swapper/310:05:453
31139991915,1cyclictest39-21ksoftirqd/309:21:023
31136991916,1cyclictest0-21swapper/209:40:432
3113699191,5cyclictest0-21swapper/208:40:132
3113499192,14cyclictest28350-21kworker/1:111:10:141
3113499191,18cyclictest0-21swapper/111:09:001
3113499190,19cyclictest0-21swapper/111:23:591
31129991917,1cyclictest0-21swapper/012:03:470
15692190,0sleep00-21swapper/011:30:210
31139991812,1cyclictest39-21ksoftirqd/312:34:563
3113699180,18cyclictest0-21swapper/211:09:002
3113499181,16cyclictest8790-21ssh09:39:351
3112999185,6cyclictest25064-21idleruntime-cro09:30:000
31129991816,1cyclictest0-21swapper/010:50:210
31139991711,5cyclictest11168-21apache_processe10:00:133
31136991715,2cyclictest32-21ksoftirqd/212:17:562
3113699170,15cyclictest0-21swapper/210:03:252
3113499170,2cyclictest0-21swapper/109:24:231
3113499170,17cyclictest0-21swapper/110:54:401
3113499170,15cyclictest0-21swapper/111:33:301
3112999170,0cyclictest0-21swapper/009:12:290
3113999163,2cyclictest141rcu_preempt11:01:353
31139991614,1cyclictest39-21ksoftirqd/309:44:353
31139991613,2cyclictest39-21ksoftirqd/309:50:243
31139991613,2cyclictest0-21swapper/310:50:213
31139991611,2cyclictest39-21ksoftirqd/312:05:033
3113999160,1cyclictest0-21swapper/311:46:383
3113999160,0cyclictest141rcu_preempt10:16:593
3113699160,2cyclictest0-21swapper/210:44:002
3113699160,16cyclictest0-21swapper/211:33:302
3113699160,13cyclictest0-21swapper/212:07:542
31134991614,2cyclictest38150irq/131-snd_hda10:44:541
3112999168,7cyclictest27243-21munin-run12:25:000
31129991612,3cyclictest0-21swapper/011:44:390
31129991612,3cyclictest0-21swapper/011:44:390
3112999160,0cyclictest0-21swapper/009:34:460
3113999158,5cyclictest21682-21apache209:45:233
3113999157,7cyclictest23717-21diskmemload09:32:213
31139991512,2cyclictest39-21ksoftirqd/311:15:243
31139991512,2cyclictest24182-21/usr/sbin/munin09:10:183
3113999150,0cyclictest141rcu_preempt12:16:093
3113699157,7cyclictest21736-21apache208:55:192
3113699150,14cyclictest0-21swapper/212:25:182
3112999153,6cyclictest21342-21munin-run10:45:000
3112999151,0cyclictest0-21swapper/010:34:310
3112999150,2cyclictest0-21swapper/011:50:110
3112999150,1cyclictest0-21swapper/010:47:260
31139991413,1cyclictest39-21ksoftirqd/310:46:263
3113999141,12cyclictest12936-21cron10:40:003
31139991411,2cyclictest0-21swapper/311:12:273
31139991410,3cyclictest11273-21wc11:55:233
3113999140,11cyclictest24202-21/usr/sbin/munin11:25:133
3113699140,4cyclictest0-21swapper/209:20:532
3113699140,10cyclictest10059-21ssh11:35:232
3113699140,10cyclictest10059-21ssh11:35:232
3113699140,0cyclictest0-21swapper/209:15:412
31134991410,3cyclictest15115-21head08:50:171
31134991410,2cyclictest3429-21cron07:20:001
3113499140,9cyclictest4309-1kworker/u9:009:55:141
3113499140,12cyclictest0-21swapper/112:03:041
3113499140,12cyclictest0-21swapper/111:00:201
31129991410,3cyclictest5385-21tr10:35:150
31129991410,3cyclictest14119-21cron08:50:000
3112999140,2cyclictest0-21swapper/009:52:390
3112999140,13cyclictest23716-21turbostat.cron09:10:000
3112999140,13cyclictest2203-21cut09:35:160
3113999139,3cyclictest7504-21cut08:35:163
3113999139,3cyclictest7504-21cut08:35:163
3113999139,3cyclictest39-21ksoftirqd/311:42:183
3113999139,3cyclictest39-21ksoftirqd/311:42:183
3113999139,3cyclictest14263-21wc07:40:183
3113999138,4cyclictest13763-21tr10:40:143
3113999135,4cyclictest225-21systemd-journal12:25:003
3113999132,3cyclictest141rcu_preempt11:05:193
31139991310,2cyclictest4640-21wc09:55:273
31139991310,2cyclictest4332-21wc12:10:273
31139991310,2cyclictest4020-21wc08:25:283
31139991310,2cyclictest32542-21rm10:12:263
31139991310,2cyclictest13978-21grep12:35:243
31139991310,2cyclictest12655-21tr10:20:163
3113999130,8cyclictest0-21swapper/307:25:133
3113999130,3cyclictest23717-21diskmemload10:27:203
3113999130,12cyclictest8032-21dump-pmu-power09:20:003
3113999130,11cyclictest0-21swapper/309:39:303
3113699139,3cyclictest30035-21head10:30:172
31136991310,2cyclictest2914-21grep12:10:162
31136991310,2cyclictest29089-21find08:10:222
31136991310,2cyclictest25461-21grep11:25:202
31136991310,2cyclictest17638-21grep11:20:262
31136991310,2cyclictest13189-21cat12:35:172
3113699130,1cyclictest0-21swapper/212:03:042
3113699130,13cyclictest0-21swapper/209:58:442
3113499139,3cyclictest901-21cut08:20:191
3113499138,4cyclictest38150irq/131-snd_hda10:05:121
3113499138,4cyclictest38150irq/131-snd_hda10:05:121
3113499134,7cyclictest27727-21grep09:50:211
31134991310,2cyclictest32472-21cut07:10:211
31134991310,2cyclictest26139-21ssh09:30:161
31134991310,2cyclictest19968-21ssh12:40:001
3113499130,12cyclictest22501-21taskset09:47:011
3112999138,4cyclictest32613-21cut09:15:120
3112999138,3cyclictest11977-21cat10:20:010
3112999132,6cyclictest16388-21apt-get09:25:000
3112999131,2cyclictest0-21swapper/008:55:150
31129991310,2cyclictest583-21tail07:10:270
31129991310,2cyclictest23755-21grep08:00:190
31129991310,2cyclictest21868-21wc10:25:200
31129991310,2cyclictest13849-21rm12:35:240
31129991310,2cyclictest12096-21wc07:35:220
3113999129,2cyclictest8215-21grep10:55:273
3113999129,2cyclictest5637-21grep08:30:183
3113999129,2cyclictest27045-21tail08:05:323
3113999129,2cyclictest231792sleep312:02:423
3113999129,2cyclictest22569-21http08:00:003
3113999129,2cyclictest10868-21tr08:40:223
3113999128,4cyclictest39-21ksoftirqd/311:30:593
3113999128,3cyclictest19674-21grep07:50:283
3113999128,2cyclictest39-21ksoftirqd/308:00:013
3113699129,2cyclictest32249-21cat09:15:012
3113699129,2cyclictest26357-21grep11:45:172
3113699129,2cyclictest2306-21cat09:55:002
3113699129,2cyclictest22158-21grep10:45:142
3113699129,2cyclictest22144-21df09:05:172
3113699129,2cyclictest20194-21grep12:20:182
3113699129,2cyclictest1797-21tr09:35:142
3113699129,2cyclictest17852-21sort09:25:162
3113699129,2cyclictest16368-21head07:45:172
3113699129,2cyclictest10649-21grep11:55:182
3113699129,2cyclictest1032-21cron07:15:012
3113699126,2cyclictest0-21swapper/210:15:172
3113699123,6cyclictest5697-21cron10:55:002
31136991210,1cyclictest17991-21sed11:40:182
31136991210,1cyclictest17991-21sed11:40:182
3113499129,2cyclictest7095-21tail10:55:181
3113499129,2cyclictest5159-21grep08:30:151
3113499129,2cyclictest26890-21wc08:05:321
3113499129,2cyclictest23717-21diskmemload09:18:301
3113499129,2cyclictest22980-21cut10:45:191
3113499129,2cyclictest19058-21grep11:40:281
3113499129,2cyclictest19058-21grep11:40:281
3113499129,2cyclictest18942-21cat07:50:201
3113499129,2cyclictest10440-21grep11:35:271
3113499129,2cyclictest10440-21grep11:35:271
3113499128,3cyclictest9356-21cron08:40:011
3113499128,3cyclictest9356-21cron08:40:001
3113499127,4cyclictest8495-21cut07:30:131
3113499127,4cyclictest24761-21cat09:10:161
3113499127,2cyclictest13042-21ssh11:18:181
3113499123,9cyclictest0-21swapper/110:24:571
3113499121,6cyclictest23717-21diskmemload12:05:151
3113499120,10cyclictest0-21swapper/110:17:171
3113499120,0cyclictest0-21swapper/112:25:181
3112999129,2cyclictest6772-21cut07:25:160
3112999129,2cyclictest3095-21grep09:55:150
3112999129,2cyclictest27885-21head12:05:200
3112999129,2cyclictest1951-21cut07:15:170
3112999129,2cyclictest14428-21grep08:50:140
3112999129,2cyclictest11458-21tr10:00:150
3112999128,3cyclictest29961-21grep10:10:260
3112999128,2cyclictest16561-21tail11:20:170
3112999122,9cyclictest31831-21munin-node08:43:320
31129991211,1cyclictest0-21swapper/011:16:510
3112999120,1cyclictest0-21swapper/011:38:280
3112999120,1cyclictest0-21swapper/011:38:280
3112999120,12cyclictest0-21swapper/012:15:400
3113999118,2cyclictest3406-21grep11:50:293
3113999118,2cyclictest21929-21ssh09:27:343
3113999118,2cyclictest19890-21grep09:00:173
3113999118,2cyclictest17389-21cut08:55:183
3113999117,3cyclictest10592-21sleep07:35:003
3113999117,2cyclictest39-21ksoftirqd/308:50:253
3113999116,2cyclictest225-21systemd-journal07:25:003
3113999114,2cyclictest21543-21apt-get09:05:153
3113699119,1cyclictest5589-21grep08:30:182
3113699119,1cyclictest22038-21grep07:55:292
3113699119,1cyclictest19231-21tr09:00:142
3113699118,3cyclictest8672-21ntpq11:15:222
3113699118,2cyclictest9565-21cat07:30:192
3113699118,2cyclictest7683-21grep10:55:222
3113699118,2cyclictest5802-21rm11:52:162
3113699118,2cyclictest4649-21ls08:30:002
3113699118,2cyclictest31460-21grep11:10:142
3113699118,2cyclictest28051-21tr10:10:142
3113699118,2cyclictest26950-21wc08:05:322
3113699118,2cyclictest20240-21grep10:05:162
3113699118,2cyclictest20240-21grep10:05:162
3113699118,2cyclictest12124-21tr08:45:122
3113699118,2cyclictest10969-21basename07:35:152
3113699113,4cyclictest225-21systemd-journal10:40:002
3113499119,1cyclictest235352sleep110:26:021
3113499118,2cyclictest4647-21wc08:30:001
3113499118,2cyclictest29524-21cat10:30:151
3113499118,2cyclictest2902-21grep12:10:161
3113499118,2cyclictest28179-21expr08:10:141
3113499118,2cyclictest27918-21ssh11:26:461
3113499118,2cyclictest25783-21ssh12:23:521
3113499118,2cyclictest23696-21cron09:10:001
3113499118,2cyclictest18560-21chrt08:56:061
3113499118,2cyclictest15251-21ssh10:02:031
3113499118,2cyclictest13269-21ssh12:15:421
3113499118,2cyclictest12598-21chrt07:35:371
3113499118,1cyclictest20722-21expr07:55:181
3113499117,3cyclictest3665-21/usr/sbin/munin12:30:261
3113499117,3cyclictest2498-21cat11:50:211
3113499111,8cyclictest9723-21ssh09:40:071
3112999118,2cyclictest8995-21awk08:35:280
3112999118,2cyclictest8995-21awk08:35:280
3112999118,2cyclictest7109-21ssh12:12:260
3112999118,2cyclictest6870-21taskset08:33:370
3112999118,2cyclictest32350-21sh11:10:180
3112999118,2cyclictest31478-21wc08:15:220
3112999118,2cyclictest29092-21seq08:10:220
3112999118,2cyclictest28690-21chrt12:25:190
3112999118,2cyclictest25234-21ssh10:08:160
3112999118,2cyclictest25234-21ssh10:08:160
3112999118,2cyclictest239762sleep011:25:040
3112999118,2cyclictest21029-21taskset09:02:040
3112999118,2cyclictest17645-21chrt11:59:580
3112999117,3cyclictest13268-21cut07:40:110
3112999110,9cyclictest22732-21latency_hist08:00:000
3112999110,1cyclictest0-21swapper/009:45:180
3113999108,1cyclictest319582sleep308:16:143
3113999108,1cyclictest31002sleep307:15:513
3113999108,1cyclictest22172sleep308:24:513
3113999108,1cyclictest16200-21expr11:20:153
3113999108,1cyclictest1235-21ntpd07:45:203
3113999108,1cyclictest12328-21cat07:35:253
3113999107,2cyclictest707-21taskset07:10:513
3113999107,2cyclictest31168-21sh10:30:273
3113999107,2cyclictest12727-21head08:45:153
3113999100,9cyclictest28700-21cut08:10:183
3113699108,2cyclictest3131-21seq07:16:162
3113699108,1cyclictest4522-21chrt07:20:192
3113699108,1cyclictest24573-21expr08:00:312
3113699108,1cyclictest18057-21grep07:50:162
3113699108,1cyclictest14529-21wget08:50:142
3113699108,1cyclictest14329-21sh11:00:012
3113699108,1cyclictest13191-21taskset07:40:002
3113699107,2cyclictest6643-21df07:25:152
3113699107,2cyclictest26958-21ls09:30:222
3113699104,5cyclictest19102-21fschecks_count09:45:172
3113699101,6cyclictest537-21snmpd12:30:262
31136991010,0cyclictest0-21swapper/210:21:422
3113499109,1cyclictest0-21swapper/107:20:151
3113499108,1cyclictest7855-21chrt07:25:391
3113499108,1cyclictest30984-21cat08:15:181
3113499108,1cyclictest13301-21wget07:40:111
3113499107,2cyclictest25819-21cut11:45:151
3113499107,2cyclictest17841-21cat07:50:001
3113499107,2cyclictest10403-21cut08:40:171
3113499100,9cyclictest20578-21expr09:00:261
3112999108,1cyclictest5100-21rm12:30:220
3112999108,1cyclictest16442-21basename07:45:180
3112999107,2cyclictest9881-21ssh09:40:130
3112999107,2cyclictest4429-21grep07:20:180
3112999107,2cyclictest19896-21taskset07:52:410
3112999107,2cyclictest1149-21cut08:20:210
3112999106,3cyclictest25656-21tr08:05:200
311369997,1cyclictest8843-21cat08:35:272
311369997,1cyclictest8843-21cat08:35:272
311369997,1cyclictest32564-21taskset08:20:152
311369996,2cyclictest30283-21wget08:15:142
311349996,2cyclictest23313-21tr08:00:171
311299995,3cyclictest3231-21grep08:25:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional