You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 10:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Apr 09, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1479217897,24sleep20-21swapper/219:09:592
12072151100,44sleep10-21swapper/119:06:321
13312148100,41sleep30-21swapper/319:08:113
14192139102,26sleep00-21swapper/019:09:160
148227456,10sleep00-21swapper/019:10:000
19082690,3sleep01909-21diskmemload21:20:000
131242590,2sleep30-21swapper/323:04:413
56362580,1sleep20-21swapper/221:22:002
210382550,2sleep10-21swapper/121:57:371
213652540,3sleep021364-21ls23:35:000
179192540,2sleep20-21swapper/221:30:542
279352520,2sleep320800-21diskmemload22:26:453
264932520,2sleep30-21swapper/321:37:503
225542510,2sleep10-21swapper/100:23:501
294802500,2sleep30-21swapper/322:52:013
243972490,1sleep30-21swapper/322:24:563
15222470,1sleep20-21swapper/222:07:102
182102420,0sleep1211rcuc/123:08:041
155252390,2sleep10-21swapper/121:53:491
241732370,1chrt0-21swapper/221:35:272
1754993712,7cyclictest12257-21grep22:15:170
235842320,2sleep10-21swapper/122:47:501
175499322,12cyclictest29934-21ssh00:05:120
1754993213,4cyclictest5956-21sh23:22:560
175699307,14cyclictest28-21ksoftirqd/221:47:332
268502280,2sleep20-21swapper/223:38:362
267692280,18sleep10-21swapper/122:50:141
175699287,10cyclictest28-21ksoftirqd/221:42:112
1756992822,4cyclictest28-21ksoftirqd/223:20:012
1756992811,4cyclictest28-21ksoftirqd/200:08:212
1756992810,8cyclictest28-21ksoftirqd/200:02:312
1756992810,8cyclictest28-21ksoftirqd/200:02:302
1756992810,4cyclictest28-21ksoftirqd/223:04:012
296292270,2chrt0-21swapper/022:52:100
266892270,1sleep00-21swapper/022:25:290
175699273,6cyclictest28-21ksoftirqd/223:51:422
1756992711,4cyclictest28-21ksoftirqd/223:58:362
1756992711,4cyclictest28-21ksoftirqd/223:58:352
1756992710,5cyclictest28-21ksoftirqd/221:51:052
1756992710,5cyclictest28-21ksoftirqd/200:15:142
92362260,2sleep00-21swapper/023:25:180
82632260,1sleep10-21swapper/122:36:071
1756992620,3cyclictest28-21ksoftirqd/220:15:002
1756992618,6cyclictest24237-21sh22:24:462
148212260,2sleep30-21swapper/319:40:123
175699259,4cyclictest28-21ksoftirqd/222:42:212
175699257,4cyclictest28-21ksoftirqd/221:27:392
175699257,3cyclictest28-21ksoftirqd/222:32:062
175699252,17cyclictest4116-21apt-get00:10:102
1756992512,4cyclictest28-21ksoftirqd/222:45:172
1756992511,4cyclictest28-21ksoftirqd/223:11:322
1756992510,5cyclictest28-21ksoftirqd/221:57:002
1756992510,3cyclictest141rcu_preempt22:02:292
1755992523,1cyclictest0-21swapper/119:11:351
1754992517,5cyclictest23935-21/usr/sbin/munin22:00:130
161242250,11sleep122-21ksoftirqd/123:30:201
175799240,23cyclictest0-21swapper/319:11:053
175699249,4cyclictest28-21ksoftirqd/200:30:052
175699248,4cyclictest28-21ksoftirqd/222:27:032
175699248,2cyclictest141rcu_preempt23:45:512
175699246,4cyclictest28-21ksoftirqd/223:34:262
175699243,9cyclictest28-21ksoftirqd/222:52:482
1756992422,1cyclictest0-21swapper/219:11:242
1756992411,4cyclictest28-21ksoftirqd/222:18:302
1756992410,5cyclictest28-21ksoftirqd/222:57:282
1755992417,5cyclictest22-21ksoftirqd/121:29:591
1755992414,6cyclictest52850irq/140-enp3s0-21:10:111
1754992412,8cyclictest19506-21python322:20:220
1757992314,7cyclictest12335-21memory22:15:173
175699238,4cyclictest28-21ksoftirqd/200:21:042
175699237,5cyclictest28-21ksoftirqd/223:40:052
175699232,10cyclictest28-21ksoftirqd/222:35:122
1756992316,5cyclictest28-21ksoftirqd/220:10:012
1756992316,5cyclictest28-21ksoftirqd/220:10:002
1756992312,4cyclictest28-21ksoftirqd/223:09:302
1756992310,4cyclictest28-21ksoftirqd/221:13:552
175599232,14cyclictest22393-21ssh21:35:091
175599231,16cyclictest29151-21ssh21:40:041
1754992314,7cyclictest12104-21python300:15:220
1757992213,6cyclictest34-21ksoftirqd/300:00:003
1757992213,6cyclictest34-21ksoftirqd/300:00:003
175699229,3cyclictest141rcu_preempt00:26:592
175699229,2cyclictest141rcu_preempt23:20:552
175699228,8cyclictest28-21ksoftirqd/219:18:422
175699228,8cyclictest28-21ksoftirqd/219:18:412
175699226,5cyclictest28-21ksoftirqd/221:18:332
1756992216,4cyclictest28-21ksoftirqd/219:40:012
175599229,8cyclictest52850irq/140-enp3s0-23:20:041
175599222,14cyclictest23238-21ssh00:00:121
175599222,14cyclictest23238-21ssh00:00:121
1755992213,7cyclictest24102-21seq23:35:481
1755992213,6cyclictest5137-21ntpq00:10:191
1754992214,6cyclictest17279-21fw_packets19:45:140
63482210,1sleep20-21swapper/219:20:122
175799212,12cyclictest15095-21ssh23:30:123
1756992116,4cyclictest28-21ksoftirqd/220:20:002
1756992115,4cyclictest28-21ksoftirqd/220:00:012
1756992112,6cyclictest278-21systemd-journal23:30:012
1756992111,7cyclictest19280-21df_inode19:50:122
1756992111,2cyclictest141rcu_preempt20:20:192
1756992110,3cyclictest141rcu_preempt20:45:442
175599213,15cyclictest4447-21cut20:30:151
175599212,17cyclictest16559-21cron19:45:011
1755992114,4cyclictest22571-40logrotate00:00:011
1755992114,4cyclictest22571-40logrotate00:00:001
1755992113,6cyclictest2940-21seq22:08:361
1755992112,7cyclictest28034-21seq22:02:581
1755992112,7cyclictest25434-21tune2fs22:25:181
1754992113,6cyclictest5791-21cron20:35:000
1754992113,6cyclictest25850-21seq21:13:190
1754992113,6cyclictest23307-21swap23:35:220
1754992113,6cyclictest10778-21perl20:45:130
1754992112,7cyclictest6398-21perl22:35:150
1754992112,6cyclictest19864-21pluginstate21:05:190
175799202,13cyclictest75-21kswapd022:00:113
175799201,8cyclictest17495-21ssh00:20:123
1757992013,4cyclictest20497-21/usr/sbin/munin23:10:133
1757992012,6cyclictest8657-21expr23:25:123
1757992012,6cyclictest32598-21smartctl20:20:153
1757992011,6cyclictest8555-21forks20:40:153
1757992011,6cyclictest278-21systemd-journal00:05:073
1757992010,7cyclictest3028-21cat20:25:243
1757992010,7cyclictest13109-21idleruntime19:35:163
1757992010,6cyclictest8200-21grep23:25:013
1757992010,6cyclictest52950irq/141-enp3s0-21:15:263
175699209,4cyclictest28-21ksoftirqd/200:37:272
175699206,2cyclictest141rcu_preempt22:13:472
1756992011,6cyclictest15282-21if_enp3s019:40:152
1756992010,7cyclictest14942-21df20:55:132
175599201,8cyclictest8414-21apt-get23:25:111
1755992014,4cyclictest22-21ksoftirqd/100:20:001
1755992012,5cyclictest30941-21seq23:40:591
1755992012,5cyclictest278-21systemd-journal22:30:001
1755992011,6cyclictest30697-21ls20:15:181
1755992011,6cyclictest17752-21cron22:20:001
175599200,5cyclictest16575-21ssh22:42:311
175499209,8cyclictest560-21dbus-daemon23:45:220
175499202,7cyclictest23219-21apt-get00:00:110
175499202,7cyclictest23219-21apt-get00:00:110
175499202,13cyclictest3610-21ssh21:45:130
1754992013,5cyclictest9991-21entropy23:50:140
1754992013,5cyclictest13230-21load20:50:170
1754992013,5cyclictest10632-21cstates19:30:110
1754992012,6cyclictest27306-21ls20:10:010
1754992012,6cyclictest27306-21ls20:10:000
1754992012,6cyclictest14950-21df20:55:140
1754992012,6cyclictest1136-21sed23:20:010
1754992012,6cyclictest10436-21cat21:50:100
1754992011,6cyclictest32624-21idleruntime20:20:160
1754992011,6cyclictest32004-21grep22:05:230
175799193,5cyclictest22578-21apt-get00:00:043
175799193,5cyclictest22578-21apt-get00:00:033
175799193,10cyclictest24224-21kworker/u8:022:05:103
175799192,12cyclictest19565-21apt-get22:45:183
175799192,11cyclictest9017-21apt-get21:25:133
175799191,5cyclictest0-21swapper/321:45:183
1757991914,3cyclictest19106-21cstates19:50:123
1757991913,4cyclictest1659-21sh21:43:293
1757991911,6cyclictest25175-21sed20:05:013
1757991911,6cyclictest22750-21cut23:35:173
1757991911,6cyclictest21329-21seq21:34:213
1757991911,6cyclictest11643-21cat20:45:223
1757991911,6cyclictest10529-21awk21:50:133
1757991911,5cyclictest6090-21cat19:20:103
1757991911,5cyclictest17769-21hddtemp_smartct21:55:163
175699195,4cyclictest28-21ksoftirqd/221:03:502
175699192,5cyclictest28-21ksoftirqd/220:35:132
1756991914,4cyclictest28-21ksoftirqd/221:10:002
1756991914,4cyclictest28-21ksoftirqd/220:30:012
1756991913,4cyclictest23820-21hwlatdetect20:00:152
1756991911,6cyclictest9124-21ntpq19:25:192
1756991911,6cyclictest16889-21cat19:45:132
1756991910,3cyclictest141rcu_preempt20:40:122
175599199,8cyclictest16535-21basename21:00:001
175599192,12cyclictest1769-21apt-get20:25:121
1755991914,3cyclictest30821-21wc00:05:181
1755991913,3cyclictest17434-21sh21:30:241
1755991911,6cyclictest8071-21seq00:36:401
1755991911,6cyclictest4011-21ntpq23:45:211
1755991911,6cyclictest21298-21cut23:10:171
1755991911,6cyclictest19671-21memory21:05:181
1755991911,6cyclictest10271-21grep20:45:011
1755991911,5cyclictest28797-21cut21:15:191
1755991911,5cyclictest278-21systemd-journal23:19:591
1755991911,5cyclictest1705-21pmu-power22:55:291
175499197,3cyclictest141rcu_preempt22:55:320
175499192,14cyclictest20951-21grep23:10:140
1754991914,3cyclictest29738-21apt-config20:15:010
1754991914,3cyclictest1549-21sh21:43:230
1754991913,4cyclictest9694-21sh00:38:180
1754991913,4cyclictest7681-21sh22:11:370
1754991913,4cyclictest3910-21sh22:33:280
1754991913,3cyclictest11659-21sh21:26:210
1754991912,5cyclictest22842-21cut21:35:130
1754991912,5cyclictest17585-21uniq21:55:150
1754991911,6cyclictest4686-21ls19:15:180
1754991911,6cyclictest4686-21ls19:15:170
1754991911,6cyclictest29010-21cat23:40:120
1754991911,6cyclictest1841-21cat20:25:140
1754991911,6cyclictest16496-21cat23:55:130
1754991911,6cyclictest16496-21cat23:55:120
111322190,1sleep30-21swapper/300:15:133
175799189,7cyclictest560-21dbus-daemon20:39:593
175799189,7cyclictest560-21dbus-daemon19:20:013
175799189,7cyclictest560-21dbus-daemon19:20:003
175799189,7cyclictest52950irq/141-enp3s0-23:20:003
175799188,6cyclictest52950irq/141-enp3s0-00:10:103
175799182,12cyclictest4763-21apt-get22:10:133
175799182,11cyclictest30824-21apt-get00:30:143
175799182,11cyclictest18913-21apt-get21:05:093
1757991813,3cyclictest4458-21hddtemp_smartct20:30:153
1757991813,3cyclictest3623-21sendmail_mailst21:20:223
1757991813,3cyclictest28909-21sh00:28:303
1757991813,3cyclictest20800-21diskmemload22:30:043
1757991813,3cyclictest16778-21apt-get21:00:113
1757991813,3cyclictest14259-21fschecks_count23:05:153
1757991812,3cyclictest3124-21cstates23:45:133
1757991811,5cyclictest8245-21seq22:36:063
1757991811,5cyclictest6836-21vmstat00:35:253
175799181,15cyclictest31631-21ssh23:41:413
1757991811,5cyclictest26656-21sed20:05:243
1757991811,5cyclictest26656-21sed20:05:233
1757991811,5cyclictest21882-21idleruntime21:10:153
1757991810,6cyclictest31956-21sed20:20:003
1757991810,6cyclictest13873-21cut22:40:203
1757991810,6cyclictest1303-21irqstats22:55:263
1757991810,5cyclictest16804-21basename19:45:103
1756991810,6cyclictest4816-21ntpq20:30:192
175599186,8cyclictest22-21ksoftirqd/120:00:101
175599181,8cyclictest4522-21ssh21:45:191
1755991813,4cyclictest22-21ksoftirqd/120:25:011
1755991813,3cyclictest17777-21proc_pri21:00:211
1755991812,4cyclictest19473-21head19:50:151
1755991811,5cyclictest9695-21seq22:13:371
1755991811,4cyclictest20800-21diskmemload00:27:151
175599181,13cyclictest18749-21tail22:20:141
1755991810,6cyclictest6257-21expr19:20:121
1755991810,6cyclictest11089-21sed19:30:171
175599180,16cyclictest11532-21ssh23:03:041
175599180,15cyclictest26122-21cat20:05:171
175599180,15cyclictest26122-21cat20:05:171
175499189,6cyclictest16268-21grep21:30:140
175499188,7cyclictest19869-21sh19:50:180
1754991813,3cyclictest8951-21sh23:00:280
1754991813,3cyclictest17863-21sh23:07:440
1754991812,4cyclictest2845-21sh00:33:090
1754991812,4cyclictest12571-21gdbus22:40:010
1754991811,5cyclictest5449-21python300:10:210
1754991811,5cyclictest3389-21ntpq21:20:200
1754991811,5cyclictest25839-21unixbench_singl00:25:270
1754991811,5cyclictest21637-21expr19:55:150
1754991811,5cyclictest19919-21df22:45:190
1754991811,5cyclictest16939-21seq21:00:130
1754991811,4cyclictest20800-21diskmemload00:20:050
1754991810,6cyclictest9601-21cat19:25:250
1754991810,6cyclictest12518-21apt-get19:35:120
175799179,6cyclictest21198-21expr19:55:123
175799179,5cyclictest8427-21expr19:25:143
1757991712,3cyclictest14594-21/usr/sbin/munin20:55:153
1757991711,4cyclictest11583-21sh23:50:413
175799171,11cyclictest27527-21apt-get20:10:123
1757991710,5cyclictest11162-21cat19:30:173
1756991713,2cyclictest10377-21apt19:30:092
175599178,7cyclictest4899-21cut19:15:201
175599178,7cyclictest4899-21cut19:15:201
1755991712,3cyclictest8216-21/usr/sbin/munin19:25:131
1755991712,3cyclictest28176-21irqstats20:10:171
1755991711,4cyclictest3517-21sh21:20:221
1755991711,3cyclictest32163-21sh00:30:241
175599171,11cyclictest10342-21apt-get20:45:111
1755991710,5cyclictest9871-21expr23:50:121
1755991710,5cyclictest21989-21ntp_kernel_pll_19:55:191
175499179,6cyclictest16738-21cut19:45:010
175499179,5cyclictest6936-21ntp_kernel_pll_20:35:180
175499172,10cyclictest23251-21apt-get20:00:110
1754991711,4cyclictest28422-21ntp_kernel_pll_20:10:190
1757991611,3cyclictest12453-21/usr/sbin/munin20:50:183
175699169,5cyclictest13767-21sed20:50:232
1755991610,4cyclictest6428-21sort20:35:141
175499168,6cyclictest8977-21cat20:40:180
1754991611,3cyclictest636-21runrttasks19:20:000
175599159,4cyclictest12654-21tr19:35:131
175599157,6cyclictest18627-21seq19:49:151
175599151,9cyclictest13637-21python320:50:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional