You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-02 - 06:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Thu Jul 02, 2026 00:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1745217197,25sleep20-21swapper/219:08:042
1642217097,25sleep10-21swapper/119:06:421
18452169147,15sleep30-21swapper/319:09:223
1723216996,25sleep00-21swapper/019:07:470
279042710,3sleep20-21swapper/221:14:512
34542630,2sleep30-21swapper/321:43:413
171762590,2sleep00-21swapper/022:39:450
262862580,3chrt0-21swapper/123:55:191
297892570,2sleep30-21swapper/323:11:103
176292570,2sleep00-21swapper/023:25:230
160672570,2sleep30-21swapper/323:01:183
20822560,7sleep012-21ksoftirqd/023:38:040
70482530,2sleep30-21swapper/321:45:303
173612520,2sleep00-21swapper/021:53:290
58802460,1sleep00-21swapper/000:04:170
37632460,4chrt28-21ksoftirqd/222:30:032
173742430,1sleep20-21swapper/223:25:222
226092360,2chrt0-21swapper/321:56:503
105652350,2chrt0-21swapper/222:11:022
308022330,2chrt0-21swapper/123:12:081
55562320,2sleep00-21swapper/020:30:240
300262310,4chrt21226-21diskmemload21:15:250
12252310,2sleep10-21swapper/121:41:311
69172280,5sleep334-21ksoftirqd/319:20:123
232382270,3sleep0131rcuc/021:10:240
148072270,1sleep10-21swapper/100:10:191
76032260,2sleep10-21swapper/119:20:181
50562260,2chrt0-21swapper/223:40:152
216899261,23cyclictest0-21swapper/219:10:252
216699261,24cyclictest0-21swapper/019:11:110
152572250,3sleep10-21swapper/123:24:491
298012240,2sleep00-21swapper/022:48:280
225232240,1sleep20-21swapper/223:52:432
2169992422,1cyclictest0-21swapper/319:11:103
2167992421,2cyclictest0-21swapper/119:11:071
2166992417,5cyclictest18278-21df_inode22:40:130
118712240,2sleep00-21swapper/021:25:330
2168992314,7cyclictest8947-21df_inode20:40:142
216699239,10cyclictest21746-21sendmail-msp22:20:000
2166992312,9cyclictest23185-21sshd23:53:200
2169992216,4cyclictest34-21ksoftirqd/322:30:003
2169992216,4cyclictest34-21ksoftirqd/322:29:593
2169992213,7cyclictest12898-21apt-get20:50:113
2168992212,7cyclictest25117-21proc_pri21:35:232
2168992212,6cyclictest27207-21sed22:00:152
2167992212,7cyclictest278-21systemd-journal21:29:591
2167992212,6cyclictest9872-21grep22:10:241
216799220,11cyclictest149472sh21:51:121
216699223,12cyclictest28072-21sh00:20:120
2166992213,7cyclictest11875-21diskstats23:45:140
2166992211,3cyclictest18872-21sh00:13:280
86612210,2sleep30-21swapper/321:24:203
315972210,2sleep30-21swapper/320:15:253
22052210,18sleep121226-21diskmemload21:19:451
216999216,3cyclictest79482sh22:32:363
2169992114,5cyclictest32099-40mandb00:00:003
2169992113,6cyclictest20406-21ls19:50:193
2169992112,7cyclictest32757-21memory22:50:203
2169992112,7cyclictest2672-21cut21:20:003
2169992110,8cyclictest6380-21apt-get22:55:113
216899218,11cyclictest21757-21sed23:05:232
216899216,5cyclictest141rcu_preempt21:55:152
216899216,3cyclictest141rcu_preempt20:40:012
216899213,13cyclictest1186-21apt-get22:05:112
216899212,15cyclictest32552-21ssh00:00:122
216899212,15cyclictest31859-21ssh22:50:132
2168992113,6cyclictest23013-21sed22:20:182
2168992111,7cyclictest29140-21ntp_states00:20:212
2168992111,7cyclictest14054-21gltestperf23:00:182
2168992111,7cyclictest10977-21/usr/sbin/munin19:30:212
216899211,16cyclictest2025-21ssh23:15:112
2168992111,6cyclictest15570-21sed20:55:162
2168992111,6cyclictest15570-21sed20:55:152
2168992110,9cyclictest13972-21sh00:33:022
2168992110,7cyclictest52850irq/140-enp3s0-22:40:112
216899210,7cyclictest30987-21sh23:35:202
216799213,10cyclictest10902-21ssh22:35:101
2167992112,7cyclictest560-21dbus-daemon19:20:001
2167992112,7cyclictest10291-21seq20:42:271
216799211,17cyclictest15194-21ssh22:15:121
2167992110,3cyclictest81902sh22:32:501
2166992115,4cyclictest12-21ksoftirqd/000:40:000
216699211,4cyclictest0-21swapper/021:02:590
2166992113,6cyclictest7509-21pluginstate22:55:210
2166992113,6cyclictest28223-21grep22:00:230
2166992113,6cyclictest26926-21ntp_states20:05:200
2166992113,6cyclictest21023-21awk21:55:250
2166992112,7cyclictest21386-21tune2fs00:15:140
201862210,2sleep0131rcuc/021:05:180
176572210,2sleep20-21swapper/222:16:062
88832200,2chrt0-21swapper/000:29:490
87872200,3sleep20-21swapper/219:25:012
216999209,7cyclictest52950irq/141-enp3s0-23:15:113
216999202,13cyclictest32449-21apt-get20:20:123
2169992014,3cyclictest61212sleep323:40:233
2169992013,5cyclictest19248-21sed23:50:163
2169992013,4cyclictest10811-21cpuspeed_turbos20:45:133
2169992012,6cyclictest3290-21cat00:25:173
2169992012,6cyclictest29081-21ntp_kernel_err00:20:203
2169992012,6cyclictest11449-21ls21:25:243
216999201,17cyclictest5239-21ntp_states20:30:213
2169992011,7cyclictest21844-21date22:20:013
216999201,11cyclictest12760-21df21:50:143
2169992010,7cyclictest25245-21df_inode22:45:143
2169992010,6cyclictest52950irq/141-enp3s0-00:30:123
216899203,13cyclictest12177-21gdbus21:50:042
216899202,14cyclictest30815-21ssh21:40:112
216899202,14cyclictest11504-21ssh23:45:112
2168992016,3cyclictest24911-21cron23:59:542
2168992014,4cyclictest32174-21sh20:20:002
2168992012,6cyclictest9792-21sed21:25:022
2168992011,5cyclictest0-21swapper/220:00:122
216799202,14cyclictest22173-21ssh22:20:121
216799202,13cyclictest15947-21sh23:25:031
2167992014,4cyclictest3404-21sh22:05:521
2167992013,5cyclictest29362-21sh22:48:021
2167992013,5cyclictest1727-21grep00:00:241
2167992012,6cyclictest5558-21ntpq23:40:191
2167992012,6cyclictest31214-21seq22:03:091
2167992012,6cyclictest30626-21diskstats20:15:161
2167992012,6cyclictest17787-21entropy19:45:161
2167992012,6cyclictest17782-21ls21:00:161
2167992012,6cyclictest16233-21cut00:35:011
2167992012,6cyclictest12592-21seq20:49:331
2167992012,6cyclictest12311-21sed23:45:181
2167992010,7cyclictest640-21snmpd00:26:041
216699202,13cyclictest20366-21ssh23:05:130
2166992012,6cyclictest7406-21seq21:45:520
2166992012,6cyclictest560-21dbus-daemon00:00:000
2166992012,6cyclictest28588-21perl20:10:160
2166992012,6cyclictest28588-21perl20:10:150
2166992011,7cyclictest918-21ntpq20:20:190
2166992011,6cyclictest560-21dbus-daemon20:40:000
2166992011,4cyclictest12-21ksoftirqd/022:55:000
2166992010,6cyclictest3921-21apt-get22:30:110
165402200,2sleep30-21swapper/320:55:263
165402200,2sleep30-21swapper/320:55:253
241022190,5sleep124119-21ssh00:16:511
216999198,7cyclictest52950irq/141-enp3s0-23:30:123
216999197,3cyclictest5632-21timerandwakeup19:15:253
216999192,13cyclictest18154-21ssh21:30:213
216999191,7cyclictest21226-21diskmemload21:39:293
2169991913,4cyclictest2944-21vmstat22:05:253
2169991913,4cyclictest22685-21sh00:39:443
2169991913,4cyclictest21538-21hwlatdetect00:15:153
2169991913,4cyclictest13328-21sh23:22:593
2169991912,5cyclictest19645-21expr21:05:123
2169991912,4cyclictest21226-21diskmemload00:00:113
2169991912,4cyclictest0-21swapper/323:35:143
2169991911,6cyclictest9174-21which20:40:173
2169991911,6cyclictest9115-21cat19:25:143
2169991911,6cyclictest27114-21readlink20:05:223
216999191,10cyclictest17816-21users23:25:243
2169991910,7cyclictest24661-21cat22:45:003
2169991910,7cyclictest13612-21fw_packets19:35:153
2169991910,6cyclictest17479-21systemctl22:40:013
216899199,7cyclictest24161-21open_inodes23:30:202
216899199,3cyclictest15500-21apt-key23:25:002
216899198,7cyclictest52850irq/140-enp3s0-00:25:132
216899198,6cyclictest5244-21ssh21:45:132
216899193,11cyclictest20904-21apt-get00:15:122
2168991912,4cyclictest7602-21ssh00:05:202
2168991911,5cyclictest27281-21sed20:05:232
216899191,14cyclictest27262-21apt-get23:10:112
216899191,10cyclictest23614-21dump-pmu-power20:00:002
2168991910,6cyclictest278-21systemd-journal22:35:072
216799199,4cyclictest3004-21sh23:38:571
216799197,7cyclictest53050irq/142-enp3s0-23:50:101
216799191,8cyclictest18593-21rm00:35:491
2167991914,3cyclictest27679-21sh23:33:111
2167991913,4cyclictest8884-21sh21:47:161
2167991913,4cyclictest25325-21sh21:12:191
2167991913,4cyclictest22028-21unixbench_singl23:05:261
2167991912,5cyclictest32714-21sed22:50:201
2167991911,6cyclictest7216-21memory20:35:191
2167991911,6cyclictest3021-21cat21:20:111
2167991911,6cyclictest30188-21ntp_kernel_pll_22:25:201
2167991911,6cyclictest30188-21ntp_kernel_pll_22:25:191
2167991911,6cyclictest26241-21expr20:05:121
2167991911,6cyclictest18016-21cat21:30:201
2167991911,6cyclictest14427-21netstat23:00:211
2167991911,6cyclictest11075-21cat19:30:111
2167991910,7cyclictest21630-21cut19:55:011
2167991910,7cyclictest15018-21cat20:55:111
2167991910,7cyclictest15018-21cat20:55:101
216699199,6cyclictest52750irq/139-enp3s0-23:40:100
216699195,2cyclictest141rcu_preempt22:10:480
216699193,11cyclictest8618-21apt-get20:40:130
216699192,12cyclictest2920-21apt-get21:20:110
216699191,5cyclictest29602-21munin-run23:35:000
2166991914,3cyclictest32107-21sendmail_mailtr21:40:220
2166991913,4cyclictest4872-21sh22:07:160
2166991913,4cyclictest30325-21sh21:40:000
2166991912,5cyclictest18895-21unixbench_singl19:45:250
2166991912,5cyclictest1683-21seq22:28:290
2166991912,5cyclictest1683-21seq22:28:290
2166991911,6cyclictest4884-21sed19:15:170
2166991911,6cyclictest17620-21fw_packets21:30:160
2166991911,6cyclictest13397-21find23:00:130
216699191,16cyclictest10381-21sh00:30:170
2166991910,7cyclictest21880-21cat19:55:120
216999189,6cyclictest13043-21timerwakeupswit23:45:243
216999188,7cyclictest19266-21kworker/u8:122:10:103
216999182,11cyclictest11002-21apt-get19:30:113
216999181,8cyclictest8225-21ntp_states00:05:253
216999181,8cyclictest24609-21ssh22:21:053
2169991813,3cyclictest24637-21missed_timers20:00:203
2169991813,3cyclictest24326-21sh23:07:353
2169991813,3cyclictest17771-21sh19:45:143
2169991811,5cyclictest8336-21cron20:40:003
2169991811,5cyclictest29147-21grep20:10:213
2169991811,5cyclictest29147-21grep20:10:203
2169991811,5cyclictest18428-21ls21:00:243
216999181,12cyclictest14040-21ssh00:10:143
2169991810,6cyclictest929-21cat22:05:003
2169991810,6cyclictest22293-21cat21:10:163
2169991810,6cyclictest15316-21grep19:40:133
216899189,7cyclictest17892-21cut19:45:162
216899181,7cyclictest18149-21/usr/sbin/munin21:30:202
216899181,6cyclictest22979-21latency_hist00:40:002
2168991813,3cyclictest3751-21chrt20:26:082
2168991812,4cyclictest32413-21sh22:26:532
2168991812,4cyclictest32413-21sh22:26:522
2168991811,5cyclictest8843-21cat19:25:112
2168991811,5cyclictest5789-21grep20:30:242
2168991811,4cyclictest21226-21diskmemload22:47:222
2168991810,6cyclictest3988-21ntpq21:20:182
2168991810,6cyclictest19144-21grep21:05:012
2168991810,6cyclictest14810-21grep20:55:002
2168991810,5cyclictest6406-21ssh22:55:112
216799188,8cyclictest24120-21cpuspeed_turbos21:35:141
216799181,9cyclictest0-21swapper/122:40:191
216799181,8cyclictest6455-21grep22:55:121
2167991813,3cyclictest7028-21sh23:18:381
2167991813,3cyclictest3273-21sendmail_mailst20:25:231
2167991811,5cyclictest21226-21diskmemload00:05:161
2167991811,5cyclictest20068-21basename21:55:161
2167991811,5cyclictest16259-21pluginstate19:40:211
2167991810,6cyclictest4690-21tune2fs20:30:141
2167991810,6cyclictest28301-21grep20:10:131
2167991810,6cyclictest28301-21grep20:10:121
2167991810,6cyclictest28220-21tr00:20:141
2167991810,6cyclictest21201-21seq21:10:001
216699189,7cyclictest11642-21ntpq20:45:190
216699182,1cyclictest141rcu_preempt19:55:000
2166991813,3cyclictest2968-21missed_timers20:25:180
2166991813,3cyclictest10915-21sh23:20:390
2166991813,3cyclictest10743-21sh19:30:010
216699181,2cyclictest141rcu_preempt23:11:280
2166991812,4cyclictest21226-21diskmemload22:20:350
2166991812,4cyclictest12898-21sh00:09:220
2166991811,5cyclictest11607-21cat19:30:160
216699181,14cyclictest21226-21diskmemload23:16:450
2166991810,6cyclictest30363-21cat20:15:130
2166991810,6cyclictest25036-21python320:00:230
2166991810,6cyclictest15408-21grep19:40:130
2166991810,6cyclictest14820-21seq19:37:260
149602180,1sleep20-21swapper/200:10:202
216999172,11cyclictest21717-21apt-get19:55:123
2169991710,5cyclictest2469-21expr20:25:133
216899179,5cyclictest20563-21head19:50:212
216899172,11cyclictest19331-21apt-get21:05:112
2168991712,3cyclictest15280-21/usr/sbin/munin19:40:232
2168991711,4cyclictest28125-21/usr/sbin/munin20:10:192
2168991711,4cyclictest28125-21/usr/sbin/munin20:10:192
2168991710,5cyclictest28476-21/usr/sbin/munin21:15:202
216799179,6cyclictest22954-21ls19:55:231
2167991711,4cyclictest9102-21expr19:25:141
2167991710,5cyclictest13303-21cat19:35:141
216699177,7cyclictest15036-21apt-get20:55:120
216699177,7cyclictest15036-21apt-get20:55:120
2166991712,3cyclictest6698-21/usr/sbin/munin19:20:220
2166991710,5cyclictest13865-21cut20:50:220
11292170,2sleep10-21swapper/120:20:201
216899169,5cyclictest5421-21processes19:15:222
216899168,6cyclictest14307-21python319:35:232
216899161,11cyclictest32452-21chrt20:20:092
216799169,5cyclictest24108-21expr20:00:141
2167991611,3cyclictest13318-21fschecks_time20:50:161
216899157,6cyclictest11095-21entropy20:45:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional