You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-20 - 18:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Thu Mar 20, 2025 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
46810993727,7cyclictest181265-21grep11:15:230
46820993127,2cyclictest151960-21sh10:35:003
46810993125,4cyclictest57074-21wireplumber07:37:120
46816992923,4cyclictest57074-21wireplumber07:37:122
46820992724,2cyclictest128698-21sh10:00:003
46812992722,3cyclictest712-21tpm2-abrmd09:40:001
46812992622,3cyclictest165661-21idleruntime-cro10:55:001
4681099267,5cyclictest95835-21sh09:15:010
46810992522,1cyclictest0-21swapper/011:25:010
46816992420,2cyclictest71090-21sendmail08:20:012
46816992420,2cyclictest703-21polkitd11:00:232
46812992420,2cyclictest0-21swapper/110:40:001
46812992412,3cyclictest96718-21ntpq09:15:191
4681299241,10cyclictest205871-21proc_pri11:50:221
4681099247,6cyclictest104574-21sh09:26:370
46810992417,2cyclictest20473-21kworker/0:0+events_freezable07:20:010
46820992319,2cyclictest0-21swapper/311:00:003
46820992317,4cyclictest219296-21/usr/sbin/munin12:10:113
46816992319,3cyclictest1090808-21systemd-journal10:00:002
46816992314,6cyclictest1090808-21systemd-journal11:41:242
4681699231,20cyclictest223653-21ntpq12:15:182
46812992320,2cyclictest132279-21munin-plugin-st10:05:001
46812992319,3cyclictest187593-21sh11:25:011
46812992319,2cyclictest241045-21sh12:40:001
46812992315,6cyclictest213637-21sort12:00:231
4681099233,16cyclictest133191-21sh10:06:330
46810992318,3cyclictest95591-21cron09:15:000
46810992318,3cyclictest0-21swapper/011:40:240
46810992316,4cyclictest188083-21sh11:25:370
46810992311,5cyclictest91335-21diskmemload09:49:280
4682099221,7cyclictest940-21runrttasks10:51:523
4682099221,7cyclictest940-21runrttasks09:44:203
4682099221,7cyclictest214574-21runrttasks12:01:523
46820992217,3cyclictest198695-21cpuspeed_turbos11:40:123
46820992212,6cyclictest124260-21runrttasks09:53:513
46820992212,6cyclictest124260-21runrttasks09:53:513
4681699221,7cyclictest146090-21cron10:25:012
46816992212,7cyclictest191006-21cut11:30:092
46816992211,8cyclictest942-21snmpd12:01:242
4681699220,20cyclictest216452-21sshd12:05:412
4681699220,20cyclictest129792-21ssh10:00:172
4681299221,7cyclictest229412-21ssh12:24:001
46812992212,8cyclictest115545-21ntpq09:40:171
46812992212,8cyclictest0-21swapper/112:15:111
46812992210,5cyclictest69323-21kworker/u16:0+xprtiod11:00:181
4681299220,19cyclictest74356-21ntpq08:25:171
46810992218,2cyclictest213463-21sendmail_mailqu12:00:210
4681099221,5cyclictest69323-21kworker/u16:0+rpciod09:56:470
4682099211,7cyclictest155570-21ssh10:39:433
46820992116,3cyclictest67729-21gltestperf08:10:333
46820992116,3cyclictest138994-21cpuspeed_turbos10:15:113
4682099211,17cyclictest228715-21sh12:21:213
4682099211,17cyclictest134339-21seq10:09:293
4682099211,17cyclictest109529-21sh09:33:113
4682099211,17cyclictest102928-21runrttasks09:23:443
46820992110,7cyclictest178350-21ssh11:12:213
4681699219,9cyclictest47798-21ntpq07:10:182
4681699211,8cyclictest82182-21/usr/sbin/munin08:45:152
46816992118,2cyclictest212190-21sendmail12:00:002
4681699211,7cyclictest221654-21taskset12:13:192
4681699211,7cyclictest207768-21sshd11:53:412
46816992117,2cyclictest198191-21cron11:40:002
4681699211,6cyclictest97713-21sh09:16:002
4681699211,6cyclictest176691-21ssh11:10:172
46816992116,3cyclictest149839-21pmu-power10:31:122
4681699211,17cyclictest940-21runrttasks09:11:572
46816992111,7cyclictest185925-21ssh11:21:392
4681699211,17cyclictest162646-21smartctl10:50:102
4681699210,18cyclictest141393-21ssh10:18:002
4681699210,18cyclictest120498-21seq09:48:162
46812992117,3cyclictest57074-21wireplumber07:37:121
46812992116,3cyclictest77634-21cpuspeed_turbos08:35:431
46812992112,7cyclictest104065-21runrttasks09:25:351
46812992112,6cyclictest161149-21sh10:48:201
4681299211,18cyclictest192552-21taskset11:31:221
4681299211,17cyclictest128588-21sh09:59:481
4681299211,10cyclictest208683-21ssh11:55:121
46810992116,3cyclictest75779-21fschecks_time08:30:170
4681099211,5cyclictest57950-21fschecks_time07:40:160
46810992112,2cyclictest191rcu_preempt12:13:260
46810992111,7cyclictest192463-21taskset11:31:110
46810992111,7cyclictest112258-21taskset09:35:320
4682099209,8cyclictest237185-21apt-get12:35:123
4682099205,13cyclictest216412-21sh12:05:363
4682099201,6cyclictest230418-21sh12:25:473
46820992015,3cyclictest85987-21gltestperf08:55:153
46820992012,5cyclictest105428-21sh09:28:113
4682099201,17cyclictest91335-21diskmemload11:21:053
4682099200,17cyclictest98297-21sh09:16:583
4681699201,7cyclictest189640-21runrttasks11:28:452
46816992016,3cyclictest148861-21kworker/u16:3+xprtiod11:09:002
46816992015,3cyclictest87818-21cpuspeed_turbos09:00:122
46816992015,3cyclictest110939-21cpuspeed_turbos09:35:142
46816992012,5cyclictest114786-21/usr/sbin/munin09:40:232
46816992011,7cyclictest62843-21ntpq07:50:202
4681699201,17cyclictest228942-21ssh12:21:472
4681699201,17cyclictest180203-21grep11:15:122
46816992011,6cyclictest137532-21ssh10:12:552
46816992011,6cyclictest133170-21sshd10:06:312
4681699200,17cyclictest231992-21ssh12:29:162
4681699200,17cyclictest100074-21sh09:20:112
4681299209,8cyclictest178756-21seq11:13:071
4681299209,7cyclictest218655-21date12:10:011
46812992016,2cyclictest156734-21cut10:40:171
46812992015,3cyclictest703-21polkitd08:25:011
46812992015,3cyclictest100205-21cpuspeed_turbos09:20:121
46812992011,7cyclictest195998-21ssh11:35:321
46812992011,6cyclictest134128-21ssh10:08:541
4681299201,15cyclictest120919-21runrttasks09:49:191
46812992010,7cyclictest140291-21ssh10:15:261
4681099208,8cyclictest129837-21awk10:00:180
4681099206,2cyclictest164122-21sh10:51:270
4681099201,9cyclictest22699150/usr/sbin/munin12:20:250
46810992018,1cyclictest0-21swapper/012:27:070
4681099201,7cyclictest78182-21needreboot08:35:470
46810992015,3cyclictest712-21tpm2-abrmd08:25:210
4681099201,17cyclictest218411-21runrttasks12:09:520
46810992010,6cyclictest1090808-21systemd-journal08:16:430
4681099200,18cyclictest155611-21ssh10:39:480
4682099191,6cyclictest57050-31sort07:37:243
46820991916,1cyclictest0-21swapper/311:00:163
46820991912,5cyclictest940-21runrttasks12:32:523
46820991912,5cyclictest180676-21ntpq11:15:173
4682099191,17cyclictest197247-21ssh11:37:553
46820991911,6cyclictest143387-21ldconfig10:20:163
46820991911,5cyclictest210211-21sh11:55:443
4682099190,8cyclictest174260-21taskset11:07:173
4682099190,7cyclictest191493-21ntp_states11:30:153
4682099190,18cyclictest0-21swapper/311:51:173
4681699199,7cyclictest942-21snmpd10:28:242
4681699197,9cyclictest122323-21ntpq09:50:132
4681699197,9cyclictest122323-21ntpq09:50:132
46816991914,3cyclictest57870-21fschecks_count07:40:152
4681699191,16cyclictest159286-21sh10:44:232
4681699191,16cyclictest105665-21seq09:28:362
46816991911,5cyclictest240794-21sh12:39:372
46816991910,6cyclictest160904-21ssh10:47:512
4681699190,6cyclictest84689-21/usr/sbin/munin08:50:242
4681699190,5cyclictest153235-21ssh10:35:222
4681699190,17cyclictest66703-21/usr/sbin/munin08:05:402
4681699190,17cyclictest203918-21runrttasks11:48:372
4681699190,17cyclictest203918-21runrttasks11:48:372
46812991915,3cyclictest1089881-21rs:main1
46812991915,3cyclictest1089881-21rs:main1
46812991914,3cyclictest86545-21pmu-power08:55:211
46812991914,3cyclictest712-21tpm2-abrmd07:15:211
46812991914,3cyclictest47090-21cpuspeed_turbos07:10:101
46812991911,6cyclictest218818-21ssh12:10:081
46812991911,6cyclictest182184-21ssh11:17:161
46812991911,6cyclictest167518-21ssh10:55:381
46812991911,6cyclictest148994-21seq10:30:581
46812991911,5cyclictest88043-21sed09:00:131
4681299191,15cyclictest173596-21taskset11:05:321
4681299191,15cyclictest146938-21ssh10:26:131
4681299190,16cyclictest203460-21ssh11:47:421
4681299190,16cyclictest203460-21ssh11:47:421
46810991915,2cyclictest0-21swapper/010:30:000
46810991915,2cyclictest0-21swapper/009:50:120
46810991915,2cyclictest0-21swapper/009:50:120
46810991914,3cyclictest176145-21cpuspeed_turbos11:10:120
46810991914,3cyclictest156185-21cpuspeed_turbos10:40:120
46810991911,6cyclictest239243-21taskset12:36:420
4681099191,16cyclictest173777-21runrttasks11:06:000
4681099191,16cyclictest165982-21/usr/sbin/munin10:55:150
4681099191,15cyclictest103229-21sh09:24:170
46810991910,7cyclictest53085-21sshd07:27:050
4681099190,17cyclictest170031-21fschecks_time11:00:150
4681099190,17cyclictest141380-21ssh10:17:590
4681099190,17cyclictest116747-21sh09:41:230
4682099189,6cyclictest1090808-21systemd-journal08:20:243
4682099183,5cyclictest66228-21ldconfig08:05:363
46820991815,2cyclictest148482-21latency_hist10:30:003
46820991814,3cyclictest84955-21unixbench_multi08:50:263
46820991814,2cyclictest50699-21dump-pmu-power07:20:013
46820991814,2cyclictest0-21swapper/312:20:003
46820991812,4cyclictest91335-21diskmemload10:40:263
46820991811,5cyclictest91335-21diskmemload11:46:573
46820991811,5cyclictest91335-21diskmemload11:46:573
4682099181,15cyclictest91335-21diskmemload09:35:453
4682099181,15cyclictest188926-21taskset11:27:283
4682099180,16cyclictest820960-21sshd07:49:243
4682099180,16cyclictest820960-21sshd07:49:243
4682099180,15cyclictest94738-21sh09:13:253
4681699189,6cyclictest236423-21anacron12:34:122
4681699188,7cyclictest722-21gdbus07:45:202
4681699188,7cyclictest722-21gdbus07:45:202
4681699181,15cyclictest108569-21ssh09:31:222
46816991810,5cyclictest74740-21/usr/sbin/munin08:25:232
4681699180,6cyclictest72153-21/usr/sbin/munin08:20:152
4681699180,16cyclictest86291-21/usr/sbin/munin08:55:172
4681299189,6cyclictest190182-21taskset11:29:451
46812991812,4cyclictest67534-21cpuspeed_turbos08:10:311
46812991810,6cyclictest233500-21/usr/sbin/munin12:30:171
4681299180,16cyclictest51241-21grep07:20:151
4681099189,6cyclictest72296-21ntpq08:20:170
4681099188,7cyclictest83974-21perl08:50:150
4681099188,7cyclictest726-21gdbus08:55:230
4681099188,7cyclictest59912-21apt-get07:45:110
4681099188,7cyclictest59912-21apt-get07:45:100
4681099186,2cyclictest91335-21diskmemload12:17:210
4681099181,5cyclictest91335-21diskmemload11:54:430
46810991814,3cyclictest67484-21cut08:10:310
46810991813,3cyclictest90282-21needreboot09:05:440
46810991813,3cyclictest149054-21cpuspeed_turbos10:31:020
46810991811,5cyclictest54270-21ldconfig07:30:120
4681099181,14cyclictest82376-21tail08:45:160
46810991810,6cyclictest65812-21smartctl08:05:310
4681099180,6cyclictest208651-21ssh11:55:110
4681099180,5cyclictest134855-21/usr/sbin/munin10:10:120
4682099179,5cyclictest91029-21sshd09:06:153
4682099171,3cyclictest135253-21sed10:10:103
46820991713,3cyclictest81479-21turbostat08:45:003
46820991713,3cyclictest51730-21ntp_kernel_pll_07:20:193
46820991712,3cyclictest82985-21grep08:45:233
46820991712,3cyclictest75881-21gltestperf08:30:173
46820991712,3cyclictest69333-21/usr/sbin/munin08:15:183
46820991711,4cyclictest91335-21diskmemload10:46:483
4681699179,6cyclictest48975-21/usr/sbin/munin07:15:152
4681699177,7cyclictest53739-21df07:30:082
4681699171,5cyclictest77836-21/usr/sbin/munin08:35:442
46816991715,1cyclictest488-21hwrng08:12:172
46816991714,2cyclictest488-21hwrng07:55:542
46816991714,2cyclictest488-21hwrng07:25:112
4681299179,6cyclictest82445-21ntpq08:45:181
4681299179,5cyclictest53162-21sshd07:28:051
4681299178,7cyclictest66197-21modprobe08:05:351
4681299177,7cyclictest1090808-21systemd-journal09:07:241
4681299171,6cyclictest200534-21ssh11:41:261
46812991714,2cyclictest488-21hwrng10:22:201
46812991714,2cyclictest488-21hwrng08:00:001
46812991712,3cyclictest122251-21needreboot09:50:131
46812991712,3cyclictest122251-21needreboot09:50:121
46812991712,3cyclictest107539-21ntpq09:30:151
46812991711,5cyclictest94465-21ssh09:12:541
4681299171,14cyclictest62138-21/usr/sbin/munin07:50:161
4681299171,13cyclictest76262-21ntpq08:30:211
46812991710,2cyclictest191rcu_preempt10:10:301
4681299170,7cyclictest232264-21ssh12:29:421
4681299170,5cyclictest69568-21grep08:15:151
4681099171,2cyclictest79783-21apt-get08:40:120
46810991712,3cyclictest48255-21unixbench_multi07:10:230
46810991712,3cyclictest143056-21ls10:20:140
46810991712,3cyclictest0-21swapper/012:30:250
46810991712,2cyclictest1090808-21systemd-journal08:01:240
4681099171,14cyclictest195087-21memory11:35:160
4681099171,13cyclictest63790-21sshd07:54:250
46810991710,5cyclictest91335-21diskmemload09:30:130
4681099170,15cyclictest160861-21ssh10:47:450
4682099168,6cyclictest58473-21/usr/sbin/munin07:40:213
4682099167,6cyclictest79235-21systemctl08:40:003
4682099167,6cyclictest59873-21gdbus07:55:063
46820991614,1cyclictest488-21hwrng09:04:303
4682099161,2cyclictest121395-21latency_hist09:50:003
46820991611,3cyclictest511ktimers/308:25:163
4681699167,6cyclictest65138-21cron08:05:002
46816991614,1cyclictest488-21hwrng09:07:352
46816991614,1cyclictest488-21hwrng08:33:472
46816991614,1cyclictest488-21hwrng07:20:032
4681699160,14cyclictest79765-21/usr/sbin/munin08:40:142
4681299166,7cyclictest699-21dbus-daemon07:35:001
46812991614,1cyclictest488-21hwrng08:01:011
46812991612,2cyclictest80040-21runrttasks08:40:151
4681099168,5cyclictest87992-21tail09:00:140
4681099161,2cyclictest202564-21sh11:45:130
4681099161,2cyclictest202564-21sh11:45:130
4682099156,6cyclictest47467-21irqstats07:10:143
4682099151,3cyclictest62852-21ntpq07:50:213
4681299151,2cyclictest58022-21grep07:40:161
46812991511,3cyclictest83916-21fschecks_count08:50:141
4681099151,2cyclictest0-21swapper/007:20:150
4682099145,5cyclictest55420-21cat07:34:593
4681099146,5cyclictest64569-21cut08:00:010
4682099135,5cyclictest53196-21seq07:28:293
4682099120,8cyclictest65009-21taskset08:03:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional