You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 18:28
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Fri Nov 14, 2025 12:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27382204188,11sleep10-21swapper/107:08:521
5962186170,11sleep30-21swapper/307:05:133
26952183166,12sleep00-21swapper/007:08:190
27222117100,12sleep20-21swapper/207:08:412
146092410,0sleep10-21swapper/109:50:011
258252100,0sleep20-21swapper/212:37:492
6354290,0sleep10-21swapper/110:17:521
29899991,7cyclictest0-21swapper/211:40:122
29919970,6cyclictest0-21swapper/310:05:103
29769973,4cyclictest7884-21idleruntime-cro11:39:590
29769960,3cyclictest6119-21basename08:20:000
29769960,3cyclictest13725-21awk08:35:000
29769960,3cyclictest12471-21munin-run11:09:590
2955260,0sleep20-21swapper/212:25:472
29919955,0cyclictest24779-21kworker/3:012:20:393
29919955,0cyclictest24779-21kworker/3:011:58:153
29919955,0cyclictest17347-21kworker/3:210:39:523
29919955,0cyclictest17347-21kworker/3:210:17:293
29919955,0cyclictest17347-21kworker/3:209:55:053
29919955,0cyclictest17347-21kworker/3:209:43:563
29919955,0cyclictest17347-21kworker/3:209:21:313
29919955,0cyclictest17347-21kworker/3:208:59:073
29919955,0cyclictest17347-21kworker/3:208:47:553
29919955,0cyclictest17347-21kworker/3:208:36:433
29919955,0cyclictest17347-21kworker/3:208:25:293
29919952,3cyclictest714-21snmpd10:14:153
29919951,1cyclictest0-21swapper/309:45:113
29919950,5cyclictest12915-21apt-get11:10:023
29919950,5cyclictest10362-21python07:20:203
29919950,5cyclictest0-21swapper/309:13:173
29919950,1cyclictest18486-21diskstats10:40:153
29899955,0cyclictest0-21swapper/212:31:112
29899950,5cyclictest0-21swapper/211:21:212
29899950,0cyclictest31591-21ssh12:07:552
29819951,2cyclictest0-21swapper/109:40:131
29819950,5cyclictest0-21swapper/111:45:371
29819950,5cyclictest0-21swapper/110:24:341
29819950,5cyclictest0-21swapper/110:08:161
29819950,5cyclictest0-21swapper/109:14:261
29819950,5cyclictest0-21swapper/108:19:591
29769955,0cyclictest0-21swapper/011:40:160
29769955,0cyclictest0-21swapper/009:25:140
29769955,0cyclictest0-21swapper/009:15:370
29769950,5cyclictest18678-21munin-run08:45:000
29769950,5cyclictest0-21swapper/010:43:220
29769950,5cyclictest0-21swapper/010:32:100
29769950,5cyclictest0-21swapper/009:23:030
29769950,4cyclictest28239-21ssh11:50:110
29769950,4cyclictest10399-21ssh10:20:150
29769950,0cyclictest0-21swapper/011:10:170
29919944,0cyclictest17347-21kworker/3:210:28:413
29919944,0cyclictest17347-21kworker/3:208:03:063
29919944,0cyclictest17347-21kworker/3:207:18:213
29919944,0cyclictest0-21swapper/311:33:333
29919944,0cyclictest0-21swapper/311:08:483
29919943,1cyclictest55950irq/130-eth012:30:103
29919943,1cyclictest40-21ksoftirqd/308:20:003
29919943,1cyclictest12822-21/usr/sbin/munin12:15:213
29919942,2cyclictest548-21ssh11:35:203
29919942,1cyclictest32394-21ssh12:25:043
29919940,4cyclictest8019-21df_inode09:30:153
29919940,4cyclictest0-21swapper/312:05:113
29919940,4cyclictest0-21swapper/311:40:383
29919940,4cyclictest0-21swapper/311:25:143
29919940,4cyclictest0-21swapper/311:20:343
29919940,4cyclictest0-21swapper/311:15:113
29919940,4cyclictest0-21swapper/310:20:153
29919940,3cyclictest32486-21sh10:46:553
29919940,3cyclictest30335-21diskstats09:25:153
29919940,3cyclictest30278-21/usr/sbin/munin10:30:133
29919940,3cyclictest20583-21grep12:35:153
29919940,3cyclictest18483-21sh09:51:103
29919940,3cyclictest18483-21sh09:51:103
29919940,3cyclictest13709-21latency_hist08:35:003
29919940,3cyclictest0-21swapper/307:40:113
29919940,3cyclictest0-21swapper/307:25:123
29919940,1cyclictest3112-21/usr/sbin/munin07:10:133
29919940,1cyclictest23881-21/usr/sbin/munin07:50:173
29919940,1cyclictest18326-21/usr/sbin/munin11:45:143
29919940,1cyclictest15585-21/usr/sbin/munin10:55:173
29919940,1cyclictest1355-21/usr/sbin/munin08:10:163
29919940,0cyclictest0-21swapper/312:04:233
29899944,0cyclictest0-21swapper/212:03:242
29899944,0cyclictest0-21swapper/211:16:062
29899944,0cyclictest0-21swapper/210:05:242
29899942,1cyclictest711-21runrttasks08:15:182
29899942,1cyclictest0-21swapper/208:25:162
29899941,3cyclictest18107-21ssh10:55:432
29899941,2cyclictest14972-21ssh12:15:222
29899941,2cyclictest0-21swapper/209:10:122
29899941,1cyclictest0-21swapper/207:55:122
29899940,4cyclictest21395-21apt-get07:45:012
29899940,4cyclictest0-21swapper/211:28:402
29899940,4cyclictest0-21swapper/211:00:222
29899940,4cyclictest0-21swapper/210:46:082
29899940,4cyclictest0-21swapper/210:28:522
29899940,4cyclictest0-21swapper/210:10:562
29899940,4cyclictest0-21swapper/209:46:332
29899940,4cyclictest0-21swapper/209:26:552
29899940,4cyclictest0-21swapper/208:35:152
29899940,3cyclictest7493-21ssh09:30:132
29899940,3cyclictest5725-21/usr/sbin/munin11:55:132
29899940,3cyclictest5706-21ssh10:17:282
29899940,3cyclictest4352-21ssh12:10:172
29899940,3cyclictest364-21python08:05:192
29899940,3cyclictest32588-21ssh11:35:202
29899940,3cyclictest30502-21wc10:30:122
29899940,3cyclictest27851-21cat11:49:592
29899940,3cyclictest25879-21ssh12:21:032
29899940,3cyclictest25757-21/usr/sbin/munin09:55:152
29899940,3cyclictest2499-21latency_hist11:05:002
29899940,3cyclictest21749-21perl11:30:182
29899940,3cyclictest19975-21ssh09:52:072
29899940,3cyclictest19975-21ssh09:52:062
29899940,3cyclictest15085-21ssh10:38:352
29899940,3cyclictest10941-21diskstats10:20:152
29899940,3cyclictest0-21swapper/209:35:132
29899940,1cyclictest12863-21apt-key11:10:012
29899940,0cyclictest0-21swapper/211:53:392
29819944,0cyclictest0-21swapper/112:27:481
29819944,0cyclictest0-21swapper/111:33:321
29819944,0cyclictest0-21swapper/111:01:141
29819944,0cyclictest0-21swapper/108:10:151
29819944,0cyclictest0-21swapper/107:35:291
29819941,2cyclictest0-21swapper/112:20:121
29819941,2cyclictest0-21swapper/110:55:161
29819941,2cyclictest0-21swapper/110:35:151
29819941,2cyclictest0-21swapper/110:10:131
29819941,2cyclictest0-21swapper/109:55:131
29819941,2cyclictest0-21swapper/107:25:111
29819940,4cyclictest0-21swapper/111:20:001
29819940,4cyclictest0-21swapper/111:05:451
29819940,4cyclictest0-21swapper/110:46:591
29819940,4cyclictest0-21swapper/108:57:151
29819940,2cyclictest0-21swapper/111:35:121
29819940,2cyclictest0-21swapper/108:30:121
29819940,0cyclictest0-21swapper/112:10:301
29769944,0cyclictest0-21swapper/011:21:300
29769944,0cyclictest0-21swapper/010:38:070
29769944,0cyclictest0-21swapper/009:50:070
29769944,0cyclictest0-21swapper/009:50:070
29769943,1cyclictest7157-21sed07:15:150
29769943,1cyclictest17159-21wc10:55:230
29769942,1cyclictest32584-21ssh10:15:070
29769941,3cyclictest3-21ksoftirqd/008:00:150
29769941,2cyclictest2641-21wc12:10:010
29769941,2cyclictest0-21swapper/011:30:160
29769941,2cyclictest0-21swapper/009:30:130
29769941,2cyclictest0-21swapper/008:50:110
29769941,2cyclictest0-21swapper/007:10:110
29769940,4cyclictest5073-21/usr/sbin/munin09:45:200
29769940,4cyclictest31035-21diskmemload12:35:130
29769940,4cyclictest31035-21diskmemload09:42:510
29769940,4cyclictest28955-21ssh11:17:420
29769940,4cyclictest21280-21ssh09:36:270
29769940,4cyclictest0-21swapper/012:30:100
29769940,4cyclictest0-21swapper/012:15:120
29769940,4cyclictest0-21swapper/012:10:120
29769940,4cyclictest0-21swapper/012:00:130
29769940,4cyclictest0-21swapper/011:45:130
29769940,4cyclictest0-21swapper/010:50:140
29769940,4cyclictest0-21swapper/010:45:500
29769940,4cyclictest0-21swapper/010:25:490
29769940,4cyclictest0-21swapper/010:06:500
29769940,4cyclictest0-21swapper/010:00:130
29769940,4cyclictest0-21swapper/009:10:140
29769940,4cyclictest0-21swapper/008:55:120
29769940,4cyclictest0-21swapper/008:10:120
29769940,4cyclictest0-21swapper/007:45:140
29769940,3cyclictest6830-21ssh12:28:160
29769940,3cyclictest30688-21ssh11:02:200
29769940,3cyclictest30065-21ntp_states09:05:170
29769940,3cyclictest27619-21ntp_states07:55:180
29769940,3cyclictest24975-21memory07:50:180
29769940,3cyclictest23284-21df_inode10:10:140
29769940,3cyclictest20596-21sed11:30:000
29919933,0cyclictest0-21swapper/307:35:023
29919932,1cyclictest714-21snmpd09:00:113
29919932,1cyclictest26952-21grep07:55:143
29919930,3cyclictest8996-21rm09:15:033
29919930,3cyclictest5329-21sh10:50:053
29919930,3cyclictest28005-21ssh11:50:023
29919930,3cyclictest25519-21sh11:00:143
29919930,3cyclictest21775-21expr08:50:123
29919930,3cyclictest0-21swapper/312:10:083
29919930,3cyclictest0-21swapper/310:00:133
29919930,3cyclictest0-21swapper/309:35:133
29919930,3cyclictest0-21swapper/309:05:113
29919930,3cyclictest0-21swapper/308:40:133
29919930,3cyclictest0-21swapper/308:20:013
29919930,3cyclictest0-21swapper/308:05:013
29919930,3cyclictest0-21swapper/307:45:013
29919930,0cyclictest0-21swapper/307:30:133
29899932,1cyclictest714-21snmpd07:35:052
29899932,1cyclictest714-21snmpd07:30:112
29899931,1cyclictest28798-21apt-get09:05:112
29899931,1cyclictest0-21swapper/208:30:122
29899930,3cyclictest9229-21cstates07:20:142
29899930,3cyclictest9055-21ssh09:15:052
29899930,3cyclictest5873-21grep07:15:002
29899930,3cyclictest5239-21ssh10:50:022
29899930,3cyclictest18942-21apt-get09:20:022
29899930,3cyclictest18827-21basename07:40:112
29899930,3cyclictest1879-21df08:10:142
29899930,3cyclictest0-21swapper/210:00:062
29899930,3cyclictest0-21swapper/209:40:122
29899930,3cyclictest0-21swapper/209:00:112
29899930,3cyclictest0-21swapper/208:55:122
29899930,3cyclictest0-21swapper/208:50:122
29899930,3cyclictest0-21swapper/208:45:122
29899930,3cyclictest0-21swapper/208:40:062
29899930,3cyclictest0-21swapper/208:20:012
29899930,3cyclictest0-21swapper/208:00:152
29899930,3cyclictest0-21swapper/207:50:132
29899930,3cyclictest0-21swapper/207:25:032
29899930,3cyclictest0-21swapper/207:10:112
29899930,2cyclictest17623-21ssh10:40:022
29819933,0cyclictest0-21swapper/111:41:501
29819933,0cyclictest0-21swapper/111:25:141
29819931,2cyclictest0-21swapper/109:35:121
29819931,1cyclictest177382chrt12:33:551
29819931,1cyclictest0-21swapper/112:05:121
29819931,1cyclictest0-21swapper/111:20:121
29819931,1cyclictest0-21swapper/111:10:141
29819931,1cyclictest0-21swapper/110:40:141
29819931,1cyclictest0-21swapper/110:30:121
29819931,1cyclictest0-21swapper/110:00:131
29819931,1cyclictest0-21swapper/109:50:121
29819931,1cyclictest0-21swapper/109:50:121
29819931,1cyclictest0-21swapper/109:25:131
29819931,1cyclictest0-21swapper/109:20:131
29819931,1cyclictest0-21swapper/109:05:101
29819931,1cyclictest0-21swapper/108:45:121
29819931,1cyclictest0-21swapper/108:40:121
29819931,1cyclictest0-21swapper/108:35:131
29819931,1cyclictest0-21swapper/108:20:121
29819931,1cyclictest0-21swapper/108:05:121
29819931,1cyclictest0-21swapper/107:40:111
29819931,1cyclictest0-21swapper/107:20:121
29819931,1cyclictest0-21swapper/107:10:111
29819930,3cyclictest0-21swapper/107:57:531
29819930,2cyclictest0-21swapper/112:35:131
29819930,2cyclictest0-21swapper/112:00:131
29819930,2cyclictest0-21swapper/111:55:121
29819930,2cyclictest0-21swapper/107:50:121
29819930,2cyclictest0-21swapper/107:45:111
29769931,2cyclictest9565-21fschecks_count08:25:180
29769931,1cyclictest5337-21ssh11:55:010
29769930,3cyclictest26854-21df_inode09:00:130
29769930,3cyclictest22757-21rm12:20:120
29769930,3cyclictest18901-21apt-get08:45:110
29769930,3cyclictest18855-21grep07:40:120
29769930,3cyclictest0-21swapper/009:55:130
29769930,3cyclictest0-21swapper/008:35:150
29769930,3cyclictest0-21swapper/008:20:100
29769930,3cyclictest0-21swapper/008:05:030
29769930,3cyclictest0-21swapper/007:35:070
29769930,3cyclictest0-21swapper/007:30:120
29769930,3cyclictest0-21swapper/007:25:120
29769930,2cyclictest8829-21taskset07:20:010
29819922,0cyclictest0-21swapper/108:51:171
29819921,1cyclictest0-21swapper/111:50:111
29819921,0cyclictest0-21swapper/109:00:121
29819920,2cyclictest0-21swapper/110:25:121
29819920,1cyclictest0-21swapper/112:15:111
29819920,1cyclictest0-21swapper/110:50:141
29819920,1cyclictest0-21swapper/109:30:131
29819920,1cyclictest0-21swapper/109:15:121
29819920,1cyclictest0-21swapper/108:25:161
29819920,1cyclictest0-21swapper/108:00:151
29819920,1cyclictest0-21swapper/107:30:121
29819920,1cyclictest0-21swapper/107:15:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional