You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 11:49
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot2.osadl.org (updated Mon Dec 01, 2025 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
289792201184,12sleep00-21swapper/019:07:180
289822200183,12sleep30-21swapper/319:07:213
290622191177,9sleep10-21swapper/119:08:231
288382162142,15sleep20-21swapper/219:05:322
142652450,0sleep30-21swapper/321:52:393
226472110,0sleep00-21swapper/023:34:480
2936299102,4cyclictest23206-21munin-run22:30:000
293729991,4cyclictest0-21swapper/323:00:133
293629992,4cyclictest25500-21munin-run23:20:000
293629991,4cyclictest17430-21awk00:05:010
23686290,0sleep10-21swapper/123:02:111
293689970,5cyclictest0-21swapper/222:20:102
293629970,4cyclictest10923-21munin-run22:39:590
293729966,0cyclictest391ktimersoftd/319:15:133
293729965,0cyclictest40-21ksoftirqd/300:35:173
293689960,5cyclictest0-21swapper/221:55:152
293639961,3cyclictest0-21swapper/122:55:121
293629960,5cyclictest0-21swapper/021:35:130
293629960,3cyclictest10137-21munin-run20:40:000
293729955,0cyclictest40-21ksoftirqd/323:25:123
293729955,0cyclictest40-21ksoftirqd/322:00:133
293729955,0cyclictest40-21ksoftirqd/321:45:133
293729955,0cyclictest40-21ksoftirqd/319:55:113
293729955,0cyclictest0-21swapper/319:30:013
293729953,2cyclictest40-21ksoftirqd/320:55:003
293729953,1cyclictest40-21ksoftirqd/320:15:183
293729952,3cyclictest2088-21irqstats23:40:163
293729950,5cyclictest7273-21ssh00:33:323
293729950,5cyclictest0-21swapper/323:59:393
293729950,5cyclictest0-21swapper/323:05:493
293729950,5cyclictest0-21swapper/320:55:253
293729950,5cyclictest0-21swapper/320:30:013
293729950,5cyclictest0-21swapper/320:25:203
293729950,5cyclictest0-21swapper/300:03:263
293729950,4cyclictest0-21swapper/322:30:133
293729950,3cyclictest0-21swapper/322:20:063
293689955,0cyclictest0-21swapper/223:24:282
293689953,2cyclictest20275-21kworker/2:022:28:012
293689950,5cyclictest0-21swapper/223:48:022
293689950,5cyclictest0-21swapper/221:36:562
293689950,5cyclictest0-21swapper/220:20:162
293689950,5cyclictest0-21swapper/220:10:212
293689950,5cyclictest0-21swapper/200:34:442
293689950,5cyclictest0-21swapper/200:15:152
293689950,0cyclictest0-21swapper/222:17:322
293689950,0cyclictest0-21swapper/200:29:302
293639952,2cyclictest0-21swapper/120:40:131
293639952,2cyclictest0-21swapper/120:35:141
293639951,3cyclictest0-21swapper/121:25:131
293639951,3cyclictest0-21swapper/100:00:121
293639951,2cyclictest0-21swapper/123:30:011
293639951,2cyclictest0-21swapper/121:05:151
293639951,2cyclictest0-21swapper/120:55:131
293629953,1cyclictest225-21systemd-journal21:05:010
293629950,5cyclictest0-21swapper/021:30:140
293629950,5cyclictest0-21swapper/000:34:210
26279250,4sleep026282-21/usr/sbin/munin22:15:150
16945250,0sleep10-21swapper/121:37:481
7527240,0sleep10-21swapper/122:21:231
293729944,0cyclictest40-21ksoftirqd/323:10:133
293729944,0cyclictest40-21ksoftirqd/322:55:153
293729944,0cyclictest40-21ksoftirqd/322:55:003
293729944,0cyclictest40-21ksoftirqd/322:35:143
293729944,0cyclictest40-21ksoftirqd/321:55:163
293729944,0cyclictest40-21ksoftirqd/321:15:153
293729944,0cyclictest40-21ksoftirqd/321:15:143
293729944,0cyclictest40-21ksoftirqd/321:10:133
293729944,0cyclictest40-21ksoftirqd/321:05:163
293729944,0cyclictest40-21ksoftirqd/320:00:123
293729944,0cyclictest40-21ksoftirqd/319:20:163
293729944,0cyclictest391ktimersoftd/322:10:133
293729944,0cyclictest31907-21kworker/3:022:26:013
293729944,0cyclictest31907-21kworker/3:019:49:163
293729944,0cyclictest0-21swapper/323:45:143
293729944,0cyclictest0-21swapper/323:35:213
293729943,1cyclictest9490-21wc20:35:223
293729943,1cyclictest6615-21pmu-power19:25:173
293729943,1cyclictest11111-21grep20:40:153
293729943,1cyclictest0-21swapper/321:40:123
293729943,0cyclictest40-21ksoftirqd/300:15:083
293729942,2cyclictest40-21ksoftirqd/321:35:293
293729942,2cyclictest40-21ksoftirqd/300:10:223
293729942,2cyclictest0-21swapper/319:35:183
293729940,4cyclictest26285-21apt-get22:15:143
293729940,4cyclictest0-21swapper/322:40:123
293729940,4cyclictest0-21swapper/321:25:133
293729940,4cyclictest0-21swapper/320:20:133
293729940,4cyclictest0-21swapper/319:50:143
293729940,4cyclictest0-21swapper/300:27:473
293729940,4cyclictest0-21swapper/300:06:023
293729940,3cyclictest8754-21sh22:05:363
293729940,3cyclictest27926-21grep20:10:153
293729940,3cyclictest22776-21ntpq22:45:183
293729940,3cyclictest21194-21sh23:17:173
293729940,3cyclictest20784-21/usr/sbin/munin23:50:183
293729940,3cyclictest1576-21df21:30:153
293729940,3cyclictest12955-21/usr/sbin/munin00:20:103
293729940,3cyclictest0-21swapper/323:30:113
293729940,1cyclictest26005-21/usr/sbin/munin23:20:173
293729940,1cyclictest25251-21/usr/sbin/munin20:05:163
293689944,0cyclictest20275-21kworker/2:023:56:012
293689944,0cyclictest0-21swapper/222:36:092
293689944,0cyclictest0-21swapper/200:11:272
293689943,1cyclictest31-21ksoftirqd/222:50:002
293689943,1cyclictest19411-21tail19:50:232
293689943,1cyclictest12978-21kworker/2:100:20:252
293689942,2cyclictest20275-21kworker/2:022:04:092
293689942,1cyclictest8273-21ssh21:33:392
293689941,2cyclictest25863-21date22:15:012
293689940,4cyclictest24980-21diskmemload22:41:172
293689940,4cyclictest0-21swapper/223:35:492
293689940,4cyclictest0-21swapper/223:31:422
293689940,4cyclictest0-21swapper/223:25:122
293689940,4cyclictest0-21swapper/223:15:122
293689940,4cyclictest0-21swapper/223:05:142
293689940,4cyclictest0-21swapper/222:50:142
293689940,4cyclictest0-21swapper/221:50:122
293689940,4cyclictest0-21swapper/221:40:132
293689940,4cyclictest0-21swapper/221:29:462
293689940,4cyclictest0-21swapper/221:00:102
293689940,4cyclictest0-21swapper/220:40:132
293689940,4cyclictest0-21swapper/219:10:172
293689940,3cyclictest6207-21ssh23:10:132
293689940,3cyclictest25690-21/usr/sbin/munin21:10:152
293689940,3cyclictest18231-21latency_hist23:00:012
293689940,3cyclictest10324-21ls20:39:592
293689940,3cyclictest0-21swapper/200:00:122
293639944,0cyclictest0-21swapper/123:22:591
293639944,0cyclictest0-21swapper/122:19:331
293639944,0cyclictest0-21swapper/122:13:351
293639944,0cyclictest0-21swapper/122:02:221
293639944,0cyclictest0-21swapper/121:30:381
293639944,0cyclictest0-21swapper/100:37:201
293639944,0cyclictest0-21swapper/100:08:401
293639942,1cyclictest0-21swapper/119:35:121
293639942,1cyclictest0-21swapper/119:25:101
293639941,2cyclictest0-21swapper/123:50:121
293639941,2cyclictest0-21swapper/123:40:121
293639941,2cyclictest0-21swapper/123:10:131
293639941,2cyclictest0-21swapper/122:50:141
293639941,2cyclictest0-21swapper/122:30:131
293639941,2cyclictest0-21swapper/121:50:131
293639941,2cyclictest0-21swapper/121:45:131
293639941,2cyclictest0-21swapper/121:20:121
293639941,2cyclictest0-21swapper/120:45:131
293639941,2cyclictest0-21swapper/120:30:121
293639941,2cyclictest0-21swapper/120:25:131
293639941,2cyclictest0-21swapper/120:00:121
293639941,2cyclictest0-21swapper/119:50:151
293639941,2cyclictest0-21swapper/119:30:131
293639941,2cyclictest0-21swapper/119:10:141
293639941,2cyclictest0-21swapper/100:10:081
293639941,1cyclictest0-21swapper/121:00:111
293639941,1cyclictest0-21swapper/120:05:111
293639940,4cyclictest0-21swapper/122:25:311
293639940,4cyclictest0-21swapper/122:07:371
293639940,2cyclictest0-21swapper/123:05:141
293639940,2cyclictest0-21swapper/119:55:111
293639940,0cyclictest0-21swapper/122:45:531
293639940,0cyclictest0-21swapper/121:44:381
293629944,0cyclictest0-21swapper/023:56:400
293629944,0cyclictest0-21swapper/022:43:110
293629944,0cyclictest0-21swapper/000:20:050
293629943,1cyclictest20867-21grep19:55:130
293629941,3cyclictest2645-21kworker/0:122:55:130
293629941,2cyclictest0-21swapper/020:55:130
293629941,0cyclictest0-21swapper/022:50:140
293629940,4cyclictest32355-21chrt23:06:220
293629940,4cyclictest28721-21apt-get22:00:130
293629940,4cyclictest22764-21apt-get20:00:010
293629940,4cyclictest13870-21ssh23:45:570
293629940,4cyclictest0-21swapper/023:37:200
293629940,4cyclictest0-21swapper/023:25:120
293629940,4cyclictest0-21swapper/023:21:420
293629940,4cyclictest0-21swapper/023:10:240
293629940,4cyclictest0-21swapper/022:47:470
293629940,4cyclictest0-21swapper/022:31:330
293629940,4cyclictest0-21swapper/022:20:110
293629940,4cyclictest0-21swapper/021:55:150
293629940,4cyclictest0-21swapper/021:25:130
293629940,4cyclictest0-21swapper/021:10:130
293629940,4cyclictest0-21swapper/020:10:140
293629940,4cyclictest0-21swapper/019:45:110
293629940,4cyclictest0-21swapper/019:35:120
293629940,4cyclictest0-21swapper/000:15:210
293629940,4cyclictest0-21swapper/000:11:480
293629940,3cyclictest674-21users21:45:220
293629940,3cyclictest6288-21ssh22:05:130
293629940,3cyclictest5020-21latency_hist19:25:000
293629940,3cyclictest3347-21ssh21:15:140
293629940,3cyclictest3347-21ssh21:15:140
293629940,3cyclictest32499-21munin-node19:15:010
293629940,3cyclictest20656-21cat23:00:210
293629940,3cyclictest17157-21ssh00:38:360
293629940,2cyclictest14925-21grep19:45:000
293629940,1cyclictest7660-21munin-run20:35:000
293629940,1cyclictest30511-21idleruntime-cro21:45:010
293729931,2cyclictest55950irq/130-eth021:20:063
293729930,3cyclictest29744-21/usr/sbin/munin19:10:153
293729930,3cyclictest20966-21perl21:00:143
293729930,3cyclictest13017-21cat20:45:133
293729930,3cyclictest0-21swapper/319:40:113
293689933,0cyclictest20275-21kworker/2:022:05:092
293689933,0cyclictest20275-21kworker/2:020:25:052
293689933,0cyclictest20275-21kworker/2:019:55:092
293689932,1cyclictest714-21snmpd23:40:032
293689932,1cyclictest714-21snmpd23:00:112
293689932,1cyclictest714-21snmpd19:45:012
293689932,1cyclictest714-21snmpd00:35:062
293689931,1cyclictest714-21snmpd19:25:012
293689930,3cyclictest25254-21apt-get20:05:112
293689930,3cyclictest20777-21ssh23:50:122
293689930,3cyclictest17719-21ssh00:05:042
293689930,3cyclictest13342-21apt-get21:20:112
293689930,3cyclictest10942-21grep19:35:152
293689930,3cyclictest0-21swapper/222:30:132
293689930,3cyclictest0-21swapper/221:45:022
293689930,3cyclictest0-21swapper/221:15:022
293689930,3cyclictest0-21swapper/221:15:022
293689930,3cyclictest0-21swapper/221:05:152
293689930,3cyclictest0-21swapper/220:55:132
293689930,3cyclictest0-21swapper/220:50:132
293689930,3cyclictest0-21swapper/220:45:112
293689930,3cyclictest0-21swapper/220:30:012
293689930,3cyclictest0-21swapper/220:15:062
293689930,3cyclictest0-21swapper/220:00:012
293689930,3cyclictest0-21swapper/219:40:132
293689930,3cyclictest0-21swapper/219:30:112
293689930,3cyclictest0-21swapper/219:20:142
293689930,3cyclictest0-21swapper/219:15:012
293639933,0cyclictest0-21swapper/100:20:011
293639931,2cyclictest0-21swapper/123:45:121
293639931,2cyclictest0-21swapper/120:50:131
293639931,2cyclictest0-21swapper/119:40:131
293639931,1cyclictest0-21swapper/123:55:121
293639931,1cyclictest0-21swapper/123:35:131
293639931,1cyclictest0-21swapper/123:25:121
293639931,1cyclictest0-21swapper/121:55:151
293639931,1cyclictest0-21swapper/121:15:141
293639931,1cyclictest0-21swapper/121:15:141
293639931,1cyclictest0-21swapper/121:10:021
293639931,1cyclictest0-21swapper/120:15:111
293639931,1cyclictest0-21swapper/119:20:131
293639931,1cyclictest0-21swapper/119:20:001
293639930,2cyclictest7029-21taskset00:16:291
293639930,2cyclictest114682sleep122:40:131
293639930,2cyclictest0-21swapper/120:10:141
293639930,2cyclictest0-21swapper/119:45:111
293629932,1cyclictest9171-21ssh21:50:140
293629930,3cyclictest25254-21apt-get20:05:110
293629930,3cyclictest22924-21cut00:25:110
293629930,3cyclictest21372-21fschecks_count23:50:150
293629930,3cyclictest1012-21tune2fs20:20:160
293629930,3cyclictest0-21swapper/023:40:120
293629930,3cyclictest0-21swapper/022:10:110
293629930,3cyclictest0-21swapper/021:20:110
293629930,3cyclictest0-21swapper/021:05:130
293629930,3cyclictest0-21swapper/020:50:130
293629930,3cyclictest0-21swapper/020:45:130
293629930,3cyclictest0-21swapper/020:40:130
293629930,3cyclictest0-21swapper/020:25:130
293629930,3cyclictest0-21swapper/020:15:110
293629930,3cyclictest0-21swapper/019:53:170
293629930,3cyclictest0-21swapper/019:30:120
293629930,3cyclictest0-21swapper/019:25:100
293629930,3cyclictest0-21swapper/019:10:140
293629930,3cyclictest0-21swapper/000:05:340
293639921,1cyclictest0-21swapper/100:30:141
293639920,1cyclictest0-21swapper/123:15:131
293639920,1cyclictest0-21swapper/122:35:141
293639920,1cyclictest0-21swapper/120:20:131
293639920,1cyclictest0-21swapper/100:25:111
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional