You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 09:27
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Wed Nov 26, 2025 00:44:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
235642205189,11sleep10-21swapper/119:08:121
234152194177,12sleep30-21swapper/319:06:173
214752193176,12sleep00-21swapper/019:05:020
236622173157,12sleep20-21swapper/219:09:302
25882520,0sleep30-21swapper/322:54:453
304682450,0sleep30-21swapper/323:40:213
161512450,0sleep30-21swapper/323:33:483
104832430,0sleep10-21swapper/100:03:371
55522330,0sleep00-21swapper/023:11:480
238699992,3cyclictest712-21idleruntime-cro23:10:000
238799981,6cyclictest0-21swapper/319:25:133
238739971,5cyclictest0-21swapper/219:30:122
238739965,1cyclictest31-21ksoftirqd/223:10:122
238699960,3cyclictest19762-21idleruntime-cro20:05:010
238799950,5cyclictest9195-21sh00:02:483
238799950,4cyclictest25273-21munin-run22:50:013
238739951,2cyclictest0-21swapper/222:15:112
238739951,2cyclictest0-21swapper/221:00:122
238739951,2cyclictest0-21swapper/219:25:132
238739951,1cyclictest0-21swapper/223:50:112
238739950,0cyclictest0-21swapper/221:59:512
238739950,0cyclictest0-21swapper/221:43:502
238709954,1cyclictest0-21swapper/123:15:141
238699955,0cyclictest3-21ksoftirqd/022:25:000
238699955,0cyclictest19906-21diskmemload21:25:010
238699955,0cyclictest0-21swapper/023:22:080
238699950,5cyclictest0-21swapper/022:47:120
238699950,5cyclictest0-21swapper/021:56:520
238699950,5cyclictest0-21swapper/021:36:210
238699950,5cyclictest0-21swapper/000:37:300
238699950,5cyclictest0-21swapper/000:37:300
238699950,5cyclictest0-21swapper/000:27:450
238699950,0cyclictest81rcu_preempt00:30:120
238699950,0cyclictest0-21swapper/022:02:590
32690240,0sleep30-21swapper/322:36:483
238799944,0cyclictest10674-21kworker/3:219:39:163
238799944,0cyclictest0-21swapper/323:12:023
238799944,0cyclictest0-21swapper/322:21:313
238799941,3cyclictest10356-21sh00:20:043
238799941,2cyclictest0-21swapper/323:35:113
238799940,4cyclictest3782-21sh00:32:423
238799940,4cyclictest1403-21perl00:15:143
238799940,4cyclictest12341-21/usr/sbin/munin19:50:213
238799940,4cyclictest0-21swapper/323:56:023
238799940,4cyclictest0-21swapper/323:50:523
238799940,4cyclictest0-21swapper/323:47:263
238799940,4cyclictest0-21swapper/323:20:133
238799940,4cyclictest0-21swapper/323:19:303
238799940,4cyclictest0-21swapper/323:00:153
238799940,4cyclictest0-21swapper/322:44:313
238799940,4cyclictest0-21swapper/322:18:073
238799940,4cyclictest0-21swapper/321:52:183
238799940,4cyclictest0-21swapper/321:24:493
238799940,4cyclictest0-21swapper/321:11:473
238799940,4cyclictest0-21swapper/320:55:113
238799940,4cyclictest0-21swapper/320:15:123
238799940,4cyclictest0-21swapper/320:00:013
238799940,4cyclictest0-21swapper/319:55:113
238799940,4cyclictest0-21swapper/319:30:123
238799940,4cyclictest0-21swapper/300:35:133
238799940,4cyclictest0-21swapper/300:35:133
238799940,4cyclictest0-21swapper/300:26:343
238799940,4cyclictest0-21swapper/300:11:133
238799940,4cyclictest0-21swapper/300:05:143
238799940,3cyclictest6257-21ssh22:55:453
238799940,3cyclictest5100-21apt-get20:40:113
238799940,3cyclictest2961-21ssh21:17:083
238799940,3cyclictest2961-21ssh21:17:083
238799940,3cyclictest26243-21/usr/sbin/munin21:45:143
238799940,3cyclictest26183-21ssh21:29:073
238799940,3cyclictest23482-21rm23:05:053
238799940,3cyclictest20440-21ssh22:30:213
238799940,3cyclictest13606-21ssh22:27:193
238799940,3cyclictest13383-21/usr/sbin/munin21:55:133
238799940,3cyclictest10661-21date22:10:003
238799940,2cyclictest0-21swapper/321:44:593
238799940,1cyclictest32537-21ntp_states23:25:183
238799940,1cyclictest30085-21/usr/sbin/munin20:25:203
238799940,1cyclictest29603-21ntpq21:30:193
238799940,1cyclictest19971-21/usr/sbin/munin20:05:163
238799940,1cyclictest10403-21diskstats19:45:133
238799940,1cyclictest10238-21/usr/sbin/munin20:50:133
238739944,0cyclictest0-21swapper/222:55:262
238739944,0cyclictest0-21swapper/222:41:452
238739944,0cyclictest0-21swapper/222:25:022
238739944,0cyclictest0-21swapper/221:20:252
238739944,0cyclictest0-21swapper/220:12:362
238739943,1cyclictest28973-21grep21:30:172
238739943,1cyclictest27342-21ssh21:29:512
238739942,1cyclictest26029-21ssh00:27:322
238739942,1cyclictest16119-21ls19:55:172
238739941,2cyclictest0-21swapper/222:30:112
238739941,2cyclictest0-21swapper/221:45:132
238739941,1cyclictest0-21swapper/200:10:122
238739941,1cyclictest0-21swapper/200:00:142
238739941,0cyclictest81rcu_preempt22:47:012
238739940,4cyclictest26935-21perl19:15:002
238739940,4cyclictest19906-21diskmemload23:20:142
238739940,4cyclictest0-21swapper/223:45:132
238739940,4cyclictest0-21swapper/223:35:112
238739940,4cyclictest0-21swapper/223:05:142
238739940,4cyclictest0-21swapper/221:15:122
238739940,4cyclictest0-21swapper/221:15:122
238739940,4cyclictest0-21swapper/220:48:402
238739940,4cyclictest0-21swapper/219:52:222
238739940,4cyclictest0-21swapper/200:39:282
238739940,4cyclictest0-21swapper/200:39:272
238739940,4cyclictest0-21swapper/200:05:122
238739940,3cyclictest8064-21wc22:25:002
238739940,3cyclictest6542-21ssh00:34:272
238739940,3cyclictest30007-21dpkg20:25:002
238739940,1cyclictest698-21ssh22:37:052
238739940,0cyclictest0-21swapper/223:28:442
238739940,0cyclictest0-21swapper/222:05:572
238709944,0cyclictest0-21swapper/121:51:081
238709944,0cyclictest0-21swapper/121:26:511
238709944,0cyclictest0-21swapper/120:44:551
238709944,0cyclictest0-21swapper/100:06:541
238709941,2cyclictest0-21swapper/122:25:111
238709941,1cyclictest0-21swapper/123:25:121
238709940,4cyclictest0-21swapper/123:59:201
238709940,4cyclictest0-21swapper/123:41:521
238709940,2cyclictest0-21swapper/123:10:121
238709940,2cyclictest0-21swapper/121:35:161
238709940,2cyclictest0-21swapper/120:20:121
238709940,1cyclictest0-21swapper/122:05:151
238709940,0cyclictest0-21swapper/122:16:271
238699943,1cyclictest9873-21apt-get19:45:120
238699943,1cyclictest2741-21ssh22:05:210
238699942,2cyclictest26449-21kworker/0:120:05:200
238699942,1cyclictest4486-21munin-run19:35:010
238699942,1cyclictest0-21swapper/022:29:550
238699941,1cyclictest0-21swapper/023:50:110
238699941,1cyclictest0-21swapper/019:25:130
238699941,1cyclictest0-21swapper/000:10:120
238699940,4cyclictest21183-21ssh21:25:590
238699940,4cyclictest19906-21diskmemload23:35:310
238699940,4cyclictest0-21swapper/023:55:110
238699940,4cyclictest0-21swapper/023:40:120
238699940,4cyclictest0-21swapper/023:25:120
238699940,4cyclictest0-21swapper/023:02:540
238699940,4cyclictest0-21swapper/022:55:100
238699940,4cyclictest0-21swapper/022:40:060
238699940,4cyclictest0-21swapper/022:35:120
238699940,4cyclictest0-21swapper/022:30:110
238699940,4cyclictest0-21swapper/022:15:480
238699940,4cyclictest0-21swapper/021:50:130
238699940,4cyclictest0-21swapper/021:12:050
238699940,4cyclictest0-21swapper/019:20:520
238699940,4cyclictest0-21swapper/000:08:510
238699940,3cyclictest8633-21/usr/sbin/munin23:30:160
238699940,3cyclictest5622-21df_inode20:40:130
238699940,3cyclictest32761-21rm00:15:010
238699940,3cyclictest31023-21diskstats21:15:140
238699940,3cyclictest31023-21diskstats21:15:140
238699940,3cyclictest26638-21fw_packets21:45:160
238699940,3cyclictest25911-21ssh22:50:090
238699940,3cyclictest23438-21irqstats20:10:190
238699940,3cyclictest17665-21/usr/sbin/munin21:05:190
238699940,3cyclictest15409-21grep19:55:130
238699940,3cyclictest15395-21ssh22:11:510
238699940,3cyclictest14060-21ssh23:15:530
238699940,1cyclictest26233-21latency_hist19:15:000
238699940,0cyclictest0-21swapper/000:04:280
238799930,3cyclictest902-21sh22:05:093
238799930,3cyclictest27417-21apt-get20:20:113
238799930,3cyclictest26954-21ntpq19:15:013
238799930,3cyclictest22633-21cat20:10:163
238799930,3cyclictest0-21swapper/322:00:013
238799930,3cyclictest0-21swapper/321:05:063
238799930,3cyclictest0-21swapper/321:00:033
238799930,3cyclictest0-21swapper/320:45:053
238799930,3cyclictest0-21swapper/320:35:133
238799930,3cyclictest0-21swapper/320:30:133
238799930,3cyclictest0-21swapper/319:40:133
238799930,3cyclictest0-21swapper/319:20:113
238799930,3cyclictest0-21swapper/319:10:073
238799930,2cyclictest5799-21ssh21:35:143
238739932,1cyclictest8088-21ssh21:35:302
238739932,1cyclictest18024-21df20:00:142
238739931,1cyclictest9436-21apt-get19:45:022
238739931,1cyclictest0-21swapper/220:55:112
238739931,1cyclictest0-21swapper/220:50:122
238739931,1cyclictest0-21swapper/219:10:142
238739930,3cyclictest9355-21perl23:30:142
238739930,3cyclictest5453-21perl19:35:152
238739930,3cyclictest3446-21ssh21:50:132
238739930,3cyclictest25516-21ssh23:55:012
238739930,3cyclictest0-21swapper/223:15:022
238739930,3cyclictest0-21swapper/223:00:052
238739930,3cyclictest0-21swapper/222:10:062
238739930,3cyclictest0-21swapper/222:00:002
238739930,3cyclictest0-21swapper/221:05:172
238739930,3cyclictest0-21swapper/220:40:022
238739930,3cyclictest0-21swapper/220:35:052
238739930,3cyclictest0-21swapper/220:30:122
238739930,3cyclictest0-21swapper/220:20:172
238739930,3cyclictest0-21swapper/220:15:122
238739930,3cyclictest0-21swapper/219:20:112
238739930,3cyclictest0-21swapper/200:15:122
238739930,1cyclictest81rcu_preempt22:50:022
238739930,1cyclictest29732-21memory23:40:162
238739930,1cyclictest20141-21ssh21:10:032
238739930,1cyclictest11691-21ls00:20:162
238739930,0cyclictest0-21swapper/220:07:212
238709933,0cyclictest0-21swapper/121:45:021
238709933,0cyclictest0-21swapper/119:46:351
238709931,1cyclictest0-21swapper/123:45:111
238709931,1cyclictest0-21swapper/123:35:111
238709931,1cyclictest0-21swapper/123:30:121
238709931,1cyclictest0-21swapper/123:05:141
238709931,1cyclictest0-21swapper/123:00:131
238709931,1cyclictest0-21swapper/122:50:091
238709931,1cyclictest0-21swapper/122:45:111
238709931,1cyclictest0-21swapper/122:35:121
238709931,1cyclictest0-21swapper/122:00:121
238709931,1cyclictest0-21swapper/121:30:141
238709931,1cyclictest0-21swapper/121:15:131
238709931,1cyclictest0-21swapper/121:15:121
238709931,1cyclictest0-21swapper/121:05:111
238709931,1cyclictest0-21swapper/120:50:121
238709931,1cyclictest0-21swapper/120:45:121
238709931,1cyclictest0-21swapper/119:30:121
238709931,1cyclictest0-21swapper/100:35:131
238709931,1cyclictest0-21swapper/100:35:131
238709931,1cyclictest0-21swapper/100:25:121
238709931,1cyclictest0-21swapper/100:15:121
238709930,3cyclictest0-21swapper/120:38:491
238709930,3cyclictest0-21swapper/119:54:101
238709930,3cyclictest0-21swapper/119:35:051
238709930,3cyclictest0-21swapper/119:12:491
238709930,2cyclictest197362sleep121:58:021
238709930,2cyclictest13627-21chrt22:10:431
238709930,2cyclictest0-21swapper/122:20:131
238709930,2cyclictest0-21swapper/121:40:121
238709930,2cyclictest0-21swapper/120:05:101
238709930,2cyclictest0-21swapper/100:30:101
238709930,1cyclictest0-21swapper/120:30:121
238699931,1cyclictest4712-21apt19:35:140
238699931,1cyclictest0-21swapper/021:00:120
238699930,3cyclictest647-21df20:30:150
238699930,3cyclictest28159-21apt21:30:130
238699930,3cyclictest27924-21perl19:15:120
238699930,3cyclictest19906-21diskmemload23:45:020
238699930,3cyclictest19906-21diskmemload00:20:010
238699930,3cyclictest12839-21expr19:50:130
238699930,3cyclictest0-21swapper/021:40:050
238699930,3cyclictest0-21swapper/020:55:120
238699930,3cyclictest0-21swapper/020:50:120
238699930,3cyclictest0-21swapper/020:45:010
238699930,3cyclictest0-21swapper/020:35:100
238699930,3cyclictest0-21swapper/020:25:010
238699930,3cyclictest0-21swapper/020:20:110
238699930,3cyclictest0-21swapper/020:15:020
238699930,3cyclictest0-21swapper/019:40:110
12851230,0sleep10-21swapper/122:43:261
238739920,2cyclictest0-21swapper/219:40:002
238709922,0cyclictest0-21swapper/121:00:121
238709921,1cyclictest229042chrt23:20:211
238709921,1cyclictest0-21swapper/120:25:121
238709921,1cyclictest0-21swapper/120:00:121
238709921,1cyclictest0-21swapper/119:15:171
238709921,1cyclictest0-21swapper/100:20:121
238709921,1cyclictest0-21swapper/100:10:121
238709920,1cyclictest0-21swapper/123:50:111
238709920,1cyclictest0-21swapper/122:55:051
238709920,1cyclictest0-21swapper/122:30:371
238709920,1cyclictest0-21swapper/121:20:141
238709920,1cyclictest0-21swapper/121:10:131
238709920,1cyclictest0-21swapper/120:55:111
238709920,1cyclictest0-21swapper/120:15:121
238709920,1cyclictest0-21swapper/120:10:151
238709920,1cyclictest0-21swapper/119:40:111
238709920,1cyclictest0-21swapper/119:25:131
238709920,1cyclictest0-21swapper/119:20:121
238709910,1cyclictest0-21swapper/119:55:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional