You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 06:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot0.osadl.org (updated Fri May 09, 2025 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1984321760,4sleep2970299cyclictest23:53:372
1812421680,3sleep2970299cyclictest23:27:192
317121590,4sleep1969799cyclictest21:42:161
2417021230,3sleep3970399cyclictest23:47:293
346021190,4sleep2970299cyclictest00:22:312
1004321160,3sleep2970299cyclictest22:31:192
30921140,4sleep2970299cyclictest00:01:212
3276621130,0sleep11846-21nfsd22:30:231
1587521100,3sleep215873-21sshd22:54:412
9238210833,45sleep30-21swapper/319:07:283
1719521060,3sleep3970399cyclictest00:34:573
3197721030,4sleep20-21swapper/223:58:062
1002321030,2sleep10-21swapper/100:37:481
787221000,1sleep30-21swapper/321:17:063
34672980,3sleep10-21swapper/100:19:081
324882970,1sleep10-21swapper/121:35:261
27292960,1sleep10-21swapper/100:22:261
165082950,2sleep116312-21sshd22:45:011
228242940,1sleep30-21swapper/300:28:083
180522940,0sleep00-21swapper/021:30:480
217392910,1sleep10-21swapper/123:14:431
150232910,0sleep00-21swapper/023:30:170
107132910,3sleep3970399cyclictest22:28:113
36672900,4sleep10-21swapper/123:25:461
27752900,2sleep22770-21sshd21:32:352
209532900,2sleep10-21swapper/123:50:221
147422900,1sleep014730-21sshd23:17:140
216552890,4sleep3970399cyclictest21:28:033
216302890,0sleep00-21swapper/022:03:320
154272870,0sleep00-21swapper/022:31:530
161742860,2sleep116175-21bash21:21:031
127892860,1sleep10-21swapper/121:17:351
261812850,1sleep026179-21bash22:13:370
261812850,1sleep026179-21bash22:13:370
64552840,4sleep3970399cyclictest22:08:273
304112840,1sleep130403-21sshd21:48:131
283802840,4sleep20-21swapper/223:08:472
203112840,2sleep220310-21bash00:10:242
310022830,2sleep230999-21bash22:49:432
186702820,3sleep30-21swapper/322:19:183
170042820,3sleep20-21swapper/221:27:342
148952820,1sleep014894-21bash22:44:530
136912820,1sleep20-21swapper/221:43:212
127072820,1sleep20-21swapper/223:49:402
918428134,43sleep10-21swapper/119:06:461
48522810,0sleep00-21swapper/023:22:430
304152800,2sleep330408-21sshd00:01:063
218322770,0sleep11846-21nfsd00:07:071
218322770,0sleep11846-21nfsd00:07:061
216842760,3sleep30-21swapper/323:17:583
928427535,9sleep20-21swapper/219:08:042
288542750,1sleep041ktimersoftd/021:48:040
298522740,2sleep20-21swapper/223:35:042
278702710,3sleep0969199cyclictest23:54:260
304422620,0sleep00-21swapper/022:17:110
923526135,4sleep00-21swapper/019:07:250
10192550,2sleep1969799cyclictest23:19:121
115832510,2sleep30-21swapper/321:36:383
969799411,9cyclictest0-21swapper/122:58:001
216462410,39sleep326950irq/26-ahci[00023:50:253
9697993632,3cyclictest0-21swapper/121:54:061
9691993430,2cyclictest0-21swapper/023:41:120
183822340,31sleep3135650irq/27-eth023:04:363
969199330,3cyclictest3722-21sshd00:19:100
969199330,32cyclictest41ktimersoftd/000:14:210
969799320,12cyclictest0-21swapper/121:59:191
9691993214,16cyclictest3-21ksoftirqd/022:23:550
9691993113,16cyclictest3-21ksoftirqd/000:25:440
969799300,11cyclictest0-21swapper/123:40:231
9691993011,17cyclictest3-21ksoftirqd/023:11:510
969799290,11cyclictest0-21swapper/122:52:131
9691992915,12cyclictest3-21ksoftirqd/021:20:590
9691992914,10cyclictest3-21ksoftirqd/023:27:050
9691992913,12cyclictest3-21ksoftirqd/000:04:070
9691992911,12cyclictest3-21ksoftirqd/021:52:140
66672290,0sleep00-21swapper/023:45:370
969199288,9cyclictest3-21ksoftirqd/023:01:570
9691992814,6cyclictest3-21ksoftirqd/022:38:050
9691992813,12cyclictest3-21ksoftirqd/023:37:280
9691992812,8cyclictest3-21ksoftirqd/022:51:390
969199281,16cyclictest0-21swapper/000:20:010
9691992810,16cyclictest3-21ksoftirqd/021:16:010
317922280,23sleep1135650irq/27-eth023:58:061
235622280,3sleep30-21swapper/321:50:393
970399270,19cyclictest44-21ksoftirqd/322:49:243
9702992723,3cyclictest1847-21nfsd23:21:392
969199272,16cyclictest24506-21sshd21:25:120
9703992622,3cyclictest10963-21sshd21:55:523
970399260,19cyclictest44-21ksoftirqd/322:30:043
9697992613,2cyclictest10950irq/23-ehci_hcd23:09:181
969199265,3cyclictest3-21ksoftirqd/021:44:420
969199264,14cyclictest0-21swapper/023:09:590
9691992614,6cyclictest3-21ksoftirqd/022:06:450
72432260,2sleep32405-21runrttasks21:42:423
208982260,2sleep20-21swapper/200:31:322
125832260,2sleep20-21swapper/223:00:412
970399250,19cyclictest44-21ksoftirqd/323:32:303
969799252,15cyclictest15180-21sshd23:36:481
9691992512,6cyclictest3-21ksoftirqd/022:59:440
969199251,22cyclictest3-21ksoftirqd/021:39:270
9691992512,12cyclictest3-21ksoftirqd/022:47:480
9691992512,12cyclictest3-21ksoftirqd/000:05:050
9691992512,12cyclictest3-21ksoftirqd/000:05:050
9691992511,6cyclictest3-21ksoftirqd/000:31:420
969199251,11cyclictest23665-21sshd21:57:110
9691992510,13cyclictest3-21ksoftirqd/023:59:590
5792250,2sleep10-21swapper/100:32:591
295532250,2sleep129547-21bash23:48:021
200992250,22sleep1135650irq/27-eth000:10:231
970399244,10cyclictest29301-21sshd22:43:023
970399241,20cyclictest44-21ksoftirqd/322:04:273
9702992412,10cyclictest29665-21sshd21:16:002
9697992412,6cyclictest10950irq/23-ehci_hcd21:14:081
260712240,0sleep226061-21sshd00:18:062
256742240,22sleep226950irq/26-ahci[00022:29:482
198252240,21sleep1135650irq/27-eth022:00:081
192612240,2sleep20-21swapper/222:22:332
135482240,2sleep10-21swapper/123:04:061
9703992315,6cyclictest25-21rcuop/123:05:183
9703992315,5cyclictest0-21swapper/323:14:593
9703992314,6cyclictest22402-21sshd00:14:183
9703992314,6cyclictest1407-21sshd23:25:323
9703992312,4cyclictest1847-21nfsd22:23:063
970399231,13cyclictest15525-21bash22:38:243
9702992319,3cyclictest1936-21cp21:38:592
969199233,2cyclictest3-21ksoftirqd/022:26:110
9691992310,6cyclictest3-21ksoftirqd/021:10:080
157842230,1sleep30-21swapper/323:40:113
125622230,21sleep1135650irq/27-eth022:12:151
125622230,21sleep1135650irq/27-eth022:12:151
9703992216,4cyclictest1848-21nfsd23:36:403
9703992215,4cyclictest135650irq/27-eth023:59:093
9703992214,6cyclictest22588-21bash21:34:373
9703992214,6cyclictest1847-21nfsd23:20:523
9703992214,5cyclictest476-21rm00:15:213
9703992212,6cyclictest1848-21nfsd00:23:283
9703992210,6cyclictest135650irq/27-eth000:39:283
9702992216,5cyclictest3869-21sadc23:00:002
9702992214,6cyclictest22526-21sshd23:40:412
969799220,3cyclictest1848-21nfsd21:26:211
969199221,3cyclictest8-21rcu_preempt00:38:460
248922220,20sleep1135650irq/27-eth022:23:091
159032220,2sleep1417-21jbd2/dm-0-822:15:431
9703992115,4cyclictest1845-21nfsd22:56:303
9702992115,4cyclictest1902-21nscd21:59:472
9702992114,3cyclictest1848-21nfsd00:35:572
9697992110,2cyclictest21240-21sshd22:42:111
970399205,11cyclictest11205-21sshd21:11:053
9703992013,4cyclictest12972-21sshd21:24:033
970399200,11cyclictest0-21swapper/321:47:253
970299209,5cyclictest135650irq/27-eth022:43:412
9702992014,5cyclictest26412-21sshd00:28:332
9697992010,1cyclictest0-21swapper/123:22:461
970399194,6cyclictest1846-21nfsd22:54:253
9703991914,3cyclictest21980-21sshd00:07:083
9703991914,3cyclictest21980-21sshd00:07:073
9703991912,5cyclictest0-21swapper/322:13:103
9703991912,5cyclictest0-21swapper/322:13:103
970299199,3cyclictest10950irq/23-ehci_hcd22:08:552
970299193,2cyclictest18586-21kworker/2:022:11:212
970299193,2cyclictest18586-21kworker/2:022:11:212
9702991914,4cyclictest1846-21nfsd23:12:262
969799199,1cyclictest1280-21sshd22:07:551
969799198,5cyclictest10950irq/23-ehci_hcd21:33:071
9697991913,4cyclictest0-21swapper/123:30:151
969799190,4cyclictest29555-21bash22:26:471
969799190,13cyclictest1845-21nfsd00:04:561
9702991812,1cyclictest15830-21bash22:35:122
9702991811,5cyclictest15365-21sshd22:18:562
9702991810,5cyclictest23721-21sshd21:47:342
970299180,16cyclictest0-21swapper/223:15:222
969799189,3cyclictest10950irq/23-ehci_hcd00:26:471
969799180,15cyclictest32759-21bash22:36:501
970399174,6cyclictest10950irq/23-ehci_hcd19:10:133
970299179,7cyclictest25615-21bash00:07:312
970299179,7cyclictest25615-21bash00:07:312
9702991712,4cyclictest23196-21sshd21:21:492
9702991711,4cyclictest19003-21sshd22:03:142
9702991711,4cyclictest10715-21sshd23:33:062
9702991710,4cyclictest1711-21bash21:51:442
970299170,10cyclictest0-21swapper/221:14:512
969799170,15cyclictest0-21swapper/119:15:151
970399164,11cyclictest10950irq/23-ehci_hcd20:41:413
970399160,14cyclictest0-21swapper/320:50:113
970399158,5cyclictest9413-21fschecks_time20:20:113
970399150,13cyclictest12298-21sh19:15:133
970399150,13cyclictest0-21swapper/320:45:193
970299153,11cyclictest10950irq/23-ehci_hcd20:36:432
969799157,6cyclictest0-21swapper/119:30:161
9697991513,1cyclictest0-21swapper/119:24:481
9697991511,3cyclictest10950irq/23-ehci_hcd20:47:321
9691991511,3cyclictest0-21swapper/020:59:510
969199150,12cyclictest0-21swapper/020:15:400
970399149,4cyclictest10950irq/23-ehci_hcd20:27:243
970399149,4cyclictest10950irq/23-ehci_hcd20:27:243
970399143,10cyclictest10950irq/23-ehci_hcd20:32:383
970399143,10cyclictest10950irq/23-ehci_hcd20:15:033
9703991410,3cyclictest24722-21munin-node20:55:173
9703991410,3cyclictest10950irq/23-ehci_hcd20:12:103
970299140,2cyclictest0-21swapper/221:05:542
969799147,3cyclictest10950irq/23-ehci_hcd19:10:161
969799146,2cyclictest3237-21grep20:05:191
969799142,11cyclictest135650irq/27-eth021:03:111
969799142,11cyclictest0-21swapper/120:30:351
969199148,5cyclictest0-21swapper/020:32:300
969199147,6cyclictest0-21swapper/020:26:510
969199147,6cyclictest0-21swapper/020:26:510
9691991411,2cyclictest3-21ksoftirqd/021:10:020
969199140,6cyclictest0-21swapper/019:50:180
969199140,12cyclictest0-21swapper/020:35:180
969199140,11cyclictest0-21swapper/020:50:100
970399137,5cyclictest0-21swapper/319:51:003
970399131,8cyclictest31461-21timerwakeupswit19:55:213
970399131,11cyclictest10950irq/23-ehci_hcd20:07:213
9703991311,1cyclictest0-21swapper/319:26:473
970399130,5cyclictest0-21swapper/320:02:393
970399130,12cyclictest2405-21runrttasks19:46:543
970399130,12cyclictest1848-21nfsd19:40:193
970399130,12cyclictest0-21swapper/321:05:543
970399130,12cyclictest0-21swapper/321:00:173
970399130,12cyclictest0-21swapper/319:36:493
970299139,3cyclictest135650irq/27-eth020:30:242
970299138,3cyclictest26168-21munin-node19:45:152
970299137,5cyclictest135650irq/27-eth020:01:032
970299137,5cyclictest0-21swapper/219:28:122
970299135,5cyclictest14339-21atenall_r5power19:20:092
970299133,1cyclictest0-21swapper/219:17:202
970299132,10cyclictest135650irq/27-eth020:45:272
970299131,3cyclictest25498-21irqstats20:55:182
970299131,3cyclictest19882-21ntp_states19:30:192
9702991311,1cyclictest0-21swapper/220:29:112
9702991311,1cyclictest0-21swapper/220:29:112
9702991311,1cyclictest0-21swapper/219:50:432
970299130,5cyclictest31388-21xargs19:55:212
970299130,5cyclictest0-21swapper/220:07:312
970299130,12cyclictest0-21swapper/220:53:262
970299130,11cyclictest6669-21munin-node20:15:202
970299130,11cyclictest4746-21diskstats20:10:142
970299130,11cyclictest2405-21runrttasks20:20:142
970299130,11cyclictest19096-21ntp_states20:40:172
969799139,3cyclictest135650irq/27-eth020:55:381
969799138,3cyclictest19061-21nfsd420:40:171
969799137,5cyclictest10652-21runrttasks20:21:101
969799131,4cyclictest26950irq/26-ahci[00020:25:021
969799131,4cyclictest26950irq/26-ahci[00020:25:021
9697991311,1cyclictest0-21swapper/120:15:261
9697991311,1cyclictest0-21swapper/119:51:141
969799130,12cyclictest0-21swapper/121:05:471
969799130,12cyclictest0-21swapper/120:52:371
969799130,12cyclictest0-21swapper/120:35:341
969799130,12cyclictest0-21swapper/120:14:441
969799130,12cyclictest0-21swapper/119:35:161
969799130,12cyclictest0-21swapper/119:27:241
969799130,11cyclictest564-21iostat20:00:141
969799130,11cyclictest25499-21munin-node19:45:121
969199137,5cyclictest0-21swapper/019:31:190
969199131,5cyclictest0-21swapper/020:05:220
9691991311,1cyclictest0-21swapper/019:55:360
9691991311,1cyclictest0-21swapper/019:10:020
969199130,8cyclictest0-21swapper/019:45:150
969199130,1cyclictest0-21swapper/020:10:180
969199130,1cyclictest0-21swapper/019:41:210
969199130,12cyclictest0-21swapper/021:00:560
969199130,12cyclictest0-21swapper/020:00:220
969199130,11cyclictest21258-21df19:35:130
969199130,10cyclictest0-21swapper/020:40:540
969199130,10cyclictest0-21swapper/020:20:500
970399128,3cyclictest1848-21nfsd19:30:063
970399127,3cyclictest15394-21munin-node19:20:163
970399120,10cyclictest15863-21munin-node20:35:103
970299120,11cyclictest10479-21iostat19:10:142
970299120,11cyclictest0-21swapper/221:00:042
970299120,11cyclictest0-21swapper/219:40:182
970299120,10cyclictest0-21swapper/219:35:132
969799120,11cyclictest24500-21munin-node19:40:181
969799120,10cyclictest30337-21diskstats19:55:121
969199127,3cyclictest1131-21jbd2/dm-2-819:25:040
969199126,5cyclictest0-21swapper/019:20:570
969199120,1cyclictest0-21swapper/020:45:420
969199120,11cyclictest0-21swapper/019:15:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional