You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-08 - 11:27
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Sun Sep 08, 2024 00:50:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
444629389,2sleep40-21swapper/419:05:204
533529261,26sleep10-21swapper/119:07:001
524028653,29sleep70-21swapper/719:05:397
544428551,30sleep20-21swapper/219:08:312
537228349,29sleep60-21swapper/619:07:306
544528167,9sleep30-21swapper/319:08:323
529427844,30sleep50-21swapper/519:06:255
543327642,29sleep00-21swapper/019:08:210
94092510,0sleep10-21swapper/121:22:311
198012480,0sleep50-21swapper/500:30:515
34242450,0sleep00-21swapper/022:05:140
146652440,0sleep20-21swapper/200:13:432
265602430,0sleep341-21ksoftirqd/323:48:303
5978992417,5cyclictest9-21ksoftirqd/019:10:010
5983992210,9cyclictest0-21swapper/500:27:195
5983992110,5cyclictest0-21swapper/523:11:185
5983992010,9cyclictest0-21swapper/523:07:585
5983992010,6cyclictest0-21swapper/523:02:145
5983992010,6cyclictest0-21swapper/522:36:045
5983991910,9cyclictest0-21swapper/522:30:145
5983991910,9cyclictest0-21swapper/522:30:145
5983991910,9cyclictest0-21swapper/521:31:315
5983991910,8cyclictest0-21swapper/523:38:535
5983991910,8cyclictest0-21swapper/522:45:565
5983991910,8cyclictest0-21swapper/522:20:065
5983991910,8cyclictest0-21swapper/521:41:255
5983991910,8cyclictest0-21swapper/521:39:145
5983991910,8cyclictest0-21swapper/521:39:145
5983991910,8cyclictest0-21swapper/521:21:445
5983991910,8cyclictest0-21swapper/519:15:185
5983991910,8cyclictest0-21swapper/500:35:315
5983991910,8cyclictest0-21swapper/500:06:365
5983991910,6cyclictest0-21swapper/522:58:595
5983991910,6cyclictest0-21swapper/522:13:265
5983991910,6cyclictest0-21swapper/522:01:035
5983991910,6cyclictest0-21swapper/521:27:565
5983991810,8cyclictest0-21swapper/523:31:105
5983991810,7cyclictest1560-21runrttasks22:05:465
5983991810,7cyclictest0-21swapper/523:55:015
5983991810,7cyclictest0-21swapper/523:45:025
5983991810,7cyclictest0-21swapper/523:40:095
5983991810,7cyclictest0-21swapper/523:25:035
5983991810,7cyclictest0-21swapper/523:20:025
5983991810,7cyclictest0-21swapper/523:15:025
5983991810,7cyclictest0-21swapper/522:50:045
5983991810,7cyclictest0-21swapper/522:40:045
5983991810,7cyclictest0-21swapper/522:25:015
5983991810,7cyclictest0-21swapper/522:15:205
5983991810,7cyclictest0-21swapper/521:55:095
5983991810,7cyclictest0-21swapper/521:50:035
5983991810,7cyclictest0-21swapper/521:45:035
5983991810,7cyclictest0-21swapper/521:10:045
5983991810,7cyclictest0-21swapper/521:06:355
5983991810,7cyclictest0-21swapper/521:00:115
5983991810,7cyclictest0-21swapper/520:50:115
5983991810,7cyclictest0-21swapper/520:40:225
5983991810,7cyclictest0-21swapper/520:35:185
5983991810,7cyclictest0-21swapper/520:30:165
5983991810,7cyclictest0-21swapper/520:25:035
5983991810,7cyclictest0-21swapper/520:15:175
5983991810,7cyclictest0-21swapper/520:10:015
5983991810,7cyclictest0-21swapper/520:05:335
5983991810,7cyclictest0-21swapper/520:00:125
5983991810,7cyclictest0-21swapper/519:55:015
5983991810,7cyclictest0-21swapper/519:50:105
5983991810,7cyclictest0-21swapper/519:50:105
5983991810,7cyclictest0-21swapper/519:45:145
5983991810,7cyclictest0-21swapper/519:40:125
5983991810,7cyclictest0-21swapper/519:35:025
5983991810,7cyclictest0-21swapper/519:20:015
5983991810,7cyclictest0-21swapper/500:20:015
5983991810,7cyclictest0-21swapper/500:15:005
5983991810,7cyclictest0-21swapper/500:10:035
5983991810,7cyclictest0-21swapper/500:00:045
5978991816,1cyclictest101ktimersoftd/000:08:100
5978991811,3cyclictest121rcu_preempt23:00:010
5983991710,7cyclictest0-21swapper/521:15:015
5983991710,7cyclictest0-21swapper/519:25:095
5983991710,6cyclictest0-21swapper/523:50:005
5983991710,6cyclictest0-21swapper/520:55:145
5983991710,6cyclictest0-21swapper/520:45:015
5983991710,6cyclictest0-21swapper/520:20:025
5983991710,6cyclictest0-21swapper/519:30:025
5983991710,6cyclictest0-21swapper/519:10:015
5982991715,1cyclictest5882sleep419:50:274
5982991715,1cyclictest5882sleep419:50:274
5979991715,1cyclictest5928-21ssh22:06:131
5978991716,1cyclictest101ktimersoftd/022:02:500
5978991715,1cyclictest101ktimersoftd/021:45:140
5982991615,1cyclictest2734-21ssh00:07:514
5982991615,1cyclictest25242-21vmstat22:30:204
5982991615,1cyclictest25242-21vmstat22:30:204
5982991615,1cyclictest12928-21ssh21:55:014
5979991616,0cyclictest0-21swapper/120:00:121
5978991615,1cyclictest101ktimersoftd/023:25:420
5978991615,1cyclictest101ktimersoftd/023:00:180
5978991615,1cyclictest101ktimersoftd/021:52:420
5978991615,1cyclictest101ktimersoftd/000:16:500
5978991615,1cyclictest101ktimersoftd/000:13:340
5978991614,1cyclictest101ktimersoftd/023:57:540
5978991614,1cyclictest101ktimersoftd/022:13:570
5978991614,1cyclictest101ktimersoftd/021:20:140
5978991614,1cyclictest101ktimersoftd/000:30:180
5978991614,1cyclictest101ktimersoftd/000:00:140
1737121610,3sleep2121rcu_preempt22:27:032
5985991515,0cyclictest0-21swapper/723:39:487
5985991515,0cyclictest0-21swapper/722:52:057
5985991510,5cyclictest0-21swapper/700:03:307
5984991514,1cyclictest11371-21ssh22:24:546
5984991510,5cyclictest0-21swapper/623:09:456
5984991510,0cyclictest0-21swapper/623:34:376
5982991510,5cyclictest0-21swapper/423:30:284
5981991514,1cyclictest26425-21ssh22:15:393
5981991513,1cyclictest0-21swapper/300:20:153
5980991514,1cyclictest24964-21ls22:30:192
5980991514,1cyclictest24964-21ls22:30:182
5980991513,1cyclictest13475-21diskmemload22:42:162
5979991515,0cyclictest0-21swapper/100:15:401
5979991514,1cyclictest1935-21ssh21:34:491
5979991513,1cyclictest13475-21diskmemload00:39:091
597899159,3cyclictest121rcu_preempt00:25:200
5978991514,1cyclictest20730-21ssh23:15:120
5978991514,1cyclictest101ktimersoftd/023:50:500
5978991514,1cyclictest101ktimersoftd/023:45:340
5978991514,1cyclictest101ktimersoftd/023:12:300
5978991514,1cyclictest101ktimersoftd/022:45:170
5978991514,1cyclictest101ktimersoftd/022:41:020
5978991514,1cyclictest101ktimersoftd/022:30:300
5978991514,1cyclictest101ktimersoftd/022:30:300
5978991514,1cyclictest101ktimersoftd/021:25:060
5978991514,1cyclictest101ktimersoftd/000:36:500
5978991514,0cyclictest32287-21tune2fs23:20:130
5978991513,1cyclictest9-21ksoftirqd/020:35:000
5978991513,1cyclictest133992sleep021:09:090
5978991513,1cyclictest101ktimersoftd/023:07:300
5978991513,1cyclictest101ktimersoftd/021:34:380
5978991513,1cyclictest101ktimersoftd/019:20:140
5978991513,1cyclictest101ktimersoftd/000:23:180
5978991511,3cyclictest121rcu_preempt22:19:080
5978991511,1cyclictest131rcu_sched19:50:200
5978991511,1cyclictest131rcu_sched19:50:200
5978991510,3cyclictest9-21ksoftirqd/021:35:110
5978991510,3cyclictest9-21ksoftirqd/021:35:110
520521510,4sleep10-21swapper/122:36:311
2353321510,3sleep3121rcu_preempt00:33:143
5985991414,0cyclictest0-21swapper/721:58:387
5985991414,0cyclictest0-21swapper/700:12:357
5985991413,1cyclictest727-21gltestperf23:05:137
5985991410,3cyclictest121rcu_preempt22:02:207
5985991410,1cyclictest0-21swapper/723:46:007
5985991410,0cyclictest0-21swapper/721:42:087
5984991410,4cyclictest0-21swapper/622:08:416
5982991413,1cyclictest9937-21ssh21:22:524
5982991413,1cyclictest26100-21ssh00:19:194
5982991413,1cyclictest23951-21ssh00:33:294
5982991412,1cyclictest4619-21sh22:36:074
5982991412,1cyclictest153492sleep400:14:094
5981991414,0cyclictest4399-21sshd20:54:013
5981991413,1cyclictest9254-21ssh22:39:023
5981991413,1cyclictest625-21rpcbind23:17:123
5981991413,1cyclictest20435-21ssh22:13:403
5981991413,1cyclictest19500-21ssh21:57:433
5981991413,1cyclictest15526-21ssh22:41:113
5981991413,1cyclictest13475-21diskmemload23:31:093
5981991412,1cyclictest641-21nscd23:43:223
5981991412,1cyclictest13475-21diskmemload21:22:473
5980991413,1cyclictest18000-21awk19:25:202
5980991413,1cyclictest16487-21ssh22:11:112
5980991412,1cyclictest13475-21diskmemload21:52:012
5979991414,0cyclictest0-21swapper/100:28:491
5979991413,1cyclictest20478-21ssh23:30:141
5979991412,1cyclictest4816-21kworker/u16:322:44:311
5979991410,0cyclictest0-21swapper/121:17:381
597899149,1cyclictest9431-21df_inode19:15:020
597899148,3cyclictest121rcu_preempt23:30:400
5978991414,0cyclictest101ktimersoftd/022:25:100
5978991413,1cyclictest101ktimersoftd/023:40:180
5978991413,1cyclictest101ktimersoftd/022:50:140
5978991413,1cyclictest101ktimersoftd/022:35:100
5978991413,1cyclictest101ktimersoftd/021:57:180
5978991413,1cyclictest101ktimersoftd/021:41:380
5978991413,1cyclictest101ktimersoftd/019:45:180
5978991413,1cyclictest101ktimersoftd/019:40:460
5978991413,1cyclictest101ktimersoftd/019:35:130
5978991413,0cyclictest7037-21ssh22:22:110
5978991412,1cyclictest9-21ksoftirqd/021:20:010
5978991410,1cyclictest9-21ksoftirqd/023:37:000
3241721410,3sleep70-21swapper/721:18:227
90922130,0sleep60-21swapper/621:22:206
5985991312,1cyclictest648-21nscd21:31:237
5985991312,1cyclictest641-21nscd23:22:167
5985991312,1cyclictest1706-21sendmail-mta23:53:417
5985991312,1cyclictest14138-21kworker/u16:123:00:087
5985991312,1cyclictest13475-21diskmemload21:27:417
5985991312,1cyclictest10155-21kworker/u16:200:16:187
5985991311,1cyclictest6436-21diskmemload21:35:587
5985991311,1cyclictest6436-21diskmemload21:35:587
5985991311,1cyclictest18846-21timerwakeupswit00:30:237
5985991311,0cyclictest121rcu_preempt00:28:027
5985991310,3cyclictest0-21swapper/723:57:277
5985991310,3cyclictest0-21swapper/723:40:187
5985991310,3cyclictest0-21swapper/723:34:497
5985991310,3cyclictest0-21swapper/723:16:087
5985991310,3cyclictest0-21swapper/722:46:277
5985991310,3cyclictest0-21swapper/722:25:137
5985991310,3cyclictest0-21swapper/722:21:107
5985991310,3cyclictest0-21swapper/722:15:197
5985991310,3cyclictest0-21swapper/722:09:297
5985991310,3cyclictest0-21swapper/721:50:307
5985991310,3cyclictest0-21swapper/721:46:047
5985991310,3cyclictest0-21swapper/721:06:567
5985991310,3cyclictest0-21swapper/721:01:097
5985991310,3cyclictest0-21swapper/720:49:197
5985991310,3cyclictest0-21swapper/720:44:197
5985991310,3cyclictest0-21swapper/720:35:117
5985991310,3cyclictest0-21swapper/720:26:347
5985991310,3cyclictest0-21swapper/720:16:587
5985991310,3cyclictest0-21swapper/720:14:327
5985991310,3cyclictest0-21swapper/719:50:417
5985991310,3cyclictest0-21swapper/719:50:417
5985991310,3cyclictest0-21swapper/719:45:477
5985991310,3cyclictest0-21swapper/719:40:167
5985991310,3cyclictest0-21swapper/719:35:317
5985991310,3cyclictest0-21swapper/719:32:047
5985991310,3cyclictest0-21swapper/719:29:537
5985991310,3cyclictest0-21swapper/719:20:007
5985991310,3cyclictest0-21swapper/719:18:067
5985991310,3cyclictest0-21swapper/719:11:287
5985991310,3cyclictest0-21swapper/700:37:567
5985991310,3cyclictest0-21swapper/700:07:117
5985991310,1cyclictest121rcu_preempt21:13:397
5985991310,0cyclictest0-21swapper/723:28:017
5984991313,0cyclictest0-21swapper/600:18:286
5984991312,1cyclictest4816-21kworker/u16:300:06:206
5984991312,1cyclictest31012-1kworker/u17:122:36:186
5984991312,1cyclictest30545-1kworker/u17:123:04:036
5984991312,1cyclictest14138-21kworker/u16:121:35:286
5984991312,1cyclictest14138-21kworker/u16:121:35:286
5984991312,1cyclictest13475-21diskmemload21:54:336
5984991311,1cyclictest641-21nscd21:55:286
5984991311,1cyclictest23126-21apt-key22:15:016
5984991310,3cyclictest0-21swapper/620:25:446
5984991310,3cyclictest0-21swapper/619:51:426
5984991310,3cyclictest0-21swapper/619:51:416
5982991312,1cyclictest9000-21ssh22:23:244
5982991312,1cyclictest69450irq/124-eth023:44:454
5982991312,1cyclictest69450irq/124-eth023:25:354
5982991312,1cyclictest69450irq/124-eth022:16:304
5982991312,1cyclictest30928-21sh23:04:434
5982991312,1cyclictest30100-21ssh00:35:434
5982991312,1cyclictest28533-21ssh21:31:114
5982991311,1cyclictest8498-21entropy23:55:144
5982991311,1cyclictest21912-21latency_hist22:45:004
5982991311,1cyclictest12810-21expr22:25:154
5981991313,0cyclictest0-21swapper/323:11:243
5981991312,1cyclictest9805-21ssh21:38:043
5981991312,1cyclictest9805-21ssh21:38:043
5981991312,1cyclictest26909-21ssh21:45:293
5981991312,1cyclictest22178-21ssh00:01:173
5981991312,1cyclictest13475-21diskmemload22:23:593
5981991312,1cyclictest13475-21diskmemload21:28:103
5981991312,1cyclictest13475-21diskmemload21:18:363
5981991312,1cyclictest13475-21diskmemload21:12:353
5981991312,1cyclictest11452-1kworker/u17:200:07:243
5981991311,1cyclictest649-21nscd23:50:143
5981991311,1cyclictest641-21nscd23:35:213
5981991311,1cyclictest5187-21rm23:22:473
5981991311,1cyclictest31754-21rm21:33:123
5981991311,1cyclictest14138-21kworker/u16:121:51:083
5981991311,1cyclictest13475-21diskmemload22:55:553
5981991311,1cyclictest13475-21diskmemload22:32:163
5981991311,1cyclictest13475-21diskmemload22:32:163
5981991310,3cyclictest0-21swapper/319:41:203
5981991310,3cyclictest0-21swapper/319:10:263
5981991310,1cyclictest0-21swapper/322:06:393
5980991313,0cyclictest0-21swapper/221:26:412
5980991312,1cyclictest648-21nscd00:06:302
5980991312,1cyclictest6039-21ssh23:38:492
5980991312,1cyclictest25807-21ssh23:32:282
5980991312,1cyclictest2485-21ssh23:21:042
5980991312,1cyclictest1580-21ssh00:38:122
5980991312,1cyclictest15239-21ssh23:27:272
5980991312,1cyclictest14138-21kworker/u16:121:11:562
5980991311,1cyclictest641-21nscd21:56:542
5979991313,0cyclictest0-21swapper/123:35:341
5979991313,0cyclictest0-21swapper/123:07:221
5979991313,0cyclictest0-21swapper/100:00:501
5979991312,1cyclictest649-21nscd00:31:031
5979991312,1cyclictest641-21nscd23:50:181
5979991312,1cyclictest192-21md0_raid120:43:401
5979991312,1cyclictest13475-21diskmemload23:23:461
5979991311,1cyclictest9690-21processes19:15:021
5979991311,1cyclictest641-21nscd23:40:351
5979991311,1cyclictest641-21nscd23:00:571
5979991311,1cyclictest641-21nscd21:59:001
5979991311,1cyclictest641-21nscd21:10:391
5979991311,1cyclictest641-21nscd00:12:481
5979991311,1cyclictest610-21rm00:22:031
5979991311,1cyclictest28152-21ssh22:16:451
5979991311,1cyclictest24618-21rm22:00:131
5979991310,3cyclictest0-21swapper/121:01:211
5979991310,3cyclictest0-21swapper/120:53:251
5979991310,3cyclictest0-21swapper/120:09:341
5979991310,3cyclictest0-21swapper/119:45:011
5979991310,3cyclictest0-21swapper/119:38:141
5979991310,3cyclictest0-21swapper/119:25:361
5979991310,0cyclictest0-21swapper/121:38:061
5979991310,0cyclictest0-21swapper/121:38:061
597899139,3cyclictest9-21ksoftirqd/020:05:120
597899139,1cyclictest9-21ksoftirqd/019:55:120
597899139,1cyclictest121rcu_preempt20:45:130
597899139,1cyclictest121rcu_preempt20:25:170
597899139,1cyclictest121rcu_preempt19:30:110
5978991312,1cyclictest101ktimersoftd/020:20:140
5978991312,1cyclictest101ktimersoftd/020:10:180
5978991312,1cyclictest101ktimersoftd/019:26:540
5978991312,0cyclictest6688-21taskset20:00:370
5978991311,1cyclictest5518-21df_abs20:55:130
5978991311,1cyclictest29957-21tr20:40:200
5978991311,1cyclictest15148-21fschecks_count21:10:120
5978991311,1cyclictest0-21swapper/020:50:140
3009921310,2sleep60-21swapper/622:48:456
2984521310,2sleep60-21swapper/621:47:206
2738821310,2sleep40-21swapper/419:44:524
1070721310,2sleep10-21swapper/122:24:291
5985991211,1cyclictest7911-21kworker/7:020:05:317
5985991211,1cyclictest5976-21cyclictest22:56:197
5985991211,1cyclictest5976-21cyclictest20:50:417
5985991211,1cyclictest5976-21cyclictest20:21:297
5985991211,1cyclictest4857-21/usr/sbin/munin20:55:207
5985991211,1cyclictest4816-21kworker/u16:322:30:017
5985991211,1cyclictest4816-21kworker/u16:322:30:017
5985991211,1cyclictest2938-21ntp_states19:55:177
5985991211,1cyclictest22837-21cstates20:30:117
5985991211,1cyclictest11497-21sh22:40:047
5985991210,2cyclictest0-21swapper/722:10:277
5985991210,2cyclictest0-21swapper/720:01:047
5985991210,1cyclictest662-21ptp4l22:35:027
5985991210,1cyclictest3095-21ssh21:20:017
5985991210,1cyclictest1706-21sendmail-mta00:20:017
5985991210,1cyclictest11255-21gltestperf23:10:137
5984991212,0cyclictest0-21swapper/619:16:566
5984991211,1cyclictest9645-1kworker/u17:222:55:186
5984991211,1cyclictest7660-21wc19:10:186
5984991211,1cyclictest662-21ptp4l19:45:486
5984991211,1cyclictest641-21nscd23:56:046
5984991211,1cyclictest641-21nscd23:20:196
5984991211,1cyclictest641-21nscd21:25:086
5984991211,1cyclictest641-21nscd21:15:166
5984991211,1cyclictest630-21avahi-daemon20:10:516
5984991211,1cyclictest5976-21cyclictest21:00:076
5984991211,1cyclictest5976-21cyclictest20:56:026
5984991211,1cyclictest5976-21cyclictest20:40:316
5984991211,1cyclictest5976-21cyclictest20:37:166
5984991211,1cyclictest5976-21cyclictest20:18:056
5984991211,1cyclictest5976-21cyclictest20:00:396
5984991211,1cyclictest5976-21cyclictest19:38:576
5984991211,1cyclictest5976-21cyclictest19:25:156
5984991211,1cyclictest28634-21strings00:35:176
5984991211,1cyclictest272-21jbd2/md0-819:55:326
5984991211,1cyclictest24862-21cpuspeed_turbos21:30:106
5984991211,1cyclictest23262-21fschecks_time20:30:126
5984991211,1cyclictest2054-21strings22:50:196
5984991211,1cyclictest16856-21cstates20:20:126
5984991211,1cyclictest16356-21sh00:30:056
5984991211,1cyclictest15195-21grep21:40:176
5984991211,1cyclictest14965-21sh22:25:316
5984991211,1cyclictest14636-21ssh22:40:376
5984991211,1cyclictest14138-21kworker/u16:123:15:326
5984991211,1cyclictest13475-21diskmemload23:50:296
5984991211,1cyclictest12752-21switchtime21:05:206
5984991211,1cyclictest12736-21cpuspeed_turbos22:10:116
5984991211,1cyclictest12128-21kworker/u16:023:25:036
5984991211,1cyclictest11452-1kworker/u17:200:11:296
5984991211,1cyclictest0-21swapper/620:50:196
5984991210,2cyclictest0-21swapper/623:45:236
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional