You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-25 - 06:15
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3.osadl.org (updated Sun Feb 25, 2024 00:50:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1443229964,31sleep10-21swapper/119:08:311
1434528844,39sleep60-21swapper/619:07:156
1431328752,30sleep30-21swapper/319:06:483
1442928348,30sleep70-21swapper/719:08:297
1435728045,30sleep00-21swapper/019:07:250
1442727945,29sleep50-21swapper/519:08:275
1436127843,30sleep40-21swapper/419:07:294
1435027843,30sleep20-21swapper/219:07:192
112452710,0sleep10-21swapper/123:31:151
220672560,0sleep40-21swapper/421:45:344
91442540,0sleep10-21swapper/121:57:081
96112510,0sleep40-21swapper/423:30:304
267652500,0sleep10-21swapper/123:21:381
171432490,0sleep10-21swapper/123:15:531
160792460,0sleep30-21swapper/321:24:433
61342430,0sleep60-21swapper/623:10:236
14967991912,6cyclictest682-21nscd23:47:092
14969991716,1cyclictest19938-21ssh23:18:104
14966991710,0cyclictest0-21swapper/122:15:341
51692160,0sleep30-21swapper/323:46:313
14972991610,6cyclictest0-21swapper/723:30:257
1497299160,15cyclictest0-21swapper/700:20:217
948821510,4sleep10-21swapper/123:11:471
56362150,0sleep60-21swapper/622:14:266
282162150,0sleep10-21swapper/120:25:001
14972991510,5cyclictest0-21swapper/723:00:157
14972991510,5cyclictest0-21swapper/722:08:287
14972991510,5cyclictest0-21swapper/700:10:367
14972991510,5cyclictest0-21swapper/700:06:587
14972991510,0cyclictest0-21swapper/722:35:187
14972991510,0cyclictest0-21swapper/722:17:397
14971991514,1cyclictest24651-21ssh00:16:566
14971991510,0cyclictest0-21swapper/600:30:116
14967991510,5cyclictest0-21swapper/221:27:282
14966991514,1cyclictest29304-21ssh21:31:191
14966991510,5cyclictest0-21swapper/123:54:541
14966991510,5cyclictest0-21swapper/123:41:191
14966991510,5cyclictest0-21swapper/122:32:181
14966991510,5cyclictest0-21swapper/122:06:311
14966991510,5cyclictest0-21swapper/100:21:571
14966991510,5cyclictest0-21swapper/100:17:021
14966991510,0cyclictest0-21swapper/121:46:161
506121410,3sleep50-21swapper/523:46:265
2187621410,3sleep60-21swapper/621:05:336
14972991414,0cyclictest0-21swapper/723:05:027
14972991414,0cyclictest0-21swapper/700:27:327
14971991413,1cyclictest30908-21ssh22:46:556
14971991412,1cyclictest22675-21diskmemload23:58:416
14970991414,0cyclictest0-21swapper/523:50:565
14970991414,0cyclictest0-21swapper/523:01:265
14970991410,4cyclictest0-21swapper/522:45:355
14970991410,4cyclictest0-21swapper/522:37:135
14970991410,0cyclictest0-21swapper/523:19:495
14970991410,0cyclictest0-21swapper/521:18:075
14970991410,0cyclictest0-21swapper/521:10:475
14970991410,0cyclictest0-21swapper/500:22:035
14969991413,1cyclictest20288-21ssh22:04:064
14969991413,1cyclictest20288-21ssh22:04:054
14969991412,1cyclictest31615-21ssh21:51:064
14969991412,1cyclictest22675-21diskmemload21:18:014
14968991414,0cyclictest0-21swapper/323:50:453
14968991414,0cyclictest0-21swapper/323:40:573
14968991414,0cyclictest0-21swapper/322:43:503
14968991410,4cyclictest0-21swapper/321:44:003
14968991410,4cyclictest0-21swapper/300:14:083
14968991410,0cyclictest0-21swapper/323:18:553
14968991410,0cyclictest0-21swapper/321:28:143
14967991414,0cyclictest0-21swapper/223:30:472
14967991414,0cyclictest0-21swapper/222:40:332
14967991414,0cyclictest0-21swapper/221:19:332
14967991410,4cyclictest0-21swapper/221:57:022
14967991410,4cyclictest0-21swapper/221:47:562
14967991410,4cyclictest0-21swapper/200:23:122
14966991414,0cyclictest0-21swapper/122:44:071
14966991414,0cyclictest0-21swapper/100:10:531
14966991410,4cyclictest0-21swapper/122:26:211
14966991410,4cyclictest0-21swapper/119:55:291
14966991410,0cyclictest0-21swapper/123:48:441
14966991410,0cyclictest0-21swapper/122:57:221
14966991410,0cyclictest0-21swapper/100:01:031
14965991412,1cyclictest1567-21grep19:40:220
14965991410,0cyclictest0-21swapper/000:13:190
813421310,2sleep40-21swapper/423:48:574
3195921310,2sleep30-21swapper/322:47:463
3094321310,2sleep30-21swapper/320:27:273
2886521310,2sleep00-21swapper/022:27:090
2575221310,2sleep60-21swapper/623:20:486
247692130,0sleep00-21swapper/022:43:590
14972991313,0cyclictest0-21swapper/722:14:057
14972991312,1cyclictest682-21nscd23:22:077
14972991312,1cyclictest682-21nscd21:59:187
14972991312,1cyclictest29845-21nscd23:50:277
14972991311,1cyclictest704-21ptp4l20:03:397
14972991311,1cyclictest682-21nscd22:55:317
14972991311,1cyclictest682-21nscd22:50:287
14972991311,1cyclictest682-21nscd20:05:207
14972991311,1cyclictest28598-21idleruntime00:00:307
14972991311,1cyclictest2504-21ssh21:15:467
14972991311,1cyclictest24194-21rm22:25:097
14972991310,3cyclictest0-21swapper/720:38:147
14972991310,3cyclictest0-21swapper/720:29:427
14972991310,3cyclictest0-21swapper/719:53:597
14971991312,1cyclictest893-21kworker/u16:122:06:216
14971991312,1cyclictest893-21kworker/u16:121:41:266
14971991312,1cyclictest24709-21ssh21:47:416
14971991312,1cyclictest1871-21ssh22:30:256
14971991312,1cyclictest13674-21ssh23:51:236
14971991311,1cyclictest8110-21cpuspeed_turbos23:30:236
14971991311,1cyclictest641ktimersoftd/622:55:016
14971991311,1cyclictest18850-21cpu19:15:226
14971991311,1cyclictest18850-21cpu19:15:216
14971991311,1cyclictest15096-21processes20:00:296
14971991311,1cyclictest11519-21timerandwakeup22:35:286
14970991313,0cyclictest0-21swapper/522:56:335
14970991313,0cyclictest0-21swapper/521:53:485
14970991313,0cyclictest0-21swapper/500:35:385
14970991312,1cyclictest73550irq/125-eth023:32:245
14970991312,1cyclictest73550irq/125-eth022:25:545
14970991312,1cyclictest73550irq/125-eth022:00:205
14970991312,1cyclictest73550irq/125-eth022:00:205
14970991312,1cyclictest73550irq/125-eth021:56:035
14970991312,1cyclictest73550irq/125-eth000:29:575
14970991311,2cyclictest73550irq/125-eth021:28:045
14970991311,2cyclictest1752-21ntpd22:10:205
14970991311,1cyclictest8746-21latency_hist22:35:005
14970991311,1cyclictest73550irq/125-eth023:13:035
14970991311,1cyclictest73550irq/125-eth000:16:545
14970991311,1cyclictest682-21nscd23:39:565
14970991311,1cyclictest682-21nscd00:34:155
14970991311,1cyclictest31775-21sh23:43:505
14970991311,1cyclictest22913-21entropy22:05:225
14970991311,1cyclictest17717-21diskmemload22:40:015
14970991311,1cyclictest1752-21ntpd22:50:295
14970991311,1cyclictest1752-21ntpd21:30:305
14970991311,1cyclictest1752-21ntpd20:40:325
14970991311,1cyclictest1752-21ntpd19:25:255
14970991311,1cyclictest1624-21runrttasks19:15:235
14970991311,1cyclictest1624-21runrttasks19:15:225
14970991310,2cyclictest692-21nscd23:08:565
14970991310,0cyclictest0-21swapper/520:25:225
14970991310,0cyclictest0-21swapper/520:22:315
14969991312,1cyclictest6265-21ssh23:29:164
14969991312,1cyclictest31003-21kworker/u16:223:14:164
14969991312,1cyclictest28862-21ssh00:20:074
14969991312,1cyclictest23489-21kworker/u16:100:12:414
14969991312,1cyclictest21500-1kworker/u17:122:49:384
14969991312,1cyclictest17969-21munin-node21:05:284
14969991312,1cyclictest14863-21ssh22:38:014
14969991312,1cyclictest14075-21ssh21:41:094
14969991312,0cyclictest0-21swapper/422:50:234
14969991311,1cyclictest7556-21ssh21:37:574
14969991311,1cyclictest32117-21kernelversion22:10:244
14969991311,1cyclictest20481-21nscd22:15:504
14969991310,2cyclictest0-21swapper/419:45:014
14969991310,0cyclictest0-21swapper/400:05:224
14968991313,0cyclictest0-21swapper/300:17:473
14968991312,1cyclictest31003-21kworker/u16:222:59:273
14968991312,1cyclictest17969-21munin-node22:32:063
14968991311,1cyclictest6954-21ssh21:55:303
14968991311,1cyclictest6900-21sh00:06:153
14968991311,1cyclictest682-21nscd22:23:353
14968991311,1cyclictest682-21nscd22:13:403
14968991311,1cyclictest22675-21diskmemload21:35:153
14968991311,1cyclictest1390-21sh23:25:323
14968991310,3cyclictest0-21swapper/321:10:293
14968991310,3cyclictest0-21swapper/321:09:203
14968991310,3cyclictest0-21swapper/320:44:293
14968991310,3cyclictest0-21swapper/320:23:243
14968991310,3cyclictest0-21swapper/320:05:143
14968991310,3cyclictest0-21swapper/319:56:523
14968991310,3cyclictest0-21swapper/300:37:363
14968991310,0cyclictest0-21swapper/323:22:023
14968991310,0cyclictest0-21swapper/322:35:383
14968991310,0cyclictest0-21swapper/320:59:383
14967991313,0cyclictest0-21swapper/222:56:202
14967991313,0cyclictest0-21swapper/222:36:552
14967991313,0cyclictest0-21swapper/222:02:402
14967991313,0cyclictest0-21swapper/222:02:402
14967991312,1cyclictest24387-21rm22:25:192
14967991312,1cyclictest23334-21kworker/u16:321:31:302
14967991312,1cyclictest201-21md0_raid121:50:322
14967991312,0cyclictest22675-21diskmemload23:35:532
14967991311,1cyclictest8596-21sh22:53:082
14967991311,1cyclictest7468-21sendmail_mailtr23:10:312
14967991311,1cyclictest682-21nscd00:29:132
14967991311,1cyclictest25132-21diskmemload23:40:132
14967991311,1cyclictest22675-21diskmemload21:44:482
14967991310,3cyclictest0-21swapper/219:48:052
14967991310,3cyclictest0-21swapper/219:18:442
14967991310,3cyclictest0-21swapper/219:18:442
14967991310,0cyclictest0-21swapper/221:02:182
14967991310,0cyclictest0-21swapper/220:38:252
14967991310,0cyclictest0-21swapper/219:51:072
14967991310,0cyclictest0-21swapper/219:26:352
14966991313,0cyclictest0-21swapper/121:21:311
14966991313,0cyclictest0-21swapper/121:10:281
14966991312,1cyclictest1621-21snmpd23:05:081
14966991311,1cyclictest682-21nscd22:10:101
14966991311,1cyclictest17110-21sh22:01:311
14966991311,1cyclictest17110-21sh22:01:311
14966991311,1cyclictest0-21swapper/120:25:301
14966991310,3cyclictest0-21swapper/100:26:501
14966991310,1cyclictest0-21swapper/123:37:431
14966991310,1cyclictest0-21swapper/122:21:511
14966991310,0cyclictest0-21swapper/100:33:491
14965991312,1cyclictest18532-21kworker/u16:000:39:250
14965991311,2cyclictest201-21md0_raid119:45:330
14965991311,1cyclictest704-21ptp4l23:50:140
14965991311,1cyclictest682-21nscd22:57:000
14965991311,1cyclictest682-21nscd22:32:530
14965991311,1cyclictest682-21nscd22:17:200
14965991311,1cyclictest682-21nscd21:28:410
14965991311,1cyclictest30225-21pmu-power21:50:300
14965991310,3cyclictest0-21swapper/023:32:230
14965991310,3cyclictest0-21swapper/019:12:170
14965991310,0cyclictest0-21swapper/023:27:160
14965991310,0cyclictest0-21swapper/021:35:440
14965991310,0cyclictest0-21swapper/021:13:440
14965991310,0cyclictest0-21swapper/020:50:260
14972991211,1cyclictest9239-1kworker/u17:221:52:087
14972991211,1cyclictest692-21nscd22:40:387
14972991211,1cyclictest682-21nscd21:45:167
14972991211,1cyclictest681-21dbus-daemon20:50:287
14972991211,1cyclictest4957-1kworker/u17:200:35:067
14972991211,1cyclictest411-21sh22:30:087
14972991211,1cyclictest27178-1kworker/u17:023:45:207
14972991211,1cyclictest26937-21nscd23:15:517
14972991211,1cyclictest2513-21nscd23:55:267
14972991211,1cyclictest21773-1kworker/u17:121:10:037
14972991211,1cyclictest20235-21ssh00:15:027
14972991211,1cyclictest1624-21runrttasks21:00:547
14972991211,1cyclictest15361-21strings00:30:267
14972991211,1cyclictest14963-21cyclictest21:08:017
14972991211,1cyclictest14963-21cyclictest20:33:017
14972991211,1cyclictest14963-21cyclictest20:22:447
14972991211,1cyclictest14963-21cyclictest19:36:577
14972991211,1cyclictest14963-21cyclictest19:31:447
14972991211,1cyclictest14963-21cyclictest19:23:497
14972991211,1cyclictest14963-21cyclictest19:11:157
14972991211,1cyclictest14789-1kworker/u17:223:40:067
14972991211,1cyclictest14789-1kworker/u17:223:35:347
14972991211,1cyclictest11768-1kworker/u17:022:21:577
14972991211,1cyclictest10879-21cstates19:55:207
14972991211,1cyclictest10494-21ssh21:40:017
14972991210,1cyclictest682-21nscd23:25:077
14972991210,1cyclictest682-21nscd21:20:037
14972991210,1cyclictest28793-21ssh21:30:537
14972991210,1cyclictest27573-21tune2fs22:45:217
14972991210,1cyclictest20382-21gltestperf20:10:267
14972991210,1cyclictest20382-21gltestperf20:10:267
14972991210,1cyclictest19140-21python21:25:277
14972991210,1cyclictest1876-21ssh21:35:067
14972991210,1cyclictest17969-21munin-node20:45:387
14972991210,1cyclictest17969-21munin-node19:46:117
14972991210,1cyclictest1752-21ntpd23:10:017
14972991210,1cyclictest1752-21ntpd19:40:017
14972991210,1cyclictest1624-21runrttasks20:40:267
14972991210,1cyclictest13206-21ssh22:00:097
14972991210,1cyclictest13206-21ssh22:00:097
14972991210,1cyclictest0-21swapper/719:15:017
14972991210,1cyclictest0-21swapper/719:15:007
14971991211,1cyclictest691-21nscd23:42:106
14971991211,1cyclictest5255-21strings00:05:296
14971991211,1cyclictest5110-21ssh00:25:146
14971991211,1cyclictest3341-21kernelversion20:35:236
14971991211,1cyclictest31003-21kworker/u16:223:25:126
14971991211,1cyclictest31003-21kworker/u16:221:16:506
14971991211,1cyclictest30135-21ssh00:00:496
14971991211,1cyclictest30007-1kworker/u17:122:25:016
14971991211,1cyclictest29302-21grep00:20:196
14971991211,1cyclictest28213-21ssh21:50:076
14971991211,1cyclictest24368-21nscd23:00:186
14971991211,1cyclictest24143-1kworker/u17:021:11:096
14971991211,1cyclictest24125-21kworker/u16:322:21:026
14971991211,1cyclictest22465-21cpuspeed_turbos00:35:216
14971991211,1cyclictest18027-21fschecks_time21:00:236
14971991211,1cyclictest16473-21cpuspeed_turbos20:05:176
14971991211,1cyclictest14963-21cyclictest19:21:566
14971991211,1cyclictest14251-21switchtime00:10:386
14971991211,1cyclictest13579-1kworker/u17:221:27:456
14971991210,1cyclictest8279-21cpuspeed_turbos20:45:236
14971991210,1cyclictest8257-21ssh21:20:096
14971991210,1cyclictest6287-21cat20:40:256
14971991210,1cyclictest29254-21/usr/sbin/munin20:25:226
14971991210,1cyclictest28967-21rm23:05:086
14971991210,1cyclictest26174-21cat21:30:206
14971991210,1cyclictest25665-21cpuspeed_turbos20:20:216
14971991210,1cyclictest22675-21diskmemload23:16:286
14971991210,1cyclictest22675-21diskmemload22:01:226
14971991210,1cyclictest22675-21diskmemload22:01:226
14971991210,1cyclictest21099-21ssh23:37:126
14971991210,1cyclictest21072-21ssh22:40:576
14971991210,1cyclictest1977-21ssh21:35:126
14971991210,1cyclictest19359-21/usr/sbin/munin20:10:256
14971991210,1cyclictest19359-21/usr/sbin/munin20:10:246
14971991210,1cyclictest16962-21sed19:10:336
14971991210,1cyclictest15639-21ntp_states20:55:276
14971991210,1cyclictest1548-21/usr/sbin/munin23:45:316
14971991210,1cyclictest11491-21ssh22:55:026
14971991210,1cyclictest11387-21head19:55:226
14971991210,1cyclictest10696-21runrttasks22:16:286
14971991210,1cyclictest10302-21sh21:58:066
14970991212,0cyclictest0-21swapper/519:58:435
14970991212,0cyclictest0-21swapper/519:10:045
14970991211,1cyclictest73550irq/125-eth023:55:175
14970991211,1cyclictest73550irq/125-eth023:25:075
14970991211,1cyclictest73550irq/125-eth023:20:035
14970991211,1cyclictest73550irq/125-eth022:15:075
14970991211,1cyclictest73550irq/125-eth021:45:095
14970991211,1cyclictest73550irq/125-eth021:40:065
14970991211,1cyclictest73550irq/125-eth021:35:075
14970991211,1cyclictest73550irq/125-eth021:20:085
14970991211,1cyclictest73550irq/125-eth020:50:245
14970991211,1cyclictest73550irq/125-eth020:30:025
14970991211,1cyclictest73550irq/125-eth020:05:055
14970991211,1cyclictest73550irq/125-eth020:00:205
14970991211,1cyclictest73550irq/125-eth019:40:275
14970991211,1cyclictest73550irq/125-eth019:35:265
14970991211,1cyclictest73550irq/125-eth000:10:045
14970991211,1cyclictest73550irq/125-eth000:05:115
14970991211,1cyclictest704-21ptp4l20:36:245
14970991211,1cyclictest14963-21cyclictest20:10:105
14970991211,1cyclictest14963-21cyclictest20:10:105
14970991210,2cyclictest0-21swapper/522:20:025
14970991210,1cyclictest4811-21cstates19:45:185
14970991210,1cyclictest14357-21cstates20:55:205
14970991210,0cyclictest0-21swapper/521:05:125
14970991210,0cyclictest0-21swapper/521:00:155
14970991210,0cyclictest0-21swapper/520:45:275
14970991210,0cyclictest0-21swapper/520:15:245
14970991210,0cyclictest0-21swapper/519:50:235
14970991210,0cyclictest0-21swapper/519:30:445
14970991210,0cyclictest0-21swapper/519:22:125
14970991210,0cyclictest0-21swapper/500:00:015
14969991211,1cyclictest893-21kworker/u16:122:05:314
14969991211,1cyclictest893-21kworker/u16:121:55:224
14969991211,1cyclictest682-21nscd21:25:184
14969991211,1cyclictest682-21nscd21:20:284
14969991211,1cyclictest30616-21diskmemload23:24:474
14969991211,1cyclictest27427-1kworker/u17:123:35:054
14969991211,1cyclictest27368-21/usr/sbin/munin00:00:364
14969991211,1cyclictest26792-21kernelversion20:20:264
14969991211,1cyclictest26176-21cstates21:30:204
14969991211,1cyclictest21378-1kworker/4:1H23:06:074
14969991211,1cyclictest20023-21mailstats22:40:304
14969991211,1cyclictest19148-1kworker/u17:123:00:114
14969991211,1cyclictest16650irq/127-ahci[0023:41:464
14969991211,1cyclictest16650irq/127-ahci[0021:00:264
14969991211,1cyclictest16650irq/127-ahci[0020:39:374
14969991211,1cyclictest16094-21cstates22:20:214
14969991211,1cyclictest14963-21cyclictest21:12:084
14969991211,1cyclictest14963-21cyclictest20:31:524
14969991211,1cyclictest14963-21cyclictest20:05:184
14969991211,1cyclictest10410-21grep23:50:214
14969991210,2cyclictest0-21swapper/423:57:404
14969991210,2cyclictest0-21swapper/422:27:204
14969991210,2cyclictest0-21swapper/400:30:114
14969991210,1cyclictest7108-21mailstats00:25:284
14969991210,1cyclictest704-21ptp4l00:16:204
14969991210,1cyclictest3300-21rm22:30:394
14969991210,1cyclictest23871-21latency20:15:264
14969991210,1cyclictest22356-21/usr/sbin/munin00:35:234
14969991210,1cyclictest14126-21cpuspeed_turbos20:55:214
14969991210,1cyclictest11570-21ssh22:55:054
14969991210,1cyclictest11248-21fschecks_time19:55:214
14969991210,0cyclictest0-21swapper/420:54:314
14969991210,0cyclictest0-21swapper/419:40:004
14968991212,0cyclictest0-21swapper/322:15:013
14968991212,0cyclictest0-21swapper/320:10:313
14968991212,0cyclictest0-21swapper/320:10:313
14968991212,0cyclictest0-21swapper/300:30:013
14968991211,1cyclictest682-21nscd22:25:283
14968991211,1cyclictest5838-21cstates23:10:223
14968991211,1cyclictest5697-21cstates00:25:203
14968991211,1cyclictest31003-21kworker/u16:221:30:133
14968991211,1cyclictest30807-21/usr/sbin/munin19:35:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional