You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-04 - 08:01
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot1.osadl.org (updated Mon Dec 04, 2023 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3056921250,1sleep630551-21C212:39:406
2571699107105,1cyclictest61-21ksoftirqd/500:26:135
25717999688,7cyclictest70-21ksoftirqd/600:26:136
25716998583,1cyclictest61-21ksoftirqd/523:56:205
25711998380,1cyclictest34-21ksoftirqd/200:01:132
25711998380,1cyclictest34-21ksoftirqd/200:01:132
124922830,2sleep22571199cyclictest00:21:032
25711998278,3cyclictest34-21ksoftirqd/200:06:262
50502750,0sleep232-21rcuc/223:34:002
25717997572,2cyclictest70-21ksoftirqd/600:21:036
235622740,3sleep42571599cyclictest19:23:584
25714997371,1cyclictest43-21ksoftirqd/323:46:173
55962720,0sleep30-21swapper/323:34:013
276292720,3sleep02567999cyclictest00:26:140
25679997269,2cyclictest27646-21snmpget21:10:050
25679997269,2cyclictest27646-21snmpget21:10:050
27432710,0sleep20-21swapper/221:28:062
25711997169,1cyclictest34-21ksoftirqd/223:46:172
229362710,0sleep30-21swapper/300:24:323
25698997068,1cyclictest25-21ksoftirqd/100:21:031
223832680,0sleep70-21swapper/700:24:047
295552670,1sleep529559-21needreboot23:03:595
25716996663,2cyclictest61-21ksoftirqd/521:10:055
25716996663,2cyclictest61-21ksoftirqd/521:10:055
118732660,0sleep50-21swapper/500:20:585
25679996563,1cyclictest9-21ksoftirqd/023:46:170
25711996462,1cyclictest32-21rcuc/221:10:052
25711996462,1cyclictest32-21rcuc/221:10:052
270382620,0sleep40-21swapper/400:26:024
25711996259,2cyclictest34-21ksoftirqd/200:26:132
88792580,3sleep02567999cyclictest19:16:010
25718995856,1cyclictest79-21ksoftirqd/700:21:037
25714995754,2cyclictest43-21ksoftirqd/320:10:063
25711995754,2cyclictest34-21ksoftirqd/223:56:202
25716995552,2cyclictest61-21ksoftirqd/523:46:175
25715995552,2cyclictest3790-21check_snmp23:46:174
25711995552,2cyclictest34-21ksoftirqd/220:10:062
25679995452,1cyclictest9-21ksoftirqd/000:21:030
25679995451,2cyclictest9-21ksoftirqd/000:01:130
25679995451,2cyclictest9-21ksoftirqd/000:01:130
25714995351,1cyclictest12396-21check_snmp00:21:033
25716995249,2cyclictest61-21ksoftirqd/520:10:065
25715995148,2cyclictest52-21ksoftirqd/400:21:034
25679995147,3cyclictest9-21ksoftirqd/020:10:060
25679995146,3cyclictest9-21ksoftirqd/023:56:200
25716995047,2cyclictest61-21ksoftirqd/500:06:265
25714994946,2cyclictest43-21ksoftirqd/323:56:203
25715994845,2cyclictest52-21ksoftirqd/423:56:204
25698994743,3cyclictest12017-21sshd20:10:061
25679994742,3cyclictest27440-21bash21:24:050
25717994644,1cyclictest70-21ksoftirqd/623:56:206
25714994643,2cyclictest43-21ksoftirqd/319:35:023
25714994642,2cyclictest43-21ksoftirqd/321:24:053
25698994542,2cyclictest25-21ksoftirqd/123:46:161
25718994442,1cyclictest79-21ksoftirqd/721:24:057
25716994341,1cyclictest61-21ksoftirqd/521:24:055
25714994341,1cyclictest43-21ksoftirqd/300:06:263
25679994339,3cyclictest9-21ksoftirqd/000:06:260
25714994240,1cyclictest43-21ksoftirqd/300:01:133
25714994240,1cyclictest43-21ksoftirqd/300:01:133
25698994240,1cyclictest25-21ksoftirqd/121:24:051
25716994038,1cyclictest15550-21check_snmp00:01:135
25716994038,1cyclictest15550-21check_snmp00:01:135
25717993935,2cyclictest70-21ksoftirqd/621:24:056
25716993936,1cyclictest61-21ksoftirqd/519:59:025
25715993936,1cyclictest52-21ksoftirqd/400:01:134
25715993936,1cyclictest52-21ksoftirqd/400:01:134
25679993934,3cyclictest9-21ksoftirqd/019:59:020
25718993835,2cyclictest79-21ksoftirqd/723:56:207
25717993834,2cyclictest70-21ksoftirqd/600:06:266
25717993734,2cyclictest70-21ksoftirqd/621:10:056
25717993734,2cyclictest70-21ksoftirqd/621:10:056
25714993734,2cyclictest43-21ksoftirqd/321:10:053
25714993734,2cyclictest43-21ksoftirqd/321:10:053
25718993633,2cyclictest79-21ksoftirqd/700:01:137
25718993633,2cyclictest79-21ksoftirqd/700:01:137
25714993634,1cyclictest43-21ksoftirqd/319:59:023
25698993634,1cyclictest25-21ksoftirqd/123:56:201
25717993529,5cyclictest70-21ksoftirqd/600:01:136
25717993529,5cyclictest70-21ksoftirqd/600:01:136
2569899351,32cyclictest16279-21postmaster00:26:141
25711993431,2cyclictest11673-21sshd19:35:022
25711993430,3cyclictest34-21ksoftirqd/219:59:022
25679993431,2cyclictest9-21ksoftirqd/019:35:020
25715993128,2cyclictest52-21ksoftirqd/420:10:064
25717993028,1cyclictest70-21ksoftirqd/619:59:026
25716992926,2cyclictest21457-21snmpget20:50:015
25715992927,1cyclictest52-21ksoftirqd/419:59:024
25715992824,2cyclictest27364-21sshd21:24:054
25718992724,2cyclictest79-21ksoftirqd/720:10:067
25716992724,2cyclictest61-21ksoftirqd/523:36:005
25716992624,1cyclictest59-21rcuc/519:35:025
2571599260,13cyclictest7266-21ntop19:33:544
25698992624,1cyclictest25-21ksoftirqd/119:35:021
224182264,17sleep50-21swapper/519:06:065
224182264,17sleep50-21swapper/519:06:065
25716992523,1cyclictest61-21ksoftirqd/520:35:015
25715992523,1cyclictest26114-21sshd20:35:014
25711992510,11cyclictest28596-21kworker/2:020:04:072
25698992523,1cyclictest25-21ksoftirqd/121:10:051
25698992523,1cyclictest25-21ksoftirqd/121:10:051
25698992523,1cyclictest25-21ksoftirqd/119:59:021
25679992523,1cyclictest9-21ksoftirqd/020:50:010
72212240,2sleep777-21rcuc/722:26:597
25714992421,2cyclictest43-21ksoftirqd/320:35:013
25714992416,1cyclictest0-21swapper/321:04:013
25711992415,8cyclictest20918-21perl19:19:062
299272230,0sleep40-21swapper/421:11:024
299272230,0sleep40-21swapper/421:11:024
25718992320,2cyclictest79-21ksoftirqd/721:10:057
25718992320,2cyclictest79-21ksoftirqd/721:10:057
25718992315,7cyclictest7265-21ntop21:03:427
25714992321,1cyclictest43-21ksoftirqd/321:45:383
25711992318,3cyclictest331ktimersoftd/220:32:312
25698992321,1cyclictest25-21ksoftirqd/100:06:261
25679992315,7cyclictest190550irq/25-eth023:05:010
178302230,0sleep50-21swapper/522:03:025
25718992219,2cyclictest302-21sshd20:04:007
2571899221,19cyclictest22399-21perl22:05:067
2571499228,13cyclictest0-21swapper/319:21:163
25711992213,6cyclictest0-21swapper/221:52:242
25698992211,9cyclictest2453-21openvpn20:18:581
2569899220,17cyclictest0-21swapper/122:52:561
2567999228,13cyclictest0-21swapper/022:05:200
25679992220,1cyclictest9-21ksoftirqd/023:41:020
237702223,5sleep70-21swapper/719:07:207
237702223,5sleep70-21swapper/719:07:207
25717992114,6cyclictest0-21swapper/621:04:496
25717992113,6cyclictest1190-21perl23:05:386
2571799210,2cyclictest3258-21perl20:04:006
25716992119,1cyclictest0-21swapper/523:29:585
2571699210,2cyclictest12916-21perl21:03:345
25715992118,2cyclictest9057-21postmaster22:15:424
2571499210,19cyclictest22453-21perl22:05:053
25711992118,2cyclictest22384-21perl22:05:052
2571199211,18cyclictest31068-21perl23:04:132
2571199211,17cyclictest34-21ksoftirqd/221:04:552
2569899211,18cyclictest31051-21perl23:04:131
25679992119,1cyclictest9-21ksoftirqd/020:35:010
25679992117,3cyclictest6105-21kworker/0:223:19:160
25679992117,3cyclictest4596-21kworker/0:122:50:440
25679992117,3cyclictest19838-21kworker/0:021:51:420
25679992117,3cyclictest1050-21kworker/0:020:26:090
25679992116,4cyclictest21097-21perl19:19:060
2567999210,2cyclictest1824-21perl20:04:220
2567999210,1cyclictest0-21swapper/021:19:070
224702215,3sleep10-21swapper/119:06:101
224702215,3sleep10-21swapper/119:06:101
195982210,0sleep70-21swapper/700:33:267
292802200,0sleep30-21swapper/320:54:023
2571899209,9cyclictest15248-21perl21:19:047
25718992017,1cyclictest79-21ksoftirqd/719:35:027
25718992016,2cyclictest79-21ksoftirqd/720:35:017
2571899200,1cyclictest0-21swapper/723:19:237
2571899200,1cyclictest0-21swapper/723:04:317
25717992017,2cyclictest20456-21perl22:04:136
25717992013,6cyclictest2913-21snmpd22:32:346
25717992013,6cyclictest2913-21snmpd20:39:186
2571799201,17cyclictest15299-21perl19:19:106
25716992018,1cyclictest61-21ksoftirqd/500:11:365
25716992016,3cyclictest27482-21kworker/5:123:22:125
25716992015,3cyclictest23184-21kworker/5:020:19:035
2571599202,1cyclictest15382-21JNI-ICMP-14335-20:04:224
25715992017,2cyclictest31440-21perl23:04:294
25715992017,2cyclictest13128-21perl21:03:424
2571599201,2cyclictest15301-21perl21:19:064
2571499200,1cyclictest0-21swapper/323:19:233
2571499200,1cyclictest0-21swapper/323:04:313
25711992018,1cyclictest34-21ksoftirqd/220:50:012
25711992016,2cyclictest34-21ksoftirqd/223:41:022
25711992015,1cyclictest331ktimersoftd/220:14:532
25711992014,5cyclictest0-21swapper/200:29:022
25698992015,2cyclictest2725-21nscd23:36:001
2569899200,2cyclictest12916-21perl21:03:341
2567999202,1cyclictest190550irq/25-eth021:04:040
25679992018,1cyclictest9-21ksoftirqd/023:50:020
25679992018,1cyclictest190550irq/25-eth023:36:000
25679992018,1cyclictest15447-21kworker/0:221:31:040
25679992018,1cyclictest10946-21kworker/0:122:14:400
25679992018,1cyclictest1050-21kworker/0:020:32:390
25679992017,2cyclictest10946-21kworker/0:122:02:220
25679992017,1cyclictest4596-21kworker/0:122:53:420
25679992017,1cyclictest10946-21kworker/0:122:26:240
302022190,0sleep10-21swapper/119:10:011
290662190,0sleep20-21swapper/223:17:102
25718991916,2cyclictest14926-21perl19:19:067
25718991915,2cyclictest2453-21openvpn22:19:097
25718991913,5cyclictest2913-21snmpd22:59:197
25718991912,6cyclictest2913-21snmpd22:51:497
25718991912,6cyclictest21728-21perl20:18:587
25718991910,8cyclictest32089-21cat19:28:537
25718991910,8cyclictest32089-21cat19:28:537
25717991917,1cyclictest70-21ksoftirqd/620:35:016
25717991910,8cyclictest2913-21snmpd20:54:036
25716991917,1cyclictest61-21ksoftirqd/523:50:025
25716991917,1cyclictest10515-21kworker/5:221:38:305
25716991916,2cyclictest30204-21kworker/5:119:29:055
25716991916,2cyclictest30204-21kworker/5:119:29:055
25716991916,2cyclictest21434-21perl22:04:475
25716991915,2cyclictest15520-21perl21:49:065
25715991916,2cyclictest21097-21perl19:19:064
25714991917,1cyclictest0-21swapper/322:55:123
25714991916,2cyclictest43-21ksoftirqd/323:15:573
25714991913,1cyclictest26801-21QueuingRrdStrat19:24:003
25711991917,1cyclictest34-21ksoftirqd/220:35:012
25711991916,2cyclictest14788-21kworker/2:019:12:572
25711991915,3cyclictest34-21ksoftirqd/221:55:402
25711991913,4cyclictest30079-21kworker/2:119:48:552
25698991916,2cyclictest21551-21perl22:18:591
25679991917,1cyclictest6105-21kworker/0:223:16:400
25679991917,1cyclictest4596-21kworker/0:123:02:520
25679991917,1cyclictest31271-21kworker/0:020:56:400
25679991917,1cyclictest2578-21kworker/0:100:29:000
25679991917,1cyclictest23794-21kworker/0:019:54:570
25679991917,1cyclictest23794-21kworker/0:019:50:090
25679991917,1cyclictest23282-21kworker/0:120:18:410
25679991917,1cyclictest19838-21kworker/0:021:53:480
25679991917,1cyclictest16956-21kworker/0:223:29:480
25679991917,1cyclictest15747-21kworker/0:219:24:290
25679991917,1cyclictest11541-21kworker/0:220:14:290
25679991916,2cyclictest9-21ksoftirqd/000:11:360
25679991916,2cyclictest24678-21kworker/0:121:47:240
25679991916,2cyclictest24241-21kworker/0:120:39:250
25679991915,3cyclictest9842-21kworker/0:222:32:040
229332192,13sleep40-21swapper/419:07:014
229332192,13sleep40-21swapper/419:07:014
212732192,3sleep20-21swapper/219:05:142
212732192,3sleep20-21swapper/219:05:142
2571899182,8cyclictest16080-21postmaster19:45:117
25718991812,5cyclictest2913-21snmpd22:47:047
25718991811,6cyclictest2913-21snmpd19:49:187
2571799189,2cyclictest2641-21ssh23:19:216
2571799181,5cyclictest22965-21postmaster22:02:166
2571799180,17cyclictest0-21swapper/623:42:116
25716991816,1cyclictest61-21ksoftirqd/523:41:025
25716991816,1cyclictest5011-21kworker/5:222:39:165
25716991816,1cyclictest3090-21kworker/5:120:59:065
25716991816,1cyclictest3090-21kworker/5:120:59:065
25716991816,1cyclictest24745-21kworker/5:019:53:015
25716991816,1cyclictest24057-21kworker/5:020:46:075
25716991816,1cyclictest24057-21kworker/5:020:40:485
25716991816,1cyclictest23743-21kworker/5:200:28:345
25716991816,1cyclictest23184-21kworker/5:020:24:075
25716991816,1cyclictest22136-21kworker/5:222:08:485
25716991816,1cyclictest15603-21kworker/5:122:34:565
25716991816,1cyclictest14786-21kworker/5:121:57:585
25716991816,1cyclictest10515-21kworker/5:221:41:405
25716991815,2cyclictest9514-21QueuingRrdStrat20:04:195
25716991815,1cyclictest18075-21kworker/5:000:14:585
25716991814,3cyclictest3960-21kworker/5:021:45:445
25716991814,3cyclictest24745-21kworker/5:019:53:595
25716991814,3cyclictest18323-21kworker/5:219:14:295
2571599181,9cyclictest15375-21JNI-ICMP-10072-23:19:394
25715991816,1cyclictest52-21ksoftirqd/420:50:014
25715991815,2cyclictest20755-21perl22:03:564
2571599180,17cyclictest26650-21sshd19:08:494
25714991816,1cyclictest43-21ksoftirqd/320:50:013
25714991815,1cyclictest43-21ksoftirqd/323:30:593
25714991815,1cyclictest43-21ksoftirqd/323:10:503
25714991814,2cyclictest21023-21perl22:18:563
25711991816,1cyclictest34-21ksoftirqd/223:50:022
25711991815,2cyclictest16244-21Collectd-Thread20:19:092
25711991813,4cyclictest0-21swapper/221:01:002
25711991813,4cyclictest0-21swapper/221:01:002
25711991812,5cyclictest25373-21smartctl22:48:482
25711991812,4cyclictest331ktimersoftd/222:10:042
25711991810,5cyclictest331ktimersoftd/222:28:542
2571199180,13cyclictest0-21swapper/220:27:412
25698991814,3cyclictest3255-21perl20:04:001
2569899181,12cyclictest15375-21JNI-ICMP-10072-22:42:241
25679991816,1cyclictest9-21ksoftirqd/022:40:480
25679991816,1cyclictest6105-21kworker/0:223:13:180
25679991816,1cyclictest4596-21kworker/0:122:45:080
25679991816,1cyclictest3811-21kworker/0:123:28:240
25679991816,1cyclictest24678-21kworker/0:121:40:280
25679991816,1cyclictest190550irq/25-eth020:59:060
25679991816,1cyclictest190550irq/25-eth020:59:060
25679991816,1cyclictest14651-21kworker/0:219:12:130
25679991815,2cyclictest24241-21kworker/0:120:44:560
25679991814,3cyclictest9-21ksoftirqd/000:14:470
238292180,0sleep30-21swapper/300:34:583
39112170,1sleep73915-21cpuspeed20:23:467
2571899174,5cyclictest15909-21Poller-Thread-519:55:267
2571899174,4cyclictest30375-21JNI-ICMP-12819-00:04:497
2571899173,5cyclictest15375-21JNI-ICMP-10072-21:49:287
2571899170,9cyclictest0-21swapper/719:58:507
2571799178,2cyclictest15422-21ssh21:19:106
2571799177,1cyclictest1093-21ntop22:18:586
25716991715,1cyclictest7927-21kworker/5:023:26:525
25716991715,1cyclictest3775-21kworker/5:022:14:365
25716991715,1cyclictest3079-21kworker/5:122:56:465
25716991715,1cyclictest30204-21kworker/5:119:24:495
25716991715,1cyclictest27630-21kworker/5:300:35:285
25716991715,1cyclictest24745-21kworker/5:019:44:375
25716991715,1cyclictest24478-21kworker/5:220:53:465
25716991715,1cyclictest17933-21kworker/5:019:09:255
25716991715,1cyclictest15745-21kworker/5:022:59:065
25716991715,1cyclictest15603-21kworker/5:122:30:525
25716991715,1cyclictest15603-21kworker/5:122:25:345
25716991715,1cyclictest13182-21kworker/5:023:14:025
25716991715,1cyclictest13182-21kworker/5:023:10:485
25716991714,2cyclictest61-21ksoftirqd/521:16:025
25716991714,2cyclictest30204-21kworker/5:119:19:035
25716991714,1cyclictest24738-21kworker/5:021:32:005
25716991714,1cyclictest1176-21kworker/5:120:29:515
25716991713,1cyclictest15603-21kworker/5:122:44:365
2571599173,9cyclictest0-21swapper/422:09:484
25714991715,1cyclictest43-21ksoftirqd/323:41:003
25714991715,1cyclictest43-21ksoftirqd/322:25:443
25714991714,2cyclictest17404-21perl21:19:033
2571499171,1cyclictest43-21ksoftirqd/322:38:343
2571499170,16cyclictest0-21swapper/322:09:163
25711991715,1cyclictest2831-21perl23:19:272
25711991714,1cyclictest19688-21perl20:54:052
25711991712,4cyclictest331ktimersoftd/222:48:062
25711991712,3cyclictest0-21swapper/219:57:272
25711991712,0cyclictest13950irq/24-ahci[00000:10:562
2571199170,1cyclictest0-21swapper/222:38:342
25698991714,2cyclictest20384-21perl22:04:071
25698991713,2cyclictest23117-21perl23:19:231
2569899171,14cyclictest15298-21perl19:19:101
2569899170,15cyclictest20690-21ssh21:49:071
25679991715,1cyclictest9-21ksoftirqd/022:10:480
25679991715,1cyclictest27024-21kworker/0:119:39:510
25679991715,1cyclictest27024-21kworker/0:119:31:250
25679991715,1cyclictest27024-21kworker/0:119:31:250
25679991715,1cyclictest2578-21kworker/0:100:33:440
25679991714,2cyclictest20395-21kworker/0:021:16:560
25679991713,3cyclictest9842-21kworker/0:222:35:520
25679991713,3cyclictest9-21ksoftirqd/019:45:170
25679991713,3cyclictest24678-21kworker/0:121:38:040
227952174,7sleep60-21swapper/619:06:476
227952174,7sleep60-21swapper/619:06:476
2571899163,5cyclictest15107-21JNI-ICMP-24208-22:09:497
2571899162,6cyclictest13241-21DefaultUDPTrans20:45:347
2571899161,2cyclictest2327-21ntop00:15:027
2571899161,13cyclictest30934-21Poller-Thread-123:44:347
2571899160,9cyclictest2935-21perl23:48:457
2571899160,1cyclictest0-21swapper/723:29:157
2571899160,13cyclictest0-21swapper/721:28:557
2571799162,4cyclictest5091-21postmaster23:30:596
2571799161,5cyclictest30375-21JNI-ICMP-12819-00:11:466
2571799160,1cyclictest0-21swapper/619:50:316
25716991614,1cyclictest24738-21kworker/5:021:20:165
25716991614,1cyclictest15603-21kworker/5:122:20:005
25716991614,1cyclictest14885-21kworker/5:319:43:075
25716991614,1cyclictest12227-21kworker/5:320:15:075
25716991613,2cyclictest16164-21Collectd-Thread22:48:385
2571599164,7cyclictest15375-21JNI-ICMP-10072-22:55:234
2571599163,6cyclictest15381-21JNI-ICMP-14335-22:38:374
2571199168,6cyclictest32-21rcuc/200:18:202
2571199167,7cyclictest29635-21kworker/2:121:15:522
2571199165,10cyclictest11-21rcu_preempt22:55:502
25711991615,0cyclictest13950irq/24-ahci[00022:19:182
25711991614,1cyclictest34-21ksoftirqd/223:31:002
25711991614,1cyclictest32-21rcuc/223:11:482
25711991613,2cyclictest30771-21PollerBackEnd_W00:38:202
25711991613,2cyclictest13488-21Timer-48697322:15:502
25711991612,3cyclictest32-21rcuc/220:43:572
25711991612,3cyclictest0-21swapper/219:24:492
2571199160,14cyclictest0-21swapper/221:20:382
2569899163,8cyclictest15107-21JNI-ICMP-24208-20:23:561
25698991613,2cyclictest25-21ksoftirqd/120:50:011
2569899161,2cyclictest4577-21pool-14-thread-21:42:171
25679991613,2cyclictest10946-21kworker/0:122:20:320
2571899155,3cyclictest15104-21JNI-ICMP-8335-S23:34:387
2571899154,3cyclictest15113-21JNI-ICMP-23157-22:56:137
2571899153,5cyclictest15107-21JNI-ICMP-24208-20:54:047
2571899153,5cyclictest15104-21JNI-ICMP-8335-S23:27:557
2571899153,5cyclictest15104-21JNI-ICMP-8335-S21:56:397
2571899153,4cyclictest15104-21JNI-ICMP-8335-S19:14:447
2571899152,8cyclictest15113-21JNI-ICMP-23157-22:39:277
2571899152,5cyclictest15372-21JNI-ICMP-4388-S19:10:277
2571899152,3cyclictest2327-21ntop00:09:477
2571899151,4cyclictest15107-21JNI-ICMP-24208-20:33:027
2571899150,8cyclictest0-21swapper/723:40:277
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional