You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 21:34
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sat Apr 27, 2024 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21567213727,72sleep021591-21apt07:05:030
217449912355,20cyclictest13750-21fschecks_time12:15:222
217449912355,20cyclictest13750-21fschecks_time12:15:222
217219911037,37cyclictest29783-21/usr/sbin/munin10:00:330
217219911037,37cyclictest29783-21/usr/sbin/munin10:00:330
1449721020,4chrt14498-21ssh10:29:112
21752991019,55cyclictest31151-21ssh11:50:483
288452990,4chrt28844-21ssh09:59:523
288452990,4chrt28844-21ssh09:59:523
2175299962,70cyclictest6472-21kworker/u8:211:40:323
21721999641,21cyclictest3017-21ssh10:10:240
21721999641,21cyclictest3017-21ssh10:10:240
21744999511,38cyclictest30950-21apt-get11:50:282
21721999545,27cyclictest11819-21ssh10:25:040
21721999535,35cyclictest27675-21/usr/sbin/munin07:21:170
21721999447,14cyclictest24682-21ssh10:45:250
21721999336,30cyclictest10386-21cut12:10:060
21721999335,30cyclictest27374-21apt-get10:50:150
21721999247,25cyclictest29027-21diskmemload10:37:140
21721999245,31cyclictest0-21swapper/010:34:580
21721999243,31cyclictest29027-21diskmemload09:28:060
21721999042,25cyclictest0-21swapper/011:04:080
21721999032,37cyclictest23214-21cat12:30:370
21721999032,23cyclictest0-21swapper/010:06:140
21721998937,17cyclictest5685-21sed07:50:440
21721998937,17cyclictest5685-21sed07:50:440
21721998836,28cyclictest15814-21sh12:19:130
21721998836,28cyclictest15814-21sh12:19:130
21721998833,22cyclictest19163-21latency_hist12:25:040
21721998832,33cyclictest30728-21cpu07:30:190
21721998832,21cyclictest0-21swapper/007:55:190
21721998821,41cyclictest5200-21runrttasks11:06:240
21721998739,17cyclictest0-21swapper/010:17:530
21721998737,28cyclictest12854-21ssh12:14:140
21721998737,27cyclictest399-21runrttasks08:20:280
21721998737,19cyclictest15105-21/usr/sbin/munin09:35:280
21721998737,17cyclictest15943-21fschecks_count11:25:240
21721998737,17cyclictest15943-21fschecks_count11:25:240
21721998736,29cyclictest2294-21ssh09:14:280
21721998734,26cyclictest29027-21diskmemload11:15:580
21721998734,26cyclictest29027-21diskmemload11:15:580
21721998732,30cyclictest19365-21sh12:25:100
21721998732,30cyclictest19365-21sh12:25:100
21721998732,28cyclictest3574-21diskmemload11:58:310
21721998732,28cyclictest3574-21diskmemload11:58:310
21721998732,20cyclictest26736-21chrt08:55:360
21721998731,30cyclictest29027-21diskmemload10:26:340
21721998729,17cyclictest5990-21ssh09:20:230
21721998726,23cyclictest0-21swapper/010:41:200
21721998724,25cyclictest0-21swapper/011:45:020
21721998636,18cyclictest24354-21sed07:10:320
21721998633,32cyclictest4685-21runrttasks09:17:470
21721998630,16cyclictest0-21swapper/011:51:270
21721998629,18cyclictest29228-21ssh09:05:060
21721998539,23cyclictest29027-21diskmemload11:31:260
21721998534,30cyclictest449-21ssh10:59:450
21721998531,30cyclictest25233-21sh09:53:030
21721998528,23cyclictest27760-21/usr/sbin/munin11:45:210
21721998524,25cyclictest0-21swapper/009:47:400
21721998520,18cyclictest11537-21cstates08:10:220
2175299843,56cyclictest399-21runrttasks10:06:033
21721998437,16cyclictest11655-21ssh09:30:160
21721998437,16cyclictest11655-21ssh09:30:160
21721998432,27cyclictest29027-21diskmemload11:10:490
21721998431,30cyclictest10844-21seq08:06:050
21721998430,20cyclictest4617-21ssh12:00:220
21721998429,22cyclictest17461-21/usr/sbin/munin09:40:410
21744998311,59cyclictest24262-21apt-get11:40:162
21733998365,12cyclictest7705-21munin-plugin-st08:00:021
21721998334,19cyclictest19416-21latency_hist08:35:050
21721998334,17cyclictest399-21runrttasks08:00:350
21721998334,17cyclictest399-21runrttasks08:00:350
21721998331,18cyclictest1294-21cat07:40:050
21721998329,32cyclictest4231-21seq07:47:340
21721998329,23cyclictest2355-21sed07:40:370
21744998236,8cyclictest141rcu_preempt12:20:182
21721998232,20cyclictest24741-21cat08:50:250
21721998232,20cyclictest24741-21cat08:50:250
21721998232,20cyclictest12949-21fschecks_count11:20:240
2025628222,24sleep10-21swapper/107:00:201
21721998162,10cyclictest26263-21apt-get09:55:140
21721998162,10cyclictest26263-21apt-get09:55:140
21721998133,19cyclictest29425-21/usr/sbin/munin07:25:300
21721998133,17cyclictest14589-21wc08:20:050
21721998133,17cyclictest14589-21wc08:20:050
21721998131,18cyclictest0-21swapper/008:35:210
21721998128,23cyclictest21632-21find11:35:130
2060928122,28sleep20-21swapper/207:00:332
151162812,9sleep30-21swapper/310:30:143
21721998038,14cyclictest29593-21fschecks_count09:05:230
21721998038,14cyclictest29593-21fschecks_count09:05:230
21721998032,19cyclictest25578-21/usr/sbin/munin07:15:260
21721998032,19cyclictest25578-21/usr/sbin/munin07:15:260
205402809,31sleep30-21swapper/307:00:303
21721997928,18cyclictest0-21swapper/008:45:300
2175299779,30cyclictest0-21swapper/312:10:573
2175299772,50cyclictest8198-21apt-get08:00:273
2175299772,50cyclictest8198-21apt-get08:00:273
21744997725,32cyclictest0-21swapper/211:15:392
21744997725,32cyclictest0-21swapper/211:15:392
21721997732,17cyclictest23316-21latency_hist07:10:050
21721997723,22cyclictest21165-21/usr/sbin/munin08:40:380
21752997653,14cyclictest23384-21unixbench_multi12:30:383
21752997538,12cyclictest141rcu_preempt09:30:483
21752997538,12cyclictest141rcu_preempt09:30:483
21752997517,27cyclictest41-21ksoftirqd/308:20:293
21721997529,19cyclictest17056-21sed08:25:380
21744997452,15cyclictest17283-21unixbench_singl08:25:402
21744997439,13cyclictest33-21ksoftirqd/207:25:392
21744997417,35cyclictest10506-21apt-get12:10:562
21733997452,16cyclictest13159-21idleruntime-cro12:15:021
2174499739,46cyclictest21681-21ssh10:40:452
21744997350,14cyclictest10257-21apt-key12:10:052
21744997239,9cyclictest116-21pr/ttymxc012:33:352
21744997232,23cyclictest0-21swapper/211:10:092
2174499717,48cyclictest0-21swapper/210:15:342
21744997140,10cyclictest141rcu_preempt09:29:512
21733997146,14cyclictest29593-21fschecks_count09:05:231
21733997146,14cyclictest29593-21fschecks_count09:05:231
21752997017,21cyclictest41-21ksoftirqd/310:55:403
21752997016,22cyclictest41-21ksoftirqd/307:15:383
21752997016,22cyclictest41-21ksoftirqd/307:15:383
21744997038,9cyclictest0-21swapper/211:33:142
21752996941,15cyclictest141rcu_preempt10:25:083
21752996918,14cyclictest0-21swapper/310:39:133
21752996916,22cyclictest41-21ksoftirqd/308:00:073
21752996916,17cyclictest0-21swapper/312:23:483
21744996931,22cyclictest141rcu_preempt09:20:482
2175299688,24cyclictest399-21runrttasks07:23:413
21752996841,7cyclictest141rcu_preempt11:25:133
21752996841,7cyclictest141rcu_preempt11:25:133
21752996837,8cyclictest141rcu_preempt10:00:263
21752996837,8cyclictest141rcu_preempt10:00:263
21744996811,26cyclictest32637-21ssh10:06:002
21744996732,20cyclictest141rcu_preempt10:10:402
21744996732,20cyclictest141rcu_preempt10:10:402
21744996727,24cyclictest347-21in:imklog09:42:002
21744996725,8cyclictest141rcu_preempt10:05:032
21744996725,8cyclictest141rcu_preempt10:05:032
2173399672,30cyclictest32367-21apt-get10:05:161
21733996717,14cyclictest22796-21apt-get08:45:121
21752996621,12cyclictest141rcu_preempt08:15:223
21752996621,12cyclictest141rcu_preempt08:15:223
21752996618,19cyclictest41-21ksoftirqd/311:05:283
21744996653,8cyclictest21708-21cyclictest10:34:052
21752996542,15cyclictest21516-21ssh12:28:333
21752996542,15cyclictest21516-21ssh12:28:333
21752996536,16cyclictest141rcu_preempt09:15:423
21752996523,18cyclictest41-21ksoftirqd/312:15:213
21752996523,18cyclictest41-21ksoftirqd/312:15:213
21752996512,21cyclictest41-21ksoftirqd/311:00:293
21744996548,9cyclictest2514-21munin-run09:15:022
21744996546,13cyclictest25394-21timerwakeupswit10:45:392
21744996544,8cyclictest33-21ksoftirqd/209:46:202
2173399656,20cyclictest9821-21ssh12:09:391
21733996547,11cyclictest27233-21apt-key07:20:061
21733996547,11cyclictest27233-21apt-key07:20:061
21733996514,12cyclictest141rcu_preempt12:20:411
2175299649,23cyclictest41-21ksoftirqd/307:05:333
21752996438,7cyclictest141rcu_preempt11:15:453
21752996438,7cyclictest141rcu_preempt11:15:453
21752996436,7cyclictest141rcu_preempt10:50:373
21752996429,19cyclictest6472-21kworker/u8:212:09:033
21744996444,13cyclictest29565-21fschecks_count09:05:222
21744996444,13cyclictest29565-21fschecks_count09:05:222
21744996438,16cyclictest0-21swapper/208:55:382
2174499642,35cyclictest11613-21cut08:10:272
2175299639,30cyclictest41-21ksoftirqd/309:25:313
21752996341,8cyclictest141rcu_preempt07:32:013
21752996334,16cyclictest141rcu_preempt09:13:343
21752996330,7cyclictest141rcu_preempt11:58:313
21752996330,7cyclictest141rcu_preempt11:58:313
21752996329,20cyclictest17153-21kworker/u8:309:40:253
21752996329,13cyclictest141rcu_preempt07:55:053
21752996329,13cyclictest141rcu_preempt07:55:053
21752996327,13cyclictest141rcu_preempt08:50:053
2175299632,43cyclictest41-21ksoftirqd/307:48:473
21752996322,11cyclictest0-21swapper/310:47:033
21752996321,20cyclictest0-21swapper/309:50:543
2174499637,42cyclictest8225-21taskset08:00:272
2174499637,42cyclictest8225-21taskset08:00:272
21744996348,9cyclictest19282-21unixbench_singl10:35:402
21744996334,19cyclictest11168-21idleruntime-cro08:10:022
21744996331,9cyclictest2072-21ssh11:55:302
21744996331,9cyclictest2072-21ssh11:55:302
21744996323,20cyclictest141rcu_preempt09:30:132
21744996323,20cyclictest141rcu_preempt09:30:132
21733996344,12cyclictest276-21systemd-journal11:05:271
21733996338,17cyclictest9579-21netstat10:20:281
21733996314,11cyclictest141rcu_preempt09:45:161
21752996239,11cyclictest0-21swapper/312:05:023
21752996238,11cyclictest0-21swapper/308:30:033
21752996217,18cyclictest41-21ksoftirqd/309:35:373
21752996216,18cyclictest41-21ksoftirqd/309:25:053
2175299621,31cyclictest23916-21ssh10:44:273
21752996213,18cyclictest41-21ksoftirqd/310:21:453
21752996210,23cyclictest401ktimers/311:10:143
2174499629,26cyclictest22190-21kworker/2:209:50:252
2174499625,19cyclictest16209-21latency_hist08:25:052
21744996231,11cyclictest141rcu_preempt07:10:202
21744996228,12cyclictest33-21ksoftirqd/207:30:252
21733996241,11cyclictest0-21swapper/109:20:391
21733996235,18cyclictest32649-21fschecks_time09:10:251
57482610,6sleep15422-21apt-get10:15:071
57482610,6sleep15422-21apt-get10:15:071
2175299616,24cyclictest41-21ksoftirqd/308:40:343
21752996133,14cyclictest141rcu_preempt11:45:333
21752996130,17cyclictest141rcu_preempt08:10:333
21752996115,17cyclictest41-21ksoftirqd/309:46:473
21752996110,35cyclictest41-21ksoftirqd/310:11:183
21752996110,35cyclictest41-21ksoftirqd/310:11:183
21744996130,11cyclictest141rcu_preempt11:05:252
21744996118,16cyclictest141rcu_preempt07:21:362
21744996113,24cyclictest0-21swapper/210:24:092
21733996141,12cyclictest0-21swapper/111:10:251
21733996139,13cyclictest11363-21kworker/u8:409:59:471
21733996139,13cyclictest11363-21kworker/u8:409:59:471
21733996131,19cyclictest30541-21grep11:50:041
2175299609,27cyclictest0-21swapper/308:55:063
2175299609,24cyclictest401ktimers/311:35:053
2175299609,24cyclictest141rcu_preempt08:50:353
2175299609,24cyclictest141rcu_preempt08:50:353
2175299608,25cyclictest41-21ksoftirqd/309:05:223
2175299608,25cyclictest41-21ksoftirqd/309:05:223
21752996028,7cyclictest141rcu_preempt11:20:433
21752996016,16cyclictest41-21ksoftirqd/307:25:313
21744996035,9cyclictest141rcu_preempt09:20:062
21744996035,8cyclictest141rcu_preempt08:30:392
21744996034,12cyclictest0-21swapper/207:15:252
21744996034,12cyclictest0-21swapper/207:15:252
21744996030,9cyclictest141rcu_preempt08:55:052
21744996030,9cyclictest141rcu_preempt08:55:052
21744996028,11cyclictest141rcu_preempt10:57:502
21744996020,17cyclictest141rcu_preempt11:45:152
2173399609,19cyclictest399-21runrttasks11:03:021
21733996044,8cyclictest555-21kworker/u8:111:00:021
21733996038,15cyclictest18750-21journal-offline08:30:431
149992600,9sleep125-21ksoftirqd/108:20:291
21752995936,12cyclictest0-21swapper/308:05:253
21752995915,18cyclictest41-21ksoftirqd/307:10:063
21752995915,17cyclictest41-21ksoftirqd/307:35:383
21744995936,7cyclictest0-21swapper/209:36:492
21744995934,7cyclictest33-21ksoftirqd/211:21:132
21744995930,15cyclictest52-21kcompactd011:35:202
21744995927,9cyclictest141rcu_preempt09:58:292
21744995927,9cyclictest141rcu_preempt09:58:292
21744995927,16cyclictest141rcu_preempt08:20:062
21744995910,17cyclictest141rcu_preempt08:00:032
21733995943,10cyclictest29027-21diskmemload10:20:041
21733995942,10cyclictest116-21pr/ttymxc007:05:141
21733995941,12cyclictest23040-21wc12:30:341
2175299589,29cyclictest41-21ksoftirqd/307:41:073
21752995829,16cyclictest141rcu_preempt10:17:583
21752995818,18cyclictest41-21ksoftirqd/309:05:063
21752995816,19cyclictest41-21ksoftirqd/311:35:373
21744995838,12cyclictest0-21swapper/210:55:022
21744995834,18cyclictest33-21ksoftirqd/212:00:302
21744995827,14cyclictest33-21ksoftirqd/211:29:052
21744995827,14cyclictest33-21ksoftirqd/211:29:052
21744995819,13cyclictest23373-21latency_hist07:10:062
21733995838,10cyclictest0-21swapper/110:50:031
2173399583,35cyclictest8344-21sshd08:01:001
2173399583,35cyclictest8344-21sshd08:01:001
21733995833,17cyclictest26093-21latency_hist09:55:061
2174499579,23cyclictest141rcu_preempt07:51:112
2174499579,23cyclictest141rcu_preempt07:51:112
21744995741,10cyclictest1-21systemd11:05:032
21744995729,8cyclictest141rcu_preempt07:39:122
21744995725,17cyclictest116-21pr/ttymxc012:25:242
21744995725,17cyclictest116-21pr/ttymxc012:25:242
2173399575,17cyclictest25-21ksoftirqd/109:37:551
21733995741,8cyclictest0-21swapper/111:40:061
21733995739,12cyclictest21257-21apt-key10:40:061
21733995737,13cyclictest15859-21processes10:30:381
2173399572,17cyclictest24810-21apt-get11:40:161
2173399571,24cyclictest0-21swapper/111:32:241
2175299564,29cyclictest41-21ksoftirqd/308:32:203
21752995639,9cyclictest401ktimers/308:35:363
21744995633,16cyclictest347-21in:imklog07:45:242
2173399566,21cyclictest27352-21/usr/sbin/munin10:50:241
21733995613,14cyclictest141rcu_preempt08:15:031
21744995536,12cyclictest347-21in:imklog08:15:202
21744995533,12cyclictest0-21swapper/209:00:392
2174499552,14cyclictest141rcu_preempt08:40:272
21733995536,10cyclictest19670-21fschecks_time12:25:221
21733995536,10cyclictest19670-21fschecks_time12:25:221
21733995531,15cyclictest3861-21unixbench_multi09:15:401
2173399552,25cyclictest399-21runrttasks08:58:501
21744995428,11cyclictest141rcu_preempt08:35:252
21733995438,11cyclictest0-21swapper/108:55:031
21733995438,11cyclictest0-21swapper/108:55:031
21733995433,12cyclictest0-21swapper/107:50:441
21733995433,12cyclictest0-21swapper/107:50:441
21733995433,11cyclictest0-21swapper/109:40:321
21733995432,12cyclictest0-21swapper/111:55:021
21744995330,12cyclictest0-21swapper/208:50:052
2173399539,18cyclictest25-21ksoftirqd/109:25:261
2173399539,17cyclictest16321-21ssh11:25:351
2173399539,17cyclictest16321-21ssh11:25:351
21733995337,8cyclictest0-21swapper/108:45:021
21733995335,9cyclictest0-21swapper/107:45:051
2173399533,17cyclictest14709-21ssh11:23:111
21733995331,14cyclictest0-21swapper/108:15:401
21733995331,14cyclictest0-21swapper/108:15:401
21733995331,12cyclictest0-21swapper/107:10:431
21733995331,11cyclictest0-21swapper/109:00:241
21733995330,12cyclictest0-21swapper/107:30:031
21733995310,11cyclictest29027-21diskmemload11:57:251
21733995310,11cyclictest29027-21diskmemload11:57:251
21733995310,11cyclictest12099-21ssh10:25:141
2174499522,17cyclictest1654-21cut07:40:222
2173399528,21cyclictest25-21ksoftirqd/112:18:001
2173399528,21cyclictest25-21ksoftirqd/112:18:001
2173399523,19cyclictest10351-21ntp_states08:05:331
21733995231,10cyclictest0-21swapper/107:35:061
21733995230,12cyclictest0-21swapper/107:25:051
21733995220,9cyclictest0-21swapper/111:15:251
21733995220,9cyclictest0-21swapper/111:15:251
2173399519,10cyclictest19643-21cat08:35:211
21733995136,9cyclictest17676-21idleruntime-cro08:30:031
2173399508,11cyclictest25-21ksoftirqd/112:00:341
2173399504,18cyclictest30579-21sh10:01:261
2173399504,18cyclictest30579-21sh10:01:261
21733995016,17cyclictest0-21swapper/107:48:361
2173399498,12cyclictest29027-21diskmemload10:40:581
2173399498,11cyclictest12042-21ssh09:30:271
2173399498,11cyclictest12042-21ssh09:30:271
21733994937,7cyclictest0-21swapper/107:39:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional