You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 05:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Fri May 09, 2025 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
146839980983,8081cyclictest23432sleep320:09:283
146839980983,8081cyclictest23432sleep320:09:283
13519215910,114sleep013046-21/usr/sbin/munin19:05:010
1321821169,33sleep213224-21diskstats19:04:512
14464211313,72sleep114524-21cat19:09:311
146619910657,38cyclictest0-21swapper/100:34:311
146529910466,31cyclictest330-21systemd-logind00:39:270
14661999944,48cyclictest21686-21apt-key19:29:301
14661999944,48cyclictest21686-21apt-key19:29:301
14669999845,10cyclictest0-21swapper/223:09:312
14669999845,10cyclictest0-21swapper/223:09:312
14652999762,24cyclictest0-21swapper/020:54:310
14661999673,13cyclictest0-21swapper/120:29:311
14652999676,12cyclictest0-21swapper/023:04:440
14652999676,12cyclictest0-21swapper/023:04:440
14652999672,15cyclictest13588-21cut22:49:290
265492950,6sleep30-21swapper/323:11:053
14669999550,11cyclictest181rcu_preempt23:11:102
14661999569,17cyclictest0-21swapper/119:44:571
14661999569,17cyclictest0-21swapper/119:44:571
14652999459,25cyclictest0-21swapper/020:14:280
14652999459,25cyclictest0-21swapper/020:14:280
14661999347,39cyclictest0-21swapper/121:29:571
14661999174,8cyclictest0-21swapper/120:54:521
14661999174,8cyclictest0-21swapper/120:54:521
14661999174,10cyclictest8212-21runrttasks00:34:571
14661999160,9cyclictest181rcu_preempt20:49:581
14669999057,10cyclictest181rcu_preempt23:38:422
14669999057,10cyclictest181rcu_preempt23:38:422
14661999062,19cyclictest667-21idleruntime-cro20:04:271
14669998942,18cyclictest21890-21ssh23:03:592
14669998942,18cyclictest21890-21ssh23:03:592
1466999893,59cyclictest21574-21apt-get19:29:412
1466999893,59cyclictest21574-21apt-get19:29:412
14661998954,26cyclictest0-21swapper/122:40:101
14661998954,26cyclictest0-21swapper/122:40:101
14652998974,8cyclictest0-21swapper/000:09:270
14652998972,11cyclictest0-21swapper/021:34:570
14652998972,11cyclictest0-21swapper/021:34:570
14652998968,15cyclictest23990-21latency_hist00:04:320
1466999883,62cyclictest20992-21apt-get21:04:522
1466999883,51cyclictest4165-21apt-get23:30:162
14661998871,8cyclictest0-21swapper/123:34:501
14661998871,8cyclictest0-21swapper/123:34:501
14661998845,33cyclictest0-21swapper/122:04:331
14652998873,8cyclictest0-21swapper/021:29:520
14652998860,21cyclictest0-21swapper/019:49:550
14652998860,21cyclictest0-21swapper/019:49:550
1466199874,49cyclictest30179-21grep00:15:041
14652998769,11cyclictest0-21swapper/022:50:110
14652998769,11cyclictest0-21swapper/022:50:110
14669998656,7cyclictest181rcu_preempt22:40:062
14669998656,7cyclictest181rcu_preempt22:40:062
14669998640,7cyclictest181rcu_preempt22:55:132
14661998653,8cyclictest181rcu_preempt21:35:001
14661998653,8cyclictest181rcu_preempt21:35:001
14661998653,10cyclictest181rcu_preempt22:55:481
14669998537,9cyclictest181rcu_preempt19:44:332
14661998562,16cyclictest20848-21munin-run23:59:271
14669998463,9cyclictest39-21ksoftirqd/222:07:292
14669998453,22cyclictest31635-21threads23:20:112
14669998453,22cyclictest31635-21threads23:20:112
1466999842,58cyclictest4109-21apt-get20:14:522
1466999842,58cyclictest4109-21apt-get20:14:522
1466999842,56cyclictest5028-21apt-get22:34:442
1466999842,56cyclictest5028-21apt-get22:34:442
14661998469,8cyclictest0-21swapper/123:39:461
14661998443,12cyclictest181rcu_preempt19:14:361
14661998443,12cyclictest181rcu_preempt19:14:361
14652998470,8cyclictest0-21swapper/023:44:500
14652998470,8cyclictest0-21swapper/023:44:500
14652998468,11cyclictest14628-21cyclictest19:30:200
14652998468,11cyclictest14628-21cyclictest19:30:200
14652998458,18cyclictest20519-21kworker/u18:1-nfsiod21:58:420
14652998458,18cyclictest20519-21kworker/u18:1-nfsiod21:58:420
14652998452,9cyclictest181rcu_preempt19:27:500
14652998452,9cyclictest181rcu_preempt19:27:500
1335228413,40sleep30-21swapper/319:04:563
14669998354,14cyclictest367-21snmpd22:29:382
1466999834,46cyclictest15103-21cat19:10:082
14661998361,8cyclictest181rcu_preempt23:03:401
14661998361,8cyclictest181rcu_preempt23:03:401
14661998360,15cyclictest207-21systemd-journal22:31:541
14661998360,14cyclictest0-21swapper/121:19:531
14661998349,27cyclictest0-21swapper/100:11:511
14661998349,27cyclictest0-21swapper/100:11:511
14661998347,8cyclictest181rcu_preempt22:44:341
14661998345,31cyclictest0-21swapper/119:12:151
14661998340,8cyclictest0-21swapper/123:19:461
14661998340,8cyclictest0-21swapper/123:19:461
14661998340,8cyclictest0-21swapper/121:09:541
14661998340,8cyclictest0-21swapper/121:09:541
14652998336,38cyclictest25063-21modprobe19:39:270
14669998261,13cyclictest13233-21ssh21:49:592
14669998248,9cyclictest181rcu_preempt00:00:162
14669998245,13cyclictest181rcu_preempt22:27:202
14669998245,13cyclictest181rcu_preempt00:16:292
14661998247,8cyclictest181rcu_preempt23:05:401
14661998247,8cyclictest181rcu_preempt23:05:401
14661998245,30cyclictest0-21swapper/119:35:111
14652998261,14cyclictest463-21rs:main0
14652998250,25cyclictest0-21swapper/022:04:310
1465299823,41cyclictest15-21ksoftirqd/019:46:160
1465299823,41cyclictest15-21ksoftirqd/019:46:160
14669998150,14cyclictest367-21snmpd22:49:392
14669998150,14cyclictest367-21snmpd22:49:392
14669998141,13cyclictest181rcu_preempt00:06:362
14661998162,9cyclictest0-21swapper/120:14:321
14661998162,9cyclictest0-21swapper/120:14:321
14661998159,12cyclictest0-21swapper/121:45:021
14661998159,12cyclictest0-21swapper/121:45:021
14652998149,22cyclictest0-21swapper/023:09:290
14652998149,22cyclictest0-21swapper/023:09:290
14652998147,25cyclictest0-21swapper/021:49:150
14652998147,25cyclictest0-21swapper/021:49:150
1465299814,44cyclictest15-21ksoftirqd/022:41:460
1465299814,44cyclictest15-21ksoftirqd/022:41:460
1465299814,42cyclictest15-21ksoftirqd/022:11:200
14669998048,10cyclictest181rcu_preempt21:13:392
14669998048,10cyclictest181rcu_preempt21:13:392
1466199807,35cyclictest31-21ksoftirqd/121:25:001
14661998040,12cyclictest31250-21kworker/u17:0-rpciod00:24:261
14652998063,8cyclictest0-21swapper/019:40:100
14652998060,11cyclictest0-21swapper/020:59:460
1465299803,42cyclictest15-21ksoftirqd/000:32:170
1465299803,40cyclictest15-21ksoftirqd/019:58:350
1465299803,40cyclictest15-21ksoftirqd/019:58:350
14669997943,8cyclictest181rcu_preempt21:46:162
14669997943,8cyclictest181rcu_preempt21:46:162
14669997936,10cyclictest181rcu_preempt20:54:302
14661997951,20cyclictest26932-21idleruntime22:14:521
14661997951,20cyclictest26932-21idleruntime22:14:521
14661997938,10cyclictest22345-21diskmemload00:04:251
14661997934,8cyclictest181rcu_preempt22:10:351
14661997928,11cyclictest181rcu_preempt23:25:011
14652997961,10cyclictest0-21swapper/023:11:030
14652997960,13cyclictest22345-21diskmemload23:19:590
14652997960,13cyclictest22345-21diskmemload23:19:590
1465299794,34cyclictest15-21ksoftirqd/020:34:520
1465299794,34cyclictest15-21ksoftirqd/020:34:520
1465299793,32cyclictest2540-21ssh22:29:430
14669997857,8cyclictest181rcu_preempt20:29:552
14669997850,18cyclictest19196-21cpu20:59:452
14669997845,17cyclictest4318-21/usr/sbin/munin21:35:022
14669997845,17cyclictest4318-21/usr/sbin/munin21:35:022
14661997837,16cyclictest22345-21diskmemload22:21:141
1466199783,37cyclictest18264-21ssh21:59:461
1465299789,35cyclictest15-21ksoftirqd/023:57:360
14652997863,8cyclictest0-21swapper/022:59:280
14652997859,9cyclictest0-21swapper/022:35:280
14652997859,9cyclictest0-21swapper/022:35:280
14652997853,18cyclictest2209-21idleruntime-cro00:24:260
14652997850,21cyclictest14574-21kworker/u17:1+nfsiod22:24:000
1465299783,44cyclictest15-21ksoftirqd/021:19:060
1465299783,29cyclictest15-21ksoftirqd/020:54:450
1465299783,29cyclictest15-21ksoftirqd/020:54:450
1465299782,40cyclictest15-21ksoftirqd/021:24:280
1465299782,37cyclictest15-21ksoftirqd/021:24:580
14669997748,22cyclictest20157-21latency_hist19:24:312
14669997748,22cyclictest20157-21latency_hist19:24:312
14669997745,8cyclictest181rcu_preempt21:42:552
14669997744,8cyclictest181rcu_preempt00:22:072
14669997743,13cyclictest181rcu_preempt20:06:172
14669997743,13cyclictest181rcu_preempt20:06:172
14669997742,14cyclictest181rcu_preempt22:16:112
14669997742,14cyclictest181rcu_preempt22:16:112
14669997739,12cyclictest181rcu_preempt21:21:372
14661997762,8cyclictest0-21swapper/123:54:301
14661997762,8cyclictest0-21swapper/123:54:301
14661997757,10cyclictest0-21swapper/100:05:361
14661997756,15cyclictest461-21in:imuxsock19:31:541
14661997756,15cyclictest461-21in:imuxsock19:31:541
14661997736,13cyclictest0-21swapper/120:09:301
14661997736,13cyclictest0-21swapper/120:09:301
14652997759,11cyclictest0-21swapper/021:10:030
14652997759,11cyclictest0-21swapper/021:10:030
14652997757,11cyclictest0-21swapper/023:32:260
14652997722,25cyclictest15-21ksoftirqd/020:05:240
14652997722,25cyclictest15-21ksoftirqd/020:05:240
14652997714,26cyclictest15-21ksoftirqd/021:43:260
14669997649,21cyclictest15048-21latency_hist23:49:302
14669997649,21cyclictest15048-21latency_hist23:49:302
14669997635,12cyclictest181rcu_preempt20:10:342
14669997635,12cyclictest181rcu_preempt20:10:342
14669997633,10cyclictest32048-21kworker/u18:1-events_unbound23:55:012
14661997661,8cyclictest0-21swapper/119:20:001
14661997661,8cyclictest0-21swapper/119:20:001
14661997657,11cyclictest0-21swapper/119:59:301
14661997657,11cyclictest0-21swapper/119:59:301
14661997639,10cyclictest17491-21kworker/u17:1-rpciod21:18:261
1466199762,36cyclictest31-21ksoftirqd/123:15:041
14652997655,15cyclictest3642-21apt-key23:29:290
14652997651,19cyclictest367-21snmpd20:30:020
1465299763,36cyclictest15-21ksoftirqd/000:14:310
1465299763,36cyclictest15-21ksoftirqd/000:14:310
1465299762,35cyclictest15-21ksoftirqd/023:36:430
1465299762,35cyclictest15-21ksoftirqd/023:36:430
325362750,5sleep30-21swapper/322:24:513
14683997541,26cyclictest461ktimers/322:34:443
14683997541,26cyclictest461ktimers/322:34:443
14669997555,8cyclictest39-21ksoftirqd/223:44:052
14669997545,9cyclictest181rcu_preempt19:59:572
14669997537,8cyclictest181rcu_preempt19:56:362
14669997537,8cyclictest181rcu_preempt19:56:362
14669997537,10cyclictest181rcu_preempt20:35:022
14669997537,10cyclictest181rcu_preempt20:35:022
14669997536,10cyclictest181rcu_preempt21:15:042
14661997559,9cyclictest0-21swapper/119:40:031
14661997558,11cyclictest359-21ntpd22:50:021
14661997558,11cyclictest359-21ntpd22:50:021
14661997545,10cyclictest181rcu_preempt21:59:311
14661997545,10cyclictest181rcu_preempt21:59:311
14652997558,10cyclictest0-21swapper/023:51:440
14652997558,10cyclictest0-21swapper/023:51:440
14652997557,8cyclictest0-21swapper/021:05:190
14652997556,9cyclictest0-21swapper/021:50:080
1465299754,42cyclictest4605-21apt-get20:14:530
1465299754,42cyclictest4605-21apt-get20:14:530
14669997441,9cyclictest181rcu_preempt20:24:482
14669997438,9cyclictest181rcu_preempt00:09:462
14669997438,9cyclictest181rcu_preempt00:09:462
14669997434,8cyclictest181rcu_preempt21:55:012
14669997434,8cyclictest181rcu_preempt21:55:012
14661997443,8cyclictest181rcu_preempt23:45:011
14661997443,8cyclictest181rcu_preempt23:45:011
14661997421,21cyclictest20704-21dpkg21:04:301
14652997443,20cyclictest0-21swapper/022:19:310
14652997443,20cyclictest0-21swapper/022:19:310
14652997429,8cyclictest181rcu_preempt20:19:590
14669997349,12cyclictest181rcu_preempt20:40:042
14669997342,7cyclictest181rcu_preempt21:28:112
14669997342,11cyclictest181rcu_preempt20:55:102
14669997342,11cyclictest181rcu_preempt20:55:102
14669997341,8cyclictest181rcu_preempt23:51:212
14669997341,8cyclictest181rcu_preempt23:51:212
14669997334,10cyclictest181rcu_preempt22:09:442
14669997333,12cyclictest181rcu_preempt23:29:302
14669997332,13cyclictest181rcu_preempt00:29:302
14669997324,27cyclictest0-21swapper/220:45:072
1466999732,32cyclictest39-21ksoftirqd/222:00:592
14661997358,8cyclictest0-21swapper/119:50:091
14661997358,8cyclictest0-21swapper/119:50:091
14661997337,7cyclictest0-21swapper/122:35:381
14661997337,7cyclictest0-21swapper/122:35:381
14661997334,11cyclictest207-21systemd-journal21:41:551
1466199733,35cyclictest4165-21apt-get23:29:411
14669997233,9cyclictest181rcu_preempt19:44:472
14669997233,9cyclictest181rcu_preempt19:44:472
14669997227,20cyclictest28848-21timerwakeupswit23:15:062
1466199728,35cyclictest31-21ksoftirqd/120:29:551
14661997226,9cyclictest181rcu_preempt21:49:501
14652997245,10cyclictest15644-21kworker/u19:2-rpciod00:18:320
1465299723,33cyclictest15-21ksoftirqd/020:04:310
14669997147,18cyclictest11660-21netstat22:44:582
14669997146,20cyclictest29912-21latency_hist19:54:292
14669997146,20cyclictest29912-21latency_hist19:54:292
14669997132,9cyclictest181rcu_preempt19:14:392
14669997132,9cyclictest181rcu_preempt19:14:392
14669997122,21cyclictest0-21swapper/200:34:542
14669997111,26cyclictest39-21ksoftirqd/222:21:282
14661997157,7cyclictest0-21swapper/121:05:231
14661997142,7cyclictest181rcu_preempt20:47:261
14652997156,8cyclictest0-21swapper/022:25:000
14652997145,19cyclictest0-21swapper/023:18:590
14652997134,9cyclictest181rcu_preempt23:40:260
14669997045,10cyclictest181rcu_preempt19:35:032
14669997042,9cyclictest181rcu_preempt00:29:492
14669997037,16cyclictest181rcu_preempt21:29:572
14661997056,9cyclictest25591-21kernelversion23:10:021
14661997039,12cyclictest12676-21/usr/sbin/munin20:39:461
14661997029,8cyclictest181rcu_preempt22:28:221
14652997034,26cyclictest14954-21apt-get19:09:480
14652997034,10cyclictest21751-21ssh22:05:040
1468399693,36cyclictest0-21swapper/323:30:083
14652996950,9cyclictest0-21swapper/019:14:430
14652996950,9cyclictest0-21swapper/019:14:430
14652996918,25cyclictest15-21ksoftirqd/019:21:410
14652996918,25cyclictest15-21ksoftirqd/019:21:410
14669996842,8cyclictest181rcu_preempt19:28:162
14669996842,8cyclictest181rcu_preempt19:28:162
14669996837,10cyclictest181rcu_preempt20:19:582
14652996853,8cyclictest0-21swapper/020:39:520
14652996849,9cyclictest0-21swapper/020:44:500
14652996836,10cyclictest58-21kcompactd020:26:110
14661996733,8cyclictest181rcu_preempt20:19:431
14661996646,10cyclictest0-21swapper/120:19:111
14661996646,10cyclictest0-21swapper/120:19:111
14661996634,8cyclictest181rcu_preempt00:26:191
14661996526,7cyclictest181rcu_preempt20:39:311
14661996526,7cyclictest181rcu_preempt20:39:311
1468399639,43cyclictest0-21swapper/319:10:023
1468399633,48cyclictest896-21kworker/3:1+events23:52:583
1468399633,48cyclictest896-21kworker/3:1+events23:52:583
1468399613,43cyclictest896-21kworker/3:1+events22:34:293
1468399613,39cyclictest896-21kworker/3:1+events21:59:313
1468399602,40cyclictest896-21kworker/3:1+pm21:25:103
86142590,4sleep3451rcuc/322:39:563
86142590,4sleep3451rcuc/322:39:563
1468399592,35cyclictest896-21kworker/3:1+events19:54:513
1468399592,35cyclictest896-21kworker/3:1+events19:54:513
1468399572,38cyclictest0-21swapper/321:04:523
1468399572,33cyclictest896-21kworker/3:1+pm21:09:423
1468399572,33cyclictest896-21kworker/3:1+pm21:09:423
1468399563,42cyclictest17326-21chrt20:54:303
1468399553,36cyclictest0-21swapper/320:14:483
1468399553,36cyclictest0-21swapper/320:14:483
1468399552,39cyclictest9879-21taskset20:29:553
14683995514,23cyclictest0-21swapper/319:29:413
14683995514,23cyclictest0-21swapper/319:29:413
14683995512,38cyclictest0-21swapper/320:10:073
14683995512,38cyclictest0-21swapper/320:10:073
1468399546,35cyclictest0-21swapper/300:05:013
1468399543,37cyclictest24746-21chrt23:09:103
1468399543,37cyclictest24746-21chrt23:09:103
1468399533,34cyclictest4057-21taskset23:29:333
1468399533,31cyclictest6765-21taskset20:19:543
1468399532,31cyclictest896-21kworker/3:1+pm23:44:403
1468399532,31cyclictest896-21kworker/3:1+pm23:44:403
77932510,4sleep30-21swapper/300:34:453
1468399513,38cyclictest29470-21chrt21:20:073
1468399513,33cyclictest13802-21chrt22:49:323
1468399513,32cyclictest22294-21chrt00:00:113
1468399512,35cyclictest896-21kworker/3:1+pm21:35:293
1468399512,35cyclictest896-21kworker/3:1+pm21:35:293
1468399506,34cyclictest6409-21chrt00:30:563
1468399503,37cyclictest25118-21chrt22:10:493
1468399503,35cyclictest11749-21chrt23:42:513
1468399502,36cyclictest3445-21taskset21:32:393
1468399493,35cyclictest18394-21taskset19:19:293
1468399493,35cyclictest18394-21taskset19:19:293
1468399493,34cyclictest896-21kworker/3:1+events23:36:193
1468399493,34cyclictest896-21kworker/3:1+events23:36:193
1468399493,34cyclictest18435-21chrt21:59:513
1468399492,36cyclictest12579-21taskset20:39:313
1468399492,36cyclictest12579-21taskset20:39:313
1468399492,34cyclictest13301-21taskset21:50:023
1468399492,32cyclictest15148-21taskset20:45:063
1468399484,27cyclictest896-21kworker/3:1+events19:39:403
1468399483,33cyclictest21402-21chrt22:04:533
1468399483,28cyclictest896-21kworker/3:1+events19:49:403
1468399483,28cyclictest896-21kworker/3:1+events19:49:403
1468399482,35cyclictest15778-21taskset21:54:483
1468399482,32cyclictest8425-21chrt21:40:463
1468399482,31cyclictest5009-21taskset00:29:303
14683994823,16cyclictest461ktimers/319:59:563
1468399473,32cyclictest28463-21chrt00:12:113
1468399473,32cyclictest28463-21chrt00:12:113
1468399473,30cyclictest15628-21taskset22:52:083
1468399473,30cyclictest15628-21taskset22:52:083
1468399473,29cyclictest19267-21taskset22:59:303
1468399472,32cyclictest30107-21chrt00:15:033
1468399472,32cyclictest29713-21chrt22:19:493
1468399472,31cyclictest29279-21chrt19:50:083
1468399472,31cyclictest29279-21chrt19:50:083
1468399472,30cyclictest9134-21chrt20:28:533
1468399463,28cyclictest32480-21chrt00:19:493
1468399462,32cyclictest21760-21chrt23:03:353
1468399462,32cyclictest21760-21chrt23:03:353
14683994623,16cyclictest461ktimers/323:59:273
1468399462,29cyclictest896-21kworker/3:1+events21:15:303
1468399453,29cyclictest19135-21chrt19:19:533
1468399453,29cyclictest19135-21chrt19:19:533
1468399443,29cyclictest18415-21taskset20:55:073
1468399443,29cyclictest18415-21taskset20:55:073
1468399443,28cyclictest30019-21taskset23:18:183
1468399442,32cyclictest11985-21taskset21:48:163
1468399442,32cyclictest11985-21taskset21:48:163
14683994420,17cyclictest461ktimers/322:17:543
14683994420,17cyclictest461ktimers/322:17:543
14683994417,17cyclictest461ktimers/323:24:333
14683994417,17cyclictest461ktimers/323:24:333
1468399433,27cyclictest13195-21chrt20:39:553
14683994319,16cyclictest461ktimers/321:00:023
14683994318,18cyclictest461ktimers/319:34:563
1468399423,28cyclictest21361-21chrt19:26:543
1468399423,28cyclictest21361-21chrt19:26:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional