You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:09
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Fri May 02, 2025 00:44:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1721421080,8sleep30-21swapper/320:50:233
14665210411,60sleep014742-21tune2fs19:05:240
159889910055,13cyclictest181rcu_preempt20:29:582
15988999541,30cyclictest0-21swapper/222:20:212
15988999541,30cyclictest0-21swapper/222:20:212
15988999472,8cyclictest181rcu_preempt22:19:482
15988999434,25cyclictest0-21swapper/220:30:142
15988999434,25cyclictest0-21swapper/220:30:142
15988999256,13cyclictest181rcu_preempt19:30:192
15988999235,15cyclictest24332-21ssh22:04:422
15988999235,15cyclictest24332-21ssh22:04:422
15967999268,13cyclictest0-21swapper/020:39:590
15967999268,13cyclictest0-21swapper/020:39:590
15988999155,13cyclictest28816-21ssh23:03:132
15988999155,13cyclictest28816-21ssh23:03:132
15988999027,18cyclictest0-21swapper/221:54:452
15979999068,14cyclictest26321-21sh21:14:591
15979999068,14cyclictest26321-21sh21:14:591
15967999072,13cyclictest15957-21cyclictest21:29:570
1598899892,23cyclictest11526-21runrttasks22:35:312
1598899892,23cyclictest11526-21runrttasks22:35:312
15967998964,17cyclictest0-21swapper/000:24:570
15967998956,25cyclictest18724-21latency_hist23:39:590
15967998956,25cyclictest18724-21latency_hist23:39:590
151742896,55sleep115214-21switchtime19:05:421
15988998861,11cyclictest181rcu_preempt00:20:352
15988998856,9cyclictest181rcu_preempt22:53:492
15988998856,9cyclictest181rcu_preempt22:53:492
15988998853,12cyclictest181rcu_preempt23:25:342
145852887,54sleep214628-21df19:05:182
1599799873,45cyclictest896-21kworker/3:1+pm21:20:183
1599799873,45cyclictest896-21kworker/3:1+pm21:20:183
15988998764,8cyclictest181rcu_preempt22:41:352
15988998751,14cyclictest181rcu_preempt21:46:192
15988998735,13cyclictest0-21swapper/200:28:052
15967998766,12cyclictest0-21swapper/020:20:220
15967998766,12cyclictest0-21swapper/020:20:220
15967998765,12cyclictest0-21swapper/019:45:230
15967998753,28cyclictest330-21systemd-logind22:54:560
15967998753,28cyclictest330-21systemd-logind22:54:560
15988998655,9cyclictest4058-21ssh00:08:312
1597999868,30cyclictest31-21ksoftirqd/121:35:431
15988998541,10cyclictest0-21swapper/222:58:472
15988998523,19cyclictest0-21swapper/223:16:122
1596799853,29cyclictest15-21ksoftirqd/022:22:450
1596799853,29cyclictest15-21ksoftirqd/022:22:450
59032840,8sleep00-21swapper/021:34:330
15988998443,9cyclictest0-21swapper/200:01:502
15967998466,12cyclictest23102-21diskmemload21:37:570
15988998362,15cyclictest31434-21sh21:22:242
15988998362,15cyclictest31434-21sh21:22:242
15988998351,16cyclictest181rcu_preempt21:29:592
15988998345,9cyclictest181rcu_preempt23:56:032
15988998345,9cyclictest181rcu_preempt23:56:032
15988998337,22cyclictest2193-21kworker/u17:1-nfsiod21:30:112
15988998327,17cyclictest0-21swapper/222:08:512
15967998317,34cyclictest15-21ksoftirqd/020:58:250
1464728319,43sleep30-21swapper/319:05:233
15988998255,11cyclictest181rcu_preempt20:00:272
15988998255,11cyclictest181rcu_preempt20:00:272
15988998247,8cyclictest181rcu_preempt00:18:102
15988998247,8cyclictest181rcu_preempt00:18:102
15988998246,7cyclictest181rcu_preempt23:44:122
15988998246,7cyclictest181rcu_preempt23:44:122
15988998241,9cyclictest0-21swapper/200:35:342
15988998239,16cyclictest15957-21cyclictest21:09:582
15988998222,19cyclictest0-21swapper/221:35:222
1596799825,47cyclictest15-21ksoftirqd/019:27:420
1596799825,43cyclictest15-21ksoftirqd/021:01:370
1596799825,43cyclictest15-21ksoftirqd/021:01:370
1599799814,35cyclictest9782-21chrt23:25:183
1599799813,46cyclictest896-21kworker/3:1+pm20:15:113
15988998154,11cyclictest181rcu_preempt20:24:582
15988998154,11cyclictest181rcu_preempt20:24:582
15988998144,16cyclictest181rcu_preempt23:51:332
15988998144,16cyclictest181rcu_preempt23:51:332
15988998143,12cyclictest181rcu_preempt23:39:582
15988998143,12cyclictest181rcu_preempt23:39:582
15988998136,13cyclictest181rcu_preempt22:31:462
15988998125,12cyclictest0-21swapper/223:30:362
15967998143,7cyclictest181rcu_preempt23:44:210
15967998143,7cyclictest181rcu_preempt23:44:210
15967998131,24cyclictest181rcu_preempt20:19:480
1596799812,34cyclictest15-21ksoftirqd/022:03:470
1596799812,34cyclictest15-21ksoftirqd/022:03:470
312862802,20sleep031295-21ssh23:06:400
15988998035,16cyclictest181rcu_preempt21:14:142
15988998035,16cyclictest181rcu_preempt21:14:142
15988998029,12cyclictest0-21swapper/220:39:582
15988998029,12cyclictest0-21swapper/220:39:582
15988998028,29cyclictest0-21swapper/200:10:162
1598899802,53cyclictest22260-21ssh23:45:332
1598899802,53cyclictest22260-21ssh23:45:332
1597999805,28cyclictest31-21ksoftirqd/122:26:291
15967998064,9cyclictest0-21swapper/019:59:590
15967998047,27cyclictest330-21systemd-logind23:54:550
15967998047,27cyclictest330-21systemd-logind23:54:550
15967998043,16cyclictest12355-21latency_hist21:44:580
1599799792,55cyclictest22261-21chrt23:45:333
1599799792,55cyclictest22261-21chrt23:45:333
1599799792,48cyclictest896-21kworker/3:1+pm20:20:363
1599799792,48cyclictest896-21kworker/3:1+pm20:20:363
15988997955,18cyclictest18582-21cpuspeed20:55:172
15988997946,8cyclictest181rcu_preempt19:14:582
15988997946,11cyclictest181rcu_preempt21:40:292
15988997942,8cyclictest181rcu_preempt22:47:572
15988997929,14cyclictest0-21swapper/222:25:592
15988997923,16cyclictest0-21swapper/221:57:592
15988997923,16cyclictest0-21swapper/221:57:592
15967997966,7cyclictest0-21swapper/023:10:320
1596799796,37cyclictest15-21ksoftirqd/019:22:320
15967997951,8cyclictest181rcu_preempt19:15:130
15967997951,8cyclictest181rcu_preempt19:15:130
15967997946,7cyclictest181rcu_preempt20:06:240
15967997940,31cyclictest31940-21ntp_kernel_pll_00:00:330
1599799784,45cyclictest896-21kworker/3:1+pm22:00:233
1599799784,45cyclictest896-21kworker/3:1+pm22:00:233
1599799783,44cyclictest896-21kworker/3:1+pm23:30:113
15988997853,17cyclictest25092-21fschecks_time19:35:182
15988997853,17cyclictest25092-21fschecks_time19:35:182
15988997847,13cyclictest181rcu_preempt22:14:592
15988997844,9cyclictest181rcu_preempt00:34:592
15988997837,10cyclictest181rcu_preempt19:49:182
15988997829,14cyclictest0-21swapper/220:50:212
15967997862,10cyclictest26254-21ssh21:14:580
15967997862,10cyclictest26254-21ssh21:14:580
1599799773,33cyclictest16257-21chrt00:29:553
1599799772,32cyclictest11014-21chrt20:30:333
1599799772,32cyclictest11014-21chrt20:30:333
15988997736,11cyclictest31289-21latency_hist19:54:592
15988997726,24cyclictest0-21swapper/219:19:592
15988997726,24cyclictest0-21swapper/219:19:592
15988997724,18cyclictest0-21swapper/220:15:172
15988997713,18cyclictest0-21swapper/221:19:492
15967997739,9cyclictest181rcu_preempt00:14:270
15967997739,8cyclictest181rcu_preempt22:28:150
1599799763,29cyclictest26809-21chrt21:15:193
15988997639,13cyclictest181rcu_preempt20:13:492
15988997638,8cyclictest181rcu_preempt23:10:372
15988997629,26cyclictest0-21swapper/223:07:182
15988997626,18cyclictest0-21swapper/219:40:262
15979997646,7cyclictest181rcu_preempt19:52:481
15967997648,7cyclictest181rcu_preempt23:33:010
15967997640,10cyclictest181rcu_preempt21:59:060
15967997640,10cyclictest181rcu_preempt21:59:060
15967997638,7cyclictest181rcu_preempt23:58:500
15967997638,7cyclictest181rcu_preempt23:58:500
1596799763,27cyclictest15-21ksoftirqd/023:16:130
15967997631,11cyclictest181rcu_preempt22:38:060
15967997631,11cyclictest181rcu_preempt22:38:060
15967997628,9cyclictest181rcu_preempt00:25:580
1599799754,24cyclictest28389-21chrt22:10:363
1599799753,23cyclictest19585-21chrt23:40:303
1599799753,23cyclictest19585-21chrt23:40:303
15988997524,10cyclictest21042-21unixbench_multi21:00:372
15988997524,10cyclictest21042-21unixbench_multi21:00:372
15988997518,27cyclictest0-21swapper/220:45:092
15979997555,11cyclictest15304-21apt-get20:45:171
1597999753,26cyclictest31-21ksoftirqd/100:15:271
1597999753,26cyclictest31-21ksoftirqd/100:15:271
15967997547,10cyclictest181rcu_preempt00:15:310
15967997547,10cyclictest181rcu_preempt00:15:310
15967997529,7cyclictest181rcu_preempt22:07:440
15967997528,17cyclictest181rcu_preempt23:03:090
15967997528,17cyclictest181rcu_preempt23:03:090
15967997513,29cyclictest15-21ksoftirqd/020:45:370
1599799743,44cyclictest896-21kworker/3:1+pm22:59:583
1599799743,34cyclictest13670-21taskset20:40:063
1599799743,34cyclictest13670-21taskset20:40:063
15997997411,46cyclictest0-21swapper/322:15:103
15988997452,8cyclictest181rcu_preempt23:20:342
15988997452,8cyclictest181rcu_preempt23:20:342
15988997434,11cyclictest14215-21ntp_kernel_pll_20:40:302
15988997434,11cyclictest14215-21ntp_kernel_pll_20:40:302
15988997431,22cyclictest0-21swapper/219:20:142
15988997426,15cyclictest0-21swapper/219:59:152
15979997446,19cyclictest29644-21grep19:49:591
1597999744,31cyclictest31-21ksoftirqd/121:57:551
1597999744,31cyclictest31-21ksoftirqd/121:57:551
1597999743,35cyclictest31-21ksoftirqd/121:25:311
1596799746,33cyclictest15-21ksoftirqd/019:52:240
1596799745,36cyclictest15-21ksoftirqd/023:49:360
1596799745,36cyclictest15-21ksoftirqd/023:49:360
15967997447,8cyclictest181rcu_preempt21:45:000
15967997446,22cyclictest9401-21latency_hist23:24:580
15967997446,22cyclictest9401-21latency_hist23:24:580
15967997427,10cyclictest181rcu_preempt00:07:190
1596799742,36cyclictest15-21ksoftirqd/022:10:220
15967997416,29cyclictest15-21ksoftirqd/023:27:460
1599799736,38cyclictest0-21swapper/320:35:273
1599799736,38cyclictest0-21swapper/320:35:273
1599799733,43cyclictest896-21kworker/3:1+pm21:40:243
1599799733,30cyclictest30307-21chrt23:05:243
1599799733,23cyclictest18169-21chrt23:39:083
1599799733,23cyclictest18169-21chrt23:39:083
15988997320,7cyclictest0-21swapper/220:05:342
15979997355,11cyclictest30967-40logrotate23:59:571
15979997355,11cyclictest30967-40logrotate23:59:571
15979997354,13cyclictest23102-21diskmemload23:19:551
15979997339,10cyclictest21896-21ssh22:00:181
15979997339,10cyclictest21896-21ssh22:00:181
1597999733,31cyclictest31-21ksoftirqd/122:16:521
1597999733,31cyclictest31-21ksoftirqd/120:23:011
1597999733,31cyclictest31-21ksoftirqd/120:23:011
1597999732,27cyclictest31-21ksoftirqd/121:00:411
1597999732,27cyclictest31-21ksoftirqd/121:00:411
15967997352,12cyclictest0-21swapper/022:55:130
15967997343,7cyclictest181rcu_preempt22:30:180
1599799723,42cyclictest896-21kworker/3:1+pm19:39:593
1599799723,42cyclictest896-21kworker/3:1+pm19:39:593
1599799723,24cyclictest2361-21chrt20:05:213
1599799722,45cyclictest896-21kworker/3:1+pm22:09:593
15988997221,17cyclictest0-21swapper/219:26:002
1597999728,24cyclictest31-21ksoftirqd/122:08:451
1597999723,28cyclictest31-21ksoftirqd/123:22:481
1597999723,28cyclictest31-21ksoftirqd/123:22:481
1597999723,26cyclictest31-21ksoftirqd/122:30:281
1597999722,27cyclictest31-21ksoftirqd/100:00:271
1596799729,33cyclictest15-21ksoftirqd/020:44:480
1596799729,33cyclictest15-21ksoftirqd/020:44:480
1596799726,33cyclictest15-21ksoftirqd/020:33:460
1596799726,33cyclictest15-21ksoftirqd/020:33:460
15967997258,8cyclictest16948-21timerandwakeup19:10:340
15967997247,7cyclictest181rcu_preempt22:41:380
15967997247,7cyclictest181rcu_preempt00:34:260
15967997243,10cyclictest181rcu_preempt22:49:520
15967997238,12cyclictest0-21swapper/020:50:220
15967997238,10cyclictest181rcu_preempt22:18:440
15967997237,11cyclictest181rcu_preempt21:23:570
15967997237,11cyclictest181rcu_preempt21:23:570
1599799713,36cyclictest896-21kworker/3:1+pm21:14:593
1599799713,36cyclictest896-21kworker/3:1+pm21:14:593
1597999717,24cyclictest31-21ksoftirqd/123:14:431
15979997151,14cyclictest18404-21threads22:45:321
15979997138,13cyclictest16621-21ssh22:43:221
1596799715,36cyclictest15-21ksoftirqd/019:33:260
15967997136,10cyclictest181rcu_preempt20:10:200
15967997130,9cyclictest181rcu_preempt00:35:540
1596799712,37cyclictest15-21ksoftirqd/020:03:570
1596799712,37cyclictest15-21ksoftirqd/020:03:570
1596799712,37cyclictest15-21ksoftirqd/019:37:390
1596799712,37cyclictest15-21ksoftirqd/019:37:390
1596799711,34cyclictest15-21ksoftirqd/021:16:240
15967997110,33cyclictest15-21ksoftirqd/021:50:010
2718127049,12sleep127214-21ssh21:15:301
1599799703,39cyclictest896-21kworker/3:1+pm19:55:173
1599799703,22cyclictest27042-21chrt19:40:343
1599799702,48cyclictest896-21kworker/3:1+pm23:10:193
1599799702,25cyclictest18170-21chrt19:15:093
1599799702,25cyclictest18170-21chrt19:15:093
1599799702,22cyclictest22221-21chrt19:25:513
1599799702,19cyclictest28260-21chrt19:45:203
1597999709,30cyclictest31-21ksoftirqd/120:07:521
15979997054,11cyclictest15957-21cyclictest19:20:191
15979997053,12cyclictest24513-21threads22:55:321
15979997052,10cyclictest0-21swapper/121:20:341
15979997052,10cyclictest0-21swapper/121:20:341
15979997042,20cyclictest15015-21cron20:44:551
15979997042,20cyclictest15015-21cron20:44:551
15979997040,9cyclictest17916-21ssh23:38:271
15979997040,9cyclictest17916-21ssh23:38:271
1597999703,26cyclictest11329-21ssh22:35:021
1597999703,26cyclictest11329-21ssh22:35:021
1597999702,34cyclictest31-21ksoftirqd/123:47:501
1597999702,34cyclictest31-21ksoftirqd/123:47:501
1597999702,29cyclictest31-21ksoftirqd/119:38:361
1597999702,29cyclictest31-21ksoftirqd/119:38:361
1597999702,27cyclictest31-21ksoftirqd/120:25:131
1597999702,26cyclictest31-21ksoftirqd/123:52:391
1597999702,26cyclictest31-21ksoftirqd/123:52:391
15967997023,19cyclictest15-21ksoftirqd/019:44:340
1599799692,40cyclictest896-21kworker/3:1+pm22:35:153
1599799692,40cyclictest896-21kworker/3:1+pm22:35:153
15997996920,8cyclictest0-21swapper/319:14:563
15997996918,11cyclictest0-21swapper/322:49:593
1597999699,30cyclictest31-21ksoftirqd/100:26:381
15979996956,9cyclictest15957-21cyclictest23:05:351
15979996955,8cyclictest15396-21modprobe23:34:551
15979996941,19cyclictest18272-21modprobe21:54:551
1597999693,35cyclictest31-21ksoftirqd/120:50:291
1597999693,26cyclictest31-21ksoftirqd/121:08:091
1597999692,24cyclictest31-21ksoftirqd/100:30:301
1597999692,23cyclictest31-21ksoftirqd/122:54:331
1597999692,23cyclictest31-21ksoftirqd/122:54:331
1596799692,26cyclictest15-21ksoftirqd/021:08:490
1599799683,43cyclictest896-21kworker/3:1+pm20:25:273
15997996821,10cyclictest0-21swapper/300:09:593
15997996818,10cyclictest0-21swapper/321:35:213
15997996811,13cyclictest0-21swapper/322:29:583
15979996854,10cyclictest15957-21cyclictest00:10:051
1597999684,26cyclictest31-21ksoftirqd/123:40:211
1597999684,26cyclictest31-21ksoftirqd/123:40:211
15979996838,9cyclictest30360-21ssh22:13:521
15979996835,9cyclictest10689-21ssh21:40:511
1597999682,30cyclictest31-21ksoftirqd/123:28:471
15967996837,11cyclictest181rcu_preempt20:28:380
1599799674,43cyclictest896-21kworker/3:1+pm21:50:363
1599799673,36cyclictest896-21kworker/3:1+pm19:50:013
1599799673,26cyclictest1989-21taskset21:27:153
1599799672,29cyclictest21513-21chrt00:38:343
1599799672,22cyclictest5519-21chrt21:33:313
15997996717,10cyclictest774098irq/304-mma8451_consumer021:56:053
15997996717,10cyclictest774098irq/304-mma8451_consumer021:56:053
1597999674,35cyclictest31-21ksoftirqd/100:36:541
15979996739,9cyclictest5066-21ssh21:32:191
15979996739,20cyclictest19504-21modprobe19:19:561
15979996739,20cyclictest19504-21modprobe19:19:561
15979996737,8cyclictest181rcu_preempt00:05:161
15979996737,22cyclictest13438-21ls21:45:341
1599799669,13cyclictest0-21swapper/300:30:183
1599799667,15cyclictest0-21swapper/300:00:303
1599799663,43cyclictest896-21kworker/3:1+pm19:24:553
1599799663,19cyclictest15145-21chrt21:49:443
1599799663,16cyclictest0-21swapper/322:34:593
1599799662,28cyclictest30909-21chrt23:59:523
1599799662,28cyclictest30909-21chrt23:59:523
15997996611,11cyclictest0-21swapper/320:45:333
1597999669,31cyclictest31-21ksoftirqd/119:42:441
15979996623,18cyclictest31-21ksoftirqd/100:24:441
1597999662,24cyclictest31-21ksoftirqd/120:55:421
1597999662,24cyclictest31-21ksoftirqd/120:04:371
1597999662,24cyclictest31-21ksoftirqd/120:04:371
1599799659,13cyclictest0-21swapper/300:20:483
1599799653,42cyclictest896-21kworker/3:1+pm22:40:293
15997996516,12cyclictest0-21swapper/323:20:153
15997996516,12cyclictest0-21swapper/323:20:153
15997996511,11cyclictest0-21swapper/323:51:463
15997996511,11cyclictest0-21swapper/323:51:463
15997996511,11cyclictest0-21swapper/322:50:223
15997996511,11cyclictest0-21swapper/322:50:223
15979996551,10cyclictest58-21kcompactd022:22:051
15979996551,10cyclictest58-21kcompactd022:22:051
1597999651,26cyclictest31-21ksoftirqd/119:34:191
1599799642,22cyclictest3270-21taskset22:21:063
1599799642,22cyclictest3270-21taskset22:21:063
15997996415,12cyclictest0-21swapper/300:15:323
15997996415,12cyclictest0-21swapper/300:15:323
15997996415,11cyclictest0-21swapper/320:55:173
15979996449,8cyclictest0-21swapper/120:39:561
15979996449,8cyclictest0-21swapper/120:39:561
15979996448,8cyclictest0-21swapper/120:00:001
15979996435,20cyclictest5088-21modprobe20:14:551
1597999642,25cyclictest31-21ksoftirqd/120:15:231
15979996410,18cyclictest26980-21fschecks_count23:00:191
15979996410,18cyclictest26980-21fschecks_count23:00:191
1599799633,20cyclictest5692-21chrt23:18:413
15997996321,11cyclictest0-21swapper/320:11:143
15997996320,10cyclictest0-21swapper/320:04:593
15997996320,10cyclictest0-21swapper/320:04:593
15997996317,14cyclictest0-21swapper/319:30:133
15997996316,10cyclictest0-21swapper/323:00:203
15997996316,10cyclictest0-21swapper/323:00:203
1597999632,24cyclictest31-21ksoftirqd/120:32:061
1597999632,24cyclictest31-21ksoftirqd/120:32:061
1599799625,18cyclictest0-21swapper/300:10:453
1599799622,28cyclictest896-21kworker/3:1-pm21:05:083
15997996210,14cyclictest0-21swapper/321:00:213
15997996210,14cyclictest0-21swapper/321:00:213
15979996248,10cyclictest15957-21cyclictest19:25:551
15979996231,21cyclictest17039-21grep19:10:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional