You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-11 - 19:12
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun May 11, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1529699963,24cyclictest25007-21/usr/sbin/munin10:14:390
15300999574,11cyclictest0-21swapper/112:14:461
15300999574,11cyclictest0-21swapper/112:14:461
15305999451,8cyclictest181rcu_preempt07:59:562
3282930,7sleep1334-21fschecks_time07:59:421
15300999371,10cyclictest0-21swapper/109:44:381
15296999254,9cyclictest181rcu_preempt08:29:130
15305999171,11cyclictest0-21swapper/210:09:452
15305999169,13cyclictest0-21swapper/208:44:202
15305999161,20cyclictest24454-21cut07:34:422
15305999067,12cyclictest0-21swapper/212:04:362
15305999067,12cyclictest0-21swapper/212:04:362
15305998971,8cyclictest0-21swapper/208:24:192
15305998968,11cyclictest0-21swapper/209:04:572
15305998870,10cyclictest0-21swapper/211:34:352
15305998870,10cyclictest0-21swapper/211:34:352
15296998758,8cyclictest181rcu_preempt09:12:520
15296998751,30cyclictest10786-21latency_hist09:49:190
15305998671,9cyclictest0-21swapper/208:59:582
15296998659,7cyclictest181rcu_preempt11:32:340
15305998570,8cyclictest0-21swapper/208:09:412
15305998570,8cyclictest0-21swapper/208:09:412
15305998567,8cyclictest0-21swapper/210:04:562
1529699859,37cyclictest15-21ksoftirqd/012:12:130
15305998467,8cyclictest0-21swapper/210:34:312
15305998364,13cyclictest8144-21latency_hist09:44:212
15305998351,8cyclictest181rcu_preempt08:04:442
15296998355,19cyclictest32735-21fschecks_count07:59:410
145652837,53sleep114629-21switchtime07:05:191
15305998258,16cyclictest11371-21idleruntime-cro11:49:172
15305998258,16cyclictest11371-21idleruntime-cro11:49:172
15300998264,8cyclictest0-21swapper/109:39:451
15296998244,22cyclictest11110-21ssh09:49:390
15296998244,22cyclictest11110-21ssh09:49:390
15296998239,9cyclictest181rcu_preempt10:25:260
15305998160,16cyclictest15287-21cyclictest10:49:382
15305998144,14cyclictest181rcu_preempt11:08:232
15305998144,14cyclictest181rcu_preempt11:08:232
15296998161,7cyclictest181rcu_preempt10:36:140
15296998160,7cyclictest181rcu_preempt08:40:470
1529699813,58cyclictest27697-21apt-get07:44:320
15305998058,15cyclictest14005-21kworker/u17:1-xprtiod11:19:572
15305998050,10cyclictest181rcu_preempt10:31:542
15300998059,9cyclictest0-21swapper/112:09:201
15300998059,9cyclictest0-21swapper/112:09:201
1529699806,37cyclictest15-21ksoftirqd/009:02:090
15296998047,7cyclictest181rcu_preempt10:06:570
1529699803,55cyclictest28826-21apt-get12:20:010
1441128011,41sleep30-21swapper/307:05:123
1422228010,38sleep00-21swapper/007:05:050
15305997965,10cyclictest15287-21cyclictest09:20:002
15305997965,10cyclictest15287-21cyclictest09:20:002
15296997960,7cyclictest181rcu_preempt11:13:460
15296997954,12cyclictest181rcu_preempt10:44:580
15296997936,28cyclictest15287-21cyclictest10:00:160
15296997936,28cyclictest15287-21cyclictest10:00:160
15296997918,32cyclictest15-21ksoftirqd/008:55:290
1463027915,32sleep20-21swapper/207:05:202
15305997863,11cyclictest15287-21cyclictest12:35:002
15305997843,11cyclictest8708-21ssh10:44:022
15300997860,14cyclictest23953-21kworker/u19:2+events_unbound10:34:171
15296997857,15cyclictest330-21systemd-logind07:44:180
15296997857,15cyclictest330-21systemd-logind07:44:180
15296997856,8cyclictest181rcu_preempt11:43:300
15296997848,11cyclictest181rcu_preempt07:20:070
15296997847,8cyclictest181rcu_preempt11:49:200
15296997847,8cyclictest181rcu_preempt11:49:200
15296997843,8cyclictest181rcu_preempt09:22:060
15296997843,8cyclictest181rcu_preempt09:22:060
15296997838,7cyclictest181rcu_preempt12:28:410
15305997754,7cyclictest181rcu_preempt11:32:112
15296997755,7cyclictest181rcu_preempt11:21:530
15296997744,23cyclictest0-21swapper/008:19:430
15296997743,7cyclictest181rcu_preempt10:50:270
15296997741,7cyclictest181rcu_preempt10:43:000
15296997738,7cyclictest181rcu_preempt11:58:010
15296997738,7cyclictest181rcu_preempt11:58:010
15296997737,7cyclictest181rcu_preempt12:03:140
15305997657,8cyclictest0-21swapper/209:54:562
15305997656,14cyclictest494-21cron07:19:182
15305997656,14cyclictest330-21systemd-logind07:34:192
15305997654,15cyclictest18810-21xargs08:54:572
15305997650,7cyclictest181rcu_preempt09:35:192
15305997645,8cyclictest181rcu_preempt11:16:412
15305997645,7cyclictest181rcu_preempt12:16:352
15305997645,7cyclictest181rcu_preempt12:16:352
15305997639,10cyclictest24296-21ssh09:10:522
15300997662,10cyclictest15287-21cyclictest11:54:551
15300997662,10cyclictest15287-21cyclictest11:54:551
15300997657,12cyclictest11542-21latency_hist11:49:201
15300997657,12cyclictest11542-21latency_hist11:49:201
15300997654,13cyclictest1-21systemd11:19:171
15300997653,16cyclictest9481-21if_err_eth010:44:441
15300997646,21cyclictest18296-21cat07:15:171
15296997653,17cyclictest2170-21netstat08:04:460
15296997643,9cyclictest181rcu_preempt10:12:440
15296997642,7cyclictest181rcu_preempt10:20:010
15296997629,11cyclictest181rcu_preempt07:30:320
15305997554,14cyclictest19490-21sh11:04:172
15305997547,7cyclictest181rcu_preempt09:28:482
15305997547,7cyclictest181rcu_preempt09:16:592
15305997547,7cyclictest181rcu_preempt09:16:592
15305997537,9cyclictest181rcu_preempt12:23:292
15296997556,12cyclictest330-21systemd-logind09:59:180
15296997553,7cyclictest181rcu_preempt12:37:400
15296997551,11cyclictest181rcu_preempt11:36:090
15296997551,11cyclictest181rcu_preempt11:36:090
15296997549,7cyclictest181rcu_preempt11:24:230
15296997548,8cyclictest181rcu_preempt07:16:400
15296997544,8cyclictest181rcu_preempt11:04:460
15296997544,8cyclictest181rcu_preempt11:04:460
15296997544,16cyclictest181rcu_preempt07:54:220
15296997517,34cyclictest15-21ksoftirqd/010:54:460
15296997517,34cyclictest15-21ksoftirqd/010:54:460
181492740,4sleep30-21swapper/307:15:153
15305997451,7cyclictest181rcu_preempt11:56:212
15305997451,7cyclictest181rcu_preempt11:56:212
15305997448,7cyclictest181rcu_preempt12:12:022
15305997443,22cyclictest6245-21modprobe08:19:172
15305997439,9cyclictest31947-21ssh11:25:452
15300997455,13cyclictest22631-21diskmemload09:54:471
15296997447,19cyclictest26353-21threads09:14:550
15296997447,19cyclictest26353-21threads09:14:550
15305997359,8cyclictest0-21swapper/207:45:112
15305997353,11cyclictest0-21swapper/211:54:192
15300997355,10cyclictest15044-21load10:54:471
15300997355,10cyclictest15044-21load10:54:471
15300997354,13cyclictest22631-21diskmemload09:14:411
15300997354,13cyclictest22631-21diskmemload09:14:411
15296997351,9cyclictest181rcu_preempt09:26:590
15296997351,7cyclictest181rcu_preempt11:50:170
15296997348,7cyclictest181rcu_preempt11:04:080
15296997343,7cyclictest181rcu_preempt08:12:580
15296997343,7cyclictest181rcu_preempt08:12:580
15296997341,8cyclictest181rcu_preempt12:14:570
15296997341,8cyclictest181rcu_preempt12:14:570
15296997329,11cyclictest181rcu_preempt07:27:080
15305997249,8cyclictest181rcu_preempt10:56:262
15305997249,8cyclictest181rcu_preempt10:56:262
15305997248,13cyclictest181rcu_preempt09:44:502
15305997241,7cyclictest181rcu_preempt07:50:282
15305997232,9cyclictest181rcu_preempt09:31:572
15305997232,9cyclictest181rcu_preempt09:31:572
15296997247,7cyclictest181rcu_preempt12:05:100
15296997247,7cyclictest181rcu_preempt12:05:100
15296997238,7cyclictest181rcu_preempt09:32:280
15296997238,7cyclictest181rcu_preempt09:32:280
15296997223,7cyclictest0-21swapper/010:32:260
15305997155,10cyclictest10022-21timerandwakeup10:44:572
15305997140,22cyclictest2190-21fschecks_count12:29:412
15305997137,7cyclictest181rcu_preempt10:15:322
15300997155,9cyclictest6559-21ssh11:39:411
15300997150,14cyclictest28753-21threads11:19:581
15300997142,20cyclictest9484-21modprobe08:29:171
15296997146,7cyclictest181rcu_preempt11:14:290
15296997139,27cyclictest15287-21cyclictest09:36:220
15305997053,12cyclictest32704-21unixbench_singl12:24:592
1530599705,27cyclictest39-21ksoftirqd/208:47:142
15300997050,14cyclictest1594-21processes11:29:541
15300997046,17cyclictest12776-21grep11:50:011
15296997050,14cyclictest21761-21netstat09:04:490
15296997027,10cyclictest0-21swapper/012:29:390
15296997021,7cyclictest0-21swapper/009:40:060
15296997021,17cyclictest0-21swapper/008:29:590
15305996948,15cyclictest17615-21entropy11:59:422
15305996939,10cyclictest13078-21ssh09:53:092
15305996939,10cyclictest13078-21ssh09:53:092
15305996935,7cyclictest181rcu_preempt11:11:482
15305996930,8cyclictest181rcu_preempt10:27:202
15305996930,10cyclictest181rcu_preempt10:03:562
15305996930,10cyclictest181rcu_preempt10:03:562
15305996926,11cyclictest181rcu_preempt10:21:562
15300996952,11cyclictest31341-21timerandwakeup09:24:591
15300996950,13cyclictest22631-21diskmemload10:16:161
15300996949,13cyclictest7441-21unixbench_multi08:19:581
15300996948,15cyclictest18382-21sh10:02:411
15300996948,15cyclictest18382-21sh10:02:411
1530099694,26cyclictest31-21ksoftirqd/107:26:091
1530099693,29cyclictest31-21ksoftirqd/112:13:101
15296996945,18cyclictest11426-21cpuspeed08:34:390
15296996932,9cyclictest181rcu_preempt08:45:260
1529699693,22cyclictest24289-21cpuspeed07:34:370
15296996917,11cyclictest0-21swapper/008:54:180
15296996913,32cyclictest15-21ksoftirqd/008:18:370
15305996842,19cyclictest27192-21latency_hist07:44:212
15305996842,19cyclictest27192-21latency_hist07:44:212
15300996854,10cyclictest31052-21kworker/u19:0-rpciod09:34:411
15300996853,9cyclictest27392-21apt-key07:44:201
15300996853,9cyclictest27392-21apt-key07:44:201
15300996847,15cyclictest6704-21fschecks_time10:39:461
1530099684,25cyclictest31-21ksoftirqd/107:21:121
1530099683,32cyclictest31-21ksoftirqd/111:07:221
1530099683,32cyclictest31-21ksoftirqd/111:07:221
1530099683,26cyclictest31-21ksoftirqd/110:23:081
1530599674,27cyclictest39-21ksoftirqd/207:57:142
15305996740,10cyclictest181rcu_preempt07:10:202
15305996725,10cyclictest181rcu_preempt07:22:152
1530599672,33cyclictest39-21ksoftirqd/207:26:472
15300996753,8cyclictest0-21swapper/110:09:471
15300996751,8cyclictest0-21swapper/111:39:181
15300996751,8cyclictest0-21swapper/111:39:181
15300996751,11cyclictest15287-21cyclictest09:12:151
1530099673,27cyclictest31-21ksoftirqd/108:04:381
1530099673,26cyclictest31-21ksoftirqd/109:31:001
1530099673,26cyclictest31-21ksoftirqd/109:31:001
1530099673,24cyclictest31-21ksoftirqd/112:01:411
1530599667,26cyclictest39-21ksoftirqd/211:40:492
15305996640,20cyclictest11543-21fschecks_count08:34:442
15300996651,8cyclictest0-21swapper/107:49:211
15300996650,10cyclictest22631-21diskmemload12:24:581
15300996647,13cyclictest5290-21latency08:14:471
1530099664,26cyclictest31-21ksoftirqd/108:47:091
1530099664,26cyclictest31-21ksoftirqd/107:29:421
1530099664,25cyclictest31-21ksoftirqd/108:53:331
1530099664,24cyclictest31-21ksoftirqd/107:55:251
1530099663,26cyclictest31-21ksoftirqd/108:35:241
15296996647,13cyclictest16155-21mailstats07:10:050
15296996638,22cyclictest0-21swapper/007:49:450
1530899652,31cyclictest896-21kworker/3:1+pm10:59:213
1530899652,31cyclictest896-21kworker/3:1+pm10:59:213
15305996551,10cyclictest15287-21cyclictest08:25:502
15300996553,7cyclictest0-21swapper/109:04:461
1530099655,26cyclictest31-21ksoftirqd/110:05:051
15300996550,10cyclictest15287-21cyclictest10:54:201
15300996549,10cyclictest17749-21latency10:59:451
15300996545,14cyclictest13123-21sh09:53:191
15300996545,14cyclictest13123-21sh09:53:191
15300996544,15cyclictest29017-21cpu12:20:061
1530099654,25cyclictest31-21ksoftirqd/110:26:421
15300996536,21cyclictest4668-21systemd-run08:14:191
15300996536,21cyclictest4668-21systemd-run08:14:191
196702640,7sleep30-21swapper/308:59:333
15305996431,8cyclictest181rcu_preempt08:49:532
15300996441,16cyclictest25733-21latency_hist07:39:201
1530599634,23cyclictest39-21ksoftirqd/208:31:592
1530099633,22cyclictest31-21ksoftirqd/109:21:011
1530099633,22cyclictest31-21ksoftirqd/109:21:011
1530099625,25cyclictest31-21ksoftirqd/112:38:281
15300996248,7cyclictest0-21swapper/110:34:461
15300996245,11cyclictest347-21runrttasks11:24:461
15308996141,14cyclictest189242sleep310:04:203
15308996141,14cyclictest189242sleep310:04:203
15300996140,15cyclictest22669-21cstates11:09:371
1530099607,16cyclictest14021-21seq08:40:291
1530099603,31cyclictest31-21ksoftirqd/107:51:301
1530099594,26cyclictest31-21ksoftirqd/108:57:191
1530099583,40cyclictest20410-21switchtime08:59:571
15300995743,9cyclictest0-21swapper/112:30:401
1530099574,26cyclictest31-21ksoftirqd/108:29:261
15300995740,10cyclictest0-21swapper/107:09:531
1530899552,36cyclictest896-21kworker/3:1+pm12:09:473
1530899552,26cyclictest896-21kworker/3:1+pm07:59:213
1530899543,35cyclictest1095-21taskset11:29:403
1530899523,34cyclictest2481-21taskset08:04:553
15308995134,11cyclictest59682sleep312:35:283
1530899513,34cyclictest22143-21taskset12:07:463
1530899513,34cyclictest22143-21taskset12:07:463
15308994935,9cyclictest451rcuc/311:50:203
15308994935,8cyclictest44532chrt09:35:203
1530899492,31cyclictest8287-21taskset09:44:243
1530899488,26cyclictest31937-21chrt09:26:203
1530899484,30cyclictest16163-21taskset09:59:173
1530899483,32cyclictest14204-21taskset08:42:513
1530899483,31cyclictest7223-21taskset09:41:163
1530899483,31cyclictest12472-21taskset09:51:103
1530899483,31cyclictest12472-21taskset09:51:103
15308994833,10cyclictest451rcuc/311:09:573
1530899483,30cyclictest24387-21chrt07:34:403
1530899483,30cyclictest22233-21taskset07:27:523
1530899483,30cyclictest19162-21taskset11:02:493
15308994828,7cyclictest774098irq/304-mma8451_consumer007:50:043
15308994827,14cyclictest451rcuc/310:16:323
1530899473,29cyclictest16576-21taskset07:11:023
1530899473,27cyclictest19108-21taskset08:55:523
1530899473,25cyclictest22556-21chrt09:08:363
1530899473,25cyclictest2230-21chrt10:30:513
1530899473,25cyclictest2160-21chrt12:29:393
1530899472,31cyclictest27151-21taskset07:43:563
1530899472,31cyclictest27151-21taskset07:43:563
1530899464,28cyclictest15825-21taskset08:48:023
1530899463,30cyclictest24496-21chrt09:11:453
1530899463,30cyclictest15025-21taskset11:54:453
1530899463,30cyclictest15025-21taskset11:54:453
1530899463,29cyclictest5003-21taskset10:35:583
1530899463,29cyclictest29142-21taskset09:20:163
1530899463,29cyclictest29142-21taskset09:20:163
1530899463,29cyclictest27339-21taskset12:16:303
1530899463,29cyclictest27339-21taskset12:16:303
1530899463,29cyclictest25681-21taskset09:14:363
1530899463,29cyclictest25681-21taskset09:14:363
1530899463,29cyclictest16558-21taskset08:49:403
1530899463,25cyclictest30070-21chrt10:23:493
1530899463,25cyclictest13236-21chrt10:50:493
1530899463,24cyclictest8420-21chrt11:43:593
1530899462,34cyclictest896-21kworker/3:1+pm11:04:353
1530899462,34cyclictest896-21kworker/3:1+pm11:04:353
1530899462,31cyclictest21009-21taskset10:07:153
1530899462,30cyclictest4379-21taskset08:11:033
1530899462,30cyclictest4379-21taskset08:11:033
1530899462,29cyclictest29380-21taskset12:20:153
1530899453,33cyclictest26728-21chrt11:16:393
1530899453,29cyclictest32288-21taskset11:26:573
1530899453,29cyclictest10869-21chrt08:31:333
1530899453,29cyclictest10274-21taskset10:45:103
1530899453,28cyclictest7767-21chrt08:22:543
1530899453,27cyclictest31724-21chrt10:25:153
1530899453,23cyclictest6022-21chrt08:16:263
15308994530,11cyclictest451rcuc/307:44:263
1530899452,27cyclictest598-21taskset07:59:483
1530899452,25cyclictest8219-21chrt10:42:263
1530899443,31cyclictest17434-21taskset11:59:353
1530899443,28cyclictest5400-21taskset11:37:183
1530899443,28cyclictest5400-21taskset11:37:183
1530899443,22cyclictest32286-21chrt12:24:533
1530899442,33cyclictest911-21taskset09:29:413
1530899442,33cyclictest911-21taskset09:29:413
1530899442,24cyclictest11232-21taskset11:48:533
1530899442,24cyclictest11232-21taskset11:48:533
1530899433,30cyclictest12759-21chrt08:39:193
1530899433,22cyclictest23610-21chrt10:11:413
15308994328,9cyclictest451rcuc/308:26:433
1530899432,32cyclictest22778-21chrt07:29:383
1530899422,28cyclictest19370-21taskset07:19:413
15308994128,6cyclictest774098irq/304-mma8451_consumer011:23:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional