You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-09 - 17:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Mon Dec 09, 2024 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3149210928,40sleep20-21swapper/207:05:072
2719210813,54sleep12753-21fschecks_count07:04:501
255629818,36sleep30-21swapper/307:04:393
3941998944,16cyclictest41-21ksoftirqd/309:03:283
3941998944,16cyclictest41-21ksoftirqd/309:03:283
3941998750,13cyclictest41-21ksoftirqd/309:32:163
394199853,47cyclictest27717-21ssh09:39:343
394199853,47cyclictest27717-21ssh09:39:343
3938998319,38cyclictest10787-21modprobe07:28:240
3938998319,38cyclictest10787-21modprobe07:28:240
3941998258,15cyclictest16679-21sh10:18:243
3941998258,15cyclictest16679-21sh10:18:243
394199822,44cyclictest5747-21ssh10:53:563
3941998158,15cyclictest30267-21idleruntime-cro08:28:253
3941998158,15cyclictest30267-21idleruntime-cro08:28:253
3941998145,8cyclictest41-21ksoftirqd/311:53:533
3941998144,13cyclictest41-21ksoftirqd/310:59:513
3941998137,15cyclictest41-21ksoftirqd/312:13:563
3941998048,10cyclictest41-21ksoftirqd/308:38:423
3941998048,10cyclictest41-21ksoftirqd/308:38:423
3938998044,9cyclictest141rcu_preempt10:59:450
3940997958,8cyclictest141rcu_preempt12:03:552
136352790,10sleep013640-21unixbench_singl07:34:120
136352790,10sleep013640-21unixbench_singl07:34:120
3941997840,30cyclictest41-21ksoftirqd/311:49:033
3941997748,18cyclictest30097-21diskmemload10:39:373
3940997760,10cyclictest31104-21tune2fs12:33:442
3939997743,15cyclictest141rcu_preempt11:04:521
3938997752,9cyclictest21153-21idleruntime-cro09:28:250
3941997660,10cyclictest2698-21idleruntime-cro09:53:253
3940997638,13cyclictest141rcu_preempt09:54:072
3940997638,13cyclictest141rcu_preempt09:54:072
3941997545,8cyclictest41-21ksoftirqd/312:09:003
250827511,39sleep0414-21lldpd07:04:030
3941997442,11cyclictest141rcu_preempt08:08:263
3941997442,11cyclictest141rcu_preempt08:08:263
3939997457,11cyclictest626-21in:imuxsock10:38:251
3939997439,18cyclictest141rcu_preempt07:15:061
3939997439,18cyclictest141rcu_preempt07:15:061
3941997349,15cyclictest28300-21cron10:38:243
3941997344,14cyclictest41-21ksoftirqd/311:18:483
3941997341,9cyclictest41-21ksoftirqd/310:23:233
3941997341,9cyclictest41-21ksoftirqd/310:23:233
3941997341,10cyclictest41-21ksoftirqd/311:03:383
3941997329,18cyclictest41-21ksoftirqd/307:49:003
3941997329,18cyclictest41-21ksoftirqd/307:49:003
3940997343,9cyclictest52-21kcompactd011:47:362
3940997343,9cyclictest52-21kcompactd011:47:362
3941997242,10cyclictest41-21ksoftirqd/310:09:113
3941997240,9cyclictest41-21ksoftirqd/307:28:283
3941997240,9cyclictest41-21ksoftirqd/307:28:283
3941997234,25cyclictest41-21ksoftirqd/308:16:273
3941997234,25cyclictest41-21ksoftirqd/308:16:273
3940997248,15cyclictest23330-21unixbench_multi08:04:092
3940997248,15cyclictest23330-21unixbench_multi08:04:092
3940997215,22cyclictest33-21ksoftirqd/210:27:092
3941997138,10cyclictest141rcu_preempt10:02:483
3941997135,13cyclictest41-21ksoftirqd/312:05:193
3940997134,13cyclictest141rcu_preempt12:16:082
3939997116,27cyclictest25-21ksoftirqd/109:22:171
3939997116,27cyclictest25-21ksoftirqd/109:22:171
393899719,37cyclictest12-21ksoftirqd/010:39:340
3940997034,13cyclictest141rcu_preempt10:04:122
3940997034,13cyclictest141rcu_preempt10:04:122
3940997015,25cyclictest0-21swapper/210:02:512
3939997025,18cyclictest141rcu_preempt11:41:471
393899707,10cyclictest25641-21if_eth011:28:440
3938997045,7cyclictest141rcu_preempt12:01:210
3938997045,7cyclictest141rcu_preempt10:46:010
3938997045,7cyclictest141rcu_preempt10:46:010
3938997043,7cyclictest141rcu_preempt09:21:560
3938997043,7cyclictest141rcu_preempt09:21:560
3941996942,9cyclictest41-21ksoftirqd/308:13:273
3941996942,9cyclictest41-21ksoftirqd/308:13:273
3941996936,10cyclictest41-21ksoftirqd/308:33:543
3940996951,12cyclictest10527-21sendmail11:58:242
3940996951,10cyclictest5886-21fschecks_time10:54:002
3940996939,7cyclictest141rcu_preempt12:28:262
3939996951,11cyclictest30944-21cron11:38:241
3939996948,12cyclictest11063-21fschecks_count11:58:451
3939996933,15cyclictest141rcu_preempt07:13:271
3939996930,22cyclictest141rcu_preempt10:18:571
3939996930,22cyclictest141rcu_preempt10:18:571
3938996948,14cyclictest1920-21ssh11:43:410
3938996948,14cyclictest1920-21ssh11:43:410
3938996943,8cyclictest141rcu_preempt09:09:500
3938996913,12cyclictest25180-21latency_hist11:28:270
3941996845,17cyclictest41-21ksoftirqd/309:24:143
3940996834,10cyclictest141rcu_preempt07:44:012
3940996834,10cyclictest141rcu_preempt07:44:012
3940996830,16cyclictest141rcu_preempt10:40:012
3940996829,15cyclictest141rcu_preempt10:18:542
3940996829,15cyclictest141rcu_preempt10:18:542
3940996822,7cyclictest0-21swapper/210:58:542
3940996822,23cyclictest3930-21cyclictest12:21:062
3940996819,17cyclictest33-21ksoftirqd/210:28:412
3940996819,17cyclictest33-21ksoftirqd/210:28:412
3940996818,25cyclictest33-21ksoftirqd/211:18:262
3939996845,15cyclictest7554-21munin-plugin-st08:58:251
3939996839,10cyclictest141rcu_preempt10:58:271
3939996837,11cyclictest141rcu_preempt07:19:021
3939996817,26cyclictest25-21ksoftirqd/108:13:291
3939996817,26cyclictest25-21ksoftirqd/108:13:291
3938996851,10cyclictest30272-21basename08:28:250
3938996851,10cyclictest30272-21basename08:28:250
3938996834,9cyclictest141rcu_preempt09:46:060
3941996731,13cyclictest41-21ksoftirqd/308:55:023
3941996726,22cyclictest141rcu_preempt09:44:233
3940996734,7cyclictest141rcu_preempt11:26:382
3940996717,20cyclictest625-21ssh09:48:562
3939996733,8cyclictest141rcu_preempt12:03:401
393899677,21cyclictest7841-21taskset11:53:400
3938996747,13cyclictest1-21systemd10:03:480
3938996747,13cyclictest1-21systemd10:03:480
3938996710,32cyclictest12-21ksoftirqd/011:38:590
3941996635,8cyclictest41-21ksoftirqd/307:54:023
3940996636,8cyclictest141rcu_preempt12:08:562
3940996633,10cyclictest141rcu_preempt11:40:202
3940996632,11cyclictest141rcu_preempt11:03:562
3940996627,14cyclictest141rcu_preempt11:35:512
3940996626,13cyclictest0-21swapper/209:43:502
3940996624,21cyclictest0-21swapper/208:34:002
3940996621,13cyclictest141rcu_preempt12:02:182
3939996632,12cyclictest141rcu_preempt10:33:261
3939996632,12cyclictest141rcu_preempt10:33:261
3938996647,13cyclictest31909-21systemd-run08:33:260
3938996644,13cyclictest4979-21ssh10:53:250
3938996620,30cyclictest141rcu_preempt11:14:020
238442660,8sleep010811-21diskmemload12:18:590
3941996545,8cyclictest41-21ksoftirqd/310:45:213
3941996545,8cyclictest41-21ksoftirqd/310:45:213
3941996543,8cyclictest41-21ksoftirqd/311:33:443
3941996538,12cyclictest41-21ksoftirqd/312:03:273
3940996538,9cyclictest141rcu_preempt07:43:292
3940996538,7cyclictest141rcu_preempt11:08:512
3940996530,11cyclictest10811-21diskmemload09:18:462
3940996530,11cyclictest10811-21diskmemload09:18:462
3940996529,7cyclictest141rcu_preempt09:03:502
3940996529,7cyclictest141rcu_preempt09:03:502
3940996529,11cyclictest141rcu_preempt10:09:042
3940996520,14cyclictest0-21swapper/210:52:532
3939996538,8cyclictest141rcu_preempt10:09:091
3939996536,8cyclictest141rcu_preempt07:39:101
3939996535,7cyclictest141rcu_preempt12:08:491
3939996527,10cyclictest141rcu_preempt12:13:491
3939996523,17cyclictest14901-21ssh11:08:571
3939996516,25cyclictest25-21ksoftirqd/110:00:481
393899657,18cyclictest21329-21taskset10:25:010
393899653,44cyclictest4940-21apt-get10:53:380
3938996521,27cyclictest22404-21/usr/sbin/munin08:03:430
3938996521,27cyclictest22404-21/usr/sbin/munin08:03:430
3941996446,11cyclictest28652-21munin-plugin-st08:23:253
3941996446,10cyclictest30545-21munin-plugin-st12:33:243
3941996446,10cyclictest30545-21munin-plugin-st12:33:243
394199643,29cyclictest10823-21munin-plugin-st09:08:253
394199643,29cyclictest10823-21munin-plugin-st09:08:253
3941996425,17cyclictest41-21ksoftirqd/309:53:583
3941996425,17cyclictest41-21ksoftirqd/309:53:583
394099648,30cyclictest1969-21awk08:38:592
394099648,30cyclictest1969-21awk08:38:592
3940996445,12cyclictest27022-21munin-plugin-st08:18:242
3940996445,12cyclictest27022-21munin-plugin-st08:18:242
3940996439,7cyclictest141rcu_preempt09:13:552
3940996439,7cyclictest141rcu_preempt09:13:552
3940996438,7cyclictest141rcu_preempt07:18:412
3940996432,9cyclictest141rcu_preempt09:09:432
3940996432,13cyclictest141rcu_preempt09:01:492
3940996432,13cyclictest141rcu_preempt09:01:492
3940996429,7cyclictest141rcu_preempt08:46:002
3940996424,9cyclictest116-21pr/ttymxc009:39:462
3940996424,9cyclictest116-21pr/ttymxc009:39:462
393999646,32cyclictest25-21ksoftirqd/109:25:351
3939996438,8cyclictest141rcu_preempt11:13:491
3939996435,9cyclictest141rcu_preempt10:47:461
3939996435,9cyclictest141rcu_preempt10:47:461
3939996419,18cyclictest141rcu_preempt07:24:091
3939996419,18cyclictest141rcu_preempt07:24:091
3938996414,14cyclictest20897-21sed12:13:590
3938996410,19cyclictest2684-21idleruntime-cro08:43:240
3938996410,19cyclictest2684-21idleruntime-cro08:43:240
3941996343,11cyclictest0-21swapper/307:23:253
3941996335,10cyclictest41-21ksoftirqd/309:09:453
3941996321,12cyclictest141rcu_preempt08:48:503
3941996321,12cyclictest141rcu_preempt08:48:503
3940996335,7cyclictest141rcu_preempt08:53:532
3940996335,14cyclictest141rcu_preempt11:18:552
3940996334,7cyclictest141rcu_preempt07:32:342
3940996333,17cyclictest141rcu_preempt11:29:442
3940996331,13cyclictest141rcu_preempt10:36:592
3940996328,17cyclictest0-21swapper/208:48:532
3940996328,17cyclictest0-21swapper/208:48:532
3940996321,16cyclictest0-21swapper/208:12:262
3940996321,16cyclictest0-21swapper/208:12:262
3939996345,10cyclictest241ktimers/112:18:391
3939996334,8cyclictest141rcu_preempt07:48:451
3939996334,8cyclictest141rcu_preempt07:48:451
3939996333,13cyclictest141rcu_preempt09:38:511
3939996333,13cyclictest141rcu_preempt09:38:511
3939996330,19cyclictest141rcu_preempt08:30:141
3939996328,8cyclictest141rcu_preempt11:20:381
3939996325,15cyclictest141rcu_preempt07:28:411
3939996324,17cyclictest0-21swapper/111:46:201
3939996324,17cyclictest0-21swapper/111:46:201
3939996313,25cyclictest25-21ksoftirqd/111:49:011
3938996314,15cyclictest141rcu_preempt09:41:510
3938996314,15cyclictest141rcu_preempt09:41:510
3941996242,14cyclictest14024-21cron07:38:253
3941996242,14cyclictest14024-21cron07:38:253
3941996239,7cyclictest141rcu_preempt11:30:133
3941996237,18cyclictest0-21swapper/312:18:593
3941996236,12cyclictest141rcu_preempt10:53:263
3941996234,8cyclictest41-21ksoftirqd/311:11:163
3941996233,8cyclictest41-21ksoftirqd/309:38:263
3941996230,17cyclictest41-21ksoftirqd/312:38:263
3941996228,10cyclictest41-21ksoftirqd/307:13:153
3940996247,9cyclictest26218-21kworker/u8:110:43:502
3940996247,9cyclictest26218-21kworker/u8:110:43:502
3940996233,7cyclictest141rcu_preempt08:18:422
3940996230,16cyclictest0-21swapper/210:13:542
3940996230,16cyclictest0-21swapper/210:13:542
3940996229,18cyclictest141rcu_preempt07:56:342
3940996215,21cyclictest0-21swapper/207:36:302
3940996215,21cyclictest0-21swapper/207:36:302
3939996234,7cyclictest141rcu_preempt12:28:261
3939996231,13cyclictest141rcu_preempt11:31:391
3939996215,26cyclictest25-21ksoftirqd/109:33:271
3938996246,9cyclictest0-21swapper/009:38:250
3938996212,8cyclictest20867-21cpuspeed07:58:410
3938996212,8cyclictest20867-21cpuspeed07:58:410
3938996211,7cyclictest9181-21latency_hist07:23:280
154962620,8sleep3401ktimers/309:14:543
154962620,8sleep3401ktimers/309:14:543
3941996139,8cyclictest31882-21ntpq11:38:553
3941996138,16cyclictest0-21swapper/307:29:093
3941996136,15cyclictest0-21swapper/307:18:243
3941996136,15cyclictest0-21swapper/307:18:243
3941996117,17cyclictest141rcu_preempt07:59:023
3941996117,17cyclictest141rcu_preempt07:59:023
3940996145,10cyclictest27049-21latency_hist09:38:282
3940996135,8cyclictest141rcu_preempt09:28:442
3940996112,20cyclictest0-21swapper/207:12:372
3939996142,12cyclictest275-21systemd-journal09:09:341
3939996130,9cyclictest141rcu_preempt08:13:511
3939996130,9cyclictest141rcu_preempt08:13:511
3939996121,21cyclictest141rcu_preempt10:03:301
3939996121,21cyclictest141rcu_preempt10:03:301
393899619,12cyclictest12-21ksoftirqd/009:04:090
393899619,12cyclictest12-21ksoftirqd/009:04:090
3938996143,11cyclictest275-21systemd-journal11:48:270
393899614,20cyclictest14073-21latency_hist10:13:270
3938996139,12cyclictest0-21swapper/009:14:090
3938996139,12cyclictest0-21swapper/009:14:090
393899613,11cyclictest12-21ksoftirqd/011:08:570
3938996128,7cyclictest141rcu_preempt12:24:420
3938996126,22cyclictest141rcu_preempt11:05:370
3938996124,7cyclictest141rcu_preempt10:16:520
3938996124,7cyclictest141rcu_preempt10:16:520
3941996038,13cyclictest41-21ksoftirqd/310:30:233
3941996038,13cyclictest41-21ksoftirqd/310:30:233
3941996034,13cyclictest41-21ksoftirqd/309:18:563
3941996034,13cyclictest41-21ksoftirqd/309:18:563
3940996036,8cyclictest141rcu_preempt12:31:442
3940996036,8cyclictest141rcu_preempt12:31:442
3940996021,7cyclictest0-21swapper/207:23:492
3940996021,7cyclictest0-21swapper/207:23:492
3940996021,18cyclictest0-21swapper/211:48:382
3939996045,8cyclictest0-21swapper/110:53:241
3939996037,7cyclictest141rcu_preempt10:43:141
3939996032,7cyclictest141rcu_preempt08:34:001
3939996030,8cyclictest141rcu_preempt07:59:021
3939996030,8cyclictest141rcu_preempt07:59:021
3939996029,7cyclictest141rcu_preempt09:07:561
3939996029,7cyclictest141rcu_preempt09:07:561
3939996023,17cyclictest141rcu_preempt07:38:291
3939996023,17cyclictest141rcu_preempt07:38:291
3939996021,22cyclictest141rcu_preempt10:27:141
3939996016,14cyclictest141rcu_preempt08:08:281
3939996016,14cyclictest141rcu_preempt08:08:281
3939996015,25cyclictest25-21ksoftirqd/108:46:171
3939996013,27cyclictest25-21ksoftirqd/112:37:591
3939996011,27cyclictest25-21ksoftirqd/109:13:341
3939996011,27cyclictest25-21ksoftirqd/109:13:341
393899607,17cyclictest12-21ksoftirqd/009:51:090
3938996045,8cyclictest0-21swapper/012:03:590
3938996028,16cyclictest141rcu_preempt09:53:570
3938996028,16cyclictest141rcu_preempt09:53:570
3938996025,6cyclictest0-21swapper/012:30:450
3938996025,6cyclictest0-21swapper/012:30:450
3938996012,13cyclictest12-21ksoftirqd/008:54:070
394199594,43cyclictest401ktimers/311:43:373
394199594,43cyclictest401ktimers/311:43:373
3941995938,8cyclictest41-21ksoftirqd/310:08:253
3941995938,8cyclictest41-21ksoftirqd/310:08:253
3940995932,8cyclictest141rcu_preempt07:49:052
3940995932,8cyclictest141rcu_preempt07:49:052
3940995930,8cyclictest141rcu_preempt07:58:522
3940995930,8cyclictest141rcu_preempt07:58:522
3940995929,7cyclictest141rcu_preempt08:28:392
3940995924,17cyclictest0-21swapper/209:26:202
3940995913,21cyclictest33-21ksoftirqd/207:13:402
3940995913,21cyclictest33-21ksoftirqd/207:13:402
393999595,22cyclictest25-21ksoftirqd/109:54:001
393999595,22cyclictest25-21ksoftirqd/109:54:001
3939995939,7cyclictest141rcu_preempt11:02:511
3939995918,18cyclictest25-21ksoftirqd/110:14:101
3939995918,18cyclictest25-21ksoftirqd/110:14:101
3939995913,12cyclictest241ktimers/109:33:491
3938995924,21cyclictest141rcu_preempt10:19:310
3938995924,21cyclictest141rcu_preempt10:19:310
3938995922,7cyclictest141rcu_preempt12:34:390
3938995916,10cyclictest9317-21latency_hist09:03:270
3938995916,10cyclictest9317-21latency_hist09:03:270
3938995915,10cyclictest12-21ksoftirqd/010:28:410
3938995915,10cyclictest12-21ksoftirqd/010:28:410
3938995911,15cyclictest21924-21latency09:28:480
3941995842,8cyclictest0-21swapper/307:48:243
3941995842,8cyclictest0-21swapper/307:48:243
3941995842,10cyclictest31882-21awk08:33:253
3941995830,9cyclictest116-21pr/ttymxc011:28:273
3941995829,7cyclictest41-21ksoftirqd/310:23:413
393999588,26cyclictest25-21ksoftirqd/112:29:161
393999588,26cyclictest25-21ksoftirqd/112:29:161
393999587,25cyclictest241ktimers/108:38:451
393999587,25cyclictest241ktimers/108:38:451
393999585,30cyclictest25-21ksoftirqd/111:23:451
3939995838,7cyclictest141rcu_preempt08:24:101
3939995838,7cyclictest141rcu_preempt08:24:101
3939995816,17cyclictest25-21ksoftirqd/108:19:021
393899588,10cyclictest12-21ksoftirqd/012:08:460
3938995841,9cyclictest0-21swapper/008:53:240
3938995841,9cyclictest0-21swapper/008:53:240
3938995839,7cyclictest141rcu_preempt10:36:370
3938995812,12cyclictest12-21ksoftirqd/008:48:290
3941995738,12cyclictest275-21systemd-journal08:44:093
3941995730,9cyclictest41-21ksoftirqd/311:13:573
394199572,27cyclictest15086-21find07:39:103
3940995718,16cyclictest0-21swapper/208:24:062
3940995718,16cyclictest0-21swapper/208:24:062
3939995719,10cyclictest141rcu_preempt07:43:521
3939995719,10cyclictest141rcu_preempt07:43:521
3939995710,29cyclictest25-21ksoftirqd/108:50:501
3939995710,29cyclictest25-21ksoftirqd/108:50:501
3938995713,16cyclictest6427-21fschecks_time07:13:520
3938995713,16cyclictest6427-21fschecks_time07:13:520
393999565,25cyclictest25-21ksoftirqd/109:48:361
3939995635,7cyclictest141rcu_preempt11:53:451
3939995611,10cyclictest141rcu_preempt09:45:231
393899564,12cyclictest12-21ksoftirqd/011:20:300
393899563,15cyclictest12-21ksoftirqd/007:39:110
393899561,20cyclictest24648-21taskset08:09:000
393899561,20cyclictest24648-21taskset08:09:000
3938995610,15cyclictest12-21ksoftirqd/007:58:240
3941995516,20cyclictest141rcu_preempt12:25:373
3939995516,16cyclictest25-21ksoftirqd/107:53:411
3938995520,16cyclictest0-21swapper/011:37:530
3938995513,9cyclictest12-21ksoftirqd/007:08:550
3938995511,10cyclictest12-21ksoftirqd/007:49:010
3938995511,10cyclictest12-21ksoftirqd/007:49:010
393899548,12cyclictest16870-21sed07:44:110
393899548,12cyclictest16870-21sed07:44:110
3938995434,10cyclictest0-21swapper/008:19:020
3938995418,9cyclictest141rcu_preempt08:37:490
393899541,18cyclictest8465-21ssh10:03:280
393999539,22cyclictest141rcu_preempt09:02:091
393999539,22cyclictest141rcu_preempt09:02:091
393899539,12cyclictest12-21ksoftirqd/008:13:570
393899539,12cyclictest12-21ksoftirqd/008:13:570
3938995129,6cyclictest141rcu_preempt07:32:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional