You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-30 - 03:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Sat Aug 30, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
512321349,93sleep10-21swapper/118:57:511
623921156,15sleep00-21swapper/018:59:580
6593991102,62cyclictest21635-21seq23:06:150
6593991102,62cyclictest21635-21seq23:06:150
6593991088,65cyclictest0-21swapper/022:09:460
6593991088,65cyclictest0-21swapper/022:09:460
65939910623,59cyclictest0-21swapper/023:17:440
65939910623,59cyclictest0-21swapper/023:17:440
5287210610,67sleep25296-21/usr/sbin/munin18:58:052
65939910523,53cyclictest0-21swapper/021:57:170
2449121050,4sleep20-21swapper/200:07:112
2449121050,4sleep20-21swapper/200:07:112
6593991024,60cyclictest0-21swapper/021:52:030
6593991024,60cyclictest0-21swapper/021:52:030
6593991022,71cyclictest10754-21runrttasks23:42:580
6593991022,60cyclictest20052-21seq21:12:520
6593991012,64cyclictest7945-21seq23:38:360
6593991012,64cyclictest24697-21seq21:20:260
6593991003,57cyclictest31591-21runrttasks21:32:500
6593991003,57cyclictest31591-21runrttasks21:32:500
6593991002,61cyclictest10881-21runrttasks20:52:510
6593991002,61cyclictest10881-21runrttasks20:52:510
659399994,85cyclictest0-21swapper/021:27:550
6593999944,49cyclictest9825-21processes20:48:130
659399992,62cyclictest29841-21seq22:23:560
659399992,60cyclictest26766-21seq22:18:210
659399992,60cyclictest26766-21seq22:18:210
659399984,57cyclictest0-21swapper/023:51:530
659399983,59cyclictest6763-21seq21:45:340
659399983,58cyclictest16655-21seq19:30:040
659399983,58cyclictest16655-21seq19:30:040
659399982,58cyclictest4217-21seq21:41:370
659399982,54cyclictest7214-21ssh23:37:390
659399974,83cyclictest0-21swapper/020:07:580
659399974,61cyclictest375-21snmpd19:58:440
659399973,59cyclictest12633-21seq22:50:260
659399972,66cyclictest19076-21seq23:02:130
659399972,60cyclictest18121-21seq23:55:090
659399972,60cyclictest18121-21seq23:55:090
659399972,59cyclictest7132-21seq20:39:510
659399972,59cyclictest27349-21seq21:24:440
659399972,59cyclictest27349-21seq21:24:440
659399972,59cyclictest1136-21chrt20:22:460
659399972,58cyclictest31618-21seq20:16:220
659399972,58cyclictest26629-21seq23:13:440
6593999719,44cyclictest0-21swapper/023:57:540
6593999712,56cyclictest5508-21seq20:34:490
659399963,54cyclictest356-21runrttasks19:13:160
659399963,54cyclictest356-21runrttasks19:13:160
659399962,61cyclictest24506-21seq22:15:130
659399962,58cyclictest0-21swapper/023:23:410
659399962,56cyclictest5691-21seq00:31:200
529729621,45sleep30-21swapper/318:58:063
659399953,82cyclictest0-21swapper/022:27:530
659399953,57cyclictest356-21runrttasks19:05:010
659399953,56cyclictest3969-21seq20:31:040
659399952,60cyclictest22007-21seq19:47:420
6606999473,14cyclictest27966-21systemd-journal21:46:401
659399943,80cyclictest0-21swapper/019:32:540
659399943,80cyclictest0-21swapper/019:32:540
659399943,56cyclictest0-21swapper/022:35:590
659399942,59cyclictest29822-21runrttasks00:16:120
659399942,59cyclictest29822-21runrttasks00:16:120
659399942,54cyclictest694-21seq20:20:190
659399942,54cyclictest694-21seq20:20:190
6593999415,42cyclictest0-21swapper/019:42:320
6593999342,44cyclictest13847-21diskmemload23:27:540
6593999342,44cyclictest13847-21diskmemload23:27:540
659399932,57cyclictest356-21runrttasks00:21:310
659399923,82cyclictest27508-21ssh00:12:380
659399923,79cyclictest22461-21ssh23:07:540
659399922,60cyclictest9817-21seq19:08:220
659399922,53cyclictest15466-21runrttasks21:03:530
661499912,61cyclictest7904-21apt-get22:43:132
659399914,61cyclictest10336-21kworker/u17:4+rpciod22:03:040
659399914,61cyclictest10336-21kworker/u17:4+rpciod22:03:040
659399913,60cyclictest356-21runrttasks22:00:540
659399912,61cyclictest2408-21seq00:24:230
6593999120,44cyclictest356-21runrttasks21:08:270
659399904,81cyclictest0-21swapper/000:02:540
659399904,81cyclictest0-21swapper/000:02:540
659399903,54cyclictest356-21runrttasks19:23:240
6614998960,19cyclictest15295-21apt-key19:27:422
659399882,76cyclictest0-21swapper/019:42:530
659399873,75cyclictest0-21swapper/020:57:500
659399873,48cyclictest356-21runrttasks22:41:430
659399872,58cyclictest13164-21taskset19:19:360
659399872,58cyclictest13164-21taskset19:19:360
6614998654,23cyclictest379-21cron00:02:372
659399863,54cyclictest8688-21seq20:44:080
659399862,59cyclictest10592-21sed22:47:410
659399852,55cyclictest356-21runrttasks19:53:460
659399842,52cyclictest356-21runrttasks20:04:030
6606998343,13cyclictest356-21runrttasks00:08:071
659399832,72cyclictest13739-21apt-get22:52:490
6606998246,13cyclictest356-21runrttasks22:44:371
6614998135,34cyclictest25530-21date21:22:402
6606998161,14cyclictest19232-21netstat19:38:071
6614998059,14cyclictest27966-21systemd-journal00:21:322
6614998056,15cyclictest27966-21systemd-journal21:31:052
662099795,32cyclictest1781-21chrt00:23:123
6614997960,13cyclictest6584-21cyclictest23:52:382
661499793,56cyclictest1860-21/usr/sbin/munin22:33:022
6606997932,42cyclictest0-21swapper/121:27:551
662099784,32cyclictest17519-21chrt23:53:223
662099784,32cyclictest17519-21chrt23:53:223
662099783,35cyclictest31722-21chrt20:17:383
6606997858,13cyclictest0-21swapper/123:53:071
6606997858,13cyclictest0-21swapper/123:53:071
6606997856,14cyclictest1-21systemd23:19:041
6606997856,14cyclictest1-21systemd23:19:041
6606997852,17cyclictest20047-21kworker/u17:2-nfsiod21:18:001
6606997843,26cyclictest10861-21grep23:43:001
661499773,42cyclictest21142-21vmstat21:13:182
6606997722,22cyclictest27985-21seq22:22:011
6606997722,22cyclictest27985-21seq22:22:011
6606997721,20cyclictest10056-21seq23:42:051
6606997721,18cyclictest23029-21seq19:48:191
6606997711,29cyclictest0-21swapper/100:13:001
6606997711,29cyclictest0-21swapper/100:13:001
662099763,30cyclictest3350-21taskset20:28:113
6606997620,22cyclictest31510-21seq20:15:001
6606997620,19cyclictest11976-21seq20:54:371
6606997620,19cyclictest11976-21seq20:54:371
6606997619,22cyclictest30045-21seq22:24:331
6606997618,26cyclictest18277-21seq19:34:541
6606997618,26cyclictest18277-21seq19:34:541
6606997612,29cyclictest5364-21sed23:33:131
6614997551,14cyclictest0-21swapper/219:17:592
6614997551,14cyclictest0-21swapper/219:17:592
661499753,44cyclictest3845-21ssh00:27:562
661499752,40cyclictest15023-21apt-get19:28:082
661499752,40cyclictest15023-21apt-get19:28:082
300952750,6sleep130092-21modprobe20:12:381
662099744,27cyclictest25987-21chrt23:13:103
6614997446,20cyclictest13774-21cpu22:52:542
661499743,40cyclictest18983-21modprobe23:57:382
661499743,40cyclictest18983-21modprobe23:57:382
6614997411,44cyclictest14053-21apt-get21:57:532
6606997427,38cyclictest0-21swapper/100:02:541
6606997427,38cyclictest0-21swapper/100:02:541
6606997426,35cyclictest0-21swapper/119:12:501
6606997426,35cyclictest0-21swapper/119:12:501
660699742,52cyclictest14078-21ssh21:57:541
6606997416,26cyclictest356-21runrttasks19:59:251
6614997354,13cyclictest4185-21latency_hist20:32:402
6614997351,12cyclictest0-21swapper/221:03:152
6614997347,16cyclictest28984-21sendmail_mailqu23:18:112
6614997347,16cyclictest28984-21sendmail_mailqu23:18:112
6614997327,32cyclictest10666-21gpgv21:52:422
6614997327,32cyclictest10666-21gpgv21:52:422
6606997351,12cyclictest19948-21kernelversion23:03:041
6606997351,12cyclictest19948-21kernelversion23:03:041
6606997344,24cyclictest26369-21kworker/u19:1+events_unbound23:32:381
6606997344,24cyclictest26369-21kworker/u19:1+events_unbound23:32:381
6606997321,17cyclictest21529-21seq21:14:311
6606997318,20cyclictest13235-21seq19:20:331
6606997318,20cyclictest13235-21seq19:20:331
661499727,50cyclictest7134-21apt-get23:37:522
6614997249,17cyclictest4221-21idleruntime-cro23:32:382
6614997249,17cyclictest4221-21idleruntime-cro23:32:382
6614997222,36cyclictest5868-21cat20:37:412
6606997250,16cyclictest17800-21timerandwakeup22:03:161
6606997250,16cyclictest17800-21timerandwakeup22:03:161
6606997235,27cyclictest2027-21ssh21:37:531
6614997148,16cyclictest13684-21latency_hist22:52:452
661499712,43cyclictest10984-21sed23:43:032
661499712,38cyclictest2623-21latency_hist20:27:412
6606997152,15cyclictest6584-21cyclictest22:28:581
6606997119,28cyclictest2164-21seq00:23:341
662099704,29cyclictest27336-21taskset20:03:003
6620997013,20cyclictest0-21swapper/321:33:143
6620997013,20cyclictest0-21swapper/321:33:143
6614997051,11cyclictest0-21swapper/223:03:002
6614997051,11cyclictest0-21swapper/223:03:002
6614997049,14cyclictest356-21runrttasks21:46:052
6614997045,16cyclictest8229-21latency_hist19:07:412
6614997034,20cyclictest22545-21latency_hist22:12:412
6614997034,20cyclictest22545-21latency_hist22:12:412
6606997055,10cyclictest6584-21cyclictest21:23:121
6606997055,10cyclictest6584-21cyclictest21:23:121
6606997052,11cyclictest15509-21/usr/sbin/munin19:28:301
6606997052,11cyclictest15509-21/usr/sbin/munin19:28:301
6606997027,38cyclictest0-21swapper/121:32:491
6606997027,38cyclictest0-21swapper/121:32:491
6606997019,18cyclictest6499-21seq22:40:091
6606997015,23cyclictest356-21runrttasks23:13:401
6614996951,12cyclictest28822-21sh22:23:012
661499693,40cyclictest2308-21fschecks_time21:38:042
6614996924,11cyclictest181rcu_preempt21:57:402
6614996923,33cyclictest30247-21cat20:12:412
6606996953,11cyclictest365-21systemd-logind20:22:401
6606996953,11cyclictest365-21systemd-logind20:22:401
6606996947,16cyclictest6584-21cyclictest19:57:421
6606996947,13cyclictest0-21swapper/120:27:401
6606996927,36cyclictest0-21swapper/120:37:521
6606996916,20cyclictest0-21swapper/123:02:051
6606996914,22cyclictest0-21swapper/119:11:091
6606996912,24cyclictest4066-21runrttasks20:32:171
6614996850,12cyclictest8505-21timerwakeupswit20:43:122
6614996819,34cyclictest39-21ksoftirqd/223:09:532
6606996816,18cyclictest20769-21runrttasks23:59:271
6620996711,20cyclictest0-21swapper/319:28:083
6620996711,20cyclictest0-21swapper/319:28:083
661499673,37cyclictest28483-21latency_hist20:07:412
6614996731,27cyclictest5649-21cat22:38:132
6614996723,34cyclictest23381-21apt-get19:52:412
6606996751,10cyclictest10336-21kworker/u17:4-xprtiod22:13:051
6606996749,13cyclictest15024-21latency_hist19:27:421
6606996727,36cyclictest0-21swapper/123:07:541
6614996649,11cyclictest17892-21switchtime21:08:162
6614996645,15cyclictest25988-21ntp_offset23:13:112
6614996635,19cyclictest7406-21apt-key23:37:412
661499662,26cyclictest13847-21diskmemload21:23:022
661499662,26cyclictest13847-21diskmemload21:23:022
6606996644,14cyclictest27966-21systemd-journal22:55:041
6614996548,11cyclictest27966-21systemd-journal23:24:042
6614996546,13cyclictest27779-21/usr/sbin/munin00:13:122
6614996546,13cyclictest27779-21/usr/sbin/munin00:13:122
6614996539,17cyclictest27966-21systemd-journal22:31:462
6614996536,22cyclictest1213-21fschecks_count00:22:592
661499653,48cyclictest22716-21/usr/sbin/munin22:13:142
661499652,45cyclictest31002-21processes20:13:112
6606996549,10cyclictest1-21systemd21:48:041
6606996549,10cyclictest1-21systemd21:48:041
6606996548,11cyclictest27966-21systemd-journal19:47:391
6606996546,13cyclictest13847-21diskmemload00:22:371
6606996544,13cyclictest22418-21latency_hist22:12:411
6606996544,13cyclictest22418-21latency_hist22:12:411
6606996529,32cyclictest0-21swapper/123:22:531
6606996527,24cyclictest9401-21seq20:48:011
6606996521,39cyclictest0-21swapper/120:32:531
6606996515,22cyclictest356-21runrttasks21:00:041
6614996447,11cyclictest365-21systemd-logind20:42:392
6614996444,9cyclictest0-21swapper/222:18:012
6614996444,9cyclictest0-21swapper/222:18:012
6614996437,15cyclictest0-21swapper/220:18:002
6614996437,15cyclictest0-21swapper/220:18:002
661499642,21cyclictest23908-21apt-get19:53:192
6606996421,31cyclictest13848-21/usr/sbin/munin23:48:031
6620996345,12cyclictest0-21swapper/322:43:243
6620996324,23cyclictest0-21swapper/323:38:293
6614996342,14cyclictest8955-21kworker/u17:1-rpciod21:32:552
6614996342,14cyclictest8955-21kworker/u17:1-rpciod21:32:552
6614996341,10cyclictest0-21swapper/222:07:402
6614996341,10cyclictest0-21swapper/222:07:402
6614996335,21cyclictest8496-21fschecks_count19:07:472
661499633,42cyclictest10955-21ntp_kernel_err19:13:062
661499633,42cyclictest10955-21ntp_kernel_err19:13:062
6606996345,10cyclictest0-21swapper/121:08:181
6606996333,18cyclictest27138-21apt-get20:02:501
6606996317,38cyclictest4939-1kworker/1:1H+kblockd21:53:151
662099623,38cyclictest14613-21chrt19:23:153
6614996243,13cyclictest21203-21timerwakeupswit19:43:142
6614996238,17cyclictest6584-21cyclictest20:48:132
661499623,41cyclictest356-21runrttasks20:53:522
661499623,41cyclictest356-21runrttasks20:53:522
6606996230,24cyclictest10844-21ssh22:47:571
6606996228,27cyclictest3734-21ssh00:27:511
6614996139,13cyclictest0-21swapper/222:57:552
661499613,39cyclictest12477-21/usr/sbin/munin20:57:492
661499612,37cyclictest17793-21grep19:33:202
661499612,37cyclictest17793-21grep19:33:202
6606996139,12cyclictest23909-21kworker/u17:2-xprtiod22:32:491
6620996045,8cyclictest0-21swapper/323:52:403
662099604,41cyclictest2853-21chrt22:33:153
662099603,37cyclictest25277-21chrt00:08:013
6620996019,31cyclictest0-21swapper/319:52:543
6614996032,19cyclictest18976-21fschecks_count19:38:022
6606996042,13cyclictest7854-21fschecks_time20:42:581
6606996042,12cyclictest15509-21sh21:04:011
6606996038,12cyclictest0-21swapper/119:03:011
662099592,41cyclictest13864-21chrt21:02:383
6614995936,12cyclictest0-21swapper/219:57:562
661499592,28cyclictest13847-21diskmemload00:08:092
662099583,42cyclictest18580-21taskset19:37:413
662099583,42cyclictest18580-21taskset19:37:413
662099583,38cyclictest32135-21chrt23:23:153
662099583,38cyclictest19459-21taskset23:58:003
6620995811,36cyclictest14076-21chrt21:57:533
662099573,36cyclictest19854-21chrt23:03:013
662099573,36cyclictest19854-21chrt23:03:013
662099567,36cyclictest15924-21taskset21:05:143
662099565,32cyclictest0-21swapper/321:53:053
662099564,35cyclictest27048-21chrt21:23:493
662099564,35cyclictest27048-21chrt21:23:493
662099564,34cyclictest401-21chrt22:29:183
662099563,41cyclictest19720-21chrt22:07:423
662099563,41cyclictest19720-21chrt22:07:423
662099563,37cyclictest18034-21chrt21:08:173
662099563,37cyclictest10794-21chrt19:13:023
662099563,37cyclictest10794-21chrt19:13:023
662099562,38cyclictest7833-21chrt20:42:583
662099554,36cyclictest21936-21taskset19:47:413
662099554,35cyclictest28498-21chrt22:22:493
662099553,38cyclictest7043-21chrt19:03:023
662099553,37cyclictest23696-21chrt22:13:133
662099553,35cyclictest10105-21chrt20:48:163
662099544,36cyclictest29015-21chrt21:28:033
662099543,40cyclictest30641-21taskset00:17:533
662099543,38cyclictest8206-21taskset21:48:053
662099543,38cyclictest8206-21taskset21:48:053
662099543,37cyclictest23455-21taskset19:52:393
662099543,37cyclictest11022-21taskset22:48:023
662099543,36cyclictest4580-21taskset21:42:383
662099543,34cyclictest27437-21chrt22:20:223
662099543,34cyclictest27437-21chrt22:20:223
662099542,35cyclictest8975-21taskset19:08:003
662099542,34cyclictest29863-21taskset23:19:433
662099542,34cyclictest29863-21taskset23:19:433
662099533,38cyclictest3752-21taskset00:27:543
662099533,37cyclictest23226-21chrt21:18:023
662099533,37cyclictest12720-21taskset23:46:253
662099533,35cyclictest22891-21chrt00:03:153
662099533,35cyclictest22891-21chrt00:03:153
662099532,34cyclictest32612-21chrt20:18:113
662099532,34cyclictest32612-21chrt20:18:113
662099524,29cyclictest900-21kworker/3:1+events23:32:453
662099523,38cyclictest18580-21chrt23:00:473
662099523,33cyclictest5103-21chrt22:38:013
662099522,35cyclictest4557-21chrt20:32:573
662099514,35cyclictest29095-21taskset20:08:073
662099514,33cyclictest26001-21chrt19:58:063
662099513,34cyclictest23173-21taskset23:08:123
662099513,33cyclictest5143-21taskset21:43:023
662099504,33cyclictest20603-21chrt22:08:113
662099504,33cyclictest20603-21chrt22:08:113
662099503,35cyclictest21039-21taskset21:13:163
662099503,34cyclictest13437-21chrt19:22:383
662099503,34cyclictest13437-21chrt19:22:383
662099502,37cyclictest14839-21chrt22:53:173
662099492,34cyclictest12293-21chrt20:57:403
662099492,34cyclictest12293-21chrt20:57:403
662099492,33cyclictest29538-21taskset00:15:183
662099492,33cyclictest29538-21taskset00:15:183
662099483,29cyclictest19434-21chrt19:38:133
662099483,29cyclictest1935-21taskset20:23:133
662099453,29cyclictest2272-21chrt23:28:103
662099453,29cyclictest2272-21chrt23:28:103
662099423,26cyclictest6497-21taskset20:38:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional