You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 22:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Tue Dec 02, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
77929912152,46cyclictest0-21swapper/109:44:061
6788212112,77sleep16815-21/usr/sbin/munin07:04:181
7789991192,28cyclictest7232-21/usr/sbin/munin09:54:070
7789991192,23cyclictest28258-21/usr/sbin/munin08:09:110
7789991192,23cyclictest28258-21/usr/sbin/munin08:09:110
77929911653,43cyclictest0-21swapper/108:43:441
77929911653,43cyclictest0-21swapper/108:43:441
7789991153,32cyclictest7791-21date07:08:460
7789991093,22cyclictest4188-21taskset11:45:160
7789991093,22cyclictest4188-21taskset11:45:160
77899910850,52cyclictest0-21swapper/011:24:030
7789991082,26cyclictest9207-21ssh10:57:050
7789991082,24cyclictest16873-21sh10:11:320
7789991074,7cyclictest0-21swapper/009:03:570
77899910746,53cyclictest0-21swapper/010:59:390
7789991073,27cyclictest18912-21grep10:14:200
7789991073,25cyclictest6237-21runrttasks10:51:180
7789991073,25cyclictest6237-21runrttasks10:51:180
7789991072,25cyclictest9727-21seq08:52:310
7789991072,23cyclictest15971-21ssh12:07:230
7789991072,23cyclictest15971-21ssh12:07:230
77899910652,49cyclictest0-21swapper/010:33:440
7789991064,25cyclictest30548-21sh10:36:590
7789991062,40cyclictest19359-21ssh12:13:550
7789991062,24cyclictest355-21runrttasks11:23:200
7789991062,24cyclictest28303-21ssh11:30:430
7789991054,22cyclictest23462-21ssh12:20:020
7789991053,26cyclictest14714-21taskset10:08:250
7789991052,26cyclictest21561-21seq09:19:170
77899910517,60cyclictest9560-21sh11:54:460
78019910449,9cyclictest0-21swapper/208:53:442
7789991044,25cyclictest355-21runrttasks07:56:540
77899910354,43cyclictest0-21swapper/009:24:100
7789991033,25cyclictest27371-21ssh12:28:210
7789991033,22cyclictest29604-21ssh12:31:230
7789991032,28cyclictest4848-21seq08:37:120
7789991032,28cyclictest4848-21seq08:37:120
7789991032,24cyclictest13116-21ssh12:02:310
7789991032,24cyclictest13116-21ssh12:02:310
886221020,6sleep30-21swapper/309:57:263
77929910257,38cyclictest292-21dbus-daemon09:18:421
7789991022,87cyclictest5447-21grep08:39:050
7789991022,87cyclictest5447-21grep08:39:050
7789991013,25cyclictest30840-21ssh11:35:060
7789991013,25cyclictest30840-21ssh11:35:060
7789991013,25cyclictest18504-21ssh12:11:500
7789991013,23cyclictest30871-21ssh09:38:380
7789991012,23cyclictest22954-21runrttasks10:23:430
7789991003,23cyclictest11274-21taskset07:17:410
7789991002,25cyclictest13294-21seq09:03:330
7789991002,24cyclictest3731-21taskset10:47:030
7789991002,21cyclictest19349-21taskset07:42:350
7789999941,50cyclictest0-21swapper/007:09:060
778999993,25cyclictest14615-21modprobe07:28:420
778999992,21cyclictest25505-21ssh10:28:250
7792999861,30cyclictest0-21swapper/112:09:171
778999983,26cyclictest31486-21rm10:38:590
778999983,26cyclictest27287-21taskset08:06:000
778999983,26cyclictest27287-21taskset08:06:000
7801999741,17cyclictest23135-21latency_hist09:23:432
7792999753,33cyclictest0-21swapper/110:59:541
7792999747,42cyclictest0-21swapper/111:14:151
779299972,27cyclictest5143-21ssh10:49:111
779299972,27cyclictest5143-21ssh10:49:111
778999973,26cyclictest376-21sshd08:21:020
778999973,22cyclictest623-21ssh09:41:570
778999973,22cyclictest623-21ssh09:41:570
778999972,23cyclictest2873-21ssh09:46:190
778999972,23cyclictest19701-21seq11:14:410
778999972,23cyclictest11996-21ssh10:03:320
778999972,22cyclictest7175-21ssh11:50:560
7792999654,36cyclictest14938-21diskmemload12:06:131
7792999654,36cyclictest14938-21diskmemload12:06:131
778999963,26cyclictest15198-21ssh09:08:470
778999963,26cyclictest15198-21ssh09:08:470
778999963,22cyclictest1286-21taskset08:25:480
778999962,25cyclictest20727-21taskset07:44:370
778999962,23cyclictest8206-21runrttasks08:48:390
778999962,23cyclictest12957-21taskset07:23:210
778999962,22cyclictest2192-21ssh11:42:370
778999962,21cyclictest708-21seq12:38:330
778999962,21cyclictest19130-21ssh09:14:420
780199953,42cyclictest10286-21/usr/sbin/munin10:59:462
780199953,39cyclictest10970-21unixbench_singl07:14:202
778999952,22cyclictest14790-21ssh11:06:240
778999952,21cyclictest4753-21switchtime09:49:170
7801999442,9cyclictest0-21swapper/210:18:532
780199943,39cyclictest14059-21wc07:24:172
778999943,22cyclictest22556-21taskset07:52:350
778999942,22cyclictest17734-21taskset07:37:510
778999942,21cyclictest11374-21taskset08:54:530
778999942,21cyclictest11374-21taskset08:54:530
778999933,22cyclictest25585-21taskset07:59:430
778999933,22cyclictest25585-21taskset07:59:430
779299923,8cyclictest0-21swapper/112:23:531
7789999218,66cyclictest29580-21cut08:14:020
779299912,33cyclictest13669-21cpuspeed09:03:591
778999912,28cyclictest15766-21grep07:29:220
7801999043,11cyclictest23594-21ssh12:20:262
780199903,37cyclictest28626-21/usr/sbin/munin10:34:042
780199903,37cyclictest24010-21/usr/sbin/munin11:24:022
7792999049,35cyclictest27002-21sh11:29:031
778999903,21cyclictest18267-21latency_hist11:13:430
7801998971,12cyclictest16460-21/usr/sbin/munin07:34:102
7801998969,14cyclictest26378-21fschecks_count08:04:072
7801998969,14cyclictest26378-21fschecks_count08:04:072
780199894,40cyclictest32148-21apt-key11:38:452
780199894,40cyclictest32148-21apt-key11:38:452
7792998951,31cyclictest0-21swapper/108:24:031
779299893,7cyclictest0-21swapper/108:34:271
779299893,7cyclictest0-21swapper/108:34:271
779299892,31cyclictest26551-21netstat09:29:091
779299892,31cyclictest26551-21netstat09:29:091
779299892,27cyclictest27756-21/usr/sbin/munin08:09:141
779299892,27cyclictest27756-21/usr/sbin/munin08:09:141
778999893,27cyclictest26600-21taskset09:29:100
778999893,27cyclictest26600-21taskset09:29:100
7792998850,31cyclictest0-21swapper/112:19:051
778999883,27cyclictest1770-21/usr/sbin/munin08:29:000
7801998743,12cyclictest355-21runrttasks11:58:592
7801998743,12cyclictest355-21runrttasks11:58:592
7792998751,30cyclictest0-21swapper/109:09:021
7792998751,30cyclictest0-21swapper/109:09:021
779299873,8cyclictest0-21swapper/111:53:581
779299873,7cyclictest0-21swapper/107:14:001
779299872,35cyclictest15788-21/usr/sbin/munin11:08:551
779299872,29cyclictest355-21runrttasks07:21:021
7792998664,16cyclictest14938-21diskmemload12:14:191
7792998651,29cyclictest0-21swapper/110:34:071
7792998646,33cyclictest0-21swapper/108:53:441
779299862,37cyclictest23047-21sort10:23:441
7792998562,15cyclictest207-21systemd-journal08:32:011
7792998546,32cyclictest0-21swapper/108:44:081
7792998544,33cyclictest0-21swapper/109:39:171
7792998544,33cyclictest0-21swapper/109:39:171
779299853,8cyclictest0-21swapper/109:33:591
779299853,38cyclictest31364-21/usr/sbin/munin08:19:051
779299853,32cyclictest11953-21/usr/sbin/munin08:58:531
7792998531,44cyclictest0-21swapper/107:59:141
7792998531,44cyclictest0-21swapper/107:59:141
7801998463,15cyclictest7293-21sh10:54:012
779299843,7cyclictest0-21swapper/107:54:041
779299843,7cyclictest0-21swapper/107:43:531
779299843,39cyclictest10319-21/usr/sbin/munin08:54:371
779299843,39cyclictest10319-21/usr/sbin/munin08:54:371
779299842,40cyclictest4136-21/usr/sbin/munin09:49:021
779299842,37cyclictest1762-21apt-get10:43:531
779299842,34cyclictest18106-21/usr/sbin/munin07:38:531
7792998347,29cyclictest0-21swapper/110:09:151
779299833,8cyclictest0-21swapper/111:48:541
779299833,36cyclictest25912-21ssh10:28:531
779299832,35cyclictest22331-21ssh09:21:041
7792998246,30cyclictest20126-21diskmemload10:18:361
779299813,7cyclictest0-21swapper/108:04:001
779299813,7cyclictest0-21swapper/108:04:001
779299812,35cyclictest387-21runrttasks10:40:161
7801998058,16cyclictest17734-21latency_hist09:13:442
7801998058,16cyclictest17734-21latency_hist09:13:442
779299803,7cyclictest0-21swapper/108:13:531
779299803,38cyclictest1665-21ssh11:40:491
779299803,35cyclictest25107-21/usr/sbin/munin11:24:181
779299803,34cyclictest13838-21latency_hist12:03:451
779299803,34cyclictest13838-21latency_hist12:03:451
779299802,34cyclictest21313-21ssh11:18:531
779299802,34cyclictest16957-21/usr/sbin/munin07:34:121
779299802,28cyclictest24659-21runrttasks09:24:531
779299802,23cyclictest9815-21/usr/sbin/munin09:59:031
7792997961,7cyclictest0-21swapper/111:34:181
7792997961,7cyclictest0-21swapper/111:34:181
7792997939,32cyclictest0-21swapper/111:43:591
7792997939,32cyclictest0-21swapper/111:43:591
779299793,36cyclictest30961-21fschecks_time12:34:061
779299793,36cyclictest23255-21ssh10:23:581
779299793,26cyclictest7043-21fschecks_time09:54:041
779299793,23cyclictest355-21runrttasks07:23:591
779299792,32cyclictest7427-21/usr/sbin/munin10:54:011
779299792,29cyclictest8625-21mailstats07:09:141
779299782,32cyclictest12761-21ssh10:04:081
779299782,31cyclictest13129-21taskset11:03:581
7792997815,48cyclictest15198-21if_eth007:29:081
73742787,57sleep20-21swapper/207:05:042
779299773,31cyclictest27991-21df_inode12:29:011
7801997657,13cyclictest30423-21latency_hist12:33:432
779299762,35cyclictest21462-21/usr/sbin/munin07:48:581
7801997555,12cyclictest16890-21ssh12:09:062
7801997455,11cyclictest3725-21timerwakeupswit11:44:182
7801997455,11cyclictest3725-21timerwakeupswit11:44:182
780999733,54cyclictest13889-21taskset12:03:453
780999733,54cyclictest13889-21taskset12:03:453
7801997351,15cyclictest316-21rs:main2
780999723,56cyclictest10505-21chrt10:59:323
780999723,52cyclictest11034-21chrt10:00:223
7801997256,10cyclictest25433-21timerwakeupswit07:59:172
7801997256,10cyclictest25433-21timerwakeupswit07:59:172
7801997254,13cyclictest7781-21cyclictest08:19:172
7801997242,22cyclictest17082-21kworker/u19:0+nfsiod10:32:572
7801997240,23cyclictest21846-21sed11:19:092
7801997156,11cyclictest7781-21cyclictest08:44:172
7801997154,10cyclictest5497-21timerwakeupswit10:49:202
7801997154,10cyclictest5497-21timerwakeupswit10:49:202
7801997152,12cyclictest309-21systemd-logind09:58:432
7801997152,12cyclictest1-21systemd07:43:022
7801997150,15cyclictest2660-21sh11:43:442
780999703,56cyclictest32584-21taskset11:38:563
780999703,53cyclictest29438-21taskset10:34:193
780999702,53cyclictest0-21swapper/312:23:533
7801997045,18cyclictest17563-21idleruntime-cro10:13:412
7801997020,37cyclictest11044-21latency_hist11:58:452
7801996952,9cyclictest0-21swapper/208:09:192
7801996952,9cyclictest0-21swapper/208:09:192
7801996952,10cyclictest28088-21kworker/u17:2-writeback10:03:432
780199694,48cyclictest20972-21modprobe11:18:412
780199694,20cyclictest15810-21apt-get11:08:572
780999683,51cyclictest12984-21chrt10:04:153
780999682,51cyclictest3801-21taskset10:47:163
780999673,51cyclictest16009-21chrt11:09:053
780999673,50cyclictest5248-21taskset09:50:003
780999673,48cyclictest0-21swapper/311:53:583
780999672,51cyclictest13854-21chrt09:04:043
780999672,51cyclictest0-21swapper/309:33:593
780999672,49cyclictest18080-21chrt10:14:043
7801996752,10cyclictest21782-21kworker/u17:0-xprtiod12:24:052
7801996750,11cyclictest7781-21cyclictest08:39:192
7801996750,11cyclictest7781-21cyclictest08:39:192
7801996750,10cyclictest0-21swapper/212:14:182
780199674,15cyclictest3634-21apt-get08:34:272
780199674,15cyclictest3634-21apt-get08:34:272
780199674,12cyclictest13587-21apt-get09:03:552
780999663,51cyclictest29149-21chrt08:13:423
780999663,51cyclictest29149-21chrt08:13:423
780999663,50cyclictest0-21swapper/312:08:553
780999663,49cyclictest25376-21chrt11:25:063
780999663,49cyclictest23360-21taskset10:24:013
780999663,49cyclictest18479-21taskset11:13:513
780999663,46cyclictest26053-21chrt09:28:493
780999663,46cyclictest26053-21chrt09:28:493
780999662,51cyclictest31301-21taskset12:34:153
780999662,50cyclictest8186-21taskset11:53:443
780999662,50cyclictest31529-21chrt08:19:093
780999662,50cyclictest0-21swapper/311:03:543
7801996652,8cyclictest0-21swapper/208:53:562
7801996652,8cyclictest0-21swapper/208:53:562
7801996649,10cyclictest0-21swapper/208:18:412
7801996648,10cyclictest0-21swapper/209:43:592
7801996645,15cyclictest29053-21sh11:33:142
7801996633,24cyclictest5596-21ssh09:51:102
780999653,48cyclictest0-21swapper/312:18:543
780999652,49cyclictest17918-21taskset09:13:533
780999652,48cyclictest27605-21chrt10:31:193
7801996549,9cyclictest0-21swapper/210:44:192
7801996547,12cyclictest18442-21ntp_offset10:14:122
7801996546,10cyclictest0-21swapper/209:38:592
7801996546,10cyclictest0-21swapper/209:38:592
780199653,46cyclictest11953-21/usr/sbin/munin08:59:102
780999643,48cyclictest21000-21chrt09:19:043
780999643,47cyclictest11541-21taskset08:57:043
780999643,47cyclictest11541-21taskset08:57:043
780999642,50cyclictest4799-21chrt11:47:153
780999642,50cyclictest4799-21chrt11:47:153
780999642,50cyclictest24358-21chrt09:24:193
780999642,48cyclictest9155-21chrt07:11:433
780999642,48cyclictest7521-21taskset10:54:053
780999642,48cyclictest22095-21chrt10:21:013
7801996448,9cyclictest0-21swapper/208:33:452
7801996433,22cyclictest28208-21cron09:33:412
7801996433,22cyclictest28208-21cron09:33:412
780999634,45cyclictest0-21swapper/308:38:553
780999634,45cyclictest0-21swapper/308:38:553
780999633,47cyclictest27253-21chrt08:05:333
780999633,47cyclictest27253-21chrt08:05:333
780999633,47cyclictest19718-21chrt07:43:543
780999633,46cyclictest1348-21taskset09:43:483
780999633,46cyclictest13094-21chrt09:00:573
780999633,46cyclictest0-21swapper/307:13:553
780999633,45cyclictest0-21swapper/308:23:573
780999632,48cyclictest30220-21taskset12:33:133
780999632,47cyclictest30461-21chrt08:15:293
780999632,47cyclictest25629-21taskset08:00:183
780999632,47cyclictest25629-21taskset08:00:183
780999632,47cyclictest19231-21chrt07:41:033
780999632,44cyclictest8464-21chrt08:48:453
7801996345,10cyclictest0-21swapper/207:48:422
780199633,44cyclictest22230-21grep07:49:152
780199633,43cyclictest23217-21/usr/sbin/munin10:24:192
780199633,34cyclictest13213-21latency_hist07:23:452
780999624,45cyclictest29492-21chrt11:33:493
780999624,45cyclictest29492-21chrt11:33:493
780999623,47cyclictest28634-21chrt11:31:513
780999623,46cyclictest31631-21chrt10:39:013
780999623,46cyclictest15299-21chrt10:09:023
780999623,45cyclictest24040-21taskset07:56:013
780999622,47cyclictest3660-21chrt08:34:073
780999622,47cyclictest3660-21chrt08:34:073
780999622,47cyclictest22296-21chrt07:49:173
780999622,47cyclictest16022-21chrt07:31:273
780999622,46cyclictest17660-21chrt07:36:533
780999622,46cyclictest12691-21chrt07:19:533
780199625,40cyclictest23753-21timerwakeupswit07:54:252
780199624,41cyclictest304-21df08:24:002
7801996235,20cyclictest30096-21kworker/u19:1+nfsiod10:42:132
780199623,46cyclictest20370-21modprobe09:18:412
780199623,43cyclictest13747-21proc_pri11:04:162
7801996233,9cyclictest0-21swapper/211:50:202
780999613,46cyclictest21127-21chrt12:16:393
780999613,46cyclictest15565-21chrt09:09:073
780999613,46cyclictest15565-21chrt09:09:073
780999613,44cyclictest6856-21chrt08:43:513
780999612,46cyclictest6329-21chrt10:51:423
780999612,46cyclictest6329-21chrt10:51:423
7801996145,9cyclictest0-21swapper/209:24:112
780199614,41cyclictest355-21runrttasks10:04:172
7801996144,10cyclictest0-21swapper/207:29:172
780999603,46cyclictest32427-21chrt09:39:513
780999603,46cyclictest32427-21chrt09:39:513
780999603,46cyclictest14707-21taskset12:04:363
780999603,46cyclictest14707-21taskset12:04:363
780999602,46cyclictest14485-21chrt07:27:223
7801996045,9cyclictest0-21swapper/212:34:032
780199603,41cyclictest14291-21df12:04:242
780199603,41cyclictest14291-21df12:04:242
780199602,43cyclictest28587-21cpu09:33:592
780999593,44cyclictest2880-21chrt08:30:393
780999582,45cyclictest21278-21chrt11:18:503
634925010,22sleep30-21swapper/307:04:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional