You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-16 - 00:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Tue Jul 15, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26531214018,90sleep226614-21sed07:01:312
26207211610,69sleep025638-21/usr/sbin/munin07:01:250
271539910948,10cyclictest0-21swapper/108:45:461
27155991082,70cyclictest20018-21chrt08:21:123
1819121070,7sleep118169-21ssh10:36:021
1819121070,7sleep118169-21ssh10:36:021
27155999918,41cyclictest0-21swapper/310:06:153
2715399994,32cyclictest20312-21latency_hist12:35:451
27150999958,34cyclictest0-21swapper/011:46:260
27150999958,34cyclictest0-21swapper/011:46:260
2715399974,38cyclictest4334-21latency_hist11:10:451
2715399974,38cyclictest4334-21latency_hist11:10:451
2715399973,36cyclictest4144-21timerandwakeup07:31:191
2715099974,82cyclictest15749-21ssh11:30:580
2715099974,82cyclictest15749-21ssh11:30:580
225382960,7sleep30-21swapper/309:43:453
27150999569,17cyclictest14556-21latency_hist08:05:450
2698029314,59sleep30-21swapper/307:05:443
2715099923,59cyclictest19997-21apt-get08:21:120
27154998920,48cyclictest0-21swapper/211:50:412
27154998920,48cyclictest0-21swapper/211:50:412
27153998934,27cyclictest0-21swapper/109:51:181
27153998761,12cyclictest181rcu_preempt11:04:511
27154998614,62cyclictest18769-21apt-get11:36:192
27150998641,31cyclictest1903-21apt-get10:05:570
27150998629,23cyclictest181rcu_preempt07:55:570
27154998559,19cyclictest2828-21sh09:06:172
27154998510,66cyclictest1995-21runrttasks10:06:152
27153998552,13cyclictest181rcu_preempt09:21:041
27153998360,17cyclictest27605-21idleruntime07:06:071
27153998340,21cyclictest0-21swapper/111:59:131
27154998262,13cyclictest27142-21cyclictest10:16:052
27154998262,13cyclictest27142-21cyclictest10:16:052
27153998262,14cyclictest202-21systemd-journal10:45:411
27150998256,17cyclictest12900-21sh11:25:490
27150998256,17cyclictest12900-21sh11:25:490
27150998246,11cyclictest181rcu_preempt11:20:410
2715499813,41cyclictest7393-21apt-get11:16:222
27154998115,27cyclictest31224-21ssh10:00:462
2715399819,29cyclictest18769-21apt-get11:35:591
27153998159,16cyclictest202-21systemd-journal10:33:301
27153998159,16cyclictest202-21systemd-journal10:33:301
27153998149,11cyclictest181rcu_preempt12:11:171
27150998158,16cyclictest27142-21cyclictest12:06:410
27150998158,16cyclictest27142-21cyclictest12:06:410
27150998155,17cyclictest18342-21fschecks_time10:36:070
27150998155,17cyclictest18342-21fschecks_time10:36:070
27150998148,10cyclictest181rcu_preempt11:21:090
27150998133,16cyclictest29573-21latency_hist08:50:450
27153998055,18cyclictest356-21runrttasks09:10:441
2715399803,49cyclictest1903-21apt-get10:06:251
27150998060,13cyclictest23461-21fschecks_time08:31:040
27150998060,13cyclictest23461-21fschecks_time08:31:040
27150998059,13cyclictest7418-21ssh10:16:000
27150998059,13cyclictest7418-21ssh10:16:000
27150998057,13cyclictest21103-21ssh10:41:090
27150998044,8cyclictest181rcu_preempt07:50:560
27154997963,10cyclictest5932-21unixbench_singl07:36:182
27154997923,24cyclictest0-21swapper/210:26:212
27154997823,40cyclictest0-21swapper/208:10:552
27153997858,11cyclictest0-21swapper/107:46:001
27153997858,11cyclictest0-21swapper/107:46:001
27153997856,16cyclictest15995-21fschecks_time11:31:061
27153997856,16cyclictest15995-21fschecks_time11:31:061
27153997855,15cyclictest25811-21latency_hist10:50:441
27150997860,12cyclictest30043-21switchtime07:11:210
27150997834,8cyclictest181rcu_preempt10:59:070
27154997761,11cyclictest27142-21cyclictest12:05:442
27154997751,18cyclictest12638-21latency_hist10:25:452
27154997710,42cyclictest0-21swapper/208:17:022
2715399779,58cyclictest19491-21apt-get08:21:031
27153997759,12cyclictest378-21rs:main1
27153997757,13cyclictest15869-21ssh12:26:351
27153997754,15cyclictest1691-21latency_hist07:25:461
27153997729,24cyclictest0-21swapper/112:21:161
27150997745,8cyclictest181rcu_preempt11:55:580
2715599764,58cyclictest12856-21taskset10:26:033
2715499769,37cyclictest0-21swapper/207:10:492
27154997652,14cyclictest7693-21runrttasks12:13:012
27153997661,11cyclictest27142-21cyclictest09:26:081
27153997652,18cyclictest12871-21latency_hist11:25:461
2715399762,33cyclictest18979-21awk08:17:161
27150997659,10cyclictest5551-21ssh10:11:230
27150997653,15cyclictest27142-21cyclictest09:49:390
27150997653,15cyclictest27142-21cyclictest09:49:390
27150997649,9cyclictest181rcu_preempt10:00:570
27150997649,9cyclictest181rcu_preempt10:00:570
2715099764,35cyclictest26371-21fschecks_time10:51:060
2715099764,35cyclictest26371-21fschecks_time10:51:060
27150997641,11cyclictest25587-21kworker/u18:1-events_unbound08:36:070
27154997560,11cyclictest27142-21cyclictest11:31:192
27154997560,11cyclictest27142-21cyclictest11:31:192
27154997536,29cyclictest3358-21apt-get07:30:562
27154997450,15cyclictest1358-21apt-get10:05:452
27154997450,15cyclictest1358-21apt-get10:05:452
27154997443,24cyclictest17425-21latency_hist12:30:452
27154997427,35cyclictest24291-21ssh10:46:202
27154997411,24cyclictest16254-21grep10:31:202
27154997411,24cyclictest16254-21grep10:31:202
27150997458,10cyclictest30611-21ssh09:59:040
27150997452,15cyclictest6491-21latency_hist07:40:460
27150997441,7cyclictest181rcu_preempt08:11:050
27150997429,12cyclictest16011-21processes10:31:170
27150997429,12cyclictest16011-21processes10:31:170
2715499738,49cyclictest17574-21kworker/u19:0+ext4-rsv-conversion08:21:122
27154997349,17cyclictest27142-21cyclictest07:51:572
27153997355,10cyclictest0-21swapper/111:26:191
27153997355,10cyclictest0-21swapper/111:26:191
27153997354,8cyclictest0-21swapper/107:56:171
27153997348,19cyclictest28997-21fschecks_count10:56:041
27153997342,11cyclictest58-21kcompactd009:46:061
27153997342,11cyclictest58-21kcompactd009:46:061
27153997318,9cyclictest181rcu_preempt08:30:561
27153997318,9cyclictest181rcu_preempt08:30:561
27150997357,10cyclictest20116-21sh12:35:420
27150997351,14cyclictest8164-21ssh12:14:250
2715099734,45cyclictest18917-21rm11:36:390
27150997322,24cyclictest21881-21if_err_eth011:41:140
2704327312,37sleep127045-21munin-run07:05:461
286172723,7sleep00-21swapper/011:52:430
2715599723,57cyclictest7493-21taskset12:12:263
2715599723,55cyclictest1681-21taskset11:05:453
27154997252,13cyclictest10547-21kernelversion09:21:092
27154997221,44cyclictest6876-21tune2fs07:41:072
27153997254,12cyclictest21773-21fschecks_count08:26:051
27153997254,12cyclictest21773-21fschecks_count08:26:051
27153997235,10cyclictest27142-21cyclictest09:04:391
27153997223,14cyclictest181rcu_preempt11:51:101
27150997258,8cyclictest0-21swapper/010:23:210
27150997254,12cyclictest10204-21ssh12:16:510
2715099723,55cyclictest26617-21memory09:51:100
27154997145,18cyclictest21056-21fschecks_time09:41:082
27154997144,20cyclictest1926-21fschecks_count07:26:052
27154997130,30cyclictest9075-21timerwakeupswit07:46:212
27154997130,30cyclictest9075-21timerwakeupswit07:46:212
27154997121,43cyclictest0-21swapper/207:10:592
27154997110,30cyclictest375-21snmpd11:10:452
27154997110,30cyclictest375-21snmpd11:10:452
27153997147,13cyclictest0-21swapper/108:40:441
2715399714,57cyclictest6805-21apt-get11:16:171
27153997138,14cyclictest0-21swapper/112:05:201
27150997154,10cyclictest11896-21ssh09:23:420
27150997144,9cyclictest58-21kcompactd007:31:110
2715599704,31cyclictest23834-21chrt10:46:093
2715599703,54cyclictest11814-21chrt07:56:093
2715599703,33cyclictest10741-21chrt09:21:153
2715599703,31cyclictest26026-21chrt08:38:583
27154997051,13cyclictest364-21ntpd09:33:192
27154997051,13cyclictest364-21ntpd09:33:192
27154997050,13cyclictest26042-21kworker/u17:1-events_unbound08:46:112
27154997045,18cyclictest356-21runrttasks12:09:302
27154997045,18cyclictest356-21runrttasks12:09:302
2715499703,44cyclictest13291-21netstat09:26:092
27154997019,21cyclictest18537-21taskset09:36:062
27154997015,39cyclictest0-21swapper/210:56:122
27154997015,23cyclictest1808-21diskmemload11:44:362
27154997010,29cyclictest26059-21/usr/sbin/munin10:51:172
27154997010,29cyclictest26059-21/usr/sbin/munin10:51:172
27154997010,25cyclictest5208-21ssh10:11:182
27153997054,10cyclictest58-21kcompactd009:15:591
27153997048,16cyclictest6528-21latency_hist07:40:461
27153997048,16cyclictest18001-21kworker/u17:4-xprtiod10:51:161
27153997048,16cyclictest18001-21kworker/u17:4-xprtiod10:51:161
27153997045,14cyclictest0-21swapper/108:46:051
27153997042,10cyclictest181rcu_preempt08:55:441
27153997036,8cyclictest0-21swapper/110:28:211
27153997030,8cyclictest181rcu_preempt09:41:021
27150997049,15cyclictest19491-21apt-get08:20:420
2715099703,51cyclictest31541-21ssh11:00:590
2715099703,41cyclictest31002-21latency_hist08:55:450
2715599693,32cyclictest17754-21chrt12:31:023
27154996936,23cyclictest21881-21cut08:26:062
27154996936,23cyclictest21881-21cut08:26:062
2715499693,45cyclictest7419-21/usr/sbin/munin09:16:102
27154996912,32cyclictest23538-21interrupts08:31:052
27154996912,32cyclictest23538-21interrupts08:31:052
27153996953,8cyclictest0-21swapper/111:15:461
27153996951,11cyclictest0-21swapper/110:01:061
27153996951,11cyclictest0-21swapper/110:01:061
27153996950,11cyclictest19146-21sh09:36:221
27153996948,12cyclictest0-21swapper/110:11:081
27153996947,15cyclictest4842-21cpuspeed09:10:591
27153996933,16cyclictest181rcu_preempt09:56:011
27150996950,11cyclictest0-21swapper/011:15:430
27150996948,15cyclictest21304-21sh09:41:130
27150996947,14cyclictest27142-21cyclictest12:24:110
27150996940,9cyclictest181rcu_preempt08:05:540
2715099693,45cyclictest15254-21ssh10:30:460
2715599684,51cyclictest8735-21chrt07:46:133
2715599684,51cyclictest8735-21chrt07:46:133
2715599684,31cyclictest17639-21chrt09:34:573
2715599684,31cyclictest17639-21chrt09:34:573
2715599683,53cyclictest28591-21chrt10:55:453
2715599683,53cyclictest28591-21chrt10:55:453
2715599683,51cyclictest16513-21chrt12:28:303
2715599683,33cyclictest24489-21taskset11:46:083
2715599683,33cyclictest24489-21taskset11:46:083
2715599682,34cyclictest4115-21chrt07:31:183
2715599682,34cyclictest27829-21chrt09:53:543
2715599682,34cyclictest13638-21chrt11:26:163
2715599682,34cyclictest13638-21chrt11:26:163
2715599682,33cyclictest8047-21chrt09:16:133
27155996823,35cyclictest0-21swapper/311:15:593
2715499683,47cyclictest25631-21grep08:36:192
2715499683,43cyclictest20336-21latency_hist12:35:452
27154996828,30cyclictest11446-21apt-get07:56:002
27154996821,13cyclictest0-21swapper/209:55:032
27154996816,20cyclictest13335-21awk11:26:072
27154996816,20cyclictest13335-21awk11:26:072
27154996815,38cyclictest13903-21ssh12:24:192
27153996850,11cyclictest24313-21diskmemload11:46:031
27153996850,11cyclictest24313-21diskmemload11:46:031
27153996847,11cyclictest0-21swapper/107:15:451
27153996846,9cyclictest0-21swapper/108:10:441
27153996845,15cyclictest0-21swapper/111:41:471
27153996838,20cyclictest0-21swapper/108:15:461
2715399683,14cyclictest5720-21idleruntime-cro12:10:411
2715399683,14cyclictest5720-21idleruntime-cro12:10:411
27153996831,27cyclictest2150-21latency07:26:091
27150996851,11cyclictest18604-21ssh09:36:100
27150996842,19cyclictest1808-21diskmemload09:15:440
27150996840,20cyclictest7925-21modprobe07:45:420
27150996838,9cyclictest181rcu_preempt09:05:580
2715099683,35cyclictest1770-21/usr/sbin/munin11:06:020
2715099683,35cyclictest1770-21/usr/sbin/munin11:06:020
2715599673,33cyclictest20728-21chrt10:40:583
2715599673,32cyclictest6816-21taskset09:14:433
2715599673,31cyclictest6853-21chrt11:15:423
2715599673,31cyclictest32340-21chrt08:56:563
2715599672,32cyclictest31151-21chrt10:00:433
2715599672,32cyclictest15938-21chrt10:31:143
2715599672,32cyclictest15938-21chrt10:31:143
27154996743,15cyclictest25275-21sh09:48:532
27154996743,15cyclictest25275-21sh09:48:532
2715499673,45cyclictest32569-21latency_hist12:00:452
2715499673,37cyclictest31495-21/usr/sbin/munin11:01:122
27154996729,29cyclictest464-21apt-get09:00:592
27154996716,38cyclictest20566-21cat10:40:452
27154996716,38cyclictest20566-21cat10:40:452
27154996710,25cyclictest30522-21sort08:51:202
27153996738,20cyclictest32635-21modprobe09:00:421
2715099674,48cyclictest356-21runrttasks09:32:100
2715099674,48cyclictest356-21runrttasks09:32:100
2715099673,53cyclictest17443-21ssh12:30:450
2715099673,52cyclictest1922-21grep07:26:050
27150996733,15cyclictest181rcu_preempt09:01:090
27150996726,13cyclictest29189-21kworker/u17:2+xprtiod09:16:050
2715599665,31cyclictest13065-21chrt08:00:593
2715599662,34cyclictest18230-21taskset08:16:553
2715599662,33cyclictest2536-21taskset09:06:083
2715599662,32cyclictest16978-21taskset11:31:513
2715599662,32cyclictest16978-21taskset11:31:513
27154996651,10cyclictest20692-21/usr/sbin/munin10:41:202
2715499664,41cyclictest356-21runrttasks08:57:232
2715499664,32cyclictest0-21swapper/211:51:172
27154996643,13cyclictest0-21swapper/211:11:192
27154996643,12cyclictest0-21swapper/212:16:182
27154996626,11cyclictest181rcu_preempt11:25:452
27153996643,12cyclictest0-21swapper/110:16:061
27153996643,12cyclictest0-21swapper/110:16:061
27153996629,14cyclictest0-21swapper/107:51:221
27153996626,15cyclictest0-21swapper/112:17:391
2715099663,44cyclictest22502-21unixbench_multi08:26:190
2715099663,44cyclictest22502-21unixbench_multi08:26:190
2715599653,33cyclictest11771-21chrt11:23:043
2715599653,30cyclictest5711-21chrt10:11:293
2715599652,31cyclictest19671-21chrt09:38:093
2715499653,41cyclictest15622-21grep08:06:192
2715499653,37cyclictest30968-21/usr/sbin/munin07:16:162
27154996531,21cyclictest32595-21apt-get07:20:582
2715099654,26cyclictest27527-21fschecks_count07:06:060
2715099653,40cyclictest9106-21unixbench_multi07:46:210
2715099653,40cyclictest9106-21unixbench_multi07:46:210
2715099652,45cyclictest384-21sshd09:27:300
2715599644,23cyclictest18040-21chrt10:35:563
2715599644,23cyclictest18040-21chrt10:35:563
2715599643,31cyclictest30500-21chrt10:58:243
2715599643,31cyclictest12908-21taskset12:21:213
2715599642,31cyclictest29357-21chrt07:11:013
2715599642,31cyclictest21437-21taskset11:40:503
2715599642,26cyclictest8888-21chrt12:15:463
2715499644,41cyclictest7270-21latency_hist09:15:452
27154996427,27cyclictest26446-21apt-get08:40:552
27154996419,36cyclictest13278-21grep08:01:092
27153996442,13cyclictest0-21swapper/109:31:081
27153996442,13cyclictest0-21swapper/109:31:081
2715399642,47cyclictest13828-21ls08:01:221
2715099643,41cyclictest31472-21fschecks_count08:56:060
2715099643,40cyclictest26427-21/usr/sbin/munin08:40:540
2715099643,25cyclictest424-21fschecks_time07:21:070
2715599632,32cyclictest501-21chrt10:02:543
2715599632,32cyclictest501-21chrt10:02:543
2715599632,29cyclictest14996-21chrt09:30:113
2715599632,28cyclictest24473-21chrt08:35:013
2715599632,28cyclictest24473-21chrt08:35:013
27153996346,12cyclictest0-21swapper/107:41:061
27153996339,13cyclictest0-21swapper/107:16:171
27150996338,16cyclictest0-21swapper/012:01:270
2715099633,33cyclictest25965-21latency_hist10:50:450
2715599623,28cyclictest20105-21chrt11:37:253
2715599623,25cyclictest5136-21chrt07:36:003
2715599623,25cyclictest14943-21chrt08:06:053
2715599622,30cyclictest26450-21chrt08:40:563
2715599622,30cyclictest23431-21chrt09:45:523
2715599622,30cyclictest23431-21chrt09:45:523
2715599612,26cyclictest9949-21chrt07:51:033
2715599603,33cyclictest31273-21chrt11:57:063
2715599603,26cyclictest10771-21taskset10:21:173
2715599603,25cyclictest9371-21chrt10:19:433
2715599603,25cyclictest9371-21chrt10:19:433
2715599602,25cyclictest1430-21chrt07:25:153
2715099602,45cyclictest356-21runrttasks07:17:090
2715599593,23cyclictest3176-21chrt12:05:583
2715599593,23cyclictest3176-21chrt12:05:583
2715599593,23cyclictest28209-21taskset11:51:273
2715599593,22cyclictest30964-21chrt08:55:263
2715599592,22cyclictest1276-21chrt12:01:283
2715599592,22cyclictest1002-21chrt09:01:133
2715599583,22cyclictest17783-21chrt08:15:453
2715599582,24cyclictest3815-21chrt11:09:263
2715599582,24cyclictest3815-21chrt11:09:263
2715599582,24cyclictest31143-21chrt07:16:033
27155995818,31cyclictest286332chrt07:10:413
2715599574,44cyclictest0-21swapper/308:26:183
2715599574,44cyclictest0-21swapper/308:26:183
2715599572,24cyclictest7588-21chrt07:41:233
2715599572,22cyclictest29061-21chrt08:46:313
2715599562,43cyclictest1803-21taskset07:26:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional