You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-30 - 07:39
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Tue Jun 30, 2026 00:44:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20509211612,75sleep220579-21awk18:53:462
21292991053,28cyclictest1178-21/usr/sbin/munin22:53:292
212789910455,38cyclictest17597-21latency_hist21:33:050
21292991033,28cyclictest15314-21seq00:09:572
212789910327,42cyclictest25200-21/usr/sbin/munin23:33:260
21292991013,30cyclictest2051-21sh23:48:582
21292991013,30cyclictest2051-21sh23:48:582
21292991002,30cyclictest18366-21seq23:22:432
212839910057,37cyclictest352-21runrttasks22:18:191
2129299982,28cyclictest29916-21runrttasks23:40:552
21278999867,21cyclictest741-21apt-get19:33:160
21278999828,39cyclictest24787-21ssh21:43:410
114842980,6sleep20-21swapper/220:03:352
21278999764,25cyclictest30123-21apt-get22:48:170
21278999764,25cyclictest30123-21apt-get22:48:170
2129299963,74cyclictest27151-21ssh22:43:172
2129299962,27cyclictest15091-21seq20:16:132
21292999449,38cyclictest24171-21sh22:38:162
21292999449,38cyclictest24171-21sh22:38:162
21278999464,20cyclictest5831-21apt-get19:49:090
21278999464,20cyclictest5831-21apt-get19:49:090
21278999455,31cyclictest0-21swapper/021:48:150
21278999455,31cyclictest0-21swapper/021:48:150
21278999441,43cyclictest13677-21ssh00:08:050
21292999334,32cyclictest0-21swapper/220:01:352
21278999329,38cyclictest0-21swapper/000:16:490
2129299922,27cyclictest23679-21taskset22:37:382
21278999270,11cyclictest6620-21apt-get23:03:170
21278999270,11cyclictest6620-21apt-get23:03:170
21278999270,11cyclictest6620-21apt-get23:03:170
21278999263,20cyclictest24159-21apt-get22:38:180
21278999263,20cyclictest24159-21apt-get22:38:180
21278999249,28cyclictest23627-21apt-get20:43:180
21278999249,28cyclictest23627-21apt-get20:43:180
21292999133,33cyclictest0-21swapper/220:22:572
21283999143,41cyclictest19393-21taskset00:18:001
21278999156,23cyclictest21282-21latency_hist18:58:060
21278999136,45cyclictest32650-21/usr/sbin/munin21:58:120
21278999123,33cyclictest6203-21kworker/u17:0+rpciod22:14:140
21278999123,33cyclictest6203-21kworker/u17:0+rpciod22:14:140
21292999049,35cyclictest18775-21ntp_offset22:28:312
21278999057,24cyclictest31595-21apt-get19:28:180
21278999035,19cyclictest31110-21sh21:54:360
21292998948,35cyclictest3358-21sh22:03:192
21283998934,47cyclictest0-21swapper/120:28:381
21283998934,47cyclictest0-21swapper/120:28:381
21278998969,10cyclictest26754-21apt-get19:13:190
21278998963,16cyclictest7760-21apt-get23:58:200
21278998963,16cyclictest7760-21apt-get23:58:200
21278998953,26cyclictest21809-21apt-get23:28:160
21278998947,34cyclictest6209-21taskset22:08:130
21278998942,42cyclictest0-21swapper/022:23:130
21278998942,42cyclictest0-21swapper/022:23:130
21278998933,21cyclictest4855-21rm22:58:590
21278998933,21cyclictest4855-21rm22:58:590
204732899,55sleep120508-21swap18:53:451
2129299883,40cyclictest25010-21http20:48:022
2129299883,40cyclictest25010-21http20:48:022
2128399884,30cyclictest26479-21modprobe19:13:021
2128399883,33cyclictest25753-21fschecks_time20:48:461
2128399883,33cyclictest25753-21fschecks_time20:48:461
21278998836,17cyclictest19248-21sh21:34:530
2129299874,6cyclictest0-21swapper/223:03:172
2129299874,6cyclictest0-21swapper/223:03:172
2129299874,6cyclictest0-21swapper/223:03:172
2129299873,8cyclictest0-21swapper/219:13:192
21292998737,44cyclictest0-21swapper/223:23:232
2129299872,23cyclictest352-21runrttasks19:41:372
2128399873,36cyclictest7303-21latency_hist19:53:051
2128399873,36cyclictest7303-21latency_hist19:53:051
2128399873,33cyclictest16022-21runrttasks23:18:451
2128399873,31cyclictest16842-21modprobe20:23:021
21278998761,18cyclictest12305-21apt-get20:08:170
21278998758,20cyclictest660-21apt-get22:53:160
21278998758,20cyclictest25497-21apt-get20:48:390
21278998758,20cyclictest25497-21apt-get20:48:390
2129299863,39cyclictest12303-21idleruntime-cro23:13:012
2129299863,13cyclictest7760-21apt-get23:58:172
2129299863,13cyclictest7760-21apt-get23:58:172
21292998617,48cyclictest0-21swapper/221:29:192
2128399864,28cyclictest5195-21/usr/sbin/munin23:53:321
21283998637,40cyclictest0-21swapper/100:08:041
21283998634,44cyclictest0-21swapper/100:23:181
2128399863,42cyclictest8484-21/usr/sbin/munin21:18:301
2128399863,42cyclictest8484-21/usr/sbin/munin21:18:301
2128399863,33cyclictest29040-21fschecks_count20:58:241
2128399863,30cyclictest5599-21apt-key19:48:051
2128399863,30cyclictest26734-21/usr/sbin/munin19:13:301
2128399863,28cyclictest15136-21/usr/sbin/munin22:23:391
2128399863,28cyclictest15136-21/usr/sbin/munin22:23:391
21283998618,30cyclictest3583-21irqstats22:03:261
21278998666,11cyclictest18752-21apt-get20:28:170
21278998666,11cyclictest18752-21apt-get20:28:170
21278998661,15cyclictest21158-21apt-get22:33:130
21278998653,26cyclictest27119-21apt-get22:43:150
21278998652,22cyclictest3181-21apt-get22:03:170
21278998648,31cyclictest7068-21kworker/u19:0-nfsiod23:17:370
21278998637,40cyclictest12236-21latency_hist20:08:050
21292998537,41cyclictest0-21swapper/222:48:172
21292998537,41cyclictest0-21swapper/222:48:172
2129299853,27cyclictest356-21seq19:31:452
2129299853,23cyclictest31038-21seq19:24:482
2129299853,23cyclictest31038-21seq19:24:482
2129299852,8cyclictest0-21swapper/220:35:432
2129299852,8cyclictest0-21swapper/220:35:432
2128399854,28cyclictest22223-21fschecks_count20:38:251
21283998539,39cyclictest0-21swapper/123:58:281
21283998539,39cyclictest0-21swapper/123:58:281
2128399853,44cyclictest18208-21cpu22:28:181
2128399853,42cyclictest535-21idleruntime21:58:251
2128399853,35cyclictest15568-21sh00:10:411
2128399853,33cyclictest23660-21/usr/sbin/munin21:43:131
2128399853,32cyclictest18613-21latency_hist20:28:051
2128399853,32cyclictest18613-21latency_hist20:28:051
2128399853,31cyclictest20705-21unixbench_multi00:18:381
2128399853,30cyclictest21132-21/usr/sbin/munin22:33:361
2128399852,30cyclictest2221-21latency_hist19:38:051
21292998439,39cyclictest28456-21diskmemload00:04:022
2129299843,7cyclictest0-21swapper/221:13:152
21292998436,40cyclictest0-21swapper/221:38:192
21292998436,40cyclictest0-21swapper/221:38:192
2129299843,37cyclictest28495-21runrttasks20:58:022
2129299843,34cyclictest12294-21find20:08:152
2129299842,8cyclictest0-21swapper/219:49:092
2129299842,8cyclictest0-21swapper/219:49:092
21283998464,14cyclictest1969-21latency_hist21:08:041
21283998435,42cyclictest0-21swapper/120:13:351
21283998435,37cyclictest0-21swapper/121:13:061
21283998435,37cyclictest0-21swapper/121:13:061
2128399843,33cyclictest22384-21grep18:58:401
2128399843,32cyclictest1-21systemd22:58:021
2128399843,32cyclictest11695-21/usr/sbin/munin21:24:151
2128399843,32cyclictest11695-21/usr/sbin/munin21:24:151
2128399843,30cyclictest14705-21/usr/sbin/munin21:28:341
2128399843,30cyclictest10001-21unixbench_singl19:58:381
2128399842,39cyclictest23627-21apt-get20:43:141
2128399842,39cyclictest23627-21apt-get20:43:141
21278998466,10cyclictest27113-21apt-get20:53:170
21278998445,29cyclictest13907-21apt-get20:13:190
21278998427,32cyclictest5812-21fschecks_time21:13:250
198142849,54sleep30-21swapper/318:53:143
2129299833,7cyclictest0-21swapper/220:58:152
2129299833,32cyclictest24049-21/usr/sbin/munin21:43:262
2129299833,31cyclictest17689-21munin-node21:33:122
2129299833,29cyclictest14277-21sh23:15:162
2128399834,33cyclictest30104-21/usr/sbin/munin22:48:351
2128399834,33cyclictest30104-21/usr/sbin/munin22:48:351
2128399834,30cyclictest4650-21grep22:58:381
2128399834,30cyclictest4650-21grep22:58:381
2128399833,26cyclictest24700-21latency_hist23:33:051
2128399833,26cyclictest11597-21grep20:03:361
21283998330,47cyclictest0-21swapper/123:03:251
21283998330,47cyclictest0-21swapper/123:03:251
21283998330,47cyclictest0-21swapper/123:03:251
21283998329,47cyclictest0-21swapper/123:39:331
21283998324,8cyclictest0-21swapper/119:03:271
2128399832,36cyclictest28456-21diskmemload23:13:181
2128399832,32cyclictest21111-21cut21:38:271
2128399832,32cyclictest21111-21cut21:38:271
21283998315,23cyclictest24782-21sh22:38:351
21283998315,23cyclictest24782-21sh22:38:351
21278998338,33cyclictest22194-21grep20:38:250
21278998338,31cyclictest11715-21apt-get21:24:060
21278998338,31cyclictest11715-21apt-get21:24:060
21278998337,36cyclictest23461-21cat00:23:360
21278998335,20cyclictest11227-21systemd-run21:23:030
21278998335,20cyclictest11227-21systemd-run21:23:030
21278998332,25cyclictest11819-21runrttasks23:11:440
21278998325,21cyclictest0-21swapper/021:12:070
21278998325,21cyclictest0-21swapper/021:12:070
2129299823,38cyclictest5412-21/usr/sbin/munin23:53:372
2129299823,36cyclictest24561-21latency_hist23:33:032
2129299823,33cyclictest6987-21/usr/sbin/munin22:08:342
2129299822,36cyclictest30764-21uptime21:53:402
2129299822,36cyclictest14322-21seq21:27:532
2129299822,36cyclictest14322-21seq21:27:532
2129299822,32cyclictest12371-21/usr/sbin/munin22:18:222
2129299822,29cyclictest17248-21/usr/sbin/munin00:13:292
2128399824,29cyclictest6030-21latency21:13:291
2128399824,29cyclictest29945-21/usr/sbin/munin19:23:241
2128399824,29cyclictest29945-21/usr/sbin/munin19:23:241
2128399823,33cyclictest7216-21unixbench_multi22:08:371
2128399823,31cyclictest32169-21ps19:28:341
2128399823,31cyclictest19021-21cut23:23:221
2128399823,30cyclictest21857-21latency_hist20:38:041
2128399823,30cyclictest21857-21latency_hist20:38:041
2128399823,26cyclictest29650-21/usr/sbin/munin21:53:391
2128399822,32cyclictest11971-21latency_hist22:18:041
2128399822,32cyclictest11971-21latency_hist22:18:041
2128399822,31cyclictest7858-21wc19:53:291
2128399822,31cyclictest7858-21wc19:53:291
2128399822,30cyclictest9778-21sh23:08:221
21278998261,11cyclictest9050-21apt-get19:58:170
21278998255,18cyclictest4796-21apt-get23:53:230
21278998254,18cyclictest29964-21apt-get19:23:170
21278998254,18cyclictest29964-21apt-get19:23:170
21278998242,32cyclictest29100-21mailstats23:39:320
21278998238,33cyclictest15808-21apt-get23:18:160
2127899823,56cyclictest1769-21apt-get23:48:200
2127899823,56cyclictest1769-21apt-get23:48:200
2129299815,64cyclictest18752-21apt-get20:28:152
2129299815,64cyclictest18752-21apt-get20:28:152
21292998141,33cyclictest0-21swapper/221:48:232
21292998141,33cyclictest0-21swapper/221:48:232
2129299813,6cyclictest0-21swapper/223:33:172
2129299813,42cyclictest31774-21apt-get21:03:142
2129299813,39cyclictest1270-40mandb23:48:062
2129299813,38cyclictest8770-21runrttasks21:18:242
2129299813,38cyclictest8770-21runrttasks21:18:242
2129299813,31cyclictest28849-21/usr/sbin/munin19:18:302
2129299813,31cyclictest28849-21/usr/sbin/munin19:18:302
2129299813,31cyclictest22809-21idleruntime-cro19:03:022
2129299813,30cyclictest759-21/usr/sbin/munin21:58:302
2128399818,29cyclictest1730-21modprobe23:48:071
2128399818,29cyclictest1730-21modprobe23:48:071
2128399813,29cyclictest17999-21fschecks_time21:33:241
2128399813,28cyclictest27836-21latency_hist23:38:061
2128399813,27cyclictest2549-21grep19:38:231
2128399813,25cyclictest27452-21fschecks_time22:43:241
21283998130,43cyclictest0-21swapper/123:43:371
21278998160,13cyclictest31981-21entropy21:03:240
21278998148,22cyclictest25149-21apt-get19:08:140
21278998136,38cyclictest31156-21ssh23:43:180
21278998132,43cyclictest0-21swapper/000:03:110
21278998125,23cyclictest0-21swapper/020:59:080
21278998125,21cyclictest0-21swapper/022:18:290
2129299804,7cyclictest0-21swapper/219:33:162
2129299803,37cyclictest2374-21systemd-journal00:23:012
2129299803,33cyclictest4570-21sendmail_mailqu19:43:342
2129299803,30cyclictest3239-21ls21:09:232
2129299803,30cyclictest3239-21ls21:09:232
2129299803,29cyclictest24693-21sh00:26:092
2129299803,28cyclictest7803-21/usr/sbin/munin19:53:282
2129299803,28cyclictest7803-21/usr/sbin/munin19:53:282
2129299802,6cyclictest0-21swapper/220:48:402
2129299802,6cyclictest0-21swapper/220:48:402
2129299802,31cyclictest9173-21/usr/sbin/munin22:13:132
2129299802,31cyclictest9173-21/usr/sbin/munin22:13:132
2128399803,32cyclictest28673-21ssh20:58:051
2128399803,27cyclictest12576-21fschecks_time20:08:271
21283998014,21cyclictest26677-21basename21:48:131
21283998014,21cyclictest26677-21basename21:48:131
21278998059,12cyclictest23509-21apt-get19:03:160
21278998030,19cyclictest18991-21df_inode23:23:200
2129299794,34cyclictest17285-21ssh22:26:322
2129299794,34cyclictest17285-21ssh22:26:322
2129299793,6cyclictest0-21swapper/219:03:162
2129299793,30cyclictest6284-21cron23:03:012
2129299793,30cyclictest6284-21cron23:03:012
2128399793,30cyclictest29854-21latency_hist19:23:041
2128399793,30cyclictest29854-21latency_hist19:23:041
21278997963,9cyclictest20694-21apt-get21:38:150
21278997963,9cyclictest20694-21apt-get21:38:150
21278997961,10cyclictest7446-21apt-get19:53:150
21278997961,10cyclictest7446-21apt-get19:53:150
21278997950,19cyclictest17114-21apt-get20:23:130
21278997950,19cyclictest17114-21apt-get20:23:130
2129299782,28cyclictest23563-21latency_hist20:43:052
21278997862,10cyclictest18174-21apt-get22:28:130
21278997845,23cyclictest28360-21apt-get19:18:180
21278997845,23cyclictest28360-21apt-get19:18:180
21278997843,30cyclictest0-21swapper/020:35:430
21278997843,30cyclictest0-21swapper/020:35:430
21278997832,39cyclictest0-21swapper/000:18:190
2129299773,29cyclictest18036-21grep20:23:372
2129299773,29cyclictest18036-21grep20:23:372
21278997746,23cyclictest0-21swapper/020:18:160
21278997740,27cyclictest2630-21fschecks_time19:38:240
2129299762,30cyclictest25344-21/usr/sbin/munin19:08:252
2129999754,33cyclictest12799-21taskset23:13:213
2129999753,36cyclictest27484-21chrt23:38:023
2129999744,29cyclictest17866-21chrt21:33:193
2129999743,31cyclictest24839-21chrt00:26:343
2129999743,30cyclictest17118-21taskset20:23:133
2129999743,30cyclictest17118-21taskset20:23:133
21278997435,29cyclictest352-21runrttasks19:43:210
2129999734,28cyclictest23362-21chrt20:43:013
2129999733,30cyclictest4691-21taskset21:11:343
2129999733,30cyclictest4691-21taskset21:11:343
2129999733,29cyclictest3731-21chrt22:58:183
2129999733,29cyclictest3731-21chrt22:58:183
2129999723,29cyclictest9241-21chrt22:13:193
2129999723,29cyclictest9241-21chrt22:13:193
2129999722,34cyclictest11254-21chrt20:03:323
2129999713,30cyclictest26581-21chrt19:13:053
2129999703,30cyclictest10380-21chrt21:20:513
2129999703,30cyclictest10380-21chrt21:20:513
2129999703,29cyclictest1332-21chrt23:48:033
21278997038,20cyclictest21568-21/usr/sbin/munin18:58:200
2129999694,29cyclictest29859-21chrt21:53:213
2129999693,29cyclictest28453-21chrt19:18:193
2129999693,29cyclictest28453-21chrt19:18:193
4732660,10sleep3451rcuc/319:33:023
21299996510,21cyclictest0-21swapper/321:49:093
21299996510,21cyclictest0-21swapper/321:49:093
2129999643,47cyclictest31154-21taskset21:02:143
2129999624,50cyclictest0-21swapper/321:24:153
2129999624,50cyclictest0-21swapper/321:24:153
2129999623,44cyclictest3344-21taskset22:58:013
21299996113,19cyclictest0-21swapper/323:48:313
21299996113,19cyclictest0-21swapper/323:48:313
2129999603,43cyclictest7348-21taskset21:15:403
2129999603,41cyclictest8406-21taskset19:53:403
2129999603,41cyclictest8406-21taskset19:53:403
2129999594,40cyclictest8816-21chrt23:58:413
2129999594,40cyclictest8816-21chrt23:58:413
2129999594,39cyclictest28827-21chrt23:39:243
2129999583,43cyclictest2008-21chrt21:08:023
2129999574,37cyclictest27737-21chrt20:53:353
2129999573,41cyclictest25189-21chrt22:38:403
2129999573,41cyclictest25189-21chrt22:38:403
2129999573,38cyclictest10782-21taskset23:09:003
2129999563,38cyclictest3604-21chrt22:03:263
2129999554,34cyclictest18824-21chrt20:28:193
2129999554,34cyclictest18824-21chrt20:28:193
2129999553,39cyclictest21771-21taskset22:33:303
2129999553,38cyclictest6018-21chrt19:49:103
2129999553,38cyclictest6018-21chrt19:49:103
2129999553,35cyclictest22494-21chrt18:59:073
2129999544,37cyclictest510-21taskset21:58:243
2129999544,34cyclictest20120-21chrt00:18:273
2129999543,39cyclictest26381-21taskset20:49:013
2129999543,39cyclictest26381-21taskset20:49:013
2129999543,38cyclictest13053-21chrt20:08:393
2129999543,36cyclictest14092-21chrt20:13:243
2129999542,38cyclictest825-21chrt19:33:183
2129999534,37cyclictest7646-21chrt23:58:053
2129999534,36cyclictest12854-21chrt22:18:343
2129999533,37cyclictest9074-21taskset19:58:183
2129999533,35cyclictest27562-21chrt22:43:273
2129999533,33cyclictest21082-21chrt21:38:263
2129999533,33cyclictest21082-21chrt21:38:263
2129999533,32cyclictest2632-21chrt19:38:243
2129999532,36cyclictest25948-21chrtset21:46:573
2129999523,36cyclictest29962-21chrt19:23:133
2129999523,36cyclictest29962-21chrt19:23:133
2129999523,33cyclictest18014-21chrt00:14:103
2129999522,38cyclictest22489-21chrt23:28:333
2129999513,36cyclictest20125-21chrt23:24:163
2129999513,33cyclictest16856-21chrt21:31:333
2129999512,34cyclictest16551-21chrt00:13:043
2129999512,34cyclictest11995-21chrt00:04:043
2129999504,33cyclictest24416-21chrt20:43:383
2129999504,33cyclictest24416-21chrt20:43:383
2129999503,33cyclictest5462-21chrt19:48:043
2129999494,33cyclictest19262-21chrt22:28:393
2129999494,32cyclictest30778-21chrt22:48:333
2129999494,32cyclictest30778-21chrt22:48:333
2129999494,31cyclictest7718-21taskset23:03:413
2129999494,31cyclictest7718-21taskset23:03:413
2129999494,31cyclictest7718-21taskset23:03:413
2129999493,33cyclictest20977-21taskset20:33:343
2129999493,33cyclictest20977-21taskset20:33:343
2129999493,32cyclictest15844-21chrt23:18:163
2129999484,33cyclictest8065-21taskset22:10:483
2129999483,31cyclictest24900-21taskset19:08:023
2129999483,31cyclictest16659-21taskset22:24:463
2129999483,31cyclictest16659-21taskset22:24:463
2129999472,30cyclictest15891-21chrt20:18:273
2129999443,30cyclictest27921-21taskset19:16:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional