You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-05 - 05:40
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue May 05, 2026 00:44:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1251721589,121sleep112567-21sort18:57:351
1251721589,121sleep112567-21sort18:57:351
132889911489,19cyclictest13290-21latency_hist19:01:530
132889911489,19cyclictest13290-21latency_hist19:01:530
2414021100,5sleep224133-21ssh21:07:192
2414021100,5sleep224133-21ssh21:07:192
13302999674,15cyclictest13279-21cyclictest20:07:222
1329399944,39cyclictest8827-21latency_hist20:26:531
1330299933,42cyclictest39-21ksoftirqd/222:42:452
13302999246,12cyclictest181rcu_preempt19:52:202
13293999228,57cyclictest1794-21ssh23:12:281
1329399922,80cyclictest12281-21apt-get21:42:081
1329399922,80cyclictest12281-21apt-get21:42:081
13302999137,15cyclictest181rcu_preempt22:59:152
1329399912,45cyclictest10246-21apt-get20:32:031
1329399912,45cyclictest10246-21apt-get20:32:031
1329399912,28cyclictest22837-21ssh22:52:281
1329399912,28cyclictest22837-21ssh22:52:281
13288999172,10cyclictest0-21swapper/000:06:490
13302999066,17cyclictest28952-21sh23:02:522
13302999066,17cyclictest28952-21sh23:02:522
13288999035,9cyclictest181rcu_preempt23:16:430
1328899901,57cyclictest15-21ksoftirqd/000:22:150
13302998956,25cyclictest25713-21kworker/u19:0+nfsiod22:16:522
13302998956,25cyclictest25713-21kworker/u19:0+nfsiod22:16:522
13302998937,17cyclictest181rcu_preempt22:48:112
13302998914,46cyclictest39-21ksoftirqd/221:30:522
13302998914,46cyclictest39-21ksoftirqd/221:30:522
1329399895,39cyclictest20350-21munin-node19:22:001
1329399893,40cyclictest12977-21fschecks_count22:37:101
13288998969,11cyclictest0-21swapper/020:02:250
13288998968,15cyclictest13279-21cyclictest23:11:530
1329399889,14cyclictest0-21swapper/119:07:241
13293998842,39cyclictest19321-21latency_hist23:41:511
13293998839,43cyclictest20236-21diskmemload00:22:521
1329399883,42cyclictest16833-21apt-key00:31:521
1329399883,42cyclictest16833-21apt-key00:31:521
1329399883,36cyclictest352-21runrttasks22:08:451
1329399882,29cyclictest17207-21/usr/sbin/munin20:52:061
119162887,54sleep211969-21cstates18:57:112
119162887,54sleep211969-21cstates18:57:112
13302998767,13cyclictest13279-21cyclictest20:17:102
13293998742,37cyclictest0-21swapper/121:27:171
13293998742,37cyclictest0-21swapper/121:27:171
1329399873,27cyclictest553-21latency_hist20:01:521
13302998645,10cyclictest58-21kcompactd000:07:052
13302998637,12cyclictest181rcu_preempt00:23:442
13302998631,8cyclictest181rcu_preempt20:06:532
13293998663,17cyclictest30417-21threads21:17:251
13293998663,17cyclictest30417-21threads21:17:251
13293998658,20cyclictest20236-21diskmemload00:19:321
13293998658,20cyclictest20236-21diskmemload00:19:321
13293998636,43cyclictest708-21ssh22:16:531
1329399863,40cyclictest10627-21apt-key20:31:531
1329399863,39cyclictest28215-21sh23:56:511
1329399863,27cyclictest23663-21cstates21:07:071
1329399863,27cyclictest23663-21cstates21:07:071
1329399862,28cyclictest7763-21grep22:27:251
13293998617,30cyclictest19173-21grep22:47:151
1328899862,52cyclictest15-21ksoftirqd/022:19:030
13302998536,13cyclictest181rcu_preempt22:23:112
13302998536,13cyclictest181rcu_preempt22:23:112
13302998515,45cyclictest39-21ksoftirqd/219:22:352
13293998559,18cyclictest8733-21sh23:22:581
1329399854,34cyclictest6357-21mailstats20:17:221
13293998539,37cyclictest0-21swapper/122:02:061
13293998513,32cyclictest2795-21sh21:26:501
1328899856,24cyclictest12281-21apt-get21:42:130
1328899856,24cyclictest12281-21apt-get21:42:130
13302998441,12cyclictest19676-21snmpd23:13:532
13293998456,20cyclictest31539-21switchtime23:07:241
1329399843,38cyclictest18834-21awk21:52:181
1329399843,34cyclictest16014-21cat20:47:151
1329399843,34cyclictest16014-21cat20:47:151
1329399843,33cyclictest28156-21latency_hist19:46:521
1329399843,32cyclictest352-21runrttasks19:12:521
1329399843,26cyclictest31684-21fschecks_count00:02:101
1329399843,24cyclictest10023-21fschecks_time22:32:091
1329399843,24cyclictest10023-21fschecks_time22:32:091
1329399843,23cyclictest12050-21cat21:41:531
1329399842,28cyclictest26709-21rm21:12:071
13293998415,27cyclictest8732-21sh21:36:481
1328899842,48cyclictest15-21ksoftirqd/023:30:030
13302998334,16cyclictest181rcu_preempt21:14:432
13302998334,12cyclictest181rcu_preempt00:27:432
13302998334,12cyclictest181rcu_preempt00:27:432
13293998357,17cyclictest20682-21sh21:02:071
13293998355,21cyclictest25389-21fschecks_count19:37:101
13293998355,21cyclictest25389-21fschecks_count19:37:101
1329399834,37cyclictest352-21runrttasks21:50:281
1329399834,37cyclictest352-21runrttasks21:50:281
1329399833,40cyclictest3880-21cstates22:22:051
1329399833,40cyclictest3880-21cstates22:22:051
1329399833,38cyclictest15744-21apt-get22:42:061
1329399833,36cyclictest2063-21cstates00:07:071
13293998316,30cyclictest21734-21diskmemload23:45:291
13293998316,30cyclictest21734-21diskmemload23:45:291
13302998252,20cyclictest22797-21fschecks_time23:47:102
13302998252,20cyclictest22797-21fschecks_time23:47:102
13302998238,18cyclictest181rcu_preempt23:54:082
13302998229,12cyclictest181rcu_preempt00:00:462
13293998257,19cyclictest22772-21fschecks_count23:47:101
13293998257,19cyclictest22772-21fschecks_count23:47:101
1329399824,37cyclictest13966-21/usr/sbin/munin20:42:151
1329399824,35cyclictest352-21runrttasks22:14:211
1329399823,36cyclictest27675-21/usr/sbin/munin23:02:261
1329399823,36cyclictest27675-21/usr/sbin/munin23:02:261
1329399823,36cyclictest25035-21grep22:57:141
1329399823,25cyclictest13439-21ssh23:32:001
1329399823,24cyclictest24943-21cron19:36:491
1329399823,24cyclictest24943-21cron19:36:491
1329399823,24cyclictest23958-21modprobe22:01:491
1329399822,38cyclictest4569-21sed20:12:171
1329399822,31cyclictest12502-21cstates20:37:071
1329399822,26cyclictest1330-21netstat22:17:181
1328899829,41cyclictest15-21ksoftirqd/000:19:240
1328899829,41cyclictest15-21ksoftirqd/000:19:240
13288998262,12cyclictest5273-21ssh23:17:260
1328899822,43cyclictest15-21ksoftirqd/019:27:000
13302998159,14cyclictest29904-21sh22:11:482
13302998156,13cyclictest0-21swapper/223:42:232
13302998156,13cyclictest0-21swapper/223:42:232
13302998154,19cyclictest2781-21sh21:26:492
13302998154,19cyclictest10976-21runrttasks00:21:532
13302998154,18cyclictest23225-21latency_hist21:06:532
13302998153,21cyclictest29918-21latency_hist19:51:522
13302998145,15cyclictest181rcu_preempt21:32:172
13302998137,15cyclictest181rcu_preempt23:22:442
13302998137,12cyclictest181rcu_preempt23:38:132
13302998136,9cyclictest181rcu_preempt21:51:532
13302998136,9cyclictest181rcu_preempt21:51:532
1329399815,30cyclictest352-21runrttasks19:03:351
1329399815,30cyclictest352-21runrttasks19:03:351
1329399813,34cyclictest19276-21latency20:57:171
1329399812,38cyclictest2364-21cat20:06:531
13288998165,9cyclictest16683-21ssh23:37:050
13288998161,12cyclictest17949-21sh21:51:490
13288998161,12cyclictest17949-21sh21:51:490
1328899814,45cyclictest15-21ksoftirqd/000:15:490
1328899814,45cyclictest15-21ksoftirqd/000:15:490
1328899812,47cyclictest15-21ksoftirqd/023:43:540
1328899812,47cyclictest15-21ksoftirqd/023:43:540
13302998035,13cyclictest181rcu_preempt21:46:282
13302998035,13cyclictest181rcu_preempt21:46:282
13302998029,24cyclictest0-21swapper/219:37:062
13302998029,24cyclictest0-21swapper/219:37:062
1329399804,33cyclictest28672-21tune2fs19:47:111
1329399803,35cyclictest5566-21sh00:12:191
1329399803,35cyclictest5566-21sh00:12:191
1329399803,27cyclictest19490-21grep19:17:241
1329399803,24cyclictest4824-21ls23:17:141
1329399802,28cyclictest3908-21taskset20:11:511
1329399802,24cyclictest30378-21/usr/sbin/munin19:52:141
1329399802,22cyclictest28589-21/usr/sbin/munin23:57:101
13288998059,15cyclictest21017-21latency_hist21:56:510
13288998047,26cyclictest13279-21cyclictest23:04:530
13288998047,26cyclictest13279-21cyclictest23:04:530
1328899802,42cyclictest15-21ksoftirqd/023:22:380
1328899802,41cyclictest15-21ksoftirqd/022:44:350
13302997955,18cyclictest1696-21latency_hist00:06:512
13302997954,14cyclictest0-21swapper/219:06:512
13302997954,14cyclictest0-21swapper/219:06:512
1329399793,36cyclictest11517-21sh23:27:271
1329399793,35cyclictest23480-21latency_hist19:31:531
1330299785,41cyclictest39-21ksoftirqd/222:03:412
13302997844,8cyclictest181rcu_preempt20:16:532
1328899783,44cyclictest15-21ksoftirqd/021:03:380
1328899782,43cyclictest15-21ksoftirqd/021:08:010
1328899782,43cyclictest15-21ksoftirqd/021:08:010
13302997759,12cyclictest20284-21latency_hist21:01:512
13302997753,18cyclictest2374-21systemd-journal21:41:522
13302997745,13cyclictest181rcu_preempt21:17:002
13302997745,13cyclictest181rcu_preempt21:17:002
13302997741,11cyclictest13279-21cyclictest20:49:412
13302997741,11cyclictest13279-21cyclictest20:49:412
13302997739,20cyclictest170-21jbd2/mmcblk2p2-820:52:062
13302997728,8cyclictest181rcu_preempt22:19:092
13288997753,15cyclictest20236-21diskmemload22:06:520
13288997715,30cyclictest15-21ksoftirqd/021:37:120
13302997655,10cyclictest0-21swapper/220:36:512
13302997655,10cyclictest0-21swapper/220:36:512
13302997648,21cyclictest30981-21sh23:07:112
13302997632,12cyclictest181rcu_preempt22:32:092
13302997632,12cyclictest181rcu_preempt22:32:092
13302997631,13cyclictest181rcu_preempt20:43:222
13302997630,11cyclictest181rcu_preempt22:27:452
13302997627,11cyclictest0-21swapper/200:12:232
13302997627,11cyclictest0-21swapper/200:12:232
13288997652,18cyclictest20236-21diskmemload22:12:280
13288997652,18cyclictest20236-21diskmemload22:12:280
13288997650,18cyclictest20236-21diskmemload21:57:220
13288997646,22cyclictest23430-21sort23:47:240
13288997646,22cyclictest23430-21sort23:47:240
13288997623,38cyclictest21860-21cstates22:52:080
13288997623,38cyclictest21860-21cstates22:52:080
13302997552,12cyclictest0-21swapper/222:01:522
13302997532,8cyclictest181rcu_preempt19:32:122
13302997532,8cyclictest181rcu_preempt19:32:122
13302997532,12cyclictest181rcu_preempt19:28:122
13302997530,7cyclictest181rcu_preempt23:18:142
13302997528,12cyclictest181rcu_preempt23:27:222
1328899758,28cyclictest15-21ksoftirqd/020:15:430
13288997552,14cyclictest14708-21proc_pri00:27:210
13288997552,14cyclictest14708-21proc_pri00:27:210
13288997542,11cyclictest19297-21ssh22:47:180
13288997521,30cyclictest15-21ksoftirqd/022:27:040
13288997516,31cyclictest15-21ksoftirqd/019:21:540
13302997430,16cyclictest181rcu_preempt00:17:222
13302997430,12cyclictest181rcu_preempt19:12:182
1328899742,33cyclictest15-21ksoftirqd/000:11:050
13302997344,8cyclictest181rcu_preempt19:07:282
13302997339,13cyclictest181rcu_preempt21:53:192
13302997336,9cyclictest22495-21ssh22:52:242
13302997336,9cyclictest22495-21ssh22:52:242
13302997313,22cyclictest0-21swapper/220:25:552
13288997359,9cyclictest6452-21modprobe22:26:490
13288997359,9cyclictest6452-21modprobe22:26:490
13288997355,12cyclictest20236-21diskmemload21:24:530
1328899734,44cyclictest15-21ksoftirqd/022:58:290
1328899733,44cyclictest15-21ksoftirqd/022:07:470
1328899732,40cyclictest15-21ksoftirqd/022:35:330
1328899732,40cyclictest15-21ksoftirqd/022:35:330
1328899732,40cyclictest15-21ksoftirqd/019:21:330
13288997310,34cyclictest15-21ksoftirqd/019:08:370
1330999722,29cyclictest24431-21chrt22:02:073
13302997248,8cyclictest181rcu_preempt19:57:232
13302997238,11cyclictest181rcu_preempt20:40:102
1328899725,40cyclictest15-21ksoftirqd/021:32:350
13288997238,8cyclictest181rcu_preempt19:12:150
1328899723,43cyclictest15-21ksoftirqd/022:38:480
1328899722,35cyclictest15-21ksoftirqd/021:11:560
1330999713,26cyclictest9541-21taskset20:27:223
13302997150,8cyclictest181rcu_preempt20:30:142
13302997149,12cyclictest0-21swapper/219:21:522
13302997125,17cyclictest0-21swapper/219:45:122
13288997136,7cyclictest0-21swapper/019:32:230
13288997136,7cyclictest0-21swapper/019:32:230
1328899713,42cyclictest15-21ksoftirqd/020:22:220
13288997112,33cyclictest15-21ksoftirqd/020:42:140
13288997112,33cyclictest15-21ksoftirqd/019:42:560
1330999702,29cyclictest1786-21chrt22:17:273
13302997047,8cyclictest181rcu_preempt22:41:282
13302997035,12cyclictest181rcu_preempt23:33:562
1328899705,36cyclictest15-21ksoftirqd/020:58:200
1328899703,44cyclictest15-21ksoftirqd/020:07:590
13288997010,34cyclictest15-21ksoftirqd/023:36:090
1330999693,29cyclictest9872-21chrt00:19:593
1330999693,29cyclictest9872-21chrt00:19:593
1330999692,31cyclictest25654-21chrt23:52:063
1328899694,31cyclictest352-21runrttasks20:32:040
1328899694,31cyclictest352-21runrttasks20:32:040
1328899693,40cyclictest15-21ksoftirqd/021:21:480
1328899693,40cyclictest15-21ksoftirqd/021:21:480
13288996932,8cyclictest181rcu_preempt23:57:260
1328899692,35cyclictest3569-21ssh21:27:150
1328899692,35cyclictest3569-21ssh21:27:150
1330999686,23cyclictest1537-21chrt23:12:083
1330999683,31cyclictest23264-21chrt21:06:513
1330999683,25cyclictest26504-21taskset22:59:233
1330999682,29cyclictest10814-21taskset23:27:133
13288996848,14cyclictest26107-21sh23:52:180
1330999674,27cyclictest14339-21chrt21:45:083
1330999674,27cyclictest14339-21chrt21:45:083
1330999673,27cyclictest1096-21taskset23:11:513
1330999673,26cyclictest29809-21chrt22:11:353
1330999672,27cyclictest5058-21chrt22:22:583
1330999672,27cyclictest5058-21chrt22:22:583
1330999672,25cyclictest15563-21taskset22:41:513
1328899677,28cyclictest15-21ksoftirqd/020:37:120
1330999668,25cyclictest25753-21chrt19:37:203
1330999668,25cyclictest25753-21chrt19:37:203
1330999664,27cyclictest27968-21chrt23:02:113
1330999664,27cyclictest27968-21chrt23:02:113
1330999663,26cyclictest22925-21chrt22:52:423
1330999663,26cyclictest22925-21chrt22:52:423
1330999663,24cyclictest1434-21chrt21:23:093
1330999663,23cyclictest16675-21chrt21:48:243
1330999663,23cyclictest16675-21chrt21:48:243
1330999662,26cyclictest4463-21taskset21:28:153
1330999662,26cyclictest4463-21taskset21:28:153
1330999662,26cyclictest19149-21chrt23:41:323
1328899665,35cyclictest15-21ksoftirqd/019:06:010
1328899665,35cyclictest15-21ksoftirqd/019:06:010
1328899663,38cyclictest15-21ksoftirqd/020:00:530
1328899662,39cyclictest15-21ksoftirqd/020:52:010
1328899662,35cyclictest15-21ksoftirqd/020:17:300
1330999653,25cyclictest5710-21taskset20:17:053
1330999653,25cyclictest25624-21chrt21:10:113
1330999653,25cyclictest25624-21chrt21:10:113
1330999653,25cyclictest23183-21chrt21:59:433
1330999653,21cyclictest2289-21chrt20:06:523
1328899653,44cyclictest352-21runrttasks19:47:030
1328899652,24cyclictest15-21ksoftirqd/019:39:220
1328899652,24cyclictest15-21ksoftirqd/019:39:220
1330999644,29cyclictest32622-21chrt22:16:013
1330999644,29cyclictest32622-21chrt22:16:013
1328899644,32cyclictest15575-21/usr/sbin/munin20:47:130
1328899644,32cyclictest15575-21/usr/sbin/munin20:47:130
1328899643,39cyclictest31592-21latency_hist19:56:530
1330999633,24cyclictest3766-21taskset00:09:293
1330999632,26cyclictest11387-21taskset00:22:313
1328899632,36cyclictest15-21ksoftirqd/020:27:020
1295326313,24sleep30-21swapper/318:59:203
1295326313,24sleep30-21swapper/318:59:203
1330999629,43cyclictest0-21swapper/321:21:493
1330999629,43cyclictest0-21swapper/321:21:493
1330999624,25cyclictest6421-21chrt23:19:513
1330999614,25cyclictest23906-21chrt23:48:133
1330999614,25cyclictest23906-21chrt23:48:133
1330999613,19cyclictest8156-21chrt23:22:223
1330999612,23cyclictest21664-21chrt23:45:173
1330999612,23cyclictest21664-21chrt23:45:173
1330999602,27cyclictest8606-21taskset22:29:353
1330999602,26cyclictest7371-21taskset21:32:583
1330999602,24cyclictest26823-21taskset19:42:023
1330999602,23cyclictest19430-21chrt20:57:233
1330999602,19cyclictest15328-21chrt23:34:393
1330999599,12cyclictest0-21swapper/321:52:273
1330999593,23cyclictest20705-21chrt19:22:133
1330999593,22cyclictest19293-21chrt19:17:193
1330999593,20cyclictest29558-21chrt19:49:233
1330999592,25cyclictest15422-21taskset00:28:153
1330999592,25cyclictest15422-21taskset00:28:153
1330999592,24cyclictest7565-21chrt20:22:113
1330999592,24cyclictest19872-21chrt22:47:353
1330999592,23cyclictest16886-21chrt20:51:233
1330999592,23cyclictest16886-21chrt20:51:233
1330999592,22cyclictest31651-21taskset19:57:003
1330999592,22cyclictest17135-21chrt22:43:173
1330999592,21cyclictest11832-21chrt20:33:293
1330999592,21cyclictest11832-21chrt20:33:293
1330999592,20cyclictest11757-21taskset22:34:583
1330999592,20cyclictest11757-21taskset22:34:583
1330999592,19cyclictest28098-21chrt21:13:403
1330999592,19cyclictest2432-21chrt20:07:003
1330999592,19cyclictest14609-21chrt19:04:193
1330999592,19cyclictest14609-21chrt19:04:193
1330999592,19cyclictest10229-21chrt21:37:363
1330999583,20cyclictest17164-21chrt19:12:053
1330999582,22cyclictest29391-21chrt23:57:293
1330999582,22cyclictest21978-21chrt19:27:033
1330999582,20cyclictest6479-21chrt00:13:433
1330999582,20cyclictest6479-21chrt00:13:433
1330999582,20cyclictest31456-21chrt00:02:033
1330999582,19cyclictest4052-21chrt20:11:593
1330999582,19cyclictest24766-21chrt19:34:413
1330999582,19cyclictest24766-21chrt19:34:413
1330999582,19cyclictest18353-21chrt20:53:303
1330999579,10cyclictest0-21swapper/319:52:203
1330999574,13cyclictest0-21swapper/320:37:113
1330999572,20cyclictest15053-21chrt20:43:303
1330999572,19cyclictest16063-21chrt19:07:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional