You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 06:09
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Dec 30, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13953991356,44cyclictest13954-21latency_hist19:09:330
13953991073,32cyclictest9294-21latency_hist21:44:320
13953991053,28cyclictest16268-21grep21:55:070
13953991053,26cyclictest24260-21cat21:14:320
13953991052,28cyclictest21547-21ls23:00:070
139539910439,57cyclictest0-21swapper/023:20:060
13953991043,32cyclictest9037-21/usr/sbin/munin23:34:580
13953991042,28cyclictest3216-21fschecks_count00:19:510
139539910342,52cyclictest17944-21sh23:49:460
13953991032,30cyclictest346-21runrttasks21:14:580
13953991032,30cyclictest346-21runrttasks21:14:580
13953991032,21cyclictest27272-21apt-get21:19:430
13953991013,29cyclictest22208-21unixbench_singl22:05:080
13953991013,28cyclictest346-21runrttasks22:02:010
13953991013,28cyclictest346-21runrttasks22:02:010
13953991007,23cyclictest14505-21latency_hist00:39:320
13953991007,23cyclictest14505-21latency_hist00:39:320
13953991002,23cyclictest6125-21sh23:29:460
13953991002,23cyclictest6125-21sh23:29:460
13953991002,20cyclictest31311-21chrt00:12:040
13953991002,20cyclictest31311-21chrt00:12:040
13953999943,49cyclictest9833-21fschecks_time20:34:530
1395399993,29cyclictest346-21runrttasks22:11:340
1395399993,22cyclictest15106-21unixbench_singl19:10:070
1395399992,39cyclictest1044-21cat22:25:350
1395399992,31cyclictest27697-21head00:05:070
1395399992,29cyclictest346-21runrttasks22:19:250
1395399992,21cyclictest10225-21ssh00:31:120
13953999915,27cyclictest29754-21/usr/sbin/munin23:15:040
13953999915,27cyclictest29754-21/usr/sbin/munin23:15:040
1395399983,34cyclictest7330-21head20:25:070
1395399983,31cyclictest16022-21/usr/sbin/munin23:45:060
1395399983,31cyclictest14023-21sh23:42:480
1395399983,31cyclictest14023-21sh23:42:480
1395399983,30cyclictest3273-21ssh22:29:430
1395399983,29cyclictest24228-21latency_hist21:14:320
1395399982,32cyclictest346-21runrttasks23:58:120
1395399982,22cyclictest9716-21fschecks_time21:44:500
1395399982,21cyclictest19728-21grep21:04:530
1395399982,11cyclictest258502ssh23:07:300
1395399982,11cyclictest258502ssh23:07:300
13953999813,28cyclictest3523-21basename23:24:500
13953999734,56cyclictest0-21swapper/020:49:490
1395399973,35cyclictest6438-21fschecks_time22:34:520
1395399973,30cyclictest346-21runrttasks22:44:120
1395399972,32cyclictest19408-21/usr/sbin/munin19:24:580
1395399972,32cyclictest19408-21/usr/sbin/munin19:24:580
1395399972,28cyclictest30791-21timerandwakeup22:20:070
1395399972,28cyclictest30791-21timerandwakeup22:20:070
1395399964,31cyclictest27562-21processes23:10:020
1395399963,31cyclictest4097-21needreboot21:34:580
1395399963,29cyclictest30707-21/usr/sbin/munin19:59:520
1395399962,31cyclictest13673-21timerwakeupswit20:45:060
1395399962,31cyclictest13673-21timerwakeupswit20:45:060
1395399962,30cyclictest18458-21ntp_states21:00:030
1396799953,75cyclictest20872-21apt-get21:09:442
1396799953,75cyclictest20872-21apt-get21:09:442
1395399954,30cyclictest27464-21/usr/sbin/munin19:49:480
1395399954,30cyclictest27464-21/usr/sbin/munin19:49:480
1395399953,33cyclictest12883-21timerandwakeup22:45:040
1395399953,31cyclictest29073-21/usr/sbin/munin19:55:060
1395399952,35cyclictest25884-21cpu19:44:450
1395399952,35cyclictest25884-21cpu19:44:450
1395399952,30cyclictest21197-21/usr/sbin/munin19:29:510
1395399952,30cyclictest21197-21/usr/sbin/munin19:29:510
13953999522,26cyclictest18038-21/usr/sbin/munin22:54:500
1395399943,34cyclictest2226-21ssh21:31:310
1395399943,33cyclictest16350-21fschecks_count19:14:520
1395399943,33cyclictest16350-21fschecks_count19:14:520
1395399942,36cyclictest4813-21cpuspeed20:19:460
1395399942,35cyclictest30237-21cpu21:24:480
1395399934,32cyclictest24334-21ls19:39:450
1395399934,32cyclictest24334-21ls19:39:450
1395399934,31cyclictest23256-21sendmail_mailqu19:35:030
1395399934,31cyclictest23256-21sendmail_mailqu19:35:030
1395399933,33cyclictest16664-21grep20:54:550
1395399933,28cyclictest8461-21modprobe00:29:280
1395399933,26cyclictest23928-21fschecks_count23:59:510
1395399932,29cyclictest11415-21fschecks_count20:39:520
1395399932,27cyclictest18487-21mailstats19:20:040
1395399932,27cyclictest18487-21mailstats19:20:040
1395399932,26cyclictest17678-21latency_hist22:54:320
1395399932,26cyclictest17678-21latency_hist22:54:320
1395399923,29cyclictest8065-21cstates20:29:450
1395399923,29cyclictest2726-21cat00:19:310
1395399922,29cyclictest2835-21modprobe20:14:290
13958999146,12cyclictest181rcu_preempt20:09:431
1395399912,31cyclictest3209-21cstates20:14:480
1395399912,25cyclictest12930-21ntpq21:49:570
1395399912,25cyclictest12930-21ntpq21:49:570
13967999065,16cyclictest18595-21if_eth021:59:532
13967999065,16cyclictest18595-21if_eth021:59:532
1395399903,29cyclictest32323-21/usr/sbin/munin20:04:520
13967998912,65cyclictest14665-21apt-get20:49:442
13958998853,11cyclictest31-21ksoftirqd/120:24:511
1256628815,40sleep10-21swapper/119:04:511
13958998769,11cyclictest13943-21cyclictest22:39:511
13967998657,20cyclictest20883-21diskmemload22:09:582
13967998463,14cyclictest26685-21timerandwakeup19:45:032
13967998463,14cyclictest26685-21timerandwakeup19:45:032
13967998355,20cyclictest1056-21wc22:25:352
13958998348,12cyclictest181rcu_preempt00:04:431
13967998263,15cyclictest18040-21kworker/u17:1-xprtiod00:19:292
13967998160,8cyclictest181rcu_preempt19:53:202
13967998160,8cyclictest181rcu_preempt19:53:202
13958998162,13cyclictest28827-21idleruntime-cro19:54:291
13958998162,13cyclictest28827-21idleruntime-cro19:54:291
13958998160,7cyclictest181rcu_preempt23:00:071
13958998160,14cyclictest13943-21cyclictest20:00:051
13958998150,22cyclictest18892-21sh23:50:071
1397199803,48cyclictest0-21swapper/321:09:573
1397199803,48cyclictest0-21swapper/321:09:573
13967998064,9cyclictest5055-21kworker/u17:3+nfsiod22:44:512
1396799803,24cyclictest39-21ksoftirqd/200:03:252
13958998055,16cyclictest14282-21apt-key20:49:311
13958998055,16cyclictest14282-21apt-key20:49:311
13967997943,27cyclictest15477-21sh19:14:292
1396799793,35cyclictest8481-21awk20:29:562
13958997957,12cyclictest6451-21fschecks_time22:34:521
1396799787,58cyclictest21021-21apt-get22:59:432
13967997850,19cyclictest1-21systemd21:59:092
13958997857,15cyclictest3678-21kernelversion22:29:551
13967997760,11cyclictest1678-21ssh23:21:012
13967997658,12cyclictest20883-21diskmemload22:36:432
13967997658,11cyclictest9115-21cpu23:34:462
1396799762,38cyclictest39-21ksoftirqd/222:40:442
13958997636,11cyclictest181rcu_preempt20:29:401
13958997632,32cyclictest11470-21cut20:39:521
1396799758,27cyclictest39-21ksoftirqd/219:54:582
13967997551,16cyclictest346-21runrttasks23:10:072
1396799752,36cyclictest39-21ksoftirqd/223:34:072
1396799752,36cyclictest39-21ksoftirqd/223:34:072
1396799752,30cyclictest39-21ksoftirqd/221:49:572
1396799752,30cyclictest39-21ksoftirqd/221:49:572
13958997558,10cyclictest23973-21kworker/u18:119:39:281
13958997558,10cyclictest23973-21kworker/u18:119:39:281
13958997551,14cyclictest15343-21ssh22:49:551
13958997551,14cyclictest15343-21ssh22:49:551
13958997538,14cyclictest16896-21find20:55:021
13967997454,14cyclictest30620-21kernelversion21:24:572
1396799743,34cyclictest39-21ksoftirqd/222:55:482
13958997445,21cyclictest31857-21kworker/u17:2-xprtiod22:49:311
13958997440,15cyclictest29575-21awk22:19:291
1395899743,52cyclictest21464-21ssh22:04:541
13967997357,11cyclictest13943-21cyclictest23:24:592
13967997348,19cyclictest29852-21cut00:09:522
13967997348,19cyclictest29852-21cut00:09:522
13958997345,10cyclictest181rcu_preempt21:44:431
13958997331,10cyclictest181rcu_preempt20:34:431
13967997240,24cyclictest14153-21modprobe20:49:292
13967997240,24cyclictest14153-21modprobe20:49:292
13967997235,21cyclictest39-21ksoftirqd/221:24:062
1396799722,26cyclictest39-21ksoftirqd/219:59:522
13958997257,10cyclictest31-21ksoftirqd/120:59:541
13958997247,11cyclictest181rcu_preempt23:34:231
13958997247,11cyclictest181rcu_preempt23:34:231
13958997235,16cyclictest181rcu_preempt21:04:591
1395899723,46cyclictest17615-21latency_hist19:19:311
1395899723,46cyclictest17615-21latency_hist19:19:311
1395899722,57cyclictest14316-21cut19:09:521
13958997218,25cyclictest6760-21cut21:39:511
13967997157,10cyclictest13943-21cyclictest23:04:592
13967997157,10cyclictest13943-21cyclictest23:04:592
1396799715,41cyclictest39-21ksoftirqd/222:51:452
1396799715,41cyclictest39-21ksoftirqd/222:51:452
13967997149,13cyclictest0-21swapper/200:25:022
13967997147,18cyclictest0-21swapper/221:44:312
1396799714,39cyclictest39-21ksoftirqd/222:19:072
13967997143,19cyclictest12228-21latency_hist21:49:322
1396799714,29cyclictest39-21ksoftirqd/219:41:562
1396799714,29cyclictest39-21ksoftirqd/219:41:562
13958997153,10cyclictest0-21swapper/121:20:021
13958997147,16cyclictest15486-21timerandwakeup20:50:251
13958997146,8cyclictest181rcu_preempt00:20:591
13958997146,11cyclictest181rcu_preempt00:30:201
1395899713,28cyclictest18453-21ssh21:59:521
1395899713,28cyclictest18453-21ssh21:59:521
13967997055,9cyclictest13943-21cyclictest00:30:122
13967997055,10cyclictest13943-21cyclictest23:49:542
1396799703,47cyclictest6338-21runrttasks21:39:312
13967997031,15cyclictest181rcu_preempt22:04:572
1396799702,54cyclictest5014-21fschecks_time20:19:512
1396799702,48cyclictest23497-21latency_hist23:59:322
1396799702,28cyclictest39-21ksoftirqd/221:17:272
1396799702,28cyclictest39-21ksoftirqd/221:17:272
13958997053,12cyclictest329-21systemd-logind23:24:291
13958997051,13cyclictest9533-21sh23:34:571
13958997032,14cyclictest29855-21grep19:55:041
1387727012,38sleep213947-21date19:09:332
13967996954,10cyclictest13943-21cyclictest19:14:412
13967996954,10cyclictest13943-21cyclictest19:14:412
13967996943,15cyclictest0-21swapper/220:10:052
13958996955,8cyclictest31-21ksoftirqd/120:14:571
13958996952,10cyclictest13943-21cyclictest21:29:001
13958996948,16cyclictest6129-21kworker/u19:2+events_unbound23:14:281
13958996927,9cyclictest0-21swapper/123:59:471
1395899692,37cyclictest12646-21fschecks_time21:49:511
1395899692,37cyclictest12646-21fschecks_time21:49:511
13971996810,45cyclictest146722sleep320:49:443
13967996852,12cyclictest13943-21cyclictest20:34:582
13967996845,12cyclictest0-21swapper/219:29:512
13967996845,12cyclictest0-21swapper/219:29:512
1396799684,41cyclictest346-21runrttasks20:09:232
13967996843,14cyclictest0-21swapper/220:25:052
13967996839,20cyclictest0-21swapper/223:42:442
13967996839,20cyclictest0-21swapper/223:42:442
1396799683,44cyclictest0-21swapper/200:39:312
1396799683,44cyclictest0-21swapper/200:39:312
13958996854,10cyclictest13943-21cyclictest21:39:251
13958996853,9cyclictest914-21fschecks_time21:29:511
13958996849,13cyclictest8652-21latency_hist00:29:311
13958996835,20cyclictest18040-21kworker/u17:1-xprtiod00:19:271
1395899683,44cyclictest16014-21uptime23:45:061
1395899682,27cyclictest22314-21ssh21:10:511
1395899682,27cyclictest22314-21ssh21:10:511
1396799674,45cyclictest32655-21rm22:24:332
1396799674,45cyclictest32655-21rm22:24:332
1396799673,43cyclictest20995-21latency_hist19:29:322
1396799673,39cyclictest18006-21cstates20:59:502
1396799671,29cyclictest39-21ksoftirqd/223:44:552
13958996747,14cyclictest13073-21unixbench_singl23:40:071
13958996747,14cyclictest13073-21unixbench_singl23:40:071
1395899673,43cyclictest346-21runrttasks19:48:091
1395899673,43cyclictest346-21runrttasks19:48:091
13971996619,29cyclictest0-21swapper/322:59:483
13967996648,7cyclictest181rcu_preempt23:17:182
13967996648,7cyclictest181rcu_preempt23:17:182
13967996646,12cyclictest0-21swapper/219:24:312
13967996646,12cyclictest0-21swapper/219:24:312
1396799662,49cyclictest28045-21ssh00:06:012
1396799662,26cyclictest39-21ksoftirqd/200:21:342
13958996651,9cyclictest21452-21kworker/u17:2-nfsiod21:54:541
13958996641,10cyclictest181rcu_preempt22:19:571
13958996641,10cyclictest181rcu_preempt22:19:571
1395899663,47cyclictest3608-21kernelversion23:24:521
1395899663,45cyclictest346-21runrttasks19:24:211
1395899663,45cyclictest346-21runrttasks19:24:211
1396799655,24cyclictest39-21ksoftirqd/220:15:022
13967996546,7cyclictest181rcu_preempt21:07:212
1396799652,37cyclictest39-21ksoftirqd/222:29:372
1395899656,14cyclictest181rcu_preempt00:35:101
1395899656,14cyclictest181rcu_preempt00:35:101
13958996552,9cyclictest19034-21kworker/u17:1-rpciod21:15:551
13958996552,9cyclictest19034-21kworker/u17:1-rpciod21:15:551
13958996545,14cyclictest30073-21sh00:09:581
13958996545,14cyclictest30073-21sh00:09:581
13958996443,11cyclictest0-21swapper/123:16:291
13958996443,11cyclictest0-21swapper/123:16:291
1395899643,42cyclictest346-21runrttasks20:07:531
1395899643,41cyclictest346-21runrttasks22:10:561
1395899643,27cyclictest19349-21ssh22:56:191
1395899642,45cyclictest24001-21cpu23:04:451
1395899642,45cyclictest24001-21cpu23:04:451
13958996414,32cyclictest0-21swapper/119:29:451
13958996414,32cyclictest0-21swapper/119:29:451
13971996315,35cyclictest8784-21taskset00:29:423
1396799634,37cyclictest346-21runrttasks20:39:452
1396799632,21cyclictest39-21ksoftirqd/221:33:462
13958996344,14cyclictest20951-21sh23:54:491
13958996340,12cyclictest0-21swapper/119:39:571
13958996340,12cyclictest0-21swapper/119:39:571
13958996314,33cyclictest5264-21ntp_kernel_pll_20:19:571
1396799624,40cyclictest20348-21grep19:25:072
1396799622,40cyclictest23970-21modprobe19:39:282
1396799622,40cyclictest23970-21modprobe19:39:282
1397199614,37cyclictest18046-21taskset19:19:533
1397199614,37cyclictest18046-21taskset19:19:533
1397199613,41cyclictest7157-21chrt23:30:083
1397199613,41cyclictest7157-21chrt23:30:083
1395899612,46cyclictest3078-21latency_hist22:29:311
1397199602,44cyclictest25236-21chrt00:01:213
1395899603,37cyclictest20754-21modprobe19:29:281
1395899603,37cyclictest20754-21modprobe19:29:281
1397199593,41cyclictest1386-21taskset21:30:023
1397199593,39cyclictest29498-21taskset22:19:213
13967995921,30cyclictest16491-21fschecks_count20:54:512
1397199584,35cyclictest5912-21taskset23:29:313
1397199583,39cyclictest31374-21chrt00:12:153
1397199583,39cyclictest31374-21chrt00:12:153
1397199583,37cyclictest17881-21taskset22:54:463
1397199573,38cyclictest24360-21taskset22:09:533
1397199573,36cyclictest7310-21chrt20:25:073
1397199564,34cyclictest12547-21chrt23:39:573
1397199564,34cyclictest12547-21chrt23:39:573
1397199563,36cyclictest16706-21taskset19:14:593
1397199563,36cyclictest16706-21taskset19:14:593
1397199562,40cyclictest25062-21taskset21:15:033
1397199562,40cyclictest25062-21taskset21:15:033
1397199562,39cyclictest17458-21chrt23:49:173
1397199562,32cyclictest18070-21chrt21:59:323
13971995618,25cyclictest14353-21taskset20:49:313
13971995618,25cyclictest14353-21taskset20:49:313
13971995615,28cyclictest2040-21taskset20:10:013
1397199554,35cyclictest9890-21chrt21:44:533
1397199553,40cyclictest31202-21taskset23:16:063
1397199553,40cyclictest31202-21taskset23:16:063
1397199553,38cyclictest20687-21chrt23:54:323
1397199553,37cyclictest2762-21chrt23:24:123
1397199553,36cyclictest3853-21chrt21:34:523
1397199552,41cyclictest7606-21taskset00:27:033
1397199552,38cyclictest21907-21chrt19:30:053
1397199552,38cyclictest21907-21chrt19:30:053
1397199552,36cyclictest20281-21chrt21:05:073
1397199543,38cyclictest28005-21chrt19:49:573
1397199543,38cyclictest28005-21chrt19:49:573
1397199543,34cyclictest14425-21taskset19:09:533
1397199542,38cyclictest2893-21chrt00:19:333
1397199542,36cyclictest15748-21chrt22:50:043
1397199542,36cyclictest15748-21chrt22:50:043
1397199542,34cyclictest9442-21taskset23:34:543
13971995417,26cyclictest25644-21taskset19:44:303
13971995417,26cyclictest25644-21taskset19:44:303
1397199533,38cyclictest31201-21taskset21:25:083
1397199533,36cyclictest22255-21chrt22:05:113
1397199533,35cyclictest12516-21chrt21:49:473
1397199533,35cyclictest12516-21chrt21:49:473
1397199533,35cyclictest12196-21chrt20:40:083
1397199533,34cyclictest23254-21taskset19:35:033
1397199533,34cyclictest23254-21taskset19:35:033
1397199533,33cyclictest6169-21chrt22:34:403
1397199533,33cyclictest32733-21chrt22:24:363
1397199533,33cyclictest32733-21chrt22:24:363
1397199532,38cyclictest6630-21chrt21:39:473
1397199532,38cyclictest1128-21chrt22:25:373
1397199532,37cyclictest28896-21chrt23:12:543
1397199532,36cyclictest29458-21chrt19:54:553
1397199532,35cyclictest30882-21taskset19:59:493
1397199525,35cyclictest27978-21taskset21:20:023
1397199523,34cyclictest27234-21chrt00:04:583
1397199523,34cyclictest22380-21chrt23:56:393
1397199523,34cyclictest12353-21chrt00:35:053
1397199523,34cyclictest12353-21chrt00:35:053
1397199523,32cyclictest32707-21chrt20:04:553
1397199522,38cyclictest3897-21chrt22:30:033
13971995215,25cyclictest10500-21taskset20:35:063
1312025212,23sleep30-21swapper/319:05:143
1397199513,33cyclictest16807-21chrt20:54:583
1397199512,36cyclictest18251-21taskset20:59:563
13971995118,21cyclictest20786-21chrt22:04:263
13971995118,21cyclictest20786-21chrt22:04:263
13971995117,20cyclictest3614-21chrt20:14:583
13971995017,21cyclictest19769-21chrt19:24:553
13971995017,21cyclictest19769-21chrt19:24:553
13971995016,20cyclictest24254-21chrt23:04:513
13971995016,20cyclictest24254-21chrt23:04:513
13971995015,21cyclictest26020-21chrt19:44:473
13971995015,21cyclictest26020-21chrt19:44:473
13971995015,21cyclictest10849-21chrt22:42:023
1397199493,32cyclictest4745-21chrt20:19:403
13971994815,19cyclictest32469-21chrt00:14:433
13971994615,19cyclictest13182-21chrt22:45:183
13971994512,22cyclictest9036-21chrt20:31:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional