You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-21 - 05:28
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Oct 21, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16869213414,83sleep216885-21gpgv19:09:492
16869213414,83sleep216885-21gpgv19:09:492
17065991143,89cyclictest5837-21apt-get00:20:020
170739911156,50cyclictest17051-21cyclictest23:30:321
170739910450,48cyclictest1489-21latency_hist19:59:521
170739910026,24cyclictest27235-21modprobe19:39:481
1668621005,74sleep00-21swapper/019:07:520
1668621005,74sleep00-21swapper/019:07:520
69402990,3sleep0211rcuc/021:34:500
69402990,3sleep0211rcuc/021:34:500
17065999955,37cyclictest8322-21sh21:38:270
17065999815,51cyclictest15-21ksoftirqd/020:27:170
17077999777,10cyclictest352-21runrttasks00:20:242
17077999710,76cyclictest0-21swapper/223:30:052
17073999637,10cyclictest0-21swapper/120:29:511
17073999632,12cyclictest0-21swapper/121:25:061
17073999436,14cyclictest23489-21latency_hist23:54:491
17073999434,8cyclictest0-21swapper/122:09:471
1707399943,28cyclictest27690-21switchtime22:10:241
1707399938,43cyclictest6015-21diskmemload00:20:241
1707399934,36cyclictest26373-21latency_hist23:59:521
17073999339,11cyclictest6022-21ssh21:32:311
17073999339,11cyclictest6022-21ssh21:32:311
1707399933,36cyclictest20511-21ntp_offset20:55:171
1707399933,36cyclictest20511-21ntp_offset20:55:171
1707399933,35cyclictest15499-21ntpq21:50:171
1707399933,35cyclictest15499-21ntpq21:50:171
17073999329,12cyclictest0-21swapper/100:19:501
17073999328,14cyclictest0-21swapper/120:34:511
1707399922,41cyclictest11889-21cpu23:35:051
1707399914,32cyclictest23631-21/usr/sbin/munin23:00:141
17073999136,8cyclictest0-21swapper/122:30:111
17073999136,8cyclictest0-21swapper/122:30:111
1707399913,36cyclictest8375-21idleruntime-cro23:29:471
1707399912,35cyclictest17494-21idleruntime-cro23:44:471
1707799903,76cyclictest20508-21apt-get22:00:032
1707399904,34cyclictest2255-21fschecks_time20:00:281
1707399904,34cyclictest2255-21fschecks_time20:00:281
17073999038,45cyclictest12230-21sh21:45:081
1707399903,33cyclictest9386-21fschecks_count21:40:071
1707399903,32cyclictest12231-21fschecks_time22:40:111
1707399903,31cyclictest18142-21fschecks_time21:55:151
17073999032,12cyclictest0-21swapper/120:35:381
17073999029,11cyclictest0-21swapper/123:45:201
17073999029,11cyclictest0-21swapper/123:45:201
1707399902,30cyclictest25245-21idleruntime21:10:111
1706599903,48cyclictest24147-21cstates22:05:070
1707399893,36cyclictest17258-21nfs4_client20:45:151
1707399893,35cyclictest32089-21latency_hist19:54:481
17073998928,11cyclictest0-21swapper/123:20:261
1707399892,40cyclictest19465-21runrttasks22:52:111
1707399892,40cyclictest19465-21runrttasks22:52:111
1707399892,39cyclictest24666-21ssh23:55:231
1707399892,34cyclictest31263-21cut21:20:091
17065998969,14cyclictest24593-21diskmemload00:25:270
17065998964,19cyclictest15023-21latency_hist20:39:510
17065998913,52cyclictest15-21ksoftirqd/019:38:230
17077998842,39cyclictest0-21swapper/219:59:482
17077998830,50cyclictest0-21swapper/222:05:162
1707399884,36cyclictest6823-21/usr/sbin/munin20:15:241
1707399884,33cyclictest5471-21fschecks_time20:10:091
1707399884,31cyclictest24577-21modprobe21:09:481
1707399883,38cyclictest21262-21fschecks_count19:20:131
1707399883,37cyclictest4039-21seq20:05:141
1707399883,37cyclictest4039-21seq20:05:141
1707399883,36cyclictest23450-21latency_hist22:59:501
1707399883,36cyclictest21332-21cstates22:00:271
1707399883,35cyclictest19129-21wc19:14:571
1707399883,35cyclictest15128-21unixbench_singl00:35:261
1707399883,34cyclictest802-21timerandwakeup23:15:231
1707399883,33cyclictest22964-21modprobe21:04:471
1707399883,32cyclictest30504-21timerandwakeup00:05:231
1707399883,32cyclictest16111-21head20:40:251
1707399883,31cyclictest24192-21latency_hist19:29:511
1707399883,30cyclictest18563-21cut20:50:111
1707399883,30cyclictest18563-21cut20:50:111
1707399882,34cyclictest28918-21latency_hist19:44:501
17065998865,8cyclictest181rcu_preempt21:56:560
1707399873,39cyclictest3044-21latency_hist22:24:511
1707399873,33cyclictest9981-21latency_hist20:24:511
1707399873,33cyclictest8834-21/usr/sbin/munin00:25:171
1707399873,30cyclictest30090-21grep19:45:291
1707399873,29cyclictest8984-21/usr/sbin/munin22:35:271
1707799863,66cyclictest0-21swapper/220:05:032
1707799863,66cyclictest0-21swapper/220:05:032
17077998636,38cyclictest0-21swapper/222:25:082
1707799862,71cyclictest0-21swapper/219:35:022
1707399863,38cyclictest8797-21cron21:39:471
1707399863,35cyclictest15745-21grep22:45:241
1707399863,32cyclictest32341-21/usr/sbin/munin00:10:051
17073998628,9cyclictest0-21swapper/119:30:121
17065998652,24cyclictest1272-21latency_hist21:24:510
1706599864,44cyclictest15-21ksoftirqd/022:46:240
1706599863,73cyclictest32618-21latency_hist22:19:510
17065998621,44cyclictest15-21ksoftirqd/023:30:180
17065998610,43cyclictest15-21ksoftirqd/020:13:100
1707799852,77cyclictest0-21swapper/223:40:042
1707799852,68cyclictest0-21swapper/223:05:042
1707399854,27cyclictest30003-21fschecks_count22:15:091
1707399853,35cyclictest26577-21cpu23:05:061
1707399853,32cyclictest18006-21proc_pri19:10:411
1707399853,30cyclictest3184-21/usr/sbin/munin22:25:091
1706599852,45cyclictest15-21ksoftirqd/022:21:150
1707399844,29cyclictest30770-21latency_hist21:19:511
1707399843,30cyclictest29742-21fschecks_time23:10:111
1707399843,30cyclictest29742-21fschecks_time23:10:111
17073998430,13cyclictest0-21swapper/100:30:201
1706599843,33cyclictest15-21ksoftirqd/020:31:130
17077998338,38cyclictest15127-21sh21:50:102
17077998338,38cyclictest15127-21sh21:50:102
17077998337,39cyclictest0-21swapper/220:10:072
1707799833,65cyclictest0-21swapper/200:29:592
1706599839,63cyclictest21096-21apt-get22:00:020
1706599837,46cyclictest15-21ksoftirqd/000:34:510
1706599836,50cyclictest15-21ksoftirqd/023:25:210
17065998357,20cyclictest1971-21sh23:18:160
17065998355,20cyclictest24231-21latency_hist19:29:510
17065998353,14cyclictest181rcu_preempt21:44:310
1706599833,58cyclictest15-21ksoftirqd/022:58:530
1706599833,47cyclictest15-21ksoftirqd/022:27:150
1707799824,38cyclictest8946-21ssh00:25:082
17077998237,38cyclictest0-21swapper/223:20:102
1707799823,68cyclictest0-21swapper/221:09:592
1707799822,65cyclictest0-21swapper/221:30:012
1707799822,65cyclictest0-21swapper/221:30:012
1707399823,33cyclictest17214-21latency_hist19:09:531
1707399823,33cyclictest17214-21latency_hist19:09:531
1706599823,54cyclictest15-21ksoftirqd/020:00:100
1706599823,54cyclictest15-21ksoftirqd/020:00:100
1706599823,42cyclictest15-21ksoftirqd/022:41:290
1706599822,43cyclictest15-21ksoftirqd/021:15:180
1706599822,35cyclictest15-21ksoftirqd/019:15:010
1707799812,74cyclictest0-21swapper/222:30:022
1707799812,74cyclictest0-21swapper/222:30:022
1706599817,37cyclictest32572-21/usr/sbin/munin19:55:080
1706599814,46cyclictest15-21ksoftirqd/022:38:480
1706599814,46cyclictest15-21ksoftirqd/019:21:230
1706599813,48cyclictest15-21ksoftirqd/000:12:510
1706599813,42cyclictest15-21ksoftirqd/020:08:250
1706599813,42cyclictest15-21ksoftirqd/020:08:250
1707799805,63cyclictest0-21swapper/200:14:592
1707799803,67cyclictest0-21swapper/200:35:002
1707799802,65cyclictest0-21swapper/220:15:032
1707799802,34cyclictest519-21fschecks_count22:20:102
1706599803,37cyclictest15-21ksoftirqd/022:11:270
1706599803,36cyclictest17756-21/usr/sbin/munin22:49:580
1706599803,36cyclictest17756-21/usr/sbin/munin22:49:580
17077997937,36cyclictest0-21swapper/222:55:272
1707799792,62cyclictest0-21swapper/220:55:002
1707799792,62cyclictest0-21swapper/220:55:002
17077997917,55cyclictest17196-21grep20:45:132
1706599795,36cyclictest2888-21grep00:15:090
17065997952,18cyclictest14339-21sh23:39:270
1706599793,45cyclictest15-21ksoftirqd/022:32:250
1706599793,45cyclictest15-21ksoftirqd/022:32:250
1706599792,38cyclictest15-21ksoftirqd/000:08:590
17077997837,33cyclictest0-21swapper/200:14:502
1707799783,70cyclictest0-21swapper/222:35:052
17077997834,36cyclictest0-21swapper/223:04:502
17077997833,39cyclictest0-21swapper/222:50:052
17077997833,39cyclictest0-21swapper/222:50:052
1707799782,70cyclictest0-21swapper/223:45:002
1707799782,70cyclictest0-21swapper/223:45:002
1707799782,65cyclictest0-21swapper/220:40:012
1707799782,61cyclictest10088-21apt-get20:25:012
17077997822,27cyclictest0-21swapper/222:45:162
1706599786,27cyclictest15-21ksoftirqd/023:02:200
1706599784,37cyclictest25516-21ntp_states21:10:180
1706599782,42cyclictest15-21ksoftirqd/000:29:570
17077997753,17cyclictest18741-21timerandwakeup21:55:282
17077997753,13cyclictest0-21swapper/221:45:102
1707799773,61cyclictest0-21swapper/223:54:582
17077997734,31cyclictest18710-21if_eth020:50:132
17077997734,31cyclictest18710-21if_eth020:50:132
17077997732,38cyclictest0-21swapper/221:40:192
1707799772,70cyclictest0-21swapper/219:50:022
1706599779,40cyclictest15-21ksoftirqd/019:33:190
1706599774,34cyclictest4087-21ls21:29:510
1706599773,48cyclictest15-21ksoftirqd/023:05:170
1706599773,47cyclictest26456-21latency_hist23:59:500
1706599773,46cyclictest15-21ksoftirqd/020:57:120
1706599773,46cyclictest15-21ksoftirqd/020:57:120
17065997726,40cyclictest23653-21cat23:54:500
17065997718,37cyclictest15-21ksoftirqd/021:00:240
17065997716,38cyclictest15-21ksoftirqd/020:54:370
17065997716,38cyclictest15-21ksoftirqd/020:54:370
1707799763,69cyclictest0-21swapper/223:35:022
1707799763,62cyclictest0-21swapper/219:39:592
1707799763,58cyclictest27974-21ssh21:15:062
1707799762,68cyclictest0-21swapper/220:35:302
1706599762,29cyclictest15-21ksoftirqd/000:00:280
1709399758,48cyclictest0-21swapper/323:30:213
17077997552,16cyclictest14754-21latency_hist22:44:512
17077997534,34cyclictest0-21swapper/220:24:522
17077997532,36cyclictest0-21swapper/220:00:572
17077997532,36cyclictest0-21swapper/220:00:572
17077997529,39cyclictest0-21swapper/221:20:072
1707799752,68cyclictest0-21swapper/219:30:042
1707799752,67cyclictest0-21swapper/200:00:052
1707799752,66cyclictest0-21swapper/221:00:002
1706599758,44cyclictest15-21ksoftirqd/020:17:490
1706599755,43cyclictest15-21ksoftirqd/019:48:110
17065997552,15cyclictest14965-21ssh23:40:060
1706599754,42cyclictest15-21ksoftirqd/020:23:470
1706599753,46cyclictest15-21ksoftirqd/020:42:070
1706599753,36cyclictest15-21ksoftirqd/019:12:500
17065997522,29cyclictest15-21ksoftirqd/021:45:240
1707799744,63cyclictest0-21swapper/219:10:202
1707799743,66cyclictest0-21swapper/219:20:062
1707799743,60cyclictest11694-21apt-get20:30:052
1707799742,67cyclictest0-21swapper/223:50:052
1707799742,65cyclictest0-21swapper/221:25:022
1707799742,57cyclictest0-21swapper/219:24:592
1706599742,38cyclictest352-21runrttasks19:49:570
1635527412,34sleep30-21swapper/319:05:343
1635527412,34sleep30-21swapper/319:05:343
17077997341,18cyclictest181rcu_preempt00:05:242
1707799733,65cyclictest0-21swapper/221:05:042
1707799733,62cyclictest29321-21grep19:45:142
1707799733,61cyclictest5984-21forks23:25:132
17077997333,34cyclictest29469-21ssh23:10:042
17077997333,34cyclictest29469-21ssh23:10:042
1706599736,37cyclictest17979-21df23:45:060
1706599736,37cyclictest17979-21df23:45:060
1706599733,36cyclictest15-21ksoftirqd/020:49:510
1706599732,39cyclictest17607-21sed21:54:500
1706599732,39cyclictest17607-21sed21:54:500
17077997230,34cyclictest0-21swapper/222:10:132
1706599725,43cyclictest15-21ksoftirqd/023:14:250
1706599725,43cyclictest15-21ksoftirqd/023:14:250
17065997211,36cyclictest15-21ksoftirqd/021:08:420
1707799712,64cyclictest0-21swapper/223:14:592
1707799702,56cyclictest20008-21/usr/sbin/munin19:15:222
1706599703,41cyclictest15-21ksoftirqd/023:22:560
1709399693,37cyclictest0-21swapper/322:00:123
1707799692,58cyclictest8851-21sendmail-msp21:39:482
1707799682,56cyclictest31557-21ssh22:17:242
17065996823,37cyclictest28251-21switchtime19:40:210
17093996417,29cyclictest0-21swapper/300:20:033
17093996341,14cyclictest451rcuc/321:24:203
1709399593,42cyclictest20449-21chrt23:49:483
1709399593,42cyclictest20449-21chrt23:49:483
17093995845,9cyclictest451rcuc/322:23:323
17093995843,11cyclictest451rcuc/319:15:273
17093995840,11cyclictest451rcuc/321:17:313
17093995840,11cyclictest451rcuc/321:01:353
17093995836,14cyclictest451rcuc/300:17:313
1709399583,38cyclictest18348-21taskset20:49:523
17093995739,11cyclictest451rcuc/320:20:453
1709399568,31cyclictest0-21swapper/320:35:063
17093995643,9cyclictest451rcuc/322:55:493
1709399563,44cyclictest28150-21taskset00:01:223
17093995542,9cyclictest451rcuc/323:59:213
17093995542,9cyclictest451rcuc/320:18:143
17093995538,10cyclictest451rcuc/323:11:533
17093995538,10cyclictest451rcuc/323:11:533
17093995536,12cyclictest451rcuc/320:56:043
17093995536,12cyclictest451rcuc/320:56:043
1709399553,39cyclictest26832-21chrt22:09:543
1709399544,37cyclictest24002-21chrt23:00:133
17093995441,9cyclictest451rcuc/322:06:593
17093995441,9cyclictest451rcuc/319:43:243
17093995435,12cyclictest451rcuc/320:01:263
17093995435,12cyclictest451rcuc/320:01:263
1709399543,37cyclictest29756-21chrt19:45:253
17093995341,8cyclictest451rcuc/321:42:193
17093995340,9cyclictest451rcuc/322:41:353
17093995340,8cyclictest451rcuc/321:36:073
1709399533,33cyclictest32297-21taskset23:14:523
1709399524,36cyclictest12451-21taskset23:35:183
1709399524,35cyclictest18518-21chrt22:50:213
1709399524,35cyclictest18518-21chrt22:50:213
17093995238,9cyclictest451rcuc/320:08:593
17093995238,9cyclictest451rcuc/320:08:593
1709399523,38cyclictest6778-21chrt21:34:363
1709399523,38cyclictest6778-21chrt21:34:363
1709399523,35cyclictest20182-21chrt21:58:563
1709399522,39cyclictest11153-21taskset20:26:273
1709399522,36cyclictest28027-21taskset23:06:373
1709399514,36cyclictest10346-21taskset22:36:113
17093995139,8cyclictest451rcuc/322:24:543
17093995138,9cyclictest451rcuc/321:09:083
17093995138,9cyclictest451rcuc/319:31:413
1709399513,35cyclictest11831-21chrt20:30:083
1709399513,34cyclictest5974-21chrt23:25:123
17093995131,14cyclictest451rcuc/319:51:133
1709399512,40cyclictest32373-21taskset00:10:013
1709399512,33cyclictest8783-21chrt00:24:523
1709399512,33cyclictest19844-21chrt20:54:503
1709399512,33cyclictest19844-21chrt20:54:503
1709399512,31cyclictest32718-21chrt19:55:093
1709399503,38cyclictest15917-21chrt21:50:253
1709399503,38cyclictest15917-21chrt21:50:253
1709399503,35cyclictest25589-21taskset21:10:213
1709399503,34cyclictest29698-21chrt22:14:523
1709399503,33cyclictest3762-21chrt21:29:133
1709399502,39cyclictest13326-21taskset00:33:103
17093994936,9cyclictest451rcuc/320:13:283
1709399493,36cyclictest15453-21taskset23:40:203
1709399493,34cyclictest3517-21chrt23:20:223
1709399493,33cyclictest12487-21chrt21:45:143
17093994834,9cyclictest451rcuc/319:35:303
17093994735,8cyclictest451rcuc/322:45:233
17093994734,9cyclictest451rcuc/323:53:173
1709399473,32cyclictest21870-21chrt19:20:273
17093994726,14cyclictest451rcuc/320:42:073
1709399472,31cyclictest30589-21taskset00:05:233
1709399462,33cyclictest15002-21chrt00:35:253
1709399442,28cyclictest23318-21chrt19:25:213
17093994128,8cyclictest451rcuc/322:32:243
17093994128,8cyclictest451rcuc/322:32:243
17093994026,9cyclictest451rcuc/319:11:293
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional