You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 04:16
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Mon May 05, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2341521076,16sleep10-21swapper/119:06:361
2341521076,16sleep10-21swapper/119:06:361
238099910693,8cyclictest0-21swapper/100:25:301
22500210624,50sleep30-21swapper/319:05:113
22500210624,50sleep30-21swapper/319:05:113
238139910590,10cyclictest23792-21cyclictest21:50:052
238139910590,10cyclictest23792-21cyclictest21:50:052
23809991054,83cyclictest1699-21apt-get23:54:571
23804991022,69cyclictest26185-21cron21:54:580
23818991002,86cyclictest5122-21chrt23:59:463
23809999972,20cyclictest13496-21apt-key20:14:461
23809999972,20cyclictest13496-21apt-key20:14:461
2380999993,34cyclictest31481-21sh23:49:581
23809999917,36cyclictest13829-21basename00:14:431
23804999810,77cyclictest12021-21/usr/sbin/munin20:09:580
23804999810,77cyclictest12021-21/usr/sbin/munin20:09:580
2380999977,73cyclictest3736-21apt-get19:44:581
2380499978,72cyclictest2247-21apt-get23:55:150
23804999747,38cyclictest20685-21sed21:45:090
23813999572,16cyclictest321-21dbus-daemon19:59:442
23809999541,46cyclictest0-21swapper/120:54:441
23809999541,46cyclictest0-21swapper/120:54:441
23804999560,26cyclictest24967-21ssh22:45:130
26102940,7sleep20-21swapper/221:15:102
26102940,7sleep20-21swapper/221:15:102
223382949,61sleep2370-21sshd19:04:572
223382949,61sleep2370-21sshd19:04:572
2381399933,66cyclictest31117-21seq19:30:012
2380999934,33cyclictest30789-21modprobe22:54:591
23809999341,45cyclictest31377-21diskmemload22:05:061
23809999341,44cyclictest0-21swapper/123:31:591
2380999925,26cyclictest13982-21cstates20:15:061
2380999925,26cyclictest13982-21cstates20:15:061
23809999231,52cyclictest0-21swapper/123:04:431
23809999215,52cyclictest31105-21apt-get19:30:091
23804999210,63cyclictest6374-21unixbench_singl00:00:220
23804999210,63cyclictest6374-21unixbench_singl00:00:220
23813999172,13cyclictest31377-21diskmemload22:28:432
2380999913,36cyclictest24222-21ssh21:50:301
2380999913,36cyclictest24222-21ssh21:50:301
2380999913,35cyclictest9112-21timerandwakeup21:25:211
2380999913,35cyclictest9112-21timerandwakeup21:25:211
2380999913,33cyclictest22415-21ssh23:34:551
23809999131,52cyclictest0-21swapper/123:14:461
23813999011,68cyclictest32048-21ssh22:55:482
2380999904,38cyclictest17149-21grep20:25:041
2380999904,36cyclictest20478-21cut21:45:051
2380999904,27cyclictest8683-21cstates20:00:041
2380999904,27cyclictest8683-21cstates20:00:041
2380999903,37cyclictest29552-21latency_hist21:59:461
2380999903,35cyclictest3232-21idleruntime-cro19:44:441
2380999903,35cyclictest27441-21latency_hist19:19:451
2380999902,33cyclictest18295-21ssh22:34:461
23809999018,23cyclictest16754-21cat23:25:071
23809999018,23cyclictest16754-21cat23:25:071
2381399893,63cyclictest19576-21apt-get00:24:582
2380999894,33cyclictest17271-21/usr/sbin/munin21:40:051
23813998867,14cyclictest28138-21latency_hist23:44:472
23813998867,14cyclictest28138-21latency_hist23:44:472
2380999884,33cyclictest27904-21fschecks_time19:20:091
2380999884,32cyclictest30228-21latency_hist22:54:471
2380999883,34cyclictest5325-21ssh21:20:031
2380999883,33cyclictest23967-21/usr/sbin/munin19:10:121
2380999883,33cyclictest23967-21/usr/sbin/munin19:10:121
2380999882,38cyclictest347-21runrttasks21:09:001
2380999882,38cyclictest347-21runrttasks21:09:001
2380999882,36cyclictest11648-21taskset22:23:231
2380999882,36cyclictest11648-21taskset22:23:231
23804998869,13cyclictest15941-21latency22:30:100
23813998758,19cyclictest13148-21latency_hist23:19:452
23809998760,18cyclictest31377-21diskmemload22:10:191
2380999874,34cyclictest23935-21uptime00:30:231
2380999874,34cyclictest23935-21uptime00:30:231
2380999874,34cyclictest18717-21/usr/sbin/munin20:30:141
2380999874,32cyclictest8826-21latency00:05:151
2380999873,39cyclictest15806-21tune2fs20:20:101
2380999873,35cyclictest30482-21processes22:00:191
2380999873,34cyclictest7353-21cat22:15:231
2380999873,34cyclictest17787-21mailstats00:20:181
2380999873,33cyclictest367-21snmpd21:14:181
2380999873,33cyclictest25431-21/usr/sbin/munin23:40:081
2380999873,33cyclictest25431-21/usr/sbin/munin23:40:081
2380999873,33cyclictest14699-21ntpq00:15:111
2380999873,33cyclictest14213-21/usr/sbin/munin21:35:191
2380999873,31cyclictest30770-21sh23:48:451
2380999873,31cyclictest27074-21cut20:55:101
23809998730,52cyclictest23792-21cyclictest22:45:231
23809998724,12cyclictest0-21swapper/122:35:541
23809998724,12cyclictest0-21swapper/122:35:541
23804998733,32cyclictest30608-21apt-get19:29:580
23813998666,14cyclictest16011-21netstat22:30:112
2380999864,34cyclictest4342-21/usr/sbin/munin23:05:161
2380999863,35cyclictest13839-21cat23:20:111
2380999863,33cyclictest347-21runrttasks00:03:431
2380999863,33cyclictest347-21runrttasks00:03:431
2380999862,36cyclictest21885-21ssh22:40:081
2380999862,32cyclictest29987-21switchtime19:25:201
2380999862,32cyclictest29987-21switchtime19:25:201
23809998621,10cyclictest0-21swapper/122:25:191
23809998616,27cyclictest22689-21switchtime20:40:211
297472850,2sleep30-21swapper/319:25:143
297472850,2sleep30-21swapper/319:25:143
2380999854,34cyclictest347-21runrttasks21:03:101
2380999854,33cyclictest21274-21unixbench_singl20:35:221
2380999853,35cyclictest10583-21latency20:05:151
2380999853,34cyclictest24154-21proc_pri20:45:181
2380999852,33cyclictest2838-21ntpq21:15:151
2380999852,33cyclictest2838-21ntpq21:15:151
23804998553,26cyclictest9258-21latency_hist22:19:460
23813998461,16cyclictest13376-21latency_hist20:14:462
23813998461,16cyclictest13376-21latency_hist20:14:462
2380999844,30cyclictest8357-21grep19:59:461
2380999844,30cyclictest347-21runrttasks21:30:011
23809998430,46cyclictest0-21swapper/123:14:551
2380999842,31cyclictest26319-21wc00:35:181
23804998467,10cyclictest27930-21ssh22:50:090
2369428414,45sleep023686-21/usr/sbin/munin19:09:450
2369428414,45sleep023686-21/usr/sbin/munin19:09:450
23818998337,34cyclictest461ktimers/300:24:583
2380999833,29cyclictest331-21cpuspeed19:35:021
23813998267,11cyclictest23792-21cyclictest20:40:202
23813998259,16cyclictest347-21runrttasks22:48:342
23813998256,19cyclictest19326-21latency_hist23:29:462
23809998216,15cyclictest0-21swapper/119:54:461
23804998261,14cyclictest26726-21kworker/u18:2-events_unbound23:00:160
23804998251,24cyclictest23792-21cyclictest23:27:300
23804998251,24cyclictest23792-21cyclictest23:27:300
23804998242,30cyclictest14091-21processes23:20:190
23818998166,10cyclictest461ktimers/321:54:583
23813998161,14cyclictest30600-21latency_hist19:29:462
23813998161,14cyclictest30600-21latency_hist19:29:462
23804998158,14cyclictest27281-21ssh23:42:200
23804998158,14cyclictest27281-21ssh23:42:200
23804998156,17cyclictest671-21latency19:35:110
2380499814,32cyclictest15-21ksoftirqd/000:07:200
23804998139,29cyclictest0-21swapper/019:44:480
2380499813,53cyclictest13550-21grep22:25:220
23804998054,17cyclictest22309-21ssh22:40:190
23804998051,19cyclictest20737-21latency00:25:390
2380499804,37cyclictest17034-21/usr/sbin/munin00:20:120
2380499803,34cyclictest25444-21latency_hist19:14:460
2380499803,34cyclictest25444-21latency_hist19:14:460
23804998011,44cyclictest0-21swapper/020:20:240
23813997959,14cyclictest3443-21grep19:44:472
23813997958,15cyclictest20134-21latency_hist20:34:452
23813997930,24cyclictest181rcu_preempt00:31:232
23813997930,24cyclictest181rcu_preempt00:31:232
23813997926,26cyclictest28682-21fschecks_count23:45:072
23804997956,18cyclictest27489-21kworker/u19:2-writeback00:10:120
2380499793,41cyclictest30793-21acpi22:54:590
23813997854,9cyclictest181rcu_preempt23:20:142
23813997847,23cyclictest23792-21cyclictest00:20:022
23813997842,18cyclictest0-21swapper/222:12:012
23813997831,15cyclictest0-21swapper/200:00:092
23813997831,15cyclictest0-21swapper/200:00:092
23804997862,9cyclictest31566-21ssh23:50:020
23804997852,21cyclictest23792-21cyclictest20:40:190
2380499784,50cyclictest4945-21ntp_kernel_pll_23:05:130
2380499783,45cyclictest20104-21cat21:44:460
23813997754,12cyclictest0-21swapper/221:10:212
23813997754,10cyclictest181rcu_preempt19:14:472
23813997741,12cyclictest181rcu_preempt21:20:022
23813997737,13cyclictest0-21swapper/219:50:022
2381399771,46cyclictest25164-21/usr/sbin/munin20:50:122
2381399771,46cyclictest25164-21/usr/sbin/munin20:50:122
23804997754,15cyclictest31945-21ssh21:10:060
23804997738,25cyclictest0-21swapper/021:50:100
23804997738,25cyclictest0-21swapper/021:50:100
23804997736,28cyclictest0-21swapper/000:15:060
23804997711,39cyclictest0-21swapper/021:05:200
23804997711,39cyclictest0-21swapper/021:05:200
23813997632,13cyclictest0-21swapper/200:11:442
23804997653,14cyclictest23792-21cyclictest22:10:030
2380499764,58cyclictest10325-21/usr/sbin/munin23:15:210
251362750,10sleep225107-21ssh23:39:432
23813997548,14cyclictest181rcu_preempt23:01:532
23813997548,11cyclictest181rcu_preempt21:59:382
23804997553,15cyclictest9030-21sh23:11:570
23804997552,17cyclictest31377-21diskmemload21:16:370
23804997552,17cyclictest31377-21diskmemload21:16:370
23804997543,25cyclictest347-21runrttasks22:38:290
23804997543,25cyclictest347-21runrttasks22:38:290
2380499753,50cyclictest347-21runrttasks21:19:480
2380499753,42cyclictest9745-21fschecks_time22:20:060
2380499753,42cyclictest9745-21fschecks_time22:20:060
23804997524,36cyclictest26082-21ntp_offset00:35:110
23813997456,11cyclictest23792-21cyclictest21:25:232
23813997456,11cyclictest23792-21cyclictest21:25:232
23813997412,30cyclictest14221-21acpi21:34:552
2380499748,29cyclictest30389-21ntpq22:00:150
23804997455,14cyclictest19778-21kworker/u18:1+events_unbound20:34:430
2380499744,55cyclictest30605-21munin-run19:29:430
2380499744,55cyclictest30605-21munin-run19:29:430
23804997427,27cyclictest31138-21munin-run23:49:430
23813997356,11cyclictest347-21runrttasks23:13:542
23813997349,8cyclictest181rcu_preempt20:22:422
23813997336,11cyclictest181rcu_preempt00:34:582
2380499734,37cyclictest708-21idleruntime22:05:080
2380499733,35cyclictest27056-21fschecks_time20:55:090
23804997313,48cyclictest8527-21sh21:25:080
23804997313,48cyclictest8527-21sh21:25:080
23813997249,17cyclictest28382-21latency_hist20:59:472
23813997245,19cyclictest1817-21apt-key23:54:452
23813997225,31cyclictest0-21swapper/221:30:192
2380499724,53cyclictest25207-21latency_hist00:34:460
2380499724,53cyclictest25207-21latency_hist00:34:460
23813997156,10cyclictest31377-21diskmemload22:38:472
23813997156,10cyclictest31377-21diskmemload22:38:472
23813997149,16cyclictest9859-21kernelversion22:20:092
23813997149,16cyclictest9859-21kernelversion22:20:092
23813997146,8cyclictest181rcu_preempt23:28:062
23813997146,19cyclictest11789-21apt-key20:09:472
23813997129,12cyclictest181rcu_preempt23:56:382
23813997126,25cyclictest24925-21munin-run20:49:442
23813997126,12cyclictest181rcu_preempt00:07:092
2380499714,52cyclictest22539-21cstates23:35:010
2380499714,52cyclictest13380-21apt-get20:14:570
2380499714,52cyclictest13380-21apt-get20:14:570
2380499713,40cyclictest14486-21fschecks_count21:35:060
23804997115,41cyclictest11520-21modprobe20:09:430
23804997115,12cyclictest26904-21/usr/sbin/munin19:15:220
23804997114,25cyclictest347-21runrttasks20:25:020
23813997049,13cyclictest0-21swapper/219:20:152
23813997046,10cyclictest181rcu_preempt21:49:472
23813997041,20cyclictest21680-21taskset22:40:022
2381399703,47cyclictest912-21ntp_kernel_err22:05:122
2380499706,23cyclictest9161-21grep20:00:190
2380499706,23cyclictest9161-21grep20:00:190
2380499703,37cyclictest27728-21cpuspeed19:20:040
2380499703,33cyclictest5859-21ntp_kernel_pll_19:50:130
2380499702,55cyclictest11462-21grep21:30:090
2381399699,49cyclictest3736-21apt-get19:44:592
23813996952,11cyclictest23792-21cyclictest21:04:462
23813996951,11cyclictest31557-21grep21:09:472
23813996951,11cyclictest31557-21grep21:09:472
23813996941,10cyclictest181rcu_preempt00:17:382
23813996940,13cyclictest181rcu_preempt22:18:322
23813996929,12cyclictest181rcu_preempt22:50:082
23804996955,10cyclictest23792-21cyclictest21:04:120
2380499694,53cyclictest3237-21apt-get19:44:580
2380499693,51cyclictest21176-21timerwakeupswit20:35:210
23813996852,10cyclictest19523-21ssh23:30:012
2381399684,47cyclictest1731-21latency_hist19:39:462
23813996834,9cyclictest181rcu_preempt19:10:062
23813996834,9cyclictest181rcu_preempt19:10:062
23813996831,10cyclictest0-21swapper/223:04:552
23813996829,15cyclictest181rcu_preempt20:34:562
23804996849,13cyclictest0-21swapper/023:30:060
23804996849,12cyclictest207-21systemd-journal20:49:570
23804996849,12cyclictest207-21systemd-journal20:49:570
2380499683,33cyclictest23554-21/usr/sbin/munin20:45:090
268602670,3sleep3451rcuc/322:48:463
23813996746,15cyclictest0-21swapper/221:40:232
23813996729,11cyclictest0-21swapper/222:02:402
2380499673,51cyclictest347-21runrttasks19:55:560
23813996633,9cyclictest20429-21kworker/u18:3-events_unbound20:15:292
23813996633,9cyclictest20429-21kworker/u18:3-events_unbound20:15:292
23813996538,9cyclictest181rcu_preempt20:04:452
23813996538,9cyclictest181rcu_preempt20:04:452
2381399653,45cyclictest18044-21unixbench_multi20:25:242
237422650,11chrt0-21swapper/321:50:183
237422650,11chrt0-21swapper/321:50:183
23818996451,8cyclictest461ktimers/320:10:053
23818996451,8cyclictest461ktimers/320:10:053
2381899633,36cyclictest25028-21chrt23:39:323
23818996128,19cyclictest0-21swapper/319:44:583
2381899612,39cyclictest20776-21taskset21:45:113
2381899604,50cyclictest0-21swapper/322:55:483
2381899583,38cyclictest16872-21taskset20:24:433
23818995741,11cyclictest461ktimers/322:19:583
23818995741,11cyclictest461ktimers/322:19:583
2381899573,36cyclictest0-21swapper/319:30:043
2381899563,39cyclictest7325-21chrt22:15:223
2381899562,42cyclictest19455-21taskset20:30:223
2381899562,40cyclictest896-21kworker/3:1+pm00:20:063
2381899555,40cyclictest20939-21chrt23:31:333
2381899554,34cyclictest5550-21chrt23:05:363
23818995535,12cyclictest461ktimers/320:59:433
23818995514,31cyclictest0-21swapper/322:29:433
2381899542,36cyclictest1890-21chrt21:14:423
23818995335,13cyclictest461ktimers/322:39:573
23818995335,11cyclictest461ktimers/319:54:563
2381899533,39cyclictest1854-21chrt22:06:273
2381899533,38cyclictest10952-21chrt00:09:433
2381899524,36cyclictest891-21taskset19:35:183
2381899523,36cyclictest28625-21chrt23:45:033
2381899523,35cyclictest6106-21chrt21:20:223
2381899523,35cyclictest17982-21chrt23:26:433
2381899523,35cyclictest17982-21chrt23:26:433
2381899523,34cyclictest4596-21chrt21:18:563
2381899523,34cyclictest4596-21chrt21:18:563
2381899523,32cyclictest18370-21chrt21:40:253
2381899522,38cyclictest16877-21chrt22:31:113
2381899522,36cyclictest31407-21taskset22:01:453
2381899522,36cyclictest29833-21chrt21:04:453
2381899513,37cyclictest10369-21chrt21:28:173
2381899513,37cyclictest10369-21chrt21:28:173
2381899513,35cyclictest3826-21chrt22:10:123
2381899513,34cyclictest14325-21chrt00:15:023
2381899513,33cyclictest6283-21taskset19:50:213
2381899513,33cyclictest12847-21chrt21:31:493
23818995130,12cyclictest461ktimers/320:25:163
2381899512,36cyclictest25061-21chrt20:49:473
2381899503,35cyclictest24047-21chrt00:30:433
2381899503,35cyclictest24047-21chrt00:30:433
2381899503,34cyclictest21796-21taskset20:39:463
2381899503,34cyclictest14032-21taskset20:15:073
2381899503,34cyclictest14032-21taskset20:15:073
2381899503,33cyclictest15802-21chrt23:24:043
23818995026,17cyclictest461ktimers/300:10:183
2381899502,38cyclictest19108-21chrt22:35:163
2381899502,38cyclictest19108-21chrt22:35:163
2381899493,38cyclictest31477-21chrt23:49:573
2381899493,35cyclictest12592-21chrt23:18:323
2381899493,32cyclictest27640-21taskset23:43:213
2381899493,32cyclictest27640-21taskset23:43:213
23818994930,12cyclictest461ktimers/300:00:063
23818994930,12cyclictest461ktimers/300:00:063
23818994928,15cyclictest461ktimers/319:15:223
23818994924,17cyclictest461ktimers/320:54:473
23818994924,17cyclictest461ktimers/320:54:473
23818994924,17cyclictest461ktimers/320:04:463
23818994924,17cyclictest461ktimers/320:04:463
2381899492,38cyclictest15367-21chrt21:35:263
2381899492,35cyclictest29669-21taskset22:53:113
2381899483,35cyclictest27689-21chrt19:20:033
2381899483,33cyclictest7570-21chrt23:10:053
2381899483,33cyclictest2411-21chrt19:40:123
2381899483,32cyclictest4033-21chrt23:04:423
2381899482,33cyclictest30970-21chrt21:05:223
2381899482,33cyclictest30970-21chrt21:05:223
2381899482,33cyclictest27304-21chrt00:37:393
23818994729,13cyclictest461ktimers/320:43:563
23818994729,13cyclictest461ktimers/320:05:113
23818994725,16cyclictest461ktimers/319:10:123
23818994725,16cyclictest461ktimers/319:10:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional