You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-02 - 06:58
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Wed Jul 02, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
210349912181,34cyclictest2063-21sh23:06:032
2072121187,15sleep00-21swapper/019:04:510
21024991173,71cyclictest18967-21apt-get20:37:020
21034991112,28cyclictest27800-21cron19:26:372
2057521106,82sleep10-21swapper/119:02:561
21024991072,83cyclictest28961-21apt-get21:07:060
21024991072,83cyclictest28961-21apt-get21:07:060
21034991022,22cyclictest969-21seq19:41:562
21034991022,22cyclictest969-21seq19:41:562
21024991012,70cyclictest31568-21kworker/u18:2+ext4-rsv-conversion19:42:060
21024991012,70cyclictest31568-21kworker/u18:2+ext4-rsv-conversion19:42:060
2894821000,6sleep00-21swapper/022:56:400
2894821000,6sleep00-21swapper/022:56:400
210249910055,37cyclictest5585-21apt-get22:16:530
2103499992,30cyclictest16640-21seq20:29:212
2103499992,30cyclictest16640-21seq20:29:212
208432997,17sleep20-21swapper/219:06:282
2103499983,29cyclictest3758-21seq19:49:492
2103499983,29cyclictest3758-21seq19:49:492
2103499982,28cyclictest10174-21seq20:09:172
2103499982,28cyclictest10174-21seq20:09:172
2103499973,27cyclictest29066-21sh21:07:062
2103499973,27cyclictest29066-21sh21:07:062
2103499973,23cyclictest5329-21seq19:54:232
2103499973,23cyclictest5329-21seq19:54:232
2103499972,24cyclictest28270-21seq21:05:162
2103499972,24cyclictest28270-21seq21:05:162
21024999729,32cyclictest19573-21seq21:46:080
2103499963,28cyclictest15045-21runrttasks20:24:272
2103499963,26cyclictest22682-21runrttasks22:45:582
2103499962,29cyclictest30207-21ntp_offset19:32:072
2103499962,24cyclictest8627-21seq20:05:002
2103499962,24cyclictest8627-21seq20:05:002
2103499962,22cyclictest22514-21seq19:11:352
21024999666,19cyclictest10533-21latency_hist20:11:400
21024999666,19cyclictest10533-21latency_hist20:11:400
2103499943,28cyclictest11867-21runrttasks20:15:162
2103499942,31cyclictest13378-21runrttasks20:18:402
2103499932,29cyclictest17080-21runrttasks20:31:482
2103499932,29cyclictest17080-21runrttasks20:31:482
21028999249,12cyclictest11361-21ssh23:21:401
21034999148,36cyclictest16600-21sh22:35:262
21034999147,35cyclictest0-21swapper/221:37:122
21034999144,40cyclictest0-21swapper/223:12:162
21024999145,9cyclictest23241-21apt-get22:46:510
2102499913,57cyclictest31559-21apt-get19:37:020
21034999048,35cyclictest32680-21latency23:57:052
21024999020,39cyclictest0-21swapper/022:43:100
21034998946,36cyclictest20096-21sh00:32:092
2102899894,36cyclictest5708-21latency_hist19:56:401
2102899894,36cyclictest5708-21latency_hist19:56:401
21024998943,33cyclictest14028-21apt-get00:21:490
21024998943,33cyclictest14028-21apt-get00:21:490
2102499892,56cyclictest2711-21taskset22:12:080
21024998923,30cyclictest16753-21runrttasks22:35:530
2103499884,76cyclictest0-21swapper/221:21:552
2103499882,8cyclictest0-21swapper/221:51:542
21028998840,41cyclictest0-21swapper/100:36:361
2102899883,38cyclictest1-21systemd00:16:391
2102899883,26cyclictest21345-21df19:06:581
21034998746,34cyclictest0-21swapper/221:16:542
21028998735,44cyclictest0-21swapper/119:17:141
21024998766,16cyclictest31568-21kworker/u18:2+events_unbound21:06:360
21024998766,16cyclictest31568-21kworker/u18:2+events_unbound21:06:360
21024998759,17cyclictest31877-21apt-get22:06:540
21024998740,40cyclictest0-21swapper/021:31:480
21024998740,40cyclictest0-21swapper/021:31:480
21034998665,15cyclictest451-21sh21:12:162
21034998643,33cyclictest0-21swapper/222:27:132
21034998643,33cyclictest0-21swapper/222:27:132
2103499862,29cyclictest24261-21timerandwakeup23:42:162
2102899864,35cyclictest23433-21fschecks_count19:12:021
2102899862,29cyclictest17264-21ntpq00:27:061
21024998663,16cyclictest11119-21apt-key00:16:400
21024998652,25cyclictest31920-21apt-get21:11:550
21024998618,38cyclictest16709-21ssh00:26:480
21034998548,31cyclictest26914-21diskmemload22:52:132
21034998548,31cyclictest26914-21diskmemload22:52:132
21034998544,35cyclictest28931-21timerwakeupswit19:27:142
2102899853,41cyclictest31903-40mandb23:56:401
2102899853,41cyclictest31903-40mandb23:56:401
2102899853,41cyclictest21131-21grep21:47:151
21024998554,21cyclictest29687-21apt-get19:31:540
21024998527,18cyclictest18522-21taskset23:32:200
21024998527,18cyclictest18522-21taskset23:32:200
2102899844,40cyclictest32100-21df21:11:591
21028998435,42cyclictest0-21swapper/120:52:031
2102899843,36cyclictest29348-21kernelversion22:02:031
21028998433,44cyclictest0-21swapper/120:36:401
21028998433,44cyclictest0-21swapper/120:36:401
2102899842,32cyclictest14327-21latency_hist22:31:411
2102899842,30cyclictest13267-21ssh23:24:101
2102899842,30cyclictest13267-21ssh23:24:101
21034998340,35cyclictest0-21swapper/223:32:002
21034998340,35cyclictest0-21swapper/223:32:002
21034998339,36cyclictest0-21swapper/219:37:262
21034998338,38cyclictest0-21swapper/200:16:402
2103499832,40cyclictest17138-21ssh21:41:482
2103499832,39cyclictest14051-21taskset00:21:542
2103499832,39cyclictest14051-21taskset00:21:542
2102899834,38cyclictest20605-21fschecks_time22:42:011
2102899834,35cyclictest5252-21latency_hist00:06:381
2102899834,25cyclictest11457-21apt-get00:17:071
2102899833,40cyclictest30593-21grep19:32:151
2102899833,28cyclictest24620-21diskmemload23:42:471
2102899833,27cyclictest9329-21grep21:27:151
2102899833,27cyclictest14446-21grep21:37:001
21028998332,43cyclictest0-21swapper/123:02:021
21028998330,46cyclictest0-21swapper/119:56:571
21028998330,46cyclictest0-21swapper/119:56:571
2102899832,44cyclictest5397-21ssh21:21:551
21024998349,22cyclictest14202-21apt-get21:36:530
21024998331,43cyclictest20852-21/usr/sbin/munin21:47:130
21024998322,23cyclictest31227-21ssh23:00:020
21034998243,31cyclictest18409-21sh00:29:132
21034998241,34cyclictest24637-21sh19:16:382
2103499823,7cyclictest0-21swapper/223:21:572
2103499823,7cyclictest0-21swapper/223:21:572
2103499823,67cyclictest29150-21latency_hist23:51:402
2103499823,34cyclictest30175-21tail22:57:172
2102899824,36cyclictest8507-21/usr/sbin/munin22:22:141
2102899824,34cyclictest356-21runrttasks20:09:531
2102899824,34cyclictest356-21runrttasks20:09:531
2102899824,30cyclictest23117-21unixbench_multi20:47:131
21028998242,34cyclictest24231-21sh22:47:161
21028998236,37cyclictest0-21swapper/120:58:231
21028998236,37cyclictest0-21swapper/120:58:231
2102899823,41cyclictest14618-21ssh22:32:011
2102899823,38cyclictest7812-21grep20:02:041
2102899823,38cyclictest7812-21grep20:02:041
2102899823,38cyclictest19234-21fschecks_time20:37:271
2102899823,37cyclictest11800-21sed21:32:041
2102899823,37cyclictest11800-21sed21:32:041
2102899823,36cyclictest356-21runrttasks22:27:141
2102899823,36cyclictest25223-21ssh21:55:281
2102899823,36cyclictest1-21systemd19:41:381
2102899823,27cyclictest17717-21fschecks_time22:37:061
2102899823,27cyclictest17717-21fschecks_time22:37:061
2102899823,25cyclictest20114-21ls23:36:391
2102899823,25cyclictest20114-21ls23:36:391
2102899823,23cyclictest26344-21cstates22:51:571
2102899823,23cyclictest26344-21cstates22:51:571
21024998257,16cyclictest11524-21apt-get23:21:570
21024998257,16cyclictest11524-21apt-get23:21:570
2103499816,66cyclictest0-21swapper/200:17:412
2103499814,67cyclictest11234-21apt-get21:31:482
2103499814,67cyclictest11234-21apt-get21:31:482
21034998136,34cyclictest0-21swapper/220:46:502
2103499813,37cyclictest28940-21sh22:01:512
2103499812,35cyclictest9035-21memory22:22:082
2102899814,36cyclictest8400-21latency_hist22:21:401
2102899813,39cyclictest3195-21mailstats21:17:131
2102899813,39cyclictest30024-21timerandwakeup22:57:151
2102899813,37cyclictest758-21unixbench_multi23:57:161
2102899813,30cyclictest356-21runrttasks20:16:591
2102899813,28cyclictest28609-21taskset21:06:391
2102899813,28cyclictest28609-21taskset21:06:391
2102899813,28cyclictest2057-21apt-get22:11:371
2102899812,26cyclictest20807-21fschecks_time20:42:001
21028998115,29cyclictest26685-21fschecks_count19:22:011
21024998151,21cyclictest8543-21sh21:27:000
21024998151,20cyclictest22192-21apt-get20:46:480
21024998129,22cyclictest28646-21unin-run22:01:380
21024998129,22cyclictest28646-21unin-run22:01:380
21024998121,21cyclictest29328-21ssh22:02:030
21034998064,9cyclictest4958-21kworker/u17:3-rpciod00:06:402
2103499804,64cyclictest0-21swapper/219:56:532
2103499804,64cyclictest0-21swapper/219:56:532
21034998040,33cyclictest0-21swapper/222:47:022
21034998035,38cyclictest0-21swapper/223:17:132
2103499803,36cyclictest2623-21apt22:11:552
2103499802,37cyclictest20576-21/usr/sbin/munin23:36:592
2103499802,33cyclictest5661-21cstates00:06:592
2103499802,33cyclictest356-21runrttasks22:21:102
2102899804,37cyclictest17165-21cpu21:41:531
2102899803,35cyclictest14826-21idleruntime23:27:051
2102899803,31cyclictest26448-21cstates23:46:531
2102899803,24cyclictest29121-21apt-get21:07:161
2102899803,24cyclictest29121-21apt-get21:07:161
21024998055,16cyclictest28403-21diskmemload23:57:450
2104199794,47cyclictest0-21swapper/322:11:543
2103499794,69cyclictest0-21swapper/220:56:492
2103499794,69cyclictest0-21swapper/220:56:492
2103499792,66cyclictest1785-21ssh22:10:552
2103499792,37cyclictest20078-21taskset21:46:492
2103499792,33cyclictest3139-21netstat23:07:052
2103499792,33cyclictest16621-21seq23:30:192
2102899793,37cyclictest2947-21fschecks_time22:12:141
2102899793,30cyclictest28186-21cstates19:26:561
2102899793,27cyclictest5972-21cut23:12:031
2102899793,27cyclictest11526-21grep20:12:131
2102899793,26cyclictest25966-21/usr/sbin/munin21:56:551
2102899793,26cyclictest25966-21/usr/sbin/munin21:56:551
2102899793,25cyclictest5861-21taskset00:07:041
2102899793,24cyclictest21140-21mailstats23:37:141
21028997927,44cyclictest0-21swapper/123:06:481
2102899792,28cyclictest4071-21latency_hist19:51:401
2102899792,28cyclictest4071-21latency_hist19:51:401
21028997912,25cyclictest1352-21uname19:42:231
21028997912,25cyclictest1352-21uname19:42:231
21024997954,19cyclictest4032-21latency_hist19:51:390
21024997954,19cyclictest4032-21latency_hist19:51:390
21024997954,15cyclictest12282-21apt-get20:16:530
21024997941,25cyclictest28085-21apt-get19:26:500
21024997927,22cyclictest4073-21rm21:18:510
21024997926,25cyclictest13492-21ssh22:29:570
21024997926,25cyclictest13492-21ssh22:29:570
21024997924,25cyclictest10085-21rm22:23:370
21024997923,23cyclictest0-21swapper/023:50:490
2103499783,34cyclictest21017-21cat20:42:042
2103499783,34cyclictest17978-21ntp_kernel_pll_22:37:122
2103499783,34cyclictest17978-21ntp_kernel_pll_22:37:122
2103499783,29cyclictest18967-21apt-get20:37:022
2103499782,7cyclictest0-21swapper/221:56:512
2103499782,7cyclictest0-21swapper/221:56:512
2102899786,12cyclictest0-21swapper/100:22:031
2102899786,12cyclictest0-21swapper/100:22:031
2102899783,30cyclictest15903-21latency20:27:041
2102899783,30cyclictest15903-21latency20:27:041
21024997863,10cyclictest21015-21cyclictest23:40:220
21024997841,25cyclictest19487-21apt-get00:31:540
21024997831,21cyclictest17396-21apt-get22:36:540
21024997831,21cyclictest17396-21apt-get22:36:540
21024997827,43cyclictest24981-21chrt23:43:490
21024997825,28cyclictest17066-21cron23:31:360
21024997823,28cyclictest25549-21rm21:56:240
2010827827,30sleep30-21swapper/319:02:213
2103499772,37cyclictest29373-21cpuspeed23:51:582
2103499772,37cyclictest29373-21cpuspeed23:51:582
2103499772,35cyclictest25287-21latency_hist20:56:402
2102899772,31cyclictest14094-21fschecks_count20:22:011
21024997727,18cyclictest31562-21rm23:55:490
21024997727,18cyclictest31562-21rm23:55:490
2103499763,23cyclictest26134-21runrttasks19:20:502
2103499762,33cyclictest8398-21cpuspeed21:26:552
2104199754,39cyclictest0-21swapper/321:07:013
2104199754,39cyclictest0-21swapper/321:07:013
21024997559,8cyclictest25437-21apt-get20:56:490
21024997559,8cyclictest25437-21apt-get20:56:490
21024997547,20cyclictest7361-21latency_hist20:01:400
21024997547,20cyclictest7361-21latency_hist20:01:400
21024997540,23cyclictest17104-21apt-get20:31:560
21024997540,23cyclictest17104-21apt-get20:31:560
21024997533,32cyclictest10799-21apt-get00:16:510
21024997522,25cyclictest3888-21ssh23:07:470
21024997456,10cyclictest13892-21apt-get20:21:510
21024997454,10cyclictest5540-21apt-get23:11:500
21024997442,21cyclictest23804-21apt-get20:51:490
21024997347,18cyclictest8548-21ssh23:16:510
21024997345,19cyclictest24872-21apt-get19:16:540
21024997343,21cyclictest7429-21apt-get20:01:500
21024997343,21cyclictest7429-21apt-get20:01:500
21024997249,13cyclictest2632-21ssh00:01:420
21024997243,19cyclictest26480-21apt-get19:21:500
21024997240,19cyclictest5542-21apt-get00:06:570
21024997232,30cyclictest23213-21/usr/sbin/munin19:11:590
2104199713,59cyclictest11383-21chrt00:16:523
21024997127,29cyclictest21126-21ntp_states20:42:090
21024997041,19cyclictest21195-21apt-get19:06:540
21024997039,21cyclictest32123-21apt-get23:01:580
21024997035,27cyclictest5772-21/usr/sbin/munin21:22:050
21024996950,9cyclictest15495-21apt-get20:26:530
21024996950,9cyclictest15495-21apt-get20:26:530
21024996638,19cyclictest4181-21apt-get19:51:550
21024996638,19cyclictest4181-21apt-get19:51:550
2104199644,43cyclictest23247-21taskset22:46:503
2104199644,41cyclictest11341-21taskset22:26:403
21041996312,39cyclictest0-21swapper/319:37:023
21024996336,19cyclictest0-21swapper/020:12:030
194282630,6sleep30-21swapper/321:45:433
2104199595,38cyclictest5119-21chrt19:52:153
2104199595,38cyclictest5119-21chrt19:52:153
2104199594,37cyclictest18516-21taskset00:29:333
2104199584,30cyclictest0-21swapper/320:37:023
2104199583,24cyclictest0-21swapper/319:41:523
2104199583,24cyclictest0-21swapper/319:41:523
2104199574,36cyclictest6209-21chrt19:57:053
2104199574,36cyclictest6209-21chrt19:57:053
2104199563,41cyclictest29525-21taskset22:57:033
2104199563,41cyclictest10940-21chrt21:31:373
2104199562,42cyclictest16098-21taskset22:33:573
2104199554,35cyclictest20699-21chrt23:37:033
2104199553,38cyclictest31311-21chrt19:36:403
2104199553,37cyclictest15067-21taskset23:27:103
2104199544,38cyclictest29245-21chrt22:02:003
2104199544,37cyclictest24741-21taskset23:43:073
2104199544,33cyclictest10779-21chrt20:11:553
2104199543,38cyclictest390-21chrt21:12:163
2104199543,38cyclictest28336-21chrt19:27:013
2104199543,36cyclictest20246-21taskset22:41:433
2104199543,35cyclictest6585-21taskset21:22:373
2104199543,35cyclictest26281-21chrt20:57:133
2104199543,35cyclictest26281-21chrt20:57:133
2104199542,39cyclictest31648-21taskset23:56:053
2104199542,39cyclictest31648-21taskset23:56:053
2104199534,35cyclictest18492-21chrt22:37:223
2104199534,35cyclictest18492-21chrt22:37:223
2104199533,39cyclictest3768-21chrt00:02:173
2104199533,36cyclictest28178-21chrt22:00:263
2104199533,36cyclictest28178-21chrt22:00:263
2104199533,35cyclictest2521-21chrt00:01:403
2104199533,32cyclictest28751-21taskset21:06:413
2104199533,32cyclictest28751-21taskset21:06:413
21041995332,15cyclictest451rcuc/320:06:543
21041995332,15cyclictest451rcuc/320:06:543
2104199532,32cyclictest23229-21taskset19:11:503
2104199523,39cyclictest900-21kworker/3:1+events00:36:233
2104199523,39cyclictest900-21kworker/3:1+events00:11:143
2104199523,36cyclictest14122-21chrt22:31:383
2104199523,36cyclictest14122-21chrt22:31:383
2104199522,40cyclictest8514-21chrt23:16:493
2104199522,37cyclictest32317-21taskset23:02:023
2104199514,36cyclictest21766-21chrt21:48:593
2104199514,34cyclictest14822-21chrt20:22:153
2104199513,36cyclictest3871-21taskset23:07:453
2104199513,36cyclictest3048-21chrt19:47:053
2104199513,36cyclictest3048-21chrt19:47:053
2104199513,36cyclictest14870-21chrt00:22:153
2104199513,36cyclictest14870-21chrt00:22:153
2104199513,36cyclictest11081-21chrt00:16:403
2104199513,35cyclictest12838-21chrt23:22:573
2104199513,35cyclictest12838-21chrt23:22:573
2104199513,34cyclictest5969-21chrt23:12:033
2104199513,34cyclictest17855-21chrt20:32:153
2104199513,34cyclictest17855-21chrt20:32:153
21041995112,22cyclictest0-21swapper/321:51:563
2104199503,35cyclictest19722-21chrt23:35:453
2104199503,35cyclictest19722-21chrt23:35:453
2104199503,33cyclictest8047-21chrt20:02:113
2104199503,33cyclictest8047-21chrt20:02:113
2104199503,33cyclictest15769-21chrt20:27:013
2104199503,33cyclictest15769-21chrt20:27:013
2104199493,35cyclictest26812-21chrt19:22:033
2104199493,34cyclictest12617-21chrt21:33:033
2104199493,34cyclictest12617-21chrt21:33:033
2104199493,33cyclictest15565-21chrt21:38:013
2104199492,38cyclictest26913-21taskset22:52:133
2104199492,38cyclictest26913-21taskset22:52:133
2104199492,37cyclictest12601-21chrt20:17:013
2104199492,35cyclictest5170-21chrt21:21:403
2104199483,30cyclictest26350-21chrt23:46:523
2104199474,31cyclictest24783-21taskset20:52:153
2104199473,31cyclictest674-21chrt22:07:433
2104199463,31cyclictest5944-21chrt22:17:033
2104199443,29cyclictest21818-21chrt20:45:233
2104199443,27cyclictest25651-21chrt19:17:133
2104199412,30cyclictest900-21kworker/3:1+events20:51:403
2104199372,25cyclictest21337-21chrt19:06:573
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional