You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 21:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Nov 19, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7119211115,61sleep07136-21munin-run07:09:360
210421110,6sleep10-21swapper/109:45:171
210421110,6sleep10-21swapper/109:45:171
7290991083,77cyclictest3548-21apt-get08:34:471
7290991053,81cyclictest8954-21apt-get07:14:491
72859910376,21cyclictest0-21swapper/012:29:570
72989910283,14cyclictest7276-21cyclictest07:59:332
729099983,85cyclictest26046-21apt-get12:29:471
729099982,80cyclictest7608-21apt-get07:09:591
728599979,75cyclictest19657-21kworker/u18:1+ext4-rsv-conversion07:09:500
7298999437,50cyclictest292-21dbus-daemon08:09:342
7298999349,37cyclictest17705-21systemctl07:39:332
729899934,40cyclictest15412-21grep12:09:552
729899933,34cyclictest23242-21ssh10:25:022
7290999374,14cyclictest22222-21kworker/u19:0+events_unbound09:29:321
729099933,69cyclictest19577-21apt-get10:19:471
729099933,69cyclictest19577-21apt-get10:19:471
7285999354,10cyclictest181rcu_preempt11:17:380
7285999354,10cyclictest181rcu_preempt11:17:380
729899924,49cyclictest17433-21fschecks_time10:14:562
729899924,49cyclictest17433-21fschecks_time10:14:562
729899923,43cyclictest29390-21ssh11:35:082
7298999215,32cyclictest11822-21sh10:04:452
7290999272,14cyclictest17674-21latency_hist09:14:371
7290999129,44cyclictest0-21swapper/110:45:001
729899905,29cyclictest25275-21runrttasks11:28:302
729899903,37cyclictest4220-21/usr/sbin/munin09:50:022
729899903,34cyclictest355-21runrttasks11:45:052
7285999076,8cyclictest0-21swapper/007:14:490
7298998950,31cyclictest31376-21ntp_offset09:40:042
729899894,39cyclictest9856-21taskset07:14:512
729899894,39cyclictest10439-21/usr/sbin/munin11:00:092
729899893,37cyclictest1573-21ssh10:44:552
729899893,34cyclictest20362-21ssh10:20:282
729899893,34cyclictest20362-21ssh10:20:282
729899893,26cyclictest11586-21latency_hist08:59:372
729899884,12cyclictest0-21swapper/211:04:492
729899883,35cyclictest30465-21latency_hist09:39:362
7290998827,26cyclictest14610-21diskmemload11:27:111
703728811,54sleep27085-21http07:09:342
7298998740,40cyclictest0-21swapper/210:29:552
729899873,41cyclictest7455-21grep09:55:122
729899873,39cyclictest9053-21dpkg07:14:352
729899873,33cyclictest355-21runrttasks09:08:242
7298998732,50cyclictest0-21swapper/207:29:332
729899872,22cyclictest11655-21apt-get08:59:482
729899872,22cyclictest11655-21apt-get08:59:482
7290998729,19cyclictest6578-21seq10:54:311
7285998758,9cyclictest181rcu_preempt09:44:570
7285998758,9cyclictest181rcu_preempt09:44:570
655828717,35sleep10-21swapper/107:05:261
7298998642,39cyclictest7276-21cyclictest08:35:472
729899863,38cyclictest28418-21latency_hist11:34:352
729899863,35cyclictest355-21runrttasks07:39:482
729899863,33cyclictest29275-21/usr/sbin/munin12:35:022
7290998654,21cyclictest31904-21sh12:39:361
7285998664,16cyclictest207-21systemd-journal09:13:300
7285998658,20cyclictest32402-21latency_hist08:24:350
7285998658,20cyclictest32402-21latency_hist08:24:350
7285998626,36cyclictest15-21ksoftirqd/007:44:360
729899855,33cyclictest355-21runrttasks08:23:312
729899855,33cyclictest355-21runrttasks08:23:312
729899854,35cyclictest9108-21/usr/sbin/munin10:00:012
729899854,35cyclictest9108-21/usr/sbin/munin10:00:012
729899854,35cyclictest15023-21ntp_kernel_pll_10:10:032
729899853,41cyclictest25027-21cron09:29:322
7285998551,12cyclictest181rcu_preempt10:39:350
7285998551,11cyclictest181rcu_preempt10:41:350
7285998551,11cyclictest181rcu_preempt10:41:350
729899845,37cyclictest18031-21idleruntime09:14:572
729899844,29cyclictest20896-21/usr/sbin/munin12:20:082
7298998437,39cyclictest0-21swapper/208:00:092
729899843,40cyclictest4260-21taskset11:49:522
729899843,40cyclictest10395-21ssh12:00:032
729899843,40cyclictest10395-21ssh12:00:032
729899843,38cyclictest20478-21/usr/sbin/munin11:20:102
729899843,27cyclictest32739-21grep08:24:552
729899843,27cyclictest32739-21grep08:24:552
729899843,26cyclictest22012-21cat07:50:092
729899843,26cyclictest22012-21cat07:50:092
729099843,62cyclictest20917-21apt-get12:20:031
7290998426,22cyclictest355-21runrttasks09:55:331
7285998458,18cyclictest14610-21diskmemload11:50:000
7285998451,11cyclictest181rcu_preempt08:50:220
708728415,41sleep30-21swapper/307:09:353
729899835,35cyclictest355-21runrttasks10:41:482
729899835,35cyclictest355-21runrttasks10:41:482
729899835,28cyclictest14945-21sort07:29:572
729899834,38cyclictest8388-21/usr/sbin/munin08:49:582
729899834,37cyclictest18124-21ssh12:14:562
729899834,35cyclictest31555-21taskset11:39:572
729899834,35cyclictest31555-21taskset11:39:572
729899833,36cyclictest355-21runrttasks10:56:302
729899833,25cyclictest4761-21ssh10:50:062
7285998358,9cyclictest181rcu_preempt09:50:090
7285998349,9cyclictest19621-21ssh10:19:330
7285998349,9cyclictest19621-21ssh10:19:330
729899824,39cyclictest18545-21sendmail_mailqu11:15:082
729899824,39cyclictest18545-21sendmail_mailqu11:15:082
729899824,36cyclictest15894-21ls09:10:112
729899824,33cyclictest25337-21/usr/sbin/munin09:30:082
729899824,31cyclictest355-21runrttasks11:09:582
729899823,38cyclictest23892-21fschecks_time12:24:572
729899823,31cyclictest29494-21fschecks_count08:14:572
729899822,37cyclictest28206-21/usr/sbin/munin10:34:482
729899822,34cyclictest8142-21latency_hist08:49:362
729899822,26cyclictest28122-21taskset12:31:412
7290998226,22cyclictest32174-21ssh09:41:101
7285998264,14cyclictest7276-21cyclictest12:04:040
7285998264,14cyclictest7276-21cyclictest12:04:040
7285998236,26cyclictest181rcu_preempt10:46:420
728599822,53cyclictest15-21ksoftirqd/010:24:120
728599822,53cyclictest15-21ksoftirqd/010:24:120
728599822,33cyclictest15-21ksoftirqd/010:00:310
728599822,33cyclictest15-21ksoftirqd/010:00:310
729899815,34cyclictest12474-21/usr/sbin/munin12:05:092
729899813,40cyclictest2123-21wc08:30:012
729899813,35cyclictest28527-21timerwakeupswit08:10:092
729899812,39cyclictest11546-21cpu07:19:512
7290998123,25cyclictest15299-21timerandwakeup10:10:101
7290998121,38cyclictest0-21swapper/107:58:401
7285998147,11cyclictest181rcu_preempt10:56:400
728599812,39cyclictest15-21ksoftirqd/009:37:250
729899806,17cyclictest1076-21apt-get09:44:502
729899806,17cyclictest1076-21apt-get09:44:502
729899805,31cyclictest355-21runrttasks07:47:022
729899805,31cyclictest355-21runrttasks07:47:022
729899803,24cyclictest6531-21idleruntime-cro08:44:332
7298998028,44cyclictest0-21swapper/211:54:502
729899802,24cyclictest20150-21/usr/sbin/munin09:20:092
729899802,24cyclictest20150-21/usr/sbin/munin09:20:092
7285998055,19cyclictest16794-21kworker/u19:1+events_unbound09:24:320
7285998055,19cyclictest16794-21kworker/u19:1+events_unbound09:24:320
7285998052,21cyclictest16318-21kworker/u18:3+events_unbound08:19:330
7285998042,16cyclictest181rcu_preempt07:44:580
7285998042,16cyclictest181rcu_preempt07:44:580
7285998034,17cyclictest181rcu_preempt11:59:230
7290997928,21cyclictest5058-21ssh09:51:001
7290997921,24cyclictest0-21swapper/112:13:271
7290997916,23cyclictest14610-21diskmemload12:19:101
7285997953,11cyclictest181rcu_preempt08:31:190
7285997945,11cyclictest181rcu_preempt11:44:110
7285997945,11cyclictest181rcu_preempt11:44:110
7285997938,17cyclictest181rcu_preempt11:29:560
7290997856,16cyclictest691-21cron08:09:331
7285997861,10cyclictest21120-21ssh11:20:080
7285997841,14cyclictest181rcu_preempt12:24:320
728599783,58cyclictest3748-21runrttasks08:35:230
7290997755,16cyclictest355-21runrttasks10:15:011
7290997755,16cyclictest355-21runrttasks10:15:011
7290997745,22cyclictest17979-21apt-get07:39:451
7290997744,12cyclictest181rcu_preempt08:04:361
7290997721,25cyclictest14610-21diskmemload12:26:191
728599775,39cyclictest15-21ksoftirqd/009:32:230
7285997746,8cyclictest181rcu_preempt09:26:070
728599773,41cyclictest15-21ksoftirqd/011:34:550
7285997715,32cyclictest161ktimers/007:50:080
7285997715,32cyclictest161ktimers/007:50:080
7290997654,16cyclictest12448-21unixbench_multi07:20:111
7290997641,22cyclictest1709-21apt-get08:29:511
729099763,44cyclictest28399-21ssh10:34:551
7290997631,38cyclictest355-21runrttasks10:05:091
7290997628,18cyclictest6124-21ssh11:52:501
7290997626,17cyclictest355-21runrttasks10:00:031
7290997626,17cyclictest355-21runrttasks10:00:031
7290997625,20cyclictest22369-21taskset07:52:391
7290997625,20cyclictest22369-21taskset07:52:391
7290997616,25cyclictest0-21swapper/108:15:111
7290997616,24cyclictest0-21swapper/111:32:231
7285997650,17cyclictest7708-21unixbench_multi08:45:090
728599764,59cyclictest355-21runrttasks09:40:280
7285997634,13cyclictest181rcu_preempt12:14:040
7285997633,26cyclictest15-21ksoftirqd/010:10:000
7290997541,25cyclictest28073-21runrttasks09:34:461
7290997521,19cyclictest28964-21runrttasks08:13:511
7290997514,36cyclictest0-21swapper/111:46:101
7285997534,11cyclictest181rcu_preempt09:57:370
50212752,27sleep05142-21tail10:50:110
7290997453,15cyclictest13620-21kernelversion09:04:591
7290997420,25cyclictest9436-21runrttasks08:50:401
7290997420,22cyclictest0-21swapper/107:27:211
7290997420,20cyclictest6213-21runrttasks08:40:401
7290997419,23cyclictest31039-21ssh10:39:481
7290997419,23cyclictest31039-21ssh10:39:481
7290997417,24cyclictest0-21swapper/109:14:511
7285997435,8cyclictest181rcu_preempt10:28:390
7285997433,13cyclictest0-21swapper/007:25:230
7290997351,16cyclictest26347-21switchtime10:30:091
7290997350,17cyclictest15655-21latency11:10:011
7290997337,26cyclictest6911-21apt-get10:54:461
7290997325,19cyclictest31945-21runrttasks08:20:251
7290997325,19cyclictest31945-21runrttasks08:20:251
729099732,36cyclictest20947-21latency11:20:011
7290997319,22cyclictest15761-21runrttasks07:30:331
7285997348,9cyclictest181rcu_preempt09:15:020
7285997346,10cyclictest181rcu_preempt12:09:140
728599733,55cyclictest26046-21apt12:29:320
728599733,43cyclictest15-21ksoftirqd/009:00:020
728599733,43cyclictest15-21ksoftirqd/009:00:020
7285997333,18cyclictest181rcu_preempt08:39:550
7285997332,28cyclictest1440-21/usr/sbin/munin11:45:030
7290997257,8cyclictest0-21swapper/109:04:321
7290997257,8cyclictest0-21swapper/109:04:321
7290997239,21cyclictest32549-21apt-get08:24:451
7290997239,21cyclictest32549-21apt-get08:24:451
7290997237,24cyclictest28606-21apt-get11:34:451
7290997235,26cyclictest7669-21/usr/sbin/munin11:55:071
7290997226,24cyclictest18409-21sh11:15:031
7290997226,24cyclictest18409-21sh11:15:031
7290997226,18cyclictest20732-21taskset07:47:161
7290997226,18cyclictest20732-21taskset07:47:161
7290997219,12cyclictest0-21swapper/110:29:331
7285997234,23cyclictest181rcu_preempt11:05:040
7285997231,9cyclictest181rcu_preempt12:17:270
7285997230,34cyclictest31840-21ssh12:39:350
7285997213,17cyclictest15-21ksoftirqd/011:27:230
7290997154,12cyclictest7276-21cyclictest07:35:121
7290997150,14cyclictest14610-21diskmemload11:05:141
7290997117,26cyclictest0-21swapper/111:01:001
7285997129,10cyclictest181rcu_preempt10:29:410
7285997112,11cyclictest15-21ksoftirqd/008:00:310
262962719,52sleep00-21swapper/008:04:570
7290997042,21cyclictest7110-21if_err_eth008:44:571
7290997031,30cyclictest22676-21latency_hist09:24:361
7290997017,23cyclictest0-21swapper/112:03:211
7290997017,23cyclictest0-21swapper/112:03:211
7290997017,23cyclictest0-21swapper/111:40:441
7290997017,23cyclictest0-21swapper/111:40:441
7285997048,7cyclictest181rcu_preempt11:00:080
7285997039,8cyclictest181rcu_preempt07:19:560
7285997037,13cyclictest181rcu_preempt08:12:450
7285997036,12cyclictest181rcu_preempt11:13:440
7285997035,13cyclictest181rcu_preempt08:54:440
728599703,39cyclictest16242-21cat07:34:360
7290996931,28cyclictest14906-21idleruntime-cro12:09:321
7290996922,18cyclictest22419-21sh09:24:291
7285996947,8cyclictest181rcu_preempt10:08:440
7285996926,7cyclictest0-21swapper/009:04:530
7290996823,22cyclictest355-21runrttasks08:56:191
7285996816,15cyclictest161ktimers/007:54:530
7290996722,21cyclictest26679-21ssh09:31:021
728599673,46cyclictest32529-21/usr/sbin/munin08:24:560
728599673,46cyclictest32529-21/usr/sbin/munin08:24:560
728599673,30cyclictest15-21ksoftirqd/007:37:240
7311996450,10cyclictest451rcuc/312:02:543
7311996450,10cyclictest451rcuc/312:02:543
7311996424,31cyclictest0-21swapper/310:19:523
7311996424,31cyclictest0-21swapper/310:19:523
7311996218,23cyclictest0-21swapper/307:14:493
731199614,44cyclictest3826-21taskset10:49:323
7311995915,36cyclictest0-21swapper/312:19:483
7311995846,8cyclictest451rcuc/308:24:323
7311995846,8cyclictest451rcuc/308:24:323
7311995845,9cyclictest451rcuc/310:00:233
7311995845,9cyclictest451rcuc/310:00:233
7311995845,9cyclictest451rcuc/309:13:323
7311995644,8cyclictest451rcuc/307:52:573
7311995644,8cyclictest451rcuc/307:52:573
731199563,42cyclictest0-21swapper/312:30:013
731199562,37cyclictest0-21swapper/307:09:503
731199553,40cyclictest1925-21chrt08:29:583
731199553,35cyclictest0-21swapper/308:35:193
731199542,38cyclictest6209-21taskset09:54:353
731199534,34cyclictest18244-21chrt07:39:563
7311995340,9cyclictest451rcuc/307:26:043
7311995334,12cyclictest451rcuc/309:30:343
7311995239,9cyclictest451rcuc/311:58:283
7311995239,9cyclictest451rcuc/311:11:163
7311995239,9cyclictest451rcuc/309:05:593
7311995239,9cyclictest451rcuc/308:40:363
7311995237,10cyclictest451rcuc/311:27:333
731199523,36cyclictest18249-21taskset12:15:003
731199522,38cyclictest18061-21taskset11:14:563
731199522,38cyclictest18061-21taskset11:14:563
731199522,37cyclictest9076-21chrt08:50:063
7311995138,9cyclictest451rcuc/309:00:363
7311995138,9cyclictest451rcuc/309:00:363
731199513,39cyclictest32283-21chrt09:41:303
7311995037,9cyclictest451rcuc/312:25:273
7311995037,9cyclictest451rcuc/307:21:593
731199503,37cyclictest24460-21taskset10:27:003
731199503,33cyclictest27116-21chrt10:31:543
731199502,37cyclictest3188-21chrt09:48:523
731199502,37cyclictest3188-21chrt09:48:523
7311994937,8cyclictest451rcuc/307:47:553
7311994937,8cyclictest451rcuc/307:47:553
7311994937,8cyclictest451rcuc/307:34:303
7311994936,9cyclictest451rcuc/308:00:433
731199493,34cyclictest23717-21chrt09:25:103
7311994836,8cyclictest451rcuc/311:37:013
7311994834,9cyclictest451rcuc/310:44:363
7311994834,9cyclictest451rcuc/310:44:363
731199483,35cyclictest19706-21taskset10:19:343
731199483,35cyclictest19706-21taskset10:19:343
731199483,33cyclictest30680-21chrt09:39:363
731199483,33cyclictest22293-21chrt09:24:003
731199483,33cyclictest22293-21chrt09:24:003
731199483,32cyclictest19341-21chrt09:17:403
731199483,32cyclictest10667-21chrt08:55:093
731199482,34cyclictest1445-21taskset11:44:463
731199473,29cyclictest9088-21chrt10:58:353
731199472,32cyclictest30874-21chrt08:19:373
7311994633,9cyclictest451rcuc/310:35:523
731199463,33cyclictest5572-21chrt11:51:103
731199463,28cyclictest17779-21chrt07:39:363
731199462,32cyclictest23402-21taskset07:55:073
64392462,6sleep30-21swapper/310:54:033
731199453,33cyclictest22441-21taskset11:23:153
731199453,32cyclictest30744-21chrt12:36:283
731199453,32cyclictest12256-21chrt11:04:353
731199453,29cyclictest12668-21taskset12:04:513
731199443,33cyclictest11986-21taskset10:04:523
731199443,32cyclictest6778-21chrt08:44:443
731199443,32cyclictest15773-21taskset10:11:033
731199443,30cyclictest28535-21chrt08:10:093
7311994431,8cyclictest451rcuc/308:27:233
7311994431,8cyclictest451rcuc/308:27:233
731199442,31cyclictest27636-21taskset08:09:363
731199433,29cyclictest7442-21taskset09:55:123
731199423,31cyclictest824-21chrt11:43:323
731199423,31cyclictest824-21chrt11:43:323
731199422,30cyclictest26185-21taskset11:29:553
731199422,29cyclictest17668-21chrt12:14:353
731199413,30cyclictest12767-21taskset11:05:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional