You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 04:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Jul 01, 2025 00:44:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19852215614,120sleep019900-21cstates19:02:020
2116221289,79sleep221166-21latencyplot19:06:452
2100021200,6sleep00-21swapper/022:47:150
21228991113,23cyclictest32291-21/usr/sbin/munin00:06:570
212289910956,48cyclictest21219-21cyclictest23:27:150
212289910956,48cyclictest21219-21cyclictest23:27:150
212289910753,46cyclictest0-21swapper/022:02:490
212289910753,46cyclictest0-21swapper/022:02:490
534621060,6sleep05340-21ntp_states21:22:130
212289910650,50cyclictest3883-21ssh22:17:090
212289910646,53cyclictest10041-21apt-key21:31:430
212289910644,53cyclictest31371-21latency_hist21:11:450
212289910643,55cyclictest0-21swapper/021:47:170
212289910643,55cyclictest0-21swapper/021:47:170
20354210667,22sleep120402-21grep19:02:241
21228991053,27cyclictest15876-21sh22:37:280
212289910347,48cyclictest0-21swapper/023:26:440
21228991033,41cyclictest9166-21apt-get22:26:550
21228991033,41cyclictest9166-21apt-get22:26:550
21228991023,27cyclictest26376-21ls22:57:060
21228991023,27cyclictest26376-21ls22:57:060
21228991023,23cyclictest17327-21apt-get20:31:560
21228991023,23cyclictest17327-21apt-get20:31:560
21228991023,23cyclictest12330-21if_eth022:32:090
21228991022,30cyclictest24948-21/usr/sbin/munin21:57:080
21228991022,30cyclictest24948-21/usr/sbin/munin21:57:080
21228991013,27cyclictest356-21runrttasks23:36:540
21228991013,27cyclictest356-21runrttasks23:36:540
21228991012,28cyclictest19422-21netstat23:42:120
212289910047,46cyclictest0-21swapper/023:21:440
212289910047,46cyclictest0-21swapper/023:21:440
212289910044,47cyclictest28845-21ssh23:01:580
21228991002,21cyclictest7215-21sort22:22:210
21228991002,20cyclictest27794-21runrttasks23:57:060
21228999945,47cyclictest0-21swapper/021:52:200
21228999916,27cyclictest18424-21seq21:46:170
21228999836,57cyclictest0-21swapper/023:12:140
2122899982,19cyclictest30229-21acpi22:06:570
21249999756,32cyclictest1183-21kworker/u17:3-xprtiod21:17:182
21228999744,45cyclictest0-21swapper/020:01:520
21228999741,49cyclictest29986-21cpu19:32:040
21228999740,50cyclictest13379-21diskmemload21:36:540
2122899973,26cyclictest8023-21taskset00:21:150
2122899973,26cyclictest8023-21taskset00:21:150
2122899973,25cyclictest15826-21grep23:36:420
2122899972,31cyclictest22071-21latency_hist20:46:440
2122899972,31cyclictest22071-21latency_hist20:46:440
21249999650,38cyclictest376-21in:imuxsock23:00:142
21249999650,38cyclictest376-21in:imuxsock23:00:142
21228999641,48cyclictest0-21swapper/019:52:180
2122899963,28cyclictest765-21/usr/sbin/munin22:12:140
2122899963,24cyclictest32649-21grep23:07:200
2122899962,26cyclictest2016-21cpu21:17:010
21228999544,43cyclictest0-21swapper/000:12:140
2122899952,31cyclictest28602-21latency_hist21:06:450
2122899952,29cyclictest16822-21latency_hist20:31:410
2122899952,29cyclictest16822-21latency_hist20:31:410
2122899952,28cyclictest25247-21awk23:52:160
2122899952,28cyclictest25247-21awk23:52:160
21228999443,44cyclictest0-21swapper/019:31:440
2122899942,28cyclictest914-21fschecks_count19:42:030
2122899936,28cyclictest14850-21cat00:32:330
2122899934,27cyclictest356-21runrttasks22:52:010
2122899933,33cyclictest10926-21apt-get00:26:550
2122899933,33cyclictest10926-21apt-get00:26:550
2122899933,29cyclictest21408-21/usr/sbin/munin19:07:040
2122899933,22cyclictest31924-21/usr/sbin/munin21:12:070
2122899932,29cyclictest437-21modprobe19:41:410
2124999923,31cyclictest6961-21latency_hist20:01:432
2124999922,31cyclictest19205-21runrttasks21:47:062
2124999922,31cyclictest19205-21runrttasks21:47:062
2122899924,29cyclictest10547-21/usr/sbin/munin21:32:440
2122899923,31cyclictest25402-21/usr/sbin/munin20:57:070
2122899923,19cyclictest23446-21/usr/sbin/munin19:12:120
2122899922,26cyclictest18590-21unixbench_multi22:42:210
21249999141,43cyclictest31323-21latency_hist23:06:442
21228999158,24cyclictest23795-21cpu20:51:580
2122899913,28cyclictest22159-21/usr/sbin/munin20:47:040
2122899913,28cyclictest22159-21/usr/sbin/munin20:47:040
2122899913,26cyclictest356-21runrttasks00:03:160
21228999132,50cyclictest0-21swapper/020:41:430
21228999132,50cyclictest0-21swapper/020:41:430
21228999114,22cyclictest8692-21seq00:22:030
2124999903,35cyclictest18241-21grep22:42:172
2124999903,29cyclictest7983-21latency21:27:092
2122899904,31cyclictest25682-21proc_pri19:17:190
2122899904,31cyclictest25682-21proc_pri19:17:190
2122899903,27cyclictest10120-21latency_hist20:11:460
2122899902,29cyclictest356-21runrttasks20:21:440
2122899902,23cyclictest6912-21idleruntime-cro20:01:410
2124999894,32cyclictest30755-21timerwakeupswit19:32:242
2124999893,30cyclictest7421-21latency_hist21:26:442
2124999892,28cyclictest10925-21latency_hist00:26:432
2122899893,22cyclictest11308-21/usr/sbin/munin20:12:340
2122899893,22cyclictest11308-21/usr/sbin/munin20:12:340
2122899893,21cyclictest26774-21cpuspeed19:21:590
2122899893,21cyclictest26774-21cpuspeed19:21:590
21228998916,22cyclictest14032-21df20:22:020
2124999883,30cyclictest24195-21cron23:51:402
21249998819,11cyclictest0-21swapper/219:51:442
21228998834,46cyclictest0-21swapper/023:47:070
2122899883,11cyclictest0-21swapper/019:49:190
2124999874,31cyclictest1739-21latency_hist23:11:432
2124999873,25cyclictest22253-21ssh21:52:172
2124999872,32cyclictest7122-21ssh22:22:202
2124999872,31cyclictest10377-21latency_hist20:11:442
21249998719,27cyclictest23042-21cat20:47:172
21249998719,27cyclictest23042-21cat20:47:172
2124999862,30cyclictest2494-21sort00:11:512
21249998621,33cyclictest8472-21sed23:22:182
21249998566,8cyclictest0-21swapper/219:16:442
2124999854,23cyclictest27144-21sed22:01:442
2124999854,23cyclictest27144-21sed22:01:442
2124999853,34cyclictest23976-21fschecks_count20:52:032
2124999853,27cyclictest11964-21cron20:16:402
2124999853,27cyclictest11964-21cron20:16:402
2124999853,20cyclictest27038-21apt-get21:02:002
2124999852,31cyclictest14189-21cat00:31:452
2124999852,31cyclictest14189-21cat00:31:452
2124999852,29cyclictest1293-21proc_pri19:42:142
21249998461,12cyclictest0-21swapper/220:21:412
2124999844,34cyclictest9538-21if_eth022:27:082
2124999844,34cyclictest9538-21if_eth022:27:082
2124999843,31cyclictest6105-21fschecks_time00:17:402
2124999843,31cyclictest6105-21fschecks_time00:17:402
2124999843,31cyclictest5773-21cat23:17:212
2124999843,31cyclictest5773-21cat23:17:212
2124999843,30cyclictest19178-21fschecks_time23:42:072
2124999843,29cyclictest16448-21grep20:27:212
2124999843,29cyclictest16448-21grep20:27:212
2124999842,30cyclictest5496-21latency_hist19:56:452
2124999833,29cyclictest18056-21ssh21:45:062
2124999824,12cyclictest24230-21/usr/sbin/munin22:52:212
2124999823,31cyclictest22070-21latency_hist20:46:442
2124999823,31cyclictest22070-21latency_hist20:46:442
2124999823,28cyclictest31015-21ssh21:11:082
2124999823,27cyclictest3872-21fschecks_time22:17:082
21249998228,47cyclictest13285-21latency_hist21:36:452
2124999822,22cyclictest14376-21apt-get00:31:512
21249998222,29cyclictest32571-21grep21:12:192
21249998218,28cyclictest2158-21sh23:12:002
21249998161,10cyclictest0-21swapper/220:36:432
21249998161,10cyclictest0-21swapper/220:36:432
21249998134,40cyclictest0-21swapper/222:37:202
2124999813,28cyclictest20619-21/usr/sbin/munin22:47:042
2124999812,31cyclictest27448-21grep23:56:582
2124999812,21cyclictest30454-21fschecks_count22:07:082
2124999804,25cyclictest13863-21/usr/sbin/munin20:21:542
21249998036,36cyclictest0-21swapper/222:12:222
2124999803,35cyclictest14289-21unixbench_multi23:32:192
2124999803,29cyclictest24539-21ssh23:51:552
2124999803,29cyclictest24539-21ssh23:51:552
21249998028,44cyclictest0-21swapper/221:37:042
21237998059,15cyclictest28055-21sh22:02:421
21237998059,15cyclictest28055-21sh22:02:421
2124999794,26cyclictest26762-21latency_hist21:01:442
2124999792,32cyclictest7550-21kernelversion20:02:062
2124999792,31cyclictest614-21timerwakeupswit00:07:152
2124999792,29cyclictest21686-21cut19:07:052
21237997963,9cyclictest4351-21mailstats22:17:201
2124999788,14cyclictest0-21swapper/222:32:092
2124999783,32cyclictest16223-21cstates23:36:592
2124999783,32cyclictest16223-21cstates23:36:592
2124999783,24cyclictest25116-21taskset19:17:002
2124999783,24cyclictest25116-21taskset19:17:002
2124999782,31cyclictest18968-21cpu20:36:582
2124999782,31cyclictest18968-21cpu20:36:582
2124999782,30cyclictest28088-21latency_hist19:26:432
2124999782,30cyclictest28088-21latency_hist19:26:432
21237997862,10cyclictest24655-21wc23:51:591
21237997862,10cyclictest24655-21wc23:51:591
21237997847,21cyclictest3149-21cat19:47:181
21249997752,13cyclictest0-21swapper/222:02:382
21249997752,13cyclictest0-21swapper/222:02:382
2124999773,29cyclictest10435-21/usr/sbin/munin23:27:112
2124999773,29cyclictest10435-21/usr/sbin/munin23:27:112
2124999772,29cyclictest28302-21/usr/sbin/munin19:27:162
2124999762,29cyclictest32292-21wc19:37:212
21255997510,51cyclictest0-21swapper/300:26:553
21255997510,51cyclictest0-21swapper/300:26:553
21237997558,11cyclictest202-21systemd-journal20:41:141
21237997558,11cyclictest202-21systemd-journal20:41:141
21237997555,15cyclictest9537-21threads20:07:181
21237997554,15cyclictest14634-21threads20:22:181
2124999749,13cyclictest0-21swapper/200:01:572
21237997461,9cyclictest21219-21cyclictest21:07:241
21237997453,14cyclictest1-21systemd20:06:431
21237997359,9cyclictest31-21ksoftirqd/100:25:371
21237997352,15cyclictest20533-21sh23:44:071
201242739,37sleep30-21swapper/319:02:143
21237997257,10cyclictest58-21kcompactd020:01:441
21237997150,14cyclictest17740-21sh22:42:021
21237997148,16cyclictest356-21runrttasks00:32:381
2125599705,40cyclictest0-21swapper/320:31:563
2125599705,40cyclictest0-21swapper/320:31:563
21255997012,33cyclictest0-21swapper/322:11:553
21237997055,10cyclictest21219-21cyclictest23:57:191
21237997049,14cyclictest1-21systemd23:28:151
21237997049,14cyclictest1-21systemd23:28:151
21237997046,9cyclictest181rcu_preempt20:46:541
21237997046,9cyclictest181rcu_preempt20:46:541
21237997041,21cyclictest17238-21modprobe22:41:401
21237997035,11cyclictest25602-21ssh22:56:281
21237996955,10cyclictest21219-21cyclictest19:29:401
21237996953,10cyclictest28385-21diskmemload23:36:591
21237996953,10cyclictest28385-21diskmemload23:36:591
21237996950,13cyclictest28385-21diskmemload22:23:571
21237996947,16cyclictest376-21in:imuxsock00:03:161
21237996947,16cyclictest22126-21sh23:47:091
2125599684,29cyclictest0-21swapper/320:11:573
2125599684,29cyclictest0-21swapper/320:11:573
21237996852,10cyclictest11510-21ssh21:32:431
21237996849,11cyclictest32267-21ntpq23:07:141
2123799684,54cyclictest13383-21ssh21:36:551
2123799684,27cyclictest32323-21grep00:06:591
21237996753,10cyclictest170-21jbd2/mmcblk2p2-822:12:331
21237996752,11cyclictest21219-21cyclictest22:00:171
21237996752,11cyclictest21219-21cyclictest22:00:171
21237996752,10cyclictest21219-21cyclictest21:47:201
21237996752,10cyclictest21219-21cyclictest21:47:201
21237996752,10cyclictest21219-21cyclictest19:42:051
21237996751,10cyclictest26745-21kworker/u17:0-rpciod21:12:101
21237996749,10cyclictest31844-21perl19:37:091
21237996748,12cyclictest331-21dbus-daemon22:36:421
21237996748,12cyclictest27020-21/usr/sbin/munin21:02:211
21237996745,16cyclictest11673-21latency_hist22:31:431
21237996745,16cyclictest11673-21latency_hist22:31:431
21237996736,8cyclictest181rcu_preempt00:31:441
21237996736,8cyclictest181rcu_preempt00:31:441
21237996733,24cyclictest23090-21grep19:11:481
21237996650,10cyclictest8983-21kworker/u17:0-nfsiod23:23:091
2123799664,45cyclictest356-21runrttasks20:15:191
2123799664,45cyclictest356-21runrttasks20:15:191
21237996548,11cyclictest28385-21diskmemload23:02:021
21237996543,15cyclictest26013-21grep19:17:231
21237996543,15cyclictest26013-21grep19:17:231
2123799653,47cyclictest18658-21latency_hist21:46:431
21237996449,8cyclictest0-21swapper/123:16:561
21237996449,8cyclictest0-21swapper/123:16:561
21237996449,8cyclictest0-21swapper/121:22:171
21237996443,15cyclictest4571-21latency_hist23:16:431
21237996434,21cyclictest24025-21grep20:52:031
21237996411,37cyclictest22276-21timerwakeupswit19:07:181
2125599638,47cyclictest0-21swapper/322:02:243
2125599638,47cyclictest0-21swapper/322:02:243
2125599634,38cyclictest0-21swapper/300:17:013
2125599634,38cyclictest0-21swapper/300:17:013
2123799634,48cyclictest5068-21apt-get00:16:541
2123799634,48cyclictest5068-21apt-get00:16:541
2123799634,45cyclictest26984-21latency_hist21:01:441
2123799634,44cyclictest16070-21processes20:27:161
2123799634,44cyclictest16070-21processes20:27:161
21237996344,10cyclictest301ktimers/120:32:191
21237996344,10cyclictest301ktimers/120:32:191
2123799634,38cyclictest356-21runrttasks21:29:411
21237996342,15cyclictest20366-21sh22:46:541
21237996341,17cyclictest21697-21kworker/u17:1+events_unbound21:56:411
2123799633,38cyclictest5101-21sendmail-msp00:16:401
2123799633,30cyclictest26695-21/usr/sbin/munin19:22:151
2123799633,30cyclictest26695-21/usr/sbin/munin19:22:151
21237996246,9cyclictest0-21swapper/120:17:071
21237996241,14cyclictest27305-21sh22:57:541
21237996241,14cyclictest27305-21sh22:57:541
21237996236,19cyclictest31575-21kworker/u18:1+nfsiod22:10:591
2123799623,44cyclictest4854-21grep19:52:191
21237996145,11cyclictest21219-21cyclictest21:19:091
2123799603,42cyclictest30411-21ntp_kernel_err19:32:161
2123799603,36cyclictest14114-21xargs23:32:171
21237995933,19cyclictest21908-21sh20:46:411
21237995933,19cyclictest21908-21sh20:46:411
2125599583,38cyclictest25370-21chrt20:56:443
21255995812,33cyclictest0-21swapper/321:31:583
2125599564,36cyclictest21905-21chrt23:47:043
21255995612,26cyclictest0-21swapper/300:31:593
2125599553,38cyclictest24399-21taskset19:12:203
2125599543,38cyclictest3442-21chrt00:12:173
2125599543,36cyclictest32721-21chrt00:07:103
2125599543,34cyclictest6910-21chrt20:01:413
2125599542,39cyclictest21714-21chrt19:07:063
2125599542,37cyclictest744-21taskset23:08:493
2125599534,36cyclictest4453-21chrt22:17:213
2125599533,38cyclictest30599-21taskset00:03:013
2125599533,33cyclictest27220-21chrt22:01:443
2125599533,33cyclictest27220-21chrt22:01:443
2125599533,33cyclictest18765-21chrt21:46:443
2125599523,38cyclictest10826-21taskset00:26:243
2125599523,36cyclictest20502-21chrt21:49:073
2125599523,36cyclictest20502-21chrt21:49:073
2125599523,33cyclictest9088-21taskset22:26:453
2125599523,33cyclictest9088-21taskset22:26:453
2125599517,30cyclictest0-21swapper/323:56:433
2125599517,30cyclictest0-21swapper/323:56:433
2125599513,35cyclictest8466-21chrt22:25:323
2125599513,30cyclictest29827-21chrt19:31:443
2125599512,36cyclictest28792-21chrt23:01:523
2125599512,33cyclictest22914-21chrt21:53:013
2125599503,36cyclictest22669-21chrt20:47:093
2125599503,36cyclictest22669-21chrt20:47:093
2125599503,34cyclictest28628-21chrt23:01:443
2125599503,34cyclictest28628-21chrt23:01:443
2125599503,33cyclictest30555-21chrt19:32:213
2125599503,33cyclictest15615-21chrt23:36:183
2125599503,33cyclictest14413-21chrt21:37:203
2125599503,32cyclictest28556-21taskset21:06:443
2125599502,38cyclictest4717-21taskset19:52:183
2125599502,37cyclictest16930-21chrt20:31:433
2125599502,37cyclictest16930-21chrt20:31:433
2125599502,36cyclictest32070-21chrt19:37:153
2125599502,35cyclictest28668-21chrt23:57:503
2125599502,35cyclictest26057-21chrt20:57:173
2125599502,35cyclictest12273-21taskset20:16:563
2125599502,34cyclictest7245-21chrt20:01:583
2125599502,34cyclictest31926-21chrt21:12:073
2125599502,32cyclictest3852-21chrt19:51:453
2125599494,34cyclictest10252-21taskset21:31:443
2125599493,37cyclictest9405-21chrt20:07:163
2125599493,37cyclictest15626-21taskset22:37:203
2125599493,36cyclictest31892-21chrt22:09:143
2125599493,33cyclictest15414-21chrt20:26:443
2125599493,32cyclictest25112-21chrt19:16:583
2125599493,32cyclictest25112-21chrt19:16:583
2125599493,32cyclictest22101-21chrt22:49:153
2125599493,32cyclictest12493-21chrt22:32:133
2125599493,28cyclictest20239-21chrt23:43:113
2125599492,34cyclictest3457-21chrt23:13:323
2125599492,34cyclictest19515-21taskset20:37:133
2125599492,34cyclictest19515-21taskset20:37:133
2125599492,33cyclictest9214-21chrt23:23:493
2125599492,33cyclictest1888-21chrt21:16:453
2125599483,33cyclictest10638-21chrt23:27:013
2125599483,33cyclictest10638-21chrt23:27:013
2125599482,35cyclictest18247-21chrt23:40:473
2125599482,35cyclictest18247-21chrt23:40:473
2125599473,35cyclictest26808-21taskset19:22:003
2125599473,35cyclictest26808-21taskset19:22:003
2125599473,28cyclictest24037-21chrt22:52:193
2125599472,34cyclictest19615-21taskset22:45:183
2125599464,32cyclictest6215-21taskset23:18:143
2125599464,32cyclictest6215-21taskset23:18:143
2125599463,33cyclictest4895-21chrt21:22:003
2125599443,32cyclictest30591-21chrt21:09:483
2125599412,25cyclictest21227-21chrt20:42:153
2125599412,25cyclictest21227-21chrt20:42:153
2125599403,24cyclictest1038-21chrt19:42:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional