You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-04 - 09:40
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Fri Jul 04, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2311021920,53sleep223113-21idleruntime-cro19:06:302
2311021920,53sleep223113-21idleruntime-cro19:06:302
233469912113,85cyclictest8223-21kworker/u17:3+rpciod00:12:040
23346991134,14cyclictest0-21swapper/019:06:340
23346991134,14cyclictest0-21swapper/019:06:340
233469911252,52cyclictest0-21swapper/021:57:080
23346991093,23cyclictest3323-21/usr/sbin/munin23:06:500
23346991083,31cyclictest356-21runrttasks20:26:430
23346991083,31cyclictest356-21runrttasks20:26:430
233469910652,47cyclictest14720-21ssh23:26:390
23346991043,32cyclictest895-21rm23:02:020
23346991043,32cyclictest895-21rm23:02:020
23346991023,32cyclictest20712-21latency_hist20:36:300
23346991023,29cyclictest23667-21latency_hist22:46:330
23346991023,29cyclictest23667-21latency_hist22:46:330
23346991022,26cyclictest8113-21cmp00:11:330
23346991022,26cyclictest29171-21taskset22:55:470
23346991022,26cyclictest29171-21taskset22:55:470
21922210211,63sleep121986-21fschecks_time19:01:571
21922210211,63sleep121986-21fschecks_time19:01:571
23346991013,23cyclictest12838-21unixbench_multi23:22:030
23346991013,23cyclictest12838-21unixbench_multi23:22:030
23358991003,68cyclictest8015-21apt-get19:56:451
233469910078,15cyclictest29147-21latency_hist21:01:330
23346991007,30cyclictest1285-21cat21:11:320
23346991002,35cyclictest356-21runrttasks22:57:190
23346991002,33cyclictest19926-21grep00:31:560
23346991002,31cyclictest24498-21/usr/sbin/munin23:42:050
2334699994,31cyclictest32121-21/usr/sbin/munin19:32:070
2334699992,35cyclictest2359-21sed21:12:090
2334699992,32cyclictest30367-21runrttasks19:26:320
23346999835,56cyclictest0-21swapper/019:07:310
2334699983,29cyclictest707-21timerwakeupswit23:57:070
2334699983,29cyclictest707-21timerwakeupswit23:57:070
2334699982,30cyclictest6888-21latency_hist21:21:300
2334699982,21cyclictest9095-21taskset23:16:470
2334699975,24cyclictest13557-21/usr/sbin/munin21:32:040
2334699973,33cyclictest12140-21grep20:07:080
2334699973,31cyclictest356-21runrttasks19:27:060
2334699972,30cyclictest24200-21modprobe21:51:280
2334699972,23cyclictest13284-21ssh00:20:380
2334699972,23cyclictest13284-21ssh00:20:380
23346999722,22cyclictest9727-21cstates20:01:480
2334699972,20cyclictest5174-21latency_hist00:06:300
23346999720,8cyclictest0-21swapper/022:49:320
2334699964,33cyclictest342-21idleruntime-cro22:06:280
2334699964,33cyclictest17732-21grep00:27:090
23346999642,47cyclictest0-21swapper/019:16:540
2334699964,20cyclictest18163-21needreboot23:31:560
2334699962,32cyclictest2098-21grep22:07:450
2334699962,30cyclictest9873-21ls21:26:310
2334699953,36cyclictest4492-21find22:12:060
2334699953,35cyclictest29672-21grep21:02:000
2334699952,42cyclictest21587-21timerwakeupswit23:37:070
2334699952,32cyclictest2825-21latency_hist19:41:310
2334699952,32cyclictest2825-21latency_hist19:41:310
2334699952,31cyclictest16022-21mailstats22:32:060
2334699952,29cyclictest6125-21/usr/sbin/munin19:51:500
2334699952,25cyclictest32088-21latency_hist23:56:320
2334699945,26cyclictest4311-21latency_hist19:46:300
2334699945,26cyclictest4311-21latency_hist19:46:300
2334699944,31cyclictest356-21runrttasks22:31:440
2334699944,31cyclictest356-21runrttasks22:31:440
2334699944,30cyclictest10644-21/usr/sbin/munin22:22:170
23346999433,55cyclictest0-21swapper/020:41:570
2334699943,34cyclictest19040-21fschecks_time21:41:520
2334699943,34cyclictest19040-21fschecks_time21:41:520
2334699943,22cyclictest24623-21cstates21:51:470
2334699942,28cyclictest18181-21/usr/sbin/munin22:37:130
23381999374,13cyclictest27139-21latency_hist19:16:312
2334699934,32cyclictest12829-21/usr/sbin/munin20:12:060
2334699933,31cyclictest6578-21/usr/sbin/munin22:16:510
2334699933,31cyclictest6578-21/usr/sbin/munin22:16:510
2334699933,30cyclictest5435-21unixbench_multi19:47:060
2334699933,30cyclictest5435-21unixbench_multi19:47:060
2334699933,30cyclictest10949-21ssh21:27:060
2334699933,29cyclictest25665-21/usr/sbin/munin19:11:530
2334699933,19cyclictest16094-21apt-get20:21:460
2338199926,49cyclictest39-21ksoftirqd/223:41:052
23358999275,11cyclictest31879-40ogrotate23:56:291
2334699924,29cyclictest14747-21grep20:16:530
2334699923,34cyclictest16280-21ssh21:36:570
2334699923,32cyclictest375-21snmpd20:56:330
2334699923,32cyclictest375-21snmpd20:56:330
2334699923,28cyclictest14140-21fschecks_time00:21:510
2334699922,23cyclictest8754-21latency_hist23:16:320
23346999218,9cyclictest26706-21fschecks_time23:46:510
23381999161,21cyclictest29190-21latency_hist23:51:322
2334699913,34cyclictest23894-21apt-get20:46:560
2334699913,34cyclictest23894-21apt-get20:46:560
2334699913,33cyclictest8993-21unixbench_singl19:57:230
2334699912,29cyclictest20919-21/usr/sbin/munin20:36:570
23358999073,8cyclictest0-21swapper/122:41:291
2338199878,57cyclictest902-21apt-get22:06:582
23358998658,19cyclictest30603-21diskmemload23:11:551
23358998568,11cyclictest25946-21sh22:49:451
23381998469,9cyclictest11629-21ssh23:21:312
23381998465,13cyclictest20641-21sh20:36:292
23381998459,15cyclictest202-21systemd-journal23:23:452
23381998459,15cyclictest202-21systemd-journal23:23:452
23381998433,26cyclictest2148-21kworker/u19:0+nfsiod23:10:352
2338199842,34cyclictest39-21ksoftirqd/200:14:312
23358998412,62cyclictest23666-21modprobe19:06:571
2338199837,63cyclictest8055-21cpu19:57:022
2338199825,44cyclictest39-21ksoftirqd/222:31:172
2338199825,44cyclictest39-21ksoftirqd/222:31:172
2338199817,46cyclictest39-21ksoftirqd/220:40:042
23381998156,17cyclictest0-21swapper/222:11:562
23358998158,16cyclictest356-21runrttasks23:21:311
2338199803,39cyclictest39-21ksoftirqd/219:10:472
23358998057,15cyclictest7259-21ntp_offset22:17:021
23358998057,15cyclictest7259-21ntp_offset22:17:021
2338199792,41cyclictest39-21ksoftirqd/222:22:542
23381997910,37cyclictest39-21ksoftirqd/223:36:212
23358997931,37cyclictest8056-21date00:11:331
2338199783,40cyclictest39-21ksoftirqd/222:39:452
23381997817,10cyclictest0-21swapper/223:51:402
23358997857,14cyclictest24695-21sh21:51:501
23358997851,16cyclictest30603-21diskmemload23:36:591
2335899784,48cyclictest16813-21unixbench_multi21:37:071
2338199773,45cyclictest39-21ksoftirqd/220:55:412
2338199773,45cyclictest39-21ksoftirqd/220:55:412
2338199772,45cyclictest39-21ksoftirqd/200:31:372
2338199771,43cyclictest39-21ksoftirqd/221:57:032
23358997760,11cyclictest331-21dbus-daemon23:36:291
23358997760,10cyclictest27884-21diskmemload22:52:081
23358997760,10cyclictest27884-21diskmemload22:52:081
23381997655,13cyclictest0-21swapper/222:46:532
2338199765,30cyclictest39-21ksoftirqd/219:36:532
2338199765,30cyclictest39-21ksoftirqd/219:36:532
23381997652,16cyclictest14733-21fschecks_time20:16:522
2338199761,47cyclictest39-21ksoftirqd/222:54:022
2338199761,47cyclictest39-21ksoftirqd/222:54:022
2335899765,43cyclictest1038-21ssh22:07:071
2338199752,25cyclictest39-21ksoftirqd/223:13:152
2338199751,29cyclictest39-21ksoftirqd/220:57:172
23358997547,9cyclictest181rcu_preempt22:59:341
23358997546,10cyclictest17288-21ssh22:34:531
23358997545,10cyclictest0-21swapper/120:21:521
2335899752,35cyclictest6623-21ssh22:16:451
2338199748,30cyclictest39-21ksoftirqd/220:02:142
2338199746,53cyclictest356-21runrttasks20:47:122
2338199746,53cyclictest356-21runrttasks20:47:122
2338199745,25cyclictest39-21ksoftirqd/219:41:592
2338199745,25cyclictest39-21ksoftirqd/219:41:592
23381997448,14cyclictest0-21swapper/200:02:062
23381997445,19cyclictest32466-21latency_hist23:01:312
2338199742,35cyclictest39-21ksoftirqd/219:56:312
2338199742,32cyclictest39-21ksoftirqd/200:25:122
2338199742,28cyclictest39-21ksoftirqd/222:45:552
2338199742,28cyclictest39-21ksoftirqd/222:45:552
23358997453,14cyclictest18104-21sh00:28:121
23358997447,20cyclictest2897-21latency_hist23:06:311
23358997447,20cyclictest2897-21latency_hist23:06:311
23358997445,21cyclictest21293-21modprobe21:46:291
23358997445,21cyclictest21293-21modprobe21:46:291
2338199739,41cyclictest39-21ksoftirqd/223:26:392
2338199734,50cyclictest39-21ksoftirqd/221:12:322
2338199734,44cyclictest356-21runrttasks20:15:582
2338199732,24cyclictest39-21ksoftirqd/221:09:052
23358997349,14cyclictest0-21swapper/121:11:471
2335899733,34cyclictest7481-21latency_hist19:56:311
2339499723,31cyclictest11955-21chrt21:29:373
2338199725,36cyclictest39-21ksoftirqd/221:38:412
23381997250,16cyclictest5459-21unixbench_multi19:47:062
23381997250,16cyclictest5459-21unixbench_multi19:47:062
23381997244,20cyclictest29313-21cpuspeed21:01:472
2338199722,32cyclictest39-21ksoftirqd/221:47:052
23358997259,9cyclictest58-21kcompactd023:26:101
23358997259,9cyclictest58-21kcompactd023:26:101
23358997253,14cyclictest0-21swapper/100:26:281
23358997251,14cyclictest393-21sh22:06:301
2335899723,57cyclictest22556-21sshd20:41:471
2338199714,35cyclictest39-21ksoftirqd/220:25:212
23381997110,38cyclictest39-21ksoftirqd/220:44:452
23358997155,10cyclictest30603-21diskmemload00:06:311
23358997153,13cyclictest23321-21cyclictest20:17:561
23358997149,8cyclictest181rcu_preempt20:39:131
23358997149,16cyclictest365-21systemd-logind19:11:421
23358997145,20cyclictest31-21ksoftirqd/120:07:061
23358997136,11cyclictest181rcu_preempt19:51:321
23358997136,11cyclictest181rcu_preempt19:51:321
2335899712,35cyclictest26102-21modprobe20:51:481
2335899712,35cyclictest26102-21modprobe20:51:481
2339499705,29cyclictest17913-21chrt21:40:063
2338199704,27cyclictest39-21ksoftirqd/219:16:572
23381997042,22cyclictest1334-21sh23:02:102
23381997042,22cyclictest1334-21sh23:02:102
23381997029,9cyclictest0-21swapper/219:26:592
2338199702,40cyclictest39-21ksoftirqd/221:55:082
2338199702,29cyclictest39-21ksoftirqd/200:27:042
23358997050,14cyclictest0-21swapper/121:26:281
23358997048,9cyclictest181rcu_preempt23:10:021
23358997048,12cyclictest0-21swapper/123:41:451
2335899704,40cyclictest29155-21fschecks_time19:21:541
23358997040,22cyclictest30482-21kworker/u18:0+nfsiod21:31:551
23358997029,10cyclictest23321-21cyclictest21:21:081
23358997026,14cyclictest181rcu_preempt19:16:541
2335899702,45cyclictest14236-21lockfile-create20:16:291
2339499693,26cyclictest13884-21taskset00:21:423
2339499692,30cyclictest6908-21chrt21:21:303
23381996956,8cyclictest381ktimers/221:31:542
23381996942,20cyclictest18178-21ntp_kernel_err20:26:582
23381996942,20cyclictest18178-21ntp_kernel_err20:26:582
23381996939,11cyclictest181rcu_preempt23:58:442
23381996939,11cyclictest181rcu_preempt23:58:442
2338199693,46cyclictest6974-21fschecks_time22:16:562
2338199693,46cyclictest6974-21fschecks_time22:16:562
2338199691,35cyclictest39-21ksoftirqd/200:10:502
23358996947,15cyclictest31539-21sh21:07:101
23358996944,18cyclictest375-21snmpd00:16:531
23358996944,18cyclictest375-21snmpd00:16:531
2335899694,33cyclictest14957-21ssh23:26:501
23358996942,11cyclictest181rcu_preempt00:31:421
2335899693,50cyclictest9388-21unixbench_singl00:12:211
2183926917,27sleep30-21swapper/319:01:513
2183926917,27sleep30-21swapper/319:01:513
2338199685,34cyclictest29904-21grep19:22:092
23381996845,13cyclictest0-21swapper/221:41:572
23381996845,13cyclictest0-21swapper/221:41:572
2338199681,24cyclictest39-21ksoftirqd/222:01:522
23358996851,9cyclictest0-21swapper/122:27:131
23358996851,9cyclictest0-21swapper/122:27:131
23358996832,24cyclictest3257-21if_eth019:41:551
23358996832,24cyclictest3257-21if_eth019:41:551
23358996824,13cyclictest181rcu_preempt21:28:341
2339499673,24cyclictest850-21taskset22:06:363
23381996751,9cyclictest0-21swapper/219:32:022
2338199673,27cyclictest39-21ksoftirqd/220:07:352
2338199672,39cyclictest39-21ksoftirqd/222:36:262
2338199672,28cyclictest39-21ksoftirqd/221:23:102
2338199672,26cyclictest39-21ksoftirqd/223:46:292
2335899675,13cyclictest0-21swapper/120:26:461
2335899675,13cyclictest0-21swapper/120:26:461
23358996749,11cyclictest376-21in:imuxsock21:06:291
2339499663,27cyclictest9910-21chrt23:17:073
2339499663,24cyclictest20474-21taskset23:36:323
2339499663,24cyclictest14223-21chrt20:16:293
23358996650,8cyclictest0-21swapper/123:46:501
2335899663,35cyclictest21814-21find22:42:071
2335899663,35cyclictest21814-21find22:42:071
23358996625,30cyclictest1998-21ssh00:00:371
23358996625,30cyclictest1998-21ssh00:00:371
2339499657,39cyclictest0-21swapper/319:56:513
2339499653,24cyclictest11652-21chrt20:06:573
2339499653,23cyclictest1204-21chrt23:58:073
2339499653,23cyclictest1204-21chrt23:58:073
2339499652,30cyclictest21816-21taskset20:37:073
2339499652,24cyclictest4635-21chrt00:05:043
23394996510,39cyclictest0-21swapper/300:12:003
2338199653,37cyclictest39-21ksoftirqd/221:26:592
23358996544,15cyclictest12078-21cut22:26:301
23358996541,8cyclictest181rcu_preempt19:31:441
2335899653,45cyclictest19497-21fschecks_count20:31:521
2335899653,42cyclictest27592-21fschecks_count21:56:521
2339499643,32cyclictest366-21chrt23:01:473
2339499643,32cyclictest366-21chrt23:01:473
2339499643,28cyclictest5620-21taskset23:11:193
2339499643,26cyclictest20509-21chrt22:40:443
2339499643,20cyclictest28814-21chrt23:51:063
2339499642,20cyclictest11348-21chrt00:16:523
2339499642,20cyclictest11348-21chrt00:16:523
23381996424,13cyclictest0-21swapper/200:21:312
23381996424,13cyclictest0-21swapper/200:21:312
23358996446,11cyclictest0-21swapper/120:01:521
23358996440,13cyclictest0-21swapper/120:47:381
23358996440,13cyclictest0-21swapper/120:47:381
2335899643,29cyclictest28332-21sensors_temp20:57:151
2339499633,23cyclictest19244-21chrt00:31:313
2339499633,22cyclictest961-21chrt19:35:473
2339499633,15cyclictest30092-21chrt22:56:533
2339499632,26cyclictest6333-21chrt23:11:523
23381996330,7cyclictest181rcu_preempt21:18:472
23358996348,9cyclictest22000-21ssh21:46:541
2335899633,14cyclictest1328-21apt-get19:36:441
2335899633,14cyclictest1328-21apt-get19:36:441
2339499623,40cyclictest20841-21taskset23:36:523
2339499623,25cyclictest19837-21chrt20:32:003
2339499623,24cyclictest7533-21chrt00:10:563
2339499623,20cyclictest31333-21chrt23:54:593
2339499622,27cyclictest11845-21chrt23:21:413
2339499622,27cyclictest11845-21chrt23:21:413
2339499622,22cyclictest4057-21chrt22:11:553
2339499622,22cyclictest16877-21chrt23:30:113
2339499622,21cyclictest24798-21chrt20:47:323
2339499622,21cyclictest24798-21chrt20:47:323
2339499622,19cyclictest21500-21chrt22:41:583
2339499622,19cyclictest21500-21chrt22:41:583
2339499613,24cyclictest31823-21chrt19:31:013
2339499613,21cyclictest25767-21chrt22:49:113
2339499613,20cyclictest31846-21chrt21:07:163
2339499612,27cyclictest6839-21chrt22:16:493
2339499612,27cyclictest6839-21chrt22:16:493
2339499612,25cyclictest26824-21chrt21:55:543
2339499612,22cyclictest19021-21chrt20:31:283
2339499612,22cyclictest19021-21chrt20:31:283
2339499612,21cyclictest28187-21chrt20:57:113
2339499612,21cyclictest1419-21chrt19:36:483
2339499612,21cyclictest1419-21chrt19:36:483
2339499604,24cyclictest23703-21chrt20:44:153
2339499603,21cyclictest25581-21chrt23:45:033
2339499603,20cyclictest30685-21chrt21:06:313
2339499603,20cyclictest28608-21chrt19:20:573
2339499603,20cyclictest16813-21chrt20:22:043
2339499603,20cyclictest15095-21taskset21:35:243
2339499602,26cyclictest20444-21chrt21:43:593
2339499602,26cyclictest20444-21chrt21:43:593
2339499602,25cyclictest29757-21chrt22:01:053
2339499602,24cyclictest28233-21chrt22:53:003
2339499602,24cyclictest28233-21chrt22:53:003
2339499602,24cyclictest23862-21chrt21:50:323
2339499602,23cyclictest15625-21chrt20:19:043
2339499602,23cyclictest14492-21chrt22:30:113
2339499602,23cyclictest14492-21chrt22:30:113
2339499602,22cyclictest16987-21chrt22:33:593
2339499602,21cyclictest31405-21chrt22:02:233
2339499602,20cyclictest9440-21chrt21:25:403
2339499602,20cyclictest20056-21taskset00:31:593
2339499593,23cyclictest3217-21taskset19:41:543
2339499593,23cyclictest3217-21taskset19:41:543
2339499592,24cyclictest11864-21taskset22:25:533
2339499592,23cyclictest7278-21chrt19:54:013
2339499592,23cyclictest29976-21chrt19:22:513
2339499592,21cyclictest26692-21chrt19:12:263
2339499592,20cyclictest4684-21chrt19:46:483
2339499592,20cyclictest4684-21chrt19:46:483
2339499592,20cyclictest24109-21chrt19:07:243
23358995928,9cyclictest181rcu_preempt19:26:471
2339499587,11cyclictest0-21swapper/320:51:593
2339499587,11cyclictest0-21swapper/320:51:593
2339499582,20cyclictest1321-21chrt21:11:343
2339499582,20cyclictest10754-21chrt20:03:503
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional