You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 07:19
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Nov 18, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1825321600,19sleep10-21swapper/119:05:231
18217215910,115sleep017539-21/usr/sbin/munin19:05:230
190989912648,58cyclictest0-21swapper/022:35:110
19098991174,69cyclictest2266-21latency_hist00:04:420
19098991174,69cyclictest2266-21latency_hist00:04:420
19115991122,101cyclictest10838-21runrttasks23:25:012
19115991113,90cyclictest29817-21apt-get19:39:542
190989910951,51cyclictest1-21systemd19:29:140
19098991052,68cyclictest6364-21ssh23:17:340
19098991052,68cyclictest6364-21ssh23:17:340
191159910452,13cyclictest181rcu_preempt22:21:522
191159910452,13cyclictest181rcu_preempt22:21:522
190989910384,11cyclictest355-21runrttasks20:04:590
190989910349,47cyclictest0-21swapper/023:24:520
190989910347,47cyclictest0-21swapper/021:24:420
190989910342,49cyclictest6688-21kworker/u19:222:24:380
190989910342,49cyclictest6688-21kworker/u19:222:24:380
190989910341,53cyclictest31461-21cron23:59:380
190989910250,42cyclictest0-21swapper/020:25:020
19098991022,7cyclictest0-21swapper/022:29:530
19098991022,7cyclictest0-21swapper/022:29:530
190989910224,53cyclictest0-21swapper/021:14:410
19098991013,68cyclictest355-21runrttasks00:38:460
19098991013,68cyclictest355-21runrttasks00:38:460
19098991012,87cyclictest16607-21apt-get00:29:500
19102991002,40cyclictest355-21runrttasks21:19:551
190989910047,45cyclictest0-21swapper/000:25:100
190989910046,48cyclictest0-21swapper/023:49:480
190989910045,49cyclictest31078-21grep22:10:080
190989910044,46cyclictest0-21swapper/020:15:070
190989910044,46cyclictest0-21swapper/020:15:070
19115999935,34cyclictest39-21ksoftirqd/223:00:082
1909899993,84cyclictest0-21swapper/020:59:550
1909899993,84cyclictest0-21swapper/020:59:550
1909899993,58cyclictest3501-21seq23:12:480
1909899992,23cyclictest13182-21runrttasks00:23:550
1909899982,64cyclictest20259-21seq20:47:480
1910299974,8cyclictest0-21swapper/120:29:511
19102999746,38cyclictest0-21swapper/120:44:511
1909899978,58cyclictest0-21swapper/023:08:140
19098999743,47cyclictest0-21swapper/021:59:380
1909899973,82cyclictest0-21swapper/022:04:560
1909899973,82cyclictest0-21swapper/022:04:560
1909899973,78cyclictest21932-21ssh22:49:420
1909899973,64cyclictest18638-21cpuspeed21:49:580
1909899973,64cyclictest18638-21cpuspeed21:49:580
1909899972,83cyclictest13607-21apt-get23:29:500
1909899972,83cyclictest13607-21apt-get23:29:500
1909899972,63cyclictest5857-21seq21:28:270
1911599962,81cyclictest25859-21modprobe23:49:542
1909899963,6cyclictest0-21swapper/022:54:560
19098999632,58cyclictest0-21swapper/022:50:150
1910299954,28cyclictest355-21runrttasks21:13:231
19098999577,12cyclictest22634-21cpu23:44:560
19098999539,48cyclictest10755-21rm00:19:420
19098999534,52cyclictest0-21swapper/020:30:020
1910299943,30cyclictest32432-21sh23:06:081
1910299942,29cyclictest26890-21diskmemload22:37:401
1910299942,27cyclictest355-21runrttasks21:26:161
1909899945,73cyclictest28089-21apt-get22:59:510
19098999446,41cyclictest0-21swapper/021:40:150
19098999444,44cyclictest0-21swapper/022:40:030
1909899943,85cyclictest0-21swapper/022:14:590
19098999438,50cyclictest26890-21diskmemload21:33:400
19098999438,50cyclictest26890-21diskmemload21:33:400
1911599938,76cyclictest19448-21apt-get19:10:212
1911599937,70cyclictest14564-21kworker/u17:1+flush-179:022:50:002
1910299933,8cyclictest0-21swapper/119:54:581
1910299933,32cyclictest355-21runrttasks22:12:421
19098999345,41cyclictest0-21swapper/023:20:170
1909899933,80cyclictest0-21swapper/020:54:590
1909899933,6cyclictest0-21swapper/020:39:560
1909899933,6cyclictest0-21swapper/020:39:560
1909899932,7cyclictest0-21swapper/000:09:520
19098999263,20cyclictest5811-21latency_hist20:04:420
19098999243,42cyclictest0-21swapper/019:30:140
1909899923,76cyclictest19629-21apt-get23:39:540
1909899923,75cyclictest7026-21apt-get22:24:520
1909899923,60cyclictest0-21swapper/022:01:580
19115999149,13cyclictest181rcu_preempt00:20:062
1910299914,77cyclictest0-21swapper/119:59:531
1909899913,76cyclictest0-21swapper/021:14:510
19098999131,50cyclictest0-21swapper/000:04:560
1911599903,58cyclictest19633-21/usr/sbin/munin20:46:072
1910299904,75cyclictest0-21swapper/119:24:561
19102999041,41cyclictest13980-21fschecks_count00:25:021
1909899902,57cyclictest21934-21seq20:53:390
1909899902,19cyclictest26890-21diskmemload21:45:200
19098999018,66cyclictest0-21swapper/021:35:020
19115998967,14cyclictest355-21runrttasks00:17:192
19102998947,35cyclictest0-21swapper/123:45:001
1910299893,33cyclictest28219-21fschecks_time19:35:031
1910299892,7cyclictest0-21swapper/100:09:521
1910299892,6cyclictest0-21swapper/122:54:561
1909899894,76cyclictest25540-21apt-get21:04:530
19098998936,47cyclictest22837-21modprobe19:19:390
19115998842,17cyclictest181rcu_preempt22:30:362
19115998842,17cyclictest181rcu_preempt22:30:362
19115998841,13cyclictest181rcu_preempt23:08:332
1911599882,60cyclictest32765-21apt-get19:49:532
1911599882,60cyclictest32765-21apt-get19:49:532
1910299883,80cyclictest0-21swapper/121:54:571
1909899885,71cyclictest0-21swapper/020:09:560
1909899884,77cyclictest0-21swapper/019:49:420
1909899884,73cyclictest0-21swapper/020:19:520
1909899884,73cyclictest0-21swapper/020:19:520
1909899882,79cyclictest16513-21unixbench_multi20:35:200
1909899882,77cyclictest23114-21/usr/sbin/munin19:20:200
19098998819,62cyclictest0-21swapper/019:39:530
19115998754,13cyclictest181rcu_preempt19:54:592
19115998739,17cyclictest181rcu_preempt21:41:482
1910299873,6cyclictest0-21swapper/120:39:561
1910299873,6cyclictest0-21swapper/120:39:561
1910299873,37cyclictest7475-21ssh23:19:541
1909899873,6cyclictest0-21swapper/019:34:560
19098998728,54cyclictest15350irq/292-2188000.ethernet23:39:010
19115998640,16cyclictest181rcu_preempt00:29:492
19115998637,18cyclictest181rcu_preempt23:11:272
19102998672,8cyclictest0-21swapper/121:00:131
19102998672,8cyclictest0-21swapper/121:00:131
19102998642,36cyclictest0-21swapper/123:35:151
19102998641,37cyclictest0-21swapper/122:40:411
1910299863,8cyclictest0-21swapper/122:29:531
1910299863,8cyclictest0-21swapper/122:29:531
1909899862,56cyclictest2489-21apt-config19:54:420
1909899862,56cyclictest2489-21apt-config19:54:420
19115998559,10cyclictest181rcu_preempt20:41:142
19115998559,10cyclictest181rcu_preempt20:41:142
1910299854,9cyclictest0-21swapper/100:34:571
1910299854,9cyclictest0-21swapper/100:34:571
19102998545,33cyclictest0-21swapper/122:50:301
1910299853,9cyclictest0-21swapper/122:24:561
1910299853,8cyclictest0-21swapper/121:39:551
1910299853,7cyclictest0-21swapper/120:54:591
1910299853,34cyclictest23424-21/usr/sbin/munin19:20:051
1910299853,33cyclictest6910-21awk20:05:191
1910299853,27cyclictest355-21runrttasks21:36:211
1910299852,6cyclictest0-21swapper/119:10:231
1909899854,76cyclictest0-21swapper/019:54:580
19115998448,8cyclictest181rcu_preempt23:39:112
19115998448,8cyclictest181rcu_preempt20:04:392
19115998436,18cyclictest181rcu_preempt21:31:302
19115998436,18cyclictest181rcu_preempt21:31:302
19115998433,17cyclictest181rcu_preempt19:20:002
19115998419,26cyclictest0-21swapper/221:21:222
19102998447,28cyclictest4746-21find22:20:121
19102998447,28cyclictest4746-21find22:20:121
1910299843,8cyclictest0-21swapper/121:04:561
1910299843,76cyclictest80212sleep100:14:541
1910299843,69cyclictest16041-21memory20:35:101
1910299843,38cyclictest14187-21/usr/sbin/munin23:30:071
1910299843,38cyclictest14187-21/usr/sbin/munin23:30:071
1910299843,37cyclictest25049-21taskset22:00:051
1910299843,34cyclictest1350-21ssh22:15:061
1910299842,35cyclictest355-21runrttasks00:03:091
1910299842,35cyclictest355-21runrttasks00:03:091
1910299842,34cyclictest1885-21sshd19:51:141
1910299842,34cyclictest1885-21sshd19:51:141
1911599838,62cyclictest12983-21ssh22:34:522
19115998355,12cyclictest181rcu_preempt22:04:422
19115998342,9cyclictest181rcu_preempt20:50:122
19115998340,12cyclictest181rcu_preempt21:44:452
19115998338,18cyclictest181rcu_preempt22:08:372
19115998338,18cyclictest181rcu_preempt22:08:372
1910299833,7cyclictest0-21swapper/122:59:511
1910299833,37cyclictest19183-21modprobe00:34:381
1910299833,35cyclictest6900-21fschecks_time21:30:021
1910299833,35cyclictest6900-21fschecks_time21:30:021
1910299833,34cyclictest4820-21/usr/sbin/munin23:15:041
1910299833,34cyclictest4820-21/usr/sbin/munin23:15:041
19115998266,10cyclictest22955-21/usr/sbin/munin23:45:032
19115998258,9cyclictest181rcu_preempt21:51:182
19115998258,9cyclictest181rcu_preempt21:51:182
19115998254,8cyclictest181rcu_preempt21:29:242
19115998248,11cyclictest181rcu_preempt23:20:272
19115998245,14cyclictest181rcu_preempt19:31:502
19115998239,10cyclictest181rcu_preempt23:57:402
19115998236,18cyclictest181rcu_preempt22:42:322
19115998233,12cyclictest181rcu_preempt22:29:352
1910299824,72cyclictest0-21swapper/120:19:521
1910299824,72cyclictest0-21swapper/120:19:521
1910299824,6cyclictest0-21swapper/121:14:511
19102998241,34cyclictest0-21swapper/119:30:161
1910299823,7cyclictest0-21swapper/123:39:541
1910299823,32cyclictest12395-21taskset23:27:081
1910299823,29cyclictest15656-21cpuspeed21:44:551
1910299822,31cyclictest21042-21munin-node19:14:501
1909899822,75cyclictest0-21swapper/019:10:240
181212829,47sleep218152-21cut19:05:212
19115998144,10cyclictest181rcu_preempt20:25:162
19115998142,14cyclictest181rcu_preempt20:30:132
19115998120,33cyclictest39-21ksoftirqd/200:02:282
19115998120,33cyclictest39-21ksoftirqd/200:02:282
1910299813,7cyclictest0-21swapper/123:49:561
1910299813,35cyclictest28856-21cpu23:54:551
1910299813,34cyclictest3292-21runrttasks23:12:111
1910299812,7cyclictest0-21swapper/100:19:541
1910299812,35cyclictest30240-21kernelversion19:40:491
1910299812,35cyclictest21190-21/usr/sbin/munin20:50:081
1910299812,33cyclictest2774-21/usr/sbin/munin00:05:051
19115998053,12cyclictest181rcu_preempt19:18:212
19115998052,19cyclictest426-21apt-key19:49:422
19115998046,12cyclictest181rcu_preempt21:12:392
19115998044,13cyclictest181rcu_preempt23:42:382
19115998042,18cyclictest181rcu_preempt21:06:432
19115998035,14cyclictest181rcu_preempt22:59:312
19115998035,13cyclictest181rcu_preempt23:16:592
19115998035,13cyclictest181rcu_preempt23:16:592
19115998034,13cyclictest181rcu_preempt21:18:472
1910299803,8cyclictest0-21swapper/120:09:561
1910299803,35cyclictest20543-21ssh22:46:391
1910299802,29cyclictest20142-21seq21:51:411
1910299802,29cyclictest20142-21seq21:51:411
19115997944,13cyclictest181rcu_preempt00:29:412
19115997941,17cyclictest181rcu_preempt20:11:372
19115997937,13cyclictest181rcu_preempt22:16:212
19115997923,15cyclictest0-21swapper/220:05:002
19102997940,32cyclictest0-21swapper/120:15:151
19102997940,32cyclictest0-21swapper/120:15:151
1910299792,35cyclictest32215-21/usr/sbin/munin19:45:171
19115997853,19cyclictest0-21swapper/221:59:382
19115997839,8cyclictest181rcu_preempt20:20:132
19115997839,8cyclictest181rcu_preempt20:20:132
19115997839,12cyclictest19088-21cyclictest00:07:542
19115997838,13cyclictest181rcu_preempt19:25:482
1910299783,33cyclictest12673-21ls20:25:041
1910299782,33cyclictest27562-21ssh22:04:551
1910299782,33cyclictest27562-21ssh22:04:551
19115997743,10cyclictest181rcu_preempt21:37:542
1911599773,45cyclictest39-21ksoftirqd/222:45:182
19115997733,8cyclictest181rcu_preempt00:14:392
19115997644,9cyclictest181rcu_preempt23:31:122
19115997644,9cyclictest181rcu_preempt23:31:122
19115997642,12cyclictest181rcu_preempt00:39:232
19115997642,12cyclictest181rcu_preempt00:39:232
19115997637,17cyclictest181rcu_preempt22:10:492
1912199743,59cyclictest0-21swapper/319:10:213
19115997450,10cyclictest181rcu_preempt21:00:032
19115997450,10cyclictest181rcu_preempt21:00:032
19115997447,11cyclictest181rcu_preempt20:35:032
19115997336,14cyclictest181rcu_preempt19:34:592
19115997327,13cyclictest0-21swapper/220:19:012
19115997327,13cyclictest0-21swapper/220:19:012
1912199729,44cyclictest0-21swapper/323:49:563
19115997228,11cyclictest0-21swapper/220:55:162
19121996912,32cyclictest0-21swapper/323:24:593
1912199644,37cyclictest0-21swapper/319:55:003
1912199634,49cyclictest0-21swapper/322:50:013
19121996111,29cyclictest0-21swapper/319:39:553
19121995813,28cyclictest0-21swapper/320:44:573
19121995731,18cyclictest29678-21chrt21:14:203
1912199564,37cyclictest23727-21taskset19:20:153
1912199553,40cyclictest12599-21chrt20:25:033
1912199553,37cyclictest9492-21taskset21:34:423
1912199552,39cyclictest4046-21taskset22:19:523
1912199552,39cyclictest4046-21taskset22:19:523
19121995514,28cyclictest0-21swapper/319:49:533
19121995514,28cyclictest0-21swapper/319:49:533
1912199546,39cyclictest895-21kworker/3:1+events00:13:103
1912199543,37cyclictest17888-21chrt20:40:143
1912199543,37cyclictest17888-21chrt20:40:143
1912199543,37cyclictest15018-21taskset22:37:593
1912199533,37cyclictest27089-21chrt21:09:423
1912199533,36cyclictest24121-21chrt23:46:263
1912199525,33cyclictest27738-21taskset22:59:383
1912199524,34cyclictest22262-21chrt19:15:163
1912199523,40cyclictest13012-21chrt21:40:053
1912199523,36cyclictest2586-21taskset23:10:163
1912199523,36cyclictest16721-21chrt23:34:563
1912199523,35cyclictest25822-21chrt22:00:473
1912199523,35cyclictest16476-21taskset20:35:193
1912199523,34cyclictest895-21kworker/3:1+events00:24:443
1912199523,34cyclictest16038-21taskset23:34:023
1912199523,34cyclictest16038-21taskset23:34:023
1912199522,36cyclictest28834-21taskset23:54:533
1912199522,35cyclictest721-21chrt19:49:433
1912199514,33cyclictest18824-21taskset00:33:393
1912199513,36cyclictest9806-21chrt00:17:263
1912199513,36cyclictest7307-21chrt21:30:123
1912199513,35cyclictest2216-21chrt00:04:423
1912199513,35cyclictest2216-21chrt00:04:423
1912199513,35cyclictest1558-21chrt22:15:103
1912199513,30cyclictest28019-21taskset22:59:433
1912199512,39cyclictest31697-21chrt23:05:143
1912199512,37cyclictest15652-21chrt21:44:553
1912199512,36cyclictest22943-21taskset20:55:163
1770225110,25sleep30-21swapper/319:05:013
1912199503,36cyclictest8536-21chrt22:26:283
1912199503,33cyclictest14189-21chrt20:30:013
1912199503,31cyclictest30235-21chrt21:14:573
1912199502,36cyclictest27999-21taskset19:34:573
1912199493,33cyclictest5909-21chrt23:16:163
1912199493,33cyclictest5909-21chrt23:16:163
1912199492,36cyclictest8396-21chrt20:10:173
59772487,9sleep30-21swapper/320:04:593
1912199483,35cyclictest17211-21taskset22:40:383
1912199483,33cyclictest29881-21chrt22:08:543
1912199483,33cyclictest29881-21chrt22:08:543
1912199483,32cyclictest20800-21chrt20:49:583
1912199482,37cyclictest21498-21taskset23:42:293
1912199482,33cyclictest20778-21taskset22:47:193
1912199482,31cyclictest24033-21chrt20:59:583
1912199482,31cyclictest24033-21chrt20:59:583
1912199473,32cyclictest1175-21chrt21:20:073
1912199472,32cyclictest9273-21chrt23:22:193
1912199472,32cyclictest11723-21taskset22:32:033
1912199472,32cyclictest11723-21taskset22:32:033
1912199463,31cyclictest19222-21chrt21:50:143
1912199463,31cyclictest19222-21chrt21:50:143
1912199463,30cyclictest4726-21chrt00:09:123
1912199453,29cyclictest13103-21chrt00:23:423
1912199453,27cyclictest22233-21taskset00:39:423
1912199453,27cyclictest22233-21taskset00:39:423
1912199452,31cyclictest26881-21chrt19:30:113
1912199443,32cyclictest30848-21taskset22:10:033
1912199442,33cyclictest4790-21chrt21:25:263
1912199442,29cyclictest9398-21chrt20:15:023
1912199442,29cyclictest9398-21chrt20:15:023
1912199442,28cyclictest4645-21chrt20:00:043
1912199433,31cyclictest25080-21chrt19:25:043
1912199433,29cyclictest21528-21chrt21:54:523
1912199424,27cyclictest10746-21chrt21:35:273
1912199383,22cyclictest10746-21taskset20:19:443
1912199383,22cyclictest10746-21taskset20:19:443
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional