You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 06:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Tue Mar 24, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1863321136,64sleep018448-21/usr/sbin/munin19:00:050
199389910528,50cyclictest15448-21ssh21:38:490
199389910528,50cyclictest15448-21ssh21:38:490
19956991042,71cyclictest28353-21apt-get22:54:522
19956991042,71cyclictest28353-21apt-get22:54:522
199389910321,55cyclictest9620-21rm22:22:440
199389910321,55cyclictest9620-21rm22:22:440
199389910234,20cyclictest6101-21apt-get19:59:480
199389910227,40cyclictest352-21runrttasks22:09:520
199389910226,31cyclictest0-21swapper/023:19:480
199389910226,31cyclictest0-21swapper/023:19:480
199389910223,33cyclictest16716-21unixbench_multi20:30:150
199389910223,33cyclictest16716-21unixbench_multi20:30:150
199389910025,40cyclictest24995-21/usr/sbin/munin21:54:490
199389910024,30cyclictest0-21swapper/020:14:390
19938999836,38cyclictest14202-21ssh22:30:090
19938999833,28cyclictest11846-21grep00:15:000
19938999822,35cyclictest0-21swapper/023:00:010
19938999822,35cyclictest0-21swapper/023:00:010
19938999744,41cyclictest12611-21apt-get20:19:540
19938999726,29cyclictest8787-21/usr/sbin/munin23:15:030
19938999633,27cyclictest25923-21/usr/sbin/munin19:20:060
19938999633,27cyclictest25923-21/usr/sbin/munin19:20:060
19938999630,27cyclictest0-21swapper/023:25:070
19938999620,41cyclictest27362-21diskmemload23:34:380
19938999529,27cyclictest4562-21rm00:02:340
19938999528,28cyclictest6794-21rm22:18:070
19938999527,22cyclictest18626-21seq20:37:400
19938999527,22cyclictest18626-21seq20:37:400
19938999525,47cyclictest0-21swapper/022:01:110
19938999523,45cyclictest30278-21rm23:50:220
19938999523,45cyclictest30278-21rm23:50:220
19938999523,32cyclictest22745-21/usr/sbin/munin22:44:480
19938999523,32cyclictest22745-21/usr/sbin/munin22:44:480
19938999522,31cyclictest27998-21/usr/sbin/munin19:25:150
19938999520,38cyclictest24792-21if_eth020:55:440
19938999519,43cyclictest12703-21rm22:28:050
19956999459,11cyclictest181rcu_preempt19:27:442
19956999451,13cyclictest181rcu_preempt22:15:032
19956999448,12cyclictest181rcu_preempt22:00:002
1995099942,69cyclictest29698-21tr22:56:001
1995099942,69cyclictest29698-21tr22:56:001
19938999433,27cyclictest20412-21apt-get20:44:510
19938999431,29cyclictest18454-21ssh21:43:570
19938999428,31cyclictest26003-21/usr/sbin/munin20:59:490
19938999428,26cyclictest1781-21ntp_states19:45:080
19938999424,32cyclictest19884-21/usr/sbin/munin21:45:120
19938999423,29cyclictest21714-21munin-node19:09:490
19938999419,28cyclictest24228-21ssh21:53:270
19963999312,70cyclictest4943-21chrt22:15:323
19956999371,16cyclictest23299-21sh22:46:022
19956999371,16cyclictest23299-21sh22:46:022
19938999333,18cyclictest7129-21apt-get21:24:540
19938999333,18cyclictest7129-21apt-get21:24:540
19938999332,29cyclictest32044-21ssh22:05:170
19938999328,27cyclictest5352-21ssh23:09:470
19938999325,32cyclictest26779-21/usr/sbin/munin22:50:120
19938999325,31cyclictest19676-21snmpd23:05:110
19938999325,31cyclictest19676-21snmpd23:05:110
19938999325,28cyclictest19574-21grep20:40:080
19938999324,30cyclictest29741-21ssh21:07:560
19938999320,37cyclictest8037-21ssh00:09:030
19938999320,37cyclictest8037-21ssh00:09:030
19938999231,24cyclictest21331-21ssh23:35:160
19938999228,29cyclictest839-21rm23:01:060
19938999228,27cyclictest6393-21ssh21:23:290
19938999227,30cyclictest16040-21rm00:21:260
19938999227,27cyclictest1855-21ssh23:58:190
186362928,54sleep118689-21fschecks_count19:00:061
19950999170,11cyclictest4613-21sh22:14:421
19938999130,24cyclictest24329-21taskset19:15:070
19938999130,24cyclictest18640-21sh22:38:010
19938999130,24cyclictest18640-21sh22:38:010
19938999126,25cyclictest24017-21apt-config20:54:400
19938999125,29cyclictest19659-21/usr/sbin/munin22:39:570
19938999125,29cyclictest19659-21/usr/sbin/munin22:39:570
19938999124,30cyclictest14834-21find20:25:120
19950999072,10cyclictest17387-21latency_hist00:24:401
19938999029,26cyclictest24721-21ssh23:41:280
19938999028,25cyclictest22453-21ssh00:32:460
19938999027,28cyclictest11433-21rm21:30:540
19938999022,30cyclictest8961-21/usr/sbin/munin00:10:030
19938999021,21cyclictest1594-21irqstats21:15:030
19956998966,16cyclictest19926-21cyclictest20:04:552
19956998939,10cyclictest181rcu_preempt23:05:122
19956998939,10cyclictest181rcu_preempt23:05:122
19950998923,40cyclictest31-21ksoftirqd/120:25:051
19938998929,21cyclictest28190-21sh23:47:530
19938998925,28cyclictest20623-21/usr/sbin/munin19:05:030
19938998925,27cyclictest20056-21ssh00:29:230
19938998925,26cyclictest30663-21sh21:09:510
19956998855,11cyclictest181rcu_preempt22:23:392
19956998855,11cyclictest181rcu_preempt22:23:392
19950998846,19cyclictest0-21swapper/122:16:091
19938998826,22cyclictest0-21swapper/019:55:030
19938998822,26cyclictest32038-21cut19:39:410
19938998821,29cyclictest28349-21apt-get19:29:500
19956998725,32cyclictest0-21swapper/222:52:202
19950998722,34cyclictest31-21ksoftirqd/122:04:101
19938998726,25cyclictest8297-21uname20:05:110
19950998645,9cyclictest181rcu_preempt23:28:011
19938998625,23cyclictest10981-21seq20:14:430
19950998559,7cyclictest181rcu_preempt22:26:141
19950998547,8cyclictest181rcu_preempt19:24:411
19950998535,24cyclictest31-21ksoftirqd/123:20:071
19950998535,24cyclictest31-21ksoftirqd/123:20:071
19938998524,24cyclictest32182-21awk19:39:560
19956998450,11cyclictest181rcu_preempt21:55:052
19956998431,29cyclictest0-21swapper/222:10:022
19950998456,20cyclictest10570-21latency21:30:031
19938998419,26cyclictest0-21swapper/019:50:120
19938998419,26cyclictest0-21swapper/019:50:120
1848028426,36sleep20-21swapper/218:59:592
19956998358,8cyclictest181rcu_preempt19:34:492
19956998348,13cyclictest181rcu_preempt20:57:582
19956998341,18cyclictest181rcu_preempt21:37:002
19956998341,18cyclictest181rcu_preempt21:37:002
19950998339,32cyclictest18065-21find20:35:081
19950998339,32cyclictest18065-21find20:35:081
1995699822,45cyclictest27362-21diskmemload21:20:032
19950998258,8cyclictest181rcu_preempt23:08:321
19950998258,8cyclictest181rcu_preempt23:08:321
19956998134,17cyclictest181rcu_preempt22:09:282
19956998133,13cyclictest181rcu_preempt23:15:492
19950998157,7cyclictest181rcu_preempt23:15:141
19950998152,10cyclictest181rcu_preempt19:40:161
19950998149,9cyclictest181rcu_preempt00:31:321
19956998061,14cyclictest19926-21cyclictest23:54:382
19956998061,14cyclictest19926-21cyclictest23:54:382
19956998047,11cyclictest181rcu_preempt22:25:032
19956998025,32cyclictest0-21swapper/220:35:002
19956998025,32cyclictest0-21swapper/220:35:002
19950998059,7cyclictest181rcu_preempt19:12:101
19950998047,10cyclictest181rcu_preempt19:46:531
19950998024,33cyclictest31-21ksoftirqd/120:10:131
19950998015,37cyclictest31-21ksoftirqd/123:50:131
19950998015,37cyclictest31-21ksoftirqd/123:50:131
19950998014,31cyclictest31-21ksoftirqd/119:55:061
19956997959,8cyclictest181rcu_preempt21:49:542
19956997952,12cyclictest181rcu_preempt21:10:282
19956997949,8cyclictest181rcu_preempt19:04:512
19956997939,12cyclictest181rcu_preempt21:00:152
19956997938,8cyclictest181rcu_preempt00:00:132
19956997930,13cyclictest181rcu_preempt23:26:042
19950997957,14cyclictest3972-21latency_hist21:19:401
19950997944,8cyclictest181rcu_preempt21:27:421
19950997944,8cyclictest181rcu_preempt21:27:421
1995099794,35cyclictest31-21ksoftirqd/120:00:121
19950997942,10cyclictest181rcu_preempt22:24:211
19950997942,10cyclictest181rcu_preempt22:24:211
19950997917,36cyclictest31-21ksoftirqd/119:34:401
19950997912,41cyclictest31-21ksoftirqd/120:09:001
19956997856,16cyclictest20926-21latency00:30:052
19956997850,9cyclictest181rcu_preempt00:04:522
19956997850,9cyclictest181rcu_preempt00:04:522
19956997847,9cyclictest181rcu_preempt19:10:012
19956997836,14cyclictest15850-21ssh00:20:532
1995099788,40cyclictest31-21ksoftirqd/122:43:381
1995099788,40cyclictest31-21ksoftirqd/122:43:381
19950997813,40cyclictest31-21ksoftirqd/123:35:141
266962770,9sleep10-21swapper/122:50:111
19956997757,15cyclictest19926-21cyclictest19:59:402
19956997754,14cyclictest200-21systemd-journal23:14:372
19956997751,9cyclictest18194-21ssh21:43:112
19956997743,9cyclictest27362-21diskmemload00:17:552
19950997747,7cyclictest181rcu_preempt23:45:221
19950997745,7cyclictest181rcu_preempt00:04:401
19950997740,9cyclictest30289-21kworker/u17:2-ext4-rsv-conversion23:34:391
1995099773,63cyclictest23302-21df22:46:021
1995099773,63cyclictest23302-21df22:46:021
19950997718,29cyclictest31-21ksoftirqd/121:35:141
19950997718,29cyclictest31-21ksoftirqd/121:35:141
19950997713,31cyclictest31-21ksoftirqd/121:14:411
19956997653,8cyclictest19022-21ssh21:44:482
19956997641,10cyclictest181rcu_preempt22:32:192
19956997638,8cyclictest181rcu_preempt22:35:022
19956997638,8cyclictest181rcu_preempt22:35:022
19950997653,7cyclictest181rcu_preempt23:40:441
19950997650,14cyclictest181rcu_preempt22:36:131
19950997650,14cyclictest181rcu_preempt22:36:131
19950997648,7cyclictest181rcu_preempt00:10:111
19950997641,8cyclictest181rcu_preempt20:32:471
19950997641,8cyclictest181rcu_preempt20:32:471
1995099763,42cyclictest31-21ksoftirqd/120:45:341
19956997532,8cyclictest181rcu_preempt20:40:072
19956997525,21cyclictest0-21swapper/223:45:102
19956997524,14cyclictest0-21swapper/223:20:042
19956997524,14cyclictest0-21swapper/223:20:042
19950997559,10cyclictest304-21kworker/u17:2-rpciod21:45:091
19950997557,12cyclictest28523-21ssh21:05:151
19950997548,7cyclictest181rcu_preempt21:59:311
1995099752,32cyclictest31-21ksoftirqd/123:14:371
19950997516,36cyclictest31-21ksoftirqd/120:55:461
19950997515,38cyclictest31-21ksoftirqd/122:09:391
316152740,8sleep30-21swapper/321:10:163
19956997451,7cyclictest181rcu_preempt23:34:402
19956997443,9cyclictest181rcu_preempt22:40:032
19956997443,9cyclictest181rcu_preempt22:40:032
19956997440,7cyclictest181rcu_preempt20:00:142
19956997440,15cyclictest181rcu_preempt20:10:012
19956997440,10cyclictest181rcu_preempt23:57:452
19956997434,16cyclictest181rcu_preempt23:01:152
19956997429,32cyclictest7369-21tune2fs21:25:002
19956997429,32cyclictest7369-21tune2fs21:25:002
19956997427,21cyclictest0-21swapper/200:25:152
1995099744,37cyclictest31-21ksoftirqd/121:50:141
19950997441,8cyclictest181rcu_preempt00:05:021
19950997441,8cyclictest181rcu_preempt00:05:021
19950997440,8cyclictest181rcu_preempt19:39:381
19950997438,9cyclictest4765-21kworker/u19:0-rpciod00:27:241
19950997438,8cyclictest181rcu_preempt23:58:441
19956997339,16cyclictest181rcu_preempt23:37:392
19956997337,10cyclictest181rcu_preempt19:49:412
19956997331,31cyclictest25835-21modprobe23:44:372
19956997323,19cyclictest0-21swapper/221:32:532
19950997344,8cyclictest181rcu_preempt19:05:091
19950997342,8cyclictest181rcu_preempt22:30:101
19950997341,12cyclictest31-21ksoftirqd/120:50:151
19956997251,8cyclictest181rcu_preempt19:15:132
19956997249,8cyclictest181rcu_preempt21:17:082
19956997247,10cyclictest181rcu_preempt20:45:472
19956997238,7cyclictest181rcu_preempt19:31:362
19950997255,11cyclictest297-21systemd-logind20:19:381
19950997241,22cyclictest13948-21sh20:24:381
19950997231,10cyclictest20374-21taskset20:44:211
19956997143,9cyclictest181rcu_preempt20:29:572
19956997143,9cyclictest181rcu_preempt20:29:572
1995099712,37cyclictest31-21ksoftirqd/100:15:061
19956997045,9cyclictest181rcu_preempt20:54:392
19956997036,9cyclictest181rcu_preempt20:25:122
19956997027,9cyclictest181rcu_preempt00:12:192
19950997053,11cyclictest297-21systemd-logind21:04:391
19956996940,13cyclictest181rcu_preempt19:50:472
19956996940,13cyclictest181rcu_preempt19:50:472
19956996937,16cyclictest181rcu_preempt20:17:552
19950996940,9cyclictest181rcu_preempt23:04:341
1995099693,34cyclictest6313-21ssh21:23:151
19956996835,7cyclictest181rcu_preempt20:20:062
19950996843,7cyclictest181rcu_preempt19:15:121
19950996823,10cyclictest0-21swapper/121:39:541
19956996742,12cyclictest181rcu_preempt21:06:152
19956996723,22cyclictest0-21swapper/219:20:052
19956996723,22cyclictest0-21swapper/219:20:052
19950996732,8cyclictest181rcu_preempt19:50:021
19950996732,8cyclictest181rcu_preempt19:50:021
47862660,5sleep30-21swapper/319:55:023
1995699662,28cyclictest32415-21if_eth019:40:022
1995099669,28cyclictest31-21ksoftirqd/119:19:541
1995099669,28cyclictest31-21ksoftirqd/119:19:541
1996399659,34cyclictest0-21swapper/322:46:013
1996399659,34cyclictest0-21swapper/322:46:013
1837226410,34sleep30-21swapper/318:59:443
1996399639,38cyclictest0-21swapper/322:56:023
1996399639,38cyclictest0-21swapper/322:56:023
19963995928,22cyclictest2516-21chrt23:04:533
19963995928,22cyclictest2516-21chrt23:04:533
19963995742,10cyclictest451rcuc/321:55:113
19963995738,12cyclictest451rcuc/322:33:293
19963995437,10cyclictest451rcuc/320:39:253
19963995437,10cyclictest451rcuc/320:39:253
19963995337,11cyclictest451rcuc/319:37:343
1996399532,36cyclictest13860-21chrt00:18:393
19963995239,9cyclictest451rcuc/319:19:523
19963995239,9cyclictest451rcuc/319:19:523
1996399523,36cyclictest23352-21chrt00:34:403
1996399523,34cyclictest18895-21chrt21:44:403
1996399523,33cyclictest0-21swapper/323:54:393
1996399523,33cyclictest0-21swapper/323:54:393
1996399522,32cyclictest26036-21chrt20:59:543
1996399512,35cyclictest23105-21chrt20:50:103
98542507,11sleep30-21swapper/320:10:103
19963995037,9cyclictest451rcuc/319:41:473
19963995036,9cyclictest451rcuc/320:15:083
19963995034,9cyclictest451rcuc/320:21:393
1996399502,35cyclictest16545-21taskset23:28:353
202502490,5sleep30-21swapper/319:04:523
19963994935,9cyclictest451rcuc/322:05:033
1996399492,33cyclictest25994-21chrt23:44:403
1996399492,31cyclictest18280-21chrt00:25:123
19963994833,10cyclictest451rcuc/320:41:203
1996399483,29cyclictest28190-21chrt19:27:443
19963994832,11cyclictest451rcuc/319:47:103
19963994831,10cyclictest451rcuc/321:46:313
19963994831,10cyclictest451rcuc/320:05:013
1996399473,35cyclictest849-21chrt23:01:083
1996399473,33cyclictest14538-21taskset21:36:143
1996399473,33cyclictest14538-21taskset21:36:143
1996399473,29cyclictest20110-21chrt22:40:083
1996399473,29cyclictest20110-21chrt22:40:083
1996399472,33cyclictest4025-21chrt21:19:403
1996399472,29cyclictest10733-21taskset22:24:523
19963994634,8cyclictest451rcuc/323:24:023
19963994634,8cyclictest451rcuc/323:24:023
1996399463,34cyclictest17316-21chrt23:29:533
1996399463,33cyclictest9696-21chrt00:10:213
1996399463,33cyclictest8734-21chrt23:15:023
1996399463,33cyclictest2711-21chrt23:59:553
19963994632,10cyclictest451rcuc/320:29:263
1996399462,33cyclictest18016-21taskset22:36:153
1996399462,33cyclictest18016-21taskset22:36:153
1996399462,31cyclictest22509-21chrt21:50:033
1996399462,30cyclictest6732-21taskset21:24:263
1996399454,32cyclictest24108-21taskset19:15:013
1996399453,33cyclictest7706-21chrt22:19:543
1996399453,33cyclictest7706-21chrt22:19:543
1996399453,32cyclictest14834-21taskset00:20:013
19963994532,9cyclictest451rcuc/322:12:253
1996399452,29cyclictest29431-21chrt21:07:033
1996399452,28cyclictest0-21swapper/323:54:413
1996399443,31cyclictest20672-21chrt23:35:033
1996399443,25cyclictest24811-21chrt20:55:453
19963994431,9cyclictest451rcuc/320:01:073
1996399442,33cyclictest30190-21taskset19:34:373
1996399442,29cyclictest21713-21chrt19:09:423
1996399433,31cyclictest12251-21taskset21:33:143
1996399433,30cyclictest7302-21taskset00:06:563
1996399433,30cyclictest7302-21taskset00:06:563
1996399432,32cyclictest28146-21chrt21:59:563
1996399432,31cyclictest16058-21chrt20:30:013
1996399432,31cyclictest16058-21chrt20:30:013
19963994311,27cyclictest0-21swapper/320:45:033
1996399423,30cyclictest25979-21taskset22:49:573
1996399422,32cyclictest5640-21taskset23:09:593
1996399412,30cyclictest28229-21taskset23:48:003
1996399403,28cyclictest7235-21chrt21:24:553
1996399403,28cyclictest7235-21chrt21:24:553
1996399392,27cyclictest2859-21taskset19:49:523
1996399392,27cyclictest2859-21taskset19:49:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional