You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-24 - 16:07
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Apr 24, 2025 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11568213112,81sleep310962-21/usr/sbin/munin19:05:413
1101321196,15sleep10-21swapper/119:05:061
124369910218,75cyclictest21957-21apt-get23:14:460
124369910063,12cyclictest181rcu_preempt21:29:560
12444999984,8cyclictest0-21swapper/123:15:141
1103129811,58sleep211038-21wc19:05:192
12461999782,8cyclictest0-21swapper/321:09:543
12461999782,8cyclictest0-21swapper/321:09:543
1246199954,81cyclictest22728-21ssh23:15:123
12456999475,14cyclictest14517-21sh23:59:312
12436999271,13cyclictest2791-21kworker/u17:021:39:310
12436999267,17cyclictest11539-21sh23:54:310
12456999170,13cyclictest19314-21idleruntime-cro00:09:312
12436999165,19cyclictest2867-21latency_hist00:39:340
12456999013,43cyclictest39-21ksoftirqd/222:17:182
12456998917,44cyclictest39-21ksoftirqd/219:27:102
12456998917,44cyclictest39-21ksoftirqd/219:27:102
12436998944,11cyclictest181rcu_preempt22:40:100
12436998856,22cyclictest14625-21ssh22:59:540
1234828815,37sleep00-21swapper/019:09:330
12436998762,13cyclictest0-21swapper/000:24:340
12436998755,13cyclictest181rcu_preempt20:39:450
12436998732,19cyclictest181rcu_preempt22:50:590
12461998661,17cyclictest12427-21cyclictest20:10:013
1245699869,35cyclictest39-21ksoftirqd/222:04:172
1245699869,35cyclictest39-21ksoftirqd/222:04:172
12444998563,11cyclictest0-21swapper/120:24:341
1243699852,54cyclictest31062-21sed22:30:040
12456998451,10cyclictest181rcu_preempt00:15:492
12436998413,64cyclictest14563-21modprobe23:59:310
12436998350,9cyclictest181rcu_preempt00:00:040
12436998347,17cyclictest181rcu_preempt20:09:500
1243699834,45cyclictest20859-21munin-run19:34:310
1243699834,45cyclictest20859-21munin-run19:34:310
12461998261,14cyclictest0-21swapper/323:59:313
12444998263,7cyclictest0-21swapper/123:59:491
12436998247,10cyclictest181rcu_preempt22:34:440
12436998246,14cyclictest181rcu_preempt20:14:450
12461998111,30cyclictest1955-21sed23:35:113
12456998117,28cyclictest39-21ksoftirqd/221:10:132
12456998117,28cyclictest39-21ksoftirqd/221:10:132
12436998140,32cyclictest358-21runrttasks19:19:360
12436998140,32cyclictest358-21runrttasks19:19:360
12461998053,17cyclictest0-21swapper/300:09:523
12461998053,17cyclictest0-21swapper/300:09:523
1245699809,27cyclictest39-21ksoftirqd/221:57:532
1245699802,61cyclictest21957-21apt-get23:14:462
12444998058,16cyclictest8687-21sh22:49:491
12444998057,17cyclictest1-21systemd20:08:381
12444998057,14cyclictest0-21swapper/120:10:101
1244499803,67cyclictest18869-21df21:09:541
1244499803,67cyclictest18869-21df21:09:541
12436998055,18cyclictest21941-21cron23:14:310
12436998055,18cyclictest21941-21cron23:14:310
12436998010,57cyclictest21638-21sort21:14:440
172262790,22sleep2381ktimers/222:05:072
172262790,22sleep2381ktimers/222:05:072
12456997939,31cyclictest15330-21systemctl19:15:082
12456997939,31cyclictest15330-21systemctl19:15:082
12444997936,30cyclictest1952-21idleruntime-cro20:19:321
12436997955,15cyclictest11857-21latency22:54:580
1243699794,65cyclictest14811-21proc_pri22:00:110
1243699794,65cyclictest14811-21proc_pri22:00:110
12436997943,16cyclictest181rcu_preempt22:20:070
12436997943,16cyclictest181rcu_preempt22:20:070
12436997930,14cyclictest181rcu_preempt22:29:340
12436997930,14cyclictest181rcu_preempt22:29:340
12444997858,14cyclictest4671-21sendmail_mailqu23:40:061
12444997858,14cyclictest4671-21sendmail_mailqu23:40:061
12436997856,8cyclictest181rcu_preempt23:41:070
12436997856,8cyclictest181rcu_preempt23:41:070
12436997852,17cyclictest0-21swapper/021:04:580
12436997848,9cyclictest181rcu_preempt20:50:550
12436997848,9cyclictest181rcu_preempt20:50:550
1243699782,48cyclictest358-21runrttasks19:40:020
1245699779,38cyclictest39-21ksoftirqd/222:35:202
12456997757,13cyclictest0-21swapper/223:10:032
12456997757,13cyclictest0-21swapper/223:10:032
12456997755,16cyclictest25735-21fschecks_count23:19:542
12456997755,16cyclictest25735-21fschecks_count23:19:542
12456997755,12cyclictest0-21swapper/223:30:052
12456997750,16cyclictest0-21swapper/221:09:312
12456997734,18cyclictest26664-21ssh21:22:452
12456997713,30cyclictest39-21ksoftirqd/221:37:022
12444997763,7cyclictest0-21swapper/119:15:021
12444997763,7cyclictest0-21swapper/119:15:021
12444997761,10cyclictest14271-21cstates21:59:561
12444997761,10cyclictest14271-21cstates21:59:561
12444997757,14cyclictest576-21cron00:19:311
12436997745,12cyclictest181rcu_preempt21:09:520
12436997745,12cyclictest181rcu_preempt21:09:520
1243699773,37cyclictest29013-21/usr/sbin/munin20:00:130
12456997659,11cyclictest16502-21idleruntime-cro21:04:312
12456997656,7cyclictest181rcu_preempt21:43:562
12456997656,14cyclictest32239-21timerandwakeup20:10:092
12456997623,27cyclictest39-21ksoftirqd/219:10:012
12456997615,36cyclictest39-21ksoftirqd/200:35:572
12456997614,28cyclictest39-21ksoftirqd/220:45:232
12456997614,28cyclictest39-21ksoftirqd/220:45:232
12456997613,26cyclictest39-21ksoftirqd/220:53:532
12456997613,26cyclictest39-21ksoftirqd/220:53:532
12444997657,11cyclictest0-21swapper/122:40:071
12436997640,13cyclictest181rcu_preempt00:14:570
1243699763,36cyclictest24646-21/usr/sbin/munin21:20:020
12436997618,26cyclictest181rcu_preempt22:06:570
12436997618,26cyclictest181rcu_preempt22:06:570
12461997560,10cyclictest12427-21cyclictest23:49:493
1246199752,53cyclictest29749-21cstates21:29:453
12456997550,11cyclictest181rcu_preempt22:40:062
12456997515,36cyclictest39-21ksoftirqd/222:56:522
12456997513,36cyclictest39-21ksoftirqd/222:22:402
12456997513,36cyclictest39-21ksoftirqd/222:22:402
12456997513,29cyclictest39-21ksoftirqd/219:51:522
12456997512,32cyclictest39-21ksoftirqd/222:13:262
12456997512,29cyclictest381ktimers/223:49:582
12456997510,35cyclictest39-21ksoftirqd/220:21:302
12444997552,17cyclictest12417-21unixbench_multi22:55:091
12444997546,20cyclictest27881-21sh00:24:311
12436997554,14cyclictest60-21kcompactd000:26:270
12436997554,14cyclictest60-21kcompactd000:26:270
12436997554,12cyclictest0-21swapper/023:20:090
12436997554,12cyclictest0-21swapper/023:20:090
12436997545,11cyclictest5714-21fschecks_time20:29:570
12436997545,10cyclictest181rcu_preempt21:39:440
12436997541,14cyclictest181rcu_preempt00:10:020
12436997541,14cyclictest181rcu_preempt00:10:020
1243699753,53cyclictest31155-21grep23:30:020
1243699753,38cyclictest19214-21cstates22:09:450
12461997445,20cyclictest23128-21grep00:15:053
1246199744,46cyclictest17148-21latency_hist23:04:343
1246199744,45cyclictest16200-21latency_hist19:19:343
1246199744,45cyclictest16200-21latency_hist19:19:343
12461997426,35cyclictest25422-21cat21:20:083
12461997415,24cyclictest358-21runrttasks22:47:583
12461997415,24cyclictest358-21runrttasks22:47:583
12456997452,8cyclictest181rcu_preempt23:41:162
12456997452,8cyclictest181rcu_preempt23:41:162
12456997452,8cyclictest181rcu_preempt21:29:562
12456997450,19cyclictest12427-21cyclictest20:24:552
12456997449,13cyclictest0-21swapper/200:30:032
12456997445,20cyclictest8572-21modprobe23:49:312
1245699744,25cyclictest1261-21fschecks_time23:34:572
12456997437,7cyclictest181rcu_preempt19:47:082
12456997421,36cyclictest39-21ksoftirqd/221:17:392
12456997415,32cyclictest39-21ksoftirqd/223:01:162
12456997414,29cyclictest39-21ksoftirqd/221:45:372
12436997443,11cyclictest181rcu_preempt19:13:280
12461997360,9cyclictest22947-21kworker/u17:3-rpciod21:40:063
12461997347,12cyclictest0-21swapper/319:59:473
1246199734,45cyclictest358-21runrttasks20:42:033
12461997343,18cyclictest0-21swapper/322:39:313
12461997330,31cyclictest0-21swapper/321:04:473
12461997323,40cyclictest21678-21ssh21:14:493
12461997319,18cyclictest18109-21diskmemload23:11:513
12461997319,18cyclictest18109-21diskmemload23:11:513
12456997350,8cyclictest181rcu_preempt23:08:562
12456997350,8cyclictest181rcu_preempt23:08:562
12456997350,13cyclictest0-21swapper/219:39:312
12456997348,19cyclictest30411-21kernelversion22:29:532
12456997343,8cyclictest181rcu_preempt00:24:522
12456997343,8cyclictest181rcu_preempt00:24:522
12456997341,23cyclictest29706-21sh22:29:312
12456997341,23cyclictest29706-21sh22:29:312
12456997332,16cyclictest181rcu_preempt22:48:462
12456997332,16cyclictest181rcu_preempt22:48:462
12456997331,25cyclictest39-21ksoftirqd/200:03:362
12456997312,35cyclictest39-21ksoftirqd/221:52:332
12444997353,12cyclictest0-21swapper/120:00:081
12444997341,23cyclictest31128-21grep00:30:041
12444997327,35cyclictest28890-21modprobe19:59:311
12436997349,14cyclictest0-21swapper/019:50:130
1243699733,37cyclictest8295-21ssh21:49:460
12436997332,16cyclictest181rcu_preempt23:46:520
12436997327,25cyclictest181rcu_preempt21:28:180
12436997327,25cyclictest181rcu_preempt21:28:180
12461997256,9cyclictest0-21swapper/321:54:313
12461997255,9cyclictest0-21swapper/320:20:083
12461997252,14cyclictest6413-21unixbench_singl20:30:123
12461997252,14cyclictest20274-21threads19:30:113
12461997252,14cyclictest20274-21threads19:30:113
12461997250,14cyclictest8418-21idleruntime-cro20:39:323
12456997253,13cyclictest22953-21fschecks_count19:39:552
12456997249,13cyclictest0-21swapper/219:30:122
12456997249,13cyclictest0-21swapper/219:30:122
12456997239,12cyclictest181rcu_preempt19:21:232
12456997239,12cyclictest181rcu_preempt19:21:232
12456997223,36cyclictest20388-21wc00:10:012
12456997223,36cyclictest20388-21wc00:10:012
12456997212,36cyclictest39-21ksoftirqd/220:32:242
12444997259,8cyclictest0-21swapper/123:09:541
12444997259,8cyclictest0-21swapper/123:09:541
12444997254,10cyclictest1-21systemd23:35:081
12444997253,12cyclictest0-21swapper/121:54:511
12444997252,14cyclictest316-21in:imuxsock21:09:311
12444997250,16cyclictest12427-21cyclictest20:44:551
12444997250,16cyclictest12427-21cyclictest20:44:551
12436997251,15cyclictest358-21runrttasks00:29:560
12436997241,15cyclictest181rcu_preempt22:14:480
12436997241,10cyclictest181rcu_preempt19:14:560
12436997241,10cyclictest181rcu_preempt19:14:560
12436997239,13cyclictest181rcu_preempt20:44:580
12436997239,13cyclictest181rcu_preempt20:44:580
12436997235,11cyclictest181rcu_preempt23:05:440
12436997235,11cyclictest181rcu_preempt23:05:440
12436997229,11cyclictest181rcu_preempt21:45:510
12436997228,13cyclictest181rcu_preempt00:06:420
12436997224,8cyclictest181rcu_preempt19:45:110
181382712,6sleep00-21swapper/019:24:560
181382712,6sleep00-21swapper/019:24:560
12461997153,10cyclictest0-21swapper/320:00:233
12461997140,22cyclictest9169-21systemctl22:50:013
12456997150,7cyclictest181rcu_preempt00:21:482
12456997116,25cyclictest39-21ksoftirqd/223:29:092
12444997154,10cyclictest17086-21ssh23:04:341
12444997143,20cyclictest32263-21modprobe21:34:311
12436997142,12cyclictest181rcu_preempt22:49:320
12436997142,12cyclictest181rcu_preempt22:49:320
1246199704,44cyclictest14040-21run-parts22:59:313
1246199704,43cyclictest25307-21grep19:45:123
12461997044,17cyclictest10284-21/usr/sbin/munin20:45:013
12461997044,17cyclictest10284-21/usr/sbin/munin20:45:013
1246199703,45cyclictest22565-21latency_hist19:39:343
1246199703,44cyclictest373-21snmpd20:25:083
1245699708,25cyclictest39-21ksoftirqd/220:55:552
12456997039,12cyclictest181rcu_preempt20:41:232
12456997014,28cyclictest39-21ksoftirqd/220:38:072
12444997012,14cyclictest181rcu_preempt22:19:491
12444997012,14cyclictest181rcu_preempt22:19:491
12436997051,11cyclictest0-21swapper/023:37:020
12436997033,11cyclictest181rcu_preempt20:25:050
12436997027,10cyclictest181rcu_preempt21:54:500
12461996938,20cyclictest0-21swapper/319:20:123
12461996938,20cyclictest0-21swapper/319:20:123
12461996915,25cyclictest13487-21chrt20:54:363
1245699698,29cyclictest39-21ksoftirqd/220:05:592
12456996943,8cyclictest181rcu_preempt22:50:332
12456996913,26cyclictest39-21ksoftirqd/220:04:062
12444996954,9cyclictest1031-21ssh21:35:071
12444996945,15cyclictest22220-21ntp_kernel_pll_21:15:021
1244499693,43cyclictest11726-21latency_hist23:54:341
12436996944,8cyclictest181rcu_preempt20:34:460
12436996931,13cyclictest181rcu_preempt20:59:590
12436996928,12cyclictest181rcu_preempt20:54:500
293392680,8sleep30-21swapper/300:28:163
293392680,8sleep30-21swapper/300:28:163
12461996841,16cyclictest0-21swapper/323:59:553
12461996839,19cyclictest0-21swapper/319:14:333
12461996824,33cyclictest16599-21latency_hist21:04:343
1245699683,35cyclictest39-21ksoftirqd/221:25:302
1245699683,35cyclictest39-21ksoftirqd/221:25:302
12456996811,20cyclictest39-21ksoftirqd/220:14:442
12444996849,12cyclictest0-21swapper/121:20:061
12436996830,20cyclictest27978-21latency19:54:580
12461996750,11cyclictest19015-21ssh23:09:313
12461996750,11cyclictest19015-21ssh23:09:313
12461996745,12cyclictest0-21swapper/322:20:053
12461996745,12cyclictest0-21swapper/322:20:053
1246199674,43cyclictest3499-21/usr/sbin/munin22:40:063
12461996742,19cyclictest18109-21diskmemload23:43:383
12461996742,19cyclictest18109-21diskmemload23:43:383
12461996742,15cyclictest0-21swapper/320:54:313
12461996742,15cyclictest0-21swapper/320:54:313
1246199673,46cyclictest358-21runrttasks00:36:043
1246199673,42cyclictest8039-21latency_hist21:49:353
12461996720,36cyclictest19579-21sh22:09:533
12444996748,13cyclictest19562-21latency22:09:531
1244499673,45cyclictest31114-21ps23:30:011
1244499673,44cyclictest25327-21grep19:45:121
12444996725,31cyclictest2696-21ssh00:39:311
12436996739,9cyclictest181rcu_preempt19:35:000
1243699672,33cyclictest3147-21unixbench_multi20:20:080
12461996650,8cyclictest0-21swapper/322:29:443
12461996648,8cyclictest0-21swapper/323:20:103
12461996648,8cyclictest0-21swapper/323:20:103
12461996643,17cyclictest30294-21fschecks_count00:29:493
12461996637,20cyclictest22753-21apt-get19:39:453
12461996634,23cyclictest22209-21cstates22:14:483
12461996622,29cyclictest294-21dbus-daemon00:24:323
12444996650,9cyclictest0-21swapper/119:19:591
12444996650,9cyclictest0-21swapper/119:19:591
12444996649,10cyclictest20462-21kworker/u20:0-rpciod22:28:351
12444996649,10cyclictest20462-21kworker/u20:0-rpciod22:28:351
12444996648,12cyclictest0-21swapper/122:19:321
1244499664,27cyclictest31-21ksoftirqd/122:08:171
1244499664,27cyclictest31-21ksoftirqd/122:08:171
1244499663,44cyclictest12753-21timerandwakeup23:55:081
1244499663,42cyclictest30275-21latency_hist23:29:341
1244499662,48cyclictest12183-21fschecks_time20:49:581
1244499662,48cyclictest12183-21fschecks_time20:49:581
1246199654,33cyclictest358-21runrttasks20:05:083
12461996542,16cyclictest576-21cron20:19:323
12461996540,13cyclictest0-21swapper/321:59:563
12461996540,13cyclictest0-21swapper/321:59:563
1246199653,45cyclictest18764-21timerwakeupswit19:25:083
1246199653,45cyclictest18764-21timerwakeupswit19:25:083
1246199653,39cyclictest358-21runrttasks23:25:533
12444996549,11cyclictest12427-21cyclictest20:40:091
12444996548,9cyclictest0-21swapper/120:59:311
12444996544,11cyclictest0-21swapper/122:34:311
1244499654,23cyclictest31-21ksoftirqd/119:37:371
12444996539,16cyclictest0-21swapper/123:23:141
12444996539,16cyclictest0-21swapper/123:23:141
12444996537,19cyclictest5175-21cron20:29:321
12444996523,31cyclictest3561-21uname21:39:581
12461996450,10cyclictest12427-21cyclictest19:49:583
12461996438,16cyclictest0-21swapper/321:25:193
12461996438,16cyclictest0-21swapper/321:25:193
12461996426,29cyclictest8689-21ssh23:49:333
12456996436,7cyclictest181rcu_preempt19:55:042
12444996448,9cyclictest0-21swapper/100:09:491
12444996448,9cyclictest0-21swapper/100:09:491
12444996447,11cyclictest311-21systemd-logind19:44:331
12444996443,12cyclictest0-21swapper/120:34:341
12444996443,10cyclictest0-21swapper/122:34:551
1244499644,29cyclictest31-21ksoftirqd/119:31:561
1244499644,29cyclictest31-21ksoftirqd/119:31:561
12444996434,22cyclictest27287-21modprobe19:54:311
12444996417,33cyclictest6425-21sh22:44:381
12444996417,33cyclictest6425-21sh22:44:381
1243699644,12cyclictest29438-21sh23:27:340
12461996341,12cyclictest0-21swapper/323:29:513
12461996339,14cyclictest0-21swapper/322:29:333
12461996339,14cyclictest0-21swapper/322:29:333
1246199633,45cyclictest18771-21runrttasks00:07:563
12461996322,34cyclictest18812-21idleruntime-cro22:09:313
12461996322,34cyclictest18812-21idleruntime-cro22:09:313
1244499634,25cyclictest31-21ksoftirqd/123:49:181
1244499633,42cyclictest12820-21fschecks_count19:09:561
1244499633,23cyclictest31-21ksoftirqd/121:01:091
12436996326,15cyclictest181rcu_preempt20:09:340
12461996230,23cyclictest11366-21fschecks_time21:54:513
12444996240,12cyclictest0-21swapper/120:34:551
12444996230,11cyclictest28333-21ssh21:26:501
12444996230,11cyclictest28333-21ssh21:26:501
12444996226,12cyclictest0-21swapper/121:45:521
12444996216,32cyclictest9327-21sh21:50:241
12461996144,9cyclictest26121-21kworker/u17:1-rpciod21:34:553
12444996145,9cyclictest0-21swapper/119:24:551
12444996145,9cyclictest0-21swapper/119:24:551
12444996044,8cyclictest0-21swapper/123:09:331
12444996044,8cyclictest0-21swapper/123:09:331
1244499603,26cyclictest31-21ksoftirqd/100:28:591
1244499603,26cyclictest31-21ksoftirqd/100:28:591
1244499593,9cyclictest0-21swapper/100:08:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional