You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 08:27
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Nov 19, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12943216311,120sleep113016-21sort19:05:241
12943216311,120sleep113016-21sort19:05:241
136209912142,72cyclictest0-21swapper/021:15:130
13620991212,43cyclictest25391-21apt-get22:10:090
136289911056,47cyclictest29464-21latency_hist00:04:382
136209910846,54cyclictest17271-21latency_hist22:49:370
136209910748,53cyclictest0-21swapper/023:00:150
136209910748,53cyclictest0-21swapper/023:00:150
13620991072,95cyclictest26002-21kworker/u17:3+rpciod00:04:570
13620991043,25cyclictest6919-21processes00:20:080
13620991042,34cyclictest3151-21sed20:14:370
13620991042,34cyclictest3151-21sed20:14:370
13620991034,29cyclictest13619-21cut19:09:390
13620991034,29cyclictest13619-21cut19:09:390
13620991032,25cyclictest9286-21grep00:24:580
13620991032,21cyclictest28389-21acpi22:14:470
136209910226,12cyclictest0-21swapper/021:41:050
136209910132,62cyclictest0-21swapper/023:39:580
136209910128,8cyclictest0-21swapper/022:05:100
136209910128,8cyclictest0-21swapper/022:05:100
136209910037,56cyclictest0-21swapper/023:50:000
13620991003,22cyclictest21662-21taskset22:55:330
136209910030,8cyclictest0-21swapper/023:06:030
13620999942,51cyclictest4731-21sh00:16:060
1362099993,41cyclictest810-21fschecks_time00:10:000
1362099993,41cyclictest810-21fschecks_time00:10:000
1362099993,30cyclictest1010-21ssh23:15:160
1362099993,24cyclictest6751-21apt-get20:25:110
1362099993,22cyclictest29666-21fschecks_count23:10:050
13620999929,8cyclictest0-21swapper/021:09:350
13620999919,23cyclictest1530-21apt-key20:09:390
1362099984,24cyclictest15212-21latency_hist19:14:370
13620999838,52cyclictest0-21swapper/023:30:090
1362099983,28cyclictest19901-21open_files19:25:080
1362099983,24cyclictest29285-21fschecks_time19:54:580
1362099983,24cyclictest29285-21fschecks_time19:54:580
13620999830,59cyclictest0-21swapper/000:39:380
13620999825,8cyclictest0-21swapper/019:40:040
1362099982,28cyclictest32388-21ssh21:27:390
13620999822,21cyclictest17401-21cat19:19:380
125172980,6sleep20-21swapper/223:35:102
13623999747,14cyclictest21031-21diskmemload00:05:181
1362099973,32cyclictest30938-21ls20:00:020
1362099973,30cyclictest19746-21sh22:00:080
13620999723,13cyclictest0-21swapper/020:49:380
13620999721,8cyclictest0-21swapper/022:25:050
13628999658,32cyclictest207-21systemd-journal21:34:332
13620999629,7cyclictest0-21swapper/020:29:590
13620999625,8cyclictest0-21swapper/021:04:350
1362099962,22cyclictest6285-21taskset22:30:310
13620999622,21cyclictest4265-21grep21:34:580
13620999619,11cyclictest207-21systemd-journal20:39:350
1362099953,28cyclictest1889-21cat22:24:380
1362099953,28cyclictest1889-21cat22:24:380
13620999527,61cyclictest0-21swapper/022:49:530
13620999527,61cyclictest0-21swapper/022:49:530
13620999527,10cyclictest0-21swapper/021:30:130
1362099943,30cyclictest5168-21grep20:19:590
1362099943,29cyclictest6300-21ssh23:25:020
1362099943,22cyclictest14541-21idleruntime-cro23:39:340
13620999426,7cyclictest0-21swapper/023:24:340
13620999421,15cyclictest0-21swapper/020:40:070
13620999421,15cyclictest0-21swapper/020:40:070
13620999335,51cyclictest0-21swapper/020:59:350
1362099933,28cyclictest12377-21ssh22:40:130
1362099933,25cyclictest28129-21/usr/sbin/munin19:50:110
1362099933,24cyclictest26927-21cstates23:59:530
1362099933,24cyclictest26196-21latency19:45:030
13620999328,60cyclictest0-21swapper/019:24:380
13620999327,12cyclictest0-21swapper/020:49:550
13620999326,9cyclictest0-21swapper/021:19:570
13620999326,9cyclictest0-21swapper/021:19:570
13620999324,8cyclictest0-21swapper/021:54:580
13628999242,44cyclictest18481-21switchtime19:20:222
1362099924,32cyclictest24084-21latency_hist21:14:370
1362099924,32cyclictest24084-21latency_hist21:14:370
1362099923,30cyclictest24082-21fschecks_time23:55:000
1362099923,26cyclictest3526-21cut20:15:010
1362899913,33cyclictest32363-21ls22:20:132
1362899913,33cyclictest32363-21ls22:20:132
1362899913,27cyclictest13359-21fschecks_time21:50:002
1362899913,27cyclictest13359-21fschecks_time21:50:002
1362899912,28cyclictest8828-21fschecks_time22:35:002
1362899912,28cyclictest8828-21fschecks_time22:35:002
1362099913,34cyclictest21031-21diskmemload21:45:120
1362099913,26cyclictest21943-21/usr/sbin/munin19:30:160
1362099913,24cyclictest13683-21sh21:50:070
1362099913,24cyclictest13683-21sh21:50:070
1362099912,27cyclictest10526-21rm22:37:510
1362099912,27cyclictest10526-21rm22:37:510
1362099903,24cyclictest17814-21/usr/sbin/munin23:44:590
1362099903,24cyclictest17814-21/usr/sbin/munin23:44:590
1362899894,48cyclictest25391-21apt-get22:10:312
1362899893,30cyclictest648-21latency_hist21:29:382
13628998927,45cyclictest0-21swapper/221:59:382
1362899892,32cyclictest13135-21ssh00:30:402
1362099892,23cyclictest12197-21grep00:29:590
1362099883,26cyclictest22738-21/usr/sbin/munin19:34:580
1362899873,28cyclictest14628-21latency_hist20:49:372
13628998659,18cyclictest1270-21apt-key20:09:372
13628998647,32cyclictest0-21swapper/223:40:062
13628998639,41cyclictest2890-21/usr/sbin/munin23:20:042
1362899863,36cyclictest8058-21grep21:40:132
13628998633,45cyclictest3175-21latency_hist20:14:382
13628998633,45cyclictest3175-21latency_hist20:14:382
1362899862,27cyclictest3542-21fschecks_time20:15:012
1362899862,27cyclictest16729-21ls20:55:002
13623998665,11cyclictest14526-21cron23:39:341
1362899858,25cyclictest28481-21grep21:20:132
1362899858,25cyclictest28481-21grep21:20:132
13628998546,33cyclictest0-21swapper/223:25:122
13628998539,40cyclictest2698-21latency_hist23:19:382
1362899853,32cyclictest355-21runrttasks23:32:452
1362899853,30cyclictest24182-21/usr/sbin/munin19:40:082
1362899843,29cyclictest22129-21/usr/sbin/munin22:04:542
1362899843,29cyclictest22129-21/usr/sbin/munin22:04:542
1362899843,26cyclictest6726-21/usr/sbin/munin22:30:372
1362899843,26cyclictest23780-21/usr/sbin/munin23:54:562
1362899843,26cyclictest11762-21fschecks_count22:39:592
1362899842,41cyclictest1-21systemd00:19:352
1362899842,23cyclictest22758-21tune2fs19:34:582
13628998338,39cyclictest21481-21cstates21:09:532
13628998338,39cyclictest21481-21cstates21:09:532
1362899833,27cyclictest27401-21acpi19:49:482
1362899833,25cyclictest355-21runrttasks20:28:412
13628998331,47cyclictest0-21swapper/221:45:002
1362899832,28cyclictest24686-21ssh21:15:002
1362899832,22cyclictest20439-21apt-get22:54:522
13628998241,35cyclictest21031-21diskmemload23:46:572
13628998241,35cyclictest21031-21diskmemload23:46:572
13628998239,37cyclictest17357-21apt-key19:19:382
13628998234,40cyclictest0-21swapper/200:05:182
1362899823,37cyclictest20628-21head21:05:142
1362899823,29cyclictest19424-21fschecks_time22:00:012
1362899823,27cyclictest355-21runrttasks19:13:522
1362899823,26cyclictest2256-21ssh22:24:472
1362899823,21cyclictest25807-21apt-get19:44:542
13628998230,47cyclictest0-21swapper/223:10:062
13623998259,17cyclictest6513-21apt-key20:24:391
1362899819,21cyclictest30888-21cut20:00:012
1362899814,36cyclictest7254-21tail00:20:122
1362899814,25cyclictest29575-21/usr/sbin/munin19:55:072
1362899814,25cyclictest29575-21/usr/sbin/munin19:55:072
13628998137,37cyclictest16019-21diskmemload22:46:242
1362899813,33cyclictest18077-21ntp_states22:50:062
1362899813,33cyclictest18077-21ntp_states22:50:062
13628998124,51cyclictest9845-21sh00:25:102
1362899812,22cyclictest13102-21latency_hist20:44:372
1362899812,22cyclictest13102-21latency_hist20:44:372
1362899803,24cyclictest23427-21/usr/sbin/munin23:00:102
1362899803,24cyclictest23427-21/usr/sbin/munin23:00:102
1362899803,24cyclictest19686-21/usr/sbin/munin19:25:012
1362899803,24cyclictest1588-21/usr/sbin/munin00:10:152
1362899803,24cyclictest1588-21/usr/sbin/munin00:10:152
1362899803,23cyclictest9384-21seq20:30:402
1362899803,22cyclictest21145-21/usr/sbin/munin23:49:582
1362899802,34cyclictest6715-21sh21:39:352
1362899802,28cyclictest14781-21ssh00:34:492
13623998060,14cyclictest5145-21apt-key22:29:391
1362899793,33cyclictest18239-21fschecks_count20:59:572
1362899793,31cyclictest355-21runrttasks20:54:382
1362899793,26cyclictest20947-21/usr/sbin/munin19:30:122
1362899793,21cyclictest355-21runrttasks20:19:472
1362899792,26cyclictest28428-21sh22:14:512
13623997959,14cyclictest15687-21sh00:35:131
1362899783,27cyclictest28596-21rm23:08:202
1362899783,24cyclictest9952-21/usr/sbin/munin20:35:112
13623997764,9cyclictest58-21kcompactd021:09:581
13623997764,9cyclictest58-21kcompactd021:09:581
13623997760,9cyclictest25181-21latency_hist22:09:381
13623997760,9cyclictest25181-21latency_hist22:09:381
13623997659,11cyclictest21812-21latency_hist22:04:371
13623997656,14cyclictest1697-21latency_hist22:24:391
13623997656,14cyclictest1697-21latency_hist22:24:391
13623997656,13cyclictest791-21timerandwakeup23:15:141
1362899742,27cyclictest13724-21cat19:09:402
1362899742,27cyclictest13724-21cat19:09:402
13623997458,10cyclictest13611-21cyclictest00:00:331
13623997452,16cyclictest5895-21munin-plugin-st00:19:361
1363399733,23cyclictest13692-21chrt21:50:083
1363399733,23cyclictest13692-21chrt21:50:083
13623997358,10cyclictest16310-21kworker/u19:0-rpciod00:31:561
13623997357,10cyclictest6233-21ntp_offset22:30:281
13623997352,14cyclictest691-21cron19:14:351
13623997352,14cyclictest24265-21latency23:55:031
13623997352,13cyclictest871-21sendmail-mta00:22:211
1363399726,34cyclictest0-21swapper/300:04:573
1363399723,26cyclictest28738-21chrt19:54:353
13623997256,12cyclictest13611-21cyclictest23:01:391
13623997256,12cyclictest13611-21cyclictest23:01:391
13623997252,14cyclictest9491-21kernelversion00:25:021
13623997156,9cyclictest3027-21kworker/u19:2-rpciod20:44:591
13623997156,8cyclictest0-21swapper/119:20:221
1363399703,26cyclictest15070-21chrt23:39:573
13623997056,9cyclictest10214-21kworker/u19:1-events_unbound20:05:091
13623997052,11cyclictest15733-21kworker/u19:1-rpciod22:55:051
1363399693,26cyclictest19490-21chrt21:04:373
1363399693,23cyclictest29058-21chrt23:09:343
1363399692,28cyclictest26439-21chrt21:17:543
1363399692,25cyclictest9405-21chrt21:43:443
13623996949,14cyclictest3003-21idleruntime-cro20:14:341
13623996949,14cyclictest3003-21idleruntime-cro20:14:341
1363399683,30cyclictest28107-21chrt22:14:353
1363399683,27cyclictest31296-21chrt22:19:383
1363399683,25cyclictest9083-21taskset23:29:563
1363399683,24cyclictest13146-21taskset00:30:423
13623996852,10cyclictest21031-21diskmemload21:29:421
13623996850,12cyclictest13611-21cyclictest19:35:041
13623996847,15cyclictest13785-21sh22:43:321
1363399673,26cyclictest23022-21chrt23:53:223
1363399673,25cyclictest461-21taskset00:09:433
1363399673,25cyclictest461-21taskset00:09:433
1363399673,24cyclictest32375-21chrt23:14:533
1363399673,23cyclictest19055-21chrt23:45:323
1363399673,23cyclictest19055-21chrt23:45:323
1363399672,23cyclictest31533-21chrt21:25:163
13623996753,10cyclictest58-21kcompactd019:55:071
13623996753,10cyclictest58-21kcompactd019:55:071
13623996752,8cyclictest0-21swapper/123:29:571
13623996751,9cyclictest0-21swapper/122:50:081
13623996751,9cyclictest0-21swapper/122:50:081
13623996748,13cyclictest29906-21latency23:10:101
1363399664,24cyclictest22922-21taskset22:05:113
1363399664,24cyclictest22922-21taskset22:05:113
1363399663,24cyclictest13090-21chrt22:41:343
1363399663,24cyclictest1005-21taskset22:22:423
1363399663,24cyclictest1005-21taskset22:22:423
1363399663,23cyclictest26497-21chrt19:45:123
1363399663,23cyclictest25273-21chrt23:56:183
1363399663,23cyclictest15523-21chrt19:14:433
1363399663,22cyclictest6287-21chrt20:24:363
1363399662,27cyclictest16382-21chrt00:36:503
13623996648,12cyclictest1-21systemd23:43:551
13623996636,21cyclictest16625-21grep20:54:581
13623996632,25cyclictest28375-21cat21:20:121
13623996632,25cyclictest28375-21cat21:20:121
13623996612,40cyclictest4058-21timerandwakeup20:15:141
1363399653,25cyclictest27964-21chrt00:00:233
1363399653,25cyclictest10715-21chrt21:45:103
1363399652,24cyclictest24863-21chrt23:01:073
1363399652,24cyclictest24863-21chrt23:01:073
13623996549,9cyclictest0-21swapper/121:35:081
13623996548,11cyclictest15455-21rm22:45:151
13623996535,23cyclictest30813-21kworker/u18:0+nfsiod21:40:131
1362399653,47cyclictest25028-21wc21:15:101
13623996534,22cyclictest13506-21cut21:50:031
13623996534,22cyclictest13506-21cut21:50:031
13623996533,23cyclictest6768-21cat23:25:121
1363399643,26cyclictest24223-21chrt21:14:383
1363399643,26cyclictest24223-21chrt21:14:383
1363399643,25cyclictest16543-21chrt22:47:543
1363399643,24cyclictest29285-21taskset21:22:203
1363399643,24cyclictest29285-21taskset21:22:203
1363399643,24cyclictest21527-21chrt22:03:493
1363399642,25cyclictest21111-21chrt19:29:583
1363399642,25cyclictest12799-21chrt20:42:553
1363399642,25cyclictest12799-21chrt20:42:553
1363399642,24cyclictest13833-21chrt19:09:493
13623996449,10cyclictest13611-21cyclictest22:38:201
13623996449,10cyclictest13611-21cyclictest22:38:201
1362399644,39cyclictest31451-21unixbench_multi21:25:151
1362399644,17cyclictest18066-21apt-get20:59:521
1362399643,45cyclictest27143-21latency_hist19:49:381
1363399633,26cyclictest17243-21taskset21:55:233
1363399633,25cyclictest2992-21chrt21:32:463
1363399633,25cyclictest14276-21chrt20:46:073
1363399633,24cyclictest14228-21chrt23:38:473
1363399633,24cyclictest10753-21taskset22:38:303
1363399633,24cyclictest10753-21taskset22:38:303
1363399632,25cyclictest4622-21taskset00:15:473
1363399632,25cyclictest30111-21taskset19:56:383
1363399632,25cyclictest30111-21taskset19:56:383
1363399632,24cyclictest7608-21chrt23:26:563
13623996348,10cyclictest13611-21cyclictest23:49:591
13623996348,10cyclictest12498-21kworker/u17:0-rpciod21:56:321
13623996347,10cyclictest12181-21ssh21:48:051
1362399633,44cyclictest26597-21df_inode23:04:561
1362399633,40cyclictest19508-21df_inode19:24:561
1362399633,36cyclictest10820-21timerwakeupswit20:35:131
13623996330,24cyclictest28072-21modprobe22:14:351
13623996329,25cyclictest9692-21modprobe20:34:351
1363399623,24cyclictest8015-21chrt20:28:453
1363399623,24cyclictest32264-21chrt20:04:473
13623996248,10cyclictest13611-21cyclictest22:15:001
1362399624,44cyclictest355-21runrttasks23:49:221
1362399624,44cyclictest355-21runrttasks23:49:221
1362399623,43cyclictest355-21runrttasks19:43:011
1362399623,42cyclictest492-21/usr/sbin/munin00:09:541
1362399623,42cyclictest492-21/usr/sbin/munin00:09:541
1362399623,38cyclictest27611-21fschecks_count19:49:591
1363399613,25cyclictest23659-21taskset19:36:343
1362399613,38cyclictest8082-21modprobe20:29:351
13623996126,25cyclictest5679-21sed23:24:371
1363399603,24cyclictest18705-21chrt22:50:383
1363399603,24cyclictest18705-21chrt22:50:383
1363399603,24cyclictest11515-21taskset00:29:243
1363399602,24cyclictest2676-21taskset20:10:353
1363399602,24cyclictest2676-21taskset20:10:353
1363399602,24cyclictest11146-21chrt20:37:403
13623996044,9cyclictest0-21swapper/119:30:011
1362399604,41cyclictest15018-21fschecks_count20:49:591
1362399603,43cyclictest30901-21fschecks_time20:00:011
1362399602,49cyclictest12017-21latency20:40:051
1362399602,49cyclictest12017-21latency20:40:051
1363399592,25cyclictest4509-21taskset23:21:383
1363399592,24cyclictest18972-21taskset19:23:183
13623995944,10cyclictest58-21kcompactd019:14:431
13623995917,27cyclictest19670-21acpi21:04:481
13633995818,8cyclictest35822chrt22:25:503
1363399572,24cyclictest17713-21taskset20:59:023
1363399563,22cyclictest16081-21taskset20:53:403
1218725417,22sleep30-21swapper/319:04:573
1218725417,22sleep30-21swapper/319:04:573
13633995316,12cyclictest0-21swapper/320:29:473
1363399512,27cyclictest895-21kworker/3:1+events19:59:513
1363399483,30cyclictest20438-21chrt22:54:473
1363399482,29cyclictest29761-21chrt23:10:073
1363399473,30cyclictest19839-21taskset21:04:583
1363399454,28cyclictest7779-21taskset00:21:383
1363399453,30cyclictest4279-21taskset20:15:333
1363399453,29cyclictest24736-21taskset19:40:063
1363399453,27cyclictest895-21kworker/3:1+events21:34:533
1363399452,30cyclictest6983-21chrt22:31:173
1363399443,29cyclictest19505-21taskset19:24:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional