You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-15 - 04:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sun Nov 15, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17034214810,23sleep00-21swapper/019:04:200
185129911132,26cyclictest20830-21kworker/u17:1+rpciod22:56:070
185129911132,26cyclictest20830-21kworker/u17:1+rpciod22:56:070
185129910934,18cyclictest2243-21ssh00:04:230
185129910934,18cyclictest2243-21ssh00:04:230
185169910749,50cyclictest1154-21idleruntime-cro19:53:591
185259910573,20cyclictest11106-21/usr/sbin/munin20:24:282
18516991013,59cyclictest21034-21apt-get20:54:141
185129910137,25cyclictest21149-21taskset21:54:200
185129910136,22cyclictest0-21swapper/019:59:100
185129910131,27cyclictest0-21swapper/021:24:020
185129910035,28cyclictest22130-21latency_hist23:44:020
185129910035,28cyclictest22130-21latency_hist23:44:020
18516999940,11cyclictest0-21swapper/100:08:591
18516999940,11cyclictest0-21swapper/100:08:591
1851699993,40cyclictest26906-21modprobe19:33:591
1851699983,42cyclictest28329-21cpu23:54:151
1851699983,42cyclictest28329-21cpu23:54:151
1851699982,40cyclictest9293-21latency_hist20:19:011
18512999838,23cyclictest11700-21ps20:24:320
18512999837,28cyclictest25630-21fw_packets23:49:220
18516999738,8cyclictest0-21swapper/121:19:021
18516999738,8cyclictest0-21swapper/121:19:021
1851699973,39cyclictest971-21ssh22:14:261
18516999728,14cyclictest0-21swapper/119:14:091
18512999738,23cyclictest8546-21sh22:26:340
18512999738,23cyclictest8546-21sh22:26:340
18512999738,21cyclictest19670-21seq20:49:270
1851699964,41cyclictest357-21runrttasks22:11:291
18516999639,15cyclictest0-21swapper/120:38:591
1851699963,43cyclictest19543-21kernelversion20:49:251
1851699963,35cyclictest28826-21ssh22:59:381
1851699954,37cyclictest2030-21wc23:09:321
1851699953,36cyclictest5390-21switchtime20:04:341
1851699953,33cyclictest31582-21fschecks_count23:59:231
1851699952,38cyclictest19636-21grep23:39:221
1851699952,38cyclictest19636-21grep23:39:221
18516999518,34cyclictest19482-21mailstats22:44:321
18512999532,27cyclictest32375-21sh22:13:160
18512999532,26cyclictest6593-21open_files21:29:320
18512999530,29cyclictest18352-21rm23:37:170
1851699944,35cyclictest32055-21latency_hist19:49:011
1851699944,35cyclictest10723-21timerwakeupswit22:29:391
1851699944,33cyclictest8050-21awk23:19:321
1851699943,39cyclictest24721-21chrt21:04:311
1851699943,38cyclictest6382-21latency_hist22:24:021
1851699943,32cyclictest31614-21grep23:04:351
1851699943,32cyclictest31614-21grep23:04:351
18512999447,39cyclictest31339-21apt-get23:59:120
18512999436,18cyclictest26562-21awk19:29:380
1851699934,32cyclictest7827-21/usr/sbin/munin00:14:341
1851699933,37cyclictest28706-21latency_hist19:39:021
1851699932,41cyclictest6561-21ssh21:29:291
18512999336,18cyclictest25880-21diskmemload00:29:020
18512999336,18cyclictest25880-21diskmemload00:29:020
18512999332,24cyclictest23514-21ntp_states21:00:150
18512999332,24cyclictest23514-21ntp_states21:00:150
18512999332,19cyclictest21446-21kernelversion20:55:020
1851699924,44cyclictest6750-21apt-get22:24:151
1851699924,44cyclictest6750-21apt-get22:24:151
1851699924,33cyclictest14240-21latency_hist20:34:031
1851699923,42cyclictest17831-21latency_hist21:49:021
1851699923,40cyclictest6494-21fschecks_time20:09:211
1851699923,40cyclictest6494-21fschecks_time20:09:211
1851699923,40cyclictest357-21runrttasks23:34:131
1851699923,38cyclictest15756-21/usr/sbin/munin22:39:251
1851699923,35cyclictest27672-21latency_hist22:59:031
1851699923,35cyclictest27672-21latency_hist22:59:031
18516999229,12cyclictest0-21swapper/121:59:021
1851699922,38cyclictest4023-21unixbench_singl21:24:381
1851699922,38cyclictest13549-21munin-run00:23:591
18512999234,21cyclictest21141-21latency19:14:270
18512999232,27cyclictest15881-21wc21:44:350
18512999228,25cyclictest0-21swapper/023:24:330
1851699914,36cyclictest5247-21if_eth000:09:231
1851699914,36cyclictest15464-21sendmail-msp22:38:591
1851699914,36cyclictest15464-21sendmail-msp22:38:591
1851699914,32cyclictest357-21runrttasks22:50:571
18516999134,15cyclictest0-21swapper/120:20:241
1851699913,39cyclictest16544-21apt-get00:29:181
1851699913,39cyclictest16544-21apt-get00:29:181
1851699913,37cyclictest13020-21unixbench_singl21:39:371
1851699913,37cyclictest13020-21unixbench_singl21:39:371
18512999134,17cyclictest20942-21cat00:34:330
18512999133,24cyclictest30587-21ssh21:14:500
18512999133,24cyclictest30587-21ssh21:14:500
18512999130,26cyclictest13299-21proc_pri20:29:300
18512999121,52cyclictest0-21swapper/022:19:020
170002919,57sleep217003-21acpi19:04:092
1851699903,36cyclictest15956-21/usr/sbin/munin20:39:181
1851699903,35cyclictest17934-21/usr/sbin/munin21:49:361
1851699903,34cyclictest25605-21fschecks_time23:49:221
1851699902,37cyclictest18365-21latency_hist19:09:041
1851699902,35cyclictest357-21runrttasks00:25:501
1851699902,35cyclictest357-21runrttasks00:25:501
18512999038,18cyclictest32081-21grep19:49:020
18512999037,19cyclictest28505-21/usr/sbin/munin23:54:200
18512999037,19cyclictest28505-21/usr/sbin/munin23:54:200
18512999036,21cyclictest13586-21grep23:29:210
18512999032,22cyclictest13713-21rm21:41:270
18512999032,22cyclictest13713-21rm21:41:270
18512999031,22cyclictest19950-21sh22:44:510
18525998960,19cyclictest26861-21ssh22:04:032
1851699893,38cyclictest11879-21find20:24:351
1851699893,35cyclictest29515-21latency19:40:291
1851699893,34cyclictest357-21runrttasks19:59:361
1851699893,31cyclictest24266-21latency_hist21:04:201
1851699893,31cyclictest24266-21latency_hist21:04:201
18512998933,22cyclictest1214-21latency_hist19:54:020
18525998869,14cyclictest6957-21cpu22:24:442
18525998869,14cyclictest6957-21cpu22:24:442
18525998860,19cyclictest357-21runrttasks20:34:322
1851699883,34cyclictest15988-21idleruntime-cro23:33:591
18512998839,21cyclictest26471-21ssh21:09:190
18512998832,24cyclictest12941-21ssh22:34:200
18512998832,24cyclictest12941-21ssh22:34:200
18512998832,23cyclictest7121-21grep20:09:350
18512998832,23cyclictest7121-21grep20:09:350
18512998827,24cyclictest0-21swapper/020:19:220
18512998823,25cyclictest0-21swapper/023:44:270
1851699874,35cyclictest384-21sshd20:46:391
1851699873,36cyclictest9218-21fschecks_count21:34:231
1851699873,36cyclictest357-21runrttasks23:48:111
1851699873,33cyclictest357-21runrttasks21:12:581
18512998760,20cyclictest16544-21apt-get00:29:160
18512998760,20cyclictest16544-21apt-get00:29:160
18512998759,19cyclictest30783-21apt-get23:04:140
18512998759,19cyclictest30783-21apt-get23:04:140
18512998735,43cyclictest13412-21ssh00:23:380
18512998733,24cyclictest1297-21cat23:09:100
18512998732,24cyclictest23863-21sh22:52:260
18525998655,21cyclictest13176-21latency_hist23:29:032
1851699863,35cyclictest13178-21latency_hist23:29:031
18512998650,22cyclictest15974-21apt-get20:39:120
18512998641,41cyclictest0-21swapper/023:14:100
18512998635,19cyclictest24733-21fschecks_count21:04:310
18512998634,19cyclictest19202-21taskset21:50:000
18512998632,19cyclictest7393-21rm00:13:380
18512998631,21cyclictest27372-21rm22:04:240
18525998558,13cyclictest0-21swapper/223:49:022
1851699854,34cyclictest2645-21munin-run21:23:591
18512998539,37cyclictest24539-21ssh21:59:290
18512998539,36cyclictest22425-21sort19:19:150
18512998533,27cyclictest10515-21sh21:35:550
18512998533,22cyclictest11436-21ssh22:31:160
18512998533,20cyclictest17826-21fschecks_time20:44:210
18516998449,26cyclictest5322-21grep23:14:341
18512998465,11cyclictest7876-21apt-get20:14:100
18512998434,41cyclictest4852-21ssh22:20:120
18512998433,23cyclictest16977-21rm22:39:550
18512998432,22cyclictest29055-21rm23:00:060
18512998431,19cyclictest1-21systemd19:59:000
18525998353,20cyclictest4755-21latency_hist00:09:032
18525998353,20cyclictest4755-21latency_hist00:09:032
1851699833,32cyclictest357-21runrttasks22:06:201
18512998347,26cyclictest29026-21/usr/sbin/munin19:40:400
18512998332,22cyclictest14887-21open_files20:34:300
18525998257,17cyclictest11936-21sh21:39:102
18525998257,17cyclictest11936-21sh21:39:102
18525998257,16cyclictest25880-21diskmemload00:34:242
18516998220,25cyclictest21078-21unixbench_multi00:34:351
18512998246,26cyclictest19305-21/usr/sbin/munin19:09:330
18512998236,22cyclictest4838-21sh21:26:480
18512998230,20cyclictest0-21swapper/000:14:370
18512998229,19cyclictest0-21swapper/019:24:230
18512998227,26cyclictest0-21swapper/023:19:330
18516998061,13cyclictest18663-21/usr/sbin/munin19:09:391
1851699804,31cyclictest357-21runrttasks19:19:541
1852599799,29cyclictest29361-21head23:54:372
1852599799,29cyclictest29361-21head23:54:372
18525997854,13cyclictest0-21swapper/223:34:022
18525997852,18cyclictest15248-21sh21:44:212
1851699784,34cyclictest357-21runrttasks22:00:461
18512997821,24cyclictest0-21swapper/019:34:070
18525997754,12cyclictest0-21swapper/221:59:012
18525997754,12cyclictest0-21swapper/220:44:212
18525997752,17cyclictest25880-21diskmemload22:30:332
18525997748,20cyclictest4105-21latency_hist23:14:012
18525997725,15cyclictest181rcu_preempt22:23:592
18516997763,9cyclictest0-21swapper/119:58:451
18512997750,18cyclictest4664-21apt-get20:04:110
18532997610,40cyclictest0-21swapper/323:59:043
18532997610,40cyclictest0-21swapper/323:59:043
18525997656,13cyclictest278-21dbus-daemon22:59:012
18525997656,13cyclictest278-21dbus-daemon22:59:012
18525997654,16cyclictest18500-21cyclictest22:14:342
18525997653,13cyclictest0-21swapper/219:14:022
18525997651,12cyclictest0-21swapper/220:44:032
18525997560,7cyclictest0-21swapper/223:59:042
18525997559,9cyclictest0-21swapper/223:39:002
18525997557,14cyclictest18500-21cyclictest22:44:202
18525997551,16cyclictest22300-21netstat22:49:272
18525997551,13cyclictest0-21swapper/219:59:022
18525997547,20cyclictest0-21swapper/223:08:592
18525997547,20cyclictest0-21swapper/223:08:592
18525997451,13cyclictest0-21swapper/220:54:592
18525997445,20cyclictest25880-21diskmemload23:50:452
18525997441,20cyclictest9442-21latency_hist20:19:032
1853299733,41cyclictest0-21swapper/322:24:163
1853299733,41cyclictest0-21swapper/322:24:163
18525997333,10cyclictest0-21swapper/221:26:102
18525997252,15cyclictest18500-21cyclictest21:34:222
18525997251,11cyclictest0-21swapper/219:59:222
18525997241,19cyclictest22319-21kworker/u17:0-xprtiod21:33:002
18525997221,19cyclictest25880-21diskmemload00:26:502
18525997221,19cyclictest25880-21diskmemload00:26:502
18525997211,20cyclictest8837-21seq23:20:452
1834327217,35sleep30-21swapper/319:08:593
1852599712,35cyclictest26425-21apt-get21:09:192
18525997119,11cyclictest181rcu_preempt00:19:142
18525997054,11cyclictest203-21systemd-journal00:34:002
18525997054,11cyclictest203-21systemd-journal00:34:002
18525997052,8cyclictest0-21swapper/220:14:022
18525997052,8cyclictest0-21swapper/220:14:022
18525997051,11cyclictest0-21swapper/219:19:022
18525997047,13cyclictest0-21swapper/222:09:252
18525997046,13cyclictest0-21swapper/220:19:222
18516997029,15cyclictest181rcu_preempt19:28:591
1852599697,25cyclictest16371-21ntpq22:39:312
18525996949,13cyclictest13665-21timerandwakeup22:34:362
18525996949,13cyclictest13665-21timerandwakeup22:34:362
1852599687,24cyclictest338-21if_eth021:19:222
18525996853,8cyclictest0-21swapper/219:24:172
18525996848,10cyclictest0-21swapper/221:54:022
1852599679,27cyclictest27134-21date19:34:032
1852599674,32cyclictest28151-21/usr/sbin/munin19:34:342
18525996741,14cyclictest0-21swapper/223:44:022
18525996741,14cyclictest0-21swapper/223:44:022
1852599673,52cyclictest22396-21apt-get20:59:152
1852599673,52cyclictest22396-21apt-get20:59:152
18525996723,8cyclictest181rcu_preempt20:49:212
18525996643,14cyclictest8405-21ntp_kernel_err00:14:272
1852599663,45cyclictest29045-21apt-get19:39:162
18525996623,31cyclictest1-21systemd00:14:002
18525996550,11cyclictest18500-21cyclictest21:14:242
18525996550,11cyclictest18500-21cyclictest21:14:242
18525996542,16cyclictest18500-21cyclictest22:59:232
18525996514,22cyclictest29865-21cat22:09:032
1853299643,42cyclictest31515-21taskset23:59:193
1853299642,40cyclictest0-21swapper/320:54:543
18525996450,8cyclictest0-21swapper/223:16:022
18525996440,14cyclictest0-21swapper/220:34:022
18525996410,21cyclictest31879-21seq19:47:392
18525996339,13cyclictest0-21swapper/219:19:202
1853299624,40cyclictest932-21kworker/3:1+events21:29:293
18525996141,10cyclictest0-21swapper/221:04:322
18525996135,12cyclictest0-21swapper/220:09:022
1853299603,42cyclictest18193-21chrt20:44:323
1853299603,39cyclictest19998-21taskset23:39:333
1853299603,39cyclictest19998-21taskset23:39:333
1853299594,39cyclictest22976-21chrt19:19:323
1853299593,38cyclictest14362-21taskset00:24:243
1853299593,38cyclictest14362-21taskset00:24:243
18532995930,20cyclictest0-21swapper/321:48:593
1853299573,40cyclictest19160-21taskset22:44:243
1853299573,39cyclictest13852-21chrt22:34:393
1853299573,39cyclictest13852-21chrt22:34:393
1853299572,40cyclictest26071-21taskset22:55:143
1853299572,40cyclictest26071-21taskset22:55:143
18525995739,13cyclictest0-21swapper/219:49:222
1853299564,39cyclictest11687-21taskset21:39:013
1853299564,36cyclictest29370-21chrt19:40:253
1853299563,39cyclictest5854-21taskset21:29:033
1853299563,36cyclictest932-21kworker/3:1+events21:14:293
1853299563,36cyclictest932-21kworker/3:1+events21:14:293
1853299563,35cyclictest25289-21chrt22:00:133
1853299562,42cyclictest6263-21taskset22:24:003
1853299562,37cyclictest1446-21chrt23:09:163
1853299553,37cyclictest18696-21chrt23:38:153
1853299553,36cyclictest19440-21chrt19:09:353
1853299553,34cyclictest6374-21taskset20:09:173
1853299553,34cyclictest6374-21taskset20:09:173
1853299553,34cyclictest27786-21chrt22:04:353
1853299552,38cyclictest0-21swapper/321:09:143
1853299552,36cyclictest751-21chrt19:49:413
1853299552,36cyclictest17624-21chrt00:30:163
1853299552,36cyclictest17624-21chrt00:30:163
1853299543,39cyclictest4586-21taskset23:14:193
1853299543,38cyclictest25618-21taskset23:49:223
1853299543,37cyclictest23672-21chrt21:58:593
1853299543,35cyclictest5420-21chrt20:04:353
1853299543,33cyclictest386-21chrt22:14:033
1853299534,37cyclictest13696-21chrt23:29:233
1853299534,31cyclictest2163-21chrt00:04:223
1853299534,31cyclictest2163-21chrt00:04:223
1853299533,37cyclictest11431-21taskset00:19:263
1853299533,33cyclictest24542-21chrt21:04:213
1853299532,35cyclictest31194-21chrt19:44:303
1853299523,37cyclictest30956-21chrt23:04:183
1853299523,37cyclictest30956-21chrt23:04:183
1853299523,37cyclictest20998-21chrt00:34:343
1853299523,36cyclictest24309-21taskset19:24:263
1853299523,34cyclictest28507-21taskset19:38:593
1853299523,34cyclictest13062-21chrt21:39:383
1853299523,34cyclictest13062-21chrt21:39:383
1853299523,33cyclictest8802-21chrt20:14:353
1853299522,36cyclictest11617-21chrt22:31:473
1853299522,33cyclictest13801-21chrt20:30:393
1853299513,35cyclictest29632-21chrt23:01:423
1853299513,34cyclictest11525-21chrt23:24:533
1853299512,38cyclictest7963-21taskset23:19:303
1853299512,35cyclictest21941-21taskset22:49:183
1853299512,35cyclictest18747-21chrt21:49:353
1853299505,30cyclictest9686-21chrt20:19:213
1853299503,35cyclictest10097-21taskset00:17:473
1853299503,34cyclictest5067-21chrt00:09:203
1853299503,34cyclictest19466-21taskset20:49:223
1853299503,34cyclictest14358-21chrt20:34:153
18532995028,12cyclictest0-21swapper/319:14:223
1853299502,37cyclictest26030-21chrt19:29:283
1853299502,36cyclictest1559-21chrt19:54:183
1853299502,34cyclictest2079-21chrt21:22:273
18042500,2sleep3451rcuc/322:15:093
1853299494,33cyclictest17540-21chrt20:44:033
1853299493,32cyclictest24323-21chrt23:47:113
1853299484,29cyclictest3217-21chrt19:59:213
1853299473,32cyclictest15778-21taskset22:39:133
1853299433,29cyclictest12501-21taskset20:29:013
18532994314,16cyclictest0-21swapper/320:59:523
18532994314,16cyclictest0-21swapper/320:59:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional