You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 12:02
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Tue Mar 10, 2026 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12743991234,49cyclictest32387-21ssh00:05:480
12743991144,30cyclictest10841-21fschecks_count22:35:540
127439911421,27cyclictest18361-21latency_hist21:55:360
127439911338,65cyclictest0-21swapper/019:10:580
127439911338,65cyclictest0-21swapper/019:10:580
12743991133,31cyclictest4175-21apt-config22:25:370
12743991133,31cyclictest4175-21apt-config22:25:370
127439911325,10cyclictest0-21swapper/023:15:570
12743991123,38cyclictest352-21runrttasks22:26:180
12743991123,38cyclictest352-21runrttasks22:26:180
12743991118,95cyclictest3131-21apt-get00:10:480
12743991114,41cyclictest23973-21/usr/sbin/munin23:51:030
12743991113,32cyclictest29402-21rm00:00:480
11121211010,71sleep111123-21/usr/sbin/munin19:00:441
12743991092,36cyclictest11190-21cpuspeed23:30:480
127439910846,55cyclictest297-21systemd-logind20:55:340
127439910724,7cyclictest0-21swapper/021:55:570
12743991063,28cyclictest14424-21tune2fs23:35:550
127439910528,13cyclictest0-21swapper/019:40:360
12743991043,38cyclictest21265-21fschecks_time23:47:390
12743991042,21cyclictest9470-21fschecks_time00:20:550
12743991033,42cyclictest13709-21ps19:07:180
12743991033,33cyclictest19809-21apt-get23:45:330
12743991033,32cyclictest18173-21latency_hist19:20:360
127439910319,78cyclictest0-21swapper/023:56:060
127439910319,78cyclictest0-21swapper/023:56:060
12743991024,42cyclictest352-21runrttasks20:11:000
12743991023,33cyclictest17358-21cstates20:55:490
12743991023,33cyclictest17358-21cstates20:55:490
12743991023,32cyclictest8528-21/usr/sbin/munin22:31:480
12743991023,31cyclictest28877-21/usr/sbin/munin19:51:520
12743991023,29cyclictest7428-21rm23:24:310
12743991022,33cyclictest8053-21rm23:25:360
1227121028,75sleep00-21swapper/019:03:030
12743991014,35cyclictest27327-21ssh21:16:000
12743991014,29cyclictest12464-21latency_hist21:45:370
12743991014,22cyclictest15738-21ntp_states00:31:050
12743991014,22cyclictest15738-21ntp_states00:31:050
12743991013,34cyclictest31538-21/usr/sbin/munin23:11:000
12743991013,34cyclictest31068-21unixbench_singl19:56:160
12743991013,34cyclictest22753-21grep19:31:260
12743991013,33cyclictest16908-21fschecks_time22:45:560
12743991013,29cyclictest352-21runrttasks23:08:050
12743991013,29cyclictest352-21runrttasks23:08:050
127439910131,7cyclictest0-21swapper/000:25:590
12743991012,37cyclictest7339-21latency_hist20:25:350
127439910037,55cyclictest0-21swapper/022:05:440
12743991003,41cyclictest30856-21grep21:21:100
12743991003,34cyclictest21521-21ssh22:53:310
12743991003,28cyclictest25227-21grep19:40:550
12743999934,8cyclictest0-21swapper/022:15:360
1274399993,39cyclictest352-21runrttasks22:57:030
1274399993,35cyclictest8878-21latency_hist00:20:350
1274399993,35cyclictest8878-21latency_hist00:20:350
1274399993,33cyclictest6798-21fschecks_count21:35:550
1274399993,32cyclictest4963-21taskset20:16:060
1274399993,29cyclictest25589-21apt-get23:00:480
1274399983,34cyclictest21005-21ssh21:06:190
1274399983,33cyclictest21265-21apt-get19:30:330
1274399983,33cyclictest21265-21apt-get19:30:330
12743999827,9cyclictest0-21swapper/022:15:590
1274399982,37cyclictest9449-21fschecks_time20:30:570
1274399982,33cyclictest8896-21modprobe20:30:330
1274399982,33cyclictest12789-21fschecks_count21:45:550
1274399982,33cyclictest12789-21fschecks_count21:45:550
1274399982,29cyclictest32332-21grep20:01:100
12743999822,12cyclictest0-21swapper/021:30:500
12743999822,12cyclictest0-21swapper/021:30:500
286352970,6sleep30-21swapper/319:51:433
1274399973,37cyclictest24709-21grep21:11:130
1274399973,37cyclictest24709-21grep21:11:130
1274399973,36cyclictest27533-21tail19:46:110
1274399973,36cyclictest27533-21tail19:46:110
1274399973,36cyclictest20117-21unixbench_singl21:01:580
1274399973,33cyclictest10980-21uniq20:35:550
1274399973,32cyclictest13212-21grep20:41:100
1274399973,32cyclictest13212-21grep20:41:100
1274399973,29cyclictest1564-21timerandwakeup21:26:240
1274399973,24cyclictest13531-21latency_hist22:40:370
1274399973,24cyclictest13531-21latency_hist22:40:370
1274399972,26cyclictest19697-21latency_hist19:25:350
12743999625,8cyclictest0-21swapper/022:00:520
1274399952,36cyclictest2441-21latency_hist20:10:360
1274399952,36cyclictest2441-21latency_hist20:10:360
115282958,57sleep211594-21memory19:01:052
1277199942,66cyclictest19809-21apt-get23:45:481
1274399943,30cyclictest14264-21fschecks_time20:45:560
1274399943,30cyclictest14264-21fschecks_time20:45:560
12771999334,19cyclictest181rcu_preempt00:06:531
12771999323,31cyclictest181rcu_preempt21:25:501
1277199923,61cyclictest4347-21apt-get22:25:481
1277199923,61cyclictest4347-21apt-get22:25:481
12794999170,15cyclictest200-21systemd-journal22:07:522
12771999044,15cyclictest0-21swapper/123:52:451
12771998965,8cyclictest181rcu_preempt19:55:371
12771998959,8cyclictest181rcu_preempt22:14:421
12794998859,20cyclictest19798-21latency_hist23:45:362
12771998835,16cyclictest181rcu_preempt23:03:461
12771998753,8cyclictest181rcu_preempt22:55:581
12771998740,13cyclictest352-21runrttasks19:35:481
1279499867,51cyclictest39-21ksoftirqd/220:12:022
1279499862,56cyclictest20889-21ssh23:46:502
12771998641,26cyclictest15262-21ssh21:50:341
12771998641,26cyclictest15262-21ssh21:50:341
12771998633,19cyclictest0-21swapper/120:53:201
12771998543,14cyclictest0-21swapper/121:15:371
12771998543,14cyclictest0-21swapper/121:15:371
12771998538,13cyclictest0-21swapper/121:56:551
12771998467,10cyclictest3113-21acpi00:10:441
12771998466,13cyclictest3475-21timerandwakeup20:11:111
12771998449,17cyclictest20473-21diskmemload22:30:521
12771998441,18cyclictest8374-21kworker/u18:0+rpciod23:42:191
12771998432,10cyclictest0-21swapper/119:19:531
1277199842,44cyclictest25070-21sort19:40:501
235782830,6sleep30-21swapper/322:56:113
1279499838,32cyclictest39-21ksoftirqd/221:47:562
1279499838,32cyclictest39-21ksoftirqd/221:47:562
1279499834,43cyclictest20473-21diskmemload22:26:442
1279499834,43cyclictest20473-21diskmemload22:26:442
12794998318,53cyclictest623-21runrttasks21:25:532
12771998358,9cyclictest181rcu_preempt00:18:351
12771998358,9cyclictest181rcu_preempt00:18:351
12771998352,12cyclictest181rcu_preempt20:25:581
1279499822,36cyclictest39-21ksoftirqd/200:08:092
1279499821,54cyclictest39-21ksoftirqd/222:51:382
12771998232,12cyclictest181rcu_preempt00:02:241
1277199822,54cyclictest19113-21apt-get21:01:261
12771998224,22cyclictest0-21swapper/121:17:551
12771998216,27cyclictest18508-21fschecks_count19:20:541
12794998159,16cyclictest297-21systemd-logind19:50:342
12794998159,16cyclictest297-21systemd-logind19:50:342
1279499812,45cyclictest39-21ksoftirqd/220:08:212
1279499812,45cyclictest39-21ksoftirqd/220:08:212
12771998156,13cyclictest181rcu_preempt00:20:511
12771998149,9cyclictest181rcu_preempt21:31:031
12771998149,9cyclictest181rcu_preempt21:31:031
12771998146,25cyclictest3834-21apt-key22:25:351
12771998146,25cyclictest3834-21apt-key22:25:351
12771998139,11cyclictest181rcu_preempt20:55:481
12771998139,11cyclictest181rcu_preempt20:55:481
12771998136,14cyclictest181rcu_preempt20:00:331
12794998064,10cyclictest297-21systemd-logind23:25:332
12794998059,16cyclictest31833-21sh21:23:492
1279499804,31cyclictest39-21ksoftirqd/223:05:012
12771998053,7cyclictest181rcu_preempt19:10:361
12771998051,15cyclictest181rcu_preempt23:30:351
12771998050,13cyclictest181rcu_preempt00:32:531
12771998050,13cyclictest181rcu_preempt00:32:531
12771998050,11cyclictest181rcu_preempt20:16:101
12771998024,19cyclictest0-21swapper/123:59:421
12771998024,19cyclictest0-21swapper/123:59:421
1250528024,36sleep30-21swapper/319:05:353
12794997956,17cyclictest26524-21systemd-run21:15:342
12794997956,17cyclictest26524-21systemd-run21:15:342
1279499793,39cyclictest39-21ksoftirqd/220:58:242
1279499793,39cyclictest39-21ksoftirqd/220:58:242
12794997933,33cyclictest13746-21modprobe20:45:342
12794997933,33cyclictest13746-21modprobe20:45:342
1279499792,51cyclictest39-21ksoftirqd/221:39:332
12771997948,11cyclictest181rcu_preempt23:23:551
1277199794,41cyclictest28014-21munin-run19:50:351
1277199794,41cyclictest28014-21munin-run19:50:351
1280399782,34cyclictest0-21swapper/319:06:023
1279499788,47cyclictest39-21ksoftirqd/200:00:582
12794997861,12cyclictest12720-21cyclictest19:25:442
12794997861,12cyclictest12720-21cyclictest19:25:442
12794997852,8cyclictest181rcu_preempt20:53:062
12794997849,19cyclictest544-21cron23:05:372
12794997849,19cyclictest544-21cron23:05:372
12794997840,27cyclictest32039-21kworker/u19:2+nfsiod00:32:152
12794997840,27cyclictest32039-21kworker/u19:2+nfsiod00:32:152
1279499782,42cyclictest39-21ksoftirqd/222:11:532
1279499782,35cyclictest39-21ksoftirqd/221:34:472
1279499782,35cyclictest39-21ksoftirqd/221:34:472
12771997861,11cyclictest28709-21kworker/u17:0-xprtiod22:05:361
12771997860,10cyclictest0-21swapper/119:30:371
12771997860,10cyclictest0-21swapper/119:30:371
12771997839,13cyclictest0-21swapper/122:37:581
1277199783,65cyclictest21765-21apt-get19:30:571
12771997829,25cyclictest20473-21diskmemload21:38:521
12771997826,15cyclictest0-21swapper/120:37:451
1279499773,51cyclictest26408-21latency_hist19:45:352
12771997747,14cyclictest181rcu_preempt20:05:381
12771997747,14cyclictest181rcu_preempt20:05:381
12771997736,12cyclictest14450-21ssh23:35:571
1277199773,37cyclictest20473-21diskmemload22:08:051
12771997733,13cyclictest181rcu_preempt21:42:411
12771997732,10cyclictest181rcu_preempt20:42:011
12771997732,10cyclictest181rcu_preempt20:42:011
12771997729,13cyclictest0-21swapper/122:46:111
1279499764,47cyclictest7575-21apt-get22:30:512
12771997615,46cyclictest31-21ksoftirqd/123:07:071
12771997615,46cyclictest31-21ksoftirqd/123:07:071
12794997538,14cyclictest181rcu_preempt23:58:422
12794997538,14cyclictest181rcu_preempt23:58:422
12771997536,8cyclictest22461-21ssh21:07:461
12771997531,14cyclictest181rcu_preempt21:54:511
12771997526,19cyclictest0-21swapper/120:01:031
1279499748,44cyclictest39-21ksoftirqd/221:10:032
1279499747,45cyclictest39-21ksoftirqd/223:52:292
12794997445,8cyclictest181rcu_preempt21:52:432
12794997441,11cyclictest181rcu_preempt22:04:362
1279499742,37cyclictest39-21ksoftirqd/222:20:392
1279499742,37cyclictest39-21ksoftirqd/222:20:392
12794997419,10cyclictest381ktimers/220:30:462
12771997428,19cyclictest181rcu_preempt20:50:041
12771997428,19cyclictest181rcu_preempt20:50:041
1279499735,40cyclictest39-21ksoftirqd/222:44:322
1279499735,40cyclictest39-21ksoftirqd/222:44:322
12794997350,14cyclictest20473-21diskmemload23:11:082
12794997335,28cyclictest25623-21kworker/u19:0+nfsiod00:21:072
1279499733,51cyclictest39-21ksoftirqd/223:32:312
1279499732,33cyclictest39-21ksoftirqd/200:26:212
1279499732,31cyclictest39-21ksoftirqd/222:46:092
12771997344,8cyclictest181rcu_preempt22:18:231
12771997335,9cyclictest181rcu_preempt22:54:051
12771997331,12cyclictest181rcu_preempt23:11:001
12771997330,12cyclictest181rcu_preempt23:34:421
12771997325,13cyclictest181rcu_preempt23:18:241
1279499722,46cyclictest39-21ksoftirqd/222:59:132
12794997213,34cyclictest39-21ksoftirqd/221:17:522
1279499721,32cyclictest39-21ksoftirqd/220:21:052
12771997238,12cyclictest181rcu_preempt22:40:581
12771997238,12cyclictest181rcu_preempt22:40:581
12771997235,9cyclictest181rcu_preempt19:11:471
12771997235,9cyclictest181rcu_preempt19:11:471
12771997234,10cyclictest181rcu_preempt21:22:351
12771997231,14cyclictest181rcu_preempt20:21:101
12771997230,13cyclictest181rcu_preempt00:26:001
1280399718,41cyclictest0-21swapper/322:25:593
1280399718,41cyclictest0-21swapper/322:25:593
1279499715,34cyclictest39-21ksoftirqd/220:00:502
12794997149,15cyclictest12720-21cyclictest21:41:012
12794997136,13cyclictest181rcu_preempt22:39:072
1279499712,45cyclictest39-21ksoftirqd/220:49:562
1279499712,45cyclictest39-21ksoftirqd/220:49:562
1279499712,30cyclictest39-21ksoftirqd/220:35:482
12771997132,9cyclictest181rcu_preempt20:31:071
1279499705,31cyclictest39-21ksoftirqd/219:50:382
12794997053,11cyclictest277-21dbus-daemon00:15:342
12794997041,20cyclictest30851-21threads19:56:142
12794997016,27cyclictest39-21ksoftirqd/223:36:102
12794997014,27cyclictest39-21ksoftirqd/220:29:062
1279499697,25cyclictest39-21ksoftirqd/221:00:512
1279499695,30cyclictest39-21ksoftirqd/222:17:372
1279499695,27cyclictest39-21ksoftirqd/223:16:212
1280399689,50cyclictest0-21swapper/323:45:523
1279499684,30cyclictest39-21ksoftirqd/200:15:502
1279499684,30cyclictest39-21ksoftirqd/200:15:502
1279499682,26cyclictest39-21ksoftirqd/221:57:022
46942670,7sleep30-21swapper/323:20:143
12794996740,18cyclictest352-21runrttasks19:10:412
12794996740,18cyclictest352-21runrttasks19:10:412
1279499672,39cyclictest39-21ksoftirqd/223:28:092
12794996631,26cyclictest5685-21modprobe20:20:332
1279499662,26cyclictest39-21ksoftirqd/219:20:442
1279499647,35cyclictest39-21ksoftirqd/219:06:212
1279499643,38cyclictest18175-21latency_hist19:20:362
1279499642,27cyclictest39-21ksoftirqd/219:31:172
1279499633,42cyclictest23689-21/usr/sbin/munin19:35:562
12803995811,39cyclictest1141-21chrt20:06:133
12803995811,39cyclictest1141-21chrt20:06:133
1280399553,38cyclictest2935-21taskset21:29:433
1280399552,41cyclictest9386-21chrt22:33:093
1280399552,40cyclictest16890-21taskset00:33:043
1280399552,40cyclictest16890-21taskset00:33:043
1280399543,39cyclictest9945-21taskset23:28:123
1280399543,39cyclictest977-21taskset22:20:363
1280399542,35cyclictest0-21swapper/321:00:593
1280399533,38cyclictest17477-21taskset23:40:573
1280399533,37cyclictest31824-21chrt20:00:503
1280399533,37cyclictest13038-21chrt21:46:003
1280399533,37cyclictest13038-21chrt21:46:003
1280399533,36cyclictest2848-21chrt22:22:563
1280399533,36cyclictest2848-21chrt22:22:563
1280399533,36cyclictest11815-21chrt21:44:263
1280399524,35cyclictest15623-21chrt23:37:163
1280399523,38cyclictest5053-21chrt00:13:273
1280399523,37cyclictest29625-21chrt00:00:553
1280399523,37cyclictest21051-21taskset21:06:193
1280399523,35cyclictest13545-21chrt22:40:393
1280399523,35cyclictest13545-21chrt22:40:393
1280399523,34cyclictest26073-21chrt22:07:193
1280399513,39cyclictest3495-21taskset20:11:113
1280399513,33cyclictest23946-21chrt23:51:023
1280399512,38cyclictest32120-21chrt21:24:363
1280399504,36cyclictest26792-21chrt19:45:563
1280399504,36cyclictest26792-21chrt19:45:563
1280399503,37cyclictest16979-21taskset19:15:553
1280399503,36cyclictest18112-21chrt20:56:073
1280399503,36cyclictest18112-21chrt20:56:073
1280399503,36cyclictest14943-21taskset20:46:103
1280399503,36cyclictest14943-21taskset20:46:103
1280399503,35cyclictest5213-21chrt21:32:433
1280399503,35cyclictest5213-21chrt21:32:433
1280399503,35cyclictest28526-21chrt23:05:393
1280399503,35cyclictest28526-21chrt23:05:393
1280399503,35cyclictest20949-21taskset22:51:533
1280399503,35cyclictest13719-21taskset00:27:333
1280399503,34cyclictest7380-21chrt23:24:233
1280399503,29cyclictest21592-21taskset19:30:373
1280399503,29cyclictest21592-21taskset19:30:373
1280399502,35cyclictest20082-21chrt21:57:183
1280399493,34cyclictest21984-21taskset19:31:103
1280399493,33cyclictest16439-21chrt20:51:093
1280399492,36cyclictest32562-21chrt00:05:533
1280399492,34cyclictest16619-21chrt21:51:103
1280399492,32cyclictest24735-21chrt19:40:333
1280399483,34cyclictest18901-21chrt19:21:053
1280399483,34cyclictest13735-21chrt23:35:253
1280399483,34cyclictest11149-21chrt20:35:583
1280399483,33cyclictest26787-21chrt23:01:263
1280399483,31cyclictest25832-21chrt21:13:493
1280399483,31cyclictest25832-21chrt21:13:493
1280399483,31cyclictest1674-21chrt23:15:153
1280399482,35cyclictest5202-21chrt20:16:103
1280399473,35cyclictest28183-21taskset22:11:043
1280399473,31cyclictest28746-21chrt21:18:323
1280399463,34cyclictest27671-21chrt23:56:413
1280399463,34cyclictest27671-21chrt23:56:413
1280399463,31cyclictest17567-21chrt22:46:103
1280399453,33cyclictest8568-21taskset21:38:423
1280399453,32cyclictest8155-21taskset00:18:423
1280399453,32cyclictest8155-21taskset00:18:423
1280399453,28cyclictest11663-21chrt22:36:093
12803994526,12cyclictest0-21swapper/320:45:343
12803994526,12cyclictest0-21swapper/320:45:343
1280399452,34cyclictest6032-21chrt20:20:513
1280399433,30cyclictest8278-21taskset20:26:143
1280399422,27cyclictest25613-21taskset19:41:063
1280399422,27cyclictest10451-21chrt00:21:463
1280399412,28cyclictest21586-21chrt22:00:473
1280399403,24cyclictest14714-21taskset19:10:403
1280399403,24cyclictest14714-21taskset19:10:403
1280399393,27cyclictest10454-21taskset20:35:013
1280399393,26cyclictest31328-21taskset19:59:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional