You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 18:21
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Jul 28, 2026 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2789921379,100sleep00-21swapper/006:51:320
29220991323,87cyclictest3157-21apt-get11:36:241
29220991323,87cyclictest3157-21apt-get11:36:241
292209912010,94cyclictest21245-21apt-get12:06:201
292209910813,88cyclictest8417-21ls09:57:201
27901210610,66sleep127929-21fschecks_count06:51:331
29226999563,11cyclictest181rcu_preempt07:41:342
2808629319,40sleep20-21swapper/206:51:382
2922099907,61cyclictest13608-21apt-get10:06:201
2922099907,61cyclictest13608-21apt-get10:06:201
29220999064,7cyclictest181rcu_preempt11:56:181
29226998961,19cyclictest2737-21apt-config11:36:072
29226998961,19cyclictest2737-21apt-config11:36:072
29208998854,28cyclictest29137-21sendmail11:26:050
29208998854,28cyclictest29137-21sendmail11:26:050
29226998639,34cyclictest6767-21kworker/u19:1+nfsiod11:44:172
2922099868,17cyclictest31-21ksoftirqd/112:21:221
2922099862,43cyclictest22490-21fschecks_time09:26:271
29226998565,8cyclictest181rcu_preempt09:32:312
29226998538,12cyclictest28987-21ssh10:31:082
29226998454,20cyclictest0-21swapper/209:56:392
29226998449,10cyclictest181rcu_preempt07:01:152
29226998446,12cyclictest29195-21cyclictest09:52:042
29226998446,12cyclictest29195-21cyclictest09:52:042
2922099847,66cyclictest24242-21kworker/u18:1+flush-179:007:11:331
2920899843,34cyclictest15-21ksoftirqd/012:05:160
29226998350,7cyclictest181rcu_preempt11:37:022
29226998350,7cyclictest181rcu_preempt11:37:022
29226998341,11cyclictest24242-21kworker/u18:1-rpciod09:11:372
29220998341,9cyclictest181rcu_preempt09:16:261
29208998366,11cyclictest315-21idleruntime-cro07:06:050
29226998266,10cyclictest31137-21rm09:41:142
29226998266,10cyclictest31137-21rm09:41:142
29226998163,13cyclictest5630-21latency_hist07:21:092
2922699812,42cyclictest39-21ksoftirqd/211:21:222
2922699812,42cyclictest39-21ksoftirqd/211:21:222
2920899819,46cyclictest15-21ksoftirqd/007:16:390
29208998111,39cyclictest16837-21/usr/sbin/munin09:16:370
29226998059,15cyclictest29947-21ssh12:21:072
29226998049,10cyclictest181rcu_preempt10:02:432
29226998045,28cyclictest24035-21kworker/u19:2+nfsiod11:31:072
29226998044,17cyclictest0-21swapper/210:46:262
29226998040,10cyclictest58-21kcompactd010:14:102
2922699803,39cyclictest39-21ksoftirqd/209:04:012
2922699803,39cyclictest39-21ksoftirqd/209:04:012
29226998032,14cyclictest0-21swapper/211:51:312
29226998032,14cyclictest0-21swapper/211:51:312
29220998050,11cyclictest181rcu_preempt10:27:501
2920899808,44cyclictest15-21ksoftirqd/007:40:190
2920899808,44cyclictest15-21ksoftirqd/007:40:190
29208998012,39cyclictest15-21ksoftirqd/010:51:360
29208998012,39cyclictest15-21ksoftirqd/010:51:360
14032800,8sleep01405-21munin-run09:46:050
14032800,8sleep01405-21munin-run09:46:050
29226997963,10cyclictest277-21dbus-daemon07:11:072
29226997963,10cyclictest277-21dbus-daemon07:11:072
29226997957,15cyclictest15555-21fschecks_count07:51:272
29226997950,8cyclictest181rcu_preempt09:50:172
29226997950,10cyclictest181rcu_preempt12:10:192
29226997936,9cyclictest14874-21sshd07:48:222
29208997963,12cyclictest27183-21kworker/u19:1-nfsiod10:16:380
2920899793,37cyclictest15-21ksoftirqd/012:19:390
29226997863,9cyclictest3882-21diskmemload11:20:242
29226997858,15cyclictest11269-21threads07:36:412
29226997858,15cyclictest11269-21threads07:36:412
29226997856,9cyclictest181rcu_preempt10:55:212
29226997856,9cyclictest181rcu_preempt10:55:212
29226997854,16cyclictest17236-21unixbench_singl09:16:422
2920899786,36cyclictest15-21ksoftirqd/009:40:060
29226997761,10cyclictest28972-21kworker/u17:3-rpciod12:22:242
29226997759,13cyclictest0-21swapper/211:56:412
29226997758,14cyclictest25598-21sh12:12:302
29226997758,14cyclictest25598-21sh12:12:302
29226997753,9cyclictest181rcu_preempt10:23:392
29226997753,9cyclictest181rcu_preempt10:23:392
29220997753,17cyclictest29195-21cyclictest07:06:271
29220997753,17cyclictest29195-21cyclictest07:06:271
29220997753,16cyclictest0-21swapper/108:01:071
29220997753,14cyclictest0-21swapper/107:06:051
29208997761,10cyclictest23299-21ssh09:26:430
29226997661,10cyclictest29195-21cyclictest09:06:322
29220997660,11cyclictest3882-21diskmemload12:02:021
29220997655,16cyclictest30531-21unixbench_singl11:26:421
29220997655,10cyclictest0-21swapper/109:56:091
29220997655,10cyclictest0-21swapper/109:56:091
29220997645,9cyclictest181rcu_preempt07:21:061
29220997610,48cyclictest26226-21cat11:21:071
2920899766,28cyclictest15-21ksoftirqd/010:08:150
2920899766,28cyclictest15-21ksoftirqd/010:08:150
29208997657,13cyclictest8941-21latency_hist11:46:080
29208997656,14cyclictest30128-21unixbench_multi10:31:410
29208997656,14cyclictest30128-21unixbench_multi10:31:410
2920899762,52cyclictest15-21ksoftirqd/009:05:370
2920899762,52cyclictest15-21ksoftirqd/009:05:370
2920899762,40cyclictest22228-21ssh12:07:520
2922699756,37cyclictest39-21ksoftirqd/208:58:172
2922699756,37cyclictest39-21ksoftirqd/208:58:172
29226997559,11cyclictest29195-21cyclictest08:26:052
29226997544,8cyclictest181rcu_preempt10:56:142
29226997535,10cyclictest31839-21systemd-run10:36:062
29226997535,10cyclictest31839-21systemd-run10:36:062
2922099753,53cyclictest6695-21head10:46:421
29220997529,22cyclictest0-21swapper/108:51:231
2920899757,40cyclictest15-21ksoftirqd/009:32:080
29208997558,11cyclictest670-21cpuspeed07:06:220
29208997558,11cyclictest670-21cpuspeed07:06:220
29208997556,13cyclictest29897-21kernelversion11:26:300
29208997556,12cyclictest8364-21systemd-run10:51:060
29208997546,21cyclictest30720-21modprobe07:01:060
29208997546,21cyclictest30720-21modprobe07:01:060
2920899754,43cyclictest15-21ksoftirqd/010:30:510
2920899753,35cyclictest15-21ksoftirqd/010:40:090
29226997460,9cyclictest17908-21fschecks_time11:06:272
29226997456,13cyclictest7127-21latency_hist07:26:082
29226997454,14cyclictest19373-21runrttasks09:21:262
29226997454,14cyclictest19373-21runrttasks09:21:262
29226997454,11cyclictest0-21swapper/207:36:082
29226997448,7cyclictest181rcu_preempt10:44:102
29226997448,7cyclictest181rcu_preempt10:39:242
29226997440,17cyclictest0-21swapper/208:11:332
29226997430,12cyclictest0-21swapper/207:26:212
29220997459,9cyclictest4983-21kworker/u17:2-xprtiod11:06:061
29220997452,14cyclictest0-21swapper/107:51:081
29220997432,8cyclictest181rcu_preempt10:01:501
29220997432,12cyclictest181rcu_preempt07:31:211
29220997430,24cyclictest0-21swapper/109:46:171
2922099742,44cyclictest5900-21fschecks_time07:21:281
2920899747,35cyclictest15-21ksoftirqd/009:08:500
2920899746,34cyclictest15-21ksoftirqd/008:01:370
29208997457,11cyclictest5731-21cpuspeed07:21:230
2920899744,44cyclictest15-21ksoftirqd/011:31:170
2920899744,44cyclictest15-21ksoftirqd/011:31:170
2920899744,27cyclictest15-21ksoftirqd/011:53:190
2920899744,27cyclictest15-21ksoftirqd/011:53:190
2920899743,45cyclictest3157-21apt-get11:36:560
2920899743,45cyclictest3157-21apt-get11:36:560
2920899742,33cyclictest15-21ksoftirqd/012:13:260
2920899742,33cyclictest15-21ksoftirqd/012:13:260
29226997355,9cyclictest0-21swapper/208:31:092
29226997346,9cyclictest181rcu_preempt07:12:162
29226997343,8cyclictest181rcu_preempt08:46:392
29226997343,8cyclictest181rcu_preempt08:46:392
29226997330,10cyclictest181rcu_preempt11:46:512
29220997351,13cyclictest11085-21ntpd08:31:361
2922099733,22cyclictest18659-21head11:06:411
29220997329,7cyclictest181rcu_preempt11:01:061
29220997329,37cyclictest301ktimers/108:46:211
29220997329,37cyclictest301ktimers/108:46:211
29208997356,12cyclictest29195-21cyclictest10:25:470
29208997356,12cyclictest29195-21cyclictest10:25:470
29208997356,11cyclictest24190-21ntp_offset08:16:340
29208997356,11cyclictest24190-21ntp_offset08:16:340
29208997354,12cyclictest0-21swapper/010:11:410
29226997256,10cyclictest28671-21latency09:36:322
29226997249,13cyclictest0-21swapper/208:41:382
29226997244,7cyclictest181rcu_preempt09:28:152
29226997239,8cyclictest181rcu_preempt07:56:452
29226997235,10cyclictest181rcu_preempt10:09:492
29226997235,10cyclictest181rcu_preempt10:09:492
29226997228,13cyclictest181rcu_preempt11:15:052
29220997253,13cyclictest5306-21idleruntime-cro10:46:051
29220997251,16cyclictest19846-21kworker/u17:0+events_unbound10:21:051
29220997241,25cyclictest11320-21vmstat09:06:441
29220997231,12cyclictest181rcu_preempt10:56:091
29220997231,12cyclictest181rcu_preempt10:56:091
29220997228,10cyclictest181rcu_preempt09:21:261
29220997228,10cyclictest181rcu_preempt09:21:261
29220997227,11cyclictest181rcu_preempt07:53:151
2922099722,38cyclictest614-21timerwakeupswit10:36:421
29208997254,12cyclictest12140-21ssh10:56:320
2920899724,42cyclictest15-21ksoftirqd/008:10:490
2920899724,40cyclictest15-21ksoftirqd/007:12:110
2920899724,29cyclictest15-21ksoftirqd/011:06:100
2920899723,41cyclictest15-21ksoftirqd/007:28:370
2920899722,45cyclictest15-21ksoftirqd/012:23:400
29208997224,20cyclictest15-21ksoftirqd/011:16:420
76362710,8sleep30-21swapper/307:26:293
29226997155,8cyclictest0-21swapper/211:01:312
29226997151,10cyclictest0-21swapper/208:36:072
29226997139,7cyclictest181rcu_preempt08:41:072
29226997138,8cyclictest181rcu_preempt08:51:322
2922699712,43cyclictest39-21ksoftirqd/212:01:162
29220997125,13cyclictest181rcu_preempt09:40:131
2920899718,36cyclictest15-21ksoftirqd/008:41:570
29208997156,9cyclictest16253-21kworker/u19:0+events_unbound12:01:040
29208997155,10cyclictest9333-21rm11:46:260
29208997155,10cyclictest3159-21ssh10:41:320
29208997154,12cyclictest29195-21cyclictest09:55:470
29208997154,12cyclictest29195-21cyclictest09:55:470
29208997150,14cyclictest13824-21ntp_states09:11:370
2920899714,33cyclictest352-21runrttasks08:11:180
2920899713,40cyclictest15-21ksoftirqd/010:03:260
2920899713,31cyclictest15-21ksoftirqd/008:34:130
29226997034,8cyclictest181rcu_preempt08:16:382
29226997034,8cyclictest181rcu_preempt08:16:382
2922699703,29cyclictest39-21ksoftirqd/210:16:172
29226997027,10cyclictest181rcu_preempt08:06:282
29220997047,17cyclictest297-21systemd-logind07:01:091
29220997047,17cyclictest297-21systemd-logind07:01:091
29220997038,7cyclictest181rcu_preempt11:46:401
2922099703,38cyclictest3882-21diskmemload11:51:441
2922099703,38cyclictest3882-21diskmemload11:51:441
2922099702,41cyclictest28260-21awk08:31:051
29220997021,22cyclictest0-21swapper/111:21:181
29220997021,22cyclictest0-21swapper/111:21:181
2922099701,29cyclictest2374-21systemd-journal12:17:341
29208997051,12cyclictest3328-21ssh09:48:050
29208997049,15cyclictest14586-21/usr/sbin/munin11:01:390
2920899704,35cyclictest8172-21ssh09:56:540
2920899703,35cyclictest13498-21sendmail-msp07:46:050
29208997028,8cyclictest161ktimers/008:26:050
29220996923,12cyclictest181rcu_preempt10:32:471
29220996923,12cyclictest181rcu_preempt10:32:471
2920899698,34cyclictest2623-21cstates08:51:240
29208996948,16cyclictest20767-21sh11:11:260
29208996924,8cyclictest161ktimers/007:36:070
29208996921,23cyclictest15-21ksoftirqd/009:21:280
29208996921,23cyclictest15-21ksoftirqd/009:21:280
29226996851,11cyclictest20171-21latency_hist08:06:092
29226996839,10cyclictest181rcu_preempt06:56:312
29226996839,10cyclictest181rcu_preempt06:56:312
29220996849,12cyclictest29195-21cyclictest11:45:501
29220996844,15cyclictest0-21swapper/112:16:071
29220996844,15cyclictest0-21swapper/112:16:071
29220996826,9cyclictest0-21swapper/109:41:181
29220996826,9cyclictest0-21swapper/109:41:181
2922099682,41cyclictest31-21ksoftirqd/107:41:441
29208996824,34cyclictest16946-21gpgv07:56:080
29208996810,34cyclictest15-21ksoftirqd/008:39:290
29220996753,10cyclictest29195-21cyclictest08:06:411
29220996735,13cyclictest9185-21ssh09:04:121
29220996735,13cyclictest9185-21ssh09:04:121
2922099672,22cyclictest32427-21apt-get11:31:201
2922099672,22cyclictest32427-21apt-get11:31:201
29220996716,21cyclictest31-21ksoftirqd/110:24:531
29220996716,21cyclictest31-21ksoftirqd/110:24:531
2920899676,37cyclictest15-21ksoftirqd/009:00:570
2920899676,37cyclictest15-21ksoftirqd/009:00:570
2920899676,35cyclictest15-21ksoftirqd/008:26:370
2920899673,27cyclictest15-21ksoftirqd/007:48:230
2922099664,32cyclictest30180-21/usr/sbin/munin08:36:211
29220996627,8cyclictest181rcu_preempt08:56:301
29220996627,8cyclictest181rcu_preempt08:56:301
2922099662,45cyclictest25997-21switchtime09:31:401
2920899664,34cyclictest15-21ksoftirqd/008:46:440
2920899664,34cyclictest15-21ksoftirqd/008:46:440
2920899663,37cyclictest15-21ksoftirqd/007:58:420
2922099653,22cyclictest32364-21runrttasks08:41:381
29220996527,12cyclictest181rcu_preempt11:14:071
29220996513,9cyclictest0-21swapper/109:11:221
2922099643,43cyclictest24207-21ntp_states08:16:361
2922099643,43cyclictest24207-21ntp_states08:16:361
2922099643,24cyclictest352-21runrttasks08:13:371
29220996422,12cyclictest181rcu_preempt07:30:551
29220996349,10cyclictest29195-21cyclictest08:01:301
2922099632,23cyclictest18196-21ssh10:11:421
29220996322,11cyclictest181rcu_preempt08:21:411
2922099622,38cyclictest10701-21taskset07:36:271
2922099622,38cyclictest10701-21taskset07:36:271
2779225910,32sleep30-21swapper/306:51:273
2923299553,40cyclictest30886-21chrt07:01:093
2923299553,40cyclictest30886-21chrt07:01:093
2923299544,40cyclictest7408-21taskset09:56:053
2923299544,40cyclictest7408-21taskset09:56:053
2923299543,33cyclictest25103-21chrt09:31:083
2923299533,39cyclictest4355-21chrt11:38:043
2923299533,39cyclictest4355-21chrt11:38:043
2923299533,39cyclictest17323-21taskset11:06:063
2923299532,41cyclictest28963-21taskset11:25:393
2923299532,41cyclictest28963-21taskset11:25:393
29232995238,9cyclictest451rcuc/308:08:063
2923299523,38cyclictest14840-21chrt10:07:373
2923299523,38cyclictest14840-21chrt10:07:373
2923299523,37cyclictest6611-21taskset07:21:433
2923299523,37cyclictest19812-21taskset10:16:043
2923299522,38cyclictest27798-21chrt10:28:163
2923299522,38cyclictest20684-21taskset12:06:053
2923299522,38cyclictest14889-21taskset09:13:213
2923299514,36cyclictest3046-21taskset10:41:303
2923299513,37cyclictest3478-21chrt07:12:443
2923299513,37cyclictest11395-21chrt09:06:563
2923299513,36cyclictest9774-21chrt09:05:513
2923299513,36cyclictest9774-21chrt09:05:513
2923299513,36cyclictest10989-21chrt07:36:343
2923299513,36cyclictest10989-21chrt07:36:343
2923299513,36cyclictest10734-21chrt10:01:083
2923299513,35cyclictest13726-21chrt10:06:073
2923299513,32cyclictest3001-21chrt11:36:093
2923299513,32cyclictest3001-21chrt11:36:093
2923299512,38cyclictest1959-21chrt07:11:053
2923299512,38cyclictest1959-21chrt07:11:053
2923299505,34cyclictest17748-21chrt12:01:063
2923299504,33cyclictest25829-21chrt09:31:373
2923299503,36cyclictest9921-21chrt11:46:393
2923299503,36cyclictest25362-21taskset08:21:223
2923299503,36cyclictest22976-21chrt08:11:423
2923299503,36cyclictest13953-21taskset11:00:103
2923299503,34cyclictest495-21taskset12:26:083
2923299502,38cyclictest30793-21taskset09:40:573
2923299502,37cyclictest27390-21chrt12:16:273
2923299502,36cyclictest28583-21chrt08:31:223
2923299493,35cyclictest20212-21taskset09:21:443
2923299493,35cyclictest20212-21taskset09:21:443
2923299493,34cyclictest6786-21chrt11:41:373
2923299493,34cyclictest30114-21chrt10:31:413
2923299493,34cyclictest30114-21chrt10:31:413
2923299493,34cyclictest18108-21taskset11:06:313
2923299492,36cyclictest9897-21chrt07:31:443
2923299492,35cyclictest30408-21taskset11:26:413
2923299492,35cyclictest18667-21chrt09:20:353
2923299492,32cyclictest18766-21chrt08:01:083
2923299483,32cyclictest2365-21chrt08:51:073
2923299483,32cyclictest2365-21chrt08:51:073
2923299482,35cyclictest2587-21chrt08:51:243
2923299482,34cyclictest3122-21chrt09:47:303
2923299482,34cyclictest26333-21chrt12:14:363
2923299482,34cyclictest26333-21chrt12:14:363
2923299482,34cyclictest20776-21chrt10:16:373
2923299482,33cyclictest31543-21chrt07:01:253
2923299473,34cyclictest24379-21chrt10:22:083
2923299473,34cyclictest24379-21chrt10:22:083
2923299473,32cyclictest23256-21chrt11:16:073
2923299472,31cyclictest24966-21chrt11:17:413
2923299463,33cyclictest676-21chrt09:44:093
2923299463,33cyclictest676-21chrt09:44:093
2923299462,32cyclictest7498-21chrt10:48:473
2923299453,32cyclictest4213-21chrt07:16:243
2923299453,31cyclictest32269-21taskset08:41:353
2923299453,29cyclictest27144-21taskset08:26:283
2923299452,33cyclictest14556-21chrt11:55:353
2923299452,33cyclictest14556-21chrt11:55:353
2923299444,31cyclictest16151-21chrt07:51:413
2923299443,31cyclictest21542-21taskset12:07:063
2923299443,30cyclictest1973-21taskset10:40:083
2923299443,28cyclictest4140-21taskset08:56:103
2923299443,28cyclictest4140-21taskset08:56:103
29232994430,9cyclictest451rcuc/310:54:073
29232994430,9cyclictest451rcuc/310:54:073
2923299433,30cyclictest30131-21taskset08:36:173
2923299413,29cyclictest13839-21chrt07:46:223
2923299413,25cyclictest23676-21taskset08:16:203
2923299413,25cyclictest23676-21taskset08:16:203
2923299412,26cyclictest12231-21chrt07:41:243
2923299382,26cyclictest18936-21taskset08:01:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional