You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-09 - 12:30
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot2.osadl.org (updated Wed Sep 09, 2026 00:43:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
366290199166147,14cyclictest79-21hwrng22:29:050
366290399160100,14cyclictest10250irq/155-8000000.ethernet-tx523:20:422
36629019915725,26cyclictest3783448-21kworker/u16:2+rpciod23:00:010
36629019915725,26cyclictest3783448-21kworker/u16:2+rpciod23:00:010
3662901991511,108cyclictest3810161-21ssh23:30:210
3662901991511,108cyclictest3810161-21ssh23:30:210
366290399139104,26cyclictest0-21swapper/223:56:182
366290399139104,26cyclictest0-21swapper/223:56:182
366290499137115,15cyclictest0-21swapper/322:31:043
3662901991372,42cyclictest3648230-21kworker/u16:5+rpciod19:20:160
366290399135107,24cyclictest0-21swapper/223:40:552
36629029913553,15cyclictest3754347-21kworker/u16:2+xprtiod22:21:051
366290199135119,12cyclictest79-21hwrng21:47:060
366290399132118,6cyclictest79-21hwrng19:15:332
366290399132118,6cyclictest79-21hwrng19:15:332
36629039913197,24cyclictest0-21swapper/221:00:182
36629039913197,24cyclictest0-21swapper/221:00:182
366290399131108,13cyclictest79-21hwrng20:10:512
366290199131119,8cyclictest79-21hwrng22:45:290
366290299130122,6cyclictest0-21swapper/123:01:301
366290399129118,7cyclictest79-21hwrng21:10:152
3662901991298,64cyclictest3775412-21kworker/u16:5+xprtiod23:05:160
366290199129115,5cyclictest79-21hwrng22:40:210
366290199129115,5cyclictest79-21hwrng22:40:210
36629019912897,21cyclictest0-21swapper/020:40:500
366290299127112,11cyclictest0-21swapper/100:20:581
366290299127112,11cyclictest0-21swapper/100:20:581
3662903991267,82cyclictest3840117-21ld00:10:442
3662903991267,82cyclictest3840117-21ld00:10:442
366290399126112,7cyclictest79-21hwrng00:02:172
36629019912643,69cyclictest10550irq/205-8000000.ethernet-rx000:22:060
36629019912643,69cyclictest10550irq/205-8000000.ethernet-rx000:22:060
366290299125110,7cyclictest79-21hwrng23:30:321
366290299125110,7cyclictest79-21hwrng23:30:321
366290299124111,10cyclictest79-21hwrng19:48:191
36629029912410,65cyclictest3815695-21kworker/u16:2+rpciod00:19:591
3662902991231,86cyclictest3791942-21apt-get23:05:221
36629049912218,62cyclictest3728877-21kworker/u16:10+rpciod21:41:003
366290399122113,6cyclictest79-21hwrng20:36:272
366290199122107,9cyclictest10550irq/205-8000000.ethernet-rx022:00:040
366290199122107,9cyclictest10550irq/205-8000000.ethernet-rx022:00:040
3662904991215,111cyclictest3788926-21ssh23:01:003
36629019912090,18cyclictest10550irq/205-8000000.ethernet-rx021:41:280
36629039911960,22cyclictest0-21swapper/223:00:192
3662904991185,90cyclictest3775416-21kworker/u16:18+xprtiod23:24:593
36629019911895,14cyclictest10550irq/205-8000000.ethernet-rx021:50:370
366290299117107,7cyclictest79-21hwrng00:28:541
366290199117103,6cyclictest79-21hwrng21:25:360
366290199117103,6cyclictest79-21hwrng21:25:360
36629019911653,20cyclictest171ktimers/019:50:230
3662904991154,104cyclictest3796329-21/usr/sbin/munin23:10:393
3662904991154,104cyclictest3796329-21/usr/sbin/munin23:10:393
366290399115107,5cyclictest79-21hwrng23:53:042
36629029911555,4cyclictest10450irq/171-8000000.ethernet-tx720:20:531
366290299115105,8cyclictest79-21hwrng21:36:521
366290299115100,12cyclictest79-21hwrng19:18:381
366290299115100,12cyclictest79-21hwrng19:18:381
36629019911580,23cyclictest0-21swapper/022:10:430
3662901991152,65cyclictest3835249-21munin-run00:04:590
366290199115106,6cyclictest79-21hwrng22:09:380
36629029911311,65cyclictest31-21ksoftirqd/121:31:101
36629029911311,65cyclictest31-21ksoftirqd/121:31:101
36629019911350,15cyclictest0-21swapper/019:35:350
36629039911195,12cyclictest79-21hwrng20:20:042
3662903991112,82cyclictest3808201-21users23:26:172
3662904991106,69cyclictest3800405-21ntpq23:16:053
366290499110100,3cyclictest10550irq/205-8000000.ethernet-rx022:00:363
366290499110100,3cyclictest10550irq/205-8000000.ethernet-rx022:00:363
36629039911030,4cyclictest10450irq/171-8000000.ethernet-tx719:40:222
36629029911098,10cyclictest79-21hwrng20:39:311
3662904991091,90cyclictest3844457-21grep00:16:123
36629019910995,5cyclictest79-21hwrng21:12:180
3662901991093,54cyclictest3744447-21store22:00:000
366290199109101,5cyclictest79-21hwrng22:39:200
36629039910898,7cyclictest79-21hwrng00:31:582
36629019910899,6cyclictest79-21hwrng23:03:540
36629019910892,13cyclictest79-21hwrng00:05:210
3662904991075,93cyclictest511-21mstpd22:20:283
3662904991072,76cyclictest3815695-21kworker/u16:2+rpciod23:41:183
36629029910794,4cyclictest3785669-21ssh22:55:471
36629029910794,4cyclictest3785669-21ssh22:55:471
3662904991061,67cyclictest0-21swapper/319:30:203
36629039910689,13cyclictest79-21hwrng19:52:252
36629029910666,37cyclictest301ktimers/119:40:181
366290299106100,4cyclictest79-21hwrng21:04:061
366290299106100,4cyclictest79-21hwrng21:04:061
3662904991058,80cyclictest3837260-21users00:06:133
36629019910598,4cyclictest10550irq/205-8000000.ethernet-rx023:35:500
36629019910598,4cyclictest10550irq/205-8000000.ethernet-rx023:35:500
36629019910570,5cyclictest10450irq/171-8000000.ethernet-tx722:21:210
36629019910551,21cyclictest0-21swapper/000:30:090
3662904991043,95cyclictest3755572-21/usr/sbin/munin22:15:413
36629029910497,5cyclictest79-21hwrng20:02:401
36629029910497,5cyclictest79-21hwrng19:38:051
3662901991046,46cyclictest0-21swapper/023:25:120
36629019910432,46cyclictest21-21rcuc/023:40:400
36629039910388,12cyclictest79-21hwrng20:29:172
36629019910375,25cyclictest171ktimers/000:38:280
36629049910233,49cyclictest0-21swapper/322:42:173
36629049910233,49cyclictest0-21swapper/322:42:173
36629049910222,70cyclictest461ktimers/319:55:143
36629039910294,5cyclictest79-21hwrng22:10:402
36629029910270,26cyclictest0-21swapper/120:41:171
36629029910269,22cyclictest301ktimers/121:45:261
36629019910293,6cyclictest171ktimers/023:50:230
36629019910287,11cyclictest79-21hwrng22:17:500
36629039910194,5cyclictest79-21hwrng20:43:372
36629039910087,10cyclictest79-21hwrng20:54:532
36629039910047,41cyclictest0-21swapper/223:10:292
36629039910047,41cyclictest0-21swapper/223:10:292
36629039910017,60cyclictest3845247-21kworker/u16:5+xprtiod00:22:072
36629039910017,60cyclictest3845247-21kworker/u16:5+xprtiod00:22:072
36629029910094,4cyclictest79-21hwrng21:43:011
36629019910026,51cyclictest16-21ksoftirqd/023:55:350
36629019910026,51cyclictest16-21ksoftirqd/023:55:350
3662901991001,71cyclictest3844034-21ntpq00:16:040
366290399994,70cyclictest3814916-21ntpq23:36:062
366290399994,70cyclictest3814916-21ntpq23:36:062
3662901999957,27cyclictest3675817-21kworker/u16:19+rpciod20:21:130
3662904999832,40cyclictest3682894-21kworker/u16:6+rpciod20:40:153
3662903999892,3cyclictest3729748-21sendmail-msp21:40:552
3662903999757,27cyclictest3761275-21kworker/u16:12+rpciod22:31:262
3662903999728,62cyclictest3720504-21grep21:26:102
3662903999728,62cyclictest3720504-21grep21:26:102
366290199973,66cyclictest3840275-21ntp_offset00:10:560
366290199973,66cyclictest3840275-21ntp_offset00:10:560
3662904999675,14cyclictest0-21swapper/323:25:113
3662903999688,5cyclictest79-21hwrng20:17:002
3662903999688,5cyclictest79-21hwrng20:17:002
366290499953,86cyclictest0-21swapper/322:25:573
3662903999586,6cyclictest37-21rcuc/221:50:522
3662903999573,16cyclictest79-21hwrng20:33:232
3662901999584,4cyclictest0-21swapper/022:30:350
3662904999429,41cyclictest3726912-21tail21:35:373
3662903999476,12cyclictest0-21swapper/220:00:512
3662901999485,6cyclictest171ktimers/023:45:480
3662901999479,12cyclictest79-21hwrng20:30:180
3662904999368,16cyclictest0-21swapper/320:05:213
3662904999368,16cyclictest0-21swapper/320:05:213
366290399933,79cyclictest3749358-21ssh22:05:582
3662902999385,5cyclictest79-21hwrng21:59:241
3662901999379,10cyclictest171ktimers/021:35:300
3662904999229,50cyclictest461ktimers/300:10:113
3662904999229,50cyclictest461ktimers/300:10:113
3662901999278,5cyclictest79-21hwrng21:34:490
3662901999278,5cyclictest79-21hwrng21:34:490
366290499916,67cyclictest3791942-21apt-get23:05:163
3662904999138,44cyclictest3831658-21sendmail-msp00:01:323
3662902999183,5cyclictest3766211-21ssh22:30:001
3662904999082,4cyclictest10550irq/205-8000000.ethernet-rx023:00:023
3662904999082,4cyclictest10550irq/205-8000000.ethernet-rx023:00:023
366290499903,67cyclictest10550irq/205-8000000.ethernet-rx021:20:563
366290499902,80cyclictest3744387-21sendmail-msp22:00:003
3662903999056,27cyclictest3761276-21kworker/u16:14+rpciod22:40:552
3662903999056,27cyclictest3761276-21kworker/u16:14+rpciod22:40:552
366290299902,24cyclictest31-21ksoftirqd/122:41:151
366290299902,24cyclictest31-21ksoftirqd/122:41:151
3662904998965,17cyclictest0-21swapper/323:59:593
3662904998965,17cyclictest0-21swapper/323:59:593
3662904998922,46cyclictest3732106-21kworker/u16:21+xprtiod21:45:443
3662903998976,7cyclictest37-21rcuc/222:18:072
3662901998974,4cyclictest79-21hwrng20:49:460
3662904998815,55cyclictest0-21swapper/322:35:103
3662902998860,21cyclictest0-21swapper/123:20:591
3662903998779,5cyclictest79-21hwrng19:47:182
3662903998769,11cyclictest79-21hwrng19:39:062
3662903998737,36cyclictest0-21swapper/222:00:202
3662903998737,36cyclictest0-21swapper/222:00:202
3662904998678,5cyclictest3707950-21ssh21:10:003
3662904998678,5cyclictest3707950-21ssh21:10:003
3662903998676,6cyclictest79-21hwrng21:15:222
3662903998675,8cyclictest79-21hwrng20:56:562
3662903998657,23cyclictest0-21swapper/220:05:112
3662903998657,23cyclictest0-21swapper/220:05:112
3662902998668,16cyclictest0-21swapper/122:14:591
3662902998662,18cyclictest0-21swapper/123:41:311
3662901998659,23cyclictest3710378-21kworker/u16:3+xprtiod21:20:210
366290499851,82cyclictest0-21swapper/322:45:253
3662903998557,24cyclictest0-21swapper/219:20:492
3662903998532,29cyclictest3775412-21kworker/u16:5+rpciod23:20:022
3662902998547,19cyclictest0-21swapper/100:00:451
3662902998543,38cyclictest29-21rcuc/122:45:121
366290199856,20cyclictest3853435-21cat00:29:590
366290499845,68cyclictest0-21swapper/300:22:193
366290499845,68cyclictest0-21swapper/300:22:193
3662904998427,42cyclictest3857966-21grep00:35:383
3662901998475,6cyclictest79-21hwrng20:25:110
366290499832,65cyclictest3819318-21kworker/u16:14+xprtiod23:45:143
3662904998321,50cyclictest0-21swapper/323:30:413
3662904998321,50cyclictest0-21swapper/323:30:413
3662903998376,4cyclictest79-21hwrng19:10:262
3662903998373,7cyclictest0-21swapper/222:00:002
366290399836,55cyclictest3835540-21apt-get00:05:112
3662902998361,18cyclictest0-21swapper/122:00:201
3662902998361,18cyclictest0-21swapper/122:00:201
3662902998353,24cyclictest0-21swapper/122:30:231
3662901998379,2cyclictest0-21swapper/023:15:480
3662901998374,6cyclictest79-21hwrng20:09:500
3662901998374,6cyclictest79-21hwrng20:09:500
366290499826,41cyclictest0-21swapper/300:25:363
3662903998254,14cyclictest0-21swapper/223:46:142
3662904998172,5cyclictest79-21hwrng19:41:093
3662904998154,17cyclictest0-21swapper/321:11:153
366290299811,4cyclictest3698276-21kworker/u16:18+rpciod21:00:001
3662901998161,17cyclictest171ktimers/019:30:140
366290499808,36cyclictest3737521-21aten_r8power_en21:50:313
3662903998056,14cyclictest381ktimers/200:25:142
3662903998039,34cyclictest0-21swapper/220:46:052
366290399801,52cyclictest3780877-21latency_hist22:50:002
3662904997955,10cyclictest0-21swapper/320:26:083
3662904997924,52cyclictest3723494-21cat21:30:433
3662904997924,52cyclictest3723494-21cat21:30:433
3662903997964,13cyclictest0-21swapper/222:55:582
3662903997964,13cyclictest0-21swapper/222:55:582
3662902997911,56cyclictest31-21ksoftirqd/100:37:051
3662901997974,3cyclictest3781990-21ssh22:50:470
3662904997861,6cyclictest79-21hwrng22:51:373
3662904997831,39cyclictest3825705-21ssh23:50:563
366290499781,72cyclictest3753632-21vmstat22:11:123
366290499781,42cyclictest3748238-21apt-get22:05:153
3662903997837,37cyclictest0-21swapper/221:40:012
3662901997851,22cyclictest171ktimers/021:15:150
366290499777,63cyclictest3854517-21memory00:30:403
366290499775,59cyclictest3689119-21sendmail-msp20:20:453
3662904997727,39cyclictest3693546-21sed20:30:363
366290499772,51cyclictest3650195-21kworker/u16:21+rpciod19:40:003
3662904997718,48cyclictest0-21swapper/321:25:403
3662904997718,48cyclictest0-21swapper/321:25:403
366290499771,43cyclictest3681829-21apt-get20:00:173
3662903997729,35cyclictest3714088-21kworker/u16:5+xprtiod21:20:392
366290199772,15cyclictest3668404-21kworker/u16:12+rpciod21:00:000
366290199771,53cyclictest16-21ksoftirqd/019:15:170
366290199771,53cyclictest16-21ksoftirqd/019:15:170
3662904997652,19cyclictest0-21swapper/323:35:383
3662904997652,19cyclictest0-21swapper/323:35:383
366290499764,67cyclictest3665438-21grep19:15:363
366290499764,67cyclictest3665438-21grep19:15:363
3662904997627,43cyclictest3712858-21ntpq21:15:563
3662904997623,47cyclictest3677196-21find19:46:013
366290499761,67cyclictest0-21swapper/320:50:273
3662902997657,10cyclictest0-21swapper/119:20:251
3662904997534,37cyclictest3678934-21ntpq19:50:543
366290499751,70cyclictest0-21swapper/320:35:333
366290499751,68cyclictest0-21swapper/320:19:583
366290499751,68cyclictest0-21swapper/320:19:583
366290499751,66cyclictest3685350-21sh20:10:423
366290499751,60cyclictest3668466-21sshd-auth19:24:163
3662903997522,36cyclictest3743289-21kworker/u16:6+xprtiod22:21:232
3662901997568,4cyclictest10550irq/205-8000000.ethernet-rx023:20:130
3662904997425,40cyclictest0-21swapper/319:25:133
366290499741,63cyclictest0-21swapper/320:55:143
366290299749,37cyclictest3836096-21fw_packets00:05:371
3662901997440,29cyclictest0-21swapper/021:05:330
3662901997440,29cyclictest0-21swapper/021:05:330
366290499732,62cyclictest3663411-21fschecks_count19:10:343
366290499731,64cyclictest3698848-21/usr/sbin/munin20:45:303
3662903997362,8cyclictest79-21hwrng19:57:322
3662902997358,11cyclictest0-21swapper/123:20:001
3662902997347,17cyclictest301ktimers/120:05:231
3662902997347,17cyclictest301ktimers/120:05:231
3662902997339,31cyclictest0-21swapper/121:21:181
366290199732,12cyclictest0-21swapper/023:12:290
366290199732,12cyclictest0-21swapper/023:12:290
3662902997244,19cyclictest301ktimers/120:50:571
3662901997256,11cyclictest0-21swapper/020:01:250
3662901997253,16cyclictest0-21swapper/019:40:150
3662901997249,20cyclictest171ktimers/019:10:150
366290299715,37cyclictest3791235-21kworker/u16:22+xprtiod23:25:111
3662902997144,22cyclictest0-21swapper/123:55:091
3662902997144,22cyclictest0-21swapper/123:55:091
3662902997019,39cyclictest3715207-21cat21:19:591
366290499691,65cyclictest3704969-21missed_timers21:00:483
366290499691,65cyclictest3704969-21missed_timers21:00:483
3662903996954,11cyclictest0-21swapper/222:50:392
3662902996952,11cyclictest0-21swapper/123:55:001
3662902996945,20cyclictest3662900-21cyclictest00:10:151
3662902996945,20cyclictest3662900-21cyclictest00:10:151
3662903996831,34cyclictest3765137-21ssh22:27:332
3662902996753,11cyclictest0-21swapper/119:59:591
3662901996743,21cyclictest171ktimers/019:45:200
3662903996636,15cyclictest381ktimers/223:30:152
3662903996636,15cyclictest381ktimers/223:30:152
3662903996631,26cyclictest19-21rcu_preempt22:35:102
3662902996632,30cyclictest3749479-21ssh22:06:071
3662901996642,19cyclictest171ktimers/020:35:130
3662903996526,33cyclictest3723251-21tune2fs21:30:392
3662903996526,33cyclictest3723251-21tune2fs21:30:392
3662902996544,17cyclictest301ktimers/120:45:141
3662902996529,29cyclictest3737425-21ssh21:50:251
3662901996557,5cyclictest79-21hwrng19:29:540
3662903996435,23cyclictest0-21swapper/223:05:162
3662903996420,38cyclictest3857808-21fschecks_count00:35:362
3662902996456,5cyclictest301ktimers/123:10:591
3662902996456,5cyclictest301ktimers/123:10:591
3662902996452,8cyclictest301ktimers/120:30:151
366290199642,28cyclictest3705178-21921:01:100
366290199642,28cyclictest3705178-21921:01:100
366290399631,28cyclictest3671455-21ntpq19:30:552
3662902996341,18cyclictest0-21swapper/100:30:351
3662902996323,37cyclictest3771146-21sh22:35:531
366290199632,41cyclictest16-21ksoftirqd/020:15:140
366290199632,41cyclictest16-21ksoftirqd/020:15:140
3662903996224,33cyclictest3843974-21ssh00:15:592
366290299628,39cyclictest31-21ksoftirqd/123:48:201
3662902996245,14cyclictest0-21swapper/119:30:001
3662902996243,16cyclictest0-21swapper/121:25:181
3662902996243,16cyclictest0-21swapper/121:25:181
3662902996240,10cyclictest0-21swapper/122:15:161
3662902996226,32cyclictest3782266-21ssh22:51:081
3662902996223,23cyclictest0-21swapper/121:15:001
3662902996145,11cyclictest29-21rcuc/123:36:131
3662902996145,11cyclictest29-21rcuc/123:36:131
3662903996044,12cyclictest0-21swapper/221:05:562
3662903996044,12cyclictest0-21swapper/221:05:562
3662901996020,23cyclictest3675356-21kworker/u16:7+xprtiod20:10:110
3662903995925,29cyclictest3669618-21ntpq19:25:582
3662902995928,28cyclictest0-21swapper/119:11:081
366290399585,50cyclictest3734586-21ntpq21:45:442
3662902995840,10cyclictest301ktimers/119:50:111
3662901995836,9cyclictest0-21swapper/020:55:000
3662901995827,20cyclictest528-21systemd-logind19:57:490
366290299578,21cyclictest3650194-21kworker/u16:20+rpciod20:20:001
366290299578,21cyclictest3650194-21kworker/u16:20+rpciod20:20:001
366290499562,42cyclictest3662910-21latency_hist19:09:593
3662902995635,17cyclictest0-21swapper/120:15:001
366290299563,46cyclictest31-21ksoftirqd/121:05:461
366290299563,46cyclictest31-21ksoftirqd/121:05:461
3662902995631,19cyclictest301ktimers/120:25:231
3662902995516,35cyclictest31-21ksoftirqd/119:30:231
3662902994627,14cyclictest0-21swapper/119:10:001
3662903994427,5cyclictest10550irq/205-8000000.ethernet-rx019:10:002
366290199341,26cyclictest3663050-21latency_hist19:10:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional