You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-03-02 - 18:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7 (updated Sat Mar 02, 2024 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1572121190,22sleep1231rcuc/108:09:421
268849911859,52cyclictest4287-21munin-plugin-st10:13:453
268629910343,12cyclictest8471-21uniq07:49:130
254392970,4chrt25440-21ssh10:49:501
26878998754,13cyclictest141rcu_preempt10:52:562
120652860,3sleep20-21swapper/211:24:212
26884998443,27cyclictest41-21ksoftirqd/308:04:353
26872998311,42cyclictest24404-21cron08:38:491
2575328314,32sleep30-21swapper/307:04:263
26884998252,24cyclictest41-21ksoftirqd/310:23:423
26884998252,24cyclictest41-21ksoftirqd/310:23:423
26862998262,13cyclictest31476-21fschecks_time08:59:170
2687299813,54cyclictest0-21swapper/110:20:461
2687299813,54cyclictest0-21swapper/110:20:461
26884998047,28cyclictest41-21ksoftirqd/307:24:253
2687299802,49cyclictest4668-21ssh09:13:461
2687299802,49cyclictest31575-21ssh11:01:151
2687299802,42cyclictest3382-21df_inode11:09:171
2687299802,42cyclictest3382-21df_inode11:09:171
2604428016,31sleep20-21swapper/207:04:372
2687299793,66cyclictest303-21apt-get11:04:091
2686299793,59cyclictest7985-21/usr/sbin/munin09:19:160
26872997841,29cyclictest29731-21unixbench_multi11:54:281
26872997828,40cyclictest22865-21rm10:45:431
26872997828,40cyclictest22865-21rm10:45:431
26884997730,24cyclictest141rcu_preempt08:24:293
26884997722,41cyclictest41-21ksoftirqd/312:32:253
2687299772,48cyclictest14487-21cstates11:29:071
26872997545,19cyclictest0-21swapper/109:49:181
26872997545,19cyclictest0-21swapper/109:49:181
2687299752,42cyclictest1727-21diskmemload12:13:151
2687299752,42cyclictest1727-21diskmemload12:13:151
26884997435,26cyclictest41-21ksoftirqd/311:04:013
26878997429,29cyclictest141rcu_preempt08:09:402
2687299742,46cyclictest22829-21ssh09:47:281
2687299742,46cyclictest22829-21ssh09:47:281
26862997428,29cyclictest141rcu_preempt10:59:220
255642749,38sleep125568-21/usr/sbin/munin07:04:201
26884997346,16cyclictest41-21ksoftirqd/311:39:103
26872997314,32cyclictest12526-21ssh11:24:301
26862997343,19cyclictest12-21ksoftirqd/010:39:280
26862997339,28cyclictest12-21ksoftirqd/011:26:280
2688499727,31cyclictest5189-21df_inode07:39:233
26884997228,19cyclictest141rcu_preempt10:34:133
26884997228,19cyclictest141rcu_preempt10:34:133
26884997221,26cyclictest41-21ksoftirqd/310:24:463
26878997236,21cyclictest179-21pr/ttyS007:29:202
26878997236,21cyclictest179-21pr/ttyS007:29:202
26872997247,16cyclictest1-21systemd10:43:451
26872997246,17cyclictest4681-21apt-key07:38:481
26872997246,17cyclictest4681-21apt-key07:38:481
2687299724,60cyclictest12858-21modprobe08:03:451
2687299723,54cyclictest29153-21apt-get09:59:111
2687299723,38cyclictest5503-21ssh09:14:221
2687299723,37cyclictest0-21swapper/111:44:191
26862997248,15cyclictest11253-21systemd-run07:58:460
26878997148,8cyclictest141rcu_preempt10:48:472
26878997148,8cyclictest141rcu_preempt10:48:472
26878997123,21cyclictest25601-21ssh09:52:032
26878997123,21cyclictest25601-21ssh09:52:032
2687299712,59cyclictest2254-21ssh12:04:091
26872997123,28cyclictest0-21swapper/112:14:101
26862997148,17cyclictest12-21ksoftirqd/011:39:380
26884997053,10cyclictest0-21swapper/309:08:453
26884997038,9cyclictest141rcu_preempt12:04:343
26884997038,11cyclictest41-21ksoftirqd/309:35:533
26884997019,26cyclictest141rcu_preempt09:50:223
26884997019,26cyclictest141rcu_preempt09:50:223
26878997040,8cyclictest141rcu_preempt08:03:482
26862997050,15cyclictest0-21swapper/008:08:450
26862997034,14cyclictest12-21ksoftirqd/009:49:250
26862997034,14cyclictest12-21ksoftirqd/009:49:250
26884996942,20cyclictest41-21ksoftirqd/310:39:143
26884996936,10cyclictest41-21ksoftirqd/307:44:313
26884996935,14cyclictest41-21ksoftirqd/311:59:063
2688499693,45cyclictest7984-21ssh09:19:163
26884996921,17cyclictest41-21ksoftirqd/308:55:393
2687299693,51cyclictest17218-21apt-get11:34:091
26872996929,32cyclictest0-21swapper/108:44:231
26884996840,14cyclictest41-21ksoftirqd/309:44:243
26884996840,14cyclictest41-21ksoftirqd/309:44:243
26884996835,14cyclictest41-21ksoftirqd/311:09:143
26884996835,14cyclictest41-21ksoftirqd/311:09:143
26878996853,9cyclictest9794-21grep11:19:312
26878996846,8cyclictest141rcu_preempt07:44:132
2687299683,56cyclictest399-21runrttasks09:54:281
2687299683,56cyclictest11393-21sort07:58:481
2687299683,55cyclictest16693-21cstates08:14:091
2687299683,53cyclictest19861-21apt-get08:24:111
2687299682,59cyclictest11725-21taskset12:19:271
26862996840,18cyclictest1731-21cut07:29:180
26862996840,18cyclictest1731-21cut07:29:180
26862996838,12cyclictest141rcu_preempt10:50:480
48322670,5sleep2311rcuc/210:14:172
26884996752,10cyclictest276-21systemd-journal07:53:223
26884996747,7cyclictest41-21ksoftirqd/311:28:003
26884996746,7cyclictest41-21ksoftirqd/309:39:243
26884996746,14cyclictest29311-21fschecks_count07:14:183
26884996727,20cyclictest141rcu_preempt11:19:043
26884996722,30cyclictest141rcu_preempt09:13:483
26884996718,23cyclictest141rcu_preempt08:29:113
26878996752,8cyclictest5630-21kworker/u8:611:13:452
26878996752,8cyclictest5630-21kworker/u8:611:13:452
26878996750,9cyclictest321ktimers/210:34:122
26878996750,9cyclictest321ktimers/210:34:122
26878996738,13cyclictest141rcu_preempt08:28:482
26878996737,9cyclictest141rcu_preempt09:59:242
26878996736,17cyclictest141rcu_preempt12:24:132
26878996724,13cyclictest141rcu_preempt10:59:212
2687299673,57cyclictest1727-21diskmemload11:19:321
2687299672,57cyclictest18797-21proc_pri09:39:321
2687299672,56cyclictest342-21unixbench_singl11:59:341
26872996725,31cyclictest0-21swapper/112:24:321
2687299672,52cyclictest15729-21ssh10:34:121
2687299672,52cyclictest15729-21ssh10:34:121
26872996712,46cyclictest0-21swapper/108:59:281
26862996751,10cyclictest7973-21sh07:48:450
26862996720,22cyclictest141rcu_preempt08:14:210
26862996718,25cyclictest12-21ksoftirqd/011:04:270
26862996716,14cyclictest276-21systemd-journal11:21:180
26884996641,9cyclictest141rcu_preempt09:17:163
26884996635,26cyclictest41-21ksoftirqd/310:04:153
26884996630,11cyclictest41-21ksoftirqd/307:34:303
26884996630,11cyclictest41-21ksoftirqd/307:34:303
26884996627,11cyclictest41-21ksoftirqd/311:34:013
2687299663,53cyclictest399-21runrttasks09:04:291
2687299663,53cyclictest17692-21fschecks_count12:29:121
2687299663,52cyclictest1911-21/usr/sbin/munin07:29:211
2687299663,52cyclictest1911-21/usr/sbin/munin07:29:211
26872996628,30cyclictest0-21swapper/111:39:141
2687299662,55cyclictest7060-21apt-get10:18:491
26872996614,26cyclictest141rcu_preempt09:19:231
26862996648,12cyclictest7802-21idleruntime-cro12:13:440
26862996648,12cyclictest7802-21idleruntime-cro12:13:440
26862996640,7cyclictest141rcu_preempt09:53:590
26862996626,9cyclictest141rcu_preempt11:59:340
26862996615,33cyclictest12-21ksoftirqd/010:14:310
2688499656,34cyclictest141rcu_preempt11:29:283
2688499655,33cyclictest0-21swapper/310:31:513
26884996546,12cyclictest52-21kcompactd009:03:483
26884996544,16cyclictest41-21ksoftirqd/311:00:413
26884996529,13cyclictest41-21ksoftirqd/309:32:083
26884996525,13cyclictest141rcu_preempt12:25:593
2688499652,49cyclictest7791-21apt-get12:13:583
26878996548,10cyclictest30928-21kworker/u8:308:58:462
26878996543,13cyclictest2447-21fschecks_count12:04:152
26878996540,9cyclictest141rcu_preempt08:34:332
26878996531,13cyclictest141rcu_preempt12:23:182
26878996530,13cyclictest141rcu_preempt12:18:252
26878996515,26cyclictest141rcu_preempt11:54:292
2687299652,56cyclictest6785-21switchtime11:14:281
2687299652,52cyclictest23401-21latency_hist12:38:481
26862996548,10cyclictest6289-21fschecks_time11:14:170
26862996547,12cyclictest1735-21idleruntime-cro09:08:450
26862996531,12cyclictest12-21ksoftirqd/011:36:060
26862996518,20cyclictest52-21kcompactd010:24:170
26884996445,10cyclictest7637-21kworker/u8:210:48:453
26884996445,10cyclictest7637-21kworker/u8:210:48:453
26884996444,13cyclictest31537-21munin-plugin-st10:03:453
26884996429,13cyclictest141rcu_preempt09:24:243
26872996441,13cyclictest0-21swapper/107:44:321
2687299643,54cyclictest15059-21grep09:33:451
2687299643,53cyclictest30806-21grep07:19:111
2687299643,52cyclictest10999-21/usr/sbin/munin09:24:181
2687299643,51cyclictest2172-21fschecks_time10:09:191
2687299643,51cyclictest18243-21apt-get08:19:071
2687299643,49cyclictest27057-21apt-get07:09:061
2687299643,47cyclictest13258-21cut10:29:151
2687299642,53cyclictest15583-21sh09:34:161
2687299642,49cyclictest0-21swapper/110:58:201
26862996450,9cyclictest179-21pr/ttyS010:34:330
26862996450,9cyclictest179-21pr/ttyS010:34:330
26862996447,12cyclictest14289-21kworker/u8:011:57:340
26862996447,10cyclictest0-21swapper/012:19:430
26862996443,15cyclictest2599-21fschecks_count09:09:340
26862996442,9cyclictest12-21ksoftirqd/008:54:250
26862996440,15cyclictest276-21systemd-journal08:34:320
26862996418,13cyclictest116-21pr/ttymxc012:14:070
26884996342,10cyclictest27746-21runrttasks08:48:483
26884996335,9cyclictest141rcu_preempt12:21:173
26884996318,14cyclictest141rcu_preempt10:48:513
26878996342,8cyclictest141rcu_preempt10:10:172
26878996334,19cyclictest4041-21grep07:34:312
26878996334,19cyclictest4041-21grep07:34:312
26878996332,13cyclictest141rcu_preempt09:12:432
26878996317,17cyclictest0-21swapper/210:54:272
2687299632,53cyclictest5554-21/usr/sbin/munin07:39:331
2687299632,52cyclictest29280-21cron08:53:451
2687299632,51cyclictest21636-21/usr/sbin/munin08:29:161
2687299632,49cyclictest23878-21/usr/sbin/munin08:34:321
2687299632,39cyclictest26882-21ssh11:49:321
2687299631,39cyclictest1727-21diskmemload10:04:421
2686299635,36cyclictest28422-21kworker/0:112:04:560
26862996347,11cyclictest22441-21unixbench_singl10:44:300
26862996347,11cyclictest22441-21unixbench_singl10:44:300
26862996342,13cyclictest27260-21fschecks_count07:09:130
26862996338,15cyclictest12-21ksoftirqd/011:11:220
26862996338,15cyclictest12-21ksoftirqd/011:11:220
26884996219,8cyclictest141rcu_preempt07:58:453
2687899627,12cyclictest141rcu_preempt08:19:412
26878996243,12cyclictest0-21swapper/211:48:452
2687899624,20cyclictest32208-21latency11:59:242
26878996235,10cyclictest141rcu_preempt07:14:212
2687299623,51cyclictest10967-21rm10:24:241
2687299622,52cyclictest30453-21unixbench_multi08:54:281
2687299622,51cyclictest8870-21/usr/sbin/munin07:49:251
176712622,48sleep217677-21ssh12:29:122
26884996138,9cyclictest41-21ksoftirqd/311:47:023
26884996135,14cyclictest141rcu_preempt11:18:273
2688499613,35cyclictest15003-21taskset08:09:203
26884996128,8cyclictest141rcu_preempt07:19:293
26878996144,9cyclictest276-21systemd-journal11:34:132
26878996137,8cyclictest141rcu_preempt07:23:482
26878996132,13cyclictest141rcu_preempt07:28:492
26878996129,19cyclictest141rcu_preempt09:34:312
2687299612,53cyclictest14817-21dpkg08:08:491
26862996139,11cyclictest0-21swapper/011:44:290
26862996138,12cyclictest0-21swapper/012:24:320
254842618,31sleep025537-21expr07:04:150
26884996038,8cyclictest41-21ksoftirqd/311:50:063
26884996033,7cyclictest141rcu_preempt12:34:003
2687899605,26cyclictest33-21ksoftirqd/211:53:482
26878996040,12cyclictest7356-21sendmail09:18:462
26878996039,7cyclictest141rcu_preempt11:04:282
26878996039,7cyclictest141rcu_preempt09:08:472
26878996035,8cyclictest141rcu_preempt12:09:142
26878996035,8cyclictest141rcu_preempt12:09:142
26878996030,10cyclictest141rcu_preempt09:54:172
26878996024,9cyclictest141rcu_preempt12:38:262
26878996015,25cyclictest141rcu_preempt09:46:532
26878996015,25cyclictest141rcu_preempt09:46:532
2687299602,51cyclictest32526-21/usr/sbin/munin07:24:151
26862996036,7cyclictest141rcu_preempt08:49:260
26862996024,10cyclictest141rcu_preempt10:08:190
2686299602,36cyclictest7620-21/usr/sbin/munin10:19:050
2686299602,36cyclictest7620-21/usr/sbin/munin10:19:050
26862996015,14cyclictest141rcu_preempt08:19:270
26862996014,20cyclictest141rcu_preempt10:54:160
26884995933,13cyclictest0-21swapper/307:29:183
26884995933,13cyclictest0-21swapper/307:29:183
26884995924,14cyclictest141rcu_preempt09:58:323
2687899598,32cyclictest5194-21diskstats07:39:242
26878995942,10cyclictest0-21swapper/208:49:312
26878995938,9cyclictest141rcu_preempt10:29:382
26878995933,8cyclictest141rcu_preempt08:44:342
26878995933,11cyclictest141rcu_preempt10:04:152
26878995931,15cyclictest141rcu_preempt07:13:482
26878995927,9cyclictest141rcu_preempt09:24:282
26878995927,17cyclictest116-21pr/ttymxc009:19:152
2687299592,49cyclictest29678-21/usr/sbin/munin07:14:271
2686299598,25cyclictest141rcu_preempt12:30:340
26862995940,12cyclictest14317-21ssh10:30:390
26862995939,15cyclictest12-21ksoftirqd/007:14:300
2687899584,27cyclictest6714-21ssh11:14:272
26878995834,8cyclictest141rcu_preempt10:24:212
26878995826,18cyclictest141rcu_preempt08:59:212
26878995824,9cyclictest14375-21kworker/u8:308:14:192
26878995824,11cyclictest141rcu_preempt11:29:132
26878995820,8cyclictest141rcu_preempt10:39:182
26862995835,17cyclictest14639-21cut11:29:140
26884995734,8cyclictest41-21ksoftirqd/310:14:213
26884995732,8cyclictest41-21ksoftirqd/311:54:213
26884995721,17cyclictest41-21ksoftirqd/308:49:233
26878995741,10cyclictest8236-21sh10:19:272
26878995741,10cyclictest8236-21sh10:19:272
26878995734,15cyclictest24928-21fschecks_count08:39:182
2687899573,28cyclictest33-21ksoftirqd/207:54:262
26862995739,9cyclictest0-21swapper/011:49:270
26862995732,9cyclictest12-21ksoftirqd/008:44:320
26862995723,29cyclictest0-21swapper/009:18:490
744525635,11sleep3401ktimers/312:12:513
744525635,11sleep3401ktimers/312:12:513
2688499564,28cyclictest141rcu_preempt10:58:453
26884995630,13cyclictest141rcu_preempt08:39:133
26884995611,12cyclictest347-21in:imklog07:13:483
2687899563,31cyclictest33-21ksoftirqd/208:08:472
2687899563,19cyclictest33-21ksoftirqd/211:39:212
2686299564,35cyclictest12-21ksoftirqd/007:38:580
26862995633,16cyclictest12-21ksoftirqd/008:39:070
26862995632,8cyclictest12-21ksoftirqd/007:24:290
26862995630,7cyclictest12-21ksoftirqd/010:09:570
26862995630,15cyclictest12-21ksoftirqd/009:43:460
26862995622,13cyclictest141rcu_preempt07:59:250
26862995619,13cyclictest12-21ksoftirqd/009:34:230
2688499556,23cyclictest0-21swapper/308:14:043
26878995533,9cyclictest141rcu_preempt09:33:482
26878995533,9cyclictest141rcu_preempt08:29:192
26862995538,9cyclictest4948-21cmp07:38:490
26862995537,12cyclictest15102-21latency_hist09:33:470
26862995536,12cyclictest131ktimers/009:48:480
26862995536,12cyclictest131ktimers/009:48:480
2686299552,17cyclictest15793-21grep08:09:430
26862995519,10cyclictest12-21ksoftirqd/010:00:090
26884995432,8cyclictest141rcu_preempt07:59:163
26884995431,7cyclictest41-21ksoftirqd/308:34:203
2687899546,13cyclictest9194-21unixbench_singl07:49:292
26862995436,10cyclictest0-21swapper/008:29:310
26862995433,14cyclictest16305-21kworker/u8:009:24:190
26862995334,11cyclictest347-21in:imklog12:35:080
26862995326,9cyclictest52-21kcompactd007:34:290
26878995139,7cyclictest0-21swapper/209:39:212
26862995133,10cyclictest0-21swapper/008:28:460
26884994727,9cyclictest0-21swapper/308:19:233
26862994730,10cyclictest276-21systemd-journal07:19:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional