You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-20 - 05:26
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Jun 19, 2024 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
57299993,67cyclictest400-21runrttasks10:21:310
57299992,63cyclictest0-21swapper/007:29:030
57299992,63cyclictest0-21swapper/007:29:030
327382986,14sleep30-21swapper/307:07:053
3250229814,52sleep032586-21grep07:05:190
57299973,85cyclictest21985-21apt-get10:34:130
57299972,66cyclictest6160-21latency_hist09:04:040
57299962,65cyclictest13082-21/usr/sbin/munin12:14:400
572999519,51cyclictest0-21swapper/007:59:190
59299946,48cyclictest27002-21kworker/u8:110:54:152
57299943,60cyclictest400-21runrttasks12:05:320
11812940,5chrt1179-21ntpq11:54:360
57299925,50cyclictest0-21swapper/009:09:430
57299923,62cyclictest9130-21ssh11:09:290
57299923,61cyclictest0-21swapper/011:40:440
57299922,55cyclictest8021-21diskmemload10:44:370
57299912,65cyclictest0-21swapper/012:10:220
57299912,65cyclictest0-21swapper/012:10:220
57299912,59cyclictest19675-21ntp_states09:29:370
57299912,59cyclictest0-21swapper/011:29:430
57299912,59cyclictest0-21swapper/011:02:320
57299912,57cyclictest21635-21sh09:34:160
592999066,14cyclictest16822-21ssh10:24:242
57299902,56cyclictest0-21swapper/012:01:230
572999012,51cyclictest28377-21kworker/u8:110:09:480
603998967,14cyclictest6169-21munin-plugin-st09:04:023
57299892,79cyclictest15380-21ssh09:21:490
57299892,79cyclictest15380-21ssh09:21:490
57299892,54cyclictest416-21snmpd08:44:420
57299884,57cyclictest19028-21ssh11:28:160
57299884,57cyclictest19028-21ssh11:28:160
57299882,55cyclictest8021-21diskmemload11:23:530
57299882,54cyclictest0-21swapper/008:14:230
57299882,53cyclictest6632-21dpkg09:04:060
57299881,58cyclictest32534-21gpgv10:54:030
57299873,54cyclictest0-21swapper/008:23:460
57299872,74cyclictest1014-21fschecks_time07:09:240
57299872,66cyclictest0-21swapper/010:25:550
57299872,54cyclictest29273-21taskset11:46:500
572998712,60cyclictest0-21swapper/007:39:310
592998664,15cyclictest12558-21modprobe07:44:022
57299863,76cyclictest3755-21xargs09:59:440
57299863,75cyclictest6771-21netstat11:04:320
57299863,55cyclictest0-21swapper/012:38:360
57299863,54cyclictest0-21swapper/009:39:390
57299863,53cyclictest31458-21taskset11:50:310
572998617,50cyclictest0-21swapper/012:31:250
60399853,49cyclictest3700-21apt-get11:59:203
57299853,75cyclictest11012-21cat07:39:040
57299853,52cyclictest27126-21apt-get09:44:160
57299852,57cyclictest8021-21diskmemload09:51:300
57299852,53cyclictest8021-21diskmemload11:18:570
57299844,72cyclictest18350-21sort12:24:200
57299843,74cyclictest12984-21ssh09:17:390
57299843,50cyclictest22256-21apt-get11:34:160
57299843,50cyclictest22256-21apt-get11:34:160
57299842,59cyclictest5693-21apt-get10:04:150
592998314,34cyclictest363-21in:imklog12:11:272
592998314,34cyclictest363-21in:imklog12:11:272
57299833,74cyclictest655-21/usr/sbin/munin10:54:220
57299833,53cyclictest400-21runrttasks08:57:300
57299832,48cyclictest11819-21ntpq10:14:380
57299823,72cyclictest25637-21ssh10:39:490
57299822,72cyclictest30830-21sort08:39:190
57299822,72cyclictest30830-21sort08:39:190
57299822,63cyclictest4181-21taskset07:16:100
57299822,53cyclictest400-21runrttasks12:21:130
583998145,30cyclictest3535-21sh10:59:231
572998127,45cyclictest0-21swapper/008:34:350
57299812,72cyclictest21216-21grep08:09:250
57299812,62cyclictest21451-21ssh10:33:110
57299803,68cyclictest16078-21/usr/sbin/munin07:54:250
57299802,71cyclictest2981-21latency_hist08:54:040
57299802,70cyclictest8323-21kernelversion07:29:330
3230028022,31sleep10-21swapper/107:05:161
321902806,45sleep232233-21grep07:05:132
572997958,14cyclictest27253-21idleruntime-cro08:29:020
57299792,54cyclictest0-21swapper/007:49:280
57299773,64cyclictest12795-21/usr/sbin/munin07:44:280
603997635,24cyclictest0-21swapper/312:14:253
592997628,13cyclictest141rcu_preempt11:09:272
583997556,12cyclictest19290-21idleruntime-cro11:29:011
583997556,12cyclictest19290-21idleruntime-cro11:29:011
57299753,64cyclictest0-21swapper/007:22:040
57299748,59cyclictest0-21swapper/009:24:270
57299744,60cyclictest28860-21apt08:34:010
57299742,67cyclictest20251-21grep08:04:440
57299742,47cyclictest0-21swapper/009:58:330
603997352,12cyclictest3988-21unixbench_singl09:59:473
59299718,56cyclictest3476-21apt-get10:59:152
592997152,11cyclictest21941-21runrttasks08:09:432
592997141,21cyclictest10914-21sh07:39:022
592997119,26cyclictest0-21swapper/212:00:322
583997145,16cyclictest0-21swapper/111:19:011
583997045,16cyclictest16146-21mailstats12:19:391
603996945,7cyclictest141rcu_preempt12:09:023
592996933,13cyclictest363-21in:imklog09:37:162
592996925,26cyclictest141rcu_preempt11:53:422
592996918,9cyclictest0-21swapper/207:54:252
603996854,9cyclictest0-21swapper/307:14:313
603996849,12cyclictest12173-21runrttasks07:39:483
592996844,16cyclictest0-21swapper/210:39:012
592996843,16cyclictest3173-21fschecks_time09:59:242
603996740,20cyclictest17383-21unixbench_multi09:24:443
603996728,23cyclictest363-21in:imklog08:29:513
59299677,26cyclictest22897-21fw_packets08:14:262
592996728,18cyclictest120-21pr/ttymxc012:15:552
592996727,18cyclictest141rcu_preempt08:23:362
592996716,12cyclictest141rcu_preempt10:04:352
583996750,11cyclictest9317-21true07:34:021
603996633,17cyclictest141rcu_preempt09:32:433
59299667,25cyclictest0-21swapper/210:44:012
592996638,19cyclictest27883-21fschecks_count11:44:262
592996632,9cyclictest0-21swapper/209:13:262
592996610,34cyclictest24002-21apt-get12:34:142
583996647,8cyclictest0-21swapper/111:44:441
58399663,41cyclictest2132-21apt-get07:14:161
603996531,17cyclictest363-21in:imklog10:24:203
59299654,29cyclictest3063-21apt-get07:14:162
592996542,15cyclictest6500-21latency_hist09:04:052
59299653,32cyclictest0-21swapper/211:15:262
583996539,13cyclictest141rcu_preempt10:15:461
583996536,19cyclictest0-21swapper/110:34:011
583996530,15cyclictest141rcu_preempt07:19:511
603996446,13cyclictest0-21swapper/309:14:043
603996413,27cyclictest3476-21apt-get10:59:213
59299646,38cyclictest0-21swapper/207:29:332
592996420,12cyclictest141rcu_preempt10:14:322
592996416,12cyclictest31094-21unixbench_singl10:49:442
58399643,41cyclictest3700-21apt-get11:59:211
583996433,9cyclictest141rcu_preempt11:33:481
603996347,9cyclictest0-21swapper/310:04:523
603996334,12cyclictest141rcu_preempt09:36:193
59299636,26cyclictest0-21swapper/212:24:202
592996347,8cyclictest0-21swapper/209:19:422
592996347,8cyclictest0-21swapper/209:19:422
592996346,10cyclictest16540-21cron11:24:012
592996341,14cyclictest52-21kcompactd007:44:282
592996336,7cyclictest141rcu_preempt09:41:042
592996326,10cyclictest141rcu_preempt11:54:282
583996344,12cyclictest32651-21kworker/u8:311:54:011
583996343,13cyclictest10860-21cron10:14:011
583996331,18cyclictest52-21kcompactd012:09:211
583996331,18cyclictest52-21kcompactd012:09:211
603996241,11cyclictest0-21swapper/311:14:523
603996239,16cyclictest17456-21sendmail07:59:013
603996236,14cyclictest141rcu_preempt08:27:203
60399623,46cyclictest6683-21seq09:04:223
603996229,14cyclictest141rcu_preempt11:52:343
59299623,31cyclictest0-21swapper/208:44:472
583996248,8cyclictest25097-21cpuspeed11:39:211
583996240,13cyclictest0-21swapper/111:59:011
583996237,7cyclictest141rcu_preempt10:41:211
603996139,14cyclictest0-21swapper/312:24:043
603996139,13cyclictest0-21swapper/312:14:013
603996139,13cyclictest0-21swapper/312:14:013
603996131,15cyclictest141rcu_preempt11:22:053
603996124,18cyclictest363-21in:imklog07:38:103
603996119,14cyclictest0-21swapper/308:23:583
592996138,13cyclictest0-21swapper/207:24:262
592996138,13cyclictest0-21swapper/207:24:262
59299613,28cyclictest14188-21cut10:19:272
592996121,8cyclictest141rcu_preempt09:16:572
592996116,14cyclictest141rcu_preempt08:49:262
592996114,10cyclictest141rcu_preempt08:34:032
583996142,11cyclictest0-21swapper/111:04:451
583996132,14cyclictest141rcu_preempt09:40:451
583996130,15cyclictest141rcu_preempt08:29:351
603996038,11cyclictest0-21swapper/309:39:493
603996037,15cyclictest0-21swapper/310:19:293
603996032,13cyclictest141rcu_preempt10:11:063
603996032,11cyclictest141rcu_preempt07:25:223
603996032,11cyclictest141rcu_preempt07:25:223
603996029,14cyclictest141rcu_preempt08:55:123
60399602,42cyclictest714-21df_inode10:54:243
603996022,8cyclictest141rcu_preempt11:58:313
59299609,31cyclictest0-21swapper/211:24:382
59299609,31cyclictest0-21swapper/211:24:382
592996039,7cyclictest141rcu_preempt09:31:472
592996036,15cyclictest22169-21unixbench_singl12:29:402
59299603,30cyclictest0-21swapper/210:48:222
59299603,29cyclictest0-21swapper/211:04:302
592996021,20cyclictest0-21swapper/207:09:312
592996010,7cyclictest0-21swapper/210:29:382
583996029,15cyclictest141rcu_preempt11:35:251
583996029,15cyclictest141rcu_preempt11:35:251
603995934,13cyclictest0-21swapper/307:44:563
60399592,30cyclictest21858-21latency_hist10:34:043
592995929,8cyclictest0-21swapper/208:54:492
583995935,7cyclictest141rcu_preempt12:27:241
583995933,14cyclictest141rcu_preempt10:06:581
583995927,16cyclictest0-21swapper/109:17:101
603995842,8cyclictest0-21swapper/312:34:433
603995838,13cyclictest20770-21idleruntime-cro12:29:013
603995836,12cyclictest0-21swapper/307:14:053
603995833,7cyclictest141rcu_preempt09:15:363
603995830,12cyclictest141rcu_preempt10:52:163
603995829,11cyclictest141rcu_preempt07:21:163
603995828,11cyclictest141rcu_preempt12:32:353
603995821,16cyclictest363-21in:imklog07:33:443
603995819,21cyclictest363-21in:imklog09:55:543
59299586,29cyclictest0-21swapper/207:19:542
592995843,9cyclictest17386-21unixbench_multi09:24:442
59299583,21cyclictest0-21swapper/211:39:342
592995828,8cyclictest141rcu_preempt11:29:342
592995826,20cyclictest0-21swapper/209:44:272
58399588,11cyclictest231rcuc/109:44:371
583995843,9cyclictest23683-21idleruntime-cro12:34:011
583995839,9cyclictest27370-21ls08:29:041
583995836,12cyclictest0-21swapper/112:34:261
603995733,13cyclictest0-21swapper/309:49:023
603995730,15cyclictest0-21swapper/309:19:223
603995730,15cyclictest0-21swapper/309:19:223
603995729,10cyclictest141rcu_preempt10:38:563
59299576,28cyclictest0-21swapper/208:04:032
592995741,10cyclictest27255-21basename08:29:022
592995724,19cyclictest141rcu_preempt08:39:372
592995724,19cyclictest141rcu_preempt08:39:372
59299572,35cyclictest6956-21grep09:04:432
592995714,19cyclictest363-21in:imklog08:08:222
592995713,18cyclictest141rcu_preempt08:37:082
583995734,12cyclictest141rcu_preempt10:01:511
583995732,8cyclictest141rcu_preempt10:37:041
583995731,7cyclictest141rcu_preempt09:37:191
583995725,17cyclictest141rcu_preempt11:11:421
583995721,18cyclictest141rcu_preempt12:07:401
583995719,11cyclictest141rcu_preempt10:22:261
603995640,10cyclictest4412-21kworker/u8:208:19:023
603995635,9cyclictest141rcu_preempt11:10:183
603995633,12cyclictest0-21swapper/311:49:033
603995631,7cyclictest141rcu_preempt10:15:093
603995631,15cyclictest0-21swapper/311:39:013
603995631,15cyclictest0-21swapper/311:39:013
603995631,12cyclictest141rcu_preempt11:24:573
603995631,12cyclictest141rcu_preempt11:24:573
603995615,22cyclictest41-21ksoftirqd/309:52:253
59299565,21cyclictest17184-21ssh12:21:492
592995631,8cyclictest28377-21kworker/u8:109:51:212
592995620,21cyclictest141rcu_preempt11:34:442
592995620,21cyclictest141rcu_preempt11:34:442
583995640,8cyclictest0-21swapper/110:54:011
583995632,7cyclictest141rcu_preempt09:12:571
583995627,7cyclictest141rcu_preempt07:13:441
583995625,13cyclictest141rcu_preempt09:30:401
60399559,7cyclictest391rcuc/311:39:443
60399554,26cyclictest141rcu_preempt11:04:263
603995522,15cyclictest141rcu_preempt11:29:553
603995521,14cyclictest141rcu_preempt07:51:253
592995540,9cyclictest52-21kcompactd012:07:352
592995533,14cyclictest0-21swapper/209:54:262
592995528,15cyclictest0-21swapper/210:09:482
583995530,12cyclictest141rcu_preempt08:56:281
583995529,7cyclictest141rcu_preempt10:24:401
583995529,14cyclictest141rcu_preempt12:15:041
583995526,15cyclictest141rcu_preempt10:46:231
583995526,14cyclictest141rcu_preempt08:03:071
583995522,15cyclictest141rcu_preempt10:54:061
583995513,15cyclictest141rcu_preempt09:24:391
603995431,11cyclictest141rcu_preempt10:45:083
603995427,7cyclictest141rcu_preempt10:40:013
58399548,11cyclictest25-21ksoftirqd/107:49:141
583995436,12cyclictest0-21swapper/108:39:021
583995424,17cyclictest52-21kcompactd008:53:161
583995418,9cyclictest231rcuc/108:59:301
603995322,15cyclictest141rcu_preempt08:49:143
58399539,26cyclictest25-21ksoftirqd/107:48:211
58399537,12cyclictest25-21ksoftirqd/109:54:121
583995336,11cyclictest23953-21cron08:19:021
583995333,13cyclictest15261-21unixbench_singl11:19:471
583995327,14cyclictest141rcu_preempt08:39:351
583995327,14cyclictest141rcu_preempt08:39:351
583995321,7cyclictest141rcu_preempt09:50:121
603995222,7cyclictest141rcu_preempt07:59:363
603995222,15cyclictest141rcu_preempt08:39:353
603995222,15cyclictest141rcu_preempt08:39:353
603995130,9cyclictest0-21swapper/308:04:253
603995122,13cyclictest141rcu_preempt08:09:313
592995129,13cyclictest14918-21ntp_kernel_pll_07:49:292
583995129,6cyclictest141rcu_preempt09:22:111
583995129,6cyclictest141rcu_preempt09:22:111
583995124,7cyclictest141rcu_preempt08:04:221
583995121,15cyclictest141rcu_preempt07:38:541
603995024,7cyclictest141rcu_preempt08:44:403
603995023,8cyclictest141rcu_preempt08:34:373
58399508,9cyclictest25-21ksoftirqd/109:09:021
58399503,22cyclictest25-21ksoftirqd/108:09:191
583995028,12cyclictest0-21swapper/107:39:301
583995022,13cyclictest141rcu_preempt08:19:141
583994924,6cyclictest141rcu_preempt08:44:341
583994919,7cyclictest141rcu_preempt07:54:381
583994910,10cyclictest25-21ksoftirqd/107:27:141
583994910,10cyclictest25-21ksoftirqd/107:27:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional