You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-12-10 - 22:17
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Tue Dec 10, 2024 12:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30609217219,32sleep00-21swapper/007:03:540
30609217219,32sleep00-21swapper/007:03:540
31013211618,57sleep331101-21grep07:04:063
31013211618,57sleep331101-21grep07:04:063
31621991114,18cyclictest11378-21apt-get07:43:350
316219911044,10cyclictest0-21swapper/011:33:230
3174021040,4chrt31738-21perl08:48:430
3174021040,4chrt31738-21perl08:48:430
31632999235,49cyclictest16210-21sh12:23:511
31621998946,11cyclictest12-21ksoftirqd/008:43:560
31632998836,43cyclictest0-21swapper/110:38:221
31632998830,8cyclictest0-21swapper/110:53:201
31632998630,11cyclictest23906-21ntp_kernel_pll_10:43:491
31621998612,50cyclictest913-21runrttasks08:53:470
99582850,7sleep00-21swapper/010:18:430
31632998323,10cyclictest0-21swapper/110:03:541
31632998319,19cyclictest10999-21grep12:13:581
3114628313,45sleep131154-21/usr/sbin/munin07:04:081
3114628313,45sleep131154-21/usr/sbin/munin07:04:081
3020128312,36sleep20-21swapper/207:03:402
3020128312,36sleep20-21swapper/207:03:402
31632998234,40cyclictest0-21swapper/112:11:061
31639998159,15cyclictest28311-21munin-plugin-st09:53:202
31632998125,9cyclictest0-21swapper/108:08:441
31632998040,17cyclictest25-21ksoftirqd/108:43:231
31639997913,38cyclictest26464-21cpu10:48:412
31632997932,39cyclictest0-21swapper/109:54:461
31632997925,10cyclictest0-21swapper/110:23:551
31632997924,12cyclictest0-21swapper/108:58:211
31632997923,10cyclictest0-21swapper/112:33:201
31647997832,22cyclictest627-21in:imklog12:36:043
3163299781,30cyclictest0-21swapper/111:33:441
3163299781,30cyclictest0-21swapper/111:33:441
31632997721,18cyclictest8647-21cat09:13:231
31647997651,15cyclictest17749-21unixbench_singl11:28:563
3164799753,32cyclictest41-21ksoftirqd/309:23:423
31632997520,12cyclictest0-21swapper/107:33:421
3163299751,14cyclictest141rcu_preempt09:48:461
31621997553,14cyclictest17492-21munin-plugin-st10:33:200
31621997549,8cyclictest12-21ksoftirqd/008:24:240
31621997525,24cyclictest0-21swapper/010:33:380
31632997420,22cyclictest0-21swapper/110:57:391
31621997428,24cyclictest13556-21ssh11:23:220
31647997347,19cyclictest21489-21sendmail08:18:213
31647997347,19cyclictest21489-21sendmail08:18:213
31632997327,8cyclictest0-21swapper/108:28:231
31632997313,19cyclictest11855-21grep11:18:591
31647997256,10cyclictest27403-21unixbench_multi10:49:013
31632997219,12cyclictest0-21swapper/112:02:261
31632997210,40cyclictest0-21swapper/107:43:351
31621997212,30cyclictest141rcu_preempt11:33:460
31621997212,30cyclictest141rcu_preempt11:33:460
31647997154,11cyclictest19861-21munin-plugin-st08:13:203
31647997152,13cyclictest23863-21munin-plugin-st09:43:203
31647997152,13cyclictest23863-21munin-plugin-st09:43:203
3163999717,58cyclictest19160-21apt-get11:33:342
3163999717,58cyclictest19160-21apt-get11:33:342
31639997152,12cyclictest7636-21munin-plugin-st07:33:212
31639997149,15cyclictest19260-21apt-key11:33:212
31632997147,17cyclictest21339-21latency_hist09:38:221
3164799708,30cyclictest21545-21kworker/3:112:03:533
3164799708,30cyclictest21545-21kworker/3:112:03:533
31647997051,12cyclictest2790-21basename07:18:213
31647997036,19cyclictest627-21in:imklog07:58:453
31647997035,20cyclictest141rcu_preempt10:33:343
31621997052,11cyclictest21583-21unixbench_multi10:38:560
31621997038,13cyclictest141rcu_preempt11:44:050
31621997038,13cyclictest141rcu_preempt11:44:050
31647996937,10cyclictest141rcu_preempt10:13:403
31647996937,10cyclictest141rcu_preempt10:13:403
31639996950,11cyclictest31048-21readlink09:58:212
3163299697,18cyclictest14503-21ssh09:23:431
31632996926,7cyclictest0-21swapper/108:48:421
31632996926,7cyclictest0-21swapper/108:48:421
31621996947,14cyclictest14131-21find09:23:230
31632996820,11cyclictest0-21swapper/108:43:471
31647996747,8cyclictest141rcu_preempt09:47:483
31647996743,15cyclictest0-21swapper/312:28:203
31639996740,8cyclictest141rcu_preempt09:34:182
31639996737,17cyclictest141rcu_preempt10:07:592
31621996743,15cyclictest28624-21ssh10:52:540
3162199673,50cyclictest26064-21/usr/sbin/munin09:48:460
31621996719,34cyclictest141rcu_preempt11:08:230
31621996719,34cyclictest141rcu_preempt11:08:230
31647996611,22cyclictest41-21ksoftirqd/312:18:573
31639996644,13cyclictest3798-21apt-key12:03:222
31639996640,14cyclictest0-21swapper/207:13:572
31639996638,14cyclictest141rcu_preempt09:21:372
31621996638,7cyclictest141rcu_preempt09:33:030
31647996531,9cyclictest116-21pr/ttymxc009:18:493
31647996512,28cyclictest141rcu_preempt12:28:543
31639996539,17cyclictest23029-21modprobe10:43:202
31632996545,13cyclictest626-21in:imuxsock09:43:201
31632996545,13cyclictest626-21in:imuxsock09:43:201
31621996546,10cyclictest0-21swapper/009:08:210
3164799648,33cyclictest11545-21df07:43:533
31647996443,14cyclictest627-21in:imklog12:16:553
31647996441,9cyclictest141rcu_preempt09:03:233
31647996441,9cyclictest141rcu_preempt09:03:233
31647996440,9cyclictest141rcu_preempt07:08:443
3163999646,14cyclictest33-21ksoftirqd/211:23:202
31639996446,11cyclictest22463-21runrttasks08:18:542
31639996445,12cyclictest275-21systemd-journal12:13:552
31639996444,11cyclictest245-21jbd2/mmcblk2p2-12:04:022
31639996444,11cyclictest245-21jbd2/mmcblk2p2-12:04:022
31639996434,8cyclictest141rcu_preempt12:36:312
31632996444,14cyclictest4576-21runrttasks11:08:201
31632996444,14cyclictest4576-21runrttasks11:08:201
31632996440,15cyclictest4068-21apt-key09:03:231
31632996440,15cyclictest4068-21apt-key09:03:231
31632996439,16cyclictest627-21in:imklog07:48:571
31632996438,17cyclictest18708-21latency_hist09:33:231
31621996433,24cyclictest15903-21sh09:28:200
31621996415,22cyclictest141rcu_preempt07:13:310
3164799637,37cyclictest0-21swapper/312:08:353
31647996347,11cyclictest0-21swapper/308:58:213
31647996336,8cyclictest141rcu_preempt11:01:453
31647996310,17cyclictest41-21ksoftirqd/309:13:433
31639996346,12cyclictest31612-21cyclictest11:43:412
31639996346,12cyclictest31612-21cyclictest11:43:412
31639996340,16cyclictest19860-21idleruntime-cro08:13:202
31639996333,17cyclictest141rcu_preempt11:07:022
31639996333,17cyclictest141rcu_preempt11:07:022
31639996333,14cyclictest141rcu_preempt07:22:262
31632996343,12cyclictest0-21swapper/111:33:201
31632996341,15cyclictest275-21systemd-journal07:23:241
31632996336,7cyclictest141rcu_preempt11:50:291
31632996316,16cyclictest25-21ksoftirqd/107:38:561
31621996342,14cyclictest0-21swapper/011:58:230
31621996341,13cyclictest6461-21runrttasks07:28:420
31647996246,9cyclictest6126-21cpuspeed09:08:383
31639996246,9cyclictest2245-21sendmail08:58:212
31632996230,8cyclictest141rcu_preempt09:18:461
31621996245,10cyclictest0-21swapper/012:08:410
31621996241,12cyclictest30864-21ssh10:56:050
31621996237,15cyclictest627-21in:imklog12:37:190
31621996219,33cyclictest8150irq/69-2188000.12:18:470
31621996215,15cyclictest275-21systemd-journal08:18:570
31647996143,12cyclictest275-21systemd-journal11:43:473
31647996143,12cyclictest275-21systemd-journal11:43:473
31647996138,12cyclictest0-21swapper/310:29:003
3164799613,31cyclictest0-21swapper/308:33:443
31647996131,8cyclictest141rcu_preempt11:33:523
31647996131,8cyclictest141rcu_preempt11:33:523
31639996148,8cyclictest0-21swapper/208:03:452
31639996137,7cyclictest141rcu_preempt11:42:492
31639996133,7cyclictest141rcu_preempt09:15:232
31639996130,7cyclictest141rcu_preempt11:15:222
31639996119,23cyclictest394-21runrttasks11:02:172
31639996113,12cyclictest33-21ksoftirqd/210:08:562
3163299618,30cyclictest141rcu_preempt09:04:031
31647996030,8cyclictest41-21ksoftirqd/311:58:053
31647996026,9cyclictest141rcu_preempt07:18:453
31647996013,30cyclictest21040-21sed10:38:453
31647996013,30cyclictest0-21swapper/309:53:413
31639996041,9cyclictest0-21swapper/209:58:552
3163999604,17cyclictest33-21ksoftirqd/211:48:562
31639996035,14cyclictest13066-21sort07:48:412
31639996035,12cyclictest0-21swapper/207:13:232
31639996028,26cyclictest0-21swapper/209:03:352
31639996027,13cyclictest141rcu_preempt11:10:032
31639996026,7cyclictest141rcu_preempt12:21:322
3163299607,27cyclictest141rcu_preempt08:03:551
31647995944,9cyclictest21235-21systemd-run09:38:213
31647995943,10cyclictest4849-21runrttasks10:08:533
31647995934,12cyclictest627-21in:imklog08:48:003
31647995927,9cyclictest141rcu_preempt08:38:423
31647995924,11cyclictest141rcu_preempt11:03:463
31647995924,11cyclictest141rcu_preempt11:03:463
31647995920,12cyclictest25991-21ssh09:48:463
31647995918,8cyclictest141rcu_preempt07:23:553
31647995913,20cyclictest7903-21apt-get07:33:393
3163999595,17cyclictest33-21ksoftirqd/210:33:422
31639995943,9cyclictest0-21swapper/207:44:102
31639995934,7cyclictest141rcu_preempt09:29:002
31639995928,7cyclictest141rcu_preempt09:39:412
31639995928,7cyclictest141rcu_preempt09:39:412
3163299599,6cyclictest141rcu_preempt09:58:371
31632995942,10cyclictest25914-21grep08:29:001
31632995941,10cyclictest13842-21unixbench_singl12:18:581
31632995918,23cyclictest0-21swapper/108:38:221
31621995942,9cyclictest5305-21unixbench_multi12:04:120
31621995942,9cyclictest5305-21unixbench_multi12:04:120
31621995942,10cyclictest21479-21cron08:18:210
31621995942,10cyclictest21479-21cron08:18:210
31621995934,15cyclictest0-21swapper/007:08:440
31621995912,27cyclictest141rcu_preempt12:02:490
31647995842,8cyclictest16390-21idleruntime-cro11:28:203
31647995842,8cyclictest0-21swapper/308:18:433
31647995841,9cyclictest0-21swapper/308:48:563
31647995841,9cyclictest0-21swapper/308:48:563
31647995839,10cyclictest0-21swapper/307:43:213
31647995832,14cyclictest41-21ksoftirqd/311:59:153
31647995817,22cyclictest23890-21ntpq10:43:483
3163999589,15cyclictest33-21ksoftirqd/212:33:232
3163999583,12cyclictest33-21ksoftirqd/207:38:452
31639995828,16cyclictest141rcu_preempt10:43:262
31632995840,12cyclictest1319-21fschecks_count07:13:311
31632995839,11cyclictest0-21swapper/110:08:561
31632995811,21cyclictest141rcu_preempt10:28:571
31632995810,23cyclictest0-21swapper/109:13:551
31621995820,32cyclictest12-21ksoftirqd/010:03:420
31621995819,29cyclictest0-21swapper/011:08:490
3164799577,32cyclictest0-21swapper/311:10:203
31647995737,13cyclictest116-21pr/ttymxc010:04:033
31647995734,8cyclictest141rcu_preempt09:58:443
31647995729,13cyclictest116-21pr/ttymxc011:39:003
31647995714,27cyclictest627-21in:imklog11:14:443
31639995736,6cyclictest141rcu_preempt12:24:392
31639995731,7cyclictest141rcu_preempt10:28:322
31639995723,15cyclictest141rcu_preempt10:14:512
31639995723,15cyclictest141rcu_preempt10:14:512
31639995711,21cyclictest33-21ksoftirqd/210:18:592
31632995736,12cyclictest275-21systemd-journal11:24:241
31632995735,11cyclictest0-21swapper/108:18:231
31632995735,11cyclictest0-21swapper/108:18:231
31632995734,11cyclictest0-21swapper/112:33:561
31632995727,9cyclictest141rcu_preempt11:14:011
3163299572,18cyclictest6843-21latency_hist12:08:221
3163299572,18cyclictest6843-21latency_hist12:08:221
31621995735,6cyclictest141rcu_preempt09:36:400
31621995732,8cyclictest12-21ksoftirqd/012:23:510
31621995732,13cyclictest0-21swapper/010:13:540
31621995732,13cyclictest0-21swapper/010:13:540
31621995727,8cyclictest141rcu_preempt09:11:100
31647995624,19cyclictest141rcu_preempt07:28:383
31647995611,29cyclictest0-21swapper/309:03:313
31647995611,12cyclictest0-21swapper/311:48:423
3163999567,28cyclictest33-21ksoftirqd/207:27:062
31639995632,13cyclictest0-21swapper/208:48:422
31639995632,13cyclictest0-21swapper/208:48:422
3163999563,12cyclictest33-21ksoftirqd/208:28:232
31639995630,17cyclictest29824-21latency_hist08:43:242
31639995626,16cyclictest24718-21kworker/u8:209:11:532
3163299567,23cyclictest0-21swapper/111:53:431
31632995637,12cyclictest0-21swapper/110:38:561
31632995631,17cyclictest9132-21sh10:17:591
31632995631,17cyclictest9132-21sh10:17:591
31632995618,17cyclictest627-21in:imklog11:47:441
31632995618,17cyclictest627-21in:imklog11:47:441
31632995610,19cyclictest141rcu_preempt10:23:201
31621995640,10cyclictest5748-21diskmemload09:53:370
31621995638,11cyclictest627-21in:imklog07:23:470
31621995630,9cyclictest141rcu_preempt12:28:420
3162199562,10cyclictest26787-21fschecks_count08:33:420
3164799556,20cyclictest3594-21kworker/3:209:28:473
31647995519,17cyclictest0-21swapper/310:54:103
31639995538,9cyclictest321ktimers/210:53:552
31639995529,13cyclictest141rcu_preempt09:25:412
31639995528,14cyclictest141rcu_preempt08:35:152
31639995522,7cyclictest141rcu_preempt11:58:222
31639995517,21cyclictest33-21ksoftirqd/212:11:332
31639995510,16cyclictest141rcu_preempt09:48:202
31621995536,10cyclictest26042-21cmp10:48:220
3162199553,18cyclictest394-21runrttasks10:23:370
3162199553,16cyclictest32393-21head09:58:550
31621995518,19cyclictest141rcu_preempt09:03:050
31621995518,19cyclictest141rcu_preempt09:03:050
31621995518,12cyclictest12-21ksoftirqd/010:08:410
31621995515,19cyclictest141rcu_preempt07:34:010
31621995513,13cyclictest8150irq/69-2188000.11:38:510
3164799545,30cyclictest0-21swapper/310:18:583
3164799544,12cyclictest0-21swapper/307:51:173
31647995431,14cyclictest627-21in:imklog10:26:223
31647995429,7cyclictest41-21ksoftirqd/308:24:243
31647995422,17cyclictest141rcu_preempt08:03:373
31639995431,12cyclictest0-21swapper/211:23:562
31639995428,13cyclictest0-21swapper/208:18:232
31639995428,13cyclictest0-21swapper/208:18:232
31639995420,6cyclictest141rcu_preempt10:23:522
31639995411,15cyclictest0-21swapper/207:58:382
3163299542,25cyclictest32075-21ssh10:58:401
31632995420,20cyclictest141rcu_preempt11:38:481
31632995410,25cyclictest141rcu_preempt07:18:451
31621995438,10cyclictest14988-21idleruntime-cro07:58:200
31621995435,11cyclictest15606-21ssh11:26:030
31621995430,14cyclictest0-21swapper/008:28:570
31647995319,8cyclictest0-21swapper/308:28:563
31647995314,12cyclictest141rcu_preempt07:58:223
31639995314,10cyclictest0-21swapper/208:43:492
3163299532,14cyclictest5148-21ssh11:08:451
31647995211,10cyclictest141rcu_preempt11:18:493
31639995232,8cyclictest141rcu_preempt09:01:492
31639995232,8cyclictest141rcu_preempt09:01:492
31639995230,16cyclictest31612-21cyclictest08:28:432
31632995220,17cyclictest141rcu_preempt07:28:461
3162199524,9cyclictest0-21swapper/009:41:100
3162199524,9cyclictest0-21swapper/009:41:100
31621995236,7cyclictest0-21swapper/012:13:420
31621995236,10cyclictest275-21systemd-journal11:15:420
3162199522,27cyclictest29170-21grep08:38:550
31621995221,18cyclictest141rcu_preempt08:03:570
31632995126,7cyclictest141rcu_preempt07:58:331
31632995123,9cyclictest179-21pr/ttyS009:48:111
3162199513,38cyclictest0-21swapper/009:45:140
31621995129,13cyclictest0-21swapper/009:13:570
3162199509,11cyclictest0-21swapper/010:58:230
3162199507,20cyclictest275-21systemd-journal07:18:430
3162199506,27cyclictest141rcu_preempt07:38:360
3162199503,17cyclictest29235-21ssh11:49:040
31639994924,7cyclictest141rcu_preempt07:55:382
31639994923,7cyclictest141rcu_preempt07:33:502
31632994929,11cyclictest0-21swapper/108:18:561
3162199493,17cyclictest15447-21fschecks_count07:58:370
31621994927,11cyclictest0-21swapper/008:08:420
31632994833,9cyclictest32232-21latency07:08:471
31621994829,6cyclictest141rcu_preempt07:52:560
31632994416,14cyclictest141rcu_preempt07:58:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional