You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 04:13
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8.osadl.org (updated Fri Jul 26, 2024 12:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26092312300,1sleep20-21swapper/212:20:002
2178679992623,2cyclictest2456423-21systemd-run10:35:000
2178689992421,1cyclictest0-21swapper/308:49:593
2178684992420,3cyclictest496-21avahi-daemon08:20:002
2178689992219,1cyclictest0-21swapper/310:20:013
2178679992219,2cyclictest0-21swapper/008:35:000
2178689992118,1cyclictest0-21swapper/310:15:003
217868499216,8cyclictest2418193-21munin-run10:10:012
2178684992118,2cyclictest507-21systemd-logind09:53:032
2178679992118,2cyclictest2184707-21latency_hist07:15:010
2178679992118,2cyclictest0-21swapper/011:00:000
2178679992118,2cyclictest0-21swapper/007:20:010
2178679992118,2cyclictest0-21swapper/007:20:000
2178684992017,2cyclictest792-21gdbus08:57:152
2178684992017,2cyclictest2433364-21latency_hist10:20:002
2178684992015,3cyclictest2558342-21kthreadcore11:40:182
2178689991916,1cyclictest0-21swapper/310:00:013
2178689991916,1cyclictest0-21swapper/309:53:033
2178689991916,1cyclictest0-21swapper/309:40:013
2178684991916,2cyclictest2532471-21latency_hist11:25:002
2178680991914,3cyclictest2408872-21grep10:00:321
2178679991916,2cyclictest2547915-21tr11:35:100
2178679991915,2cyclictest2308654-21cpuspeed_turbos08:55:140
2178689991816,1cyclictest0-21swapper/311:30:193
2178689991814,1cyclictest0-21swapper/309:25:013
2178689991814,1cyclictest0-21swapper/309:25:003
2178684991814,3cyclictest2485058-21unixbench_multi10:50:282
2178680991815,2cyclictest757-21systemctl12:30:001
2178679991814,2cyclictest2517791-21gltestperf11:15:140
2178679991814,2cyclictest0-21swapper/011:55:010
2178679991814,2cyclictest0-21swapper/010:10:000
2178679991814,2cyclictest0-21swapper/009:55:010
2178679991813,4cyclictest2340178-21unixbench_multi09:15:220
2178679991811,5cyclictest361-21systemd-journal10:25:010
22750362170,1sleep10-21swapper/108:25:161
2178689991714,1cyclictest0-21swapper/312:29:593
2178689991714,1cyclictest0-21swapper/312:20:003
2178689991714,1cyclictest0-21swapper/311:40:173
2178689991714,1cyclictest0-21swapper/310:59:593
2178689991714,1cyclictest0-21swapper/309:40:183
2178689991714,1cyclictest0-21swapper/309:30:233
2178684991714,2cyclictest2533130-21ps11:25:172
2178684991714,2cyclictest2517344-21latency_hist11:15:002
2178684991714,2cyclictest2223293-21turbostat.cron07:45:002
2178684991713,3cyclictest0-21swapper/210:05:002
2178684991712,3cyclictest2266260-21cpuspeed_turbos08:20:132
2178684991711,5cyclictest3206876-21kworker/u8:2-ext4-rsv-conversion07:15:002
2178680991715,1cyclictest2357258-21latency_hist09:30:011
2178680991715,1cyclictest2184656-21cron07:15:011
2178680991713,3cyclictest2562660-21cron11:45:011
2178680991713,2cyclictest0-21swapper/110:29:591
2178680991713,2cyclictest0-21swapper/108:20:001
2178680991713,2cyclictest0-21swapper/107:25:141
2178679991714,2cyclictest503-21polkitd08:10:000
2178679991714,2cyclictest2456956-21/usr/sbin/munin10:35:200
2178679991714,2cyclictest2433734-21apt-key10:20:010
2178679991714,2cyclictest2326499-21latency_hist09:10:010
2178679991714,2cyclictest0-21swapper/010:45:240
2178679991713,3cyclictest2289110-21needreboot08:35:200
2178679991713,2cyclictest0-21swapper/009:40:010
2178679991712,4cyclictest2627814-21gltestperf12:35:130
2178679991712,4cyclictest2421137-21kthreadcore10:10:180
2178679991712,3cyclictest2596516-21grep12:05:250
2178689991613,1cyclictest0-21swapper/312:36:183
2178689991613,1cyclictest0-21swapper/311:45:133
2178689991613,1cyclictest0-21swapper/311:35:183
2178689991613,1cyclictest0-21swapper/311:10:203
2178689991613,1cyclictest0-21swapper/310:33:393
2178689991613,1cyclictest0-21swapper/309:16:163
2178689991613,1cyclictest0-21swapper/308:35:183
2178689991613,1cyclictest0-21swapper/308:30:143
2178689991613,1cyclictest0-21swapper/307:20:493
2178684991613,2cyclictest3291653-21kworker/u8:1-rpciod09:15:002
2178684991613,2cyclictest2503307-21irqrtprio11:05:172
2178684991613,2cyclictest2448827-21sh10:29:592
2178684991613,2cyclictest0-21swapper/211:04:592
2178684991612,3cyclictest2461847-21needreboot10:35:202
2178684991611,4cyclictest2627201-21cron12:34:592
2178684991611,3cyclictest2235864-21fschecks_count07:55:132
217868099163,12cyclictest0-21swapper/108:00:181
2178680991614,1cyclictest2464150-21cron10:39:591
2178680991614,1cyclictest0-21swapper/110:25:011
2178680991613,2cyclictest2525425-21gltestperf11:20:141
2178680991613,2cyclictest2494462-21sendmail11:00:001
2178680991613,2cyclictest2462562-21kworker/u8:2-rpciod10:40:171
2178680991613,2cyclictest2335369-21irqcore09:15:131
2178680991613,2cyclictest2198875-21systemd-run07:25:001
2178680991612,3cyclictest2248040-21gltestperf08:05:131
217867999162,12cyclictest2499797-21ntp_states11:00:220
2178679991613,2cyclictest503-21polkitd11:10:240
2178679991613,2cyclictest2308238-21basename08:54:590
2178679991612,2cyclictest0-21swapper/011:25:160
21785322161,12sleep22178613-21wc07:10:012
217868999152,9cyclictest25850372sleep312:00:123
2178689991513,1cyclictest0-21swapper/310:45:013
2178689991513,1cyclictest0-21swapper/307:45:003
2178689991512,1cyclictest0-21swapper/312:30:193
2178689991512,1cyclictest0-21swapper/311:59:023
2178689991512,1cyclictest0-21swapper/311:20:173
2178689991512,1cyclictest0-21swapper/310:05:203
2178689991512,1cyclictest0-21swapper/309:45:213
2178689991512,1cyclictest0-21swapper/309:27:543
2178689991512,1cyclictest0-21swapper/308:55:003
2178689991512,1cyclictest0-21swapper/308:04:183
2178689991512,1cyclictest0-21swapper/307:40:013
2178689991512,1cyclictest0-21swapper/307:25:193
2178689991512,1cyclictest0-21swapper/307:15:003
217868499159,5cyclictest361-21systemd-journal12:00:002
217868499159,4cyclictest361-21systemd-journal11:20:002
2178684991512,2cyclictest2570226-21turbostat.cron11:50:002
2178684991512,2cyclictest2370106-21ntp_kernel_pll_09:35:212
2178684991512,2cyclictest2326415-21diskmemload10:45:012
2178684991511,3cyclictest2575479-21ntp_kernel_err11:50:202
217868499150,2cyclictest2192761-21idleruntime-cro07:20:012
217868499150,2cyclictest2192761-21idleruntime-cro07:20:012
2178680991513,1cyclictest2326686-21kworker/u8:4-rpciod09:45:141
2178680991512,2cyclictest2633308-21sh12:40:001
2178680991512,2cyclictest2517708-21fschecks_time11:15:141
2178680991512,2cyclictest0-21swapper/108:57:151
2178680991511,3cyclictest2393168-21ntp_kernel_pll_09:50:211
2178680991511,3cyclictest2369964-21missed_timers09:35:191
2178680991511,3cyclictest2315671-21irqrtprio09:00:191
2178680991511,3cyclictest2301710-21unixbench-2d08:45:261
2178680991511,2cyclictest0-21swapper/110:15:241
217867999159,5cyclictest516-21rs:main0
217867999159,4cyclictest2400944-21systemctl09:55:210
2178679991512,2cyclictest521-21gdbus12:00:230
2178679991512,2cyclictest2442034-21irqcore10:25:140
2178679991512,2cyclictest2326686-21kworker/u8:4-rpciod09:35:000
2178679991512,2cyclictest2266244-21tr08:20:130
2178679991512,2cyclictest2169359-21kworker/0:0+kacpid07:50:260
2178679991512,2cyclictest0-21swapper/008:10:160
2178679991512,2cyclictest0-21swapper/007:25:000
2178679991511,3cyclictest2295631-21unixbench-2d08:40:210
217867999150,14cyclictest2320391-21awk09:04:590
217867999150,13cyclictest2627225-21turbostat12:34:590
2178689991412,1cyclictest0-21swapper/307:47:293
2178689991411,1cyclictest0-21swapper/311:55:013
2178689991411,1cyclictest0-21swapper/311:25:203
2178689991411,1cyclictest0-21swapper/311:15:183
2178689991411,1cyclictest0-21swapper/311:09:223
2178689991411,1cyclictest0-21swapper/311:00:133
2178689991411,1cyclictest0-21swapper/310:54:053
2178689991411,1cyclictest0-21swapper/310:04:553
2178689991411,1cyclictest0-21swapper/309:10:193
2178689991411,1cyclictest0-21swapper/308:55:183
2178689991411,1cyclictest0-21swapper/308:15:173
2178689991411,1cyclictest0-21swapper/307:59:473
2178689991411,1cyclictest0-21swapper/307:30:273
2178689991410,1cyclictest0-21swapper/308:25:153
2178684991411,2cyclictest496-21avahi-daemon09:00:302
2178684991411,2cyclictest2616312-21tr12:25:152
2178684991411,2cyclictest2545331-21sendmail_mailqu11:30:242
2178684991411,2cyclictest2451474-21kthreadcore10:30:172
2178684991411,2cyclictest2382422-21kthreadcore09:45:202
2178684991411,2cyclictest2296550-21cpuspeed_turbos08:45:132
2178684991411,2cyclictest2272541-21gltestperf08:25:132
2178684991410,3cyclictest2439408-21unixbench_multi10:20:252
2178684991410,3cyclictest2423643-21sendmail_mailqu10:10:252
2178684991410,3cyclictest0-21swapper/208:45:012
2178680991411,2cyclictest2602009-21wc12:10:191
2178680991411,2cyclictest2547883-21cpuspeed_turbos11:35:101
2178680991411,2cyclictest2477451-21unixbench-2d10:45:251
2178680991411,2cyclictest2423939-21unixbench-2d10:10:271
2178680991411,2cyclictest2403258-21latency_hist10:00:011
2178680991411,2cyclictest2307845-21unixbench_singl08:50:261
2178680991411,2cyclictest2254718-21irqrtprio08:10:191
2178680991411,2cyclictest2205334-21cpuspeed_turbos07:30:121
2178680991411,2cyclictest0-21swapper/107:50:281
2178680991411,1cyclictest0-21swapper/110:05:151
2178680991410,3cyclictest2453321-21kthreadcore10:30:181
2178680991410,3cyclictest2238539-21kthreadcore07:55:181
217868099140,5cyclictest26130722cut12:20:201
217868099140,5cyclictest26092292cat12:20:011
217867999149,4cyclictest0-21swapper/007:35:140
217867999148,4cyclictest2377535-21sshd09:40:180
2178679991412,1cyclictest2409722-21kworker/0:2+kacpid12:13:370
2178679991412,1cyclictest2169359-21kworker/0:0+kacpid08:00:580
2178679991412,1cyclictest2169359-21kworker/0:0+kacpid07:48:490
2178679991411,2cyclictest2609620-21cut12:20:150
2178679991411,2cyclictest2527856-21kthreadcore11:20:190
2178679991411,2cyclictest2238539-21kthreadcore07:55:170
2178679991411,2cyclictest2205348-21tr07:30:120
2178679991410,3cyclictest2260137-21gltestperf08:15:120
2178679991410,2cyclictest2352899-21kthreadcore09:25:180
217868999139,1cyclictest0-21swapper/308:20:163
217868999135,6cyclictest24793402chrt10:49:593
2178689991311,1cyclictest0-21swapper/307:16:363
2178689991311,1cyclictest0-21swapper/307:16:353
2178689991310,1cyclictest0-21swapper/312:06:173
2178689991310,1cyclictest0-21swapper/310:35:193
2178689991310,1cyclictest0-21swapper/310:20:253
2178689991310,1cyclictest0-21swapper/309:05:033
2178689991310,1cyclictest0-21swapper/308:14:343
2178689991310,1cyclictest0-21swapper/308:05:383
2178689991310,1cyclictest0-21swapper/307:50:233
217868499139,2cyclictest516-21rs:main2
217868499139,2cyclictest516-21rs:main2
217868499137,4cyclictest361-21systemd-journal10:49:592
217868499137,4cyclictest2350500-21sed09:25:162
2178684991310,2cyclictest2592067-21irqrtprio12:05:152
2178684991310,2cyclictest2364823-21idleruntime-cro09:35:012
2178684991310,2cyclictest2339101-21kthreadcore09:15:162
2178684991310,2cyclictest2305348-21kthreadcore08:50:192
2178684991310,2cyclictest2242177-21sed08:00:142
2178684991310,2cyclictest2212591-21kernelversion07:35:202
2178684991310,2cyclictest0-21swapper/208:30:152
217868099139,3cyclictest2223876-21gltestperf07:45:151
217868099138,4cyclictest2342747-21smartctl09:20:151
217868099138,4cyclictest2342747-21smartctl09:20:151
2178680991310,2cyclictest2589544-21kthreadcore12:00:191
2178680991310,2cyclictest2567104-21kthreadcore11:45:181
2178680991310,2cyclictest2542692-21kthreadcore11:30:201
2178680991310,2cyclictest2532932-21gltestperf11:25:151
2178680991310,2cyclictest2512140-21kthreadcore11:10:191
2178680991310,2cyclictest2502686-21gltestperf11:05:151
2178680991310,2cyclictest2326415-21diskmemload11:55:171
2178680991310,2cyclictest2326032-21unixbench_singl09:05:281
2178680991310,2cyclictest0-21swapper/109:30:171
2178680991310,2cyclictest0-21swapper/109:10:171
217867999139,3cyclictest2555777-21gltestperf11:40:140
217867999139,3cyclictest2219923-21kthreadcore07:40:200
217867999137,5cyclictest516-21rs:main0
217867999137,5cyclictest516-21rs:main0
217867999135,7cyclictest0-21swapper/010:42:200
2178679991311,1cyclictest2409722-21kworker/0:2+kacpid12:25:470
2178679991311,1cyclictest2409722-21kworker/0:2+kacpid12:17:560
2178679991311,1cyclictest2409722-21kworker/0:2+kacpid10:51:470
2178679991311,1cyclictest2169359-21kworker/0:0+kacpid08:45:360
2178679991311,1cyclictest2169359-21kworker/0:0+kacpid08:25:460
2178679991311,1cyclictest2169359-21kworker/0:0+kacpid07:25:350
2178679991310,2cyclictest2545112-21tr11:30:220
2178679991310,2cyclictest2403365-21/usr/sbin/munin10:00:290
217867999130,4cyclictest2565405-21cut11:45:170
217867999130,4cyclictest2326986-21grep09:10:130
217867999130,12cyclictest2506781-21kthreadcore11:05:190
217868999129,1cyclictest0-21swapper/312:10:183
217868999129,1cyclictest0-21swapper/310:26:463
217868999129,1cyclictest0-21swapper/309:00:273
217868999129,1cyclictest0-21swapper/308:41:003
2178689991210,1cyclictest0-21swapper/312:20:093
217868499129,2cyclictest2373304-21gltestperf09:40:142
217868499129,2cyclictest2223781-21fschecks_time07:45:142
217868499129,2cyclictest2204158-21sendmail_mailqu07:25:232
217868499129,2cyclictest0-21swapper/211:35:132
217868499129,2cyclictest0-21swapper/210:00:012
217868499128,3cyclictest2632370-21ntp_kernel_pll_12:35:192
217868499128,3cyclictest2539686-21kworker/u8:2-flush-8:012:05:012
217868499128,3cyclictest2254144-21cut08:10:162
217868499126,4cyclictest2320819-21diskstats09:05:132
217868499126,4cyclictest2206092-21seq07:30:172
2178684991210,1cyclictest0-21swapper/210:55:132
217868099129,2cyclictest2595338-21kthreadcore12:05:181
217868099129,2cyclictest2571888-21irqrtprio11:50:161
217868099129,2cyclictest2292934-21kthreadcore08:40:151
217868099129,2cyclictest2185270-21kthreadcore07:15:021
217868099129,2cyclictest2185270-21kthreadcore07:15:021
217868099129,2cyclictest0-21swapper/109:40:241
217868099129,2cyclictest0-21swapper/107:35:191
217868099129,2cyclictest0-21swapper/107:10:011
217868099127,4cyclictest2479656-21awk10:50:121
2178680991210,1cyclictest2495602-21irqcore11:00:171
217867999126,4cyclictest2385553-21ntpq09:45:240
217868499119,1cyclictest2609690-21grep12:20:162
217868499118,2cyclictest431ktimers/207:50:142
217868499117,2cyclictest0-21swapper/208:05:182
217868099119,1cyclictest0-21swapper/108:35:141
217868099116,4cyclictest2282971-21sed08:30:181
217868099110,1cyclictest0-21swapper/108:23:081
217868099110,10cyclictest0-21swapper/107:42:421
217867999117,2cyclictest0-21swapper/007:10:010
217868499107,2cyclictest2194854-21kthreadcore07:20:192
217868499107,2cyclictest0-21swapper/208:35:172
217868099105,4cyclictest2622774-21sed12:30:151
21786849996,2cyclictest0-21swapper/212:10:152
21786899986,1cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional