You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 21:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Fri Nov 21, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22206989913030,1299cyclictest26823352sleep312:19:563
2220698994030,400cyclictest2439289-21turbostat09:54:553
2220698993520,349cyclictest2617195-21turbostat11:44:563
2220698991916,1cyclictest0-21swapper/307:59:593
2220698991815,1cyclictest0-21swapper/310:00:153
2220698991714,1cyclictest0-21swapper/308:40:003
2220698991713,1cyclictest0-21swapper/310:45:003
222069899171,12cyclictest2317821-21turbostat08:25:003
222069899165,8cyclictest2601075-21turbostat11:30:013
2220698991613,1cyclictest0-21swapper/312:14:343
2220698991613,1cyclictest0-21swapper/309:10:123
222069899161,12cyclictest2609186-21taskset11:35:003
222069899161,11cyclictest2227003-21taskset07:12:243
222069899155,7cyclictest2631417-21taskset11:45:263
222069899155,7cyclictest2396428-21taskset09:20:203
2220698991513,1cyclictest0-21swapper/311:00:423
2220698991512,1cyclictest0-21swapper/311:05:323
222069899151,11cyclictest2714722-21turbostat12:39:593
222069899151,11cyclictest2641435-21turbostat11:55:003
222069899151,11cyclictest2512518-21taskset10:35:133
222069899151,11cyclictest2439281-21turbostat09:50:013
222069899151,11cyclictest2311545-21turbostat08:20:013
222069899151,11cyclictest2274131-21taskset07:50:143
222069899151,10cyclictest2390753-21turbostat09:20:003
190898150,4rtkit-daemon2697973-21ssh12:28:420
24557242141,3sleep20-21swapper/210:00:122
23686262140,4sleep02368659-21apt-get09:05:150
222069899146,6cyclictest25360002sleep310:48:533
222069899144,7cyclictest25438762sleep310:53:273
2220698991412,1cyclictest0-21swapper/308:08:103
2220698991412,1cyclictest0-21swapper/307:15:103
2220698991412,0cyclictest0-21swapper/312:26:363
2220698991412,0cyclictest0-21swapper/307:37:453
2220698991411,1cyclictest0-21swapper/311:11:393
2220698991411,1cyclictest0-21swapper/308:51:213
222069899141,10cyclictest2688566-21taskset12:20:293
222069899141,10cyclictest2650168-21taskset12:00:133
222069899141,10cyclictest2641724-21taskset11:55:083
222069899141,10cyclictest2585119-21taskset11:20:013
222069899141,10cyclictest2585119-21taskset11:20:013
222069899141,10cyclictest2577089-21taskset11:15:093
222069899141,10cyclictest2552584-21turbostat11:00:003
222069899141,10cyclictest2504406-21taskset10:30:163
222069899141,10cyclictest2502841-21taskset10:28:093
222069899141,10cyclictest2420198-21taskset09:35:193
222069899141,10cyclictest2398979-21taskset09:25:013
222069899141,10cyclictest2366004-21taskset09:00:173
222069899141,10cyclictest2366004-21taskset09:00:173
222069899141,10cyclictest2304910-21taskset08:10:363
222069899140,11cyclictest2330378-21chrt08:31:533
190898140,4rtkit-daemon2527346-21ssh10:43:080
190898140,2rtkit-daemon0-21swapper/110:51:221
24459312130,10sleep12445939-21ssh09:51:451
222069899131,9cyclictest2704752-21taskset12:30:343
222069899131,9cyclictest2484889-21taskset10:15:183
222069899131,9cyclictest2468792-21taskset10:05:203
222069899131,9cyclictest2374429-21taskset09:07:193
222069899131,9cyclictest2274074-21taskset07:50:013
222069899131,9cyclictest2254149-21taskset07:30:223
222069899131,9cyclictest2242844-21taskset07:25:143
222069899131,8cyclictest24476022sleep309:55:113
2220698991312,0cyclictest0-21swapper/311:35:283
2220698991312,0cyclictest0-21swapper/308:47:053
2220698991311,1cyclictest0-21swapper/307:41:423
222069899131,10cyclictest2414597-21taskset09:34:273
2220698991311,0cyclictest0-21swapper/308:44:553
2220698991311,0cyclictest0-21swapper/307:20:293
2220698991310,1cyclictest0-21swapper/308:55:193
222069899130,10cyclictest2658417-21taskset12:05:023
222069899130,10cyclictest2429616-21taskset09:41:163
222069899130,10cyclictest2323424-21taskset08:25:193
26582022120,4sleep22657930-21apt-get12:05:022
24311492121,9sleep12431157-21ssh09:44:471
22866812121,7sleep02286714-21apt-get08:00:130
222069899120,9cyclictest2493756-21taskset10:20:353
222069899120,9cyclictest2476834-21taskset10:10:183
222069899120,10cyclictest2286669-21chrt08:00:033
24996082110,1sleep02500981-21memory10:25:180
190898110,7rtkit-daemon2710209-21cut12:35:170
190898110,7rtkit-daemon2471276-21ssh10:09:340
190898110,4rtkit-daemon2412418-21ssh09:30:201
190898110,4rtkit-daemon1679647-21kworker/0:0+kacpid07:39:140
190898110,3rtkit-daemon667-21snmpd11:43:410
26270742100,4sleep22628233-21kthreadcore11:45:182
26245082101,6sleep12624514-21ssh11:43:181
25991182100,3sleep22599126-21ssh11:25:392
24559042100,7sleep02456109-21hddtemp_smartct10:00:140
22269842101,5sleep22226992-21unattended-upgr07:12:242
190898100,6rtkit-daemon2447123-21ssh09:54:320
190898100,6rtkit-daemon2350143-21tail08:50:140
190898100,5rtkit-daemon2681901-21ssh12:18:560
190898100,5rtkit-daemon2666732-21/usr/sbin/munin12:10:150
190898100,5rtkit-daemon2633364-21munin-run11:50:000
190898100,5rtkit-daemon2488000-21apt-get10:20:120
190898100,5rtkit-daemon2477675-21ssh10:10:420
190898100,5rtkit-daemon2406646-21ssh09:29:422
190898100,3rtkit-daemon667-21snmpd10:39:360
2682364290,7sleep22682365-21cron12:20:002
2644191291,5sleep22646365-21kthreadcore11:55:172
2577080290,6sleep22577088-21ssh11:15:092
2553140290,6sleep02553268-21gltestperf11:00:160
2493293290,2sleep20-21swapper/210:20:232
2405946291,5sleep02405948-21ssh09:28:030
2397875290,1sleep22374660-21diskmemload09:22:572
2329939290,6sleep22330186-21awk08:30:252
2299253291,4sleep02299286-21apt-get08:10:100
2267793291,4sleep02267826-21apt-get07:45:100
2236315290,1sleep02237015-21irqrtprio07:20:140
2231015290,6sleep02232577-21cut07:15:180
19089890,6rtkit-daemon473-21gmain09:12:571
19089890,6rtkit-daemon1928-21sendmail-mta07:55:460
19089890,5rtkit-daemon674-21lldpd11:02:431
19089890,5rtkit-daemon667-21snmpd12:02:200
19089890,5rtkit-daemon667-21snmpd09:04:460
19089890,5rtkit-daemon667-21snmpd09:04:450
19089890,4rtkit-daemon2639465-21ssh11:50:300
19089890,4rtkit-daemon2544920-21ssh10:55:121
19089890,3rtkit-daemon2227388-21munin-node11:32:142
19089890,2rtkit-daemon1679647-21kworker/0:0+kacpid07:09:450
19089890,1rtkit-daemon0-21swapper/009:45:370
19089890,1rtkit-daemon0-21swapper/008:55:210
2698870281,3sleep12698907-21apt-get12:30:131
2641732281,4sleep02641771-21apt-get11:55:110
2641345281,5sleep12641353-21ssh11:54:511
2585407281,4sleep02585615-21hddtemp_smartct11:20:160
2585407281,4sleep02585615-21hddtemp_smartct11:20:150
2577097281,4sleep00-21swapper/011:15:110
2574862280,2sleep02574870-21ssh11:10:340
2536856281,4sleep02537010-21df10:50:140
2528780281,4sleep02528942-21diskstats10:45:130
2527940280,5sleep22527948-21ssh10:44:362
2509177280,6sleep02509220-21awk10:30:240
2479144281,5sleep22479150-21ssh10:14:072
2432523281,4sleep12406659-21kworker/u16:0+rpciod09:45:161
2416137280,5sleep02417170-21cat09:35:180
2367470281,4sleep12367531-21open_files09:00:211
2367470281,4sleep12367531-21open_files09:00:201
2338190281,4sleep22339321-21cut08:40:172
2311845280,4sleep22311988-21df08:20:152
2273068281,4sleep22273196-21switchtime07:45:202
2251471280,5sleep12253436-21sed07:30:171
2242260281,4sleep02242416-21diskstats07:25:110
2241153281,4sleep22241278-21python307:20:202
2225977281,4sleep12226043-21ntp_states07:10:211
19089880,7rtkit-daemon1907-21rtkit-daemon09:10:160
19089880,5rtkit-daemon1928-21sendmail-mta08:39:500
19089880,5rtkit-daemon1928-21sendmail-mta08:28:050
19089880,4rtkit-daemon438-21wpa_supplicant08:17:560
19089880,4rtkit-daemon319-21systemd-udevd09:23:381
19089880,4rtkit-daemon2485969-21ssh10:16:052
19089880,4rtkit-daemon2428781-21ntp_states09:40:220
19089880,4rtkit-daemon2418774-21cut09:35:181
19089880,4rtkit-daemon2227388-21munin-node12:20:040
19089880,4rtkit-daemon2227388-21munin-node09:56:440
19089880,4rtkit-daemon2226514-21users07:10:240
19089880,1rtkit-daemon0-21swapper/112:12:091
19089880,1rtkit-daemon0-21swapper/008:49:240
19089880,1rtkit-daemon0-21swapper/008:43:500
19089880,1rtkit-daemon0-21swapper/008:23:370
19089880,1rtkit-daemon0-21swapper/007:33:430
2662440271,3sleep12663527-21sshd12:05:201
2623436270,4sleep22623444-21ssh11:40:462
2613398270,4sleep02614527-21memory11:35:180
2606747270,4sleep02606870-21grep11:30:220
2519030270,5sleep22519038-21ssh10:37:422
2496377270,4sleep12496637-21cut10:25:161
2465404270,4sleep12467408-21kthreadcore10:05:191
2423692270,4sleep22423819-21grep09:40:162
2399097270,4sleep12399250-21df_inode09:25:121
2398144270,4sleep02398152-21ssh09:23:400
2390393271,3sleep02390401-21ssh09:19:180
2352316270,5sleep12354488-21kthreadcore08:50:181
2337199270,3sleep12337232-21apt-get08:40:121
2305250270,5sleep12305462-21latency_hist08:15:001
2297075270,5sleep22298002-21sed08:05:162
2220539270,2sleep30-21swapper/307:09:553
2215567271,3sleep20-21swapper/207:05:172
2214586271,2sleep10-21swapper/107:05:121
19089870,4rtkit-daemon81364-21packagekitd08:30:090
19089870,4rtkit-daemon2227388-21munin-node12:05:060
19089870,4rtkit-daemon2227388-21munin-node11:05:331
19089870,3rtkit-daemon671-21ntpd07:50:020
19089870,3rtkit-daemon667-21snmpd08:08:520
19089870,3rtkit-daemon2703606-21kthreadcore12:30:190
19089870,3rtkit-daemon2582924-21ssh11:15:271
19089870,3rtkit-daemon2575637-21ssh11:12:201
19089870,3rtkit-daemon2478502-21ssh10:12:351
19089870,3rtkit-daemon2374660-21diskmemload09:16:272
19089870,3rtkit-daemon2227388-21munin-node10:48:052
19089870,3rtkit-daemon2227388-21munin-node09:59:142
2713284260,1sleep20-21swapper/212:36:442
2696539260,1sleep10-21swapper/112:25:241
2689194260,1sleep20-21swapper/212:21:572
2666493260,3sleep22666501-21ssh12:10:102
2656613260,4sleep22374660-21diskmemload12:02:012
2656105260,1sleep10-21swapper/112:00:481
2617001260,1sleep10-21swapper/111:39:321
2615908260,1sleep20-21swapper/211:37:002
2595593260,3sleep12597642-21cut11:25:171
2590728261,2sleep10-21swapper/111:20:251
2590728261,2sleep10-21swapper/111:20:241
2585971261,3sleep22586726-21kthreadcore11:20:182
2585971261,3sleep22586726-21kthreadcore11:20:172
2575224260,1sleep20-21swapper/211:11:282
2567038260,1sleep00-21swapper/011:06:050
2535865260,1sleep10-21swapper/110:48:311
2511684260,3sleep12511702-21apt-get10:35:001
2504198261,2sleep20-21swapper/210:30:162
2496638261,2sleep20-21swapper/210:25:172
2494210260,1sleep10-21swapper/110:21:341
2486223260,3sleep02486231-21ssh10:16:490
2450533260,1sleep10-21swapper/109:55:171
2444893261,2sleep20-21swapper/209:50:212
2414654260,1sleep20-21swapper/209:34:352
2413785260,3sleep02413793-21ssh09:32:360
2374563260,1sleep20-21swapper/209:09:032
2286270260,1sleep10-21swapper/107:58:221
2248043260,1sleep10-21swapper/107:27:251
2228395260,1sleep20-21swapper/207:15:032
19089860,5rtkit-daemon1907-21rtkit-daemon09:36:122
19089860,3rtkit-daemon667-21snmpd07:43:140
19089860,3rtkit-daemon2680446-21ssh12:15:331
19089860,3rtkit-daemon1928-21sendmail-mta09:15:101
2712833250,3sleep1667-21snmpd12:35:431
2705983250,3sleep20-21swapper/212:33:272
2697772250,3sleep20-21swapper/212:28:182
2688647251,2sleep12688653-21ssh12:20:401
2642962250,2sleep12644186-21kthreadcore11:55:161
2633577250,3sleep20-21swapper/211:50:032
2631966250,2sleep10-21swapper/111:46:451
2601734250,3sleep12602136-21cut11:30:151
2594423250,3sleep02594790-21kthreadcore11:25:160
2567596250,3sleep20-21swapper/211:07:242
2544826250,3sleep20-21swapper/210:55:052
2542821250,2sleep22542829-21ssh10:50:572
2527670250,3sleep12527678-21ssh10:43:511
2506208250,2sleep1321rcuc/110:30:191
2486605250,3sleep10-21swapper/110:17:461
2469963251,2sleep20-21swapper/210:06:322
2463044250,3sleep10-21swapper/110:04:121
2438683250,3sleep22438691-21ssh09:48:422
2375813250,2sleep20-21swapper/209:10:142
2374392250,1sleep10-21swapper/109:06:491
2349996250,3sleep20-21swapper/208:50:152
2343498250,2sleep10-21swapper/108:45:141
2331712250,3sleep22332653-21cut08:35:162
2331274251,2sleep10-21swapper/108:35:151
2327277250,3sleep12329444-21cut08:30:191
2321257250,3sleep12323248-21expr08:25:181
2312250250,3sleep12313044-21sed08:20:181
2299441250,3sleep20-21swapper/208:10:122
2298648250,1sleep10-21swapper/108:05:311
2274095251,1sleep20-21swapper/207:50:142
2257304251,2sleep20-21swapper/207:35:182
2248244250,1sleep20-21swapper/207:29:592
2227661251,2sleep12228393-21kthreadcore07:15:031
2560129240,1sleep20-21swapper/211:03:422
2545305240,2sleep02545816-21irqcore10:55:160
2362405240,2sleep22362369-21/usr/sbin/munin09:00:122
2362405240,2sleep22362369-21/usr/sbin/munin09:00:122
2323722240,2sleep20-21swapper/208:25:232
2311467240,3sleep10-21swapper/108:19:131
2305508240,1sleep20-21swapper/208:15:132
2286210240,2sleep2667-21snmpd07:57:342
2269247241,1sleep10-21swapper/107:45:151
2267512240,3sleep10-21swapper/107:44:401
2262321240,2sleep22263233-21kthreadcore07:40:202
2237016240,2sleep10-21swapper/107:20:151
2361958230,1sleep20-21swapper/208:58:202
2359162230,1sleep10-21swapper/108:55:191
2344954230,1sleep20-21swapper/208:45:192
2286717230,1sleep10-21swapper/108:00:141
2279773230,1sleep10-21swapper/107:50:281
2256143230,1sleep10-21swapper/107:35:171
2286668220,1sleep20-21swapper/208:00:032
2248294220,1sleep20-21swapper/207:30:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional