You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 14:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Sun Nov 16, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26535189910580,1054cyclictest28002992sleep321:04:573
290398723000,295sleep02743964-21kworker/u16:3+efi_rts_wq22:10:140
2653518992580,246cyclictest2951939-21turbostat22:44:543
2653518991916,1cyclictest0-21swapper/323:29:513
2653518991916,1cyclictest0-21swapper/323:29:503
265351899182,12cyclictest2749965-21turbostat20:25:003
265351899171,12cyclictest2756213-21turbostat20:29:593
265351899166,6cyclictest30177662sleep323:20:163
265351899154,8cyclictest2901319-21taskset22:05:233
2653518991513,1cyclictest0-21swapper/323:37:113
2653518991513,1cyclictest0-21swapper/320:30:593
2653518991512,1cyclictest0-21swapper/320:39:453
2653518991512,1cyclictest0-21swapper/320:04:533
265351899151,11cyclictest3090613-21taskset00:05:153
265351899151,11cyclictest3017006-21turbostat23:20:013
265351899151,11cyclictest2992718-21taskset23:05:013
265351899151,10cyclictest3066009-21taskset23:50:143
265351899151,10cyclictest3004019-21taskset23:10:193
265351899151,10cyclictest2863556-21taskset21:45:153
265351899151,10cyclictest2781890-21chrt20:50:133
265351899151,10cyclictest2668033-21turbostat19:20:013
265351899150,11cyclictest3104876-21taskset00:12:253
26471982150,3sleep30-21swapper/319:05:013
190898150,11rtkit-daemon667-21snmpd22:47:350
265351899145,6cyclictest29411402sleep322:30:213
265351899144,8cyclictest3081790-21turbostat00:04:563
265351899144,8cyclictest2705628-21taskset19:46:513
265351899144,7cyclictest2893798-21taskset22:00:543
265351899144,7cyclictest2781443-21taskset20:49:543
2653518991413,0cyclictest0-21swapper/319:33:323
2653518991412,1cyclictest0-21swapper/321:24:363
2653518991411,1cyclictest0-21swapper/320:10:203
2653518991411,1cyclictest0-21swapper/320:08:533
2653518991411,1cyclictest0-21swapper/300:20:133
265351899141,10cyclictest3144159-21taskset00:35:253
265351899141,10cyclictest3134293-21taskset00:30:213
265351899141,10cyclictest2991021-21taskset23:02:133
265351899141,10cyclictest2935502-21taskset22:29:153
265351899141,10cyclictest2919741-21turbostat22:19:593
265351899141,10cyclictest2820969-21chrt21:15:593
265351899141,10cyclictest2806267-21chrt21:06:003
265351899141,10cyclictest2711788-21taskset19:50:253
265351899141,10cyclictest2698992-21chrt19:40:353
29444462135,2sleep2171rcu_preempt22:35:172
28872292131,8sleep12887407-21apt-config22:00:011
27931702131,9sleep22793243-21/usr/sbin/munin20:55:262
265351899135,6cyclictest28128482sleep321:10:433
265351899134,6cyclictest29252582sleep322:20:233
265351899134,6cyclictest2793737-21taskset20:56:343
265351899131,9cyclictest3057920-21taskset23:45:133
265351899131,9cyclictest3049810-21taskset23:40:073
265351899131,9cyclictest2977831-21taskset22:55:193
265351899131,9cyclictest2950141-21taskset22:35:533
265351899131,9cyclictest2910411-21taskset22:12:033
265351899131,9cyclictest2871403-21chrt21:50:053
265351899131,9cyclictest2861203-21taskset21:40:433
265351899131,9cyclictest2853334-21taskset21:36:053
2653518991312,0cyclictest0-21swapper/322:53:483
2653518991311,0cyclictest0-21swapper/320:43:393
2653518991311,0cyclictest0-21swapper/320:18:203
2653518991310,1cyclictest0-21swapper/323:30:153
265351899130,9cyclictest3128257-21taskset00:25:363
265351899130,10cyclictest3080763-21taskset23:57:383
265351899130,10cyclictest2960469-21taskset22:45:143
265351899130,10cyclictest2837688-21taskset21:27:253
265351899130,10cyclictest2653644-21taskset19:10:023
190898130,8rtkit-daemon2648386-21cut19:05:182
27567572121,8sleep02756915-21/usr/sbin/munin20:30:200
265351899129,1cyclictest0-21swapper/319:20:393
265351899121,9cyclictest2692835-21taskset19:36:493
265351899120,9cyclictest2879490-21taskset21:55:103
265351899120,9cyclictest2839095-21chrt21:30:083
265351899120,9cyclictest2712399-21chrt19:55:013
265351899120,9cyclictest2674542-21taskset19:25:123
265351899120,10cyclictest3112385-21chrt00:16:063
190898120,1rtkit-daemon0-21swapper/122:03:251
31057412110,1sleep23105749-21ssh00:14:252
29918532110,2sleep00-21swapper/023:04:080
29576722114,3sleep12957921-21grep22:40:231
29465522110,8sleep12948678-21tr22:35:201
190898110,7rtkit-daemon3136728-21ssh00:31:250
190898110,6rtkit-daemon2786884-21ntp_states20:50:222
190898110,4rtkit-daemon3064438-21ssh23:47:140
190898110,4rtkit-daemon3049473-21ssh23:39:580
190898110,4rtkit-daemon2838767-21ssh21:29:570
190898110,3rtkit-daemon667-21snmpd22:53:570
190898110,3rtkit-daemon1928-21sendmail-mta00:05:361
190898110,2rtkit-daemon2659852-21munin-node22:12:022
29687552100,7sleep12968888-21irqcore22:50:181
28706812101,6sleep22870689-21ssh21:48:552
28068832100,2sleep10-21swapper/121:10:081
27564602101,5sleep12756493-21apt-get20:30:171
27186942102,4sleep02718721-21apt-get20:00:110
190898100,6rtkit-daemon3032277-21ssh23:28:080
190898100,6rtkit-daemon3032277-21ssh23:28:070
190898100,6rtkit-daemon2920614-21ps22:20:180
190898100,5rtkit-daemon3099813-21irqstats00:10:201
190898100,5rtkit-daemon2939656-21cut22:30:180
190898100,5rtkit-daemon2706341-21df19:50:132
190898100,3rtkit-daemon2659565-21kworker/u16:3+events_power_efficient19:15:412
190898100,3rtkit-daemon1928-21sendmail-mta19:30:462
3140657291,6sleep00-21swapper/000:35:190
3120749290,4sleep00-21swapper/000:21:570
3114478291,5sleep23114642-21diskstats00:20:142
3082234290,7sleep13082133-21/usr/sbin/munin00:00:141
3074064291,4sleep13074097-21apt-get23:55:131
3040892290,1sleep20-21swapper/223:34:302
2984880291,5sleep22985465-21ps23:00:172
2917142291,5sleep22917204-21ntp_states22:15:202
2814606290,3sleep22814614-21ssh21:14:532
2768478291,5sleep22768601-21users20:35:282
2744249290,1sleep22744426-21if_eno120:20:172
2743006290,6sleep02743227-21/usr/sbin/munin20:15:260
2737691291,4sleep12737724-21apt-get20:15:151
2736540291,5sleep22736596-21ntp_states20:10:242
2712418291,4sleep12712450-21apt-get19:55:161
2653697291,5sleep12653850-21df_inode19:10:161
2653661291,4sleep02653694-21apt-get19:10:150
19089890,6rtkit-daemon2806590-21diskmemload00:28:470
19089890,5rtkit-daemon2858905-21grep21:40:210
19089890,5rtkit-daemon2739998-21kthreadcore20:15:192
19089890,4rtkit-daemon3113760-21ssh00:19:211
19089890,4rtkit-daemon3050275-21basename23:40:160
19089890,4rtkit-daemon2918648-21ssh22:17:281
19089890,4rtkit-daemon2847440-21df_inode21:35:120
19089890,4rtkit-daemon2794465-21df21:00:142
19089890,1rtkit-daemon0-21swapper/021:11:450
3128307281,4sleep13128315-21ssh00:25:421
3091040280,5sleep03091950-21cut00:05:180
2945415280,6sleep02946551-21kthreadcore22:35:190
2933921281,4sleep02933929-21ssh22:25:370
2895530281,5sleep02895580-21munin-run22:05:010
2786757281,5sleep12786846-21awk20:50:211
2775479280,4sleep02775512-21apt-get20:45:150
2705604281,3sleep22705625-21apt-get19:46:512
2699336280,5sleep02699756-21latency_hist19:45:020
19089880,5rtkit-daemon1928-21sendmail-mta21:05:202
19089880,4rtkit-daemon81364-21packagekitd23:55:172
19089880,4rtkit-daemon667-21snmpd23:01:241
19089880,4rtkit-daemon667-21snmpd21:58:130
19089880,4rtkit-daemon667-21snmpd21:51:330
19089880,4rtkit-daemon438-21wpa_supplicant19:38:032
19089880,4rtkit-daemon436-21systemd-logind20:47:382
19089880,4rtkit-daemon3072457-21ssh23:51:580
19089880,4rtkit-daemon3024425-21ssh23:23:352
19089880,4rtkit-daemon2999666-21ssh23:07:592
19089880,4rtkit-daemon2978523-21grep22:55:200
19089880,4rtkit-daemon2806590-21diskmemload00:22:451
19089880,4rtkit-daemon2696174-21grep19:40:212
19089880,4rtkit-daemon2659852-21munin-node22:43:190
19089880,4rtkit-daemon2659852-21munin-node22:07:142
19089880,4rtkit-daemon2659852-21munin-node19:57:072
19089880,1rtkit-daemon0-21swapper/220:42:322
19089880,1rtkit-daemon0-21swapper/220:33:322
3138578271,3sleep23138617-21apt-get00:35:132
3106351270,4sleep03106512-21df_inode00:15:160
3097586270,5sleep23097594-21ssh00:09:132
3081759271,3sleep03081816-40updatedb.mlocat00:00:000
3071063270,4sleep13071211-21awk23:50:211
3064890270,5sleep13064894-21ssh23:48:191
3048875270,1sleep20-21swapper/223:38:362
3041725270,3sleep13041764-21apt-get23:35:141
3026509270,4sleep23027694-21cut23:25:212
3026509270,4sleep23027694-21cut23:25:212
3025412270,3sleep13025451-21apt-get23:25:161
3025412270,3sleep13025451-21apt-get23:25:151
2961309270,4sleep22962452-21grep22:45:182
2933503271,4sleep10-21swapper/122:25:261
2917325270,4sleep02917436-21find22:15:220
2887411270,3sleep22887254-21apt-get22:00:012
2886483270,4sleep22886491-21ssh21:58:222
2869739270,1sleep10-21swapper/121:46:511
2855743271,3sleep20-21swapper/221:40:192
2845198270,1sleep10-21swapper/121:31:001
2835640270,1sleep10-21swapper/121:25:181
2822563270,5sleep12822571-21ssh21:19:381
2780551271,3sleep12780614-21ntp_states20:45:241
2744006270,4sleep02744153-21df20:20:150
2724686271,3sleep02724702-21apt-get20:05:000
2717732270,4sleep02717963-21grep19:55:270
2689558270,4sleep12691737-21kthreadcore19:35:201
2647413270,3sleep00-21swapper/019:05:150
19089870,4rtkit-daemon671-21ntpd23:16:012
19089870,4rtkit-daemon667-21snmpd21:15:090
19089870,4rtkit-daemon421-21avahi-daemon23:10:342
19089870,4rtkit-daemon2615233-21munin-node19:13:512
19089870,3rtkit-daemon674-21lldpd21:20:260
19089870,3rtkit-daemon671-21ntpd22:59:561
19089870,3rtkit-daemon671-21ntpd21:49:150
19089870,3rtkit-daemon671-21ntpd19:20:182
19089870,3rtkit-daemon671-21ntpd00:25:522
19089870,3rtkit-daemon667-21snmpd20:25:072
19089870,3rtkit-daemon436-21systemd-logind23:54:382
19089870,3rtkit-daemon3039368-21ssh23:30:510
19089870,3rtkit-daemon2911609-21ssh22:14:511
19089870,3rtkit-daemon2845994-21ssh21:32:550
19089870,3rtkit-daemon2723651-21memory20:00:182
19089870,3rtkit-daemon2659852-21munin-node20:08:192
19089870,1rtkit-daemon0-21swapper/219:25:422
3137026260,3sleep13137034-21ssh00:32:121
3132240260,4sleep23134292-21kthreadcore00:30:212
3089710261,2sleep20-21swapper/200:04:362
3057582260,3sleep23057627-21systemd-run23:45:002
3057382260,4sleep13057388-21ssh23:44:361
3034404260,3sleep13035360-21grep23:30:161
3015026260,1sleep10-21swapper/123:15:311
3009187260,3sleep03009195-21ssh23:15:100
3007426260,3sleep03007434-21ssh23:12:160
3000888260,3sleep13000923-21apt-get23:10:131
2997986260,3sleep02998048-21ntp_states23:05:200
2968007260,3sleep12968015-21ssh22:49:571
2959024260,3sleep22959034-21ssh22:42:492
2943332260,1sleep10-21swapper/122:33:381
2928109261,2sleep20-21swapper/222:25:142
2927223260,4sleep22927231-21ssh22:23:432
2920302260,3sleep12920475-21expr22:20:171
2902997260,1sleep00-21swapper/022:08:330
2901694260,3sleep12901704-21ssh22:05:291
2862027260,3sleep12862035-21ssh21:42:381
2854316260,3sleep22854324-21ssh21:38:262
2854110260,1sleep10-21swapper/121:37:561
2825729260,4sleep12827863-21cut21:20:171
2794320260,3sleep02794429-21grep21:00:140
2787979260,1sleep00-21swapper/020:55:080
2767870260,4sleep12768042-21strings20:35:231
2756143263,2sleep1433-21hwrng20:29:221
2707653260,4sleep02709624-21kthreadcore19:50:180
2699856260,3sleep12699889-21apt-get19:45:151
19089860,3rtkit-daemon473-21gmain22:58:512
3112456250,3sleep20-21swapper/200:16:222
3104521250,3sleep03104529-21ssh00:11:350
3082173251,2sleep00-21swapper/000:00:130
3062889251,2sleep23063074-21needreboot23:45:202
3025033250,3sleep03025041-21ssh23:24:580
2998589250,1sleep10-21swapper/123:05:291
2975427250,3sleep20-21swapper/222:53:322
2941455250,3sleep22941578-21ssh22:30:232
2879028250,3sleep12879035-21diskmemload21:54:351
2878778250,1sleep20-21swapper/221:54:042
2845626250,3sleep22845632-21ssh21:32:012
2837970250,3sleep20-21swapper/221:28:012
2829530250,1sleep20-21swapper/221:22:032
2815015250,2sleep20-21swapper/221:15:142
2802017250,3sleep02803520-21kthreadcore21:05:190
2769691251,2sleep02770626-21irqrtprio20:40:180
2763161250,2sleep00-21swapper/020:35:180
2756018250,1sleep00-21swapper/020:27:450
2737107250,1sleep00-21swapper/020:10:500
2712112250,1sleep10-21swapper/119:54:271
2705696250,1sleep00-21swapper/019:47:220
2693559250,2sleep12693726-21gltestperf19:40:181
2686517250,1sleep10-21swapper/119:31:001
2683903250,2sleep02685771-21latency19:30:190
2669757251,2sleep10-21swapper/119:20:181
2668290250,2sleep00-21swapper/019:20:130
2659685250,3sleep12660034-21hddtemp_smartct19:15:021
2648480250,2sleep10-21swapper/119:05:201
3144500240,3sleep10-21swapper/100:35:441
3019013240,2sleep10-21swapper/123:20:181
2788971240,2sleep10-21swapper/120:55:211
2787508240,3sleep00-21swapper/020:52:130
2775029240,3sleep10-21swapper/120:42:311
2746585240,2sleep12748787-21kthreadcore20:20:201
2736816240,2sleep10-21swapper/120:10:271
2720155240,2sleep10-21swapper/120:00:161
2688413240,2sleep00-21swapper/019:35:190
2679752241,1sleep10-21swapper/119:25:271
2662856240,2sleep02663691-21kthreadcore19:15:210
2794433230,1sleep10-21swapper/121:00:151
2725235230,1sleep10-21swapper/120:05:131
2676005230,1sleep00-21swapper/019:25:220
2800547220,1sleep10-21swapper/121:05:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional