You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 16:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Tue Nov 25, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26344079912531,1248cyclictest26742032sleep307:39:573
2634407994151,411cyclictest2768888-21turbostat08:59:563
2634407992670,263cyclictest2941764-21turbostat10:49:543
2634407992118,1cyclictest0-21swapper/309:40:013
2634407991815,1cyclictest0-21swapper/312:20:193
2634407991815,1cyclictest0-21swapper/310:15:243
190898180,4rtkit-daemon3094255-21ssh12:16:410
263440799176,8cyclictest2884998-21turbostat10:10:003
2634407991715,1cyclictest0-21swapper/308:40:003
2634407991714,1cyclictest0-21swapper/312:05:003
2634407991613,1cyclictest0-21swapper/311:45:173
2634407991613,1cyclictest0-21swapper/311:30:163
2634407991613,1cyclictest0-21swapper/310:34:593
263440799161,11cyclictest3039157-21turbostat11:45:013
263440799156,6cyclictest29569892sleep310:52:483
2634407991513,0cyclictest0-21swapper/311:56:393
2634407991512,1cyclictest0-21swapper/311:40:003
2634407991512,1cyclictest0-21swapper/309:13:253
263440799151,11cyclictest2994720-21taskset11:15:193
263440799151,11cyclictest2994720-21taskset11:15:183
263440799151,11cyclictest2982695-21taskset11:10:013
263440799151,11cyclictest2933621-21turbostat10:39:593
263440799151,11cyclictest2909543-21chrt10:25:023
263440799145,6cyclictest26494932sleep307:20:153
263440799144,7cyclictest2965030-21taskset10:57:473
263440799144,7cyclictest2743370-21taskset08:33:373
2634407991413,0cyclictest0-21swapper/312:36:493
2634407991413,0cyclictest0-21swapper/310:43:003
2634407991413,0cyclictest0-21swapper/308:23:553
2634407991413,0cyclictest0-21swapper/307:41:393
2634407991413,0cyclictest0-21swapper/307:14:113
2634407991412,1cyclictest0-21swapper/308:43:093
2634407991412,1cyclictest0-21swapper/308:05:013
2634407991412,0cyclictest0-21swapper/309:40:123
2634407991412,0cyclictest0-21swapper/309:09:213
263440799141,11cyclictest2860623-21chrt09:55:003
2634407991411,1cyclictest0-21swapper/309:30:113
2634407991411,1cyclictest0-21swapper/308:05:173
2634407991411,1cyclictest0-21swapper/307:50:143
2634407991411,1cyclictest0-21swapper/307:25:153
263440799141,10cyclictest3107654-21chrt12:25:193
263440799141,10cyclictest2860873-21taskset09:55:043
263440799141,10cyclictest2796397-21taskset09:15:043
263440799141,10cyclictest2775422-21taskset09:00:143
263440799141,10cyclictest2768823-21taskset08:54:293
263440799141,10cyclictest2732036-21chrt08:25:173
263440799140,11cyclictest3004933-21taskset11:20:393
26342142140,3sleep30-21swapper/307:09:323
263440799139,1cyclictest0-21swapper/309:25:133
263440799135,6cyclictest29036572sleep310:20:173
263440799131,9cyclictest3089739-21taskset12:15:173
263440799131,9cyclictest30071612sleep311:25:143
263440799131,9cyclictest2972560-21taskset11:00:473
263440799131,9cyclictest2804672-21chrt09:20:143
263440799131,9cyclictest2698917-21chrt07:55:353
2634407991312,0cyclictest0-21swapper/312:30:443
2634407991312,0cyclictest0-21swapper/312:12:203
2634407991311,1cyclictest0-21swapper/307:46:103
2634407991311,1cyclictest0-21swapper/307:30:013
263440799131,10cyclictest3079383-21taskset12:09:373
263440799131,10cyclictest2981113-21taskset11:06:543
263440799131,10cyclictest2762616-21turbostat08:50:003
263440799131,10cyclictest2648869-21chrt07:18:493
2634407991310,1cyclictest0-21swapper/311:50:243
263440799130,10cyclictest2885232-21taskset10:10:023
263440799130,10cyclictest2850107-21taskset09:45:233
190898130,9rtkit-daemon2973341-21ssh11:02:371
31020092120,1sleep10-21swapper/112:20:541
263440799121,8cyclictest2712385-21chrt08:10:013
263440799121,8cyclictest0-21swapper/310:00:143
26297242121,3sleep20-21swapper/207:05:182
26932422110,3sleep02693275-21apt-get07:55:130
263440799110,9cyclictest2718643-21chrt08:15:073
190898110,8rtkit-daemon81364-21packagekitd09:31:572
190898110,7rtkit-daemon2802211-21ssh09:15:271
190898110,6rtkit-daemon3085160-21ntp_states12:10:220
190898110,4rtkit-daemon2998538-21ssh11:19:411
190898110,4rtkit-daemon2998538-21ssh11:19:401
190898110,4rtkit-daemon2640800-21munin-node11:53:270
190898110,4rtkit-daemon1928-21sendmail-mta09:41:262
190898110,3rtkit-daemon2640800-21munin-node11:41:170
31199542101,6sleep13120012-21latency_hist12:34:591
29904932101,6sleep00-21swapper/011:15:010
26672852101,7sleep02667472-21/usr/sbin/munin07:30:250
190898100,6rtkit-daemon2794207-21ssh09:10:322
190898100,6rtkit-daemon2662438-21irqcore07:30:162
190898100,6rtkit-daemon2636053-21kthreadcore07:10:172
190898100,5rtkit-daemon3127885-21ssh12:39:340
190898100,3rtkit-daemon1928-21sendmail-mta11:34:370
2957068291,6sleep12957076-21ssh10:52:571
2941738290,6sleep12941774-21systemd-run10:45:011
2940632290,6sleep22940639-21sh10:42:332
2750094290,6sleep22750156-21/usr/sbin/munin08:40:152
2744130291,6sleep10-21swapper/108:35:171
2694695291,2sleep10-21swapper/107:55:181
19089890,5rtkit-daemon3057788-21kthreadcore11:55:180
19089890,5rtkit-daemon0-21swapper/010:06:520
19089890,4rtkit-daemon3101959-21ssh12:20:460
19089890,4rtkit-daemon3076849-21ssh12:05:170
19089890,4rtkit-daemon2999251-21diskstats11:20:141
19089890,4rtkit-daemon2943371-21sed10:45:152
19089890,1rtkit-daemon0-21swapper/207:55:222
3112188281,3sleep23112227-21apt-get12:30:122
3093136281,4sleep23093197-21ntp_states12:15:212
3035882281,4sleep23036451-21memory11:40:202
3031561281,5sleep13031618-21sort11:40:151
3030734281,4sleep13030742-21ssh11:39:171
2996839281,5sleep20-21swapper/211:15:472
2996839281,5sleep20-21swapper/211:15:462
2959107281,5sleep12959954-21cut10:55:181
2933593281,4sleep22933660-21munin-run10:40:002
2899811280,2sleep20-21swapper/210:16:482
2893430281,3sleep02893467-21apt-get10:15:140
2885291281,5sleep02885438-21cpuspeed_turbos10:10:120
2867286280,5sleep02867294-21ssh09:56:420
2844769281,3sleep12844803-21apt-get09:45:131
2836643281,3sleep12836678-21apt-get09:40:121
2828982280,1sleep20-21swapper/209:35:172
2820531280,5sleep02820690-21diskstats09:30:130
2812901281,4sleep02813169-21sed09:25:170
2801664281,3sleep22801706-21ntp_states09:15:202
2786884282,4sleep12786958-21sendmail_mailqu09:05:221
2781693281,3sleep02781715-21apt-get09:05:130
2742554280,4sleep12742631-21ntp_states08:30:211
2717577280,4sleep22717702-21python308:10:212
2712433281,4sleep12712535-21grep08:10:121
2699946280,5sleep12700537-21irqrtprio08:00:191
2687403281,4sleep22687575-21sed07:50:152
2680770281,4sleep02680916-21df07:45:110
2655576281,4sleep12655673-21tr07:25:141
2641224281,4sleep02642825-21ntp_states07:15:020
19089880,5rtkit-daemon1928-21sendmail-mta08:48:372
19089880,5rtkit-daemon0-21swapper/209:54:082
19089880,4rtkit-daemon81364-21packagekitd10:48:080
19089880,4rtkit-daemon671-21ntpd07:40:132
19089880,4rtkit-daemon667-21snmpd10:53:282
19089880,4rtkit-daemon667-21snmpd10:50:560
19089880,4rtkit-daemon667-21snmpd09:12:571
19089880,4rtkit-daemon667-21snmpd08:22:302
19089880,4rtkit-daemon438-21wpa_supplicant11:14:451
19089880,4rtkit-daemon438-21wpa_supplicant10:26:220
19089880,4rtkit-daemon438-21wpa_supplicant09:26:482
19089880,4rtkit-daemon438-21wpa_supplicant07:49:032
19089880,4rtkit-daemon407-21rpcbind12:03:200
19089880,4rtkit-daemon3041512-21tr11:45:170
19089880,4rtkit-daemon3007119-21ssh11:25:080
19089880,4rtkit-daemon2909111-21ssh10:24:370
19089880,4rtkit-daemon2875776-21ssh10:02:330
19089880,4rtkit-daemon2844768-21/usr/sbin/munin09:45:152
19089880,4rtkit-daemon2705600-21/usr/sbin/munin08:05:012
19089880,3rtkit-daemon3104381-21fschecks_count12:25:160
19089880,1rtkit-daemon0-21swapper/209:57:082
19089880,1rtkit-daemon0-21swapper/111:06:241
3114266270,5sleep03116280-21kthreadcore12:30:180
3088337270,4sleep13088798-21sed12:15:161
3077851270,5sleep23077859-21ssh12:06:002
3062627270,4sleep13062635-21ssh11:58:141
3049040270,4sleep23050559-21kthreadcore11:50:162
3014591270,5sleep13014598-21sh11:29:161
3004342271,3sleep23004467-21python311:20:232
2958680271,3sleep02959106-21sed10:55:170
2940317270,5sleep02940325-21ssh10:41:460
2916167270,1sleep22916175-21ssh10:27:142
2908445270,1sleep20-21swapper/210:23:092
2891276270,4sleep22891280-21ssh10:10:502
2852902270,4sleep12852869-21/usr/sbin/munin09:50:151
2843736271,1sleep0171rcu_preempt09:43:230
2788220270,3sleep02787740-21apt-get09:10:020
2777639270,5sleep02778472-21kthreadcore09:00:200
2768857270,1sleep00-21swapper/008:54:550
2764302270,4sleep22765974-21cut08:50:162
2705618270,4sleep244-21ksoftirqd/208:05:002
2628271270,3sleep10-21swapper/107:05:131
19089870,4rtkit-daemon667-21snmpd09:20:541
19089870,4rtkit-daemon2640800-21munin-node11:36:060
19089870,4rtkit-daemon2640800-21munin-node07:21:382
19089870,3rtkit-daemon667-21snmpd09:05:372
19089870,3rtkit-daemon667-21snmpd09:00:492
19089870,3rtkit-daemon667-21snmpd08:37:192
19089870,3rtkit-daemon667-21snmpd08:17:052
19089870,3rtkit-daemon667-21snmpd07:28:452
19089870,3rtkit-daemon299-21systemd-journal08:55:292
19089870,3rtkit-daemon2643159-21diskstats07:15:162
19089870,3rtkit-daemon2640800-21munin-node10:30:251
19089870,3rtkit-daemon2640800-21munin-node10:28:341
19089870,3rtkit-daemon2640800-21munin-node08:25:392
19089870,3rtkit-daemon2640800-21munin-node07:35:432
3120302261,2sleep20-21swapper/212:35:142
3111779260,1sleep20-21swapper/212:29:442
3104806260,3sleep13055395-21kworker/1:2+events12:25:181
3085802260,3sleep13085810-21ssh12:10:521
3080329260,4sleep23080885-21irqcore12:10:162
3071777261,3sleep13071793-21acpi12:05:101
3070699260,1sleep20-21swapper/212:03:072
3061573260,1sleep20-21swapper/211:55:462
3055083260,1sleep10-21swapper/111:54:281
3029146260,1sleep20-21swapper/211:35:422
3018015261,3sleep23020056-21ssh11:30:182
2999054260,3sleep00-21swapper/011:20:130
2997337260,1sleep00-21swapper/011:16:570
2997337260,1sleep00-21swapper/011:16:560
2959956260,3sleep22961926-21kthreadcore10:55:192
2933029260,1sleep10-21swapper/110:38:421
2932229260,3sleep02932237-21ssh10:36:530
2925044260,1sleep00-21swapper/010:34:020
2892737260,3sleep12892745-21ssh10:14:101
2873877260,4sleep12874161-21missed_timers10:00:201
2851331260,1sleep00-21swapper/009:47:160
2830741261,3sleep02832738-21tr09:35:190
2820437260,1sleep10-21swapper/109:30:061
2796374260,1sleep00-21swapper/009:15:020
2762845261,2sleep10-21swapper/108:50:111
2738913260,3sleep22740336-21cut08:30:182
2724932260,2sleep12724964-21apt-get08:20:141
2718686260,3sleep12718842-21diskstats08:15:141
2718661261,2sleep00-21swapper/008:15:120
2708303261,2sleep00-21swapper/008:05:200
2680243260,1sleep00-21swapper/007:42:000
2668131260,3sleep02668194-21ls07:35:130
2650695260,3sleep10-21swapper/107:20:161
19089860,3rtkit-daemon2640800-21munin-node10:31:462
19089860,3rtkit-daemon1928-21sendmail-mta09:55:011
3101467250,2sleep20-21swapper/212:20:272
3064193250,3sleep13064513-21irqcore12:00:171
3039438250,2sleep20-21swapper/211:45:112
3016704250,3sleep13018013-21cut11:30:171
2984724250,3sleep22985914-21kthreadcore11:10:172
2982431250,2sleep02982438-21sh11:09:590
2974648250,3sleep22974650-21ssh11:05:092
2972421250,3sleep22972429-21ssh11:00:312
2949731250,3sleep12949739-21ssh10:49:441
2908728250,3sleep12908736-21ssh10:23:441
2899897250,3sleep10-21swapper/110:16:591
2878634250,2sleep20-21swapper/210:05:172
2877153250,2sleep12877190-21apt-get10:05:121
2875117250,3sleep22875125-21ssh10:01:062
2834928250,1sleep10-21swapper/109:36:441
2820013250,1sleep10-21swapper/109:29:341
2811446251,2sleep22811454-21ssh09:23:232
2810719251,2sleep00-21swapper/009:21:400
2781157250,1sleep10-21swapper/109:01:331
2769119250,2sleep10-21swapper/108:55:131
2760857251,2sleep10-21swapper/108:45:191
2756531250,2sleep00-21swapper/008:45:020
2756003250,1sleep10-21swapper/108:44:061
2736937250,1sleep10-21swapper/108:26:391
2730612250,1sleep00-21swapper/008:20:420
2705358250,1sleep00-21swapper/008:02:100
2692844250,1sleep10-21swapper/107:53:141
2674678250,3sleep12674847-21cut07:40:151
2673863250,1sleep10-21swapper/107:36:561
2662823251,2sleep10-21swapper/107:30:181
2655540250,2sleep02655573-21apt-get07:25:130
2648660250,1sleep10-21swapper/107:16:111
2640474250,3sleep10-21swapper/107:13:001
3126254240,2sleep13126262-21ssh12:35:531
3012620240,2sleep23012822-21ssh11:25:242
2966555240,2sleep00-21swapper/011:00:010
2749760240,3sleep00-21swapper/008:39:480
2743131240,3sleep00-21swapper/008:30:320
2733011240,2sleep02735025-21kthreadcore08:25:180
2712397240,1sleep00-21swapper/008:10:110
2688540240,2sleep02690556-21kthreadcore07:50:170
2683370240,2sleep10-21swapper/107:45:161
2649242240,2sleep00-21swapper/007:20:110
2640468240,3sleep00-21swapper/007:12:550
2633565240,1sleep00-21swapper/007:05:240
3039391230,1sleep10-21swapper/111:45:021
2852868230,1sleep00-21swapper/009:50:140
2769347230,1sleep00-21swapper/008:55:160
2750258230,1sleep00-21swapper/008:40:160
2706304230,1sleep10-21swapper/108:05:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional