You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 15:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Sun Nov 30, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24223019912500,1246cyclictest26567122sleep33
2422301993520,349cyclictest2475224-21turbostat19:54:563
281052222570,253sleep12695550-21kworker/u16:3+efi_rts_wq1
246101122120,1sleep00-21swapper/00
25846432950,0sleep2421rcuc/221:15:152
24714612400,2sleep1321rcuc/119:45:191
242230199221,17cyclictest2600213-21turbostat3
242230199191,15cyclictest2753679-21turbostat23:00:003
242230199181,14cyclictest2640641-21turbostat21:50:003
242230199181,13cyclictest2737939-21taskset22:50:143
242230199181,13cyclictest2525957-21turbostat20:30:003
242230199172,11cyclictest2858697-21turbostat3
242230199171,13cyclictest2737286-21taskset22:49:253
242230199171,12cyclictest2494037-21turbostat3
25220832160,4sleep12524153-21grep1
2422301991613,1cyclictest0-21swapper/33
2422301991613,1cyclictest0-21swapper/321:14:513
242230199161,12cyclictest2874061-21chrt00:12:333
242230199161,12cyclictest2867038-21taskset00:10:003
242230199161,12cyclictest2760362-21taskset3
242230199161,12cyclictest2663487-21taskset22:01:173
242230199161,12cyclictest2463229-21taskset19:40:163
242230199161,12cyclictest0-21swapper/300:35:133
242230199161,12cyclictest0-21swapper/300:20:133
242230199161,11cyclictest2551060-21turbostat3
242230199161,10cyclictest25575662sleep320:55:143
242230199160,12cyclictest25512912sleep320:50:133
190898160,12rtkit-daemon2873880-21ssh00:12:110
242230199152,11cyclictest2777862-21sleep23:15:003
2422301991513,0cyclictest0-21swapper/323:56:173
242230199151,2cyclictest0-21swapper/33
2422301991512,1cyclictest0-21swapper/33
2422301991512,1cyclictest0-21swapper/33
2422301991512,1cyclictest0-21swapper/33
2422301991512,1cyclictest0-21swapper/323:51:223
2422301991512,1cyclictest0-21swapper/323:09:013
2422301991512,1cyclictest0-21swapper/320:07:133
242230199151,12cyclictest0-21swapper/33
242230199151,11cyclictest2907435-21turbostat00:35:013
242230199151,11cyclictest2817390-21taskset23:37:543
242230199151,11cyclictest0-21swapper/33
242230199151,11cyclictest0-21swapper/33
242230199151,11cyclictest0-21swapper/322:25:113
242230199151,11cyclictest0-21swapper/320:15:143
242230199151,11cyclictest0-21swapper/319:25:133
242230199151,11cyclictest0-21swapper/319:20:133
242230199151,11cyclictest0-21swapper/319:10:133
242230199151,10cyclictest0-21swapper/322:35:123
242230199150,12cyclictest2687501-21taskset3
242230199150,12cyclictest2665342-21taskset22:05:013
242230199150,12cyclictest2591999-21taskset21:19:533
242230199150,12cyclictest0-21swapper/321:30:123
242230199141,9cyclictest0-21swapper/323:20:123
242230199141,3cyclictest0-21swapper/33
242230199141,2cyclictest0-21swapper/323:45:143
242230199141,2cyclictest0-21swapper/319:15:153
2422301991411,1cyclictest0-21swapper/33
242230199141,11cyclictest0-21swapper/33
242230199141,11cyclictest0-21swapper/322:20:133
242230199141,11cyclictest0-21swapper/320:30:153
242230199141,11cyclictest0-21swapper/320:10:153
242230199141,10cyclictest2570097-21taskset21:05:113
242230199141,10cyclictest2570097-21taskset21:05:113
242230199141,10cyclictest2475106-21chrt19:48:503
242230199141,10cyclictest0-21swapper/33
242230199141,10cyclictest0-21swapper/33
242230199141,10cyclictest0-21swapper/33
242230199141,10cyclictest0-21swapper/321:40:143
242230199141,10cyclictest0-21swapper/320:35:133
242230199140,12cyclictest0-21swapper/319:55:133
242230199140,11cyclictest2880759-21taskset3
242230199140,11cyclictest0-21swapper/323:25:153
242230199140,11cyclictest0-21swapper/323:25:153
242230199131,9cyclictest0-21swapper/33
242230199131,10cyclictest0-21swapper/319:30:153
190898130,4rtkit-daemon2428775-21munin-node00:27:421
28807002121,8sleep22875403-21/usr/sbin/munin00:15:212
28344332120,1sleep22834520-21sed2
27863412120,2sleep1321rcuc/123:20:131
27697362120,9sleep12769752-21latency_hist23:10:001
24565952121,4sleep22456739-21df19:35:152
24161962121,6sleep22416228-21apt-get19:05:142
190898120,5rtkit-daemon1928-21sendmail-mta0
190898120,3rtkit-daemon1928-21sendmail-mta23:18:590
28505202110,3sleep02850528-21ssh23:59:560
27611772110,1sleep02761185-21ssh23:03:500
190898110,8rtkit-daemon2075-21udisksd21:59:120
190898110,6rtkit-daemon2745955-21apt-get2
190898110,6rtkit-daemon2545072-21apt-get1
190898110,5rtkit-daemon2689487-21ssh0
190898110,5rtkit-daemon0-21swapper/121:40:561
190898110,1rtkit-daemon0-21swapper/121:12:591
190898110,1rtkit-daemon0-21swapper/121:12:591
26960232101,7sleep12696031-21ssh22:22:121
26139232100,7sleep12614054-21/usr/sbin/munin21:30:221
26096982101,6sleep02610807-21tr0
26005432100,7sleep12600671-21tr21:25:131
25908062100,1sleep00-21swapper/021:17:030
25826502100,4sleep22582658-21ssh21:11:362
25826502100,4sleep22582658-21ssh21:11:362
25384812101,6sleep00-21swapper/020:40:000
242230199100,2cyclictest0-21swapper/319:10:013
190898100,6rtkit-daemon2809676-21ssh23:33:510
190898100,5rtkit-daemon2834974-21df_inode23:50:140
190898100,4rtkit-daemon2735927-21ssh22:46:060
190898100,3rtkit-daemon473-21gmain22:37:260
190898100,1rtkit-daemon0-21swapper/000:05:110
2907058290,3sleep10-21swapper/100:34:161
2792127295,2sleep244-21ksoftirqd/22
2738360291,5sleep22739102-21ssh22:50:182
2696708290,1sleep20-21swapper/222:23:492
2668202291,5sleep10-21swapper/11
2647064291,5sleep22647071-21ssh2
2615380290,6sleep22615389-21ssh2
2507066291,4sleep12507099-21apt-get1
2500531291,4sleep02500563-21apt-get20:10:150
19089890,5rtkit-daemon671-21ntpd23:35:230
19089890,5rtkit-daemon2498961-21grep20:05:211
19089890,5rtkit-daemon2431196-21apt-get19:15:151
19089890,5rtkit-daemon2420823-21grep19:05:201
19089890,4rtkit-daemon2800703-21ssh23:26:500
19089890,4rtkit-daemon2800703-21ssh0
19089890,4rtkit-daemon2443175-21runrttasks19:20:471
19089890,3rtkit-daemon2777495-21ssh23:14:160
19089890,1rtkit-daemon0-21swapper/11
2883559281,3sleep12883596-21apt-get00:20:131
2841165280,1sleep10-21swapper/123:51:481
2839875281,4sleep22834775-21/usr/sbin/munin23:50:212
2775886281,5sleep22770073-21/usr/sbin/munin2
2761721281,4sleep22761883-21latency_hist2
2746340281,4sleep12746695-21sed1
2730030281,4sleep22730236-21sh2
2729112280,4sleep02729120-21ssh0
2680047280,3sleep10-21swapper/122:12:181
2647630280,3sleep10-21swapper/121:52:341
2569843280,5sleep00-21swapper/021:05:010
2557120280,3sleep00-21swapper/00
2538587280,1sleep10-21swapper/120:40:011
2442690281,4sleep02442815-21python319:20:230
2437009280,3sleep00-21swapper/00
2431199281,4sleep22431343-21df2
19089880,5rtkit-daemon1928-21sendmail-mta23:07:540
19089880,4rtkit-daemon81364-21packagekitd2
19089880,4rtkit-daemon81364-21packagekitd0
19089880,4rtkit-daemon667-21snmpd22:27:140
19089880,4rtkit-daemon667-21snmpd19:42:071
19089880,4rtkit-daemon438-21wpa_supplicant1
19089880,4rtkit-daemon421-21avahi-daemon23:20:050
19089880,4rtkit-daemon2532674-21df_inode1
19089880,4rtkit-daemon2428775-21munin-node21:52:270
19089880,3rtkit-daemon438-21wpa_supplicant19:25:241
19089880,3rtkit-daemon2856255-21ntp_states00:00:230
19089880,1rtkit-daemon0-21swapper/11
19089880,1rtkit-daemon0-21swapper/00
2889657271,3sleep22889661-21ssh00:21:142
2875400271,3sleep12875439-21apt-get00:15:121
2867737270,5sleep22867880-21grep00:10:162
2862166270,4sleep22864249-21tr2
2860729271,4sleep12862165-21cut00:05:181
2844292270,4sleep22845647-21grep23:55:172
2795324272,3sleep12796390-21kthreadcore23:25:191
2795324272,3sleep12796390-21kthreadcore23:25:191
2784024270,1sleep20-21swapper/223:15:342
2752540270,1sleep00-21swapper/00
2724538270,5sleep12726621-21kthreadcore22:40:201
2712376271,4sleep20-21swapper/222:32:422
2688310271,4sleep22688316-21ssh22:18:002
2658274271,3sleep12659169-21kthreadcore1
2632910271,2sleep20-21swapper/221:45:142
2598271270,1sleep10-21swapper/121:20:321
2576671270,4sleep02576873-21ssh21:10:170
2576671270,4sleep02576873-21ssh0
2575485271,3sleep22575729-21users2
2575485271,3sleep22575729-21users2
2550349270,4sleep02550550-21grep20:45:250
2526182270,3sleep00-21swapper/020:30:110
2512984271,4sleep0667-21snmpd0
2508233271,3sleep22509310-21grep20:15:182
2490327270,4sleep02492399-21cut20:00:220
2488435271,4sleep22489164-21irqrtprio20:00:202
2477359271,3sleep02479407-21tr19:50:200
2443500270,3sleep20-21swapper/219:24:582
2422466271,2sleep00-21swapper/00
19089870,4rtkit-daemon81364-21packagekitd19:10:101
19089870,4rtkit-daemon671-21ntpd21:15:421
19089870,4rtkit-daemon671-21ntpd20:30:531
19089870,4rtkit-daemon667-21snmpd21:45:400
19089870,4rtkit-daemon667-21snmpd21:01:541
19089870,4rtkit-daemon667-21snmpd20:58:311
19089870,4rtkit-daemon473-21gmain1
19089870,4rtkit-daemon473-21gmain00:20:020
19089870,4rtkit-daemon473-21gmain0
19089870,4rtkit-daemon1928-21sendmail-mta1
19089870,3rtkit-daemon671-21ntpd23:45:470
19089870,3rtkit-daemon671-21ntpd0
19089870,3rtkit-daemon667-21snmpd1
19089870,3rtkit-daemon421-21avahi-daemon21:05:431
19089870,3rtkit-daemon421-21avahi-daemon1
19089870,3rtkit-daemon2907931-21tune2fs00:35:152
19089870,3rtkit-daemon2665339-21ssh22:05:010
19089870,3rtkit-daemon1223-21cron20:52:011
2913986260,1sleep00-21swapper/000:36:390
2913633260,4sleep10-21swapper/100:35:481
2897483260,1sleep00-21swapper/000:25:410
2892276261,2sleep244-21ksoftirqd/200:25:192
2851162260,3sleep12851364-21hddtemp_smartct00:00:171
2850946261,2sleep20-21swapper/200:00:142
2826703261,2sleep10-21swapper/123:45:151
2818631261,2sleep22818670-21apt-get2
2816588260,1sleep20-21swapper/223:36:062
2802381260,4sleep12802336-21/usr/sbin/munin1
2784386260,4sleep12784394-21ssh23:16:251
2736104260,1sleep10-21swapper/122:46:401
2719864261,3sleep12719870-21ssh22:36:301
2719740260,1sleep20-21swapper/222:36:032
2712495261,2sleep10-21swapper/11
2697509260,1sleep10-21swapper/11
2664079261,3sleep22664086-21sh2
2656734261,2sleep22656749-21sendmail2
2655790260,1sleep10-21swapper/11
2639703260,1sleep10-21swapper/11
2623078260,1sleep00-21swapper/021:36:460
2616675261,2sleep22616708-21apt-get21:35:122
2602555260,4sleep22604605-21kthreadcore2
2600916261,2sleep00-21swapper/021:25:160
2538802261,3sleep02538958-21diskstats20:40:160
2538763260,4sleep22538777-21find2
2529841260,3sleep22531194-21memory2
2503039260,3sleep22505169-21kthreadcore20:10:212
2500122260,3sleep20-21swapper/220:08:092
2500078260,1sleep00-21swapper/020:07:350
2452050260,3sleep02454077-21tr0
2421913260,1sleep00-21swapper/00
19089860,3rtkit-daemon671-21ntpd22:15:380
19089860,3rtkit-daemon671-21ntpd1
19089860,3rtkit-daemon473-21gmain1
2904672250,3sleep22904834-21ssh00:30:202
2901154251,2sleep00-21swapper/000:30:180
2872740250,3sleep12872855-21sed00:10:231
2809626251,2sleep20-21swapper/223:33:462
2801928250,3sleep22801930-21ssh23:29:412
2801928250,3sleep22801930-21ssh2
2768485250,3sleep20-21swapper/22
2755250250,3sleep12756337-21kthreadcore23:00:181
2727204251,2sleep20-21swapper/222:40:242
2687285251,2sleep10-21swapper/11
2678727250,3sleep22678841-21mailstats2
2665389250,3sleep2299-21systemd-journal22:05:092
2632379250,3sleep22632387-21ssh2
2631951251,2sleep00-21swapper/021:43:310
2594066251,2sleep00-21swapper/021:20:190
2575841250,1sleep00-21swapper/021:06:450
2575841250,1sleep00-21swapper/021:06:450
2550854250,1sleep20-21swapper/220:47:442
2525592250,1sleep20-21swapper/22
2487525250,1sleep00-21swapper/019:57:170
2483643250,2sleep20-21swapper/219:55:192
2475076250,1sleep00-21swapper/00
2474847250,1sleep20-21swapper/22
2462883250,2sleep00-21swapper/019:40:140
2450705251,2sleep20-21swapper/219:30:182
2428431250,1sleep20-21swapper/219:13:402
2842807240,1sleep10-21swapper/123:55:121
2825613240,1sleep10-21swapper/123:43:141
2775974240,2sleep12775976-21ssh1
2737554240,2sleep12737802-21ssh22:50:011
2697338241,1sleep20-21swapper/222:25:012
2563843240,1sleep20-21swapper/221:00:142
2559021240,1sleep00-21swapper/020:55:190
2525045240,2sleep00-21swapper/020:25:220
2513615240,2sleep2667-21snmpd20:20:032
2468657240,1sleep20-21swapper/22
2449679240,1sleep20-21swapper/22
19089840,1rtkit-daemon0-21swapper/022:50:040
2557309230,1sleep20-21swapper/22
2551595230,1sleep20-21swapper/22
2532479230,1sleep20-21swapper/22
2513147230,1sleep00-21swapper/020:20:010
2475245230,1sleep20-21swapper/219:50:022
2448031230,1sleep00-21swapper/019:25:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional