You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 23:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Jul 13, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
326063222490,8sleep33260610-21perf07:10:003
326560921250,4sleep20-21swapper/207:20:182
329690621180,4sleep30-21swapper/308:50:133
330608421140,2sleep00-21swapper/009:15:200
331313921030,2sleep20-21swapper/209:35:182
325959726844,15sleep00-21swapper/007:05:200
326030326442,15sleep20-21swapper/207:06:192
326036426344,14sleep10-21swapper/107:07:061
33584772520,2sleep10-21swapper/111:45:151
33497022510,5sleep20-21swapper/211:20:132
33742202500,3sleep10-21swapper/112:30:181
33585092470,9sleep234-21ksoftirqd/211:45:152
33519442440,4sleep33056-21PK-Backend11:25:203
33281472420,3sleep00-21swapper/010:20:130
33546262330,2sleep20-21swapper/211:35:012
326073599333,18cyclictest3361033-21seq11:51:221
326073599321,18cyclictest3287671-21runrttasks08:24:161
326073599321,18cyclictest3287671-21runrttasks08:24:161
326073599321,17cyclictest3364192-21python312:00:201
326073599311,19cyclictest3285719-21runrttasks08:19:471
326073599311,18cyclictest0-21swapper/109:05:381
326073599302,15cyclictest0-21swapper/107:30:131
326073599301,17cyclictest0-21swapper/108:35:071
326073599300,18cyclictest1-21systemd12:14:551
326073599300,18cyclictest1-21systemd12:14:551
326073599300,17cyclictest3344197-21/usr/sbin/munin11:05:121
326074499290,15cyclictest3263121-21chrt07:15:063
326073599293,15cyclictest0-21swapper/112:21:191
326073599291,20cyclictest33493841gpgconf11:20:001
3260735992911,11cyclictest0-21swapper/107:36:141
3260737992816,11cyclictest3289511-21unin-run08:30:002
326073599277,18cyclictest3313755-21perf09:40:001
326073599273,22cyclictest3325352-21timerandwakeup10:10:221
3260735992710,14cyclictest3363070-21grep12:00:011
326073199270,9cyclictest3297195-21latency08:50:170
326073599269,16cyclictest3262387-21cron07:15:001
3260735992611,13cyclictest436-20systemd-journal07:54:131
3260735992611,13cyclictest3278810-21cron08:00:001
3260735992610,14cyclictest3348896-21sshd11:19:441
139099260,13rtkit-daemon0-21swapper/207:42:042
3260744992520,4cyclictest3056-21PK-Backend10:57:203
326074499250,23cyclictest3056-21PK-Backend08:41:063
326073599256,15cyclictest3341388-21/usr/sbin/munin10:55:211
326073599255,18cyclictest3316117-21df09:45:131
326073599253,20cyclictest3351405-21cstates11:25:121
326073599251,23cyclictest2175-21JS10:27:481
3260735992512,11cyclictest3371746-21latency_hist12:25:011
3260735992512,11cyclictest3368276-21latency_hist12:15:001
326073599251,20cyclictest3355210-21latency11:35:151
326073599251,20cyclictest3280799-21/usr/sbin/munin08:05:191
326073599251,19cyclictest3357868-21sh11:45:001
326073599251,19cyclictest3335293-21apt-get10:40:041
3260735992510,13cyclictest3314679-21grep09:40:191
326073199250,7cyclictest3312180-21sed09:35:010
326073199250,7cyclictest3286116-21apt-config08:20:010
326073199250,7cyclictest3286116-21apt-config08:20:010
3260744992419,4cyclictest3056-21PK-Backend11:02:133
3260744992417,6cyclictest3056-21PK-Backend11:12:003
3260744992417,6cyclictest3056-21PK-Backend11:12:003
326073599246,16cyclictest3289028-21grep08:25:231
326073599242,20cyclictest26-21ksoftirqd/108:40:101
326073599242,18cyclictest82332-21update-manager07:19:101
326073599241,21cyclictest3275147-21unin-run07:50:001
326073599241,20cyclictest3304281-21grep09:10:181
326073599241,19cyclictest3365071-21apt12:05:161
326073599241,19cyclictest3354531-21latency_hist11:35:011
326073599241,19cyclictest3279960-21python308:00:231
326073599241,19cyclictest3268094-21dump-pmu-power07:30:001
326073599240,21cyclictest3329724-21apt-key10:25:011
3260744992317,5cyclictest3056-21PK-Backend11:07:073
3260744992317,5cyclictest3056-21PK-Backend07:49:223
3260744992316,1cyclictest3056-21PK-Backend10:39:493
3260744992314,2cyclictest3056-21PK-Backend11:21:473
326074499230,21cyclictest3056-21PK-Backend08:23:223
326074499230,21cyclictest3056-21PK-Backend08:23:223
326073599237,14cyclictest3305544-21wc09:15:141
326073599237,14cyclictest3272221-21cat07:40:121
326073599236,15cyclictest3297722-21cat08:50:231
326073599236,15cyclictest3265457-21cat07:20:171
326073599235,16cyclictest3311249-21ntp_kernel_err09:30:181
326073599235,16cyclictest3307945-21smartctl09:20:211
326073599234,17cyclictest3299136-21ntp_kernel_pll_08:55:201
326073599232,17cyclictest0-21swapper/110:45:141
326073599232,17cyclictest0-21swapper/109:25:121
326073599232,16cyclictest3327873-21apt10:20:121
326073599231,20cyclictest3333541-21cstates10:35:121
326073599231,19cyclictest3327082-21find10:15:241
326073599231,19cyclictest3327082-21find10:15:241
326073599231,19cyclictest3320915-21/usr/sbin/munin10:00:271
326073599231,18cyclictest3331600-21apt10:30:111
326073599230,21cyclictest3284139-21latency_hist08:15:001
326073599230,20cyclictest3346757-21wc11:10:201
326073599230,20cyclictest3346757-21wc11:10:201
326073599230,19cyclictest436-20systemd-journal12:40:001
326073199233,8cyclictest0-21swapper/011:15:130
326073199233,6cyclictest3285477-21runrttasks08:16:420
326073199232,6cyclictest3345061-21users11:05:220
326073199230,14cyclictest3354667-21apt11:35:090
139099230,17rtkit-daemon34-21ksoftirqd/211:50:202
139099230,16rtkit-daemon3056-21PK-Backend09:13:133
139099230,16rtkit-daemon3056-21PK-Backend09:01:223
139099230,14rtkit-daemon0-21swapper/209:09:162
32641922220,2sleep20-21swapper/207:15:252
3260744992219,2cyclictest3056-21PK-Backend12:38:393
3260744992219,2cyclictest3056-21PK-Backend11:41:133
3260744992219,2cyclictest3056-21PK-Backend08:31:403
3260744992215,1cyclictest3056-21PK-Backend10:52:273
326074499220,20cyclictest3056-21PK-Backend12:14:333
326074499220,20cyclictest3056-21PK-Backend12:14:333
326074499220,20cyclictest3056-21PK-Backend11:56:553
326074499220,20cyclictest3056-21PK-Backend11:36:543
326074499220,20cyclictest3056-21PK-Backend10:02:183
326074499220,20cyclictest3056-21PK-Backend08:07:473
3260737992220,1cyclictest0-21swapper/212:32:272
3260737992220,1cyclictest0-21swapper/212:08:542
3260737992220,1cyclictest0-21swapper/210:49:452
3260737992220,1cyclictest0-21swapper/209:49:362
3260737992218,2cyclictest34-21ksoftirqd/212:39:592
326073799220,21cyclictest0-21swapper/208:06:412
326073599224,16cyclictest3289965-21wc08:30:131
326073599221,19cyclictest3343721-21sh11:05:011
326073599221,18cyclictest3318339-21python309:50:201
326073599221,17cyclictest3319208-21apt09:55:121
326073599220,20cyclictest3295074-21grep08:45:131
326073599220,19cyclictest3339287-21kernelversion10:50:171
326073599220,19cyclictest3300124-21acpi09:00:111
326073199221,8cyclictest3266404-21latency_hist07:25:000
326073199221,7cyclictest3346582-21python311:10:190
326073199221,7cyclictest3346582-21python311:10:180
139099220,15rtkit-daemon3056-21PK-Backend09:31:323
139099220,15rtkit-daemon3056-21PK-Backend09:19:373
139099220,15rtkit-daemon3056-21PK-Backend08:13:443
139099220,15rtkit-daemon3056-21PK-Backend07:32:193
33056092210,6sleep2331rcuc/209:15:142
3260744992119,1cyclictest3056-21PK-Backend12:16:423
3260744992119,1cyclictest3056-21PK-Backend11:45:333
3260744992119,1cyclictest3056-21PK-Backend10:45:223
3260744992119,1cyclictest3056-21PK-Backend10:12:043
3260744992119,1cyclictest3056-21PK-Backend08:03:283
3260744992119,1cyclictest0-21swapper/310:21:373
3260744992118,1cyclictest3056-21PK-Backend07:50:093
3260744992114,1cyclictest3056-21PK-Backend12:34:203
326074499210,20cyclictest3056-21PK-Backend07:13:533
326074499210,1cyclictest3056-21PK-Backend10:05:223
326074499210,19cyclictest3056-21PK-Backend12:20:143
326074499210,19cyclictest3056-21PK-Backend09:20:473
3260737992119,1cyclictest0-21swapper/210:25:042
3260737992119,1cyclictest0-21swapper/209:53:142
3260737992119,1cyclictest0-21swapper/209:34:082
3260737992119,1cyclictest0-21swapper/208:32:232
3260737992119,1cyclictest0-21swapper/207:57:152
3260737992119,1cyclictest0-21swapper/207:25:182
3260737992113,6cyclictest3325854-21cron10:15:002
326073799210,20cyclictest0-21swapper/211:15:262
326073799210,20cyclictest0-21swapper/208:53:122
326073799210,20cyclictest0-21swapper/208:13:232
326073199213,6cyclictest3262425-21systemctl07:15:010
326073199212,5cyclictest0-21swapper/011:45:140
326073199212,4cyclictest0-21swapper/009:22:410
3260731992113,6cyclictest3334198-21seq10:35:200
326073199210,20cyclictest0-21swapper/010:32:470
326073199210,1cyclictest0-21swapper/008:07:480
139099210,15rtkit-daemon3056-21PK-Backend12:26:223
139099210,15rtkit-daemon3056-21PK-Backend12:00:373
139099210,14rtkit-daemon3056-21PK-Backend08:27:173
3260744992018,1cyclictest3056-21PK-Backend12:06:083
3260744992017,2cyclictest3056-21PK-Backend10:33:073
326074499200,19cyclictest3056-21PK-Backend11:54:463
326074499200,18cyclictest3056-21PK-Backend10:42:533
326073799207,5cyclictest141rcu_preempt10:05:242
3260737992018,1cyclictest0-21swapper/212:15:232
3260737992018,1cyclictest0-21swapper/212:14:352
3260737992018,1cyclictest0-21swapper/212:14:342
3260737992018,1cyclictest0-21swapper/211:08:242
3260737992018,1cyclictest0-21swapper/210:50:332
3260737992018,1cyclictest0-21swapper/210:37:222
3260737992018,1cyclictest0-21swapper/210:02:262
3260737992018,1cyclictest0-21swapper/208:23:172
3260737992018,1cyclictest0-21swapper/208:23:172
3260737992018,1cyclictest0-21swapper/207:34:172
3260737992010,8cyclictest3298812-21sh08:55:172
326073799200,1cyclictest0-21swapper/207:46:272
326073799200,19cyclictest3056-21PK-Backend09:12:452
326073799200,19cyclictest0-21swapper/208:01:342
326073199203,16cyclictest3330488-21ls10:25:190
326073199203,15cyclictest3365169-21sort12:05:170
3260731992018,1cyclictest3056-21PK-Backend07:35:500
3260731992018,1cyclictest0-21swapper/010:51:330
3260731992018,1cyclictest0-21swapper/009:50:150
3260731992018,1cyclictest0-21swapper/007:30:570
326073199201,6cyclictest3360239-21df_inode11:50:160
326073199201,6cyclictest3353508-21memory11:30:180
326073199201,6cyclictest3319579-21df_inode09:55:140
326073199201,6cyclictest13-21ksoftirqd/009:45:170
3260731992013,5cyclictest436-20systemd-journal08:00:000
326073199200,7cyclictest3279957-21python308:00:230
326073199200,1cyclictest0-21swapper/012:10:280
326073199200,1cyclictest0-21swapper/012:10:280
326073199200,1cyclictest0-21swapper/011:21:540
326073199200,1cyclictest0-21swapper/007:45:370
326073199200,19cyclictest13-21ksoftirqd/008:44:040
326073199200,19cyclictest0-21swapper/012:36:100
326073199200,19cyclictest0-21swapper/011:56:070
326073199200,19cyclictest0-21swapper/011:42:230
326073199200,19cyclictest0-21swapper/010:07:530
326073199200,19cyclictest0-21swapper/007:54:500
326073199200,19cyclictest0-21swapper/007:28:470
139099200,14rtkit-daemon3056-21PK-Backend09:42:453
139099200,14rtkit-daemon3056-21PK-Backend07:23:073
139099200,14rtkit-daemon0-21swapper/211:40:462
139099200,14rtkit-daemon0-21swapper/210:57:372
139099200,12rtkit-daemon3056-21PK-Backend08:38:263
3260744991917,1cyclictest3056-21PK-Backend09:54:273
3260744991917,1cyclictest3056-21PK-Backend09:25:193
3260744991917,1cyclictest3056-21PK-Backend08:49:453
3260744991917,1cyclictest3056-21PK-Backend07:43:403
326074499190,1cyclictest3056-21PK-Backend11:30:253
326074499190,1cyclictest3056-21PK-Backend09:06:553
326074499190,18cyclictest3056-21PK-Backend10:25:503
3260737991917,1cyclictest0-21swapper/212:26:452
3260737991917,1cyclictest0-21swapper/212:03:132
3260737991917,1cyclictest0-21swapper/211:56:442
3260737991917,1cyclictest0-21swapper/210:42:562
3260737991917,1cyclictest0-21swapper/210:22:012
3260737991917,1cyclictest0-21swapper/207:10:582
326073799190,1cyclictest0-21swapper/212:22:262
326073799190,1cyclictest0-21swapper/211:29:472
326073799190,1cyclictest0-21swapper/211:11:562
326073799190,1cyclictest0-21swapper/211:11:562
326073799190,1cyclictest0-21swapper/209:00:562
326073799190,1cyclictest0-21swapper/208:45:072
326073799190,1cyclictest0-21swapper/208:40:412
326073799190,1cyclictest0-21swapper/207:52:562
326073799190,18cyclictest0-21swapper/210:19:302
326073799190,18cyclictest0-21swapper/210:19:302
326073799190,18cyclictest0-21swapper/208:36:492
326073199192,5cyclictest0-21swapper/012:27:510
3260731991917,1cyclictest0-21swapper/012:21:490
3260731991917,1cyclictest0-21swapper/012:00:270
3260731991917,1cyclictest0-21swapper/011:04:170
3260731991917,1cyclictest0-21swapper/010:57:140
3260731991917,1cyclictest0-21swapper/010:19:220
3260731991917,1cyclictest0-21swapper/010:19:220
3260731991917,1cyclictest0-21swapper/010:10:230
3260731991917,1cyclictest0-21swapper/010:01:440
3260731991917,1cyclictest0-21swapper/008:58:310
3260731991917,1cyclictest0-21swapper/007:17:180
3260731991916,2cyclictest13-21ksoftirqd/009:02:230
3260731991912,5cyclictest3375165-21seq12:34:300
3260731991912,5cyclictest3337623-21cat10:45:190
326073199190,6cyclictest3289754-21apt08:30:120
326073199190,1cyclictest0-21swapper/012:16:090
326073199190,1cyclictest0-21swapper/009:40:420
326073199190,1cyclictest0-21swapper/008:26:190
326073199190,18cyclictest0-21swapper/010:41:240
326073199190,18cyclictest0-21swapper/008:35:120
3260744991816,1cyclictest3056-21PK-Backend10:16:513
3260744991816,1cyclictest3056-21PK-Backend10:16:513
3260744991816,1cyclictest3056-21PK-Backend08:56:413
326073799183,13cyclictest3331730-21tr10:30:122
3260737991813,3cyclictest3271627-21munin-plugin-st07:40:002
3260737991811,5cyclictest3307882-21cut09:20:202
3260731991816,1cyclictest0-21swapper/011:29:590
3260731991815,2cyclictest161rcuc/008:45:180
3260731991814,3cyclictest3313736-21cron09:40:000
3260731991812,5cyclictest3304231-21ls09:10:180
3260731991812,4cyclictest3272254-21seq07:40:130
3260744991715,1cyclictest3056-21PK-Backend09:58:043
3260744991715,1cyclictest3056-21PK-Backend07:56:253
3260737991715,1cyclictest0-21swapper/209:55:442
3260737991715,1cyclictest0-21swapper/208:18:442
3260731991715,1cyclictest0-21swapper/008:11:330
3260731991711,5cyclictest3309706-21sed09:25:220
3260744991614,1cyclictest3056-21PK-Backend09:49:403
3260744991614,1cyclictest3056-21PK-Backend07:38:473
3260744991611,4cyclictest42-21ksoftirqd/308:20:003
3260737991611,3cyclictest34-21ksoftirqd/211:35:202
326073799160,3cyclictest141rcu_preempt11:00:192
3260731991611,3cyclictest3302274-21sh09:05:150
3260744991513,1cyclictest3056-21PK-Backend07:25:423
3260737991511,3cyclictest1095-21runrttasks09:27:522
3260744991411,2cyclictest3056-21PK-Backend11:18:153
3260737991412,1cyclictest0-21swapper/209:44:492
139099120,4rtkit-daemon3056-21PK-Backend09:36:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional