You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 23:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Thu Mar 12, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
371392721390,5sleep23713934-21gltestperf09:50:142
3510547212155,61sleep30-21swapper/307:09:313
351030128955,7sleep00-21swapper/007:06:240
351047628153,9sleep10-21swapper/107:08:371
350558727556,12sleep20-21swapper/207:05:092
3510792992512,7cyclictest4003106-21turbostat12:35:003
351079299231,19cyclictest0-21swapper/310:56:463
3510792992216,3cyclictest3730987-21taskset10:00:093
351079299221,18cyclictest0-21swapper/309:49:593
351079299221,17cyclictest3625687-21taskset08:51:223
351079299221,13cyclictest0-21swapper/310:32:403
3510792992118,1cyclictest0-21swapper/312:04:393
3510792992117,2cyclictest0-21swapper/310:35:153
351079299211,16cyclictest66950irq/132-enp1s0-TxRx-311:04:163
37997752201,2sleep00-21swapper/010:39:000
3510792992017,1cyclictest0-21swapper/311:30:523
351079299201,6cyclictest3540208-21taskset07:34:233
351079299201,15cyclictest3675313-21taskset09:26:573
351079299201,15cyclictest0-21swapper/311:40:213
351079299201,15cyclictest0-21swapper/311:14:063
3510792992010,7cyclictest3651305-21taskset09:15:033
351079299200,7cyclictest3782792-21taskset10:29:393
351079299200,16cyclictest0-21swapper/309:57:513
39989532191,5sleep13999016-21ntpq12:30:201
351079299194,12cyclictest0-21swapper/309:32:493
351079299192,12cyclictest3752706-21taskset10:10:193
3510792991917,1cyclictest0-21swapper/311:17:343
3510792991917,1cyclictest0-21swapper/308:15:183
3510792991916,1cyclictest0-21swapper/310:20:003
351079299191,15cyclictest66950irq/132-enp1s0-TxRx-310:50:143
351079299191,14cyclictest0-21swapper/312:23:163
351079299191,14cyclictest0-21swapper/310:05:333
351079299191,14cyclictest0-21swapper/309:41:023
351079299191,14cyclictest0-21swapper/309:37:033
351079299191,13cyclictest0-21swapper/311:25:113
39055622181,4sleep00-21swapper/011:39:120
351079299181,14cyclictest36481812sleep309:11:153
351079299181,14cyclictest0-21swapper/312:13:103
351079299181,14cyclictest0-21swapper/312:06:483
351079299181,14cyclictest0-21swapper/311:20:473
351079299181,14cyclictest0-21swapper/310:45:213
351079299181,14cyclictest0-21swapper/310:41:273
351079299181,14cyclictest0-21swapper/309:23:493
351079299181,13cyclictest0-21swapper/311:56:243
351079299181,13cyclictest0-21swapper/311:50:023
351079299181,13cyclictest0-21swapper/310:22:113
351079299181,13cyclictest0-21swapper/309:50:113
351079299180,14cyclictest0-21swapper/311:05:283
40023352171,3sleep00-21swapper/012:34:010
38617712171,3sleep20-21swapper/211:14:302
38338072171,2sleep20-21swapper/210:57:172
37822982171,3sleep00-21swapper/010:29:010
37013352171,3sleep00-21swapper/009:41:110
36945772171,2sleep20-21swapper/209:38:522
351079299171,4cyclictest0-21swapper/311:35:123
3510792991712,2cyclictest0-21swapper/308:05:153
351079299171,13cyclictest0-21swapper/312:15:403
38940092161,2sleep00-21swapper/011:30:300
37577262161,4sleep03757848-21cut10:15:140
37001032161,4sleep23700175-21ntpq09:40:202
36505782161,4sleep13642252-21diskmemload09:14:261
351079299161,12cyclictest0-21swapper/309:10:013
351079299161,12cyclictest0-21swapper/309:10:003
351079299161,12cyclictest0-21swapper/307:55:153
351079299161,12cyclictest0-21swapper/307:45:263
351079299161,12cyclictest0-21swapper/307:35:103
351079299160,14cyclictest0-21swapper/311:45:103
351079299160,14cyclictest0-21swapper/308:55:183
351079299160,13cyclictest0-21swapper/312:35:403
351079299160,13cyclictest0-21swapper/312:25:273
351079299160,13cyclictest0-21swapper/307:20:133
351079299160,13cyclictest0-21swapper/307:10:173
40097422151,3sleep03642252-21diskmemload12:37:130
40084652151,9sleep13642252-21diskmemload12:35:331
38953992150,2sleep20-21swapper/211:32:212
38866432151,3sleep23642252-21diskmemload11:27:202
37981232151,3sleep23642252-21diskmemload10:36:492
37648372151,3sleep13642252-21diskmemload10:19:041
36833962151,3sleep13642252-21diskmemload09:30:411
36661422151,3sleep00-21swapper/009:21:500
3510792991510,3cyclictest0-21swapper/307:50:113
351079299150,13cyclictest0-21swapper/309:00:163
351079299150,13cyclictest0-21swapper/308:25:163
351079299150,13cyclictest0-21swapper/307:40:193
351079299150,13cyclictest0-21swapper/307:15:173
351079299150,12cyclictest0-21swapper/308:35:183
351079299150,12cyclictest0-21swapper/308:20:183
351079299150,12cyclictest0-21swapper/307:25:153
39865242146,4sleep134-21ksoftirqd/112:25:151
39575662141,8sleep13642252-21diskmemload12:08:121
38854602141,9sleep03885470-21ssh11:25:460
38014802141,7sleep23801560-21irqstats10:40:132
37743252141,5sleep13642252-21diskmemload10:25:011
37566492141,2sleep20-21swapper/210:14:482
37471982141,3sleep13642252-21diskmemload10:08:501
37064182141,5sleep13708111-21cut09:45:171
36928612140,1sleep00-21swapper/009:36:380
36767092145,2sleep1171rcu_preempt09:28:491
36489232141,3sleep23642252-21diskmemload09:12:142
36379182141,6sleep03639589-21cut09:05:150
36379182141,6sleep03639589-21cut09:05:150
351079299140,13cyclictest0-21swapper/308:30:143
351079299140,12cyclictest0-21swapper/308:10:153
351079299140,11cyclictest0-21swapper/308:40:163
39479832131,3sleep03947997-21ssh12:02:050
39399982131,7sleep13642252-21diskmemload11:58:051
39300692131,6sleep13642252-21diskmemload11:51:571
39043572130,3sleep13642252-21diskmemload11:37:371
38758472131,6sleep03875915-21ntpq11:20:200
37906172131,5sleep00-21swapper/010:33:260
37450282130,2sleep20-21swapper/210:05:582
37305892131,8sleep1255-21systemd-journal10:00:001
37271522131,5sleep20-21swapper/209:55:292
36781392131,6sleep23677631-21apt-get09:30:022
351079299130,11cyclictest0-21swapper/308:45:133
351079299130,11cyclictest0-21swapper/308:00:193
40016462121,3sleep24001659-21sh12:33:062
39644532121,5sleep00-21swapper/012:10:430
39490322121,5sleep23642252-21diskmemload12:03:292
39460012121,7sleep13946053-21ntpq12:00:181
39060222121,5sleep23642252-21diskmemload11:39:512
38686572120,8sleep13868672-21ssh11:16:401
38543222121,7sleep03854596-21grep11:10:140
38490952121,5sleep23849187-21ntpq11:05:192
38421512121,5sleep00-21swapper/011:01:450
37729952121,7sleep03773007-21rm10:23:160
37624632121,6sleep23642252-21diskmemload10:15:562
36659212121,5sleep20-21swapper/209:21:322
36263132121,5sleep03626513-21df_inode08:55:130
36250082121,6sleep03625059-21ntpq08:50:210
36117352121,7sleep23613509-21grep08:40:162
35726362123,5sleep03572654-21apt-get08:05:110
35456382121,7sleep13545585-31find07:39:271
35294032121,6sleep23529315-21apt-get07:25:012
39928522110,3sleep23992863-21ssh12:28:062
39814172111,5sleep03981475-21ntpq12:20:210
39733872111,6sleep13642252-21diskmemload12:15:581
39660092111,4sleep23642252-21diskmemload12:12:462
39605952111,6sleep13962263-21cut12:10:161
39390622110,2sleep00-21swapper/011:56:520
39388342111,3sleep20-21swapper/211:56:342
39314662110,5sleep00-21swapper/011:53:480
39304352111,6sleep23930449-21ssh11:52:262
39194132111,5sleep03919466-21ntpq11:45:180
39145392111,3sleep20-21swapper/211:44:312
38882582111,4sleep13642252-21diskmemload11:29:281
38682482111,7sleep03868262-21ssh11:16:070
38452422111,5sleep03845330-21diskstats11:05:130
38245452111,3sleep00-21swapper/010:51:350
38163522111,3sleep00-21swapper/010:47:170
38069942112,4sleep13642252-21diskmemload10:41:571
37977132111,4sleep13642252-21diskmemload10:36:171
37909902111,3sleep10-21swapper/110:33:551
37907942111,3sleep20-21swapper/210:33:402
37731092111,5sleep13642252-21diskmemload10:23:251
37559292111,4sleep13642252-21diskmemload10:13:521
37351862111,4sleep03735239-21ssh10:00:190
37091442110,1sleep00-21swapper/009:45:240
37049462111,6sleep23705153-21gltestperf09:45:132
37000482111,5sleep13700100-21ssh09:40:191
35784582111,6sleep23577950-21/usr/sbin/munin08:10:152
35445902111,5sleep23544636-21ntpq07:35:192
39837962101,2sleep10-21swapper/112:22:431
39578152100,1sleep00-21swapper/012:08:310
39227512101,3sleep20-21swapper/211:48:482
39213082100,1sleep10-21swapper/111:46:551
39120022101,3sleep13642252-21diskmemload11:41:101
39065252101,4sleep03642252-21diskmemload11:40:090
38712092101,4sleep13642252-21diskmemload11:20:011
38694902101,3sleep23642252-21diskmemload11:17:462
38536612101,3sleep13642252-21diskmemload11:10:031
38348912100,1sleep00-21swapper/010:58:430
38317152101,4sleep13831773-21ntpq10:55:201
38263372101,2sleep20-21swapper/210:53:562
38240962100,1sleep10-21swapper/110:50:591
38174902101,3sleep13642252-21diskmemload10:48:461
38161242101,2sleep20-21swapper/210:46:592
38067242101,2sleep00-21swapper/010:41:360
37741882101,2sleep20-21swapper/210:24:512
37487752101,5sleep03748965-21head10:10:140
37379092100,1sleep20-21swapper/210:03:052
37376572101,4sleep13642252-21diskmemload10:02:441
37292592101,2sleep00-21swapper/009:58:140
36946082100,4sleep10-21swapper/109:38:551
36719572101,4sleep23673243-21memory09:25:172
36607582101,5sleep13661482-21tr09:20:161
36597442101,5sleep23659793-21dump-pmu-power09:20:002
36574232101,3sleep03642252-21diskmemload09:16:570
36564232101,5sleep13656436-21ssh09:15:381
36432482101,6sleep03644360-21tr09:10:170
36413032101,4sleep13641356-21ntp_states09:05:201
36413032101,4sleep13641356-21ntp_states09:05:201
36359892101,5sleep23636046-21ntpq09:00:212
36248142101,3sleep23624955-21ntpq08:50:182
36155812101,2sleep20-21swapper/208:45:132
36051082101,4sleep13605179-21irqstats08:35:151
35993652101,4sleep23599568-21diskstats08:30:102
35940202101,4sleep23594238-21df_inode08:25:132
35936942101,5sleep13593724-21perf08:25:001
35926782101,4sleep03592725-21ntpq08:20:180
35820772101,5sleep03582130-21ntp_states08:10:200
35664202101,5sleep03566665-21users07:55:250
35606882101,5sleep23560728-21ntpq07:50:182
35556922101,2sleep10-21swapper/107:45:241
35500872101,5sleep13550141-21ntpq07:40:191
35305482101,5sleep13532210-21cut07:25:161
35283882105,2sleep2421rcuc/207:20:212
35164902104,3sleep03517249-21kthreadcore07:15:010
4007621291,4sleep24007669-21ntpq12:35:182
3991203290,5sleep03991217-21ssh12:25:540
3982649291,2sleep20-21swapper/212:21:112
3974497291,4sleep03974511-21ssh12:17:260
3968080291,4sleep23968111-21apt-get12:15:112
3950492292,3sleep2431ktimers/212:05:042
3894528291,2sleep10-21swapper/111:31:121
3877703291,3sleep23642252-21diskmemload11:21:592
3850364291,3sleep13642252-21diskmemload11:06:021
3842637291,3sleep23642252-21diskmemload11:02:222
3841399290,3sleep10-21swapper/111:00:461
3780250290,1sleep20-21swapper/210:26:182
3746624291,2sleep00-21swapper/010:08:040
3719897291,2sleep00-21swapper/009:52:290
3719658290,1sleep10-21swapper/109:52:101
3678181291,2sleep00-21swapper/009:30:040
3674309291,2sleep00-21swapper/009:25:390
3635839291,4sleep03635932-21ld09:00:180
3599853291,4sleep13601190-21cut08:30:131
3598015291,4sleep13598065-21ntpq08:25:201
3588149291,3sleep2802-21snmpd08:17:322
3576736290,4sleep13576795-21ntpq08:05:201
3572918291,4sleep23573042-21perl08:05:132
3567343291,4sleep13567501-21ls08:00:111
3565989291,4sleep13566037-21ntpq07:55:181
3551338290,1sleep00-21swapper/007:45:110
3545647290,5sleep03545585-31find07:39:350
3528332291,4sleep13528381-21ntpq07:20:191
3515559291,4sleep03515615-21sed07:10:250
3632423281,3sleep13633414-21kthreadcore09:00:161
3614598281,4sleep13614608-21awk08:40:231
3610230280,3sleep03609713-21apt-get08:40:020
3599281282,3sleep00-21swapper/008:30:080
3578243281,3sleep13578457-21smartctl08:10:131
3567035281,4sleep23567284-21cat08:00:002
3548799281,3sleep23549895-21memory07:40:162
3529842280,2sleep00-21swapper/007:25:110
3524758281,3sleep03525116-21kthreadcore07:20:140
3515950280,1sleep20-21swapper/207:11:152
3641941270,1sleep20-21swapper/209:06:132
3641941270,1sleep20-21swapper/209:06:132
3630411271,3sleep13630495-21ps08:55:221
3615864271,3sleep00-21swapper/008:45:150
3609635270,1sleep20-21swapper/208:39:132
3588619270,1sleep20-21swapper/208:20:042
3583412271,2sleep00-21swapper/008:15:130
3567321271,2sleep03567340-21apt-get08:00:110
3561020271,3sleep0151ktimers/007:50:240
3545990270,2sleep00-21swapper/007:40:100
3523943271,3sleep20-21swapper/207:20:002
3523926270,1sleep10-21swapper/107:19:511
3630201260,2sleep20-21swapper/208:55:192
3625289261,2sleep10-21swapper/108:50:251
3620497260,1sleep10-21swapper/108:48:091
3587117260,2sleep13587290-21strings08:15:181
3572355260,4sleep20-21swapper/208:04:492
3556630260,1sleep10-21swapper/107:50:011
3551605260,2sleep20-21swapper/207:45:142
3540148260,1sleep20-21swapper/207:33:392
3539359261,2sleep00-21swapper/007:30:200
3539143260,2sleep10-21swapper/107:30:171
3510964260,2sleep10-21swapper/107:10:011
3604898250,1sleep00-21swapper/008:35:130
3597925250,1sleep00-21swapper/008:25:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional