You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 23:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Sun Nov 30, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24176422070,2sleep10-21swapper/111:46:571
4066735212161,55sleep30-21swapper/307:09:423
406676028554,9sleep20-21swapper/207:09:572
406655728252,10sleep10-21swapper/107:07:241
406650828057,6sleep00-21swapper/007:06:450
4066885993019,7cyclictest130996-21taskset10:34:273
4066885992823,3cyclictest0-21swapper/311:20:283
406688599261,19cyclictest281670-21taskset12:10:293
4066885992521,2cyclictest0-21swapper/309:50:163
4066885992520,2cyclictest0-21swapper/309:15:183
4066885992415,4cyclictest1158582sleep310:24:063
406688599240,20cyclictest108454-21taskset10:19:323
4066885992321,1cyclictest0-21swapper/309:35:193
4066885992320,1cyclictest0-21swapper/311:45:173
4066885992318,2cyclictest66550irq/131-enp1s0-TxRx-311:55:013
4066885992313,5cyclictest2344742sleep311:43:443
406688599229,10cyclictest153146-21turbostat10:54:543
406688599222,14cyclictest24894-21taskset09:25:123
406688599222,13cyclictest321436-21taskset12:35:223
4066885992219,1cyclictest0-21swapper/310:07:513
4066885992219,1cyclictest0-21swapper/307:35:223
4066885992212,7cyclictest123732-21turbostat10:30:003
3237982221,2sleep20-21swapper/212:38:232
406688599218,6cyclictest290394-21taskset12:16:263
406688599212,13cyclictest180681-21taskset11:06:383
4066885992112,4cyclictest3074612sleep312:27:483
406688599211,14cyclictest2168-21turbostat09:10:003
406688599202,12cyclictest78251-21taskset09:59:553
4066885992017,1cyclictest0-21swapper/311:56:103
4066885992015,2cyclictest0-21swapper/312:00:163
406688599201,1cyclictest0-21swapper/311:30:553
406688599201,15cyclictest4143302-21turbostat08:20:003
406688599201,15cyclictest0-21swapper/309:20:213
406688599201,14cyclictest187955-21chrt11:11:113
406688599201,14cyclictest146085-21chrt10:45:023
4066885992010,6cyclictest66550irq/131-enp1s0-TxRx-310:36:143
3002952209,1sleep1171rcu_preempt12:24:311
3002952209,1sleep1171rcu_preempt12:24:311
1281342204,10sleep1128231-21ldconfig10:30:211
406688599192,12cyclictest84915-21taskset10:02:563
406688599192,12cyclictest60718-21taskset09:45:483
406688599192,11cyclictest220274-21taskset11:35:083
4066885991916,1cyclictest0-21swapper/310:55:173
4066885991916,1cyclictest0-21swapper/309:00:183
4066885991916,1cyclictest0-21swapper/307:30:213
4066885991915,2cyclictest0-21swapper/307:40:203
4066885991913,3cyclictest0-21swapper/309:10:173
406688599191,14cyclictest38133-21chrt09:32:063
406688599191,14cyclictest0-21swapper/309:40:153
2518072191,9sleep1251817-21ssh11:54:401
1310812191,6sleep02144-21diskmemload10:34:370
406688599182,12cyclictest195167-21taskset11:15:563
406688599182,12cyclictest173382-21taskset11:01:253
406688599182,11cyclictest93969-21taskset10:10:063
4066885991815,1cyclictest0-21swapper/307:20:223
4066885991814,2cyclictest0-21swapper/308:05:013
4066885991814,2cyclictest0-21swapper/307:15:163
4066885991813,3cyclictest0-21swapper/312:05:173
406688599181,15cyclictest210701-21chrt11:26:433
4066885991811,4cyclictest312825-21taskset12:30:173
406688599181,13cyclictest0-21swapper/312:22:043
406688599181,13cyclictest0-21swapper/312:22:043
4066885991714,1cyclictest0-21swapper/307:10:163
4066885991713,2cyclictest0-21swapper/308:10:233
406688599171,12cyclictest0-21swapper/308:05:133
406688599171,10cyclictest4158206-21taskset08:30:213
406688599171,10cyclictest4110645-21taskset07:45:193
2985922171,3sleep00-21swapper/012:21:410
2985922171,3sleep00-21swapper/012:21:410
2910052171,3sleep12144-21diskmemload12:17:251
2758712171,3sleep00-21swapper/012:08:580
1967982171,2sleep10-21swapper/111:19:231
1813412171,2sleep00-21swapper/011:07:520
1657882171,4sleep12144-21diskmemload10:55:351
1453932171,2sleep00-21swapper/010:43:430
926672161,2sleep20-21swapper/210:08:102
676562161,2sleep10-21swapper/109:50:251
557092161,9sleep22144-21diskmemload09:45:082
41634362161,5sleep14163483-21ntpq08:35:211
4066885991613,1cyclictest0-21swapper/310:40:123
406688599161,13cyclictest0-21swapper/308:40:203
406688599161,12cyclictest0-21swapper/307:50:163
406688599161,11cyclictest4179292-21taskset08:50:203
406688599161,11cyclictest4122191-21turbostat07:59:593
406688599160,13cyclictest0-21swapper/308:55:133
406688599160,12cyclictest0-21swapper/308:20:123
383742161,2sleep00-21swapper/009:32:320
28082161,2sleep03027-21ssh09:10:150
2666402161,1sleep20-21swapper/212:02:032
2422332161,2sleep20-21swapper/211:47:442
1958612161,3sleep02144-21diskmemload11:17:310
1604222161,2sleep20-21swapper/210:54:362
1080522161,4sleep02144-21diskmemload10:18:460
480752151,10sleep248252-21expr09:40:112
480452152,6sleep148071-21apt-get09:40:101
41159962151,10sleep04116120-21/usr/sbin/munin07:50:230
406688599151,9cyclictest4174667-21taskset08:46:203
4066885991512,1cyclictest0-21swapper/308:25:183
406688599151,10cyclictest4088889-21chrt07:25:203
3227392150,1sleep00-21swapper/012:36:320
3069622150,3sleep00-21swapper/012:27:020
2894652155,6sleep0289743-21sed12:15:280
2499552151,7sleep02144-21diskmemload11:51:540
2334432151,8sleep0233445-21sh11:42:040
2260812151,3sleep22144-21diskmemload11:37:252
2119292151,8sleep22144-21diskmemload11:29:022
2103332151,9sleep0210343-21ssh11:26:050
934552141,7sleep12144-21diskmemload10:09:371
767542141,9sleep276764-21ssh09:57:192
619682141,2sleep10-21swapper/109:48:281
40670612141,9sleep04067077-21awk07:10:110
406688599141,9cyclictest4159673-21taskset08:35:143
369212141,2sleep22144-21diskmemload09:30:252
31052141,8sleep24469-21cut09:10:172
299022141,7sleep22144-21diskmemload09:25:512
1504022141,6sleep0150457-21ntp_states10:45:240
1373602140,2sleep00-21swapper/010:37:260
1160972141,7sleep12144-21diskmemload10:24:351
104482141,5sleep211109-21kthreadcore09:15:182
41330292131,6sleep24133051-21sshd08:10:132
2984422131,8sleep22144-21diskmemload12:21:272
2984422131,8sleep22144-21diskmemload12:21:262
2691642131,6sleep22144-21diskmemload12:05:152
2684672136,2sleep0221rcuc/012:04:560
2680932131,5sleep10-21swapper/112:04:221
2199412131,8sleep2220193-21cut11:35:012
2191542131,6sleep12144-21diskmemload11:33:271
2111442131,6sleep12144-21diskmemload11:27:341
1679392133,2sleep2171rcu_preempt11:00:012
1606172131,8sleep0160627-21ssh10:54:580
148412131,7sleep12144-21diskmemload09:15:481
1140082131,3sleep2114014-21ssh10:20:402
934452121,7sleep093454-21ssh10:09:360
771872120,1sleep10-21swapper/109:58:061
452522120,2sleep20-21swapper/209:36:052
41901242121,7sleep14190338-21grep09:00:261
41221802121,7sleep04122421-21sort08:00:010
41171582121,7sleep14117220-21cpuspeed_turbos07:55:131
402922127,2sleep10-21swapper/109:35:131
3134522121,7sleep02144-21diskmemload12:30:230
2841472121,4sleep22144-21diskmemload12:14:212
2582992121,4sleep12144-21diskmemload11:56:451
232242121,3sleep023230-21ssh09:22:010
1981132121,7sleep2198794-21cut11:20:172
1735122121,4sleep12144-21diskmemload11:01:401
1534922121,4sleep12144-21diskmemload10:50:141
150412121,4sleep02144-21diskmemload09:16:120
1381502121,5sleep20-21swapper/210:38:522
1379582121,5sleep12144-21diskmemload10:38:311
1160932121,7sleep014-21ksoftirqd/010:24:350
988972111,3sleep10-21swapper/110:10:371
858202111,3sleep10-21swapper/110:04:361
847602111,4sleep02144-21diskmemload10:02:420
762612111,4sleep02144-21diskmemload09:56:210
75432111,4sleep12144-21diskmemload09:10:521
673282111,4sleep067375-21ntpq09:50:200
480322111,4sleep02144-21diskmemload09:40:090
458332111,4sleep02144-21diskmemload09:37:090
41915572111,5sleep24191646-21irqstats09:05:162
41494172111,6sleep04151054-21kthreadcore08:25:180
41172712111,5sleep24117427-21diskstats07:55:142
41008492111,5sleep04100797-31find07:40:010
41008462111,5sleep24100797-31find07:39:582
40751572113,4sleep24076838-21tr07:15:182
40711022111,5sleep14071150-21ntpq07:10:201
395992111,3sleep10-21swapper/109:34:491
3141722111,4sleep20-21swapper/212:30:452
3090712111,4sleep12144-21diskmemload12:30:041
3070672111,4sleep22144-21diskmemload12:27:122
3061602111,3sleep10-21swapper/112:25:311
2900212111,4sleep22144-21diskmemload12:15:542
2836812111,3sleep02144-21diskmemload12:13:390
2832182110,1sleep10-21swapper/112:12:551
2740842110,3sleep12144-21diskmemload12:06:071
2588322110,1sleep00-21swapper/011:57:360
2585182111,4sleep22144-21diskmemload11:57:062
2482662111,4sleep20-21swapper/211:50:202
2365322112,5sleep0237672-21kthreadcore11:45:170
234882111,4sleep12144-21diskmemload09:22:301
2330822111,3sleep10-21swapper/111:41:291
231622111,3sleep20-21swapper/209:21:552
2257502111,4sleep12144-21diskmemload11:36:521
2195762111,2sleep00-21swapper/011:34:190
2036782111,3sleep00-21swapper/011:23:000
2034972111,4sleep12144-21diskmemload11:22:411
1900712111,3sleep20-21swapper/211:15:082
1887902111,1sleep00-21swapper/011:12:540
1882902111,4sleep22144-21diskmemload11:11:482
1802412111,4sleep12144-21diskmemload11:05:531
1682652111,4sleep22144-21diskmemload11:00:092
1653832111,4sleep02144-21diskmemload10:55:260
1517632111,4sleep22144-21diskmemload10:46:582
1514142111,3sleep12144-21diskmemload10:46:171
1446692111,3sleep10-21swapper/110:42:201
1440012111,3sleep20-21swapper/210:41:052
1247242111,6sleep2126161-21kthreadcore10:30:172
1234932111,4sleep22144-21diskmemload10:29:342
1066672111,3sleep10-21swapper/110:16:021
1000532111,4sleep02144-21diskmemload10:12:480
1000312111,3sleep20-21swapper/210:12:472
850882100,3sleep20-21swapper/210:03:162
676932100,2sleep20-21swapper/209:50:262
41804662101,4sleep14180485-21apt-get08:55:131
41699922101,4sleep04170211-21diskstats08:45:140
41648622101,4sleep24164985-21diskstats08:40:142
41593782101,4sleep04159397-21apt-get08:35:120
41541042101,3sleep10-21swapper/108:30:121
41380402101,4sleep14138088-21munin-run08:15:011
41318252101,4sleep04131879-21ntpq08:05:220
41227592101,4sleep24122973-21smartctl08:00:152
41068742102,4sleep04106949-21fschecks_time07:45:140
41011692101,4sleep24101421-21df07:40:152
41011202100,4sleep14100755-31sort07:40:021
40951942101,5sleep04095202-21apt-get07:35:000
40942202101,4sleep24094269-21ntpq07:30:212
3239662101,3sleep12144-21diskmemload12:38:381
297002101,3sleep10-21swapper/109:25:311
2278892101,3sleep2227916-21apt-get11:40:142
2252452101,3sleep00-21swapper/011:35:430
1876252101,3sleep10-21swapper/111:10:381
1801342100,3sleep20-21swapper/211:05:422
1731552100,4sleep02144-21diskmemload11:00:470
1226762101,3sleep12144-21diskmemload10:27:411
1015722101,4sleep2101602-21ssh10:15:112
61708291,2sleep00-21swapper/009:47:590
4180158290,1sleep00-21swapper/008:54:280
4173911291,4sleep24173959-21ntpq08:45:202
4154126291,3sleep24154378-21df08:30:122
4142092291,4sleep14142232-21netstat08:15:211
4140371291,4sleep04142091-21sed08:15:200
4110593291,3sleep24110643-21ntpq07:45:192
4089238291,4sleep14089486-21awk07:25:261
4086238291,4sleep04087988-21grep07:25:180
4083961291,4sleep14084014-21cat07:20:301
24772290,1sleep00-21swapper/009:25:020
123077291,2sleep00-21swapper/010:28:480
4189657281,3sleep24189797-21ntp_192.168.11409:00:192
4132767281,3sleep24132782-21latency_hist08:10:012
4124659281,3sleep14126335-21sshd08:00:181
4079625280,1sleep20-21swapper/207:20:072
4074859281,3sleep14075156-21kthreadcore07:15:161
4190651270,1sleep00-21swapper/009:02:570
4179969271,3sleep2802-21snmpd08:52:052
4175213271,2sleep10-21swapper/108:50:121
4170212271,3sleep10-21swapper/108:45:151
4164694270,2sleep00-21swapper/008:40:130
4154432271,3sleep00-21swapper/008:30:150
4148200270,1sleep10-21swapper/108:20:331
4131881271,3sleep10-21swapper/108:05:231
4126780270,1sleep00-21swapper/008:00:250
4111863270,2sleep10-21swapper/107:50:121
4100482270,1sleep00-21swapper/007:36:400
4095731270,2sleep10-21swapper/107:35:131
4068676271,2sleep20-21swapper/207:10:172
4192791261,2sleep00-21swapper/009:05:170
4185244260,1sleep00-21swapper/008:56:280
4181388260,2sleep20-21swapper/208:55:182
4165269261,2sleep10-21swapper/108:40:181
4153158261,2sleep10-21swapper/108:25:271
4149106260,2sleep24149320-21smartctl08:25:162
4148272260,4sleep00-21swapper/008:21:290
4138301260,2sleep20-21swapper/208:15:142
4112447261,2sleep24072170-21kworker/2:2+events07:50:172
4110458261,1sleep10-21swapper/107:45:191
4090153260,2sleep10-21swapper/107:30:121
4085294260,2sleep20-21swapper/207:25:162
4080212261,2sleep00-21swapper/007:20:210
4079006260,1sleep00-21swapper/007:15:400
4191366250,1sleep10-21swapper/109:05:151
4163485250,1sleep20-21swapper/208:35:222
4143851250,1sleep20-21swapper/208:20:152
4133308240,1sleep00-21swapper/008:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional