You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Fri May 02, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4146843211557,52sleep30-21swapper/319:05:263
414726228466,12sleep10-21swapper/119:09:291
414725128062,13sleep00-21swapper/019:09:200
414700227862,10sleep20-21swapper/219:06:092
4147457993833,2cyclictest0-21swapper/323:00:003
4147455993732,3cyclictest87618-21pmu-power22:35:202
4147455993026,2cyclictest4152461-21cut19:20:142
4147457992924,2cyclictest0-21swapper/323:45:263
4147455992924,3cyclictest4188253-21cpuspeed_turbos20:50:152
4147455992924,3cyclictest195356-21cpuspeed_turbos00:25:132
4147455992924,3cyclictest1-21systemd00:03:472
4147455992817,8cyclictest7357-21cut21:15:142
4147455992813,12cyclictest1809-21diskmemload00:06:122
4147455992724,2cyclictest4147447-21cyclictest20:40:012
4147455992723,3cyclictest4190249-21cpuspeed_turbos20:55:172
4147455992722,3cyclictest96716-21cpuspeed_turbos22:45:122
4147455992722,3cyclictest706-21gpm23:46:542
4147455992718,6cyclictest57761-21cut22:05:232
4147455992621,3cyclictest4172287-21fschecks_count20:10:172
4147455992621,3cyclictest0-21swapper/200:15:002
4147455992616,6cyclictest92364-21netstat22:40:172
4147455992615,8cyclictest944-21snmpd22:15:082
4147457992522,1cyclictest0-21swapper/323:01:483
4147455992520,3cyclictest51473-21munin-run22:00:012
4147455992513,8cyclictest2766-21sed21:10:182
4147455992511,12cyclictest0-21swapper/222:20:182
4147455992420,2cyclictest0-21swapper/223:40:222
4147455992419,3cyclictest83112-21unixbench_singl22:30:262
4147455992419,3cyclictest147106-21head23:35:172
4147455992419,2cyclictest0-21swapper/219:10:142
4147455992411,9cyclictest41220-21sshd21:49:322
4147457992320,1cyclictest0-21swapper/321:41:483
4147457992320,1cyclictest0-21swapper/300:31:483
4147457992319,2cyclictest0-21swapper/323:10:243
4147457992318,2cyclictest0-21swapper/322:50:233
414745599231,8cyclictest34363-21sh21:41:432
4147455992318,3cyclictest27782-21ntp_kernel_err21:35:212
4147455992318,3cyclictest186704-21fschecks_time00:15:142
414745599231,7cyclictest4181563-21runrttasks20:33:332
414745599231,7cyclictest4181563-21runrttasks20:33:322
414745599231,19cyclictest64458-21kworker/u16:1+rpciod23:24:582
414745599231,18cyclictest45661-21sh21:53:452
414745599230,20cyclictest52269-21runrttasks22:00:142
1143982230,2sleep00-21swapper/023:02:120
4147457992219,1cyclictest0-21swapper/322:49:473
4147457992219,1cyclictest0-21swapper/322:05:273
4147457992219,1cyclictest0-21swapper/321:37:483
4147457992219,1cyclictest0-21swapper/321:33:483
4147455992218,2cyclictest4162658-21ntp_kernel_pll_19:45:222
4147455992218,2cyclictest0-21swapper/222:30:002
4147455992218,2cyclictest0-21swapper/222:29:592
4147455992218,2cyclictest0-21swapper/220:10:002
414745599221,7cyclictest63237-21sh22:10:252
414745599221,7cyclictest19115-21runrttasks21:26:192
414745599221,7cyclictest19115-21runrttasks21:26:192
4147455992217,3cyclictest4192460-21gltestperf21:00:182
4147455992217,3cyclictest203464-21latency_hist00:35:002
4147455992217,3cyclictest107979-21threads22:55:242
4147455992216,4cyclictest0-21swapper/223:15:002
4147455992212,7cyclictest25518-21runrttasks21:33:242
4147455992212,6cyclictest192700-21sh00:21:012
414745599221,18cyclictest941-21runrttasks00:37:362
4147455992211,8cyclictest142258-21ntpq23:30:212
428472212,13sleep142937-21/usr/sbin/munin21:50:211
414745799219,8cyclictest179809-21taskset00:08:593
4147457992117,2cyclictest0-21swapper/321:24:593
4147457992116,2cyclictest0-21swapper/322:20:003
414745599213,4cyclictest12844-21strings21:20:232
414745599212,16cyclictest161283-21ssh23:50:152
414745599212,15cyclictest136585-21diskstats23:25:142
414745599211,9cyclictest106438-21awk22:55:012
4147455992117,2cyclictest4157743-21idleruntime-cro19:35:002
4147455992117,2cyclictest4157743-21idleruntime-cro19:34:592
414745599211,6cyclictest130546-21ssh23:19:122
4147455992110,7cyclictest112324-21wc23:00:172
4147457992017,1cyclictest0-21swapper/319:51:483
4147457992016,2cyclictest0-21swapper/323:15:253
4147457992012,4cyclictest1842242sleep300:12:483
414745799201,15cyclictest168560-21taskset23:56:523
414745799201,14cyclictest142329-21chrt23:30:223
414745599201,6cyclictest4155700-21sshd19:29:302
4147455992016,2cyclictest4186108-21apt-key20:45:002
4147455992016,2cyclictest4175763-21cron20:19:592
4147455992014,3cyclictest4168393-21sort20:00:172
4147455992012,5cyclictest4157968-21/usr/sbin/munin19:35:172
4147455992011,6cyclictest119872-21ssh23:08:002
414745599200,17cyclictest4166824-21ldconfig19:55:202
130352201,13sleep013134-21ps21:20:260
4147457991916,1cyclictest0-21swapper/320:45:003
4147457991916,1cyclictest0-21swapper/320:11:483
4147457991915,2cyclictest0-21swapper/322:35:583
4147457991913,2cyclictest0-21swapper/300:16:513
414745799190,1cyclictest0-21swapper/300:20:163
414745799190,16cyclictest0-21swapper/323:09:153
414745799190,16cyclictest0-21swapper/322:40:533
414745599199,6cyclictest4186859-21ntp_192.168.11420:45:222
414745599196,8cyclictest700-21dbus-daemon20:29:592
414745599194,2cyclictest0-21swapper/219:50:182
414745599191,5cyclictest166343-21fschecks_count23:55:132
4147455991915,2cyclictest0-21swapper/219:45:002
414745599190,16cyclictest4177795-21munin-run20:25:002
542442180,2sleep30-21swapper/322:01:513
484342180,2sleep11809-21diskmemload21:55:351
4147457991815,1cyclictest0-21swapper/321:55:153
4147457991815,1cyclictest0-21swapper/321:55:013
4147457991815,1cyclictest0-21swapper/300:37:203
4147457991815,1cyclictest0-21swapper/300:27:283
4147457991814,2cyclictest0-21swapper/323:50:213
4147457991814,2cyclictest0-21swapper/323:42:423
4147457991814,2cyclictest0-21swapper/321:00:143
4147457991814,2cyclictest0-21swapper/320:46:563
4147457991814,2cyclictest0-21swapper/300:05:003
4147457991811,4cyclictest736852sleep322:21:073
414745799181,13cyclictest8257-21chrt21:15:233
414745799181,13cyclictest19106-21chrt21:26:183
414745799181,13cyclictest19106-21chrt21:26:173
414745799181,13cyclictest0-21swapper/322:25:213
414745799181,13cyclictest0-21swapper/322:25:213
414745799181,12cyclictest81576-21chrt22:30:043
414745799180,14cyclictest82250irq/126-enp1s0-TxRx-323:25:173
414745599181,3cyclictest4150261-21/usr/sbin/munin19:15:142
414745799173,10cyclictest36979-21chrt21:45:103
4147457991714,1cyclictest0-21swapper/321:07:253
4147457991714,1cyclictest0-21swapper/320:53:043
4147457991714,1cyclictest0-21swapper/320:30:333
4147457991714,1cyclictest0-21swapper/320:30:323
4147457991714,1cyclictest0-21swapper/320:29:313
4147457991714,1cyclictest0-21swapper/320:16:473
4147457991714,1cyclictest0-21swapper/320:00:073
4147457991714,1cyclictest0-21swapper/319:41:243
4147457991714,1cyclictest0-21swapper/319:39:213
4147457991714,1cyclictest0-21swapper/319:27:043
4147457991714,1cyclictest0-21swapper/319:22:583
4147457991714,1cyclictest0-21swapper/319:15:023
4147457991713,2cyclictest0-21swapper/322:10:103
414745799171,13cyclictest0-21swapper/323:35:123
414745599172,13cyclictest1411-21grep21:05:262
812062161,10sleep181220-21cat22:29:551
812062161,10sleep181220-21cat22:29:541
414745799162,2cyclictest0-21swapper/323:21:223
4147457991613,1cyclictest0-21swapper/319:55:243
4147457991612,2cyclictest0-21swapper/320:05:153
414745799161,13cyclictest0-21swapper/321:13:473
414745799160,13cyclictest0-21swapper/320:20:223
1306942161,10sleep11809-21diskmemload23:19:261
869462151,9sleep01809-21diskmemload22:35:150
414745799151,10cyclictest4147628-21chrt19:10:143
1798302151,4sleep11809-21diskmemload00:09:021
1750132150,2sleep00-21swapper/000:04:110
1640042151,4sleep0164006-21ssh23:52:240
1612292151,7sleep11809-21diskmemload23:50:121
1569052150,2sleep00-21swapper/023:45:180
1215312151,7sleep016-21ksoftirqd/023:10:110
1043772151,9sleep11809-21diskmemload22:51:561
61802141,8sleep11809-21diskmemload21:14:101
591272141,8sleep11809-21diskmemload22:06:421
536302141,7sleep11809-21diskmemload22:00:561
447662141,4sleep044768-21ssh21:52:250
414745799141,9cyclictest4182708-21chrt20:35:193
414745799141,2cyclictest0-21swapper/320:55:163
224752145,6sleep0171ktimers/021:30:200
1409772141,7sleep11809-21diskmemload23:29:531
1214562141,7sleep11809-21diskmemload23:10:041
936492131,8sleep093664-21ssh22:41:130
862452131,8sleep086255-21ssh22:34:590
598022131,8sleep059814-21ssh22:07:410
41586722131,8sleep14158720-21ntpq19:35:171
414745799132,8cyclictest4161921-21chrt19:45:133
414745799131,8cyclictest4155996-21chrt19:30:063
414745799131,8cyclictest4155996-21chrt19:30:063
34052130,8sleep03666-21sed21:10:280
1597992131,7sleep11809-21diskmemload23:48:291
1338302131,3sleep0133840-21ssh23:21:450
1281692131,8sleep0128180-21ssh23:15:410
871252121,5sleep187247-21memory22:35:151
548392121,4sleep00-21swapper/022:02:440
478492120,2sleep00-21swapper/021:55:220
41768082121,7sleep14176856-21ntpq20:20:231
41477782121,8sleep04147864-21grep19:10:150
1986132120,1sleep10-21swapper/100:29:181
1879492121,8sleep0187963-21ssh00:15:340
1793342123,2sleep0191rcu_preempt00:08:160
1531092121,6sleep11809-21diskmemload23:40:461
1449422121,5sleep11809-21diskmemload23:33:261
1051082121,2sleep00-21swapper/022:53:030
746582110,1sleep00-21swapper/022:22:320
633472111,4sleep11809-21diskmemload22:10:341
41847372110,7sleep04183930-21/usr/sbin/munin20:40:260
41819602111,5sleep04182103-21df20:35:130
41626682111,5sleep14162716-21ntpq19:45:231
387772111,4sleep11809-21diskmemload21:45:531
287502110,2sleep10-21swapper/121:35:511
1999962111,5sleep1200080-21ntpq00:30:171
1952352111,5sleep0195261-21apt-get00:25:120
1947952111,7sleep10-21swapper/100:24:521
1855132111,4sleep11809-21diskmemload00:14:381
1749442111,4sleep11809-21diskmemload00:04:031
1704702111,6sleep1170480-21rm23:59:441
1413552111,6sleep0141365-21ssh23:30:030
1355632111,4sleep11809-21diskmemload23:24:241
1139802111,4sleep11809-21diskmemload23:01:341
1097172110,5sleep10-21swapper/122:57:361
991612101,4sleep11809-21diskmemload22:46:581
947802101,3sleep10-21swapper/122:42:541
842962101,3sleep11809-21diskmemload22:32:081
805862100,4sleep00-21swapper/022:28:580
805862100,4sleep00-21swapper/022:28:580
722222101,3sleep11809-21diskmemload22:20:161
700752101,4sleep11809-21diskmemload22:18:081
675012101,4sleep067549-21ntpq22:15:190
634692100,3sleep00-21swapper/022:10:460
41907562101,4sleep14190804-21ntpq20:55:221
41681132101,5sleep04168114-21/usr/sbin/munin20:00:140
337732101,4sleep11809-21diskmemload21:40:511
2067222101,3sleep00-21swapper/000:38:350
2001982101,5sleep0200237-21ssh00:30:210
1863672101,2sleep10-21swapper/100:15:071
175512101,4sleep017680-21memory21:25:200
175512101,4sleep017680-21memory21:25:190
1679352101,3sleep01809-21diskmemload23:55:580
144502101,3sleep11809-21diskmemload21:21:441
1188842101,3sleep11809-21diskmemload23:06:311
99871291,4sleep099883-21ssh22:48:020
9304291,2sleep00-21swapper/021:16:280
8169290,3sleep11809-21diskmemload21:15:221
4182602291,4sleep14182658-21ld20:35:181
4174760291,4sleep14174808-21ntpq20:15:211
4170127291,4sleep04170128-21/usr/sbin/munin20:05:150
4156736291,4sleep04156777-21ntpq19:30:200
4156736291,4sleep04156777-21ntpq19:30:190
38993291,2sleep00-21swapper/021:46:120
34860291,4sleep034872-21ssh21:42:260
29140290,3sleep00-21swapper/021:36:260
205378291,3sleep11809-21diskmemload00:36:041
193035291,2sleep00-21swapper/000:21:390
181159291,3sleep0181184-21apt-get00:10:130
152395290,3sleep00-21swapper/023:40:210
148180290,2sleep00-21swapper/023:35:530
148101290,1sleep10-21swapper/123:35:461
107863291,3sleep01809-21diskmemload22:55:240
669281,3sleep0771-21ntpq21:05:190
4191870280,1sleep00-21swapper/021:00:020
4186904281,3sleep14186952-21ntpq20:45:241
4177549281,4sleep04177550-21sshd20:22:300
4163268281,4sleep04163269-21sshd19:45:310
4155461281,4sleep04143281-21kworker/0:2+events19:26:270
4153654280,1sleep10-21swapper/119:23:261
4152098280,1sleep00-21swapper/019:20:130
4149555281,3sleep10-21swapper/119:15:021
25573281,2sleep10-21swapper/121:33:291
20033281,3sleep120043-21ssh21:27:411
20033281,3sleep120043-21ssh21:27:411
138261280,1sleep00-21swapper/023:25:590
118363280,2sleep00-21swapper/023:05:460
4192789271,3sleep14192837-21ntpq21:00:231
4190917270,1sleep00-21swapper/020:55:250
4189486270,1sleep00-21swapper/020:50:360
4179974270,2sleep10-21swapper/120:30:131
4179974270,2sleep10-21swapper/120:30:121
4179628270,1sleep00-21swapper/020:29:000
4168115270,2sleep10-21swapper/120:00:151
4151599270,1sleep00-21swapper/019:17:330
4188797261,2sleep14188854-21modprobe20:50:211
4187527260,1sleep00-21swapper/020:45:390
4181544260,1sleep00-21swapper/020:33:210
4181544260,1sleep00-21swapper/020:33:210
4174127260,3sleep00-21swapper/020:15:170
4173664260,1sleep00-21swapper/020:13:240
4166119260,4sleep00-21swapper/019:55:080
4164610260,2sleep10-21swapper/119:50:211
4164567261,1sleep00-21swapper/019:50:200
4161509260,1sleep00-21swapper/019:43:060
4159563260,1sleep00-21swapper/019:38:370
4156004260,2sleep10-21swapper/119:30:131
4156004260,2sleep10-21swapper/119:30:121
4154772260,2sleep10-21swapper/119:25:201
4184552250,2sleep10-21swapper/120:40:231
4172750250,1sleep10-21swapper/120:10:211
4171146250,2sleep10-21swapper/120:05:271
4166307250,1sleep10-21swapper/119:55:171
4159964250,2sleep10-21swapper/119:40:191
1679250,2sleep10-21swapper/121:08:331
4178174240,1sleep10-21swapper/120:25:161
4147865240,1sleep10-21swapper/119:10:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional