You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 12:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot6s.osadl.org (updated Wed Dec 03, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22272212299,18sleep00-21swapper/019:09:430
22055211895,18sleep30-21swapper/319:06:563
22059211698,12sleep20-21swapper/219:07:002
22257211393,14sleep10-21swapper/119:09:321
80721040,0sleep10-21swapper/122:13:001
263462890,4sleep02232199cyclictest23:15:120
126572880,1sleep312661-21latency22:25:163
161462820,0sleep10-21swapper/121:55:121
100052760,0sleep30-21swapper/321:13:453
249442730,1sleep011-21rcuc/023:47:390
149262730,0sleep00-21swapper/021:19:150
23652670,0sleep10-21swapper/100:32:491
298122580,1sleep20-21swapper/221:35:122
2232299267,18cyclictest0-21swapper/120:25:131
2232299261,23cyclictest26812-21http00:25:001
48762250,1sleep0101ktimersoftd/023:25:330
2232499230,21cyclictest0-21swapper/300:25:013
321802220,0sleep30-21swapper/321:36:583
2232499221,19cyclictest0-21swapper/322:45:103
2232299223,2cyclictest332250irq/24-eth121:51:051
2232299210,17cyclictest0-21swapper/121:46:041
2232299202,2cyclictest0-21swapper/121:12:011
187432200,1sleep30-21swapper/321:57:103
5932190,1sleep121-21rcuc/121:38:231
2232499191,2cyclictest0-21swapper/320:50:143
2232299192,2cyclictest0-21swapper/122:40:061
22322991916,2cyclictest8135-21kworker/u8:123:34:051
22322991916,2cyclictest7898-21kworker/u8:020:15:041
22322991916,2cyclictest7898-21kworker/u8:020:15:031
22322991916,2cyclictest7898-21kworker/u8:020:12:011
22322991916,2cyclictest6718-21kworker/u8:222:57:051
22322991916,2cyclictest5069-21kworker/u8:019:23:031
22322991916,2cyclictest303-21kworker/u8:019:41:041
22322991916,2cyclictest303-21kworker/u8:019:39:041
22322991916,2cyclictest28656-21kworker/u8:219:46:001
22322991916,2cyclictest20880-21kworker/u8:122:19:041
2232299190,2cyclictest0-21swapper/100:38:051
2232499180,16cyclictest0-21swapper/321:20:133
22322991815,2cyclictest7447-21kworker/u8:200:08:011
22322991815,2cyclictest4248-21kworker/u8:119:56:011
22322991815,2cyclictest28656-21kworker/u8:219:50:041
22322991815,2cyclictest28656-21kworker/u8:219:33:041
22322991815,2cyclictest28621-21kworker/u8:120:53:041
22322991815,2cyclictest26231-21kworker/u8:119:28:011
22322991815,2cyclictest24539-21kworker/u8:020:41:041
22322991815,2cyclictest24378-21kworker/u8:023:13:051
22322991815,2cyclictest21986-21kworker/u8:220:49:041
22322991815,2cyclictest21138-21kworker/u8:223:55:041
22322991815,2cyclictest21138-21kworker/u8:223:52:041
22322991815,2cyclictest20880-21kworker/u8:122:23:041
22322991815,2cyclictest15125-21kworker/u8:023:39:041
22322991815,2cyclictest13833-21kworker/u8:021:19:041
22322991815,2cyclictest12181-21kworker/u8:123:03:041
2232299180,2cyclictest0-21swapper/100:01:041
22323991714,2cyclictest14460-21kworker/u8:021:53:042
22322991714,2cyclictest6718-21kworker/u8:222:54:041
22322991714,2cyclictest24539-21kworker/u8:020:37:031
22322991714,2cyclictest24378-21kworker/u8:023:19:051
22322991714,2cyclictest21986-21kworker/u8:220:59:041
22322991714,2cyclictest20880-21kworker/u8:122:49:051
22322991714,2cyclictest20880-21kworker/u8:122:30:051
22322991714,2cyclictest20880-21kworker/u8:122:26:031
22322991714,2cyclictest19378-21kworker/u8:022:36:041
22322991714,2cyclictest10480-21kworker/u8:220:06:011
22322991711,1cyclictest0-21swapper/123:29:041
2232299170,11cyclictest0-21swapper/123:06:021
2232499161,13cyclictest0-21swapper/322:20:053
2232499160,14cyclictest1070-21ssh22:13:253
2232399161,3cyclictest22931-21sed23:45:172
2232399161,3cyclictest21853-21ssh00:19:372
2232399161,3cyclictest0-21swapper/223:23:112
22323991613,2cyclictest9595-21kworker/u8:321:48:052
22323991613,2cyclictest14460-21kworker/u8:022:01:042
22322991613,2cyclictest7898-21kworker/u8:020:24:041
22322991613,2cyclictest7898-21kworker/u8:020:24:041
22322991613,2cyclictest27464-21kworker/u8:021:34:011
22322991613,2cyclictest21986-21kworker/u8:221:01:041
22322991613,2cyclictest21986-21kworker/u8:220:33:041
22322991613,2cyclictest20022-21kworker/u8:121:26:011
22322991613,2cyclictest19360-21kworker/u8:000:23:051
22322991613,2cyclictest19360-21kworker/u8:000:18:061
296122150,3sleep029624-21rm22:43:580
22324991512,2cyclictest6154-21diskmemload00:23:353
22324991512,2cyclictest0-21swapper/322:51:353
22324991511,2cyclictest6154-21diskmemload22:03:253
2232499151,12cyclictest0-21swapper/321:45:193
22324991510,3cyclictest22519-21munin-node3
2232499150,2cyclictest0-21swapper/321:40:153
2232499150,14cyclictest0-21swapper/323:22:153
2232399151,3cyclictest28793-21ssh23:16:592
2232399151,3cyclictest24870-21ssh23:13:172
2232399150,13cyclictest29865-21sh22:44:222
22322991512,2cyclictest21138-21kworker/u8:223:45:051
22321991512,2cyclictest15125-21kworker/u8:023:38:060
2232499149,3cyclictest0-21swapper/300:10:163
2232499141,2cyclictest5829-21ssh00:01:183
2232499141,2cyclictest0-21swapper/322:36:293
22324991411,2cyclictest0-21swapper/323:52:153
22324991411,2cyclictest0-21swapper/300:33:153
22324991411,2cyclictest0-21swapper/300:05:053
2232499140,2cyclictest0-21swapper/323:37:543
2232499140,2cyclictest0-21swapper/323:15:153
2232499140,13cyclictest0-21swapper/322:07:353
2232499140,13cyclictest0-21swapper/320:30:063
2232499140,12cyclictest6154-21diskmemload00:16:413
22323991411,2cyclictest7447-21kworker/u8:200:04:042
22323991411,2cyclictest27464-21kworker/u8:021:32:062
22323991411,2cyclictest19905-21kworker/u8:119:10:042
22323991411,2cyclictest0-21swapper/223:53:452
22323991411,2cyclictest0-21swapper/222:45:252
22323991411,2cyclictest0-21swapper/222:05:392
22323991411,2cyclictest0-21swapper/200:38:002
22323991411,2cyclictest0-21swapper/200:28:442
22323991410,2cyclictest0-21swapper/219:45:122
2232399140,2cyclictest6154-21diskmemload23:29:062
2232399140,2cyclictest5129-21runrttasks23:38:052
2232399140,2cyclictest0-21swapper/223:30:242
2232399140,2cyclictest0-21swapper/219:35:132
2232399140,2cyclictest0-21swapper/200:20:552
2232399140,12cyclictest29045-21fschecks_count20:45:162
2232399140,12cyclictest0-21swapper/222:20:262
2232399140,12cyclictest0-21swapper/200:11:002
22322991411,2cyclictest9595-21kworker/u8:322:00:031
22322991411,2cyclictest1549-21kworker/u8:121:43:041
2232299140,2cyclictest0-21swapper/121:09:041
2232299140,2cyclictest0-21swapper/120:02:011
2232199140,12cyclictest0-21swapper/023:06:540
2232199140,12cyclictest0-21swapper/022:20:060
22324991310,2cyclictest6154-21diskmemload22:41:033
22324991310,2cyclictest6154-21diskmemload00:36:303
22324991310,2cyclictest0-21swapper/323:30:203
22324991310,2cyclictest0-21swapper/323:06:553
22324991310,2cyclictest0-21swapper/322:33:053
22324991310,2cyclictest0-21swapper/322:16:183
22324991310,2cyclictest0-21swapper/319:50:163
22324991310,2cyclictest0-21swapper/319:10:203
2232499130,2cyclictest0-21swapper/323:45:063
2232499130,2cyclictest0-21swapper/323:02:153
2232499130,2cyclictest0-21swapper/321:30:203
2232499130,2cyclictest0-21swapper/321:15:443
2232499130,2cyclictest0-21swapper/320:45:233
2232499130,2cyclictest0-21swapper/319:55:163
2232499130,12cyclictest5129-21runrttasks21:25:243
2232499130,12cyclictest0-21swapper/323:41:433
2232499130,12cyclictest0-21swapper/322:56:323
2232499130,12cyclictest0-21swapper/321:50:013
22323991310,2cyclictest6154-21diskmemload23:06:172
22323991310,2cyclictest6154-21diskmemload22:33:522
22323991310,2cyclictest4068-21ssh22:15:412
22323991310,2cyclictest0-21swapper/223:58:042
22323991310,2cyclictest0-21swapper/223:40:112
22323991310,2cyclictest0-21swapper/221:22:412
22323991310,2cyclictest0-21swapper/221:17:002
22323991310,2cyclictest0-21swapper/219:55:172
22323991310,2cyclictest0-21swapper/219:25:272
22323991310,2cyclictest0-21swapper/200:30:182
2232399130,2cyclictest0-21swapper/222:55:582
2232399130,2cyclictest0-21swapper/222:50:172
2232399130,2cyclictest0-21swapper/221:40:132
2232399130,2cyclictest0-21swapper/221:05:162
2232399130,2cyclictest0-21swapper/220:10:132
2232399130,12cyclictest0-21swapper/222:35:372
2232399130,12cyclictest0-21swapper/222:25:592
2232399130,12cyclictest0-21swapper/222:10:242
2232399130,12cyclictest0-21swapper/221:55:222
2232399130,12cyclictest0-21swapper/221:27:022
2232399130,12cyclictest0-21swapper/221:00:182
2232399130,12cyclictest0-21swapper/220:25:252
2232399130,12cyclictest0-21swapper/220:20:182
2232399130,12cyclictest0-21swapper/220:20:182
2232399130,12cyclictest0-21swapper/219:20:142
2232399130,12cyclictest0-21swapper/200:05:242
2232299130,2cyclictest0-21swapper/123:43:041
2232299130,2cyclictest0-21swapper/119:12:041
2232299130,2cyclictest0-21swapper/100:12:011
22321991310,2cyclictest19360-21kworker/u8:000:25:050
22321991310,2cyclictest0-21swapper/021:45:070
22321991310,2cyclictest0-21swapper/021:10:170
2232199130,2cyclictest6154-21diskmemload22:37:250
2232199130,2cyclictest0-21swapper/022:53:240
2232199130,12cyclictest0-21swapper/023:54:080
2232199130,12cyclictest0-21swapper/023:30:140
2232199130,12cyclictest0-21swapper/021:35:170
2232199130,12cyclictest0-21swapper/019:15:000
2232199130,11cyclictest0-21swapper/000:35:180
16052130,2sleep00-21swapper/022:14:200
2232499129,2cyclictest0-21swapper/323:55:233
2232499129,2cyclictest0-21swapper/323:27:433
2232499122,6cyclictest0-21swapper/320:40:233
2232499120,2cyclictest0-21swapper/319:25:273
2232499120,2cyclictest0-21swapper/319:20:183
2232499120,11cyclictest0-21swapper/320:55:233
2232499120,11cyclictest0-21swapper/320:20:273
2232499120,11cyclictest0-21swapper/320:20:263
2232499120,11cyclictest0-21swapper/320:00:253
2232499120,11cyclictest0-21swapper/319:42:083
2232499120,11cyclictest0-21swapper/319:30:133
2232399129,2cyclictest22349-21copy20:30:042
2232399129,2cyclictest0-21swapper/223:00:032
2232399129,2cyclictest0-21swapper/220:40:272
2232399120,2cyclictest0-21swapper/221:10:312
2232399120,11cyclictest0-21swapper/220:59:202
2232399120,11cyclictest0-21swapper/220:06:232
2232299129,2cyclictest5069-21kworker/u8:019:15:041
2232299129,2cyclictest20880-21kworker/u8:122:06:041
2232299120,2cyclictest0-21swapper/123:20:061
2232299120,11cyclictest0-21swapper/121:20:001
2232199129,2cyclictest0-21swapper/022:55:420
2232199129,2cyclictest0-21swapper/022:47:250
2232199129,2cyclictest0-21swapper/021:59:410
2232199129,2cyclictest0-21swapper/021:50:200
2232199129,2cyclictest0-21swapper/021:26:390
2232199129,2cyclictest0-21swapper/019:35:250
2232199129,2cyclictest0-21swapper/019:25:180
2232199129,2cyclictest0-21swapper/000:13:250
2232199120,2cyclictest6196-21ssh00:01:530
2232199120,2cyclictest0-21swapper/022:05:520
2232199120,2cyclictest0-21swapper/019:45:230
2232199120,11cyclictest0-21swapper/023:42:450
2232199120,11cyclictest0-21swapper/022:00:160
2232199120,11cyclictest0-21swapper/019:15:190
190472120,2sleep00-21swapper/022:32:000
2232499118,2cyclictest0-21swapper/321:05:273
2232499118,2cyclictest0-21swapper/320:25:263
2232499118,2cyclictest0-21swapper/320:10:153
2232499118,2cyclictest0-21swapper/320:05:223
2232499118,2cyclictest0-21swapper/319:47:203
2232499110,2cyclictest0-21swapper/320:15:223
2232499110,2cyclictest0-21swapper/320:15:223
2232399118,2cyclictest5069-21kworker/u8:019:18:042
2232399118,2cyclictest32254-21kworker/u8:020:54:042
2232399118,2cyclictest10480-21kworker/u8:220:01:042
2232399118,2cyclictest0-21swapper/220:39:192
2232399118,2cyclictest0-21swapper/219:30:262
2232199118,2cyclictest0-21swapper/023:20:260
2232199118,2cyclictest0-21swapper/000:20:510
2232199118,2cyclictest0-21swapper/000:07:380
2232199116,3cyclictest3416-21processes22:15:210
2232199110,2cyclictest0-21swapper/023:55:210
2232199110,2cyclictest0-21swapper/021:21:550
2232199110,2cyclictest0-21swapper/000:15:060
2232199110,10cyclictest0-21swapper/023:12:120
2232199110,10cyclictest0-21swapper/023:02:120
2232199110,10cyclictest0-21swapper/022:28:510
2232499102,6cyclictest0-21swapper/319:35:263
2232499102,6cyclictest0-21swapper/319:15:253
2232399107,2cyclictest303-21kworker/u8:019:40:042
2232399107,2cyclictest0-21swapper/219:50:262
2232399105,3cyclictest16928-21fschecks_time20:15:162
2232399105,3cyclictest16928-21fschecks_time20:15:152
2232199107,2cyclictest0-21swapper/021:00:240
2232199107,2cyclictest0-21swapper/020:10:240
2232199107,2cyclictest0-21swapper/019:30:250
2232199100,9cyclictest0-21swapper/021:31:250
2232199100,9cyclictest0-21swapper/000:34:250
2232199100,2cyclictest0-21swapper/021:40:220
223249994,3cyclictest25828-21timerwakeupswit20:35:283
223249994,3cyclictest2442-21grep21:00:153
223219996,2cyclictest673-21fschecks_time20:55:140
223219996,2cyclictest0-21swapper/020:45:280
223219996,2cyclictest0-21swapper/020:25:150
223219995,3cyclictest24461-21latency_hist20:35:010
223219994,3cyclictest13441-21pzem_power20:05:220
223219991,6cyclictest16702-21grep20:15:150
223219991,6cyclictest16702-21grep20:15:150
223219985,2cyclictest0-21swapper/019:50:140
223219985,2cyclictest0-21swapper/019:20:150
223219984,3cyclictest11000-21http_loadtime20:00:160
223219983,3cyclictest9270-21ntp_offset19:55:180
223219983,3cyclictest4305-21tail21:05:140
223219983,3cyclictest28523-21cut20:45:010
223219983,3cyclictest2842-21interrupts19:40:170
223219974,2cyclictest28621-21kworker/u8:120:50:140
223219973,3cyclictest25089-21grep20:35:160
223219970,6cyclictest0-21swapper/020:22:040
223219970,6cyclictest0-21swapper/020:22:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional