You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 00:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot6s.osadl.org (updated Sun May 04, 2025 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16614211596,14sleep30-21swapper/307:06:433
16574211592,18sleep00-21swapper/007:06:110
16767210897,6sleep10-21swapper/107:08:411
16618210796,6sleep20-21swapper/207:06:462
138282990,0sleep00-21swapper/011:34:010
221012730,1sleep00-21swapper/008:40:170
13592710,1sleep11358-21if_err_eth010:15:181
16899993817,19cyclictest0-21swapper/309:10:153
223632350,1sleep20-21swapper/212:15:162
199872240,1sleep30-21swapper/307:15:193
1689699241,18cyclictest1954-21ssh11:55:130
1689899220,20cyclictest0-21swapper/209:40:162
41102210,1sleep24112-21ssh09:12:332
35542210,1sleep1221ktimersoftd/107:55:161
16898992118,2cyclictest11917-21kworker/u8:212:31:062
1689899211,2cyclictest0-21swapper/211:20:122
1689899210,19cyclictest0-21swapper/211:55:122
1689899210,16cyclictest0-21swapper/210:12:052
150122210,1sleep1221ktimersoftd/109:23:541
16898992017,2cyclictest23482-21kworker/u8:109:27:042
16898992017,2cyclictest12294-21kworker/u8:111:34:042
16898992017,2cyclictest11917-21kworker/u8:212:06:052
1689899201,18cyclictest0-21swapper/212:14:042
1689899200,15cyclictest0-21swapper/212:02:042
1689799200,18cyclictest0-21swapper/111:11:041
16898991916,2cyclictest2728-21kworker/u8:009:17:042
16898991916,2cyclictest26600-21kworker/u8:011:47:042
16898991916,2cyclictest23482-21kworker/u8:109:47:052
16898991916,2cyclictest23482-21kworker/u8:109:39:042
16898991916,2cyclictest17594-21kworker/u8:111:12:042
16898991916,2cyclictest1404-21kworker/u8:012:27:042
16898991916,2cyclictest12294-21kworker/u8:111:44:052
1689899190,18cyclictest0-21swapper/212:39:032
1689899190,14cyclictest0-21swapper/210:37:052
16897991916,2cyclictest8495-21kworker/u8:210:25:041
16897991916,2cyclictest23482-21kworker/u8:109:59:041
16898991815,2cyclictest27397-21kworker/u8:008:58:052
16898991815,2cyclictest26036-21kworker/u8:011:39:042
16898991815,2cyclictest23482-21kworker/u8:109:54:032
16898991815,2cyclictest23194-21kworker/u8:210:43:032
16898991815,2cyclictest17345-21kworker/u8:208:51:042
16898991815,2cyclictest17345-21kworker/u8:208:37:042
16898991815,2cyclictest16470-21kworker/u8:209:58:032
16898991815,2cyclictest11305-21kworker/u8:008:42:032
16898991815,2cyclictest11305-21kworker/u8:008:32:042
1689899180,2cyclictest0-21swapper/211:18:052
1689899180,2cyclictest0-21swapper/207:50:032
1689899180,13cyclictest0-21swapper/207:44:042
16898991714,2cyclictest26036-21kworker/u8:011:26:042
16898991714,2cyclictest23482-21kworker/u8:108:48:052
16898991714,2cyclictest23256-21kworker/u8:207:29:032
16898991714,2cyclictest22240-21kworker/u8:107:10:042
16898991714,2cyclictest18833-21kworker/u8:211:51:042
16898991714,2cyclictest17182-21kworker/u8:007:22:052
1689899170,12cyclictest0-21swapper/208:21:042
16897991714,2cyclictest6517-21kworker/u8:211:31:031
16897991714,2cyclictest2760-21kworker/u8:011:00:041
16897991714,2cyclictest23194-21kworker/u8:210:42:051
16897991712,3cyclictest20522-21grep07:15:271
1689799170,12cyclictest0-21swapper/109:43:041
16898991613,2cyclictest20781-21kworker/u8:108:02:042
16898991613,2cyclictest17594-21kworker/u8:111:06:032
16897991613,2cyclictest18833-21kworker/u8:211:55:031
1689699161,3cyclictest0-21swapper/008:05:000
16896991611,3cyclictest30392-21ssh10:45:120
16896991611,3cyclictest0-21swapper/010:05:150
303622150,0sleep10-21swapper/112:23:391
16899991511,3cyclictest8418-21nscd09:30:043
1689999151,12cyclictest25419-21ssh10:40:063
16899991510,3cyclictest17611-21iostat12:10:173
1689999150,3cyclictest6151-21ssh11:59:123
1689999150,3cyclictest25562-21chrt08:50:143
16898991512,2cyclictest3066-21kworker/u8:208:17:032
16898991512,2cyclictest23482-21kworker/u8:109:05:032
16898991512,2cyclictest20781-21kworker/u8:108:13:052
1689899150,2cyclictest0-21swapper/210:51:052
1689899150,2cyclictest0-21swapper/209:00:042
1689899150,14cyclictest0-21swapper/211:02:052
1689799159,4cyclictest0-21swapper/110:58:511
1689799152,7cyclictest0-21swapper/112:25:001
1689799151,3cyclictest0-21swapper/108:20:241
1689799151,3cyclictest0-21swapper/108:15:261
16897991512,2cyclictest8495-21kworker/u8:210:30:041
16897991512,2cyclictest10752-21kworker/u8:211:07:041
1689799151,12cyclictest24993-21ssh10:39:481
16897991510,3cyclictest0-21swapper/112:15:061
1689799150,2cyclictest0-21swapper/109:29:041
1689799150,13cyclictest203450irq/24-eth111:15:151
1689799150,13cyclictest0-21swapper/108:55:151
1689699152,6cyclictest0-21swapper/007:55:000
16896991512,2cyclictest10313-21ssh10:24:530
16896991512,2cyclictest0-21swapper/012:10:130
1689699151,12cyclictest13216-21df_inode11:00:160
1689999149,3cyclictest26097-21ssh11:13:233
1689999149,3cyclictest25335-21ssh10:06:103
16899991411,2cyclictest0-21swapper/312:23:023
16899991411,2cyclictest0-21swapper/308:40:123
1689999140,3cyclictest7867-21munin-node3
1689999140,2cyclictest0-21swapper/312:29:523
1689999140,2cyclictest0-21swapper/310:50:123
1689999140,13cyclictest20408-21ssh10:35:033
1689999140,13cyclictest0-21swapper/311:20:023
1689999140,13cyclictest0-21swapper/309:41:423
1689999140,13cyclictest0-21swapper/309:35:233
1689999140,12cyclictest753-21diskmemload12:04:023
1689999140,12cyclictest26452-21ssh11:45:533
16898991411,2cyclictest0-21swapper/210:27:072
1689899141,11cyclictest24876-21ssh10:05:292
1689899140,2cyclictest24145-1kworker/u9:010:47:032
1689899140,2cyclictest0-21swapper/210:56:042
1689899140,2cyclictest0-21swapper/210:20:042
1689899140,2cyclictest0-21swapper/208:25:032
1689899140,2cyclictest0-21swapper/207:30:172
1689899140,12cyclictest21864-21ssh10:03:272
1689799149,3cyclictest0-21swapper/112:11:541
1689799142,7cyclictest27028-21apt-get07:35:011
1689799141,2cyclictest753-21diskmemload11:25:251
16897991411,2cyclictest31830-21kworker/u8:112:04:031
16897991411,2cyclictest26600-21kworker/u8:011:49:031
16897991411,2cyclictest23256-21kworker/u8:207:28:031
1689799141,11cyclictest0-21swapper/111:52:181
1689799140,9cyclictest0-21swapper/107:49:041
1689799140,2cyclictest0-21swapper/109:33:471
1689799140,2cyclictest0-21swapper/109:10:181
1689799140,12cyclictest24046-21ssh11:44:411
1689799140,12cyclictest0-21swapper/112:31:301
1689799140,12cyclictest0-21swapper/108:10:221
1689699149,3cyclictest7490-21diskstats12:00:170
1689699149,3cyclictest21513-21df_abs07:20:160
16896991411,2cyclictest0-21swapper/011:22:230
1689699140,3cyclictest25085-21latency_hist07:30:000
1689699140,2cyclictest0-21swapper/012:31:440
1689699140,2cyclictest0-21swapper/011:39:130
1689699140,2cyclictest0-21swapper/009:39:140
1689699140,2cyclictest0-21swapper/009:05:130
1689699140,2cyclictest0-21swapper/007:30:140
1689699140,13cyclictest753-21diskmemload11:44:440
1689699140,12cyclictest0-21swapper/010:40:140
89222130,2sleep10-21swapper/110:22:471
1689999139,2cyclictest11187-21sed08:14:593
1689999139,2cyclictest0-21swapper/308:35:003
16899991310,2cyclictest3701-21ssh10:17:323
16899991310,2cyclictest0-21swapper/312:31:423
16899991310,2cyclictest0-21swapper/312:09:573
16899991310,2cyclictest0-21swapper/311:33:123
16899991310,2cyclictest0-21swapper/310:30:403
16899991310,2cyclictest0-21swapper/310:01:533
16899991310,2cyclictest0-21swapper/309:50:443
16899991310,2cyclictest0-21swapper/309:46:233
16899991310,2cyclictest0-21swapper/308:15:223
16899991310,2cyclictest0-21swapper/307:38:123
1689999130,2cyclictest0-21swapper/311:03:043
1689999130,2cyclictest0-21swapper/310:27:033
1689999130,2cyclictest0-21swapper/310:21:333
1689999130,2cyclictest0-21swapper/309:15:143
1689999130,2cyclictest0-21swapper/308:55:153
1689999130,12cyclictest0-21swapper/311:38:143
1689999130,12cyclictest0-21swapper/311:27:533
1689999130,12cyclictest0-21swapper/311:05:523
1689999130,11cyclictest0-21swapper/312:37:253
1689999130,11cyclictest0-21swapper/311:18:173
1689899138,3cyclictest0-21swapper/210:19:252
1689899130,2cyclictest0-21swapper/209:22:042
1689799134,7cyclictest9796-21cat09:50:281
1689799131,9cyclictest300972sleep109:00:181
1689799131,7cyclictest0-21swapper/107:40:001
1689799131,10cyclictest0-21swapper/107:40:251
16897991310,2cyclictest3066-21kworker/u8:208:06:041
16897991310,2cyclictest2728-21kworker/u8:009:15:041
16897991310,2cyclictest23482-21kworker/u8:110:03:051
16897991310,2cyclictest20781-21kworker/u8:107:52:031
16897991310,2cyclictest17182-21kworker/u8:007:23:051
16897991310,2cyclictest0-21swapper/110:50:321
16897991310,2cyclictest0-21swapper/110:45:161
16897991310,2cyclictest0-21swapper/108:30:121
16897991310,2cyclictest0-21swapper/108:25:211
1689799130,2cyclictest0-21swapper/112:37:361
1689799130,12cyclictest753-21diskmemload09:46:111
1689799130,12cyclictest0-21swapper/109:35:201
1689799130,11cyclictest0-21swapper/112:05:241
1689699139,2cyclictest0-21swapper/009:27:250
1689699132,6cyclictest0-21swapper/011:29:590
1689699132,6cyclictest0-21swapper/008:30:000
1689699131,7cyclictest0-21swapper/007:20:000
1689699131,6cyclictest0-21swapper/009:10:010
1689699131,10cyclictest3745-21runrttasks12:38:500
16896991310,2cyclictest0-21swapper/010:32:230
16896991310,2cyclictest0-21swapper/010:01:140
1689699130,2cyclictest0-21swapper/009:15:240
1689699130,2cyclictest0-21swapper/008:15:160
1689699130,12cyclictest0-21swapper/012:24:340
1689699130,12cyclictest0-21swapper/010:11:330
1689699130,12cyclictest0-21swapper/009:59:130
1689699130,12cyclictest0-21swapper/009:23:130
1689699130,11cyclictest19909-21ssh11:06:290
1689699130,11cyclictest0-21swapper/011:15:450
297812120,2sleep10-21swapper/110:10:291
23272120,1sleep02338-21ssh12:27:560
221702120,2sleep20-21swapper/209:30:252
1689999129,2cyclictest0-21swapper/312:16:003
1689999129,2cyclictest0-21swapper/311:40:003
1689999129,2cyclictest0-21swapper/309:58:403
1689999129,2cyclictest0-21swapper/308:08:143
1689999120,2cyclictest0-21swapper/310:11:283
1689999120,11cyclictest0-21swapper/310:47:113
1689999120,11cyclictest0-21swapper/309:26:053
1689999120,11cyclictest0-21swapper/309:20:403
1689999120,11cyclictest0-21swapper/307:40:193
1689899120,2cyclictest0-21swapper/207:48:042
1689899120,2cyclictest0-21swapper/207:35:032
1689899120,11cyclictest0-21swapper/212:20:232
1689899120,11cyclictest0-21swapper/210:30:472
1689799129,2cyclictest23482-21kworker/u8:110:05:041
1689799129,2cyclictest11163-21kworker/u8:207:12:031
1689799129,2cyclictest0-21swapper/111:35:131
1689799127,3cyclictest19311-21munin-node1
1689799121,9cyclictest0-21swapper/108:40:241
1689799120,2cyclictest0-21swapper/108:45:271
1689799120,11cyclictest0-21swapper/111:20:301
1689799120,11cyclictest0-21swapper/108:50:211
1689699129,2cyclictest13829-21ssh12:05:560
1689699129,2cyclictest10849-21ssh10:57:330
1689699129,2cyclictest0-21swapper/009:52:040
1689699129,2cyclictest0-21swapper/008:35:150
1689699122,7cyclictest0-21swapper/008:54:580
1689699121,6cyclictest0-21swapper/008:15:000
1689699120,2cyclictest0-21swapper/011:51:150
1689699120,2cyclictest0-21swapper/011:47:240
1689699120,2cyclictest0-21swapper/010:35:450
1689699120,11cyclictest0-21swapper/012:15:150
1689699120,11cyclictest0-21swapper/011:12:040
1689699120,11cyclictest0-21swapper/010:50:150
1689699120,11cyclictest0-21swapper/009:45:330
1689699120,11cyclictest0-21swapper/009:40:550
1689699120,11cyclictest0-21swapper/009:31:140
1689999118,2cyclictest0-21swapper/311:50:413
1689999112,5cyclictest0-21swapper/307:25:003
1689999110,10cyclictest0-21swapper/307:31:143
1689899111,6cyclictest0-21swapper/207:15:252
1689799118,2cyclictest0-21swapper/109:05:151
1689799116,3cyclictest0-21swapper/108:00:271
1689699112,5cyclictest0-21swapper/010:25:250
1689699110,10cyclictest0-21swapper/010:16:360
1689999107,2cyclictest0-21swapper/308:20:223
1689999107,2cyclictest0-21swapper/307:50:323
1689999102,6cyclictest0-21swapper/307:10:253
1689699107,2cyclictest0-21swapper/007:50:030
1689699107,2cyclictest0-21swapper/007:35:150
1689699105,3cyclictest18910-21gpgv07:15:010
1689699100,7cyclictest0-21swapper/009:03:590
140972100,7sleep00-21swapper/008:20:190
168999996,2cyclictest0-21swapper/308:45:433
168999994,3cyclictest3011-21nscd09:05:153
168999992,5cyclictest0-21swapper/308:26:583
168999990,7cyclictest20757-21chrt08:35:353
168989994,3cyclictest3163-21basename07:55:142
168969994,3cyclictest7520-21tr08:05:130
168969994,3cyclictest31524-21cpuspeed_turbos07:45:170
168969994,3cyclictest30172-21ntpdc07:40:190
168969991,6cyclictest0-21swapper/008:30:240
168999983,3cyclictest5223-21taskset08:00:043
168999983,3cyclictest23313-21aten2.4-expect07:25:143
168999980,7cyclictest0-21swapper/307:55:053
168999980,7cyclictest0-21swapper/307:49:593
168999980,3cyclictest0-21swapper/309:00:053
168989983,3cyclictest7613-21df_abs08:05:142
168969983,3cyclictest27368-21aten2.4_r3power08:55:120
168969983,3cyclictest233462sleep008:45:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional