You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 10:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot6s.osadl.org (updated Thu Nov 27, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28238212198,18sleep10-21swapper/119:08:481
28117211998,15sleep20-21swapper/219:07:132
28278211195,11sleep30-21swapper/319:09:183
28053210696,6sleep00-21swapper/019:06:230
259472770,0sleep10-21swapper/122:00:411
91722720,1sleep011-21rcuc/022:52:190
258082690,0sleep30-21swapper/323:10:143
173212690,1sleep30-21swapper/322:26:093
204892280,1sleep20-21swapper/222:30:172
41102270,1sleep00-21swapper/023:55:250
137692220,1sleep2301ktimersoftd/222:23:002
2836599210,2cyclictest0-21swapper/219:40:112
189372210,1sleep1221ktimersoftd/123:37:071
28365992015,3cyclictest24704-21ssh22:35:122
28364992017,2cyclictest0-21swapper/121:25:101
241072200,1sleep121-21rcuc/123:42:471
28363991916,2cyclictest9434-21kworker/u8:021:34:040
2836399190,2cyclictest0-21swapper/020:15:140
2836399190,2cyclictest0-21swapper/020:15:130
28363991815,2cyclictest8353-21kworker/u8:222:39:050
28363991815,2cyclictest20668-21kworker/u8:023:43:040
28363991815,2cyclictest1753-21kworker/u8:121:48:020
2836699171,3cyclictest0-21swapper/320:24:583
2836699171,14cyclictest23564-21latency_hist21:24:593
2836599173,12cyclictest0-21swapper/222:06:472
2836599171,14cyclictest0-21swapper/223:00:112
28364991714,2cyclictest2728-21kworker/u8:122:16:011
28364991713,2cyclictest8353-21kworker/u8:222:40:041
177772170,1sleep00-21swapper/023:35:250
66852160,1sleep30-21swapper/321:00:233
2836699161,13cyclictest9458-21diskmemload21:10:243
2836699161,13cyclictest13137-21iostat19:50:153
2836599163,2cyclictest0-21swapper/223:43:262
2836599161,3cyclictest9458-21diskmemload21:10:012
28365991611,3cyclictest0-21swapper/221:15:192
28365991610,4cyclictest0-21swapper/223:10:002
2836599160,14cyclictest0-21swapper/200:30:172
28364991613,2cyclictest9434-21kworker/u8:021:35:041
28364991612,2cyclictest20668-21kworker/u8:023:09:041
28363991613,2cyclictest32116-21kworker/u8:100:27:050
2836699150,2cyclictest0-21swapper/300:11:063
2836699150,13cyclictest9458-21diskmemload22:00:493
2836699150,13cyclictest23190-21ssh22:33:053
2836699150,13cyclictest0-21swapper/322:48:503
2836699150,13cyclictest0-21swapper/320:45:113
2836599153,2cyclictest0-21swapper/222:27:562
2836599152,12cyclictest0-21swapper/221:25:152
2836599152,12cyclictest0-21swapper/200:25:192
28365991512,2cyclictest0-21swapper/223:57:012
2836599151,13cyclictest301ktimersoftd/200:05:172
2836599151,12cyclictest9458-21diskmemload21:45:142
2836599151,12cyclictest0-21swapper/222:55:162
2836599151,12cyclictest0-21swapper/200:04:362
28365991510,3cyclictest17442-21ssh21:52:062
28365991510,3cyclictest0-21swapper/222:11:082
28364991512,2cyclictest31296-21kworker/u8:121:05:041
28364991511,2cyclictest19524-21kworker/u8:222:07:041
2836499150,3cyclictest0-21swapper/123:20:111
28363991512,2cyclictest5405-21kworker/u8:200:38:040
28363991512,2cyclictest32527-21kworker/u8:223:19:040
28363991512,2cyclictest30124-21kworker/u8:022:06:040
28363991512,2cyclictest20668-21kworker/u8:023:49:040
28363991512,2cyclictest17192-21kworker/u8:200:14:040
2836399150,3cyclictest0-21swapper/000:15:170
2836399150,13cyclictest0-21swapper/023:06:440
2836699149,3cyclictest19547-21ssh21:55:113
2836699142,7cyclictest4046-21gpgv19:30:003
28366991411,2cyclictest9458-21diskmemload22:21:203
2836699141,11cyclictest1590-21ssh21:35:173
2836699140,2cyclictest0-21swapper/321:52:393
2836699140,2cyclictest0-21swapper/320:50:203
2836699140,2cyclictest0-21swapper/300:15:243
2836699140,13cyclictest0-21swapper/323:00:493
2836699140,13cyclictest0-21swapper/300:29:093
2836699140,12cyclictest9458-21diskmemload22:10:173
2836699140,12cyclictest17164-21modprobe20:00:173
2836699140,12cyclictest0-21swapper/321:15:113
2836599143,2cyclictest0-21swapper/222:18:082
2836599143,2cyclictest0-21swapper/221:55:382
2836599142,7cyclictest9458-21diskmemload22:00:002
2836599142,10cyclictest0-21swapper/221:31:092
2836599141,12cyclictest301ktimersoftd/222:47:362
2836599140,12cyclictest0-21swapper/220:45:242
2836499149,3cyclictest0-21swapper/119:10:201
2836499149,3cyclictest0-21swapper/100:22:421
2836499142,2cyclictest0-21swapper/121:16:011
28364991411,2cyclictest28362-21cyclictest22:50:001
28364991411,2cyclictest20668-21kworker/u8:023:32:041
28364991411,2cyclictest1753-21kworker/u8:121:41:041
28364991411,2cyclictest12147-21kworker/u8:021:51:041
28364991410,2cyclictest8229-21fschecks_time00:00:161
2836499140,8cyclictest0-21swapper/121:30:061
2836499140,13cyclictest0-21swapper/100:30:161
2836499140,12cyclictest9458-21diskmemload22:35:141
2836499140,12cyclictest6992-21munin-node1
2836499140,12cyclictest26158-21cat00:20:001
2836499140,12cyclictest0-21swapper/123:52:251
2836499140,12cyclictest0-21swapper/123:01:241
2836499140,12cyclictest0-21swapper/122:55:371
2836499140,12cyclictest0-21swapper/100:10:161
28363991411,2cyclictest8353-21kworker/u8:222:20:040
28363991411,2cyclictest5092-21kworker/u8:223:02:050
28363991411,2cyclictest32116-21kworker/u8:100:30:040
28363991411,2cyclictest0-21swapper/023:26:350
28363991410,2cyclictest0-21swapper/019:30:060
2836399140,3cyclictest15880-21fschecks_time22:25:140
2836399140,3cyclictest15549-21sendmail_mailst21:15:220
2836399140,13cyclictest0-21swapper/023:21:240
2836399140,12cyclictest0-21swapper/022:44:570
2836399140,12cyclictest0-21swapper/019:35:140
16152140,2sleep11810-21munin-node1
74542138,2sleep00-21swapper/021:41:050
6792130,2sleep10-21swapper/100:26:101
288412130,2sleep10-21swapper/123:13:511
2836699138,3cyclictest6660-21ssh00:32:563
2836699132,7cyclictest0-21swapper/319:20:003
2836699132,6cyclictest0-21swapper/319:40:013
2836699132,5cyclictest0-21swapper/322:40:003
2836699131,6cyclictest0-21swapper/300:25:013
2836699131,2cyclictest24652-21ssh21:25:163
28366991310,2cyclictest0-21swapper/323:54:593
28366991310,2cyclictest0-21swapper/323:24:153
28366991310,2cyclictest0-21swapper/321:33:533
28366991310,2cyclictest0-21swapper/300:02:403
2836699130,2cyclictest0-21swapper/323:59:043
2836699130,2cyclictest0-21swapper/323:25:363
2836699130,2cyclictest0-21swapper/322:55:173
2836699130,2cyclictest0-21swapper/322:16:103
2836699130,11cyclictest0-21swapper/321:42:323
2836599138,3cyclictest8859-21perl00:35:172
2836599132,7cyclictest0-21swapper/219:25:012
2836599131,2cyclictest301ktimersoftd/200:16:052
2836599131,2cyclictest301ktimersoftd/200:14:072
2836599131,10cyclictest9458-21diskmemload23:49:332
28365991310,2cyclictest9458-21diskmemload22:50:072
28365991310,2cyclictest301ktimersoftd/223:53:162
28365991310,2cyclictest301ktimersoftd/223:15:322
28365991310,2cyclictest301ktimersoftd/220:20:262
28365991310,2cyclictest0-21swapper/223:25:172
28365991310,2cyclictest0-21swapper/221:38:452
28365991310,2cyclictest0-21swapper/200:24:372
2836599130,12cyclictest0-21swapper/223:23:182
2836599130,11cyclictest9458-21diskmemload23:30:232
2836599130,11cyclictest9458-21diskmemload22:40:122
2836499138,3cyclictest9458-21diskmemload22:30:151
2836499138,3cyclictest15327-21rm22:25:111
28364991310,2cyclictest19524-21kworker/u8:222:10:051
28364991310,2cyclictest1476-21kworker/u8:221:14:051
28364991310,2cyclictest0-21swapper/123:45:371
28364991310,2cyclictest0-21swapper/123:15:411
28364991310,2cyclictest0-21swapper/122:22:091
28364991310,2cyclictest0-21swapper/121:21:031
28364991310,2cyclictest0-21swapper/120:20:171
28364991310,2cyclictest0-21swapper/119:25:171
2836499130,2cyclictest9458-21diskmemload22:52:261
2836499130,2cyclictest0-21swapper/123:58:351
2836499130,2cyclictest0-21swapper/119:55:181
2836499130,12cyclictest0-21swapper/100:36:031
28363991310,2cyclictest9458-21diskmemload21:10:080
28363991310,2cyclictest8353-21kworker/u8:222:18:010
28363991310,2cyclictest6104-21kworker/u8:000:03:050
28363991310,2cyclictest18422-21kworker/u8:322:31:040
28363991310,2cyclictest1753-21kworker/u8:121:55:040
28363991310,2cyclictest16294-21kworker/u8:223:53:040
28363991310,2cyclictest11583-21kworker/u8:219:17:040
28363991310,2cyclictest0-21swapper/021:20:120
28363991310,2cyclictest0-21swapper/020:40:000
2836399130,2cyclictest9458-21diskmemload23:30:320
2836399130,2cyclictest0-21swapper/000:20:250
2836399130,12cyclictest0-21swapper/022:10:120
2836399130,12cyclictest0-21swapper/000:07:440
2836399130,11cyclictest0-21swapper/021:35:140
2836699129,2cyclictest0-21swapper/323:48:503
2836699129,2cyclictest0-21swapper/323:44:433
2836699129,2cyclictest0-21swapper/323:35:063
2836699129,2cyclictest0-21swapper/323:32:033
2836699129,2cyclictest0-21swapper/323:06:153
2836699129,2cyclictest0-21swapper/322:44:363
2836699129,2cyclictest0-21swapper/322:06:033
2836699129,2cyclictest0-21swapper/319:40:103
2836699122,7cyclictest0-21swapper/320:29:583
2836699122,6cyclictest0-21swapper/320:00:003
2836699121,8cyclictest9458-21diskmemload22:50:243
2836699120,2cyclictest9458-21diskmemload00:07:513
2836699120,2cyclictest0-21swapper/321:46:463
2836699120,11cyclictest0-21swapper/323:15:123
2836699120,11cyclictest0-21swapper/320:55:133
2836699120,11cyclictest0-21swapper/300:38:163
2836599129,2cyclictest301ktimersoftd/219:25:132
2836599129,2cyclictest0-21swapper/221:20:142
2836599122,7cyclictest0-21swapper/220:08:012
2836599120,2cyclictest0-21swapper/223:11:582
2836599120,11cyclictest0-21swapper/221:40:362
2836599120,11cyclictest0-21swapper/221:12:372
2836499129,2cyclictest24638-21kworker/u8:220:39:041
2836499129,2cyclictest18196-21kworker/u8:220:06:041
2836499129,2cyclictest0-21swapper/121:46:171
2836499123,6cyclictest0-21swapper/120:42:021
2836499120,2cyclictest0-21swapper/121:55:371
2836499120,2cyclictest0-21swapper/120:45:131
2836499120,2cyclictest0-21swapper/119:45:111
2836499120,2cyclictest0-21swapper/119:30:011
2836499120,11cyclictest332250irq/24-eth120:15:161
2836499120,11cyclictest332250irq/24-eth120:15:161
2836499120,11cyclictest0-21swapper/121:00:191
2836499120,11cyclictest0-21swapper/100:05:451
2836399129,2cyclictest6579-21kworker/u8:020:12:040
2836399129,2cyclictest32364-21kworker/u8:019:25:050
2836399129,2cyclictest0-21swapper/023:10:560
2836399129,2cyclictest0-21swapper/021:00:130
2836399122,7cyclictest0-21swapper/020:48:020
2836399122,7cyclictest0-21swapper/020:04:590
2836399122,7cyclictest0-21swapper/019:43:580
2836399122,7cyclictest0-21swapper/019:22:010
2836399120,2cyclictest0-21swapper/022:00:290
2836399120,2cyclictest0-21swapper/021:26:290
2836399120,11cyclictest0-21swapper/022:55:150
2836399120,11cyclictest0-21swapper/022:45:040
2836399120,11cyclictest0-21swapper/021:50:070
2836399120,11cyclictest0-21swapper/021:05:170
2836399120,11cyclictest0-21swapper/019:50:130
2836699112,6cyclictest0-21swapper/319:13:023
2836599116,3cyclictest18286-21ssh23:36:032
2836599112,7cyclictest0-21swapper/219:56:542
2836599112,6cyclictest0-21swapper/221:04:582
2836599112,6cyclictest0-21swapper/220:40:592
2836599111,6cyclictest0-21swapper/219:51:532
2836499118,2cyclictest6579-21kworker/u8:019:36:051
2836499118,2cyclictest0-21swapper/120:55:171
2836499118,2cyclictest0-21swapper/119:40:131
2836499117,2cyclictest27173-21munin-node1
2836499110,3cyclictest0-21swapper/119:15:111
2836399118,2cyclictest24638-21kworker/u8:220:23:050
2836399118,2cyclictest11583-21kworker/u8:219:11:050
2836399112,6cyclictest0-21swapper/020:32:010
2836399112,6cyclictest0-21swapper/020:25:010
2836399111,6cyclictest0-21swapper/020:55:230
2836399111,6cyclictest0-21swapper/020:05:230
2836399110,2cyclictest0-21swapper/020:50:110
2836399110,10cyclictest0-21swapper/019:45:190
2836699107,2cyclictest0-21swapper/319:30:043
2836699102,6cyclictest0-21swapper/320:40:203
2836599107,2cyclictest0-21swapper/219:45:142
2836599104,4cyclictest5083-21sshd20:11:202
2836599104,4cyclictest4713-21NetworkManager19:30:042
2836599104,4cyclictest27700-21cut20:30:102
2836599103,5cyclictest29325-21chrt20:36:462
2836599102,7cyclictest0-21swapper/219:19:012
2836599102,6cyclictest0-21swapper/220:01:562
2836499107,2cyclictest0-21swapper/120:00:161
2836499100,5cyclictest0-21swapper/119:55:011
2836499100,2cyclictest0-21swapper/119:20:131
2836399101,8cyclictest0-21swapper/020:43:580
263562100,2sleep10-21swapper/120:25:011
283669994,3cyclictest8563-21open_files21:05:183
283669994,3cyclictest0-21swapper/320:10:003
283659996,2cyclictest0-21swapper/220:50:102
283659995,2cyclictest6173-21apt-get19:35:062
283659994,3cyclictest5129-21runrttasks20:29:282
283659993,4cyclictest28831-21cut19:10:132
283659993,4cyclictest22574-21tail20:15:132
283659993,4cyclictest22574-21tail20:15:132
283659993,4cyclictest1387-21gdbus20:55:112
283649990,7cyclictest0-21swapper/120:10:131
20528290,1sleep30-21swapper/320:10:133
283669984,2cyclictest0-21swapper/320:30:083
283669983,3cyclictest292962sleep320:36:233
283669983,3cyclictest22669-21tail20:15:153
283669983,3cyclictest22669-21tail20:15:143
283669983,3cyclictest10473-21aten2.4_r3power19:45:123
283669980,6cyclictest0-21swapper/319:20:083
283639980,7cyclictest0-21swapper/019:55:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional