You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 05:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot6s.osadl.org (updated Thu Jul 03, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
256162116104,6sleep30-21swapper/319:05:583
25693211494,15sleep10-21swapper/119:06:581
256872114103,6sleep00-21swapper/019:06:530
24358210996,6sleep20-21swapper/219:05:132
142822730,1sleep2301ktimersoftd/222:20:272
142822730,1sleep2301ktimersoftd/222:20:272
215122710,0sleep00-21swapper/021:22:260
280922690,1sleep3381ktimersoftd/323:40:213
255762690,0sleep30-21swapper/322:32:073
267122680,0sleep30-21swapper/320:35:003
236042670,0sleep20-21swapper/223:03:122
25959993516,17cyclictest0-21swapper/223:55:152
164182240,1sleep0101ktimersoftd/021:17:120
2595999231,20cyclictest0-21swapper/200:20:122
2595899230,2cyclictest0-21swapper/121:35:261
25960992218,2cyclictest3912-21kworker/u8:021:10:043
2595899220,3cyclictest0-21swapper/122:10:141
2595899220,3cyclictest0-21swapper/122:10:141
2595899210,3cyclictest27259-21tr19:10:221
2595899210,3cyclictest0-21swapper/119:45:261
2595899210,3cyclictest0-21swapper/100:25:001
25960992017,2cyclictest12866-21kworker/u8:122:08:063
25960992017,2cyclictest0-21swapper/320:50:113
2596099200,2cyclictest8423-21diskmemload23:15:133
2595899200,3cyclictest865-21ssh22:07:181
2595899200,3cyclictest59-21kswapd023:55:071
2595899200,3cyclictest5129-21runrttasks21:29:221
2595899200,3cyclictest27960-21ssh00:13:231
2595899200,3cyclictest23638-21rm21:57:301
2595899200,3cyclictest22910-21ssh21:24:341
2595899200,3cyclictest17334-21sh21:18:341
2595899200,3cyclictest16083-21ssh22:22:381
2595899200,3cyclictest16083-21ssh22:22:381
2595899200,3cyclictest1255-21ssh23:13:381
2595899200,3cyclictest10803-21sh00:28:431
2595899200,3cyclictest0-21swapper/123:30:381
2595899200,3cyclictest0-21swapper/123:25:231
2595899200,3cyclictest0-21swapper/121:51:061
2595899200,3cyclictest0-21swapper/100:30:271
2595899200,3cyclictest0-21swapper/100:00:471
2595899200,2cyclictest22219-21ssh23:35:091
2595899200,2cyclictest0-21swapper/122:43:001
2595899200,2cyclictest0-21swapper/122:02:221
2595899200,2cyclictest0-21swapper/120:35:141
2595899200,2cyclictest0-21swapper/119:40:271
2595899200,2cyclictest0-21swapper/100:39:071
2595899200,18cyclictest0-21swapper/122:32:191
2595899200,18cyclictest0-21swapper/100:08:201
229032200,1sleep00-21swapper/022:30:110
2595999190,2cyclictest0-21swapper/220:15:122
2595899191,3cyclictest28980-21grep21:30:151
2595899191,15cyclictest0-21swapper/119:15:101
2595899190,7cyclictest5911-21gdm-simple-gree20:57:591
2595899190,3cyclictest0-21swapper/123:21:421
2595899190,3cyclictest0-21swapper/123:18:181
2595899190,3cyclictest0-21swapper/122:36:461
2595899190,2cyclictest0-21swapper/123:41:221
2595899190,2cyclictest0-21swapper/121:40:201
2595899190,2cyclictest0-21swapper/121:14:091
2595899190,2cyclictest0-21swapper/100:17:231
2595899190,18cyclictest0-21swapper/123:01:011
2595899190,17cyclictest0-21swapper/123:50:471
281622180,0sleep30-21swapper/322:35:193
25959991813,3cyclictest5010-21snmpd19:35:122
2595899182,7cyclictest0-21swapper/120:51:581
2595899181,3cyclictest894-21python23:45:281
2595899181,3cyclictest17986-21munin-node1
2595899180,3cyclictest24785-21munin-node1
2595899180,3cyclictest15284-21munin-node1
2595899180,3cyclictest0-21swapper/121:48:381
2595899180,3cyclictest0-21swapper/120:09:551
2595899180,3cyclictest0-21swapper/119:37:421
2595899180,2cyclictest0-21swapper/122:46:501
2595899180,16cyclictest0-21swapper/122:17:031
2595899180,16cyclictest0-21swapper/122:17:021
2595899180,16cyclictest0-21swapper/119:30:031
25960991712,3cyclictest15898-21gltestperf22:55:183
2595899171,2cyclictest0-21swapper/123:07:381
2595899170,3cyclictest0-21swapper/120:20:251
2595899170,2cyclictest0-21swapper/120:41:151
2595899170,15cyclictest0-21swapper/120:10:121
2595899170,15cyclictest0-21swapper/119:55:221
25960991613,2cyclictest8678-21kworker/u8:223:24:043
25960991613,2cyclictest657-21kworker/u8:322:26:043
25960991613,2cyclictest17245-21kworker/u8:223:09:043
2595899166,8cyclictest0-21swapper/120:15:121
2595899161,3cyclictest0-21swapper/119:20:141
2595899161,13cyclictest0-21swapper/121:00:131
2595899160,3cyclictest0-21swapper/120:46:501
2595899160,3cyclictest0-21swapper/119:25:191
2595899160,2cyclictest0-21swapper/119:52:491
25957991613,2cyclictest8160-21kworker/u8:022:21:050
25957991613,2cyclictest8160-21kworker/u8:022:21:040
25957991613,2cyclictest17245-21kworker/u8:223:06:040
2595799160,14cyclictest0-21swapper/021:12:050
25960991512,2cyclictest8866-21kworker/u8:121:19:043
25960991512,2cyclictest26879-21kworker/u8:221:38:053
25960991512,2cyclictest14546-21kworker/u8:100:12:043
2596099151,12cyclictest7937-21ssh23:52:583
2595899154,3cyclictest12680-21munin-node1
2595899151,3cyclictest0-21swapper/122:50:151
2595899151,3cyclictest0-21swapper/121:10:001
2595899151,3cyclictest0-21swapper/120:25:161
2595799151,2cyclictest2425-21apt-get20:55:150
2595799150,2cyclictest0-21swapper/020:15:110
2595799150,13cyclictest23102-21missed_timers23:35:190
25960991411,2cyclictest9726-21kworker/u8:220:05:043
25960991411,2cyclictest0-21swapper/300:00:283
2596099140,12cyclictest23238-21open_inodes23:35:203
2596099140,10cyclictest0-21swapper/300:27:043
2595999141,11cyclictest10368-21apt-get22:50:132
2595999140,3cyclictest17941-21timerandwakeup20:10:252
2595999140,2cyclictest0-21swapper/221:25:192
2595799149,3cyclictest9365-21sh22:49:000
2595799149,3cyclictest1314-21munin-node0
25957991411,2cyclictest8423-21diskmemload22:07:440
25957991411,2cyclictest15775-21kworker/u8:023:43:040
25957991411,2cyclictest14546-21kworker/u8:100:14:040
25957991411,2cyclictest0-21swapper/022:16:240
25957991411,2cyclictest0-21swapper/022:16:240
25957991411,2cyclictest0-21swapper/019:10:240
25957991411,2cyclictest0-21swapper/000:31:240
25957991410,2cyclictest6306-21ssh00:24:240
2595799140,2cyclictest8423-21diskmemload00:39:340
2595799140,2cyclictest13343-21taskset21:46:450
2595799140,2cyclictest0-21swapper/023:58:040
2595799140,2cyclictest0-21swapper/023:25:240
2595799140,13cyclictest0-21swapper/023:23:150
2595799140,13cyclictest0-21swapper/021:35:540
2596099139,2cyclictest29755-21ntp_states20:40:193
25960991310,2cyclictest5483-21kworker/u8:122:50:053
25960991310,2cyclictest5483-21kworker/u8:122:48:043
25960991310,2cyclictest29740-21kworker/u8:019:29:043
25960991310,2cyclictest27644-21kworker/u8:119:14:013
25960991310,2cyclictest26879-21kworker/u8:221:55:043
25960991310,2cyclictest26879-21kworker/u8:221:32:043
25960991310,2cyclictest0-21swapper/323:00:023
25960991310,2cyclictest0-21swapper/322:00:273
25960991310,2cyclictest0-21swapper/321:54:133
25960991310,2cyclictest0-21swapper/321:25:483
25960991310,2cyclictest0-21swapper/321:24:423
25960991310,2cyclictest0-21swapper/319:35:223
25960991310,2cyclictest0-21swapper/300:20:383
2596099130,2cyclictest0-21swapper/323:28:333
2596099130,12cyclictest0-21swapper/322:20:183
2596099130,12cyclictest0-21swapper/322:20:173
2596099130,12cyclictest0-21swapper/322:10:323
2596099130,12cyclictest0-21swapper/322:10:323
2596099130,12cyclictest0-21swapper/300:36:223
2596099130,11cyclictest0-21swapper/323:58:063
2596099130,11cyclictest0-21swapper/320:10:123
2596099130,11cyclictest0-21swapper/300:15:513
2595999138,3cyclictest29996-21ssh23:10:072
2595999138,3cyclictest2455-21ssh23:15:092
2595999138,3cyclictest18388-21ssh22:57:472
2595999133,6cyclictest0-21swapper/223:20:572
2595999133,6cyclictest0-21swapper/221:15:322
25959991310,2cyclictest0-21swapper/223:27:172
25959991310,2cyclictest0-21swapper/200:05:072
2595999130,2cyclictest0-21swapper/223:39:372
2595999130,2cyclictest0-21swapper/219:45:172
2595999130,2cyclictest0-21swapper/219:25:122
2595999130,12cyclictest0-21swapper/221:45:472
25957991310,2cyclictest26879-21kworker/u8:221:40:040
25957991310,2cyclictest2162-21kworker/u8:200:01:040
25957991310,2cyclictest0-21swapper/023:02:240
25957991310,2cyclictest0-21swapper/022:56:300
25957991310,2cyclictest0-21swapper/021:57:050
25957991310,2cyclictest0-21swapper/021:53:510
25957991310,2cyclictest0-21swapper/000:05:440
2595799130,8cyclictest0-21swapper/021:30:040
2595799130,6cyclictest0-21swapper/023:14:040
2595799130,4cyclictest0-21swapper/022:25:040
2595799130,2cyclictest0-21swapper/023:53:200
2595799130,2cyclictest0-21swapper/022:40:340
2595799130,2cyclictest0-21swapper/020:05:270
2595799130,2cyclictest0-21swapper/000:25:370
2595799130,12cyclictest0-21swapper/023:15:120
2595799130,12cyclictest0-21swapper/022:03:440
2595799130,12cyclictest0-21swapper/020:45:240
2595799130,11cyclictest0-21swapper/000:15:520
120652130,2sleep00-21swapper/022:50:410
314912120,1sleep20-21swapper/221:32:392
2596099129,2cyclictest9726-21kworker/u8:221:04:043
2596099129,2cyclictest6101-21kworker/u8:021:45:043
2596099129,2cyclictest10494-21copy19:55:043
2596099129,2cyclictest0-21swapper/323:31:573
2596099129,2cyclictest0-21swapper/322:15:033
2596099129,2cyclictest0-21swapper/322:15:033
2596099129,2cyclictest0-21swapper/320:35:263
2596099129,2cyclictest0-21swapper/300:31:573
2596099129,2cyclictest0-21swapper/300:05:183
2596099126,4cyclictest32584-21fschecks_count23:45:183
2596099122,9cyclictest0-21swapper/319:42:013
2596099120,11cyclictest0-21swapper/323:10:173
2596099120,11cyclictest0-21swapper/322:40:133
2596099120,11cyclictest0-21swapper/321:41:133
2596099120,11cyclictest0-21swapper/320:20:153
2595999129,2cyclictest8423-21diskmemload22:13:172
2595999129,2cyclictest8423-21diskmemload22:13:172
2595999129,2cyclictest0-21swapper/223:49:372
2595999129,2cyclictest0-21swapper/221:50:132
2595999127,3cyclictest21290-21cut20:20:142
2595999123,6cyclictest0-21swapper/220:25:222
2595999122,7cyclictest0-21swapper/222:28:012
2595999122,7cyclictest0-21swapper/220:57:012
2595999122,7cyclictest0-21swapper/219:52:012
2595999122,6cyclictest0-21swapper/221:05:002
2595999122,6cyclictest0-21swapper/220:00:002
2595999121,7cyclictest8423-21diskmemload00:00:012
2595999121,7cyclictest32427-21tr22:40:002
2595999121,6cyclictest0-21swapper/222:10:002
2595999120,11cyclictest0-21swapper/200:11:472
2595799129,2cyclictest0-21swapper/023:46:360
2595799129,2cyclictest0-21swapper/020:25:250
2595799129,2cyclictest0-21swapper/019:50:000
2595799122,6cyclictest0-21swapper/020:40:250
2595799121,9cyclictest0-21swapper/019:55:080
2595799120,2cyclictest0-21swapper/023:30:300
2595799120,2cyclictest0-21swapper/022:36:400
2595799120,2cyclictest0-21swapper/020:35:190
2595799120,1cyclictest25311-21ssh21:25:440
2595799120,11cyclictest0-21swapper/022:10:130
2595799120,11cyclictest0-21swapper/022:10:120
2595799120,11cyclictest0-21swapper/021:00:260
201872120,2sleep20-21swapper/223:32:242
127182120,8sleep212901-21grep20:00:142
2596099118,2cyclictest9726-21kworker/u8:220:04:043
2596099118,2cyclictest29740-21kworker/u8:019:46:043
2596099118,2cyclictest29740-21kworker/u8:019:20:013
2596099118,2cyclictest16720-21kworker/u8:120:18:053
2596099112,6cyclictest0-21swapper/320:26:583
2596099112,6cyclictest0-21swapper/319:17:593
2596099112,5cyclictest0-21swapper/319:35:003
2595999118,2cyclictest0-21swapper/223:42:572
2595999118,2cyclictest0-21swapper/222:40:172
2595999118,2cyclictest0-21swapper/219:20:012
2595999116,3cyclictest29338-21cut20:40:152
2595999116,3cyclictest14535-21taskset00:31:542
2595999116,3cyclictest0-21swapper/221:35:132
2595999112,6cyclictest0-21swapper/221:05:272
2595999111,7cyclictest0-21swapper/220:09:592
2595999111,6cyclictest0-21swapper/219:30:242
2595999110,2cyclictest8423-21diskmemload21:59:372
2595999110,10cyclictest0-21swapper/222:31:172
2595799116,3cyclictest1372-21sshd19:27:490
2595799113,6cyclictest0-21swapper/021:05:300
2595799112,6cyclictest3818-21apt-get19:35:120
2595799110,10cyclictest0-21swapper/020:12:490
2596099107,2cyclictest9726-21kworker/u8:220:46:043
2596099107,2cyclictest0-21swapper/321:05:243
2596099105,3cyclictest4230-21cron21:00:003
2595999107,2cyclictest11325-21ssh22:17:592
2595999107,2cyclictest11325-21ssh22:17:582
2595999107,2cyclictest0-21swapper/220:30:162
2595999105,3cyclictest6842-21load21:40:172
2595999105,3cyclictest21558-21cron00:40:002
2595999102,6cyclictest0-21swapper/200:25:242
2595999102,5cyclictest0-21swapper/220:52:012
2595999101,7cyclictest8667-21ssh23:54:052
2595999101,7cyclictest0-21swapper/220:46:012
2595999100,9cyclictest0-21swapper/223:07:332
2595999100,9cyclictest0-21swapper/222:01:012
2595999100,2cyclictest0-21swapper/221:20:122
2595799107,2cyclictest0-21swapper/019:30:060
2595799105,3cyclictest8898-21grep19:45:190
2595799105,3cyclictest30278-21telnet19:20:120
2595799105,3cyclictest21958-21wc20:20:260
2595799100,9cyclictest0-21swapper/019:16:240
259609990,8cyclictest0-21swapper/319:50:153
259599994,3cyclictest8423-21diskmemload00:15:472
259599994,3cyclictest6984-21unixbench_singl22:45:302
259599994,3cyclictest30282-21aten2.4_r3power19:20:122
259599994,3cyclictest27243-21grep20:35:142
259599994,3cyclictest26878-21latency19:10:162
259599991,6cyclictest0-21swapper/221:10:262
259599990,6cyclictest0-21swapper/219:40:132
259579994,3cyclictest7353-21sed19:40:270
259579994,3cyclictest25396-21hwlatdetect20:30:200
259579990,6cyclictest13277-21hwlatdetect20:00:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional