You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 04:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot6s.osadl.org (updated Thu Nov 20, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17801997831,28cyclictest20515-21runrttasks23:02:373
17801996928,14cyclictest18358-21runrttasks21:51:253
17801996928,14cyclictest18358-21runrttasks21:51:253
17801996916,46cyclictest29980-21df_inode22:16:173
17801996825,36cyclictest19560-21turbostat00:10:583
243799670,7rtkit-daemon0-21swapper/119:01:112
243799660,6rtkit-daemon8095-21processes20:21:382
17801996624,36cyclictest9765-21runrttasks23:48:153
243799650,7rtkit-daemon0-21swapper/022:20:112
17801996519,39cyclictest16504-21python21:46:333
17801996417,42cyclictest0-21swapper/322:51:273
276002630,6sleep127599-21uniq00:26:151
17801996312,44cyclictest26929-21cat00:25:583
17801996216,40cyclictest0-21swapper/322:06:313
17801996216,39cyclictest0-21swapper/322:36:153
17801996216,39cyclictest0-21swapper/322:36:153
17801996215,41cyclictest0-21swapper/323:21:003
17801996214,41cyclictest0-21swapper/319:31:093
17801996213,42cyclictest23264-21iostat_ios22:01:223
17801996143,12cyclictest27971-21gltestperf19:56:273
17801996122,33cyclictest3301-21wc21:21:163
17801996112,42cyclictest10784-21chrt22:41:333
1780199604,48cyclictest0-21swapper/322:46:333
17801996025,29cyclictest0-21swapper/321:17:443
17801996022,32cyclictest2815-21hwlatdetect22:26:223
17801996018,11cyclictest31667-21iostat_ios21:11:253
17801996016,38cyclictest0-21swapper/322:21:193
17801996014,39cyclictest4842-21df_abs22:31:203
17801996014,33cyclictest0-21swapper/300:16:183
135082603,11sleep00-21swapper/022:50:010
1780199597,45cyclictest15871-21gltestperf00:01:183
17801995930,21cyclictest663-21ntp_kernel_err20:06:353
17801995916,25cyclictest28132-21latency22:11:253
1780099593,45cyclictest0-21swapper/219:55:222
1780199584,47cyclictest0-21swapper/323:21:363
1780199584,47cyclictest0-21swapper/323:21:363
1780199584,47cyclictest0-21swapper/323:11:013
1780199584,47cyclictest0-21swapper/320:41:423
1780199584,47cyclictest0-21swapper/300:30:563
1780199584,46cyclictest0-21swapper/320:41:073
17801995814,38cyclictest9710-21munin-node21:31:383
17801995812,41cyclictest0-21swapper/300:11:083
17801995812,40cyclictest0-21swapper/323:55:583
1780199574,46cyclictest0-21swapper/320:46:253
1780199574,46cyclictest0-21swapper/319:55:003
1780199574,45cyclictest0-21swapper/320:04:523
17801995722,29cyclictest31726-21sort23:26:233
1780199564,45cyclictest0-21swapper/323:31:133
1780199564,45cyclictest0-21swapper/321:31:043
1780199564,45cyclictest0-21swapper/319:24:223
17801995631,17cyclictest12674-21runrttasks21:41:133
17801995630,19cyclictest23977-21runrttasks23:11:163
1780199563,46cyclictest0-21swapper/320:21:073
1780199563,46cyclictest0-21swapper/319:36:093
17801995611,23cyclictest0-21swapper/319:04:033
17800995612,37cyclictest0-21swapper/221:21:252
17799995639,11cyclictest24335-21gltestperf20:56:251
268952553,9sleep00-21swapper/021:01:290
17801995533,13cyclictest27948-21runrttasks21:05:103
17801995532,15cyclictest23099-21runrttasks19:42:103
17801995532,15cyclictest23099-21runrttasks19:42:103
17801995532,14cyclictest22477-21runrttasks19:39:533
17801995518,29cyclictest1369-21runrttasks20:27:343
17801995518,29cyclictest1369-21runrttasks20:27:343
17801995517,32cyclictest5594-21aten2.4-expect23:41:143
17800995510,40cyclictest0-21swapper/222:31:342
73462543,44sleep17460-21perl22:36:201
73462543,44sleep17460-21perl22:36:201
270262543,43sleep10-21swapper/122:11:141
200492543,7sleep020071-21python23:01:290
1780199545,44cyclictest0-21swapper/320:22:333
1780199544,43cyclictest0-21swapper/321:57:103
17801995424,25cyclictest3143-21sort20:11:333
17801995414,33cyclictest11021-21cut21:36:203
1780099547,41cyclictest0-21swapper/221:36:172
1780099544,34cyclictest0-21swapper/223:56:322
1780099543,44cyclictest1369-21runrttasks21:06:202
1780099543,44cyclictest1369-21runrttasks21:06:202
17800995414,34cyclictest8747-21grep21:31:222
17799995430,11cyclictest0-21swapper/123:11:101
1779999542,44cyclictest12655-21telnet23:56:111
99772533,44sleep10-21swapper/123:50:591
314172533,9sleep00-21swapper/022:19:140
264512533,9sleep00-21swapper/022:06:410
195672533,43sleep10-21swapper/119:16:041
1780199539,39cyclictest0-21swapper/319:51:093
17801995331,15cyclictest14526-21runrttasks23:57:463
17801995331,13cyclictest22404-21runrttasks20:51:343
17801995330,14cyclictest4928-21runrttasks23:37:313
17801995324,19cyclictest18396-21latency_hist23:01:013
17801995318,28cyclictest0-21swapper/321:10:553
17801995318,28cyclictest0-21swapper/321:10:553
17800995321,17cyclictest0-21swapper/221:56:232
1780099532,45cyclictest32721-21turbostat23:30:592
17801995230,14cyclictest19451-21runrttasks19:14:293
17801995220,26cyclictest0-21swapper/319:16:353
17801995218,28cyclictest0-21swapper/320:31:143
1780099529,27cyclictest1369-21runrttasks19:59:552
17800995235,10cyclictest23708-21grep23:11:132
1779799524,42cyclictest12628-21munin-node23:56:300
1779799524,30cyclictest0-21swapper/019:56:080
1779799523,44cyclictest41ktimersoftd/020:36:340
1779799523,43cyclictest1243-21snmpd20:32:340
74052512,43sleep10-21swapper/121:27:411
289272512,9sleep00-21swapper/000:28:450
217492512,42sleep10-21swapper/100:13:311
211692512,43sleep10-21swapper/119:28:261
17801995130,12cyclictest18985-21munin-run19:11:103
17801995125,18cyclictest23463-21runrttasks20:56:123
1780099519,27cyclictest24728-21aten_r1power_fr22:06:152
1780099514,40cyclictest0-21swapper/220:06:512
1780099514,34cyclictest0-21swapper/200:01:312
17800995135,10cyclictest28154-21tail00:26:222
17800995135,10cyclictest26324-21unixbench_singl00:21:342
1780099513,42cyclictest0-21swapper/223:41:312
1780099513,42cyclictest0-21swapper/222:21:342
17800995129,10cyclictest0-21swapper/220:49:522
17800995113,22cyclictest1369-21runrttasks21:42:022
17800995111,34cyclictest14671-21sed22:51:202
1779999514,40cyclictest0-21swapper/100:10:581
1779999512,43cyclictest18671-21munin-node20:46:351
1779999512,43cyclictest16232-21munin-node22:56:311
1779799515,29cyclictest0-21swapper/023:16:370
1779799515,28cyclictest0-21swapper/022:27:530
1779799515,28cyclictest0-21swapper/020:57:390
1779799513,44cyclictest0-21swapper/021:06:310
1779799513,44cyclictest0-21swapper/021:06:310
1779799513,43cyclictest1243-21snmpd21:57:490
17797995121,12cyclictest0-21swapper/021:46:200
74522502,41sleep10-21swapper/123:44:101
292352503,41sleep00-21swapper/020:01:050
231642503,9sleep00-21swapper/020:55:160
228492503,9sleep00-21swapper/023:07:010
218322503,40sleep10-21swapper/121:58:481
183372503,40sleep00-21swapper/019:05:400
1780099507,12cyclictest2637-21taskset21:17:422
1780099506,35cyclictest0-21swapper/223:21:382
1780099506,35cyclictest0-21swapper/223:21:382
17800995035,9cyclictest7008-21tr22:36:152
17800995035,9cyclictest7008-21tr22:36:152
17800995016,27cyclictest27878-21sed22:11:212
1779999503,41cyclictest1243-21snmpd22:52:571
1779999502,42cyclictest27096-21munin-node19:56:371
1779799506,27cyclictest0-21swapper/019:32:250
1779799506,27cyclictest0-21swapper/000:17:340
1779799505,40cyclictest0-21swapper/021:31:030
1779799505,28cyclictest0-21swapper/021:32:440
1779799505,27cyclictest0-21swapper/023:28:030
1779799505,27cyclictest0-21swapper/020:41:340
1779799505,27cyclictest0-21swapper/019:21:060
1779799504,28cyclictest0-21swapper/021:42:460
1779799503,43cyclictest0-21swapper/000:01:190
17797995022,13cyclictest0-21swapper/023:11:290
98272493,40sleep00-21swapper/023:49:040
307542493,39sleep00-21swapper/021:11:160
24972493,39sleep00-21swapper/023:32:000
186032493,39sleep10-21swapper/119:06:131
1780099495,29cyclictest0-21swapper/220:31:202
1779999493,42cyclictest0-21swapper/121:11:341
1779999493,40cyclictest0-21swapper/100:16:311
1779999493,39cyclictest30419-21munin-run21:11:051
1779999493,39cyclictest30419-21munin-run21:11:051
1779999493,39cyclictest0-21swapper/121:31:211
1779999492,42cyclictest11514-21munin-node20:31:331
1779799496,27cyclictest0-21swapper/019:06:450
1779799495,27cyclictest0-21swapper/022:32:540
1779799495,27cyclictest0-21swapper/020:17:320
1779799495,26cyclictest0-21swapper/022:37:550
1779799495,26cyclictest0-21swapper/022:37:550
1779799494,40cyclictest0-21swapper/019:30:000
1779799494,28cyclictest0-21swapper/019:22:230
1779799494,22cyclictest0-21swapper/020:01:390
17797994919,13cyclictest0-21swapper/020:06:360
156462493,39sleep00-21swapper/022:51:460
145922492,41sleep10-21swapper/121:43:051
90552482,39sleep19089-21aten_r1power_cu22:41:121
61902482,9sleep10-21swapper/122:32:531
300912482,9sleep00-21swapper/023:23:010
300912482,9sleep00-21swapper/023:23:010
243212482,9sleep10-21swapper/119:49:331
230432483,40sleep10-21swapper/123:09:321
217782483,37sleep00-21swapper/000:13:540
1780099487,6cyclictest281ktimersoftd/223:01:142
1780099486,35cyclictest0-21swapper/220:16:072
17800994820,10cyclictest21944-21latency_hist00:15:592
1779999483,40cyclictest0-21swapper/123:26:291
1779999482,41cyclictest31870-21munin-node20:06:341
1779999482,41cyclictest1243-21snmpd21:38:241
17799994819,8cyclictest11449-21cat20:31:071
17799994819,8cyclictest11449-21cat20:31:071
1779799484,27cyclictest0-21swapper/022:02:490
1779799484,27cyclictest0-21swapper/020:47:370
1779799484,27cyclictest0-21swapper/020:12:310
1779799484,26cyclictest0-21swapper/019:39:220
1779799483,13cyclictest0-21swapper/023:51:270
1779799482,24cyclictest0-21swapper/022:12:220
17797994818,12cyclictest0-21swapper/021:54:220
17797994818,12cyclictest0-21swapper/021:54:220
159382482,40sleep10-21swapper/120:39:461
111352483,39sleep00-21swapper/022:45:090
243502472,39sleep00-21swapper/019:49:560
228032472,38sleep10-21swapper/119:41:111
228032472,38sleep10-21swapper/119:41:111
1780099478,34cyclictest0-21swapper/219:21:092
1780099478,12cyclictest0-21swapper/223:16:372
1780099477,16cyclictest0-21swapper/221:31:032
1780099476,36cyclictest0-21swapper/222:46:312
1780099476,34cyclictest30980-21switchtime20:01:412
1780099476,34cyclictest0-21swapper/221:51:162
1780099476,34cyclictest0-21swapper/221:51:162
1780099476,15cyclictest0-21swapper/200:06:272
1780099475,35cyclictest11059-21tr23:51:182
1780099474,37cyclictest0-21swapper/221:01:532
17800994733,9cyclictest9984-21gltestperf20:26:262
17800994733,9cyclictest9984-21gltestperf20:26:262
17800994733,9cyclictest22364-21aten2.4-expect22:01:122
17800994732,10cyclictest15660-21head20:36:382
1780099473,34cyclictest3054-21munin-node23:36:312
1780099472,28cyclictest1243-21snmpd22:27:532
17799994732,9cyclictest25449-21tail00:21:211
1779999473,40cyclictest0-21swapper/122:20:341
1779999473,39cyclictest0-21swapper/120:01:591
1779999473,38cyclictest0-21swapper/121:55:221
1779999473,38cyclictest0-21swapper/121:55:221
1779799475,26cyclictest0-21swapper/000:07:320
1779799474,38cyclictest0-21swapper/023:41:180
1779799474,28cyclictest0-21swapper/020:21:250
1779799474,26cyclictest0-21swapper/022:56:340
17797994732,10cyclictest25203-21grep00:21:190
17797994719,12cyclictest0-21swapper/021:21:220
25612463,37sleep00-21swapper/021:16:440
243799460,5rtkit-daemon2436-21rtkit-daemon18:59:041
243799460,5rtkit-daemon2436-21rtkit-daemon18:59:041
222402462,36sleep10-21swapper/119:36:441
217092462,37sleep10-21swapper/119:32:371
1780099468,33cyclictest0-21swapper/223:06:322
1780099464,18cyclictest0-21swapper/221:51:042
1780099463,37cyclictest0-21swapper/220:16:232
1780099463,36cyclictest0-21swapper/219:41:092
1780099463,26cyclictest0-21swapper/200:16:292
1780099463,16cyclictest8272tr23:31:102
17800994621,15cyclictest1369-21runrttasks20:53:142
17799994632,9cyclictest3441-21tr21:21:171
17799994631,9cyclictest8889-21crond20:26:061
1779999463,37cyclictest0-21swapper/122:01:121
1779799464,26cyclictest0-21swapper/021:36:240
1779799464,26cyclictest0-21swapper/020:26:190
1779799464,26cyclictest0-21swapper/020:26:190
17797994620,7cyclictest23406-21latency_hist19:46:090
17797994620,7cyclictest23406-21latency_hist19:46:090
17797994618,12cyclictest0-21swapper/023:36:200
13882463,37sleep00-21swapper/022:24:140
123082462,38sleep10-21swapper/123:54:581
49702453,9sleep10-21swapper/123:38:051
300472452,37sleep10-21swapper/123:22:261
300472452,37sleep10-21swapper/123:22:261
24812452,9sleep10-21swapper/123:31:491
192492453,9sleep00-21swapper/019:11:480
1780099454,22cyclictest18978-21latency_hist19:11:102
17800994519,8cyclictest19717-1kworker/2:2H19:43:142
17800994519,8cyclictest19717-1kworker/2:2H19:43:142
17800994516,23cyclictest17992-21grep22:56:332
17800994510,30cyclictest0-21swapper/219:46:152
17799994533,8cyclictest25474-21grep22:06:221
17799994531,9cyclictest26627-21tail23:16:211
1779999453,36cyclictest0-21swapper/121:21:051
17799994517,12cyclictest0-21swapper/120:54:251
44852443,34sleep10-21swapper/120:16:211
40672442,9sleep10-21swapper/120:15:281
278492442,36sleep10-21swapper/121:03:521
218202442,9sleep20-21swapper/219:34:072
213502442,9sleep20-21swapper/219:30:522
205832442,36sleep10-21swapper/123:03:321
200502442,36sleep10-21swapper/119:19:311
17801994426,12cyclictest17848-21latency_hist19:01:103
17801994426,12cyclictest17848-21latency_hist19:01:103
1780099445,25cyclictest7861-21runrttasks23:46:102
1780099444,34cyclictest0-21swapper/222:41:132
17800994429,10cyclictest31261-21fschecks_count21:11:202
1780099442,37cyclictest0-21swapper/220:56:252
17800994415,13cyclictest0-21swapper/220:44:262
17799994429,10cyclictest32102-21cut22:21:171
17799994429,10cyclictest2613-21fschecks_time22:26:201
168922444,32sleep11243-21snmpd00:02:311
243799430,4rtkit-daemon2436-21rtkit-daemon22:46:181
193282432,9sleep20-21swapper/219:12:512
1779999433,35cyclictest0-21swapper/120:41:231
1779999432,22cyclictest0-21swapper/121:46:301
1779999423,34cyclictest0-21swapper/119:51:221
1779999423,33cyclictest1243-21snmpd19:02:201
174462424,32sleep20-21swapper/218:56:562
174462424,32sleep20-21swapper/218:56:562
17800994118,14cyclictest1369-21runrttasks19:25:332
1779999403,33cyclictest0-21swapper/119:25:411
1779799353,27cyclictest17774-21latency_hist19:01:100
1779799353,27cyclictest17774-21latency_hist19:01:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional