You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 03:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot6s.osadl.org (updated Tue Jul 15, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
123032850,6sleep312302-21expr21:14:273
125192760,8sleep00-21swapper/022:19:440
27399996529,15cyclictest0-21swapper/121:14:111
27400996233,20cyclictest6147-21grep20:59:422
27397995933,12cyclictest0-21swapper/023:09:430
325972574,8sleep232623-21switchtime19:39:482
2739999573,49cyclictest0-21swapper/118:37:401
27397995732,12cyclictest0-21swapper/020:34:350
2740199564,43cyclictest27455-21munin-node22:54:253
223099560,5rtkit-daemon2229-21rtkit-daemon22:34:402
136582563,6sleep313708-21diskstats22:24:283
44162552,9sleep00-21swapper/018:41:250
2740199554,42cyclictest16808-21munin-node19:09:283
2740099554,10cyclictest25387-21tr22:49:232
27397995530,11cyclictest0-21swapper/020:09:150
264942553,7sleep026507-21python22:49:400
205892553,13sleep30-21swapper/321:29:473
129332554,45sleep20-21swapper/223:29:312
30622543,44sleep20-21swapper/219:44:512
2740199544,43cyclictest16753-21munin-node21:24:423
27401995437,10cyclictest18828-21gltestperf20:19:333
27400995437,11cyclictest4517-21grep19:49:382
2739999549,25cyclictest0-21swapper/123:39:331
2739999549,25cyclictest0-21swapper/123:39:331
2739999544,44cyclictest0-21swapper/123:22:021
2739999544,44cyclictest0-21swapper/120:31:451
2739999544,44cyclictest0-21swapper/119:48:111
2739999543,44cyclictest0-21swapper/122:34:241
223099540,5rtkit-daemon2229-21rtkit-daemon23:19:242
223099540,5rtkit-daemon2229-21rtkit-daemon22:19:342
223099540,5rtkit-daemon2229-21rtkit-daemon21:44:353
223099540,5rtkit-daemon2229-21rtkit-daemon19:24:332
223099540,5rtkit-daemon2229-21rtkit-daemon18:59:342
2740199533,43cyclictest12040-21munin-node18:59:283
2740199533,42cyclictest9523-21munin-node23:24:303
2740199533,42cyclictest1160-21snmpd18:44:533
2740199533,42cyclictest1160-21snmpd18:44:533
2740199533,41cyclictest1160-21snmpd20:25:163
2740099535,20cyclictest149402sleep221:19:292
27399995324,12cyclictest0-21swapper/122:09:241
26802533,43sleep00-21swapper/021:59:420
258262532,44sleep30-21swapper/321:41:163
223099530,5rtkit-daemon2229-21rtkit-daemon22:39:252
223099530,5rtkit-daemon2229-21rtkit-daemon22:29:332
223099530,5rtkit-daemon2229-21rtkit-daemon19:34:553
223099530,5rtkit-daemon2229-21rtkit-daemon19:19:112
223099530,5rtkit-daemon2229-21rtkit-daemon18:49:342
223099530,5rtkit-daemon2229-21rtkit-daemon18:34:502
223099530,4rtkit-daemon2229-21rtkit-daemon19:48:543
105942533,43sleep10-21swapper/118:54:401
8222522,43sleep00-21swapper/019:44:010
285882522,43sleep00-21swapper/020:39:430
2740199524,41cyclictest1628-21diskstats21:59:273
2740199524,40cyclictest17705-21iostat_ios23:39:333
2740199524,40cyclictest17705-21iostat_ios23:39:333
2740199523,42cyclictest32393-21munin-node18:34:483
2740199523,42cyclictest13187-21munin-node20:09:473
2740199523,42cyclictest13187-21munin-node20:09:473
2740199523,41cyclictest31074-21munin-node21:54:243
2740199523,41cyclictest27541-21munin-node20:39:293
2740199522,39cyclictest0-21swapper/318:27:343
27400995236,10cyclictest26693-21tr19:29:312
27400995236,10cyclictest26693-21tr19:29:312
2740099523,22cyclictest0-21swapper/223:34:362
27399995224,13cyclictest0-21swapper/121:19:251
236172522,9sleep00-21swapper/019:22:370
223099520,5rtkit-daemon2229-21rtkit-daemon23:14:242
223099520,5rtkit-daemon2229-21rtkit-daemon22:59:242
223099520,5rtkit-daemon2229-21rtkit-daemon18:29:502
223099520,4rtkit-daemon2229-21rtkit-daemon21:49:353
221022522,43sleep30-21swapper/322:39:513
67772512,9sleep30-21swapper/321:02:023
43362512,9sleep20-21swapper/220:56:212
43362512,9sleep20-21swapper/220:56:212
289002512,42sleep00-21swapper/019:34:300
2740199519,36cyclictest14357-21telnet23:34:203
2740199514,39cyclictest8419-21munin-node19:59:403
2740199514,39cyclictest16854-21ntp_states20:14:443
2740199513,41cyclictest7268-21munin-node18:49:443
2740199513,41cyclictest20220-21iostat_ios19:14:423
2740199513,41cyclictest19649-21entropy23:44:243
2740199513,41cyclictest1160-21snmpd23:20:443
2740199513,41cyclictest1160-21snmpd19:40:023
2740199513,40cyclictest2342-21munin-node23:09:243
2740099513,33cyclictest0-21swapper/219:34:302
2739999514,40cyclictest0-21swapper/123:14:241
27399995118,11cyclictest0-21swapper/121:39:111
27399995118,11cyclictest0-21swapper/121:39:111
27397995120,26cyclictest18878-21http_loadtime20:19:340
268012513,42sleep326371-21munin-node19:29:333
268012513,42sleep326371-21munin-node19:29:333
258102513,42sleep10-21swapper/122:49:291
223099510,5rtkit-daemon2229-21rtkit-daemon22:54:322
223099510,5rtkit-daemon2229-21rtkit-daemon19:09:412
223099510,5rtkit-daemon2229-21rtkit-daemon19:04:342
175022512,9sleep00-21swapper/022:32:070
149832512,9sleep00-21swapper/022:25:220
142422512,42sleep00-21swapper/019:04:190
127122513,42sleep10-21swapper/120:06:111
91062502,9sleep00-21swapper/018:50:160
81192502,9sleep30-21swapper/319:58:313
55142502,43sleep00-21swapper/019:50:420
295842503,40sleep10-21swapper/118:26:561
2740199504,39cyclictest22672-21munin-node22:44:433
2740199503,41cyclictest30989-21sed22:59:333
2740199503,40cyclictest29927-21munin-node20:44:313
2740199503,40cyclictest1160-21snmpd22:30:543
2740199503,40cyclictest0-21swapper/321:09:133
2740199503,39cyclictest0-21swapper/318:29:333
2740199502,43cyclictest0-21swapper/318:39:483
27400995036,9cyclictest4260-21unixbench_singl18:39:522
2740099503,23cyclictest19422-21python20:19:442
27399995023,8cyclictest3458-21cat22:04:101
237062502,42sleep20-21swapper/219:23:482
223099500,5rtkit-daemon2229-21rtkit-daemon21:19:433
223099500,5rtkit-daemon2229-21rtkit-daemon20:55:523
223099500,5rtkit-daemon2229-21rtkit-daemon20:55:523
223099500,4rtkit-daemon2229-21rtkit-daemon22:24:412
175182502,42sleep00-21swapper/020:16:250
163582503,9sleep10-21swapper/119:06:331
139812502,42sleep00-21swapper/023:33:120
92312492,41sleep00-21swapper/021:07:520
79902492,9sleep30-21swapper/322:12:473
69092493,40sleep10-21swapper/121:03:461
68882492,40sleep20-21swapper/218:47:282
68882492,40sleep20-21swapper/218:47:282
48312494,38sleep00-21swapper/022:04:380
42722492,41sleep00-21swapper/020:55:310
42722492,41sleep00-21swapper/020:55:310
274272493,9sleep10-21swapper/121:44:361
2740199495,36cyclictest0-21swapper/321:34:253
2740199495,36cyclictest0-21swapper/321:34:253
2740199495,28cyclictest0-21swapper/322:35:553
2740199493,40cyclictest0-21swapper/323:52:493
2740199493,39cyclictest5249-21df_abs23:14:233
27401994919,13cyclictest0-21swapper/319:04:553
2740099494,28cyclictest0-21swapper/221:05:402
2740099494,27cyclictest0-21swapper/223:04:192
27400994934,10cyclictest25447-21cpuspeed_turbos20:34:292
27400994919,12cyclictest0-21swapper/222:09:232
2739999495,38cyclictest0-21swapper/119:19:271
2739799494,29cyclictest0-21swapper/022:44:200
27397994922,21cyclictest0-21swapper/020:30:570
27397994921,10cyclictest18961-21latency_hist19:14:170
260572493,9sleep10-21swapper/119:28:181
234342493,40sleep10-21swapper/123:52:031
233412492,41sleep00-21swapper/023:50:490
210012493,9sleep10-21swapper/121:30:341
198592492,41sleep00-21swapper/022:36:420
188312493,40sleep10-21swapper/119:12:421
104252492,9sleep20-21swapper/220:02:292
77212482,39sleep00-21swapper/022:09:460
2740199485,37cyclictest0-21swapper/322:19:233
2740199485,37cyclictest0-21swapper/319:24:433
2740199484,37cyclictest10807-21munin-node20:04:413
2740099484,27cyclictest0-21swapper/223:24:262
2740099484,26cyclictest0-21swapper/220:40:482
27400994834,9cyclictest6822-21tune2fs19:54:362
27400994834,9cyclictest653-21gltestperf20:49:332
27400994834,9cyclictest653-21gltestperf20:49:332
27400994833,7cyclictest0-21swapper/221:24:252
2740099483,39cyclictest27645-21telnet18:24:302
2740099483,28cyclictest0-21swapper/223:39:452
2740099483,28cyclictest0-21swapper/223:39:452
2740099483,28cyclictest0-21swapper/221:44:322
27399994829,13cyclictest0-21swapper/120:54:341
27399994829,13cyclictest0-21swapper/120:54:341
27399994820,11cyclictest0-21swapper/122:30:541
27399994820,11cyclictest0-21swapper/118:59:561
27399994818,11cyclictest0-21swapper/120:25:151
27399994813,13cyclictest0-21swapper/120:19:271
27397994829,9cyclictest0-21swapper/020:49:590
27397994829,9cyclictest0-21swapper/020:49:590
27397994828,10cyclictest0-21swapper/021:39:320
27397994822,21cyclictest0-21swapper/019:29:030
247602483,39sleep10-21swapper/122:48:241
212002483,40sleep10-21swapper/119:17:141
199172482,39sleep00-21swapper/019:14:390
193312483,38sleep10-21swapper/123:44:221
176882483,40sleep10-21swapper/120:18:361
151202483,38sleep10-21swapper/122:27:081
150842483,38sleep10-21swapper/120:11:021
150842483,38sleep10-21swapper/120:11:021
133062483,7sleep113383-21munin-node21:14:421
115572482,9sleep30-21swapper/318:56:083
104782483,9sleep10-21swapper/122:18:501
68272472,39sleep00-21swapper/023:18:170
57002472,9sleep30-21swapper/319:53:073
32112472,39sleep00-21swapper/019:46:470
2740199475,27cyclictest0-21swapper/320:34:343
2740199473,38cyclictest0-21swapper/320:32:563
2740099474,28cyclictest0-21swapper/222:00:492
2740099474,26cyclictest0-21swapper/222:44:302
2740099474,26cyclictest0-21swapper/221:39:232
27400994733,9cyclictest23123-21cpuspeed_turbos20:29:262
2740099473,27cyclictest0-21swapper/223:44:182
27400994718,23cyclictest0-21swapper/221:09:192
2739999476,36cyclictest0-21swapper/120:49:471
2739999476,36cyclictest0-21swapper/120:49:471
2739999475,26cyclictest0-21swapper/123:01:151
2739999475,26cyclictest0-21swapper/122:24:111
2739999475,25cyclictest0-21swapper/121:06:481
2739999473,27cyclictest0-21swapper/122:04:281
2739999473,24cyclictest0-21swapper/118:42:111
27399994722,12cyclictest0-21swapper/119:59:451
27399994720,11cyclictest0-21swapper/123:14:091
2739799478,24cyclictest0-21swapper/022:17:090
269922472,9sleep30-21swapper/322:51:243
223099470,5rtkit-daemon2229-21rtkit-daemon22:19:042
212032472,9sleep00-21swapper/021:33:110
139602472,9sleep20-21swapper/221:17:052
67302463,38sleep10-21swapper/118:45:251
67302463,38sleep10-21swapper/118:45:251
5002463,9sleep10-21swapper/119:39:551
319762462,39sleep00-21swapper/018:32:020
318842463,9sleep10-21swapper/120:46:491
317362462,37sleep00-21swapper/023:00:560
285062462,9sleep00-21swapper/019:34:010
285062462,9sleep00-21swapper/019:34:010
2740099464,23cyclictest1265-21runrttasks20:09:152
27400994632,9cyclictest16345-21tune2fs20:14:352
27400994631,9cyclictest13995-21gltestperf20:09:342
27400994631,9cyclictest13995-21gltestperf20:09:342
27400994630,11cyclictest30270-21tr20:44:292
2740099463,36cyclictest0-21swapper/221:34:392
2740099463,36cyclictest0-21swapper/221:34:392
27400994618,13cyclictest0-21swapper/221:29:292
27399994621,8cyclictest29482-21cut19:34:341
27399994619,11cyclictest0-21swapper/123:29:301
2739799465,37cyclictest0-21swapper/018:27:130
2739799464,37cyclictest0-21swapper/020:24:450
2739799464,36cyclictest0-21swapper/021:49:410
261032464,7sleep00-21swapper/018:19:380
216802463,7sleep321772-21cpu19:19:293
209912462,38sleep00-21swapper/023:46:180
187802462,38sleep10-21swapper/121:27:391
185232462,38sleep00-21swapper/023:40:130
185232462,38sleep00-21swapper/023:40:130
116902462,9sleep30-21swapper/321:13:513
114322462,9sleep00-21swapper/023:26:030
80442452,9sleep00-21swapper/019:57:310
56312453,9sleep10-21swapper/119:52:121
4582452,9sleep00-21swapper/021:55:190
306142452,9sleep20-21swapper/221:51:222
293992452,38sleep00-21swapper/022:56:330
2740199455,34cyclictest0-21swapper/323:29:193
2740199453,35cyclictest3657-21telnet22:04:233
27400994517,12cyclictest0-21swapper/223:09:392
2739999453,36cyclictest0-21swapper/123:24:171
27399994518,8cyclictest68592gltestperf19:54:371
27399994518,12cyclictest0-21swapper/121:39:261
2739799455,35cyclictest0-21swapper/021:24:300
2739799455,34cyclictest0-21swapper/022:39:240
2739799453,38cyclictest0-21swapper/021:21:150
2739799453,26cyclictest0-21swapper/021:04:130
271792453,35sleep10-21swapper/120:37:461
213492453,36sleep121433-21missed_timers22:39:331
19222453,36sleep10-21swapper/123:08:231
115042452,37sleep20-21swapper/218:55:262
55712442,9sleep20-21swapper/222:07:222
2740199443,34cyclictest32372-21telnet20:49:263
2740199443,34cyclictest32372-21telnet20:49:263
2740199443,34cyclictest32225-21munin-node23:04:293
2740099444,25cyclictest21696-21sshd20:24:452
2740099443,23cyclictest21449-21munin-node23:49:362
27400994416,11cyclictest0-21swapper/221:54:362
2739999445,33cyclictest0-21swapper/118:54:171
2739999444,33cyclictest0-21swapper/120:44:141
2739999444,23cyclictest29990-21open_inodes21:49:391
27399994421,10cyclictest0-21swapper/119:29:401
27399994421,10cyclictest0-21swapper/119:29:401
27399994419,11cyclictest0-21swapper/121:54:311
2739799449,21cyclictest106250irq/46-eth018:44:500
2739799449,21cyclictest106250irq/46-eth018:44:500
2739799444,34cyclictest0-21swapper/021:09:260
2739799442,37cyclictest0-21swapper/020:44:210
27397994415,14cyclictest0-21swapper/023:36:370
27397994413,27cyclictest10259-21sort18:54:360
27397994412,7cyclictest0-21swapper/023:09:070
138842442,37sleep00-21swapper/019:00:240
104082442,35sleep30-21swapper/322:17:553
103092442,36sleep00-21swapper/020:00:580
2739999435,32cyclictest0-21swapper/122:54:201
2739999434,32cyclictest0-21swapper/118:24:191
27399994315,11cyclictest0-21swapper/123:34:201
2739799436,32cyclictest0-21swapper/023:19:230
2739799436,31cyclictest0-21swapper/021:14:240
27399994218,10cyclictest0-21swapper/118:29:491
2739799426,31cyclictest0-21swapper/021:34:370
2739799426,31cyclictest0-21swapper/021:34:370
2740199413,9cyclictest27520-21cat18:24:193
2739799414,10cyclictest0-21swapper/021:44:380
2739799413,33cyclictest0-21swapper/018:34:480
27397994025,11cyclictest0-21swapper/020:12:210
27397994025,11cyclictest0-21swapper/020:12:210
272982394,29sleep20-21swapper/218:23:302
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional