You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 22:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot6s.osadl.org (updated Fri Nov 28, 2025 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5248998234,39cyclictest1369-21runrttasks12:20:160
5248998225,50cyclictest0-21swapper/010:47:450
5248998131,42cyclictest0-21swapper/011:22:590
5248998127,46cyclictest0-21swapper/007:23:120
5248998127,46cyclictest0-21swapper/007:23:120
5248997830,41cyclictest23371-21aten2.4-expect08:38:180
5248997830,41cyclictest14616-21gltestperf08:18:240
5248997726,43cyclictest0-21swapper/007:08:090
5248997725,43cyclictest1369-21runrttasks09:03:220
5248997720,49cyclictest0-21swapper/007:38:290
5248997636,11cyclictest15288-21grep09:28:190
5248997628,31cyclictest0-21swapper/010:17:470
5248997625,42cyclictest29506-21cat11:03:310
5248997625,42cyclictest29506-21cat11:03:310
5248997623,31cyclictest0-21swapper/010:58:020
5248997445,22cyclictest0-21swapper/007:58:340
5248997433,36cyclictest0-21swapper/008:37:290
5248997433,35cyclictest10960-21taskset11:33:190
5248997422,34cyclictest0-21swapper/007:47:360
5248997330,37cyclictest19747-21aten_r1power_en09:38:170
5248997328,15cyclictest0-21swapper/008:43:180
5248997327,40cyclictest11342-21telnet10:28:150
5248997326,40cyclictest0-21swapper/011:13:210
5248997326,34cyclictest0-21swapper/009:13:170
5248997321,45cyclictest0-21swapper/011:53:190
5248997226,40cyclictest0-21swapper/010:28:030
5248997226,29cyclictest0-21swapper/007:28:370
5248997221,45cyclictest0-21swapper/008:14:460
5248997135,29cyclictest6603-21missed_timers06:53:360
5248997133,19cyclictest22049-21telnet11:58:090
5248997127,28cyclictest6621-21munin-node08:03:410
5248997126,38cyclictest0-21swapper/011:28:180
5248997126,38cyclictest0-21swapper/009:53:150
5248997124,40cyclictest0-21swapper/008:33:060
5248997119,45cyclictest0-21swapper/010:38:140
243799710,18rtkit-daemon2436-21rtkit-daemon09:51:163
5248997034,28cyclictest1369-21runrttasks11:42:230
5248997029,27cyclictest16832-21taskset11:45:400
5248997029,27cyclictest16832-21taskset11:45:400
5248997027,36cyclictest0-21swapper/009:23:370
5248997023,31cyclictest15847-21crond10:38:020
5248996927,35cyclictest1369-21runrttasks09:33:170
5248996927,35cyclictest1369-21runrttasks09:33:170
5248996925,26cyclictest16159-21munin-node08:23:300
5248996922,40cyclictest26798-21munin-node12:08:220
243799690,18rtkit-daemon2436-21rtkit-daemon09:22:103
5248996846,15cyclictest0-21swapper/007:53:340
5248996831,31cyclictest10168-21munin-node09:18:180
5248996829,32cyclictest741-21tr08:58:210
5248996829,32cyclictest3924-21munin-run10:13:020
5248996822,40cyclictest0-21swapper/007:37:170
5248996822,40cyclictest0-21swapper/007:08:370
243799680,7rtkit-daemon7229-21sshd06:53:461
5248996742,19cyclictest0-21swapper/008:08:340
5248996742,19cyclictest0-21swapper/008:08:340
5248996730,30cyclictest29276-21munin-node09:58:140
5248996727,34cyclictest0-21swapper/009:11:430
5248996726,34cyclictest0-21swapper/007:18:370
5248996725,35cyclictest1243-21snmpd09:43:160
5248996721,15cyclictest0-21swapper/010:03:220
5248996715,46cyclictest0-21swapper/011:23:240
5248996645,16cyclictest114550irq/46-eth008:53:070
5248996645,16cyclictest114550irq/46-eth008:53:070
5248996621,38cyclictest0-21swapper/009:48:250
5248996618,40cyclictest0-21swapper/010:18:300
5248996618,40cyclictest0-21swapper/010:18:300
5250996531,26cyclictest0-21swapper/207:33:092
5248996541,19cyclictest0-21swapper/007:52:510
5248996529,29cyclictest0-21swapper/007:13:300
5248996525,33cyclictest0-21swapper/008:53:070
5248996520,38cyclictest0-21swapper/011:48:120
5251996417,39cyclictest0-21swapper/312:02:063
5248996423,34cyclictest27689-21sshd11:01:380
5248996330,26cyclictest1369-21runrttasks10:48:260
5248996330,14cyclictest10015-21idleruntime-cro07:03:110
5248996321,36cyclictest0-21swapper/012:03:050
5248996233,22cyclictest1369-21runrttasks12:13:390
524999614,49cyclictest0-21swapper/110:33:021
5249996139,15cyclictest0-21swapper/107:33:421
5248996113,42cyclictest0-21swapper/011:08:340
5251996042,11cyclictest8573-21gltestperf11:28:223
5249996038,15cyclictest0-21swapper/107:28:351
243799600,6rtkit-daemon0-21swapper/007:12:301
243799600,4rtkit-daemon2436-21rtkit-daemon09:53:233
5251995921,13cyclictest23840-21df_inode10:53:183
5249995937,15cyclictest0-21swapper/107:18:101
525199584,28cyclictest0-21swapper/309:13:253
5251995821,13cyclictest0-21swapper/308:48:073
5251995820,13cyclictest0-21swapper/310:13:223
5248995835,16cyclictest0-21swapper/006:53:110
243799580,4rtkit-daemon2436-21rtkit-daemon11:00:363
107072583,12sleep30-21swapper/311:33:183
525199575,28cyclictest0-21swapper/312:18:303
525199575,28cyclictest0-21swapper/310:23:133
525199575,28cyclictest0-21swapper/310:18:273
525199575,28cyclictest0-21swapper/310:18:273
525199575,28cyclictest0-21swapper/308:08:393
525199575,28cyclictest0-21swapper/308:08:393
525199574,30cyclictest0-21swapper/310:38:233
525199574,29cyclictest0-21swapper/310:48:153
525199574,28cyclictest0-21swapper/311:08:273
525199574,28cyclictest0-21swapper/308:28:403
5251995722,14cyclictest0-21swapper/308:38:323
525099577,44cyclictest0-21swapper/209:53:272
525199565,28cyclictest0-21swapper/309:03:283
525199564,29cyclictest0-21swapper/312:08:203
525199564,29cyclictest0-21swapper/308:33:193
525199564,29cyclictest0-21swapper/306:53:443
525199564,28cyclictest0-21swapper/311:53:343
525199564,28cyclictest0-21swapper/310:47:293
525199564,28cyclictest0-21swapper/308:23:193
525199564,27cyclictest0-21swapper/312:13:343
525199564,27cyclictest0-21swapper/308:08:083
524999564,31cyclictest0-21swapper/110:08:371
243799560,4rtkit-daemon2436-21rtkit-daemon09:24:553
525199554,29cyclictest0-21swapper/312:03:323
525199554,27cyclictest0-21swapper/307:18:423
525199554,25cyclictest31384-21sshd07:46:453
5249995539,10cyclictest0-21swapper/109:48:161
205952553,12sleep20-21swapper/210:47:302
5251995430,14cyclictest1369-21runrttasks11:18:183
525099544,26cyclictest0-21swapper/207:07:472
5250995435,12cyclictest21480-21grep07:23:442
5250995435,12cyclictest21480-21grep07:23:442
5250995430,17cyclictest2242-21head11:13:342
525099543,44cyclictest0-21swapper/209:43:282
5249995440,8cyclictest4776-21fschecks_count10:13:221
243799540,5rtkit-daemon2436-21rtkit-daemon11:23:353
243799540,5rtkit-daemon2436-21rtkit-daemon08:58:333
178652544,12sleep20-21swapper/208:23:412
87642533,43sleep18822-21sed08:08:081
5251995312,23cyclictest0-21swapper/311:38:103
5250995336,10cyclictest12914-21grep09:23:182
5250995317,7cyclictest281ktimersoftd/211:32:592
524999535,30cyclictest0-21swapper/110:53:211
524999535,29cyclictest0-21swapper/111:03:241
524999535,29cyclictest0-21swapper/111:03:241
5249995337,11cyclictest0-21swapper/110:38:151
525199526,26cyclictest0-21swapper/308:02:223
525099524,28cyclictest0-21swapper/210:33:362
525099524,28cyclictest0-21swapper/208:53:172
525099522,43cyclictest0-21swapper/211:37:192
524999525,29cyclictest0-21swapper/109:28:351
524999524,29cyclictest0-21swapper/111:28:261
5249995221,13cyclictest0-21swapper/107:03:401
273232523,41sleep10-21swapper/109:53:201
138082523,42sleep113889-21sendmail_mailst11:38:281
525099519,8cyclictest7150irq/45-ahci10:53:252
525099514,41cyclictest0-21swapper/210:17:472
525099514,28cyclictest0-21swapper/211:43:202
525099514,28cyclictest0-21swapper/211:43:202
524999515,28cyclictest0-21swapper/107:38:221
524999515,26cyclictest0-21swapper/108:53:161
524999515,13cyclictest0-21swapper/108:37:291
524999514,29cyclictest0-21swapper/111:53:341
524999514,29cyclictest0-21swapper/108:48:201
524999514,29cyclictest0-21swapper/108:48:201
5249995130,14cyclictest0-21swapper/111:23:181
5249995130,14cyclictest0-21swapper/111:08:321
524999513,43cyclictest0-21swapper/106:58:451
243799510,5rtkit-daemon2436-21rtkit-daemon10:28:222
243799510,5rtkit-daemon2436-21rtkit-daemon10:25:302
243799510,4rtkit-daemon2436-21rtkit-daemon10:03:233
525199508,31cyclictest0-21swapper/309:43:053
525199505,28cyclictest0-21swapper/307:33:403
525199504,24cyclictest0-21swapper/307:38:183
5251995033,10cyclictest11550-21runrttasks08:13:193
5251995031,11cyclictest13462-21crond10:33:023
5251995018,8cyclictest18626-21open_inodes09:33:333
5251995018,8cyclictest18626-21open_inodes09:33:333
525099504,27cyclictest0-21swapper/210:08:402
525099504,26cyclictest0-21swapper/209:48:412
525099503,28cyclictest0-21swapper/209:58:202
525099503,27cyclictest0-21swapper/208:03:252
525099503,27cyclictest0-21swapper/207:43:222
525099503,22cyclictest0-21swapper/211:18:292
5250995026,18cyclictest13936-21egrep08:18:182
525099502,32cyclictest0-21swapper/212:22:262
524999505,28cyclictest0-21swapper/111:03:011
524999505,28cyclictest0-21swapper/108:18:381
524999505,26cyclictest0-21swapper/110:52:531
524999505,14cyclictest0-21swapper/109:13:171
524999504,39cyclictest20722-21runrttasks09:38:271
524999504,29cyclictest0-21swapper/109:18:361
524999504,29cyclictest0-21swapper/108:13:421
524999503,39cyclictest0-21swapper/111:58:091
314012502,41sleep10-21swapper/110:02:471
254092502,9sleep10-21swapper/108:42:261
208502503,39sleep20-21swapper/211:53:242
11022502,9sleep20-21swapper/210:04:262
525199495,27cyclictest0-21swapper/307:52:203
525199494,28cyclictest0-21swapper/310:33:303
525199494,27cyclictest0-21swapper/309:58:223
5251994930,12cyclictest2581-21tune2fs07:53:243
5251994930,11cyclictest15293-21cpuspeed_turbos09:28:193
5251994920,12cyclictest0-21swapper/307:32:163
525099498,15cyclictest0-21swapper/208:58:192
525099494,26cyclictest0-21swapper/211:02:542
525099494,26cyclictest0-21swapper/207:13:242
5250994932,11cyclictest6274-21sed09:08:242
525099493,27cyclictest0-21swapper/212:13:182
525099493,26cyclictest0-21swapper/208:48:202
525099493,26cyclictest0-21swapper/208:48:202
525099493,26cyclictest0-21swapper/207:18:302
524999495,37cyclictest0-21swapper/108:48:061
524999495,27cyclictest0-21swapper/110:33:251
524999495,27cyclictest0-21swapper/107:58:091
524999493,29cyclictest0-21swapper/107:23:341
524999493,29cyclictest0-21swapper/107:23:341
299072492,9sleep30-21swapper/311:04:243
299072492,9sleep30-21swapper/311:04:243
243799490,4rtkit-daemon2436-21rtkit-daemon11:48:193
242172492,40sleep10-21swapper/109:47:281
205402492,41sleep10-21swapper/110:46:491
12732492,41sleep10-21swapper/110:06:301
525199484,26cyclictest0-21swapper/307:17:143
525199483,28cyclictest0-21swapper/308:53:073
5251994820,12cyclictest0-21swapper/311:17:573
5251994819,8cyclictest14953-21cat07:13:113
525099489,21cyclictest12-21rcu_preempt09:16:422
525099484,27cyclictest0-21swapper/211:07:552
525099484,27cyclictest0-21swapper/211:07:552
525099483,27cyclictest0-21swapper/208:43:262
525099483,27cyclictest0-21swapper/208:13:342
525099483,27cyclictest0-21swapper/207:08:392
5250994818,12cyclictest0-21swapper/212:08:112
524999485,36cyclictest0-21swapper/111:38:001
524999485,27cyclictest0-21swapper/111:17:571
48472482,40sleep10-21swapper/109:04:111
198522483,39sleep319939-21aten2.4-expect07:23:243
198522483,39sleep319939-21aten2.4-expect07:23:243
525199476,34cyclictest0-21swapper/310:08:293
525199474,25cyclictest0-21swapper/309:08:293
525199470,8cyclictest29921-21yum08:48:443
525199470,8cyclictest29921-21yum08:48:443
525099474,25cyclictest0-21swapper/212:03:112
525099474,25cyclictest0-21swapper/211:58:042
525099474,25cyclictest0-21swapper/209:41:422
5250994730,11cyclictest9323-21cpuspeed_turbos08:08:212
5250994730,11cyclictest9323-21cpuspeed_turbos08:08:212
525099473,26cyclictest0-21swapper/210:48:172
525099473,25cyclictest0-21swapper/207:48:382
5250994723,7cyclictest29093-21latency_hist07:43:102
525099472,9cyclictest0-21swapper/211:23:272
524999474,26cyclictest0-21swapper/111:53:001
524999474,26cyclictest0-21swapper/106:53:111
5249994732,10cyclictest0-21swapper/108:11:061
5249994732,10cyclictest0-21swapper/108:11:061
524999473,27cyclictest0-21swapper/112:04:221
5249994717,12cyclictest0-21swapper/109:38:051
5249994717,12cyclictest0-21swapper/109:38:051
293982472,9sleep10-21swapper/107:43:211
194172472,39sleep10-21swapper/107:21:371
525199466,23cyclictest34-21rcuc/309:48:053
525199465,34cyclictest0-21swapper/307:03:443
525199464,26cyclictest0-21swapper/307:02:103
525199464,24cyclictest13861-21aten2.4-expect08:18:173
525099464,26cyclictest0-21swapper/209:28:312
5250994633,7cyclictest3041-21needreboot07:53:322
5250994631,8cyclictest3814-21grep09:03:252
525099463,23cyclictest0-21swapper/210:38:142
525099463,22cyclictest0-21swapper/210:20:492
525099463,22cyclictest0-21swapper/210:20:492
5250994620,19cyclictest11876-21grep09:18:402
525099462,10cyclictest0-21swapper/206:53:352
524999464,36cyclictest0-21swapper/108:28:321
524999464,34cyclictest0-21swapper/109:08:281
5249994631,11cyclictest0-21swapper/111:22:411
5249994616,13cyclictest0-21swapper/109:03:071
42222462,37sleep20-21swapper/207:58:182
525099455,12cyclictest0-21swapper/207:33:222
525099454,31cyclictest0-21swapper/208:38:182
525099453,34cyclictest0-21swapper/208:28:192
524999455,34cyclictest0-21swapper/112:17:571
524999455,14cyclictest0-21swapper/112:08:051
524999454,34cyclictest0-21swapper/110:18:281
524999454,34cyclictest0-21swapper/110:18:281
524999453,23cyclictest0-21swapper/109:23:171
5249994515,14cyclictest0-21swapper/112:18:111
97842442,9sleep20-21swapper/207:00:262
525199444,22cyclictest0-21swapper/311:43:123
525199444,22cyclictest0-21swapper/311:43:123
525099442,9cyclictest0-21swapper/208:37:212
525099442,10cyclictest0-21swapper/211:09:532
524999444,34cyclictest0-21swapper/107:52:221
524999444,12cyclictest0-21swapper/108:23:221
193472442,36sleep20-21swapper/211:52:202
525099434,31cyclictest0-21swapper/209:33:312
525099434,31cyclictest0-21swapper/209:33:312
525099433,34cyclictest0-21swapper/206:53:112
524999434,12cyclictest0-21swapper/108:02:221
524999433,35cyclictest0-21swapper/110:23:171
525099413,30cyclictest0-21swapper/211:38:162
524999415,21cyclictest0-21swapper/111:43:121
524999415,21cyclictest0-21swapper/111:43:121
525199404,29cyclictest0-21swapper/306:53:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional