You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 19:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6s.osadl.org (updated Sat Nov 29, 2025 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2437991000,6rtkit-daemon2436-21rtkit-daemon12:19:550
2437991000,6rtkit-daemon2436-21rtkit-daemon12:19:550
13981999939,11cyclictest27900-21grep07:17:510
1398199974,13cyclictest0-21swapper/012:01:070
1398199965,29cyclictest0-21swapper/009:32:200
1398199954,27cyclictest0-21swapper/012:02:110
1398199944,30cyclictest0-21swapper/010:42:300
1398199944,30cyclictest0-21swapper/009:22:270
1398199924,30cyclictest0-21swapper/009:37:330
1398199924,29cyclictest0-21swapper/007:02:400
1398199914,14cyclictest0-21swapper/011:17:190
1398799906,78cyclictest0-21swapper/310:02:073
13985999017,55cyclictest142081telnet09:07:211
1398199904,29cyclictest0-21swapper/008:17:430
13981999021,14cyclictest0-21swapper/008:32:450
13986998919,12cyclictest10603-21ntpq08:57:362
1398199899,20cyclictest0-21swapper/010:57:050
13981998936,10cyclictest7341-21grep12:12:360
13981998931,10cyclictest8523-21tr07:47:280
1398799875,75cyclictest0-21swapper/311:22:033
1398799873,77cyclictest0-21swapper/309:17:233
1398799873,77cyclictest0-21swapper/309:17:233
1398799873,77cyclictest0-21swapper/306:52:153
13987998719,13cyclictest0-21swapper/307:47:333
1398599874,76cyclictest0-21swapper/107:32:401
13985998734,32cyclictest0-21swapper/107:22:131
1398199874,14cyclictest0-21swapper/009:42:210
13981998730,11cyclictest5215-21grep11:02:240
1398799862,26cyclictest30818-21munin-node09:42:353
1398699864,75cyclictest0-21swapper/212:17:382
1398699864,75cyclictest0-21swapper/212:17:382
13986998618,51cyclictest0-21swapper/207:40:112
1398599863,75cyclictest0-21swapper/110:46:061
13987998520,11cyclictest0-21swapper/307:55:323
13987998520,11cyclictest0-21swapper/307:55:323
1398199855,28cyclictest0-21swapper/007:27:180
13987998430,47cyclictest10708-21ls10:07:223
13987998428,49cyclictest0-21swapper/309:22:373
1398699844,74cyclictest0-21swapper/210:22:062
1398599844,72cyclictest0-21swapper/107:47:401
1398599843,8cyclictest24801-21runrttasks10:37:111
13985998422,39cyclictest104851munin-node10:07:221
1398199844,37cyclictest0-21swapper/010:27:290
1398199844,30cyclictest0-21swapper/008:17:110
13981998410,20cyclictest0-21swapper/007:46:490
1398799834,72cyclictest6495-21gltestperf12:12:233
13987998333,42cyclictest29354-21hwlatdetect09:37:313
1398599838,46cyclictest0-21swapper/111:17:021
1398599833,9cyclictest1179-21lldpd09:26:341
13981998333,9cyclictest9425-21head08:57:220
1398799824,24cyclictest15931-21fschecks_count10:17:263
13987998239,37cyclictest18961-21runrttasks11:32:163
13987998239,37cyclictest18961-21runrttasks11:32:163
13987998236,41cyclictest0-21swapper/310:42:173
13987998228,47cyclictest20470-21cpuspeed_turbos08:12:253
1398699824,72cyclictest0-21swapper/209:22:272
13986998229,47cyclictest31138-21cpuspeed_turbos09:42:222
1398599824,8cyclictest7960-21latency_hist12:17:021
1398599824,8cyclictest19776-21hddtemp_smartct09:17:301
1398599824,8cyclictest19776-21hddtemp_smartct09:17:301
1398599823,9cyclictest18794-21latency08:07:361
1398599823,8cyclictest12417-21grep09:02:271
13985998220,39cyclictest189781munin-node07:02:451
13981998228,11cyclictest25142-21cpuspeed_turbos10:37:200
1398799815,68cyclictest1369-21runrttasks10:22:203
1398699813,69cyclictest24614-21grep11:42:282
1398599814,8cyclictest31330-21diskstats11:57:211
1398599814,70cyclictest0-21swapper/107:27:441
1398599813,12cyclictest0-21swapper/110:02:301
1398599813,12cyclictest0-21swapper/110:02:301
13985998116,48cyclictest14204-21munin-node06:52:471
13981998132,9cyclictest17010-21grep08:02:440
1398799804,27cyclictest0-21swapper/307:57:443
1398799803,26cyclictest23655-21munin-node09:27:353
13987998028,43cyclictest29464-21ls07:22:353
13987998019,10cyclictest26920-21gltestperf09:32:293
13987998011,8cyclictest114550irq/46-eth008:52:113
1398699804,69cyclictest0-21swapper/212:08:392
13986998011,12cyclictest0-21swapper/207:17:292
1398599804,55cyclictest0-21swapper/107:15:071
13985998037,27cyclictest9019-21munin-run11:12:041
1398599803,9cyclictest23736-21cut09:27:211
13985998019,45cyclictest1243-21snmpd11:23:231
13985998010,49cyclictest0-21swapper/109:17:101
13981998029,9cyclictest14557-21grep10:12:390
13987997939,32cyclictest0-21swapper/309:02:213
13987997938,36cyclictest0-21swapper/310:52:053
13987997938,36cyclictest0-21swapper/310:52:053
1398799793,68cyclictest10042-21sshd10:04:273
1398799793,68cyclictest10042-21sshd10:04:273
1398799793,68cyclictest0-21swapper/312:22:013
1398799793,68cyclictest0-21swapper/312:22:013
13987997910,11cyclictest0-21swapper/308:42:403
1398699794,67cyclictest0-21swapper/208:29:262
1398699794,63cyclictest0-21swapper/211:27:372
1398699793,69cyclictest0-21swapper/209:52:212
1398699793,68cyclictest26763-21fschecks_time09:32:272
1398699793,67cyclictest1243-21snmpd10:46:572
1398599794,68cyclictest0-21swapper/108:03:321
1398599793,10cyclictest0-21swapper/110:22:321
13985997925,36cyclictest767-21munin-node09:47:361
13985997918,46cyclictest22500-21munin-node08:17:371
1398799784,69cyclictest0-21swapper/307:12:413
1398799783,20cyclictest1369-21runrttasks11:31:023
13987997824,10cyclictest21898-21fschecks_time11:37:233
1398699783,67cyclictest3795-21unixbench_multi10:57:392
13986997826,46cyclictest5773-21uname08:47:382
13986997823,49cyclictest0-21swapper/206:59:262
13986997822,50cyclictest1369-21runrttasks11:48:242
13986997822,49cyclictest1369-21runrttasks11:17:582
1398599784,68cyclictest0-21swapper/109:32:251
1398599783,13cyclictest0-21swapper/110:32:551
1398599783,13cyclictest0-21swapper/107:23:011
13985997816,46cyclictest1243-21snmpd11:37:061
1398799773,15cyclictest0-21swapper/312:02:173
13987997727,9cyclictest21127-21awk07:07:143
13987997725,47cyclictest0-21swapper/307:37:463
1398799772,10cyclictest0-21swapper/307:17:373
13987997718,9cyclictest6375-21fschecks_count07:42:313
13987997714,12cyclictest19478-21unixbench_singl08:07:463
1398699773,7cyclictest0-21swapper/210:02:262
1398699773,7cyclictest0-21swapper/210:02:262
1398699773,11cyclictest0-21swapper/209:07:272
13986997723,47cyclictest24263-21unixbench_singl08:17:452
13986997722,48cyclictest1369-21runrttasks11:13:062
13986997716,7cyclictest0-21swapper/207:55:252
13986997716,7cyclictest0-21swapper/207:55:252
1398599779,51cyclictest0-21swapper/108:25:491
1398599775,49cyclictest4478-21munin-node11:02:381
1398599774,9cyclictest8764-21perl12:17:211
1398599774,9cyclictest8764-21perl12:17:211
1398599773,8cyclictest5340-21chrt12:11:231
1398599773,8cyclictest13978-21iostat_ios07:57:351
13985997719,42cyclictest30175-21yum11:52:391
13985997715,46cyclictest28431-21munin-node09:37:241
1398799765,64cyclictest0-21swapper/310:32:193
1398799764,63cyclictest0-21swapper/311:02:173
13987997626,8cyclictest1369-21runrttasks07:33:203
13987997625,9cyclictest3939-21fschecks_time12:07:223
13987997612,13cyclictest0-21swapper/311:52:153
1398699765,11cyclictest0-21swapper/210:37:422
1398699764,65cyclictest19443-21grep08:07:462
13986997629,40cyclictest24583-21grep07:12:322
13986997623,47cyclictest1369-21runrttasks08:12:382
13986997614,12cyclictest0-21swapper/207:42:312
1398599769,58cyclictest0-21swapper/109:42:281
1398599764,11cyclictest0-21swapper/108:53:361
1398599763,9cyclictest6292-21diskstats09:57:261
1398599763,10cyclictest0-21swapper/107:37:241
13985997619,43cyclictest1243-21snmpd09:53:271
13985997616,46cyclictest10567-21munin-node07:52:431
13985997616,46cyclictest10567-21munin-node07:52:431
1398799754,64cyclictest0-21swapper/310:42:063
1398799754,32cyclictest27698-21munin-node08:27:263
13987997524,8cyclictest1369-21runrttasks09:12:553
13987997522,8cyclictest1243-21snmpd11:27:043
13987997521,9cyclictest3174-21latency10:57:293
1398799752,27cyclictest0-21swapper/308:17:423
13987997512,55cyclictest15898-21unixbench_multi06:52:473
1398699753,63cyclictest29662-21munin-node08:32:232
13986997523,46cyclictest1369-21runrttasks07:35:512
13986997522,47cyclictest1369-21runrttasks12:06:262
13986997521,48cyclictest1369-21runrttasks11:59:002
1398699752,66cyclictest17668-21ls09:12:342
13986997513,13cyclictest0-21swapper/210:07:362
1398599754,50cyclictest13079-21python11:17:351
1398599753,9cyclictest877-21ps10:52:351
1398599753,7cyclictest21937-21diskstats07:07:311
1398599753,7cyclictest21937-21diskstats07:07:311
1398599753,12cyclictest164121munin-node11:27:331
13985997514,47cyclictest0-21swapper/108:41:211
13985997510,45cyclictest0-21swapper/110:22:071
1398799745,22cyclictest1369-21runrttasks09:02:073
1398799744,64cyclictest0-21swapper/311:42:243
1398799744,64cyclictest0-21swapper/309:47:223
1398799743,13cyclictest0-21swapper/306:58:293
13987997426,10cyclictest7608-21fschecks_time11:07:263
13987997421,9cyclictest29892-21cpuspeed08:32:253
13987997421,8cyclictest14601-21unixbench_multi10:12:393
13987997420,10cyclictest1140-21timerandwakeup10:52:393
13987997412,12cyclictest0-21swapper/310:27:213
1398699744,11cyclictest0-21swapper/209:02:232
13986997427,41cyclictest20847-21gltestperf10:27:272
13986997425,44cyclictest0-21swapper/211:52:222
13986997424,43cyclictest1369-21runrttasks12:16:412
13986997423,45cyclictest1369-21runrttasks11:03:232
13986997423,44cyclictest0-21swapper/207:08:052
13986997423,44cyclictest0-21swapper/207:08:052
13986997422,46cyclictest896-21open_inodes08:37:382
13986997422,46cyclictest1369-21runrttasks09:59:592
13986997421,47cyclictest22488-21open_inodes11:37:322
1398699742,66cyclictest0-21swapper/209:27:202
13986997413,13cyclictest0-21swapper/211:07:212
1398599743,10cyclictest0-21swapper/107:45:121
13985997419,39cyclictest4110-21taskset08:44:331
13985997418,35cyclictest16713-21telnet06:57:271
13985997414,40cyclictest22199-21sshd08:16:241
1398799739,13cyclictest0-21swapper/308:22:233
1398799732,11cyclictest0-21swapper/311:12:163
13987997315,14cyclictest0-21swapper/308:02:263
13987997313,8cyclictest26138-21aten2.4-expect11:47:183
13987997312,13cyclictest0-21swapper/308:52:123
1398699736,9cyclictest114550irq/46-eth010:22:252
1398699734,24cyclictest0-21swapper/209:39:272
1398699734,24cyclictest0-21swapper/207:57:312
1398699734,13cyclictest0-21swapper/207:02:282
13986997322,44cyclictest19235-21df11:32:192
13986997322,44cyclictest19235-21df11:32:192
13986997312,11cyclictest6936-21sendmail08:52:112
1398599733,8cyclictest29841-21cpu08:32:251
1398599732,9cyclictest0-21swapper/112:05:451
13985997317,48cyclictest930-21ntpd10:57:331
13985997315,51cyclictest31134-21timerandwakeup10:47:381
13985997315,51cyclictest31134-21timerandwakeup10:47:381
1398799725,56cyclictest0-21swapper/311:57:093
1398799725,13cyclictest574-21runrttasks07:32:043
1398799723,30cyclictest21360-21munin-node07:07:343
1398799723,30cyclictest21360-21munin-node07:07:343
1398799723,13cyclictest0-21swapper/309:07:423
13987997225,9cyclictest1060-21python08:37:413
13987997213,13cyclictest0-21swapper/309:52:273
1398699723,24cyclictest0-21swapper/208:27:112
13986997223,43cyclictest16304-21if_err_eth008:02:322
13986997223,43cyclictest1369-21runrttasks10:12:122
13986997222,44cyclictest3442-21open_files08:42:392
13986997221,45cyclictest31541-21tune2fs07:27:302
1398699722,63cyclictest29603-21munin-node10:47:262
1398699722,63cyclictest29603-21munin-node10:47:262
13986997217,7cyclictest0-21swapper/209:49:152
1398599724,60cyclictest0-21swapper/111:42:371
1398599724,48cyclictest0-21swapper/110:12:301
1398599723,9cyclictest0-21swapper/111:51:251
1398599723,8cyclictest18812-21aten_r1power_cu11:32:151
1398599723,8cyclictest18812-21aten_r1power_cu11:32:151
1398599723,62cyclictest0-21swapper/108:27:261
1398699713,20cyclictest0-21swapper/206:52:462
13986997123,42cyclictest1369-21runrttasks07:50:422
13986997122,43cyclictest1369-21runrttasks09:20:172
13986997122,43cyclictest1369-21runrttasks09:20:172
13986997121,42cyclictest32215-21sort10:52:202
1398599714,49cyclictest20037-21munin-node10:27:301
1398599713,59cyclictest0-21swapper/108:47:271
13986997023,41cyclictest29863-21proc_pri07:22:422
1398599703,12cyclictest0-21swapper/106:52:141
13985997017,39cyclictest0-21swapper/108:57:211
1398699694,23cyclictest0-21swapper/210:37:062
13986996912,14cyclictest0-21swapper/211:27:042
1398199694,16cyclictest19225-21grep09:17:240
1398199694,16cyclictest19225-21grep09:17:240
1398199655,17cyclictest0-21swapper/010:17:410
13986996411,8cyclictest140742grep06:52:152
1398199593,50cyclictest0-21swapper/010:47:400
1398199593,50cyclictest0-21swapper/010:47:400
13981995840,12cyclictest7203-21grep11:07:210
13981995840,10cyclictest22940-21grep11:37:380
1398199584,27cyclictest0-21swapper/010:01:490
232532574,46sleep00-21swapper/007:08:590
232532574,46sleep00-21swapper/007:08:590
13981995640,11cyclictest1813-21wc07:32:350
13981995537,11cyclictest23743-21munin-node07:12:330
269922544,42sleep00-21swapper/008:26:380
13981995435,12cyclictest2728-21grep10:57:230
42712538,38sleep00-21swapper/008:46:370
304952534,41sleep00-21swapper/007:24:590
13981995334,11cyclictest19409-21grep10:22:410
13981995235,9cyclictest1339-21unixbench_multi08:37:430
13981995222,16cyclictest0-21swapper/006:52:420
88662513,41sleep00-21swapper/008:54:110
52112515,39sleep00-21swapper/009:53:390
13981995129,16cyclictest0-21swapper/012:07:180
186692503,41sleep00-21swapper/007:01:250
287242496,37sleep00-21swapper/008:27:430
1398199494,38cyclictest0-21swapper/011:52:300
13981994929,12cyclictest23133-21fschecks_time10:32:250
112962484,39sleep00-21swapper/011:15:490
28312473,38sleep00-21swapper/009:48:480
1398199476,24cyclictest11746-21aten2.4-expect09:02:200
13981994723,15cyclictest0-21swapper/008:47:250
1398199465,14cyclictest0-21swapper/007:40:110
1398199464,36cyclictest0-21swapper/011:32:190
1398199464,36cyclictest0-21swapper/011:32:190
1398199464,35cyclictest0-21swapper/010:07:310
1398199464,34cyclictest0-21swapper/011:27:040
1398199464,34cyclictest0-21swapper/009:12:100
1398199464,27cyclictest0-21swapper/009:12:210
1398199464,14cyclictest0-21swapper/009:27:420
1398199464,13cyclictest0-21swapper/008:07:330
13981994629,10cyclictest24446-21gltestperf11:42:240
136912464,36sleep00-21swapper/006:48:530
100872462,38sleep00-21swapper/010:05:040
100872462,38sleep00-21swapper/010:05:040
278842453,37sleep00-21swapper/011:48:540
1398199454,12cyclictest0-21swapper/011:27:170
1398199444,25cyclictest0-21swapper/007:52:270
1398199444,25cyclictest0-21swapper/007:52:270
1398199444,13cyclictest0-21swapper/008:02:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional