You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 09:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot6s.osadl.org (updated Mon Nov 24, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
97899910139,55cyclictest17571-21grep22:32:470
9789999641,49cyclictest13862-21cpuspeed_turbos20:12:360
978999964,27cyclictest0-21swapper/021:13:020
978999954,27cyclictest28872-21systemd-cgroups19:37:230
978999954,27cyclictest28872-21systemd-cgroups19:37:230
9789999538,51cyclictest4713-21gltestperf00:22:310
9789999534,11cyclictest12796-21grep22:22:510
9789999534,11cyclictest12796-21grep22:22:510
979399945,14cyclictest13535-21runrttasks20:12:303
978999949,23cyclictest0-21swapper/023:51:350
979399934,13cyclictest0-21swapper/323:17:143
978999935,27cyclictest0-21swapper/019:16:130
978999934,29cyclictest0-21swapper/020:22:330
978999924,27cyclictest0-21swapper/020:17:530
978999924,27cyclictest0-21swapper/020:17:530
978999918,25cyclictest0-21swapper/000:22:120
978999914,27cyclictest0-21swapper/023:25:370
979399903,62cyclictest1243-21snmpd22:39:153
978999905,26cyclictest0-21swapper/018:58:400
978999904,29cyclictest0-21swapper/023:42:250
978999904,28cyclictest0-21swapper/022:17:170
978999904,28cyclictest0-21swapper/019:42:400
978999904,27cyclictest0-21swapper/019:06:280
9789999036,11cyclictest22372-21grep22:42:490
9789999021,13cyclictest0-21swapper/023:32:360
979299894,78cyclictest0-21swapper/200:02:342
978999895,34cyclictest0-21swapper/020:57:490
978999895,34cyclictest0-21swapper/020:57:490
978999894,26cyclictest0-21swapper/023:41:570
978999894,13cyclictest0-21swapper/023:19:210
978999894,12cyclictest0-21swapper/021:49:060
978999893,25cyclictest0-21swapper/020:37:330
9789998927,52cyclictest0-21swapper/019:51:530
979399884,13cyclictest0-21swapper/323:22:243
979399884,12cyclictest0-21swapper/322:42:283
978999885,27cyclictest0-21swapper/020:08:340
978999884,27cyclictest0-21swapper/021:44:050
978999884,27cyclictest0-21swapper/021:44:050
9789998834,11cyclictest25781-21cpuspeed_turbos22:52:290
9789998820,12cyclictest0-21swapper/021:57:550
9789998817,12cyclictest0-21swapper/023:57:320
979399875,12cyclictest0-21swapper/322:12:173
978999875,34cyclictest0-21swapper/019:22:510
979399863,60cyclictest24248-21munin-node21:42:453
979399863,60cyclictest24248-21munin-node21:42:453
979399863,13cyclictest0-21swapper/300:17:123
979399863,10cyclictest5222-21aten_r1power_vo23:17:283
979299863,77cyclictest7302-21latency_hist21:07:202
979299862,78cyclictest0-21swapper/223:12:152
978999864,27cyclictest0-21swapper/023:09:190
978999864,24cyclictest0-21swapper/022:27:270
978999864,24cyclictest0-21swapper/022:27:270
9789998633,11cyclictest10465-21tail20:02:560
979399854,16cyclictest0-21swapper/321:22:323
9793998525,52cyclictest22672-21gltestperf23:52:353
9793998525,52cyclictest22672-21gltestperf23:52:353
979299853,76cyclictest0-21swapper/223:13:092
978999854,36cyclictest0-21swapper/020:32:480
978999853,23cyclictest0-21swapper/022:47:270
9789998522,13cyclictest0-21swapper/000:07:140
87892850,3sleep3361ktimersoftd/320:02:353
979399844,63cyclictest0-21swapper/319:51:383
9793998424,52cyclictest29870-21fschecks_time19:37:423
979299843,74cyclictest2767-21munin-node20:57:412
979299843,74cyclictest2767-21munin-node20:57:412
978999845,12cyclictest0-21swapper/022:02:340
978999844,29cyclictest0-21swapper/023:12:360
978999844,25cyclictest0-21swapper/019:38:460
9789998429,11cyclictest20561-21cat21:32:420
979399835,64cyclictest0-21swapper/319:03:413
979399834,13cyclictest0-21swapper/323:42:563
9793998325,50cyclictest6071-21hwlatdetect21:02:393
9793998325,49cyclictest15043-21df_abs23:37:293
9793998324,53cyclictest12650-21cpuspeed_turbos21:17:323
9793998320,52cyclictest18959-21fschecks_count20:22:403
978999836,20cyclictest0-21swapper/021:12:200
978999835,32cyclictest0-21swapper/020:51:260
979399825,71cyclictest0-21swapper/322:32:423
979399825,13cyclictest0-21swapper/320:37:203
979399824,12cyclictest0-21swapper/319:17:243
979399824,12cyclictest0-21swapper/300:27:113
9793998224,51cyclictest3900-21missed_timers20:57:443
9793998224,51cyclictest3900-21missed_timers20:57:443
9793998223,53cyclictest22705-21fschecks_time19:22:413
9793998221,55cyclictest26676-21needreboot20:37:463
9793998221,52cyclictest1150-21hwlatdetect20:52:413
979299823,72cyclictest0-21swapper/222:27:392
979299823,72cyclictest0-21swapper/222:27:392
979299822,73cyclictest1243-21snmpd00:08:492
979199823,61cyclictest8365-21gltestperf21:07:391
978999825,16cyclictest0-21swapper/020:52:430
978999824,25cyclictest0-21swapper/019:17:560
978999824,24cyclictest0-21swapper/022:57:290
979399814,13cyclictest0-21swapper/322:17:173
9793998125,50cyclictest1696-21cpuspeed_turbos00:17:263
9793998124,51cyclictest20147-21fschecks_time23:47:323
9793998124,50cyclictest31608-21sendmail_mailqu23:02:433
9793998124,47cyclictest11446-21sendmail_mailtr18:57:563
9793998118,54cyclictest21161-21unixbench_multi21:32:513
979299814,71cyclictest0-21swapper/220:42:492
979299814,70cyclictest0-21swapper/219:37:502
979299813,71cyclictest19322-21munin-run19:17:232
979299813,70cyclictest12955-21fschecks_count21:17:362
979299812,73cyclictest0-21swapper/223:19:382
979299812,72cyclictest2843-21open_inodes22:02:452
979299812,62cyclictest0-21swapper/223:33:422
978999813,26cyclictest0-21swapper/021:17:320
979399805,17cyclictest0-21swapper/321:27:293
979399805,15cyclictest0-21swapper/319:28:073
979399804,17cyclictest0-21swapper/321:52:393
979399803,56cyclictest11159-21munin-node20:07:483
979399803,19cyclictest1369-21runrttasks20:27:343
979399802,18cyclictest0-21swapper/322:27:273
979399802,18cyclictest0-21swapper/322:27:273
979299803,70cyclictest27808-21munin-node22:57:282
979299803,70cyclictest0-21swapper/222:57:142
979299803,70cyclictest0-21swapper/221:27:292
979299803,69cyclictest1185-21sshd23:38:372
979299802,72cyclictest0-21swapper/222:42:342
979199803,52cyclictest17085-21munin-node21:27:311
978999806,16cyclictest1369-21runrttasks22:37:340
978999804,34cyclictest0-21swapper/019:52:350
979399798,63cyclictest0-21swapper/320:47:213
979399796,53cyclictest0-21swapper/319:08:423
979399795,53cyclictest0-21swapper/323:27:373
979399795,13cyclictest11245-21pluginstate21:12:493
979399794,53cyclictest0-21swapper/322:53:303
979399793,56cyclictest0-21swapper/322:02:303
9793997921,52cyclictest22538-21fschecks_count21:37:363
9793997916,53cyclictest4678-21fschecks_time19:52:403
9793997915,46cyclictest0-21swapper/322:57:283
979299794,69cyclictest0-21swapper/221:34:032
979299794,68cyclictest23006-21munin-node22:47:412
979299794,67cyclictest0-21swapper/221:47:182
979299794,67cyclictest0-21swapper/221:47:182
979299794,43cyclictest0-21swapper/200:14:592
979299793,69cyclictest23134-21basename20:32:322
979399785,19cyclictest0-21swapper/323:57:513
979399785,13cyclictest0-21swapper/319:42:363
979399784,12cyclictest0-21swapper/322:47:343
979399784,12cyclictest0-21swapper/319:37:243
979399784,12cyclictest0-21swapper/319:37:243
979399784,12cyclictest0-21swapper/300:02:393
979299783,68cyclictest9600-21latency_hist23:27:142
979299783,68cyclictest18079-21latency_hist20:22:212
979299783,68cyclictest18079-21latency_hist20:22:212
979299782,69cyclictest17810-21chrt22:34:522
9793997718,50cyclictest16264-21tr20:17:343
9793997718,50cyclictest16264-21tr20:17:343
9793997718,41cyclictest0-21swapper/322:17:283
9793997714,54cyclictest26992-21cpuspeed_turbos21:47:333
979299774,51cyclictest0-21swapper/219:22:542
979299773,47cyclictest0-21swapper/221:47:502
979299773,45cyclictest0-21swapper/222:24:252
979299773,45cyclictest0-21swapper/222:24:252
979399767,14cyclictest0-21swapper/323:32:333
979399765,13cyclictest30314-21runrttasks20:47:343
979399764,13cyclictest12287-21needreboot22:22:423
979399764,13cyclictest12287-21needreboot22:22:423
9793997615,46cyclictest0-21swapper/323:10:373
979299764,66cyclictest0-21swapper/222:07:312
979299764,65cyclictest6145-21latency_hist19:57:232
979299763,67cyclictest31858-21cstates19:42:382
979299762,46cyclictest0-21swapper/223:27:272
979199764,64cyclictest0-21swapper/120:29:201
979199763,65cyclictest10612-21fschecks_count18:57:441
979399754,65cyclictest0-21swapper/319:57:363
979399754,32cyclictest1243-21snmpd00:08:493
979399754,12cyclictest0-21swapper/321:07:553
979299754,44cyclictest0-21swapper/219:17:582
979299753,66cyclictest0-21swapper/220:02:362
979299753,65cyclictest8057-21grep22:12:522
979299753,65cyclictest12081-21munin-run21:17:192
979299753,58cyclictest13837-21python19:02:542
979299752,68cyclictest0-21swapper/221:07:322
979299752,67cyclictest0-21swapper/223:59:122
979299752,67cyclictest0-21swapper/219:07:572
979199757,53cyclictest0-21swapper/121:37:351
979199754,64cyclictest0-21swapper/123:12:411
979199754,63cyclictest0-21swapper/123:21:411
979199753,56cyclictest13245-21iostat_ios23:32:351
9793997423,43cyclictest32199-21tune2fs21:57:373
9793997413,52cyclictest0-21swapper/319:17:373
979299743,65cyclictest0-21swapper/220:17:222
979299743,64cyclictest10544-21df_abs18:57:422
979299742,57cyclictest0-21swapper/222:39:152
979199746,61cyclictest0-21swapper/119:38:461
979299733,47cyclictest0-21swapper/200:17:232
979299732,64cyclictest2521-21latency_hist20:57:202
979299732,55cyclictest0-21swapper/223:48:452
979199735,61cyclictest13933-21munin-node21:17:501
979199734,62cyclictest20997-21python19:17:541
979299723,62cyclictest22177-21cpuspeed_turbos23:52:302
979299723,62cyclictest22177-21cpuspeed_turbos23:52:302
979299722,64cyclictest0-21swapper/221:59:162
979299722,64cyclictest0-21swapper/220:28:172
979199726,58cyclictest0-21swapper/120:22:351
979199726,45cyclictest11168-21netstat21:12:471
979199724,61cyclictest7398-21iostat19:57:451
979199724,60cyclictest0-21swapper/123:42:451
979199724,41cyclictest24359-21sshd19:27:361
979199723,40cyclictest12949-21df_abs19:02:391
9791997218,32cyclictest10559-21perl23:27:321
979299713,63cyclictest23827-21nscd20:02:212
979299713,45cyclictest13032-21sshd20:08:512
979299713,44cyclictest0-21swapper/221:22:462
979299712,62cyclictest28114-21ps19:32:532
979299712,62cyclictest28114-21ps19:32:532
979299712,56cyclictest0-21swapper/221:37:302
979199714,36cyclictest0-21swapper/100:22:411
979199713,61cyclictest0-21swapper/123:02:311
243799710,5rtkit-daemon2436-21rtkit-daemon00:12:440
979299703,60cyclictest1369-21runrttasks19:52:202
979299702,62cyclictest0-21swapper/222:17:282
979299702,62cyclictest0-21swapper/220:47:422
979199706,52cyclictest0-21swapper/122:45:381
979199705,57cyclictest1243-21snmpd19:13:421
979199704,60cyclictest0-21swapper/121:59:081
979199704,60cyclictest0-21swapper/120:52:341
979199704,59cyclictest0-21swapper/122:32:371
979199704,45cyclictest24306-21open_files22:47:421
979199704,36cyclictest32355-21yum-updatesd-he20:51:371
979199704,36cyclictest0-21swapper/122:17:401
979199703,35cyclictest0-21swapper/120:17:391
979199703,35cyclictest0-21swapper/120:17:391
9791997019,33cyclictest27858-21munin-node20:42:441
979299697,55cyclictest18346-21kworker/2:020:37:332
979299694,59cyclictest0-21swapper/223:42:322
9792996937,26cyclictest30991-21grep23:02:332
979199697,53cyclictest18236-21munin-node22:37:361
979199696,52cyclictest0-21swapper/100:03:191
979199694,58cyclictest0-21swapper/100:07:321
979199694,57cyclictest0-21swapper/123:52:321
979199694,57cyclictest0-21swapper/123:52:321
979199694,57cyclictest0-21swapper/121:22:311
979199693,59cyclictest25039-21chrt23:57:361
979199693,47cyclictest19398-21munin-node23:47:441
9791996925,29cyclictest0-21swapper/122:52:271
9791996916,35cyclictest2043-21df_inode19:47:391
979299684,40cyclictest0-21swapper/220:22:342
979299683,59cyclictest25181-21gltestperf19:27:442
979299682,60cyclictest0-21swapper/200:22:332
979199684,58cyclictest0-21swapper/123:09:201
979199684,49cyclictest0-21swapper/120:32:331
979199683,59cyclictest3512-21sshd00:21:061
979399673,13cyclictest0-21swapper/318:57:253
979299674,45cyclictest9769-21latency_hist18:57:252
979199679,38cyclictest114550irq/46-eth021:42:311
979199679,38cyclictest114550irq/46-eth021:42:311
979199676,54cyclictest0-21swapper/123:01:381
979199674,57cyclictest0-21swapper/123:22:461
979299662,40cyclictest30846-21chrt21:52:522
979199666,53cyclictest0-21swapper/121:02:481
979199663,56cyclictest14779-21munin-node19:07:581
979199663,56cyclictest12180-21iostat_ios20:07:461
979199663,52cyclictest0-21swapper/119:22:341
979199663,49cyclictest3992-21munin-node19:52:441
9791996628,31cyclictest3665-21smartctl20:57:401
9791996628,31cyclictest3665-21smartctl20:57:401
9791996621,39cyclictest1369-21runrttasks00:16:141
243799660,9rtkit-daemon0-21swapper/000:11:440
979199654,54cyclictest1243-21snmpd23:38:431
979199653,55cyclictest1628-21nscd20:17:211
979199653,48cyclictest12422-21ps22:22:461
979199653,48cyclictest12422-21ps22:22:461
9791996522,29cyclictest0-21swapper/121:52:281
9791996520,39cyclictest1369-21runrttasks22:28:141
9791996520,39cyclictest1369-21runrttasks22:28:141
978999654,16cyclictest0-21swapper/021:52:300
979199648,41cyclictest7150irq/45-ahci21:37:181
979199645,45cyclictest0-21swapper/122:02:291
979199643,55cyclictest0-21swapper/122:08:281
979199643,54cyclictest912-21taskset19:43:561
979199643,54cyclictest27468-21fschecks_time19:32:431
979199643,54cyclictest27468-21fschecks_time19:32:431
979199643,41cyclictest0-21swapper/120:37:331
9791996424,33cyclictest7360-21latency22:12:431
243799640,12rtkit-daemon0-21swapper/022:07:560
243799640,11rtkit-daemon0-21swapper/023:55:370
243799640,11rtkit-daemon0-21swapper/023:55:370
979199634,39cyclictest0-21swapper/121:47:321
979199633,54cyclictest0-21swapper/120:02:351
979199553,46cyclictest0-21swapper/118:57:251
978999535,29cyclictest0-21swapper/021:02:320
9789995321,13cyclictest0-21swapper/019:57:450
978999524,29cyclictest0-21swapper/021:22:490
9789995221,14cyclictest0-21swapper/019:27:440
978999514,29cyclictest0-21swapper/019:07:470
9789995120,13cyclictest0-21swapper/021:37:360
978999505,25cyclictest0-21swapper/020:28:540
978999504,28cyclictest0-21swapper/020:42:360
978999504,27cyclictest0-21swapper/022:20:360
170832502,41sleep00-21swapper/021:27:290
978999495,27cyclictest0-21swapper/023:29:230
9789994919,12cyclictest0-21swapper/023:02:440
94202454,6sleep00-21swapper/018:53:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional