You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-18 - 11:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6s.osadl.org (updated Fri Sep 18, 2026 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2211221030,4sleep122113-21sh20:12:101
2903499965,24cyclictest0-21swapper/122:49:461
2903499964,70cyclictest1227-21runrttasks20:48:101
2903499944,25cyclictest0-21swapper/121:59:531
2903499936,69cyclictest16823-21cstates20:04:501
2903499915,66cyclictest24350-21ntp_states20:15:041
2903499914,64cyclictest0-21swapper/121:55:061
2903499914,25cyclictest0-21swapper/122:05:031
29034999117,11cyclictest0-21swapper/121:23:551
2903499906,63cyclictest0-21swapper/122:09:541
2903499905,22cyclictest6529-21latency_hist19:49:391
2903499904,25cyclictest24117-21gltestperf18:20:011
2903499904,22cyclictest0-21swapper/122:39:321
2903499903,61cyclictest0-21swapper/120:50:121
29034999017,10cyclictest0-21swapper/120:29:511
2903499895,64cyclictest0-21swapper/120:39:571
2903499895,14cyclictest20508-21ssh20:59:481
2903499895,14cyclictest20508-21ssh20:59:481
2903499894,17cyclictest0-21swapper/120:34:531
29034998919,9cyclictest10392-21ssh22:20:141
29034998915,12cyclictest0-21swapper/121:49:451
29034998914,15cyclictest0-21swapper/122:25:051
29034998914,15cyclictest0-21swapper/122:25:051
29034998914,13cyclictest29892-21memory19:35:021
2902999893,68cyclictest0-21swapper/022:18:500
2903499884,77cyclictest16261-21grep21:40:071
2903499884,77cyclictest16261-21grep21:40:071
2903499884,60cyclictest0-21swapper/122:19:201
2903499884,23cyclictest0-21swapper/121:14:461
2903499883,24cyclictest0-21swapper/121:46:391
2903499883,24cyclictest0-21swapper/121:46:391
2903499883,21cyclictest712-21ssh19:40:001
2903499883,21cyclictest712-21ssh19:40:001
29034998817,13cyclictest0-21swapper/121:25:061
2903499875,58cyclictest0-21swapper/119:25:011
2903499875,21cyclictest11166-21munin-node21:34:581
29034998720,9cyclictest28007-21iostat21:09:571
29034998718,9cyclictest9847-21sort17:50:031
29034998715,13cyclictest0-21swapper/120:25:051
2902999878,59cyclictest0-21swapper/021:56:410
2902999874,75cyclictest0-21swapper/021:59:450
2903499865,20cyclictest0-21swapper/121:07:391
2903499864,20cyclictest8640-21perl21:29:511
2903499863,76cyclictest0-21swapper/122:39:551
29034998613,11cyclictest0-21swapper/119:29:551
2903499855,65cyclictest0-21swapper/119:00:041
2903499855,64cyclictest0-21swapper/118:10:141
2903499855,58cyclictest0-21swapper/122:33:411
29034998523,55cyclictest10172-21cpuspeed_turbos19:54:521
29034998517,14cyclictest0-21swapper/118:05:081
2902999853,76cyclictest0-21swapper/020:27:000
2903499845,22cyclictest0-21swapper/120:01:221
2903499844,25cyclictest0-21swapper/120:21:131
2903499844,25cyclictest0-21swapper/120:21:131
2903499844,21cyclictest0-21swapper/119:53:511
29034998426,52cyclictest8101-21df_abs18:54:551
29034998414,13cyclictest0-21swapper/122:46:181
2903499834,27cyclictest0-21swapper/117:26:171
29034998325,51cyclictest21380-21unixbench_singl19:20:121
29034998325,47cyclictest6034-21fschecks_time18:49:591
2902999839,41cyclictest0-21swapper/020:39:450
2902999833,73cyclictest0-21swapper/019:33:420
2902999833,61cyclictest0-21swapper/020:07:370
2902999833,44cyclictest1227-21runrttasks22:19:470
2903499824,71cyclictest0-21swapper/117:45:061
2903499824,26cyclictest17161-21basename20:54:491
29034998217,13cyclictest0-21swapper/117:40:171
2903499814,24cyclictest0-21swapper/118:33:141
2902999814,70cyclictest0-21swapper/021:25:030
2902999814,70cyclictest0-21swapper/019:04:590
2902999813,42cyclictest0-21swapper/021:39:330
2903499805,19cyclictest0-21swapper/117:35:161
2903499804,55cyclictest0-21swapper/119:05:121
2903499804,22cyclictest0-21swapper/118:34:521
2902999804,69cyclictest32091-21munin-node19:40:070
2902999804,69cyclictest32091-21munin-node19:40:070
2902999804,40cyclictest0-21swapper/022:44:120
2902999803,70cyclictest20625-21kworker/0:020:39:320
2902999803,69cyclictest0-21swapper/022:29:470
2902999803,68cyclictest10897-21crond21:34:330
2902999803,62cyclictest23332-21ntp_states19:25:060
29029998020,39cyclictest107550irq/46-eth020:34:370
2903499794,54cyclictest0-21swapper/118:16:581
2903499794,54cyclictest0-21swapper/118:16:581
2903499794,16cyclictest0-21swapper/119:14:531
29034997923,48cyclictest31953-21cpuspeed_turbos17:29:571
2902999794,55cyclictest0-21swapper/021:07:240
2902999793,69cyclictest0-21swapper/021:40:150
2902999793,69cyclictest0-21swapper/021:40:150
2902999793,57cyclictest1157-21chrt21:19:440
29029997920,38cyclictest232431munin-node18:20:020
2903499783,20cyclictest15830-21crond18:04:411
29034997819,7cyclictest1227-21runrttasks18:42:581
2902999784,41cyclictest18639-21sed21:44:540
2902999784,41cyclictest18639-21sed21:44:540
2902999783,68cyclictest29456-21cut22:49:560
2902999783,68cyclictest0-21swapper/018:39:510
2902999783,66cyclictest3004-21ntp_kernel_pll_17:35:100
2902999783,60cyclictest1227-21runrttasks22:13:150
2902999783,58cyclictest0-21swapper/020:13:310
2902999782,57cyclictest20625-21kworker/0:020:19:510
2902999782,57cyclictest20625-21kworker/0:020:19:510
2903499774,12cyclictest0-21swapper/117:58:051
2902999774,66cyclictest0-21swapper/022:27:170
2902999774,66cyclictest0-21swapper/022:27:170
2902999773,58cyclictest0-21swapper/021:14:580
2902999772,68cyclictest27571-21forks21:09:530
2902999772,57cyclictest0-21swapper/021:52:550
29029997711,46cyclictest0-21swapper/019:35:330
2903499764,12cyclictest0-21swapper/118:45:081
29034997621,47cyclictest15598-21fschecks_time19:09:571
2902999767,52cyclictest0-21swapper/020:45:020
29029997630,40cyclictest4010-21tune2fs19:44:550
2902999763,65cyclictest0-21swapper/018:44:580
2902999763,59cyclictest0-21swapper/017:41:190
2902999763,56cyclictest0-21swapper/020:55:050
29029997620,38cyclictest21780-21diskmemload20:49:480
2903499754,21cyclictest0-21swapper/118:24:521
2902999754,54cyclictest13918-21ssh19:59:560
2902999753,64cyclictest0-21swapper/019:54:500
2902999744,63cyclictest0-21swapper/019:14:510
2902999743,65cyclictest0-21swapper/018:50:010
2902999743,64cyclictest0-21swapper/022:44:500
2902999743,64cyclictest0-21swapper/022:34:460
2902999743,64cyclictest0-21swapper/017:24:590
2902999743,41cyclictest0-21swapper/022:07:590
29029997429,39cyclictest18813-21cpuspeed_turbos18:09:550
29029997423,44cyclictest20819-21grep19:20:040
2902999734,61cyclictest7170-21diskstats19:49:530
2902999733,63cyclictest28096-21aten_r1power_fr18:29:530
2902999733,63cyclictest0-21swapper/020:14:490
2902999733,62cyclictest20668-21ssh20:59:500
2902999733,62cyclictest20668-21ssh20:59:500
29029997318,38cyclictest1124-21snmpd17:51:210
2902999723,62cyclictest0-21swapper/018:00:020
29029997224,41cyclictest32346-21tune2fs17:30:010
29029997218,43cyclictest1084-21lldpd18:15:340
29029997218,43cyclictest1084-21lldpd18:15:340
2902999714,47cyclictest30402-21munin-node18:34:500
29029997110,55cyclictest10770-21cut18:59:580
2902999703,60cyclictest0-21swapper/018:55:030
29029997026,38cyclictest6876-21cpuspeed_turbos17:44:560
29029996929,34cyclictest16921-21head18:05:010
29029996916,38cyclictest0-21swapper/018:25:010
2903499685,15cyclictest0-21swapper/117:24:431
2902999683,58cyclictest1124-21snmpd19:11:350
29029996518,40cyclictest11561-21cpuspeed_turbos17:54:550
25782633,11sleep30-21swapper/321:20:003
2903799615,47cyclictest14829-21iostat20:49:593
2903799614,50cyclictest0-21swapper/322:10:053
2903699613,49cyclictest0-21swapper/220:44:562
2903699613,40cyclictest0-21swapper/220:54:362
2903799605,47cyclictest0-21swapper/322:02:293
2903799604,48cyclictest0-21swapper/320:03:463
2903799604,48cyclictest0-21swapper/319:29:503
2902999603,52cyclictest29010-21latency_hist17:24:440
219499600,6rtkit-daemon2193-21rtkit-daemon22:04:522
2903799595,47cyclictest0-21swapper/321:04:203
2903799595,47cyclictest0-21swapper/321:04:203
2903799595,46cyclictest18330-21diskstats21:44:513
2903799595,46cyclictest18330-21diskstats21:44:513
2903799595,46cyclictest0-21swapper/320:56:333
2903799594,48cyclictest0-21swapper/320:14:503
219499590,4rtkit-daemon2193-21rtkit-daemon19:41:512
219499590,4rtkit-daemon2193-21rtkit-daemon19:41:512
2903799585,46cyclictest32687-21diskstats19:39:533
2903799585,46cyclictest32687-21diskstats19:39:533
2903799585,46cyclictest0-21swapper/321:10:403
2903799584,47cyclictest17198-21ssh21:43:313
2903799584,47cyclictest17198-21ssh21:43:313
219499580,6rtkit-daemon2193-21rtkit-daemon21:14:242
2903799574,46cyclictest0-21swapper/322:54:113
2903799572,29cyclictest0-21swapper/319:28:173
2903699573,47cyclictest0-21swapper/222:45:012
29036995723,20cyclictest0-21swapper/219:48:022
219499570,5rtkit-daemon2193-21rtkit-daemon20:07:272
219499570,4rtkit-daemon2193-21rtkit-daemon20:33:412
2903799564,46cyclictest0-21swapper/321:05:483
2903799564,45cyclictest0-21swapper/319:36:083
29036995624,12cyclictest0-21swapper/222:09:422
219499560,6rtkit-daemon2193-21rtkit-daemon19:19:512
219499560,5rtkit-daemon2193-21rtkit-daemon21:15:092
219499560,5rtkit-daemon2193-21rtkit-daemon19:55:132
2903799554,46cyclictest0-21swapper/320:11:013
2903799553,45cyclictest11762-21needreboot20:45:043
29036995527,23cyclictest4776-21gltestperf20:34:552
219499550,6rtkit-daemon2193-21rtkit-daemon20:24:492
219499550,6rtkit-daemon2193-21rtkit-daemon20:14:572
219499550,6rtkit-daemon2193-21rtkit-daemon19:04:522
219499550,5rtkit-daemon2193-21rtkit-daemon21:39:462
219499550,5rtkit-daemon2193-21rtkit-daemon21:39:462
59282543,43sleep30-21swapper/319:48:143
2903799544,43cyclictest0-21swapper/320:36:123
29036995433,7cyclictest16802-21ssh22:30:062
2903699543,43cyclictest0-21swapper/219:30:082
29036995428,10cyclictest20902-21telnet18:14:532
29036995428,10cyclictest20902-21telnet18:14:532
29036995425,23cyclictest0-21swapper/221:07:402
29036995422,13cyclictest0-21swapper/222:19:502
29036995412,25cyclictest0-21swapper/221:03:082
29036995412,25cyclictest0-21swapper/221:03:082
219499540,6rtkit-daemon2193-21rtkit-daemon22:14:372
219499540,6rtkit-daemon2193-21rtkit-daemon19:34:512
219499540,5rtkit-daemon2193-21rtkit-daemon21:35:012
219499540,5rtkit-daemon2193-21rtkit-daemon20:54:552
219499540,5rtkit-daemon2193-21rtkit-daemon19:27:292
219499540,5rtkit-daemon2193-21rtkit-daemon18:40:152
2903699534,42cyclictest0-21swapper/219:50:102
2903699533,43cyclictest0-21swapper/220:40:082
29036995324,24cyclictest0-21swapper/222:34:442
219499530,7rtkit-daemon2193-21rtkit-daemon21:49:462
219499530,6rtkit-daemon2193-21rtkit-daemon21:23:392
219499530,5rtkit-daemon2193-21rtkit-daemon21:30:092
219499530,5rtkit-daemon2193-21rtkit-daemon20:09:572
219499530,5rtkit-daemon2193-21rtkit-daemon18:35:012
219499530,5rtkit-daemon2193-21rtkit-daemon18:32:082
2903799529,25cyclictest0-21swapper/319:54:503
2903799524,42cyclictest0-21swapper/322:34:453
2903799524,42cyclictest0-21swapper/322:22:193
2903799524,42cyclictest0-21swapper/321:29:533
2903799524,42cyclictest0-21swapper/321:14:523
2903699524,41cyclictest0-21swapper/221:54:502
219499520,5rtkit-daemon2193-21rtkit-daemon21:59:532
219499520,5rtkit-daemon2193-21rtkit-daemon21:24:542
219499520,5rtkit-daemon2193-21rtkit-daemon20:20:122
219499520,5rtkit-daemon2193-21rtkit-daemon20:20:122
219499520,5rtkit-daemon2193-21rtkit-daemon18:44:532
2903799515,29cyclictest0-21swapper/320:20:083
2903799515,29cyclictest0-21swapper/320:20:083
2903799514,40cyclictest18898-21yum-updatesd-he20:07:363
2903799514,40cyclictest0-21swapper/319:49:573
29037995123,13cyclictest0-21swapper/322:25:093
29037995123,13cyclictest0-21swapper/322:25:093
2903699514,21cyclictest0-21swapper/220:00:092
29036995126,11cyclictest0-21swapper/219:19:392
2903699512,21cyclictest0-21swapper/221:48:132
2903699512,21cyclictest0-21swapper/221:48:132
268592512,10sleep30-21swapper/322:45:223
205712512,10sleep20-21swapper/218:14:232
138742513,42sleep21124-21snmpd22:26:022
138742513,42sleep21124-21snmpd22:26:022
119972512,9sleep30-21swapper/319:02:043
2903799504,23cyclictest18076-21munin-run22:34:313
2903799504,13cyclictest0-21swapper/321:24:513
29037995035,10cyclictest7032-21unixbench_singl22:15:143
2903799503,14cyclictest0-21swapper/322:04:593
29037995016,27cyclictest17484-21cpuspeed_turbos19:14:533
29036995020,25cyclictest0-21swapper/222:40:262
219499500,5rtkit-daemon2193-21rtkit-daemon18:55:222
2903799495,16cyclictest0-21swapper/320:39:563
2903799494,14cyclictest0-21swapper/321:52:353
2903799493,26cyclictest0-21swapper/321:54:443
29036994936,8cyclictest9207-21cpuspeed_turbos17:49:562
29036994933,10cyclictest32494-21grep17:30:032
29036994923,12cyclictest0-21swapper/222:51:162
29036994919,23cyclictest0-21swapper/219:09:552
311272483,10sleep20-21swapper/217:26:152
2903799484,39cyclictest0-21swapper/317:44:433
2903799484,27cyclictest0-21swapper/318:35:063
2903799484,26cyclictest0-21swapper/318:44:413
2903799484,25cyclictest0-21swapper/319:14:403
29037994827,13cyclictest0-21swapper/318:00:013
29037994820,12cyclictest0-21swapper/321:36:323
29037994819,11cyclictest1543-21irqstats20:30:003
29037994813,29cyclictest11934-21fschecks_count17:54:593
2903699482,38cyclictest13678-21munin-node18:00:022
2903799475,25cyclictest0-21swapper/322:39:593
2903799475,25cyclictest0-21swapper/318:55:003
2903799474,27cyclictest0-21swapper/317:50:093
2903799474,25cyclictest0-21swapper/318:04:523
29037994733,9cyclictest18766-21tr18:09:553
29037994718,13cyclictest0-21swapper/320:27:573
111812472,10sleep20-21swapper/219:00:052
2903799463,26cyclictest0-21swapper/317:35:123
29037994618,10cyclictest29068-21cat17:24:433
2903699466,12cyclictest0-21swapper/218:04:532
2903699462,38cyclictest0-21swapper/218:20:022
275242462,10sleep20-21swapper/218:26:282
2903799458,22cyclictest6810-21grep18:50:133
2903799454,36cyclictest0-21swapper/318:25:133
2903799454,25cyclictest0-21swapper/317:26:173
29037994514,12cyclictest0-21swapper/318:20:003
2903699453,36cyclictest0-21swapper/217:45:092
2903799444,23cyclictest22164-21proc_pri18:15:093
2903799444,23cyclictest22164-21proc_pri18:15:093
29037994429,9cyclictest20492-21sed19:19:583
29037994414,13cyclictest0-21swapper/318:46:313
2903699444,25cyclictest0-21swapper/217:40:102
144482442,10sleep30-21swapper/319:07:483
2903799434,33cyclictest0-21swapper/317:44:593
219499430,4rtkit-daemon2193-21rtkit-daemon18:49:532
2903799425,32cyclictest0-21swapper/317:30:103
29036994227,9cyclictest11910-21grep17:54:592
29036994227,10cyclictest2036-21cpuspeed_turbos17:34:572
2903799414,32cyclictest0-21swapper/318:30:123
2903699362,28cyclictest0-21swapper/217:24:442
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional