You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 08:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Mar 12, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1112328875,9sleep20-21swapper/219:06:002
1126028774,9sleep00-21swapper/019:07:470
929428561,6sleep10-21swapper/119:05:031
1121128067,9sleep30-21swapper/319:07:093
102752560,0sleep00-21swapper/000:05:510
1152299312,3cyclictest20480-21awk22:55:182
11522992926,2cyclictest16498-21cron22:14:592
1152299280,2cyclictest0-21swapper/222:20:222
1152599270,3cyclictest0-21swapper/322:51:323
1152299263,3cyclictest6808-21turbostat.cron21:49:592
1152299260,4cyclictest0-21swapper/223:45:232
1152299250,2cyclictest0-21swapper/221:22:102
1152299240,5cyclictest0-21swapper/223:27:182
1152299240,3cyclictest4314-21diskmemload23:11:262
1152299240,0cyclictest0-21swapper/223:38:202
1152099240,24cyclictest0-21swapper/121:25:581
1152299236,15cyclictest503-21rm23:40:502
11522992321,1cyclictest0-21swapper/200:32:562
1152299230,1cyclictest0-21swapper/222:45:252
1152299230,1cyclictest0-21swapper/222:26:382
1152299230,1cyclictest0-21swapper/200:16:032
1152099230,23cyclictest0-21swapper/100:02:101
1152299221,20cyclictest0-21swapper/220:55:152
1152299220,17cyclictest0-21swapper/221:30:142
1152299220,0cyclictest0-21swapper/222:15:192
11520992220,1cyclictest0-21swapper/123:45:151
1152599210,1cyclictest0-21swapper/322:21:453
1152299211,3cyclictest22296-21memory19:30:182
1152299211,2cyclictest28846-21sed23:00:182
1151599218,7cyclictest5895-21munin-run23:45:000
1151599210,1cyclictest0-21swapper/021:55:030
258132200,1sleep10-21swapper/123:55:581
1152599202,15cyclictest30191-21apt-get23:40:033
1152299201,1cyclictest0-21swapper/220:00:142
1152299201,15cyclictest0-21swapper/220:35:112
1152299200,3cyclictest0-21swapper/221:35:132
1152299200,1cyclictest0-21swapper/223:55:212
1152299200,1cyclictest0-21swapper/222:50:452
1151599200,16cyclictest0-21swapper/019:50:130
1152299198,8cyclictest8092-21sh23:07:122
1152299197,11cyclictest4314-21diskmemload22:02:012
1152299192,2cyclictest1037-21gdbus00:08:192
1152299191,2cyclictest0-21swapper/200:11:272
1152299191,17cyclictest0-21swapper/222:10:002
1152299190,4cyclictest0-21swapper/222:44:012
1152299190,4cyclictest0-21swapper/222:38:072
1152299190,1cyclictest4314-21diskmemload21:58:122
11520991917,1cyclictest25-21ksoftirqd/100:20:091
1152099191,17cyclictest4314-21diskmemload23:43:531
1151599195,7cyclictest31046-21ssh23:59:590
11515991916,2cyclictest23065-21cut21:20:190
1151599190,15cyclictest0-21swapper/020:35:120
1152599180,1cyclictest0-21swapper/300:19:193
1152299182,2cyclictest4314-21diskmemload00:26:272
1152299182,2cyclictest17076-21ssh22:34:372
1152299181,2cyclictest32094-21grep21:25:252
1152299180,2cyclictest0-21swapper/200:35:392
1152299180,1cyclictest0-21swapper/221:12:512
1152299180,1cyclictest0-21swapper/200:24:032
11520991817,1cyclictest0-21swapper/122:02:011
1152099181,0cyclictest141rcu_preempt23:00:541
1151599186,8cyclictest2412-21/usr/sbin/munin22:45:140
130942170,0sleep00-21swapper/022:31:300
11525991717,0cyclictest0-21swapper/321:20:133
1152599170,16cyclictest0-21swapper/321:43:543
1152299178,4cyclictest20416-21ssh23:53:172
1152299177,8cyclictest2376-21sh23:23:222
1152299175,12cyclictest4314-21diskmemload21:16:022
1152299171,3cyclictest15262-21apt20:25:002
1152299170,3cyclictest0-21swapper/219:15:132
1152299170,16cyclictest15250-21grep23:30:262
1152099179,1cyclictest0-21swapper/122:50:291
11520991715,2cyclictest25-21ksoftirqd/121:52:181
1152599160,0cyclictest0-21swapper/321:58:543
1152299168,4cyclictest24410-21ssh23:16:512
1152299160,3cyclictest0-21swapper/221:51:042
1152099161,1cyclictest141rcu_preempt23:11:251
1152099160,15cyclictest0-21swapper/121:33:371
11515991615,1cyclictest13-21ksoftirqd/022:43:410
11515991613,2cyclictest8943-21cut21:50:230
1152599150,1cyclictest0-21swapper/323:26:303
1152599150,15cyclictest0-21swapper/300:32:453
1152299159,5cyclictest29698-21taskset21:44:082
1152299158,6cyclictest29846-21ntpq19:45:212
1152299152,1cyclictest0-21swapper/200:01:552
1152099150,15cyclictest0-21swapper/123:22:441
1152099150,15cyclictest0-21swapper/122:14:571
11515991510,3cyclictest31382-21wc20:55:230
1151599150,1cyclictest0-21swapper/023:04:050
1151599150,12cyclictest6804-21ls21:50:000
257422140,2sleep30-21swapper/319:40:003
11525991414,0cyclictest0-21swapper/300:38:013
11525991412,2cyclictest0-21swapper/323:11:263
11525991411,2cyclictest22460-21grep23:15:273
1152599140,1cyclictest0-21swapper/322:42:303
1152599140,14cyclictest0-21swapper/323:49:333
1152599140,13cyclictest721-21sensors00:00:223
1152299149,4cyclictest19593-21grep19:25:172
1152299148,5cyclictest26245-21cut20:45:212
1152299147,6cyclictest8654-21tr20:10:152
1152299147,6cyclictest29006-21cat20:50:252
1152299147,6cyclictest2314-21needreboot19:55:192
1152099149,3cyclictest843-21grep21:45:261
11520991412,1cyclictest0-21swapper/119:20:141
11520991410,2cyclictest11821-21tr19:10:131
1152099140,2cyclictest0-21swapper/121:36:591
1152099140,1cyclictest0-21swapper/122:45:041
1151599144,5cyclictest9239-21perf22:29:590
1151599143,6cyclictest24252-21awk00:15:010
1151599141,9cyclictest0-21swapper/020:05:120
11515991412,2cyclictest13-21ksoftirqd/022:37:280
11515991410,3cyclictest24681-21cat21:40:260
11515991410,3cyclictest21256-21awk23:35:000
1151599140,13cyclictest10064-21tr22:10:210
1152599138,3cyclictest6543-21df20:05:153
1152299138,5cyclictest12666-21irqstats19:10:172
1152299138,4cyclictest813-21sed21:00:182
1152299137,5cyclictest6131-21cut20:05:132
1152299137,5cyclictest3590-21python21:05:262
1152299137,5cyclictest32485-21cut19:50:212
1152299137,5cyclictest23478-21basename20:40:162
1152299137,5cyclictest18904-21cat20:30:172
1152299137,5cyclictest15919-21expr20:25:152
1152299136,6cyclictest11161-21expr20:15:172
1152299131,4cyclictest23921-21grep19:35:122
1152099139,3cyclictest29748-21wc21:25:011
1152099139,3cyclictest27136-21cron20:50:001
1152099139,3cyclictest25977-21tail00:15:211
1152099139,2cyclictest7614-21sed20:05:251
1152099138,5cyclictest25-21ksoftirqd/123:26:261
1152099136,4cyclictest537-21snmpd23:15:051
11520991311,2cyclictest241rcuc/122:07:521
11520991311,1cyclictest0-21swapper/122:26:391
11520991310,2cyclictest20164-21wc19:25:221
11520991310,2cyclictest14214-21cut23:30:161
1152099130,12cyclictest30080-21tr20:55:131
1152099130,11cyclictest22422-21sendmail-msp20:39:591
1151599139,3cyclictest5845-21cut21:10:160
1151599139,3cyclictest21310-21sendmail21:20:010
1151599139,3cyclictest1411-21wc22:05:180
1151599134,6cyclictest377-21awk00:00:190
1151599134,1cyclictest0-21swapper/000:27:440
1151599131,6cyclictest20015-21munin-run20:35:000
11515991310,2cyclictest5045-21head23:05:200
11515991310,2cyclictest29658-21awk23:40:000
11515991310,2cyclictest26746-21grep20:45:260
11515991310,2cyclictest14789-21grep20:20:260
11515991310,2cyclictest1260-21cut19:55:130
1151599130,12cyclictest23393-21grep20:40:150
1152599129,2cyclictest9659-21tail00:05:243
1152599129,2cyclictest29277-21tr23:20:143
1152599129,2cyclictest25815-21tail20:45:153
1152599129,2cyclictest2567-21tr22:45:143
1152599129,2cyclictest21801-21grep23:35:143
1152599129,2cyclictest16284-21cat21:35:243
1152599129,2cyclictest16232-21cron19:20:013
1152599129,2cyclictest12930-21basename21:15:013
1152599128,3cyclictest29017-21grep20:50:253
1152599128,3cyclictest14099-21uniq23:30:163
1152599128,1cyclictest0-21swapper/322:26:263
1152599123,5cyclictest547-21grep22:05:133
1152599122,1cyclictest0-21swapper/323:55:123
1152599121,8cyclictest17592-21dump-pmu-power22:35:003
11525991210,1cyclictest8146-21grep21:30:273
11525991210,1cyclictest4314-21diskmemload23:09:333
11525991210,1cyclictest0-21swapper/300:10:273
1152599120,10cyclictest18116-21grep20:30:133
1152299125,6cyclictest28129-21cat19:44:552
1152299125,6cyclictest17187-21grep19:20:142
1152099129,2cyclictest9974-21head00:25:191
1152099129,2cyclictest5167-21wc20:00:261
1152099129,2cyclictest26628-21grep22:20:201
1152099129,2cyclictest21897-21cut19:30:161
1152099129,2cyclictest14673-21grep21:15:201
1152099129,2cyclictest11654-21tail22:30:251
1152099122,2cyclictest225-21systemd-journal20:20:001
1152099121,3cyclictest225-21systemd-journal20:50:181
11520991211,1cyclictest0-21swapper/100:06:131
1152099120,2cyclictest0-21swapper/121:44:171
1151599129,2cyclictest7015-21sed21:30:170
1151599129,2cyclictest470-21tune2fs21:00:150
1151599129,2cyclictest27244-21ssh00:35:200
1151599129,2cyclictest26427-21ssh22:20:190
1151599129,2cyclictest24874-21df22:00:140
1151599129,2cyclictest22919-21tail19:30:250
1151599129,2cyclictest20889-21grep23:15:150
1151599129,2cyclictest20052-21head22:55:150
1151599129,2cyclictest15349-21sed19:15:210
1151599129,2cyclictest14042-21grep23:10:250
1151599120,11cyclictest28125-21head20:50:150
1152599119,1cyclictest30529-21cut21:25:143
1152599119,1cyclictest0-21swapper/321:50:083
1152599118,2cyclictest9622-21cat22:10:183
1152599118,2cyclictest8744-21expr20:10:163
1152599118,2cyclictest31495-21grep19:50:143
1152599118,2cyclictest30203-21expr20:55:123
1152599118,2cyclictest29577-21grep23:00:253
1152599118,2cyclictest25529-21uname22:00:203
1152599118,2cyclictest21950-21ssh22:36:533
1152599118,2cyclictest19766-21ssh23:52:473
1152599118,2cyclictest19552-21expr22:55:133
1152599118,2cyclictest18493-21chrt19:23:193
1152599118,2cyclictest17589-21ssh21:17:073
1152599118,2cyclictest17478-21ls22:15:153
1152599118,2cyclictest14800-21sed20:20:253
1152599118,2cyclictest11891-21ssh00:26:023
1152599117,4cyclictest29822-21apache221:47:433
1152599116,4cyclictest55550irq/126-enp2s0-00:22:043
1152099119,1cyclictest2082-21wc22:45:011
1152099118,2cyclictest9082-21tail20:10:171
1152099118,2cyclictest5677-21uniq21:10:141
1152099118,2cyclictest31991-21grep19:50:161
1152099118,2cyclictest30957-21wc19:49:591
1152099118,2cyclictest26675-21cut00:35:171
1152099118,2cyclictest24018-21which20:40:211
1152099118,2cyclictest21550-21ssh23:35:051
1152099118,2cyclictest21416-21ssh00:12:491
1152099118,2cyclictest19271-21ssh22:15:381
1152099118,2cyclictest18597-21ssh22:35:131
1152099118,2cyclictest16658-21tail23:50:261
1152099118,2cyclictest16243-21tr21:55:161
1152099117,2cyclictest18147-21tr20:30:131
1152099116,5cyclictest29819-21apache223:05:581
1152099116,1cyclictest16315-21sh20:25:171
1152099114,3cyclictest225-21systemd-journal23:00:011
1151599118,2cyclictest32061-21sed21:25:240
1151599118,2cyclictest30865-21ssh00:18:380
1151599118,2cyclictest27298-21wc19:40:200
1151599118,2cyclictest1917-21sed00:20:170
1151599118,2cyclictest15781-21cut21:35:190
1151599118,2cyclictest14509-21ssh23:50:010
1151599118,2cyclictest13366-21sed19:10:250
1151599118,2cyclictest11426-21cut20:15:190
1151599118,2cyclictest11026-21ssh23:47:300
1151599117,2cyclictest16553-21cat20:25:200
1151599116,4cyclictest17023-21apt-get22:15:130
1151599116,2cyclictest225-21systemd-journal19:40:000
1151599114,6cyclictest4301-21apt-get21:10:010
1151599113,5cyclictest10497-21idleruntime-cro20:15:000
1151599113,4cyclictest4314-21diskmemload23:27:460
1151599111,8cyclictest0-21swapper/000:32:360
1152599108,2cyclictest593-21sh21:00:163
1152599108,1cyclictest5353-21taskset20:00:273
1152599108,1cyclictest3365-21basename19:59:593
1152599108,1cyclictest30768-21taskset19:49:493
1152599108,1cyclictest26094-21grep19:40:123
1152599108,1cyclictest244222sleep320:40:373
1152599108,1cyclictest21573-21cat19:30:143
1152599108,1cyclictest19315-21expr19:25:153
1152599108,1cyclictest15839-21seq20:25:153
1152599108,1cyclictest126582sleep320:17:293
1152599108,1cyclictest12194-21expr19:10:143
1152599108,1cyclictest1142-21runrttasks20:39:243
1152599107,2cyclictest4120-21taskset21:07:503
1152099108,1cyclictest32713-21sh21:00:141
1152099108,1cyclictest17352-21sh00:30:041
1152099108,1cyclictest1313-21grep19:55:131
1152099108,1cyclictest131142sleep120:20:081
1152099107,2cyclictest28362-21readlink19:45:001
1152099107,2cyclictest14165-21cut19:15:141
1151599108,1cyclictest28990-21aten2_r7power_c23:20:130
1151599108,1cyclictest17299-21grep19:20:150
1151599108,1cyclictest10560-21ssh22:50:000
1151599107,2cyclictest30400-21head19:45:270
1151599107,2cyclictest19040-21tr19:25:150
115209997,1cyclictest4037-21taskset21:06:451
115209996,2cyclictest23784-21cut19:35:121
115159997,1cyclictest3698-21aten2_r7power_p20:00:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional