You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 05:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 08, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
240972239161,25sleep00-21swapper/019:05:410
242192238163,23sleep20-21swapper/219:07:162
242922235163,24sleep30-21swapper/319:08:153
243772211181,19sleep10-21swapper/119:09:201
3181221470,4sleep32453299cyclictest21:50:153
245269910198,2cyclictest0-21swapper/223:48:112
124782660,4sleep230-21rcuc/222:20:152
124782660,4sleep230-21rcuc/222:20:152
32762600,4sleep123-21ksoftirqd/122:00:111
136552550,4sleep113657-21proc_pri23:35:241
136552550,4sleep113657-21proc_pri23:35:241
256122540,5sleep22452699cyclictest00:05:022
75262510,1sleep00-21swapper/020:55:170
50472510,2sleep10-21swapper/123:15:251
24526995148,2cyclictest0-21swapper/200:12:452
245592480,1sleep00-21swapper/000:00:220
2451599386,15cyclictest3823-21munin-run19:35:000
57402330,4sleep081ktimersoftd/023:20:010
2452199296,13cyclictest29183-21diskstats21:45:131
2452199295,4cyclictest16549-21latency21:15:171
2452199292,12cyclictest13233-21perf21:10:011
2452199292,12cyclictest13233-21perf21:10:011
2452199288,13cyclictest5650-21fschecks_count22:05:141
2451599281,13cyclictest29240-21perf23:00:020
2451599278,16cyclictest26689-21munin-run19:15:020
2452199266,12cyclictest31948-21memory21:50:161
2452199265,9cyclictest26405-21crond21:40:011
2452199265,18cyclictest19257-21cat20:10:011
2451599261,12cyclictest3488-21basename23:15:010
2452199256,9cyclictest23818-21/usr/sbin/munin20:20:231
2452199255,12cyclictest28634-21wc21:45:011
2452199253,9cyclictest4420-21latency_hist20:50:011
2451599258,15cyclictest22640-21awk22:45:020
2451599251,13cyclictest29890-21perf00:15:010
2452199246,15cyclictest1606-21sh19:30:001
2452199245,10cyclictest18660-21expr22:35:121
2452199244,12cyclictest7482-21users23:20:261
2452199242,19cyclictest15631-21cat23:40:201
2452199241,17cyclictest0-21swapper/121:34:291
2451599242,11cyclictest24216-21munin-run21:35:010
2451599241,12cyclictest6305-21munin-run00:35:000
2451599241,12cyclictest25809-21idleruntime-cro20:25:020
2453299231,16cyclictest0-21swapper/322:52:193
2452199232,19cyclictest21-21rcuc/123:57:211
2452199231,10cyclictest21-21rcuc/123:50:121
2451599232,12cyclictest31777-21kworker/0:300:27:220
2451599232,11cyclictest4950-21kworker/0:122:04:060
2453299226,3cyclictest29594-21if_enp2s021:45:163
2453299221,5cyclictest0-21swapper/322:45:393
2453299221,5cyclictest0-21swapper/321:30:433
2453299221,11cyclictest25088-21smart_sda20:20:233
2453299220,5cyclictest18061-21kworker/u8:222:35:233
2452699228,5cyclictest0-21swapper/200:35:272
2452699227,5cyclictest28420-21proc_pri22:55:222
2452699225,3cyclictest12050irq/123-ahci[0021:15:022
2452699220,4cyclictest28968-21kworker/u8:223:01:392
2452199228,3cyclictest4331-21/usr/sbin/munin00:30:241
2452199226,2cyclictest25056-21/usr/sbin/munin22:50:251
2452199225,4cyclictest0-21swapper/122:55:221
2452199223,11cyclictest11750-21sed22:20:011
2452199221,9cyclictest28010-21latency_hist20:30:011
2452199221,9cyclictest0-21swapper/121:30:011
2452199221,16cyclictest0-21swapper/122:12:401
2452199221,11cyclictest0-21swapper/119:40:141
2451599224,15cyclictest12605-21munin-run19:55:000
2451599223,9cyclictest10199-21kworker/0:220:00:010
2451599221,11cyclictest8435-21/usr/sbin/munin19:45:110
2453299211,11cyclictest0-21swapper/323:22:143
2453299211,10cyclictest0-21swapper/319:37:183
2452699211,6cyclictest9006-21memory23:25:202
2452699210,8cyclictest0-21swapper/221:56:192
2452199217,6cyclictest15772-21sshd22:26:441
2452199215,13cyclictest2493-21grep23:10:201
2452199214,14cyclictest22725-21latency_hist22:45:011
2452199214,13cyclictest23477-21/usr/sbin/munin00:00:121
2452199213,5cyclictest26375-21taskset00:05:161
2452199213,15cyclictest8842-21grep21:00:011
2452199212,7cyclictest0-21swapper/120:05:011
2452199212,6cyclictest87650irq/129-enp1s0-23:45:071
24521992117,3cyclictest0-21swapper/119:35:221
24521992115,3cyclictest28285-21sh19:15:161
24521992114,4cyclictest2423-21grep21:55:231
24521992114,2cyclictest0-21swapper/121:23:411
2452199211,17cyclictest10383-21grep21:00:241
2452199211,11cyclictest14032-21perl21:10:161
2452199211,11cyclictest0-21swapper/122:22:471
2452199211,11cyclictest0-21swapper/122:22:471
2451599219,4cyclictest26088-21cpuspeed_turbos20:25:140
2451599218,4cyclictest15599-21date21:15:010
2451599213,16cyclictest8217-21munin-run19:45:000
2451599212,11cyclictest5537-21kworker/0:223:27:020
24515992115,3cyclictest827-21runrttasks00:15:110
2451599211,12cyclictest30413-21/usr/sbin/munin20:35:160
2451599210,10cyclictest26415-21kworker/0:319:17:010
109412210,4sleep22452699cyclictest23:30:152
2453299209,3cyclictest24437-21/usr/sbin/munin21:35:163
2453299207,2cyclictest26268-21grep19:10:253
2453299207,2cyclictest0-21swapper/320:03:553
2453299204,9cyclictest12932-21chrt21:06:243
2453299204,9cyclictest12932-21chrt21:06:243
2453299202,10cyclictest832-21/usr/sbin/munin21:55:123
2453299202,10cyclictest7476-21/usr/sbin/munin22:10:113
2453299202,10cyclictest12828-21/usr/sbin/munin19:55:183
2453299201,14cyclictest0-21swapper/323:27:183
2453299201,11cyclictest23470-21/usr/sbin/munin00:00:193
2453299201,10cyclictest0-21swapper/320:48:473
2453299201,10cyclictest0-21swapper/300:08:263
2453299200,5cyclictest0-21swapper/323:10:003
2452699208,3cyclictest0-21swapper/220:21:202
2452699206,4cyclictest9012-21latency_hist21:00:012
2452699201,3cyclictest0-21swapper/222:54:562
2452699201,16cyclictest17548-21cstates20:05:122
2452699200,6cyclictest0-21swapper/223:15:592
2452199209,3cyclictest23588-21latency_hist20:20:011
2452199208,4cyclictest29868-21latency_hist00:15:011
2452199205,9cyclictest29798-21diskstats23:00:151
2452199205,12cyclictest29892-21wc19:20:131
2452199204,5cyclictest28456-21expr20:30:121
2452199204,12cyclictest32197-21cat00:20:011
2452199203,15cyclictest1923-21awk00:25:001
2452199202,16cyclictest0-21swapper/122:31:031
24521992016,3cyclictest0-21swapper/119:33:311
2452199201,16cyclictest32240-21perl23:05:161
2452199201,11cyclictest8781-21diskstats19:45:131
2452199201,11cyclictest12612-21awk19:55:001
2452199200,6cyclictest808-21snmpd23:30:091
2452199200,18cyclictest0-21swapper/120:40:241
2452199200,18cyclictest0-21swapper/100:38:411
2451599207,7cyclictest87350irq/126-enp1s0-22:54:530
2451599207,4cyclictest14575-21kworker/0:020:17:050
2451599206,6cyclictest19162-21cpuspeed_turbos23:50:120
2451599206,6cyclictest0-21swapper/021:46:040
2451599206,5cyclictest0-21swapper/022:39:590
2451599205,6cyclictest0-21swapper/021:26:340
2451599205,3cyclictest87350irq/126-enp1s0-22:24:030
2451599205,3cyclictest87350irq/126-enp1s0-22:24:030
2451599205,3cyclictest87350irq/126-enp1s0-00:07:000
2451599205,2cyclictest87550irq/128-enp1s0-00:35:100
2451599204,2cyclictest87650irq/129-enp1s0-20:12:450
2451599203,8cyclictest87450irq/127-enp1s0-21:50:220
2451599201,11cyclictest17848-21/usr/sbin/munin21:20:170
53022191,2sleep30-21swapper/300:30:213
308272190,4sleep230-21rcuc/221:50:022
2453299192,10cyclictest0-21swapper/319:40:123
2453299191,9cyclictest0-21swapper/323:30:143
2453299191,9cyclictest0-21swapper/323:16:253
2453299191,7cyclictest2919-21sed23:10:253
2453299191,7cyclictest0-21swapper/321:41:163
24532991912,4cyclictest7256-21idleruntime-cro22:10:013
24532991912,4cyclictest6949-21tr20:55:133
24532991911,4cyclictest17897-21gltestperf20:05:163
2453299191,10cyclictest30260-21irqstats23:00:173
2452699191,6cyclictest0-21swapper/222:06:012
2452699191,5cyclictest0-21swapper/221:18:542
2452699191,4cyclictest0-21swapper/200:24:082
24526991914,3cyclictest548-21sh21:55:012
2452699191,3cyclictest0-21swapper/221:38:162
2452199195,2cyclictest808-21snmpd23:25:091
2452199193,5cyclictest0-21swapper/100:25:171
2452199191,3cyclictest0-21swapper/120:51:171
24521991913,3cyclictest23201-21diskstats22:45:131
2452199191,16cyclictest0-21swapper/120:35:041
2452199191,16cyclictest0-21swapper/120:12:221
2452199191,16cyclictest0-21swapper/119:13:131
2451599198,3cyclictest9044-21/usr/sbin/munin21:00:210
2451599196,5cyclictest0-21swapper/023:23:150
2451599195,4cyclictest0-21swapper/023:34:460
2451599194,2cyclictest87450irq/127-enp1s0-21:37:580
2451599193,4cyclictest14165-21grep22:25:130
2451599192,6cyclictest87650irq/129-enp1s0-22:46:590
2451599191,6cyclictest0-21swapper/022:30:180
2451599191,4cyclictest30152-21sshd20:34:440
2451599191,4cyclictest16985-21kworker/0:320:07:410
2451599191,3cyclictest7943-21fschecks_time22:10:130
2451599191,10cyclictest29802-21/usr/sbin/munin19:20:250
2451599191,10cyclictest0-21swapper/022:18:490
277952181,2sleep30-21swapper/320:27:403
2453299183,9cyclictest0-21swapper/323:53:453
2453299183,5cyclictest17401-1kworker/u9:222:30:203
2453299183,4cyclictest579-21systemd-journal00:15:023
2453299181,6cyclictest0-21swapper/300:16:443
24532991813,3cyclictest13223-21diskmemload21:17:263
24532991812,3cyclictest827-21runrttasks20:15:173
2453299181,11cyclictest0-21swapper/322:04:373
2453299181,10cyclictest6434-21sshd20:52:423
2452699185,4cyclictest32-21ksoftirqd/222:30:142
2452699185,4cyclictest2146-21acpi00:25:122
2452699184,3cyclictest87350irq/126-enp1s0-22:00:162
2452699182,4cyclictest22793-21ps20:15:222
2452699181,3cyclictest20380-21df21:25:142
24526991813,3cyclictest19545-21grep23:50:152
2452699181,15cyclictest0-21swapper/222:35:282
2452199189,6cyclictest14380-21unixbench_singl19:55:261
2451599188,3cyclictest32005-21/usr/sbin/munin19:25:170
2451599187,3cyclictest359-21ps23:05:240
2451599186,3cyclictest0-21swapper/021:18:560
2451599185,4cyclictest0-21swapper/023:01:440
2451599185,3cyclictest0-21swapper/023:41:070
2451599184,7cyclictest17277-21kworker/u8:221:41:450
2451599183,6cyclictest0-21swapper/023:48:530
2451599181,9cyclictest808-21snmpd19:39:260
2451599181,2cyclictest0-21swapper/000:23:090
2451599181,14cyclictest0-21swapper/023:55:120
2451599181,10cyclictest4647-21kworker/0:120:50:470
2451599181,10cyclictest10796-21kworker/0:221:05:300
2451599181,10cyclictest10796-21kworker/0:221:05:300
185102181,2sleep30-21swapper/323:45:493
137522180,2sleep30-21swapper/322:24:033
137522180,2sleep30-21swapper/322:24:033
2453299177,3cyclictest808-21snmpd23:57:433
2453299176,3cyclictest87650irq/129-enp1s0-23:35:453
2453299176,3cyclictest87650irq/129-enp1s0-23:35:453
2453299175,3cyclictest0-21swapper/319:50:353
2453299172,5cyclictest32391-21munin-run20:40:013
2453299171,9cyclictest0-21swapper/321:10:233
2453299171,7cyclictest0-21swapper/300:24:483
24532991714,2cyclictest21409-21awk20:15:023
2453299171,3cyclictest0-21swapper/323:42:183
2453299171,3cyclictest0-21swapper/300:35:313
2453299171,14cyclictest0-21swapper/321:25:523
2453299171,14cyclictest0-21swapper/319:48:083
2453299171,10cyclictest0-21swapper/322:29:093
24532991710,5cyclictest9672-21/usr/sbin/munin22:15:163
24532991710,4cyclictest32090-21cpuspeed_turbos19:25:143
2452699176,3cyclictest87650irq/129-enp1s0-22:11:422
2452699176,3cyclictest27214-21unixbench_multi00:05:272
2452699175,3cyclictest13532-21idleruntime19:55:172
2452699175,3cyclictest13188-21idleruntime23:35:182
2452699175,3cyclictest13188-21idleruntime23:35:182
2452699172,6cyclictest1482-21/usr/sbin/munin23:10:202
2452699172,5cyclictest4678-21diskstats00:30:132
2452699171,9cyclictest0-21swapper/221:44:102
2452699171,7cyclictest790-21in:imjournal22:42:452
2452699171,7cyclictest24931-21expr19:10:122
2452699171,6cyclictest18911-21ntp_states21:20:212
2452699171,6cyclictest0-21swapper/223:40:582
2452699171,5cyclictest0-21swapper/221:09:292
2452699171,5cyclictest0-21swapper/221:09:292
2452699171,3cyclictest0-21swapper/222:26:402
24526991712,3cyclictest32610-21/usr/sbin/munin20:40:132
24526991712,3cyclictest21273-21/usr/sbin/munin23:55:272
2452699171,14cyclictest0-21swapper/219:20:072
24526991710,4cyclictest23772-21sed21:30:262
2452699170,14cyclictest10276-21find21:00:222
2451599177,3cyclictest0-21swapper/022:07:570
2451599174,3cyclictest2148-21kworker/0:020:45:450
2451599171,14cyclictest13369-21ls23:35:180
2451599171,14cyclictest13369-21ls23:35:180
2451599171,10cyclictest0-21swapper/021:55:170
167272171,3sleep20-21swapper/220:01:352
2453299161,6cyclictest32610-21/usr/sbin/munin20:40:243
2453299161,6cyclictest0-21swapper/321:20:163
2453299161,5cyclictest0-21swapper/300:27:353
24532991611,3cyclictest9044-21/usr/sbin/munin21:00:223
2453299161,13cyclictest495-21jbd2/dm-0-819:15:073
24532991611,3cyclictest29384-21proc_pri20:30:223
24532991611,3cyclictest20655-21/usr/sbin/munin22:40:263
2453299160,9cyclictest0-21swapper/322:55:023
2452699169,4cyclictest32507-21/usr/sbin/munin23:05:182
2452699165,3cyclictest87650irq/129-enp1s0-19:45:182
2452699161,6cyclictest0-21swapper/220:51:112
2452699161,6cyclictest0-21swapper/220:10:022
2452699161,5cyclictest30097-21/usr/sbin/munin00:15:112
2452699161,2cyclictest0-21swapper/222:17:352
24526991612,3cyclictest0-21swapper/223:20:072
24526991612,3cyclictest0-21swapper/220:47:032
24526991612,3cyclictest0-21swapper/219:44:132
24526991612,3cyclictest0-21swapper/219:31:122
24526991612,3cyclictest0-21swapper/219:26:462
2452699161,14cyclictest0-21swapper/219:37:512
24526991611,3cyclictest29147-21cat20:30:192
24526991610,3cyclictest30413-21/usr/sbin/munin20:35:162
24526991610,3cyclictest27514-21munin-node19:15:042
24526991610,3cyclictest23797-21missed_timers22:45:182
2451599162,2cyclictest87650irq/129-enp1s0-20:00:150
24532991510,3cyclictest29802-21/usr/sbin/munin19:20:143
24532991510,2cyclictest827-21runrttasks19:33:513
2452699151,5cyclictest11477-21latency19:50:172
24526991512,2cyclictest0-21swapper/220:25:192
24515991510,3cyclictest394-21expr20:40:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional