You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-09 - 02:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Jan 09, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14502243166,25sleep10-21swapper/119:09:501
12262240164,24sleep20-21swapper/219:06:572
12822239168,23sleep00-21swapper/019:07:400
11192202165,25sleep30-21swapper/319:05:343
957321690,5sleep2156099cyclictest00:20:112
776321220,4sleep0154799cyclictest20:35:120
134322610,5sleep1155599cyclictest19:35:011
170792560,3sleep121-21rcuc/122:10:111
28402540,3sleep09-21rcu_preempt19:10:200
315342520,4sleep0154799cyclictest21:30:020
83612510,1sleep19-21rcu_preempt21:50:131
103022500,4sleep121-21rcuc/119:25:221
1560994410,22cyclictest15097-21awk23:20:012
1547994011,13cyclictest13242-21idleruntime-cro19:35:010
1547994011,13cyclictest13242-21idleruntime-cro19:35:000
156099373,7cyclictest17897-40dnf20:55:552
154799378,13cyclictest553-21munin-run20:20:010
156599361,12cyclictest0-21swapper/323:43:393
156099354,5cyclictest32408-40dnf23:57:002
156099353,7cyclictest22643-21diskmemload22:45:202
156099351,4cyclictest21597-21chrt23:34:122
1560993512,20cyclictest24097-21/usr/sbin/munin23:40:202
156099344,4cyclictest16019-21sed22:05:242
156099341,7cyclictest5901-21/usr/sbin/munin21:45:172
1560993413,18cyclictest9785-21ntp_states23:05:232
156099333,5cyclictest9129-21munin-run00:20:012
1560993313,17cyclictest1953-21ntp_states20:20:202
1560993312,18cyclictest10293-21/usr/sbin/munin21:55:222
1560993311,19cyclictest3080-21/usr/sbin/munin20:25:162
1560993310,20cyclictest11251-21/usr/sbin/munin19:30:152
1560993310,20cyclictest11251-21/usr/sbin/munin19:30:152
154799335,13cyclictest23260-21munin-run22:25:010
150702330,4sleep121-21rcuc/122:05:141
156099325,3cyclictest17290-21awk23:25:012
156099325,3cyclictest12050irq/123-ahci[0022:55:012
156099321,7cyclictest27563-21runrttasks22:34:032
1560993211,18cyclictest830-21runrttasks00:32:082
1560993211,18cyclictest32374-21threads20:15:252
1560993211,18cyclictest21303-21sensors_temp19:50:242
1560993210,3cyclictest2756-21/usr/sbin/munin00:05:212
1560993210,3cyclictest11554-21/usr/sbin/munin00:25:152
1560993210,3cyclictest11554-21/usr/sbin/munin00:25:152
156099320,2cyclictest0-21swapper/219:55:232
154799329,11cyclictest17935-21perf00:40:010
156099319,19cyclictest830-21runrttasks23:10:112
156099319,18cyclictest5574-21load19:15:192
156099315,4cyclictest2633-21pmu-power21:35:232
156099313,4cyclictest8120-21wc21:50:132
156099313,3cyclictest87450irq/129-enp1s0-22:00:162
156099311,4cyclictest0-21swapper/220:45:252
156099311,4cyclictest0-21swapper/220:45:252
1560993111,17cyclictest30291-21expr22:40:122
1560993111,17cyclictest1773-21/usr/sbin/munin19:10:232
1560993110,3cyclictest4965-21/usr/sbin/munin00:10:272
1565993026,3cyclictest0-21swapper/322:30:293
156099304,4cyclictest0-21swapper/219:45:182
156099301,6cyclictest0-21swapper/222:15:152
1560993012,15cyclictest10586-21memory20:40:182
1560993010,17cyclictest28868-21cut23:50:142
156599299,9cyclictest0-21swapper/319:53:073
156599291,12cyclictest0-21swapper/322:50:033
156599291,12cyclictest0-21swapper/319:45:113
156099299,4cyclictest17719-21uname22:10:182
156099299,3cyclictest32568-21missed_timers21:30:192
156099298,4cyclictest13451-21/usr/sbin/munin19:35:212
156099298,3cyclictest29456-21/usr/sbin/munin21:25:222
156099292,3cyclictest87450irq/129-enp1s0-21:05:122
156099291,3cyclictest87150irq/126-enp1s0-20:10:162
1560992911,15cyclictest18544-21cat21:00:112
156599280,12cyclictest0-21swapper/322:05:393
156099289,3cyclictest22749-21unixbench-2d22:20:252
156099288,3cyclictest22551-21sed23:35:162
156099285,6cyclictest0-21swapper/223:00:222
156099284,5cyclictest0-21swapper/221:45:012
156099283,4cyclictest6999-21sshd20:31:582
156099282,4cyclictest23486-21tail21:10:142
156099281,9cyclictest0-21swapper/200:02:112
156099281,7cyclictest9605-21fschecks_time19:25:152
156099281,11cyclictest17115-21grep19:40:242
1560992810,3cyclictest830-21runrttasks20:10:002
156099280,2cyclictest0-21swapper/222:55:192
154799281,11cyclictest0-21swapper/023:11:450
156599279,5cyclictest0-21swapper/321:35:213
1565992710,3cyclictest0-21swapper/323:25:553
1565992710,3cyclictest0-21swapper/320:00:033
156099279,3cyclictest29244-21basename21:25:012
156099279,3cyclictest28080-21expr22:35:122
156099277,3cyclictest25078-21cpu21:15:112
156099277,3cyclictest14280-21expr20:50:122
156099276,4cyclictest8197-21if_enp1s020:35:172
156099276,3cyclictest830-21runrttasks22:27:372
156099274,5cyclictest0-21swapper/200:35:212
156099272,5cyclictest17508-21/usr/sbin/munin23:25:282
156099272,2cyclictest87150irq/126-enp1s0-20:00:092
156099270,2cyclictest0-21swapper/219:20:132
155599271,18cyclictest0-21swapper/100:01:581
154799273,12cyclictest13536-21awk00:30:010
154799273,12cyclictest13536-21awk00:30:010
154799270,10cyclictest0-21swapper/023:30:240
156599269,3cyclictest0-21swapper/321:07:153
156599261,13cyclictest0-21swapper/321:53:513
156599261,12cyclictest0-21swapper/322:27:513
156599261,12cyclictest0-21swapper/321:26:153
156599261,12cyclictest0-21swapper/319:37:513
1565992610,3cyclictest0-21swapper/319:21:473
1555992612,12cyclictest20510-21kworker/u8:223:33:321
155599261,19cyclictest0-21swapper/123:02:581
154799261,7cyclictest3106-21head22:50:190
156599259,8cyclictest0-21swapper/320:45:593
156599259,8cyclictest0-21swapper/320:45:593
156599259,3cyclictest0-21swapper/323:21:273
156599259,3cyclictest0-21swapper/321:15:153
156599259,3cyclictest0-21swapper/320:44:513
156599259,3cyclictest0-21swapper/320:20:313
156599259,3cyclictest0-21swapper/320:14:513
156599259,3cyclictest0-21swapper/319:56:353
156599259,3cyclictest0-21swapper/319:19:343
156599259,3cyclictest0-21swapper/300:27:593
156599259,3cyclictest0-21swapper/300:27:593
156599254,9cyclictest0-21swapper/320:07:513
156599251,15cyclictest7814-21sed00:15:173
156599251,12cyclictest0-21swapper/323:36:193
156599251,12cyclictest0-21swapper/323:30:313
156599251,12cyclictest0-21swapper/323:05:383
156599251,12cyclictest0-21swapper/319:10:353
156599251,12cyclictest0-21swapper/300:04:513
1565992510,3cyclictest0-21swapper/321:23:153
1565992510,3cyclictest0-21swapper/320:30:593
1565992510,3cyclictest0-21swapper/319:41:073
1565992510,3cyclictest0-21swapper/319:25:153
156099256,4cyclictest26946-21gltestperf23:45:152
155599257,2cyclictest898-21systemd-logind23:50:231
155599251,17cyclictest0-21swapper/122:23:091
155599250,18cyclictest0-21swapper/120:45:581
155599250,18cyclictest0-21swapper/120:45:581
154799256,16cyclictest22008-21idleruntime-cro19:55:010
154799251,8cyclictest8487-21latency_hist23:05:010
154799251,12cyclictest6296-21perf23:00:010
156599249,4cyclictest0-21swapper/321:48:433
156599249,3cyclictest0-21swapper/323:59:593
156599249,3cyclictest0-21swapper/323:49:073
156599249,3cyclictest0-21swapper/323:12:463
156599249,3cyclictest0-21swapper/323:00:073
156599249,3cyclictest0-21swapper/322:16:233
156599249,3cyclictest0-21swapper/321:56:433
156599249,3cyclictest0-21swapper/321:40:473
156599249,3cyclictest0-21swapper/321:34:273
156599249,3cyclictest0-21swapper/320:25:433
156599249,2cyclictest19451-21strings21:00:193
156599248,4cyclictest0-21swapper/322:49:233
156599248,3cyclictest0-21swapper/320:38:273
156599241,12cyclictest0-21swapper/300:35:353
1565992410,3cyclictest0-21swapper/322:57:593
1565992410,3cyclictest0-21swapper/322:11:433
1565992410,3cyclictest0-21swapper/320:51:223
155599246,9cyclictest0-21swapper/121:56:021
155599242,4cyclictest25358-21sshd22:28:581
155599242,11cyclictest6639-21munin-run19:20:011
155599242,11cyclictest5051-21munin-run20:30:011
156599239,3cyclictest0-21swapper/320:58:423
156599231,11cyclictest0-21swapper/300:24:233
156599231,10cyclictest0-21swapper/320:15:273
1565992310,3cyclictest0-21swapper/322:36:033
1565992310,3cyclictest0-21swapper/322:20:593
155599232,18cyclictest27034-21munin-run21:20:011
155599231,3cyclictest8840-21perf19:25:011
155599231,20cyclictest0-21swapper/119:38:481
155599231,11cyclictest13524-21munin-run00:30:011
155599231,11cyclictest13524-21munin-run00:30:011
156599229,3cyclictest0-21swapper/323:54:153
156599229,3cyclictest0-21swapper/322:44:273
156599229,3cyclictest0-21swapper/300:05:473
156599228,3cyclictest0-21swapper/321:11:593
156599228,3cyclictest0-21swapper/300:13:193
156599227,3cyclictest0-21swapper/319:30:473
156599227,3cyclictest0-21swapper/319:30:473
1565992210,3cyclictest0-21swapper/323:17:433
155599223,10cyclictest29021-21awk22:35:211
1555992218,3cyclictest0-21swapper/120:09:231
155599221,3cyclictest30689-21acpi23:55:111
155599221,19cyclictest0-21swapper/123:15:221
155599221,19cyclictest0-21swapper/120:17:231
155599221,12cyclictest10193-21tune2fs20:40:151
155599220,17cyclictest87450irq/129-enp1s0-23:29:261
154799227,3cyclictest28241-21fschecks_count22:35:140
154799221,5cyclictest11102-21irqstats21:55:170
156599217,3cyclictest22643-21diskmemload22:03:353
155599212,17cyclictest0-21swapper/123:05:551
1555992117,3cyclictest0-21swapper/121:07:121
1555992117,3cyclictest0-21swapper/121:04:201
1555992117,3cyclictest0-21swapper/120:13:151
1555992117,3cyclictest0-21swapper/119:12:571
1555992117,3cyclictest0-21swapper/100:36:381
155599211,6cyclictest0-21swapper/122:58:441
155599211,2cyclictest21104-21ntp_states19:50:211
155599211,18cyclictest0-21swapper/121:31:581
155599211,17cyclictest22643-21diskmemload00:17:561
155599211,14cyclictest0-21swapper/122:51:251
155599211,10cyclictest0-21swapper/122:45:021
154799211,10cyclictest0-21swapper/023:56:590
154799210,8cyclictest0-21swapper/022:26:580
156599206,3cyclictest0-21swapper/300:31:513
155599206,6cyclictest0-21swapper/122:32:151
155599202,17cyclictest0-21swapper/123:12:171
155599202,17cyclictest0-21swapper/119:57:551
1555992016,3cyclictest0-21swapper/100:24:071
155599201,5cyclictest1585-21cpuspeed_turbos21:35:131
155599201,18cyclictest0-21swapper/123:38:441
155599201,17cyclictest0-21swapper/121:26:121
155599201,17cyclictest0-21swapper/121:21:311
155599201,12cyclictest0-21swapper/120:24:511
155599201,12cyclictest0-21swapper/119:40:371
155599201,11cyclictest7823-21diskstats20:35:131
155599201,11cyclictest6479-21perl21:45:151
155599201,11cyclictest6151-21latency20:30:191
155599201,10cyclictest7098-21cat00:15:011
155599200,15cyclictest0-21swapper/123:46:011
154799209,2cyclictest22145-21sed21:05:250
154799208,4cyclictest546-21dump-pmu-power20:24:560
154799206,5cyclictest831-21missed_timers22:45:200
154799206,5cyclictest0-21swapper/021:39:270
154799202,10cyclictest8096-21/usr/sbin/munin21:50:160
1547992014,3cyclictest15806-21tr19:40:110
154799201,3cyclictest16040-21munin-run20:55:010
154799201,11cyclictest18479-21users22:10:260
154799201,10cyclictest0-21swapper/021:10:080
155599198,3cyclictest13363-21latency22:00:181
155599198,3cyclictest0-21swapper/123:21:081
1555991916,2cyclictest0-21swapper/122:44:051
1555991915,3cyclictest0-21swapper/123:42:481
1555991915,3cyclictest0-21swapper/121:43:251
1555991915,3cyclictest0-21swapper/120:00:171
155599191,3cyclictest0-21swapper/122:18:371
155599191,3cyclictest0-21swapper/100:05:511
155599191,2cyclictest0-21swapper/121:10:081
155599191,17cyclictest0-21swapper/120:55:091
155599191,10cyclictest18136-21cstates19:45:131
155599190,17cyclictest0-21swapper/120:50:161
154799199,3cyclictest3080-21/usr/sbin/munin20:25:190
154799199,3cyclictest28840-21/usr/sbin/munin20:10:210
154799197,3cyclictest0-21swapper/023:23:210
154799197,3cyclictest0-21swapper/000:03:060
154799193,3cyclictest87150irq/126-enp1s0-23:26:390
154799193,3cyclictest0-21swapper/023:50:470
154799191,5cyclictest0-21swapper/023:19:580
154799191,12cyclictest0-21swapper/021:43:430
154799191,10cyclictest4668-21/usr/sbin/munin19:15:140
154799191,10cyclictest11873-21/usr/sbin/munin20:45:230
154799191,10cyclictest11873-21/usr/sbin/munin20:45:230
154799191,10cyclictest0-21swapper/021:49:040
154799191,10cyclictest0-21swapper/000:13:480
155599181,16cyclictest0-21swapper/119:30:031
155599180,16cyclictest0-21swapper/100:30:111
154799187,7cyclictest87150irq/126-enp1s0-21:16:280
154799186,7cyclictest87150irq/126-enp1s0-00:18:010
154799186,3cyclictest0-21swapper/023:41:190
154799186,10cyclictest87150irq/126-enp1s0-00:24:240
154799185,2cyclictest814-21snmpd23:35:450
154799182,3cyclictest0-21swapper/022:34:270
154799181,11cyclictest0-21swapper/022:42:070
154799181,11cyclictest0-21swapper/000:33:140
154799180,9cyclictest13803-21ls22:00:240
109862181,2sleep00-21swapper/019:29:390
154799177,3cyclictest814-21snmpd22:16:470
154799173,12cyclictest3359-21tail00:05:150
154799172,12cyclictest18728-21wc19:45:180
154799171,4cyclictest26641-21diskstats23:45:130
154799171,15cyclictest0-21swapper/020:59:560
154799171,14cyclictest0-21swapper/020:33:100
1547991710,4cyclictest23859-40dnf19:55:420
154799170,9cyclictest0-21swapper/022:05:290
154799170,2cyclictest87150irq/126-enp1s0-21:33:470
154799161,4cyclictest28073-21irqstats21:20:180
154799161,2cyclictest0-21swapper/023:06:030
154799161,14cyclictest0-21swapper/019:23:570
1547991611,3cyclictest24743-21expr20:00:100
1547991611,3cyclictest19984-21grep21:00:250
154799161,12cyclictest9907-21cat20:40:120
154799161,12cyclictest26647-21/usr/sbin/munin20:05:170
1547991610,3cyclictest13607-21grep19:35:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional