You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-09 - 07:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Aug 09, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
152332238164,23sleep10-21swapper/119:05:471
155142236163,24sleep20-21swapper/219:09:272
155302203166,24sleep30-21swapper/319:09:403
133962202165,25sleep00-21swapper/019:05:020
1982721390,5sleep01564599cyclictest19:15:220
2513921330,4sleep01564599cyclictest21:55:180
15657998682,3cyclictest0-21swapper/223:21:512
15657998682,3cyclictest0-21swapper/223:21:512
15657997875,2cyclictest0-21swapper/222:48:562
15657997167,3cyclictest0-21swapper/200:03:302
99612570,4sleep081ktimersoftd/022:35:150
309092540,5sleep11565099cyclictest19:40:221
293702540,3sleep31566199cyclictest20:50:233
102142530,1sleep00-21swapper/020:08:390
258612520,4sleep339-21rcuc/300:25:143
258612520,4sleep339-21rcuc/300:25:143
97332490,3sleep30-21swapper/320:05:253
1565099462,32cyclictest3435-21munin-run23:35:011
1566199421,29cyclictest0-21swapper/320:57:053
1565099427,24cyclictest7836-21awk23:45:011
1565099421,30cyclictest19124-21latency_hist20:30:021
1565099411,23cyclictest32027-21turbostat.cron00:40:011
1566199401,27cyclictest0-21swapper/319:23:583
1566199400,28cyclictest0-21swapper/320:25:583
1565799400,29cyclictest0-21swapper/200:16:002
1565799391,27cyclictest0-21swapper/220:10:522
1565799390,28cyclictest0-21swapper/200:20:122
1565099391,29cyclictest600-21cat22:15:011
1566199380,31cyclictest0-21swapper/320:19:583
1566199380,29cyclictest0-21swapper/322:01:183
1565799381,25cyclictest0-21swapper/219:42:042
1565099388,20cyclictest0-21swapper/120:45:011
1565799370,29cyclictest0-21swapper/200:27:002
1565799370,29cyclictest0-21swapper/200:27:002
1565799370,27cyclictest0-21swapper/200:05:562
1565099377,17cyclictest10020-21latency_hist23:50:021
1565099371,3cyclictest26370-21idleruntime-cro22:00:001
1565799351,26cyclictest0-21swapper/220:53:322
1565799351,25cyclictest0-21swapper/219:30:042
1565799350,25cyclictest0-21swapper/223:09:362
1565099355,22cyclictest20706-21latency_hist19:20:021
1565099354,19cyclictest28168-21sed23:15:191
1565099351,5cyclictest19182-21if_enp2s022:55:171
1565099351,3cyclictest6643-21sed21:15:021
1565099351,3cyclictest24159-21cat21:55:001
1564599357,13cyclictest8750-21munin-run21:20:010
1566199341,19cyclictest0-21swapper/319:13:223
1565799340,26cyclictest0-21swapper/220:39:432
1565799340,25cyclictest0-21swapper/220:02:002
1565099343,28cyclictest0-21swapper/100:05:291
1565099342,27cyclictest4307-21smartctl22:20:251
1565099341,5cyclictest20968-21sshd00:14:101
1564599348,12cyclictest19755-21perf21:45:000
1566199331,26cyclictest0-21swapper/319:38:383
1566199330,26cyclictest0-21swapper/319:27:143
1565099336,14cyclictest0-21swapper/121:47:151
1565099331,8cyclictest14265-21cat22:45:131
1565099331,6cyclictest17767-21/usr/sbin/munin21:40:151
1565099331,3cyclictest5129-21wc19:55:221
1565099331,3cyclictest4355-21munin-run21:10:011
1565099331,3cyclictest29506-21basename19:40:001
1565099331,3cyclictest13932-21cat22:45:021
1564599335,13cyclictest1514-21munin-run19:50:010
15661993218,6cyclictest0-21swapper/319:41:383
1566199321,28cyclictest0-21swapper/321:46:443
1566199321,24cyclictest0-21swapper/319:16:573
1566199320,14cyclictest0-21swapper/322:07:503
1566199320,14cyclictest0-21swapper/322:07:503
1565799321,13cyclictest0-21swapper/222:24:352
1565799320,25cyclictest0-21swapper/200:13:562
1565099329,15cyclictest966-21timerwakeupswit19:45:251
1565099328,20cyclictest20796-21timerwakeupswit20:30:261
1565099321,6cyclictest1759-21/usr/sbin/munin19:50:171
1565099321,4cyclictest663-21grep21:00:161
1565099321,3cyclictest3709-21cpu23:35:131
1565099321,3cyclictest11606-21/usr/sbin/munin20:10:211
1565099321,18cyclictest7101-21missed_timers20:00:211
1564599323,13cyclictest13812-21idleruntime-cro22:45:020
1566199311,28cyclictest0-21swapper/322:52:463
1566199311,28cyclictest0-21swapper/321:02:133
1566199311,28cyclictest0-21swapper/320:01:023
1566199311,18cyclictest4349-21diskmemload00:24:023
1566199311,14cyclictest0-21swapper/321:40:023
1566199311,12cyclictest4349-21diskmemload21:14:013
1565799311,23cyclictest0-21swapper/221:26:262
1565799311,14cyclictest0-21swapper/219:23:262
1565099319,19cyclictest20829-21expr23:00:141
1565099319,18cyclictest2822-21sleep22:20:011
1565099312,25cyclictest26933-21smart_sda00:25:251
1565099312,25cyclictest26933-21smart_sda00:25:251
1565099312,25cyclictest22770-21grep00:15:261
1565099311,7cyclictest12045-21latency21:25:201
1565099311,3cyclictest31154-21cut20:55:181
1565099311,3cyclictest13925-21perl21:30:161
1565099311,3cyclictest112222sleep122:35:281
1565099311,2cyclictest0-21swapper/122:25:151
1565099311,18cyclictest31899-21expr23:25:121
15650993111,17cyclictest9633-21gltestperf21:20:161
1565099310,20cyclictest0-21swapper/122:00:431
1566199301,27cyclictest0-21swapper/322:28:423
1566199301,13cyclictest0-21swapper/322:56:293
1566199301,13cyclictest0-21swapper/319:54:103
1566199301,12cyclictest0-21swapper/322:31:503
1566199300,25cyclictest0-21swapper/319:59:463
1565799301,27cyclictest0-21swapper/221:37:442
1565799301,22cyclictest0-21swapper/223:16:242
1565799300,23cyclictest0-21swapper/223:42:502
1565099307,19cyclictest8542-21pluginstate22:30:221
1565099302,3cyclictest28810-21/usr/sbin/munin22:05:241
1565099302,3cyclictest28810-21/usr/sbin/munin22:05:231
1565099302,3cyclictest22835-21/usr/sbin/munin23:05:201
1565099301,5cyclictest13825-21ntp_states20:15:251
1565099301,3cyclictest29156-21netstat20:50:211
1565099301,3cyclictest24385-21sendmail_mailtr19:25:261
1565099301,3cyclictest0-21swapper/123:20:211
1565099301,3cyclictest0-21swapper/123:20:211
1565099301,2cyclictest15592-21cpu21:35:131
15650993011,16cyclictest22851-21sed20:35:241
1564599302,13cyclictest19144-21perf20:30:020
1566199291,15cyclictest0-21swapper/322:10:023
1566199291,14cyclictest0-21swapper/321:06:053
1566199291,13cyclictest0-21swapper/323:36:423
1566199291,13cyclictest0-21swapper/300:17:023
1566199291,12cyclictest0-21swapper/322:17:063
1566199291,12cyclictest0-21swapper/319:46:583
1566199291,12cyclictest0-21swapper/300:30:223
1566199291,12cyclictest0-21swapper/300:05:143
1566199290,17cyclictest0-21swapper/300:02:263
1566199290,13cyclictest0-21swapper/322:21:023
1565799291,14cyclictest0-21swapper/221:06:402
1565799291,14cyclictest0-21swapper/220:41:552
1565799291,12cyclictest0-21swapper/221:51:202
1565799291,12cyclictest0-21swapper/220:17:592
1565799291,12cyclictest0-21swapper/219:55:242
1565799291,11cyclictest0-21swapper/219:51:142
1565799291,11cyclictest0-21swapper/219:16:562
1565799291,10cyclictest0-21swapper/222:58:282
1565799290,22cyclictest0-21swapper/222:44:112
1565099298,18cyclictest16333-21grep22:50:121
1565099298,18cyclictest10416-21cstates23:50:121
1565099296,20cyclictest16551-21sshd00:04:011
1565099292,3cyclictest25132-40dnf00:21:111
1565099292,2cyclictest0-21swapper/100:30:111
1565099291,3cyclictest808-21snmpd19:14:471
1565099291,3cyclictest24460-21idleruntime20:40:171
1565099291,3cyclictest16943-21awk20:25:021
1565099290,7cyclictest0-21swapper/121:19:091
1566199281,19cyclictest0-21swapper/323:06:093
1566199281,12cyclictest0-21swapper/322:47:463
1566199281,12cyclictest0-21swapper/321:23:383
1566199281,11cyclictest0-21swapper/323:55:103
1566199281,11cyclictest0-21swapper/321:55:343
1566199281,11cyclictest0-21swapper/320:38:583
1566199281,10cyclictest0-21swapper/300:35:383
1566199280,13cyclictest0-21swapper/323:27:463
1566199280,13cyclictest0-21swapper/321:51:503
1566199280,13cyclictest0-21swapper/321:27:183
1565799281,20cyclictest0-21swapper/222:34:392
1565799281,11cyclictest0-21swapper/220:57:442
1565799280,12cyclictest0-21swapper/221:21:242
1565099288,17cyclictest9350-21needreboot20:05:201
1565099287,18cyclictest26998-21latency_hist23:15:011
1565099281,4cyclictest13066-21grep23:55:161
1565099281,3cyclictest23039-21cat19:25:011
1564599287,10cyclictest2167-21awk21:05:010
1564599281,13cyclictest27305-21munin-run19:35:010
1566199279,9cyclictest0-21swapper/320:32:543
1566199271,18cyclictest0-21swapper/322:35:023
1566199271,13cyclictest0-21swapper/323:13:093
1566199271,13cyclictest0-21swapper/321:39:303
1566199271,13cyclictest0-21swapper/320:14:223
1566199271,11cyclictest0-21swapper/322:44:023
1566199271,11cyclictest0-21swapper/321:19:383
1566199270,13cyclictest0-21swapper/321:33:143
1566199270,12cyclictest26996-21runrttasks19:31:163
1565799271,12cyclictest0-21swapper/222:53:012
1565799271,11cyclictest0-21swapper/220:31:192
1565799271,11cyclictest0-21swapper/219:28:062
1565799271,10cyclictest0-21swapper/221:30:322
1565799271,10cyclictest0-21swapper/219:12:392
1565799270,18cyclictest839-21lldpd23:52:262
1565799270,17cyclictest0-21swapper/221:59:242
1565799270,13cyclictest0-21swapper/222:28:172
1565099274,2cyclictest26001-21hddtemp_smartct19:30:151
1566199261,12cyclictest0-21swapper/323:23:143
1566199261,12cyclictest0-21swapper/323:23:133
1566199260,13cyclictest0-21swapper/323:18:213
1566199260,13cyclictest0-21swapper/320:24:303
1566199260,10cyclictest0-21swapper/323:46:383
1565799261,9cyclictest0-21swapper/223:45:362
1565799261,17cyclictest0-21swapper/200:30:302
1565799261,12cyclictest0-21swapper/222:03:162
1565799261,11cyclictest0-21swapper/223:32:442
1565799261,11cyclictest0-21swapper/222:38:292
1565799261,10cyclictest0-21swapper/222:14:252
1565799260,13cyclictest0-21swapper/222:15:112
1565799260,12cyclictest0-21swapper/221:19:282
1565799260,11cyclictest0-21swapper/223:58:162
1565799260,11cyclictest0-21swapper/221:46:322
1564599262,12cyclictest20402-21munin-run23:00:010
1566199257,3cyclictest0-21swapper/300:10:463
1566199251,14cyclictest0-21swapper/320:45:093
1566199251,12cyclictest0-21swapper/323:43:143
1566199250,12cyclictest0-21swapper/323:52:343
1565799251,9cyclictest18109-21df_inode21:40:142
1565799251,11cyclictest0-21swapper/220:07:272
1565799251,10cyclictest0-21swapper/223:02:442
1565799250,9cyclictest0-21swapper/220:23:292
1565799250,11cyclictest0-21swapper/221:12:442
1565799250,11cyclictest0-21swapper/219:45:282
1566199241,11cyclictest0-21swapper/323:02:293
1566199241,11cyclictest0-21swapper/320:41:423
1565799247,8cyclictest0-21swapper/223:10:562
1565799241,9cyclictest0-21swapper/200:35:232
1565799241,11cyclictest0-21swapper/221:04:202
1565799241,11cyclictest0-21swapper/220:25:072
1565799240,10cyclictest0-21swapper/223:35:082
1564599241,6cyclictest0-21swapper/021:26:260
1564599241,14cyclictest32299-21latency_hist21:00:020
1566199230,9cyclictest0-21swapper/323:32:183
1565799236,3cyclictest0-21swapper/220:45:082
1565799236,3cyclictest0-21swapper/219:39:362
1565799221,9cyclictest0-21swapper/223:28:522
1565799221,10cyclictest0-21swapper/222:06:562
1565799221,10cyclictest0-21swapper/222:06:552
1564599226,14cyclictest0-21swapper/021:10:010
1564599223,8cyclictest4112-40dnf22:20:250
1564599214,3cyclictest87550irq/128-enp1s0-22:35:020
15645992114,4cyclictest28587-21munin-run22:05:010
15645992111,8cyclictest12333-21sshd20:12:400
1564599211,11cyclictest23438-21expr19:25:140
1564599201,7cyclictest0-21swapper/000:26:190
1564599201,7cyclictest0-21swapper/000:26:190
1564599201,6cyclictest0-21swapper/000:09:200
1564599201,11cyclictest808-21snmpd21:20:350
1564599199,3cyclictest87550irq/128-enp1s0-00:30:220
1564599198,3cyclictest12758-21/usr/sbin/munin20:15:190
1564599197,5cyclictest5478-21kworker/0:123:42:450
1564599195,3cyclictest1021-21kworker/0:323:35:050
1564599193,7cyclictest31620-21/usr/sbin/munin23:25:170
1564599192,14cyclictest14752-21tr00:00:130
1564599191,6cyclictest0-21swapper/021:54:260
15645991912,4cyclictest22609-21awk23:05:010
1564599191,10cyclictest808-21snmpd00:16:120
1564599191,10cyclictest30063-21/usr/sbin/munin00:35:230
1564599191,10cyclictest20969-21sshd00:14:100
1564599191,10cyclictest12446-21/usr/sbin/munin23:55:140
1564599186,3cyclictest6158-21/usr/sbin/munin20:00:230
1564599182,13cyclictest16010-21tr19:10:140
1564599181,9cyclictest784-21/usr/sbin/munin22:15:240
1564599181,8cyclictest2615-21latency19:50:200
1564599181,7cyclictest0-21swapper/023:46:390
1564599181,5cyclictest0-21swapper/023:07:580
1564599181,5cyclictest0-21swapper/022:47:420
1564599181,4cyclictest0-21swapper/021:48:440
15645991813,3cyclictest29037-21tail19:35:270
1564599181,10cyclictest29758-21sshd20:50:390
1564599181,10cyclictest0-21swapper/019:55:120
1564599180,12cyclictest0-21swapper/023:10:180
1564599178,3cyclictest3425-21idleruntime-cro23:35:000
1564599171,9cyclictest808-21snmpd00:20:110
1564599171,9cyclictest10248-21/usr/sbin/munin23:50:190
1564599171,7cyclictest29422-21/usr/sbin/munin23:20:210
1564599171,7cyclictest29422-21/usr/sbin/munin23:20:210
1564599171,7cyclictest0-21swapper/021:37:180
1564599171,7cyclictest0-21swapper/020:45:270
15645991712,3cyclictest5905-21cut22:25:180
15645991712,3cyclictest29722-21/usr/sbin/munin19:40:130
15645991712,3cyclictest20042-21hwlatdetect20:30:170
15645991712,3cyclictest14954-21/usr/sbin/munin20:20:120
1564599171,13cyclictest790-21in:imjournal20:35:130
1564599171,13cyclictest29800-21grep22:05:190
1564599171,13cyclictest29800-21grep22:05:190
1564599166,3cyclictest87550irq/128-enp1s0-23:15:160
1564599161,6cyclictest0-21swapper/022:52:460
15645991612,3cyclictest0-21swapper/022:10:360
15645991611,3cyclictest5282-21basename21:10:170
15645991611,3cyclictest13370-21/usr/sbin/munin21:30:230
1564599161,12cyclictest23748-21/usr/sbin/munin20:40:160
1564599161,10cyclictest0-21swapper/019:20:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional