You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-17 - 08:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun May 17, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
205232229157,21sleep20-21swapper/219:05:362
189012219144,23sleep30-21swapper/319:05:033
206132201164,25sleep00-21swapper/019:06:460
205992201163,25sleep10-21swapper/119:06:351
2667421740,4sleep32096699cyclictest23:00:183
3122821460,4sleep12095799cyclictest21:55:271
20962999188,2cyclictest0-21swapper/223:24:162
20962995956,2cyclictest0-21swapper/221:35:362
20962995754,2cyclictest0-21swapper/220:55:532
140982530,4sleep230-21rcuc/220:05:162
307442510,1sleep30-21swapper/300:25:023
107992500,3sleep00-21swapper/022:25:150
175462460,3sleep339-21rcuc/321:25:233
2095799351,10cyclictest0-21swapper/120:27:011
20950993510,12cyclictest23475-21munin-run22:55:010
267662340,2sleep00-21swapper/023:00:190
2096299321,15cyclictest2309-21latency_hist23:20:012
2095799313,21cyclictest23-21ksoftirqd/122:01:131
2095099318,11cyclictest1484-21awk22:05:010
2095099311,15cyclictest29458-21awk21:55:020
2095799301,12cyclictest8877-21cat23:35:011
2095799301,12cyclictest0-21swapper/122:25:011
2095799300,15cyclictest0-21swapper/123:16:371
2095799294,13cyclictest755-21rngd19:15:551
2095799291,16cyclictest12534-21ls23:40:251
2095099291,20cyclictest0-21swapper/023:41:220
2095099291,13cyclictest32612-21munin-run19:35:020
2095099291,13cyclictest30412-21munin-run19:30:010
213692280,11sleep011-21rcuc/020:20:210
2096299283,5cyclictest87650irq/129-enp1s0-21:12:252
2095799287,3cyclictest0-21swapper/121:47:011
2095799280,10cyclictest0-21swapper/123:30:011
2095099281,19cyclictest0-21swapper/021:21:350
2095099281,19cyclictest0-21swapper/021:21:350
2096699273,21cyclictest39-21rcuc/321:40:193
2095799271,10cyclictest0-21swapper/121:21:451
2095799271,10cyclictest0-21swapper/121:21:451
2095099272,19cyclictest17916-21cat21:25:270
2095099271,18cyclictest0-21swapper/000:20:070
2095099271,14cyclictest11238-21munin-run20:00:010
2096699262,21cyclictest41-21ksoftirqd/322:45:183
2096699261,22cyclictest30643-21/usr/sbin/munin19:30:293
2096699261,22cyclictest22142-21grep19:10:203
2096699261,21cyclictest13919-21gltestperf23:45:153
2096699261,21cyclictest13919-21gltestperf23:45:153
2096299263,7cyclictest23202-21ntp_states00:05:212
2096299261,3cyclictest14670-21latency_hist22:35:012
2096299261,16cyclictest12668-21cut21:15:162
2096299261,12cyclictest26390-21cut19:20:122
2096299261,10cyclictest0-21swapper/222:25:182
2095799261,6cyclictest9661-21diskmemload22:50:091
2095799261,23cyclictest0-21swapper/120:38:011
2095799261,10cyclictest0-21swapper/121:05:561
2095099263,11cyclictest13455-21awk20:05:010
2095099261,19cyclictest87550irq/128-enp1s0-21:40:130
2095099261,19cyclictest0-21swapper/023:15:240
2095099261,18cyclictest0-21swapper/023:55:080
2095099261,18cyclictest0-21swapper/021:38:460
2095099261,16cyclictest87350irq/126-enp1s0-21:31:500
2096699258,3cyclictest29354-21memory19:25:203
2096699252,21cyclictest39-21rcuc/320:50:113
2096699252,20cyclictest8682-21needreboot21:05:213
2096699251,21cyclictest401ktimersoftd/323:20:193
2096699251,21cyclictest13669-21/usr/sbin/munin20:05:183
2096299252,11cyclictest17444-21diskstats22:40:152
2096299251,9cyclictest32510-21grep22:00:162
2096299251,14cyclictest24915-21grep00:10:152
2096299251,10cyclictest12050irq/123-ahci[0000:25:022
2095799258,4cyclictest4097-21kworker/u8:120:55:481
2095799251,9cyclictest0-21swapper/121:17:091
2095799251,16cyclictest8133-21ntpq19:50:211
2095799251,11cyclictest14261-21grep22:30:241
2095799251,10cyclictest0-21swapper/120:03:391
2095799251,10cyclictest0-21swapper/100:07:051
2095099252,16cyclictest0-21swapper/023:10:080
20950992513,7cyclictest0-21swapper/022:47:100
2095099251,18cyclictest0-21swapper/022:09:400
2095099251,17cyclictest0-21swapper/022:10:080
2095099251,17cyclictest0-21swapper/021:15:380
2095099251,17cyclictest0-21swapper/000:10:420
246352240,8sleep324636-21gltestperf19:15:163
2096699249,3cyclictest5425-21sed23:25:183
2096699241,20cyclictest41-21ksoftirqd/322:55:253
2096699241,20cyclictest3908-21/usr/sbin/munin22:10:223
2096699241,20cyclictest16687-21ps23:50:223
2096699241,20cyclictest11185-21cat21:10:253
2096699241,20cyclictest0-21swapper/320:45:253
2096699241,12cyclictest0-21swapper/320:30:163
2096299241,9cyclictest12194-21sh23:40:222
2096299241,13cyclictest7602-21latency23:30:192
2096299241,11cyclictest87650irq/129-enp1s0-23:10:122
2096299240,9cyclictest0-21swapper/221:50:022
2096299240,10cyclictest28322-21runrttasks00:17:472
2095799249,3cyclictest20672-21latency_hist21:35:011
2095799247,9cyclictest0-21swapper/121:53:451
2095799241,3cyclictest28218-21munin-run19:25:001
2095799241,12cyclictest0-21swapper/119:38:211
2095799241,10cyclictest21063-21cstates21:35:131
2095799241,10cyclictest0-21swapper/120:10:131
2095799241,10cyclictest0-21swapper/100:15:171
2095099246,15cyclictest3063-21munin-run20:55:020
2095099242,16cyclictest87350irq/126-enp1s0-23:24:570
20950992413,6cyclictest0-21swapper/023:53:070
20950992413,3cyclictest0-21swapper/000:05:090
2095099241,17cyclictest0-21swapper/000:27:470
2095099241,15cyclictest0-21swapper/000:01:120
2096699231,6cyclictest5860-21needreboot19:45:183
2096699231,19cyclictest17793-21cpuspeed_turbos23:55:123
2096699231,11cyclictest7293-21sshd21:02:563
20966992310,2cyclictest17095-21/usr/sbin/munin22:40:273
2096299231,9cyclictest5342-21irqstats23:25:172
2096299231,19cyclictest24303-21gunzip21:40:242
2096299231,15cyclictest0-21swapper/222:16:382
2096299231,13cyclictest20751-21latency00:00:172
2096299231,12cyclictest4173-1kworker/u9:122:12:192
2096299231,11cyclictest25346-21chrt22:56:032
2096299231,10cyclictest30250-21perl21:55:162
2096299231,10cyclictest15861-21/usr/sbin/munin20:10:142
2096299230,15cyclictest14456-21ls21:20:112
2096299230,15cyclictest14456-21ls21:20:112
2095799232,18cyclictest20983-21ntp_states00:00:201
2095799231,14cyclictest0-21swapper/123:22:491
2095799231,10cyclictest18906-21latency20:15:171
2095799231,10cyclictest15359-21latency_hist23:50:011
2095799231,10cyclictest15359-21latency_hist23:50:011
2095799231,10cyclictest0-21swapper/122:09:051
2095099237,6cyclictest0-21swapper/021:45:080
2095099236,15cyclictest10295-21awk22:25:010
2095099235,10cyclictest0-21swapper/021:55:210
2095099233,13cyclictest32471-21kworker/u8:021:11:260
20950992313,5cyclictest0-21swapper/023:30:240
2095099231,16cyclictest0-21swapper/000:39:540
2095099231,15cyclictest0-21swapper/022:44:110
2095099231,10cyclictest28518-21perf00:20:010
93422220,7sleep230-21rcuc/219:55:122
278502221,17sleep3909-21sssd_nss20:35:193
2096699228,3cyclictest39-21rcuc/300:30:243
2096699221,10cyclictest808-21snmpd23:30:443
2096699221,10cyclictest7067-21/usr/sbin/munin19:50:223
2096699221,10cyclictest0-21swapper/322:00:183
2096299223,7cyclictest0-21swapper/223:50:482
2096299222,7cyclictest0-21swapper/200:30:222
2096299222,17cyclictest31243-21tail19:30:222
2096299221,8cyclictest30067-21latency_hist23:10:012
2096299221,7cyclictest24659-21/usr/sbin/munin20:30:192
2096299221,18cyclictest17700-21ps21:25:242
2096299221,12cyclictest87350irq/126-enp1s0-23:36:332
2096299221,12cyclictest11461-21/usr/sbin/munin20:00:132
2096299221,12cyclictest0-21swapper/221:50:142
2096299221,10cyclictest12050irq/123-ahci[0000:30:022
2096299221,10cyclictest0-21swapper/222:08:202
2096299221,10cyclictest0-21swapper/200:35:182
2096299220,15cyclictest0-21swapper/220:15:492
2095799228,3cyclictest0-21swapper/119:57:431
2095799228,3cyclictest0-21swapper/119:27:371
2095799227,3cyclictest0-21swapper/120:53:081
2095799226,8cyclictest0-21swapper/123:11:211
2095799223,3cyclictest0-21swapper/120:22:371
2095799223,17cyclictest30663-21grep19:30:191
2095799223,11cyclictest909-21sssd_nss19:45:231
2095799221,9cyclictest0-21swapper/120:07:251
20957992216,3cyclictest24050-21sed21:40:201
2095799221,3cyclictest17552-21ntp_states21:25:231
2095799221,16cyclictest0-21swapper/120:44:011
2095799221,13cyclictest0-21swapper/121:00:161
2095799221,10cyclictest0-21swapper/119:11:371
2095099226,8cyclictest87550irq/128-enp1s0-19:11:560
2095099221,13cyclictest6294-21cstates22:15:120
2095099220,15cyclictest0-21swapper/023:47:080
2095099220,15cyclictest0-21swapper/023:47:080
2096699217,3cyclictest11461-21/usr/sbin/munin20:00:163
2096699212,16cyclictest39-21rcuc/320:10:203
2096699212,16cyclictest24341-21/usr/sbin/munin00:10:123
2096699211,9cyclictest1716-21/usr/sbin/munin22:05:243
2096699211,9cyclictest0-21swapper/300:25:173
2096699211,18cyclictest0-21swapper/321:33:553
20966992110,1cyclictest0-21swapper/321:00:013
2096299212,8cyclictest0-21swapper/223:00:242
2096299211,6cyclictest8670-21/usr/sbin/munin21:05:202
2096299211,17cyclictest5909-21grep19:45:202
2096299211,17cyclictest30576-21grep20:40:262
2096299211,12cyclictest87350irq/126-enp1s0-23:55:142
2096299210,14cyclictest0-21swapper/223:45:122
2096299210,14cyclictest0-21swapper/223:45:122
2095799217,3cyclictest10500-21taskset22:25:041
2095799216,8cyclictest0-21swapper/121:12:571
2095799216,3cyclictest0-21swapper/120:32:371
2095799211,3cyclictest25978-21cpuspeed_turbos23:00:131
20957992113,3cyclictest0-21swapper/122:38:211
2095799211,11cyclictest0-21swapper/100:35:561
2095799210,15cyclictest0-21swapper/122:10:131
2095799210,12cyclictest0-21swapper/123:09:371
2095799210,12cyclictest0-21swapper/122:15:211
20950992117,2cyclictest11-21rcuc/020:55:210
2095099211,6cyclictest14901-21/usr/sbin/munin22:35:160
2095099211,15cyclictest87350irq/126-enp1s0-23:06:300
2095099211,15cyclictest0-21swapper/000:31:470
2095099211,14cyclictest0-21swapper/022:58:070
3702201,3sleep0656-21expr19:35:130
2096699206,3cyclictest87350irq/126-enp1s0-21:50:113
2096699205,3cyclictest4173-1kworker/u9:123:13:293
2096699205,3cyclictest0-21swapper/321:15:183
2096699203,7cyclictest0-21swapper/323:40:263
2096699203,3cyclictest87650irq/129-enp1s0-21:45:163
2096699201,8cyclictest808-21snmpd23:35:453
2096699201,7cyclictest21110-21expr21:35:123
2096699201,16cyclictest32497-21/usr/sbin/munin23:15:143
2096699200,11cyclictest0-21swapper/319:38:563
2096299205,2cyclictest0-21swapper/222:39:392
2096299202,6cyclictest87650irq/129-enp1s0-22:22:182
2096299202,15cyclictest24207-21grep19:15:132
2096299201,7cyclictest22461-21/usr/sbin/munin20:25:202
2096299201,5cyclictest32472-21gunzip20:45:232
20962992011,6cyclictest20097-21cat21:30:252
20962992011,6cyclictest0-21swapper/219:25:122
2096299201,12cyclictest0-21swapper/222:52:122
2096299201,11cyclictest21176-21sshd22:48:562
2095799208,2cyclictest0-21swapper/100:12:571
2095799207,3cyclictest0-21swapper/119:43:531
2095799205,3cyclictest24952-21sensors_temp22:55:241
2095799203,3cyclictest0-21swapper/120:48:011
2095799201,13cyclictest0-21swapper/100:22:021
2095799200,4cyclictest0-21swapper/123:50:411
2095099206,3cyclictest30710-21kworker/0:320:44:280
2095099203,14cyclictest6378-21tail19:45:250
2095099202,10cyclictest9036-21latency_hist19:55:010
20950992013,2cyclictest87350irq/126-enp1s0-22:31:230
2095099201,16cyclictest24380-21df19:15:130
2095099201,14cyclictest0-21swapper/023:35:510
2096699197,3cyclictest12050irq/123-ahci[0019:42:193
2096699195,3cyclictest87350irq/126-enp1s0-21:20:133
2096699195,3cyclictest87350irq/126-enp1s0-21:20:133
2096699195,3cyclictest8310-21/usr/sbin/munin22:20:123
2096699194,4cyclictest9661-21diskmemload00:36:093
2096699191,5cyclictest22980-21sed22:50:263
20962991914,3cyclictest23134-21awk19:15:012
20962991912,5cyclictest7615-21sed21:05:012
2096299190,7cyclictest0-21swapper/219:45:022
2095799193,8cyclictest0-21swapper/100:28:521
2095799192,3cyclictest0-21swapper/123:39:331
2095799192,14cyclictest18382-21cut23:55:171
20957991912,3cyclictest0-21swapper/122:45:091
20957991912,2cyclictest0-21swapper/122:40:291
2095099195,7cyclictest808-21snmpd19:42:340
2095099194,7cyclictest87350irq/126-enp1s0-23:25:210
2095099191,8cyclictest19568-21tail20:15:240
2095099191,6cyclictest15024-21timerandwakeup20:05:250
20950991914,3cyclictest22461-21/usr/sbin/munin20:25:220
2095099191,10cyclictest6479-21cut21:00:200
2095099191,10cyclictest24659-21/usr/sbin/munin20:30:180
2095099191,10cyclictest0-21swapper/019:20:220
2096699189,2cyclictest87650irq/129-enp1s0-00:18:203
2096699184,3cyclictest9661-21diskmemload22:28:313
2096699182,4cyclictest20913-21kworker/u8:000:05:163
2096699182,13cyclictest18060-21/usr/sbin/munin20:15:183
2096699181,7cyclictest10825-21kworker/3:120:25:153
2096699181,4cyclictest6460-21diskstats22:15:143
2096699181,15cyclictest9661-21diskmemload21:56:273
2096699181,15cyclictest29357-21sensors_temp23:05:223
2096699180,15cyclictest15783-21ls22:35:183
2096299181,8cyclictest362-21/usr/sbin/munin19:35:122
20962991812,3cyclictest27665-21interrupts20:35:162
2095799189,3cyclictest0-21swapper/100:30:151
2095099185,3cyclictest87550irq/128-enp1s0-20:35:150
2095099184,7cyclictest87650irq/129-enp1s0-20:45:110
2095099184,5cyclictest827-21runrttasks21:05:130
219372171,3sleep30-21swapper/320:21:233
2096699178,2cyclictest0-21swapper/320:40:153
2096699175,5cyclictest20291-21fschecks_count00:00:133
2096699171,5cyclictest27100-21latency19:20:183
20966991712,3cyclictest13037-21df22:30:133
2096299171,6cyclictest21075-21irqstats20:20:182
2096299171,14cyclictest0-21swapper/219:52:442
2096299171,13cyclictest2665-21grep20:50:252
2095099171,4cyclictest15926-21cpuspeed_turbos20:10:100
2096699153,3cyclictest87650irq/129-enp1s0-19:57:433
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional