You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 23:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 31, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
67812241166,23sleep20-21swapper/207:08:412
67312234162,60sleep00-21swapper/007:08:020
67522233166,19sleep10-21swapper/107:08:191
66322203166,25sleep30-21swapper/307:06:453
2326021340,5sleep2698199cyclictest11:25:152
2044221330,5sleep1697599cyclictest10:05:121
2044221330,5sleep1697599cyclictest10:05:121
6981991111,107cyclictest124222sleep208:32:152
698199108104,2cyclictest28042-21diskmemload12:10:522
6981997471,2cyclictest0-21swapper/212:30:502
144442560,2sleep10-21swapper/111:05:131
260432530,5sleep121-21rcuc/107:50:161
6981994845,2cyclictest0-21swapper/208:39:292
154412460,1sleep00-21swapper/012:20:160
6981994542,2cyclictest0-21swapper/207:50:372
697599452,24cyclictest20480-21munin-run11:20:011
697599452,24cyclictest20480-21munin-run11:20:011
697599421,31cyclictest2263-21cat11:50:181
697599402,34cyclictest18284-21munin-run11:15:011
8412390,3sleep1840-21grep11:45:261
697599388,18cyclictest0-21swapper/111:35:021
697599381,29cyclictest10097-21latency_hist12:10:021
697599381,28cyclictest23615-21idleruntime11:25:191
697599381,28cyclictest12580-21expr07:20:131
697599381,25cyclictest15271-21/usr/sbin/munin07:25:191
696999379,13cyclictest2886-21munin-run10:40:020
697599368,17cyclictest0-21swapper/108:30:011
697599361,3cyclictest4486-21awk09:30:021
697599361,3cyclictest14100-21grep09:50:141
697599352,28cyclictest3059-40dnf08:10:241
697599351,4cyclictest10012-21memory09:40:171
697599351,25cyclictest24487-21/usr/sbin/munin10:15:211
697599351,24cyclictest11569-21fgrep12:10:251
697599351,22cyclictest10142-21grep07:15:131
697599344,22cyclictest17038-21latency_hist08:45:021
697599343,24cyclictest13266-21if_enp1s012:15:181
697599342,19cyclictest5027-21df09:30:111
697599341,30cyclictest6371-21ntp_states10:45:221
697599341,23cyclictest28265-21/usr/sbin/munin09:10:171
697599341,21cyclictest29997-21cut10:25:211
697599341,21cyclictest17657-21sa110:00:001
697599340,21cyclictest7170-21expr09:35:131
697599331,4cyclictest20850-21ls07:40:011
697599331,3cyclictest27371-21cat11:35:131
697599331,30cyclictest14504-21perf12:24:551
697599331,30cyclictest0-21swapper/109:20:111
697599331,22cyclictest26067-21/usr/sbin/munin09:05:151
697599321,4cyclictest6740-21sshd12:00:191
697599321,3cyclictest13716-21chrt11:03:001
697599321,3cyclictest11582-21latency08:30:191
697599321,28cyclictest6379-21cpuspeed_turbos08:20:121
697599321,28cyclictest0-21swapper/108:15:121
697599321,23cyclictest21421-21idleruntime11:20:161
697599321,21cyclictest23202-21memory10:10:171
697599321,16cyclictest3112-21/usr/sbin/munin10:40:191
697599311,3cyclictest30526-21ntpq11:40:211
697599311,3cyclictest18624-21latency_hist07:35:021
697599311,2cyclictest0-21swapper/110:55:181
697599311,27cyclictest0-21swapper/112:35:151
697599311,20cyclictest0-21swapper/108:55:251
697599311,20cyclictest0-21swapper/108:55:251
697599310,3cyclictest13050-21cstates08:35:121
6969993121,3cyclictest0-21swapper/011:54:570
697599302,24cyclictest17616-21diskstats08:45:151
697599301,4cyclictest0-21swapper/112:30:271
697599301,3cyclictest31098-21cpu10:30:131
6975993011,16cyclictest29839-21/usr/sbin/munin08:00:261
6975993010,17cyclictest8187-21netstat07:10:181
697599300,21cyclictest0-21swapper/110:20:171
697599295,21cyclictest0-21swapper/110:35:171
697599292,4cyclictest24780-21memory09:00:201
697599291,2cyclictest3930-21cstates11:55:131
697599291,2cyclictest17598-21/usr/sbin/munin12:25:161
697599291,2cyclictest17598-21/usr/sbin/munin12:25:161
697599291,2cyclictest0-21swapper/107:45:131
697599291,25cyclictest9478-21turbostat.cron10:55:011
697599290,2cyclictest11293-21/usr/sbin/munin09:45:171
696999291,13cyclictest8872-21munin-run09:40:010
698699287,10cyclictest0-21swapper/311:47:283
697599281,2cyclictest0-21swapper/109:15:141
697599281,25cyclictest0-21swapper/107:40:151
697599281,25cyclictest0-21swapper/107:40:151
697599281,23cyclictest18949-21ntp_states10:00:201
697599280,2cyclictest87350irq/126-enp1s0-08:00:011
697599280,16cyclictest0-21swapper/108:05:131
696999281,14cyclictest7289-21munin-run10:50:020
698699275,5cyclictest0-21swapper/311:22:383
696999271,13cyclictest12304-21munin-run12:15:000
697599261,4cyclictest20189-21/usr/sbin/munin08:50:171
696999262,9cyclictest0-21swapper/007:29:180
6969992616,3cyclictest0-21swapper/009:19:190
696999261,17cyclictest0-21swapper/011:15:060
696999261,17cyclictest0-21swapper/011:15:060
698699257,5cyclictest26003-21tail10:15:253
696999251,10cyclictest28186-21latency11:35:190
696999242,7cyclictest0-21swapper/010:20:210
696999241,9cyclictest0-21swapper/010:53:090
6969992413,3cyclictest0-21swapper/011:23:440
696999241,17cyclictest0-21swapper/010:04:120
698699231,16cyclictest0-21swapper/309:48:443
698199233,7cyclictest17658-21munin-run10:00:012
698199231,5cyclictest0-21swapper/211:54:582
698199231,2cyclictest0-21swapper/208:55:512
698199231,2cyclictest0-21swapper/208:55:512
698199231,10cyclictest29953-21grep08:00:132
6981992310,3cyclictest19878-21awk10:05:012
696999235,15cyclictest5083-21munin-run10:45:020
696999232,14cyclictest87350irq/126-enp1s0-11:42:100
696999231,8cyclictest5228-21tr08:15:230
696999231,16cyclictest0-21swapper/010:55:290
696999231,13cyclictest0-21swapper/011:55:250
698699224,11cyclictest3410-21grep08:10:263
698699221,6cyclictest0-21swapper/312:13:213
698699221,11cyclictest23864-21/usr/sbin/munin09:00:153
698199229,3cyclictest1661-21awk08:10:022
698199226,4cyclictest28046-21perf09:10:022
698199225,8cyclictest25228-21munin-run07:50:012
698199225,8cyclictest0-21swapper/210:15:202
6981992214,3cyclictest0-21swapper/207:36:562
698199221,10cyclictest30056-21chrt09:12:492
696999228,2cyclictest11-21rcuc/009:55:000
696999226,5cyclictest23382-21/usr/sbin/munin10:10:210
696999221,4cyclictest10310-21processes09:40:210
6969992213,4cyclictest0-21swapper/010:25:370
696999221,15cyclictest0-21swapper/011:26:420
696999221,14cyclictest0-21swapper/012:01:290
696999221,14cyclictest0-21swapper/009:57:440
32472220,1sleep230-21rcuc/210:40:132
698699216,5cyclictest39-21rcuc/308:52:263
698699215,6cyclictest4220-21/usr/sbin/munin10:40:243
698699215,5cyclictest0-21swapper/311:00:233
698699213,3cyclictest39-21rcuc/308:00:263
698699212,6cyclictest0-21swapper/311:12:203
698699212,10cyclictest0-21swapper/311:18:153
698699212,10cyclictest0-21swapper/311:18:153
698699211,9cyclictest27125-21ntpq09:05:203
698699211,8cyclictest0-21swapper/312:33:443
698699211,6cyclictest24492-21users11:25:283
698699211,11cyclictest9094-21/usr/sbin/munin09:40:223
698699211,11cyclictest28265-21/usr/sbin/munin09:10:233
698699211,11cyclictest14722-21/usr/sbin/munin12:20:193
698699211,10cyclictest27602-21memory10:20:203
698699211,10cyclictest0-21swapper/312:28:113
698699211,10cyclictest0-21swapper/312:28:113
698199218,8cyclictest909-21sssd_nss08:10:252
698199216,4cyclictest28042-21diskmemload09:37:372
698199215,4cyclictest5734-21runrttasks08:16:072
698199215,3cyclictest13873-21fschecks_count09:50:142
698199213,4cyclictest0-21swapper/210:05:152
698199213,4cyclictest0-21swapper/210:05:152
6981992115,4cyclictest14276-21cut11:05:112
698199211,19cyclictest0-21swapper/211:15:432
698199211,19cyclictest0-21swapper/211:15:432
698199210,20cyclictest0-21swapper/210:24:552
696999218,2cyclictest0-21swapper/007:30:170
696999213,13cyclictest0-21swapper/009:27:260
696999213,10cyclictest0-21swapper/012:30:110
696999212,14cyclictest87350irq/126-enp1s0-11:12:010
696999211,9cyclictest11-21rcuc/008:55:200
696999211,9cyclictest11-21rcuc/008:55:200
696999211,7cyclictest0-21swapper/012:27:120
696999211,7cyclictest0-21swapper/012:27:120
696999211,6cyclictest0-21swapper/010:34:280
696999211,14cyclictest0-21swapper/010:15:410
696999211,14cyclictest0-21swapper/010:06:390
696999211,14cyclictest0-21swapper/010:06:390
696999211,14cyclictest0-21swapper/009:23:560
696999211,11cyclictest9293-21irqstats08:25:180
696999211,11cyclictest6611-21df08:20:140
696999211,11cyclictest28848-21gltestperf09:10:150
698699208,3cyclictest12262-21/usr/sbin/munin07:20:203
698699204,8cyclictest0-21swapper/308:40:243
698699203,15cyclictest39-21rcuc/311:50:253
698699202,6cyclictest8829-21diskstats08:25:143
698699201,7cyclictest18220-21df10:00:133
698699201,6cyclictest9287-21munin-node07:35:293
698699201,6cyclictest31803-21threads09:15:243
698699201,5cyclictest0-21swapper/312:15:253
698699201,3cyclictest0-21swapper/310:05:293
698699201,3cyclictest0-21swapper/310:05:293
698699201,11cyclictest27645-21/usr/sbin/munin07:55:193
698699201,10cyclictest7511-21/usr/sbin/munin10:50:133
698699200,10cyclictest0-21swapper/312:35:253
698199206,3cyclictest87650irq/129-enp1s0-12:05:192
698199205,4cyclictest18056-40dnf11:12:192
698199204,7cyclictest6963-21grep12:00:212
698199204,7cyclictest5850-21grep10:45:152
698199202,9cyclictest5308-21users11:55:262
698199202,9cyclictest28212-21memory11:35:192
698199202,9cyclictest1344-21ntp_states09:20:192
698199202,8cyclictest22288-21/usr/sbin/munin10:10:112
698199202,6cyclictest87550irq/128-enp1s0-11:30:162
698199201,5cyclictest31082-21/usr/sbin/munin10:30:252
698199201,5cyclictest10913-21grep07:15:182
696999206,9cyclictest0-21swapper/009:32:020
696999205,3cyclictest11631-21df09:45:130
696999203,9cyclictest12262-21/usr/sbin/munin07:20:230
696999202,9cyclictest808-21snmpd09:07:510
696999202,10cyclictest17268-21/usr/sbin/munin08:45:240
696999201,7cyclictest0-21swapper/011:30:260
696999201,12cyclictest87350irq/126-enp1s0-12:35:020
696999201,12cyclictest0-21swapper/007:37:330
696999201,11cyclictest808-21snmpd08:52:480
696999201,11cyclictest14688-21grep11:05:150
696999201,10cyclictest8516-21fschecks_count12:05:140
696999200,13cyclictest0-21swapper/011:45:090
33262201,15sleep23458-21missed_timers09:25:182
698699195,3cyclictest12050irq/123-ahci[0009:25:023
698699192,9cyclictest0-21swapper/309:35:393
698699191,9cyclictest0-21swapper/309:57:403
698699191,7cyclictest0-21swapper/311:08:433
698699191,7cyclictest0-21swapper/310:31:163
698699191,7cyclictest0-21swapper/308:20:193
698699191,4cyclictest28882-21/usr/sbin/munin10:25:183
6986991913,4cyclictest24261-21turbostat10:15:013
698699191,11cyclictest0-21swapper/309:27:453
698699191,10cyclictest0-21swapper/311:40:183
698199194,4cyclictest18430-21taskset07:32:372
698199193,7cyclictest29124-21expr10:25:112
698199193,4cyclictest31727-21smart_sda09:15:242
698199193,3cyclictest21312-21/usr/sbin/munin12:35:152
698199193,3cyclictest15404-21hostname12:20:162
698199192,9cyclictest0-21swapper/210:55:202
698199191,8cyclictest0-21swapper/211:20:162
698199191,6cyclictest8575-21ntp_offset10:50:212
698199191,6cyclictest16982-21grep12:25:132
698199191,6cyclictest16982-21grep12:25:132
698199191,5cyclictest13270-21perf09:50:012
698199191,4cyclictest28253-21grep07:55:172
696999191,9cyclictest10051-21memory07:15:050
696999191,7cyclictest13521-21head12:15:200
6969991915,3cyclictest0-21swapper/007:10:150
696999191,10cyclictest29839-21/usr/sbin/munin08:00:260
696999191,10cyclictest23864-21/usr/sbin/munin09:00:160
27652191,2sleep02862-21/usr/sbin/munin08:10:200
229932191,3sleep00-21swapper/007:44:460
229932191,3sleep00-21swapper/007:44:460
698699185,3cyclictest87650irq/129-enp1s0-09:30:113
698699183,6cyclictest0-21swapper/311:37:393
698699182,6cyclictest4415-21df08:15:143
698699181,9cyclictest808-21snmpd11:33:183
698699181,8cyclictest0-21swapper/310:55:233
698699181,6cyclictest0-21swapper/309:50:193
698699181,5cyclictest0-21swapper/310:37:563
698699181,4cyclictest0-21swapper/310:45:423
6986991813,3cyclictest4786-21ntp_states11:55:213
6986991812,3cyclictest32139-21chrt08:05:123
6986991811,4cyclictest9836-21entropy07:15:033
698199186,3cyclictest23718-21kworker/2:111:03:412
698199185,3cyclictest87650irq/129-enp1s0-08:45:152
698199185,3cyclictest13006-21if_enp2s007:20:172
698199183,3cyclictest0-21swapper/210:35:252
698199182,5cyclictest0-21swapper/211:46:072
698199182,3cyclictest9206-21cpuspeed_turbos09:40:122
698199181,8cyclictest0-21swapper/208:53:122
698199181,7cyclictest0-21swapper/209:33:122
698199181,5cyclictest0-21swapper/211:40:192
6981991812,3cyclictest23864-21/usr/sbin/munin09:00:252
6981991811,4cyclictest1-21systemd08:20:182
696999181,9cyclictest27645-21/usr/sbin/munin07:55:120
696999181,5cyclictest13585-21sshd11:01:180
6969991811,4cyclictest802-21sssd_be08:40:270
6969991811,4cyclictest13240-21fschecks_count08:35:150
698699177,2cyclictest10678-21/usr/sbin/munin08:30:183
698699171,7cyclictest8432-21expr12:05:133
698699171,7cyclictest7035-21pmu-power12:00:223
698699171,6cyclictest18184-21memory08:45:193
698699171,6cyclictest16658-21/usr/sbin/munin07:30:193
698699171,15cyclictest0-21swapper/308:56:483
698699171,15cyclictest0-21swapper/308:56:483
698699171,13cyclictest8846-21sshd07:11:193
6986991711,3cyclictest12875-21/usr/sbin/munin08:35:123
698199174,3cyclictest13031-21fschecks_time12:15:162
698199171,7cyclictest21051-21/usr/sbin/munin07:40:252
698199171,7cyclictest21051-21/usr/sbin/munin07:40:252
698199171,4cyclictest15399-21sed07:25:202
6981991712,3cyclictest15072-21/usr/sbin/munin08:40:132
698199171,14cyclictest0-21swapper/207:10:032
696999171,6cyclictest0-21swapper/008:05:160
696999171,4cyclictest12648-21ls08:35:010
696999171,13cyclictest24764-21sed07:45:250
248732171,3sleep30-21swapper/307:45:433
6986991610,3cyclictest16045-21uptime07:25:263
698699160,2cyclictest25514-21cpuspeed_turbos07:50:143
698199163,3cyclictest0-21swapper/208:28:422
696999160,4cyclictest0-21swapper/007:50:170
698699152,4cyclictest21773-21if_enp1s007:40:173
698699152,4cyclictest21773-21if_enp1s007:40:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional