You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-18 - 17:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Sep 18, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
51842238163,24sleep20-21swapper/207:06:182
53412232160,59sleep00-21swapper/007:08:210
54142203165,25sleep30-21swapper/307:09:183
51432202166,24sleep10-21swapper/107:05:451
5568991100,108cyclictest0-21swapper/211:34:372
227512580,4sleep2556899cyclictest11:25:242
145422500,4sleep1556199cyclictest11:08:331
261512470,2sleep39-21rcu_preempt09:05:263
261512470,2sleep39-21rcu_preempt09:05:263
211982460,2sleep00-21swapper/007:40:230
555699402,20cyclictest19963-21hddtemp_smartct11:20:150
556199392,18cyclictest23-21ksoftirqd/111:00:011
555699394,28cyclictest87350irq/126-enp1s0-09:50:320
555699392,26cyclictest2767-21sshd09:26:310
123052390,2sleep339-21rcuc/309:50:143
555699383,29cyclictest25527-40dnf11:33:320
555699382,21cyclictest3678-21munin-run10:45:020
555699381,3cyclictest2343-21grep11:55:120
5572993712,10cyclictest4623-21tr12:00:123
555699371,25cyclictest808-21snmpd07:17:450
555699352,4cyclictest32260-21munin-run11:50:010
557299347,14cyclictest3362-21grep11:55:203
555699342,28cyclictest15331-21ntp_states09:55:200
555699341,6cyclictest0-21swapper/012:18:070
555699341,4cyclictest0-21swapper/010:51:510
555699341,2cyclictest0-21swapper/010:35:260
555699341,2cyclictest0-21swapper/009:15:120
557299338,14cyclictest17709-21/usr/sbin/munin12:30:173
557299333,18cyclictest12871-21seq12:17:233
555699332,27cyclictest22550-21ntp_states11:25:200
555699331,4cyclictest0-21swapper/011:38:210
555699331,28cyclictest25094-21ntp_states07:50:210
555699331,18cyclictest0-21swapper/012:23:400
557299329,10cyclictest21099-21cat08:55:193
557299328,14cyclictest8971-21hddtemp_smartct10:55:153
557299327,12cyclictest25895-21tail10:20:153
557299327,12cyclictest15079-21irqstats09:55:173
557299326,11cyclictest31836-21unixbench_singl11:45:243
557299325,16cyclictest15962-21ntp_states11:10:203
557299325,14cyclictest9700-21grep08:30:133
557299324,10cyclictest0-21swapper/311:50:233
555699324,16cyclictest24652-21/usr/sbin/munin09:05:150
555699324,16cyclictest24652-21/usr/sbin/munin09:05:150
555699323,3cyclictest87550irq/128-enp1s0-09:20:130
555699321,3cyclictest87350irq/126-enp1s0-09:13:430
555699321,3cyclictest19907-21/usr/sbin/munin12:35:140
555699321,2cyclictest87350irq/126-enp1s0-12:10:290
557299318,13cyclictest9245-21expr12:10:133
557299315,14cyclictest31185-21sed10:30:243
556899311,27cyclictest0-21swapper/212:07:572
556899310,16cyclictest0-21swapper/210:44:202
555699316,14cyclictest0-21swapper/008:20:180
555699312,25cyclictest24539-21sed10:15:250
555699312,25cyclictest18672-21/usr/sbin/munin10:05:240
555699311,3cyclictest12050irq/123-ahci[0007:15:020
555699311,3cyclictest0-21swapper/010:10:190
555699311,3cyclictest0-21swapper/010:10:190
555699311,3cyclictest0-21swapper/009:40:130
555699311,2cyclictest87350irq/126-enp1s0-12:25:240
555699311,2cyclictest87350irq/126-enp1s0-12:25:240
555699311,17cyclictest15852-21/usr/sbin/munin08:45:150
5556993110,18cyclictest3928-21cpu10:45:110
557299309,11cyclictest18156-21ntp_states11:15:213
557299308,8cyclictest23035-21grep09:00:153
557299308,12cyclictest28151-21cpuspeed_turbos11:40:113
557299306,15cyclictest0-21swapper/309:35:233
557299305,16cyclictest0-21swapper/309:30:123
557299304,15cyclictest0-21swapper/311:30:193
557299303,10cyclictest1714-21cpu10:40:103
5572993012,6cyclictest14018-21cat11:05:243
556899300,16cyclictest0-21swapper/211:14:212
556899300,15cyclictest0-21swapper/209:57:242
555699307,20cyclictest11317-21fschecks_time07:20:140
555699306,20cyclictest0-21swapper/009:35:270
555699304,16cyclictest87550irq/128-enp1s0-10:20:160
555699303,2cyclictest87650irq/129-enp1s0-10:57:570
555699302,3cyclictest23090-21smart_sda07:45:240
555699302,24cyclictest4639-21munin-run08:20:020
555699302,24cyclictest16872-21munin-run11:15:010
555699301,5cyclictest11828-21/usr/sbin/munin11:00:240
555699301,4cyclictest29002-21/usr/sbin/munin11:40:180
555699301,4cyclictest0-21swapper/011:15:070
555699301,3cyclictest0-21swapper/012:30:290
555699301,26cyclictest13652-21/usr/sbin/munin08:40:220
555699301,25cyclictest22445-21/usr/sbin/munin09:00:140
555699301,20cyclictest0-21swapper/010:25:180
555699301,17cyclictest30360-21ls10:30:160
5556993011,16cyclictest28189-21idleruntime-cro08:00:010
555699300,4cyclictest0-21swapper/009:30:180
264982300,4sleep230-21rcuc/210:20:222
251832300,2sleep20-21swapper/207:50:212
557299297,13cyclictest26936-21/usr/sbin/munin11:35:203
557299297,12cyclictest15512-21/usr/sbin/munin12:25:133
557299297,12cyclictest15512-21/usr/sbin/munin12:25:133
556899291,9cyclictest0-21swapper/212:31:452
556899291,14cyclictest0-21swapper/209:06:112
556899291,14cyclictest0-21swapper/209:06:112
556899291,13cyclictest0-21swapper/207:15:442
556899291,12cyclictest0-21swapper/211:24:572
556899291,11cyclictest0-21swapper/211:06:362
556899291,11cyclictest0-21swapper/208:01:442
556899291,11cyclictest0-21swapper/207:57:242
555699299,17cyclictest7807-21open_files12:05:210
555699298,18cyclictest6068-21unixbench_singl12:00:240
555699296,19cyclictest15265-21cpu07:30:110
555699296,19cyclictest10321-21fschecks_count09:45:150
555699292,24cyclictest11460-21/usr/sbin/munin08:35:230
555699291,4cyclictest808-21snmpd08:57:320
555699291,4cyclictest12868-21/usr/sbin/munin11:05:130
555699291,24cyclictest32609-21awk08:10:020
5556992911,15cyclictest29641-21cut08:00:230
557299286,13cyclictest15240-21/usr/sbin/munin07:30:253
557299286,12cyclictest26372-21cpuspeed_turbos07:55:123
556899281,9cyclictest0-21swapper/208:56:212
556899281,18cyclictest0-21swapper/208:43:132
556899281,18cyclictest0-21swapper/207:20:092
556899281,17cyclictest0-21swapper/212:27:252
556899281,17cyclictest0-21swapper/212:27:252
556899281,17cyclictest0-21swapper/210:32:122
556899281,13cyclictest0-21swapper/207:26:252
556899281,11cyclictest0-21swapper/210:17:002
556899281,10cyclictest0-21swapper/207:30:372
556199288,13cyclictest87650irq/129-enp1s0-10:25:191
556199281,8cyclictest29676-21users11:40:261
5561992810,2cyclictest0-21swapper/112:10:241
555699289,16cyclictest16632-21expr10:00:120
555699287,18cyclictest10315-21ntpq08:30:190
555699282,22cyclictest579-21systemd-journal07:30:000
555699281,4cyclictest17783-21diskstats07:35:130
555699281,3cyclictest555-21wc08:10:130
555699281,17cyclictest0-21swapper/011:50:100
557299278,9cyclictest87350irq/126-enp1s0-08:11:123
556899271,9cyclictest0-21swapper/211:41:552
556899271,24cyclictest0-21swapper/210:05:442
556899271,12cyclictest26622-21diskmemload10:59:162
556899270,9cyclictest0-21swapper/211:17:492
556899270,15cyclictest0-21swapper/211:56:172
556899270,12cyclictest0-21swapper/211:02:282
556899270,12cyclictest0-21swapper/208:27:562
556899270,11cyclictest0-21swapper/211:37:332
556899270,11cyclictest0-21swapper/208:35:282
555699271,3cyclictest7227-21/usr/sbin/munin08:25:100
555699271,22cyclictest0-21swapper/008:50:140
48022270,1sleep10-21swapper/110:45:181
557299268,11cyclictest11186-21ls09:45:243
557299266,13cyclictest87350irq/126-enp1s0-07:40:023
557299265,7cyclictest14427-21pmu-power12:20:213
557299264,12cyclictest0-21swapper/308:15:123
557299264,10cyclictest8694-21latency_hist12:10:023
557299264,10cyclictest827-21runrttasks11:24:113
557299263,13cyclictest0-21swapper/310:50:043
557299261,14cyclictest0-21swapper/309:41:353
556899261,9cyclictest0-21swapper/210:00:082
556899261,11cyclictest0-21swapper/209:36:372
556899261,11cyclictest0-21swapper/209:11:052
556899261,11cyclictest0-21swapper/208:24:362
556899261,11cyclictest0-21swapper/207:11:442
556899260,18cyclictest0-21swapper/212:03:052
556899260,17cyclictest0-21swapper/210:47:522
556899260,11cyclictest0-21swapper/211:52:532
556199262,10cyclictest0-21swapper/110:01:361
557299258,8cyclictest24656-21vmstat10:15:263
557299256,12cyclictest472-21ntp_states10:35:233
557299255,8cyclictest0-21swapper/312:35:253
557299255,7cyclictest0-21swapper/310:45:113
557299255,12cyclictest0-21swapper/308:35:213
557299255,11cyclictest87350irq/126-enp1s0-08:50:183
557299253,13cyclictest17142-21hddtemp_smartct10:00:153
557299251,15cyclictest0-21swapper/311:00:113
5572992510,6cyclictest1061-21acpi09:25:123
556899256,10cyclictest0-21swapper/212:11:572
556899251,9cyclictest0-21swapper/209:02:152
556899251,9cyclictest0-21swapper/207:38:412
556899251,12cyclictest0-21swapper/209:32:532
556899251,12cyclictest0-21swapper/208:52:072
556899251,11cyclictest0-21swapper/209:46:202
556899251,10cyclictest0-21swapper/208:14:542
556899251,10cyclictest0-21swapper/208:08:122
556899251,10cyclictest0-21swapper/207:42:052
556899250,18cyclictest0-21swapper/209:17:252
556899250,13cyclictest0-21swapper/212:20:292
556899250,11cyclictest0-21swapper/210:28:522
556899250,10cyclictest0-21swapper/209:52:322
556899250,10cyclictest0-21swapper/209:25:232
556199251,6cyclictest87650irq/129-enp1s0-12:00:121
556199251,19cyclictest0-21swapper/109:49:211
556199251,17cyclictest0-21swapper/107:41:211
557299246,5cyclictest12050irq/123-ahci[0010:10:023
557299242,15cyclictest22253-21if_enp2s011:25:163
557299241,14cyclictest0-21swapper/309:17:313
556899249,3cyclictest0-21swapper/212:39:412
556899241,12cyclictest0-21swapper/210:10:042
556899241,12cyclictest0-21swapper/210:10:042
5568992410,3cyclictest0-21swapper/208:31:442
556899240,10cyclictest0-21swapper/211:48:332
556899240,10cyclictest0-21swapper/210:52:522
556199247,3cyclictest9267-21tail07:15:171
556199241,16cyclictest221ktimersoftd/109:30:181
557299239,5cyclictest6089-21df07:10:113
557299236,9cyclictest87350irq/126-enp1s0-10:10:123
557299236,9cyclictest87350irq/126-enp1s0-10:10:123
557299235,16cyclictest0-21swapper/310:25:193
557299235,11cyclictest0-21swapper/308:29:013
557299234,16cyclictest15852-21/usr/sbin/munin08:45:153
557299234,12cyclictest0-21swapper/309:20:223
557299234,10cyclictest87350irq/126-enp1s0-09:10:223
5572992315,5cyclictest8542-21irqstats07:15:043
5572992315,5cyclictest28773-21diskstats08:00:143
5572992313,7cyclictest31968-21/usr/sbin/munin08:05:253
556899231,8cyclictest0-21swapper/209:41:092
556899230,10cyclictest0-21swapper/210:39:082
556899230,10cyclictest0-21swapper/209:22:052
556199238,6cyclictest32252-21sh11:50:021
556199237,2cyclictest0-21swapper/108:35:131
556199236,9cyclictest221ktimersoftd/112:25:011
556199232,7cyclictest26613-21latency_hist09:10:021
556199232,7cyclictest26613-21latency_hist09:10:021
5561992318,2cyclictest28420-21chrt08:00:081
556199231,15cyclictest0-21swapper/112:30:111
5561992310,6cyclictest87350irq/126-enp1s0-10:10:111
5561992310,6cyclictest87350irq/126-enp1s0-10:10:111
556199230,6cyclictest0-21swapper/109:56:481
556199230,5cyclictest0-21swapper/109:23:251
557299226,13cyclictest19630-21/usr/sbin/munin07:40:173
557299225,14cyclictest21933-21tr07:45:123
557299221,7cyclictest0-21swapper/308:20:233
5572992213,6cyclictest24089-21/usr/sbin/munin07:50:113
5572992212,7cyclictest11055-21expr07:20:133
556899221,9cyclictest0-21swapper/208:15:342
556899221,7cyclictest0-21swapper/208:46:252
556899220,9cyclictest0-21swapper/207:49:042
556899220,8cyclictest0-21swapper/212:18:172
556199223,12cyclictest0-21swapper/111:35:131
556199221,18cyclictest5988-21grep09:35:171
556199221,16cyclictest0-21swapper/108:12:431
556199221,15cyclictest0-21swapper/108:58:401
556199220,17cyclictest0-21swapper/110:19:381
5572992114,4cyclictest13390-21diskstats07:25:143
5572992112,7cyclictest14281-21gltestperf08:40:143
556199219,4cyclictest24468-21if_err_enp2s011:30:171
556199218,6cyclictest89250irq/135-enp2s0-08:25:131
556199218,4cyclictest87550irq/128-enp1s0-11:50:231
556199217,8cyclictest0-21swapper/110:50:131
556199216,13cyclictest990-21kworker/u8:108:41:361
556199211,9cyclictest170752awk12:25:231
556199211,9cyclictest170752awk12:25:231
5561992116,3cyclictest26622-21diskmemload11:27:001
5561992113,7cyclictest0-21swapper/112:05:151
5561992113,3cyclictest0-21swapper/108:47:081
5561992113,3cyclictest0-21swapper/108:06:441
5561992112,2cyclictest30414-21timerandwakeup09:15:231
556199211,17cyclictest23360-21memory09:00:201
556199211,15cyclictest0-21swapper/112:37:521
556199211,15cyclictest0-21swapper/107:37:371
556199211,14cyclictest0-21swapper/107:53:211
5561992111,2cyclictest2862-21proc_pri10:40:211
556199209,2cyclictest87550irq/128-enp1s0-10:20:251
556199202,15cyclictest21-21rcuc/108:20:131
556199201,8cyclictest0-21swapper/109:42:291
556199201,4cyclictest23-21ksoftirqd/111:14:021
556199201,2cyclictest20508-21rt-features11:20:221
556199201,14cyclictest0-21swapper/108:18:351
556199201,14cyclictest0-21swapper/107:59:371
556199201,14cyclictest0-21swapper/107:47:091
556199201,14cyclictest0-21swapper/107:20:381
556199201,11cyclictest19251-21perl10:05:141
556199201,11cyclictest12751-21taskset12:15:461
556199201,10cyclictest17154-21egrep11:15:131
556199200,15cyclictest87650irq/129-enp1s0-09:50:401
556199200,14cyclictest0-21swapper/108:50:121
556199199,2cyclictest87550irq/128-enp1s0-07:25:151
556199199,2cyclictest15466-21cat07:30:111
556199196,3cyclictest2669-21diskstats11:55:151
556199194,6cyclictest2573-21/usr/sbin/munin09:25:241
556199192,14cyclictest21-21rcuc/111:00:151
5561991915,3cyclictest0-21swapper/110:31:261
556199191,16cyclictest0-21swapper/107:13:011
556199191,14cyclictest0-21swapper/109:11:301
556199191,10cyclictest309-21memory10:35:191
5561991815,2cyclictest0-21swapper/108:32:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional