You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 10:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 29, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
93952238163,23sleep30-21swapper/319:08:113
92652238162,25sleep00-21swapper/019:06:300
93262237164,24sleep20-21swapper/219:07:172
92562203166,24sleep10-21swapper/119:06:231
963199109105,3cyclictest0-21swapper/220:44:152
324572600,4sleep0961999cyclictest21:10:260
305772570,3sleep30-21swapper/322:20:203
190922480,1sleep30-21swapper/320:40:233
168982330,4sleep0961999cyclictest00:15:270
961999328,11cyclictest22094-21munin-run20:50:010
962699311,10cyclictest0-21swapper/119:55:031
961999308,11cyclictest11734-21awk21:40:010
962699291,9cyclictest0-21swapper/122:32:211
962699290,12cyclictest0-21swapper/100:26:171
962699290,11cyclictest0-21swapper/122:38:051
962699290,11cyclictest0-21swapper/121:50:301
962699281,11cyclictest0-21swapper/100:15:291
962699280,4cyclictest0-21swapper/123:55:211
962699280,11cyclictest0-21swapper/120:07:171
961999283,12cyclictest24925-21awk22:10:010
962699271,9cyclictest0-21swapper/123:25:371
962699271,11cyclictest0-21swapper/121:28:421
962699271,11cyclictest0-21swapper/120:31:451
962699271,11cyclictest0-21swapper/119:21:581
962699271,10cyclictest30885-21diskmemload22:21:371
962699270,9cyclictest0-21swapper/120:14:051
962699270,11cyclictest0-21swapper/100:12:131
962699270,10cyclictest0-21swapper/121:02:021
962699270,10cyclictest0-21swapper/120:53:291
963199269,3cyclictest87650irq/129-enp1s0-23:30:132
963199261,19cyclictest0-21swapper/223:01:052
962699265,11cyclictest0-21swapper/122:15:231
962699261,9cyclictest0-21swapper/119:13:531
962699261,15cyclictest0-21swapper/123:00:311
962699261,11cyclictest0-21swapper/121:09:241
962699261,10cyclictest495-21jbd2/dm-0-822:05:291
962699260,8cyclictest0-21swapper/120:27:171
962699260,8cyclictest0-21swapper/120:23:491
962699260,24cyclictest0-21swapper/122:46:091
962699260,24cyclictest0-21swapper/100:34:381
962699260,12cyclictest0-21swapper/123:40:211
962699260,10cyclictest0-21swapper/121:49:101
9636992516,3cyclictest0-21swapper/323:15:113
963199257,3cyclictest87350irq/126-enp1s0-21:10:182
9631992511,4cyclictest87650irq/129-enp1s0-22:45:022
962699251,9cyclictest0-21swapper/122:26:451
962699251,9cyclictest0-21swapper/119:47:491
962699251,9cyclictest0-21swapper/119:33:341
962699251,11cyclictest0-21swapper/123:38:111
962699251,11cyclictest0-21swapper/120:48:291
962699251,10cyclictest0-21swapper/123:05:211
962699251,10cyclictest0-21swapper/121:16:141
962699250,9cyclictest0-21swapper/123:32:451
962699250,9cyclictest0-21swapper/120:36:531
962699250,18cyclictest0-21swapper/123:10:171
962699250,11cyclictest0-21swapper/123:20:211
962699250,11cyclictest0-21swapper/123:20:211
962699250,10cyclictest0-21swapper/121:13:421
961999251,9cyclictest30885-21diskmemload23:45:200
9636992416,3cyclictest0-21swapper/322:10:543
962699241,9cyclictest0-21swapper/120:02:051
962699241,8cyclictest0-21swapper/121:57:161
962699241,8cyclictest0-21swapper/119:16:371
962699241,8cyclictest0-21swapper/100:37:381
962699241,11cyclictest0-21swapper/119:52:591
962699241,11cyclictest0-21swapper/119:52:591
962699241,10cyclictest0-21swapper/121:34:441
962699241,10cyclictest0-21swapper/120:19:151
962699240,9cyclictest0-21swapper/119:37:171
962699240,8cyclictest0-21swapper/122:44:211
962699240,8cyclictest0-21swapper/122:13:491
962699240,8cyclictest0-21swapper/120:42:331
962699240,10cyclictest0-21swapper/121:36:061
963199235,5cyclictest0-21swapper/223:50:242
962699231,7cyclictest0-21swapper/123:47:121
962699231,7cyclictest0-21swapper/100:00:411
962699231,11cyclictest0-21swapper/119:44:191
962699231,10cyclictest0-21swapper/121:20:041
962699230,9cyclictest0-21swapper/121:43:221
962699230,8cyclictest0-21swapper/122:04:381
962699230,8cyclictest0-21swapper/100:07:371
962699230,7cyclictest0-21swapper/120:59:411
962699230,7cyclictest0-21swapper/119:27:221
961999234,16cyclictest30894-21munin-run21:10:020
961999233,11cyclictest27337-21/usr/sbin/munin22:15:130
961999231,16cyclictest0-21swapper/021:46:550
963699228,2cyclictest0-21swapper/319:35:193
9636992217,3cyclictest2062-21sshd20:02:363
963699221,14cyclictest0-21swapper/322:34:213
963199226,3cyclictest0-21swapper/221:05:092
963199224,7cyclictest87350irq/126-enp1s0-23:35:122
963199223,7cyclictest0-21swapper/200:37:242
963199221,7cyclictest13045-21sshd20:26:512
962699223,3cyclictest0-21swapper/123:18:201
962699221,18cyclictest4731-21diskstats23:50:131
962699221,10cyclictest0-21swapper/122:56:591
962699220,9cyclictest0-21swapper/100:22:571
962699220,10cyclictest0-21swapper/122:51:171
961999222,5cyclictest12344-21munin-run22:55:010
961999221,3cyclictest4741-21users21:20:260
191542220,3sleep20-21swapper/200:20:262
963699213,6cyclictest41-21ksoftirqd/321:10:253
9636992117,3cyclictest0-21swapper/321:38:523
9636992115,3cyclictest16246-21grep23:00:273
963699211,14cyclictest0-21swapper/322:48:543
9636992111,3cyclictest8298-21sshd23:56:533
9636992111,3cyclictest20962-21sshd23:12:533
963199213,7cyclictest0-21swapper/221:34:542
963199213,6cyclictest0-21swapper/221:46:552
963199212,4cyclictest0-21swapper/223:40:172
963199211,7cyclictest0-21swapper/223:19:452
963199211,6cyclictest24513-21/usr/sbin/munin20:55:182
963199211,5cyclictest11828-21kernelversion00:05:182
963199211,5cyclictest0-21swapper/221:22:372
963199211,4cyclictest0-21swapper/221:17:392
963199211,14cyclictest0-21swapper/222:14:342
963199211,12cyclictest0-21swapper/220:35:222
961999212,5cyclictest22305-1kworker/u9:222:34:440
961999211,9cyclictest808-21snmpd19:10:250
961999211,9cyclictest4106-21df22:35:130
961999211,11cyclictest808-21snmpd19:40:320
961999211,11cyclictest16682-21kworker/u8:123:20:020
961999211,11cyclictest1-21systemd23:59:550
961999211,11cyclictest0-21swapper/023:32:550
9619992110,3cyclictest5971-21users23:50:260
54722210,4sleep3963699cyclictest23:50:213
188332211,2sleep30-21swapper/300:20:243
963699205,3cyclictest87350irq/126-enp1s0-23:31:093
963699202,10cyclictest4733-21/usr/sbin/munin20:10:153
9636992017,2cyclictest39-21rcuc/319:56:083
963699201,6cyclictest0-21swapper/322:19:233
963699201,6cyclictest0-21swapper/300:39:293
9636992016,3cyclictest0-21swapper/322:06:483
963699201,11cyclictest16344-21/usr/sbin/munin21:50:253
963699201,10cyclictest401ktimersoftd/321:30:173
963699201,10cyclictest20175-21sshd21:55:533
963699200,2cyclictest89150irq/134-enp2s0-19:50:013
963199208,7cyclictest20408-40dnf21:58:562
963199208,2cyclictest87650irq/129-enp1s0-21:36:572
963199204,8cyclictest87350irq/126-enp1s0-00:11:272
963199204,4cyclictest25236-21chrt23:21:232
963199204,4cyclictest25236-21chrt23:21:232
963199204,4cyclictest10772-21sshd20:20:502
963199204,2cyclictest89050irq/133-enp2s0-19:50:142
963199204,2cyclictest89050irq/133-enp2s0-19:50:142
963199203,4cyclictest30-21rcuc/220:45:192
963199202,8cyclictest87650irq/129-enp1s0-00:25:152
963199201,7cyclictest0-21swapper/220:20:012
963199201,6cyclictest32-21ksoftirqd/222:10:022
963199201,5cyclictest22578-21chrt22:03:222
9631992015,3cyclictest9436-21gltestperf00:00:162
963199201,4cyclictest0-21swapper/221:54:492
963199201,12cyclictest0-21swapper/223:55:062
963199201,12cyclictest0-21swapper/219:55:152
963199200,9cyclictest311ktimersoftd/223:50:012
963199200,8cyclictest0-21swapper/222:20:132
963199200,6cyclictest30-21rcuc/200:34:062
963199200,10cyclictest30-21rcuc/223:10:222
961999209,3cyclictest11333-21/usr/sbin/munin20:25:230
961999203,15cyclictest24299-21basename20:55:010
961999202,10cyclictest13213-21/usr/sbin/munin00:10:160
961999201,8cyclictest0-21swapper/021:55:250
961999201,7cyclictest191582sed00:20:270
961999201,7cyclictest0-21swapper/023:02:410
961999201,4cyclictest0-21swapper/021:28:280
961999201,11cyclictest808-21snmpd19:50:330
961999201,11cyclictest808-21snmpd19:50:330
961999201,11cyclictest29876-21df_inode22:20:130
961999201,11cyclictest24193-21/usr/sbin/munin00:35:180
961999201,11cyclictest0-21swapper/023:05:380
961999201,11cyclictest0-21swapper/022:41:370
961999201,11cyclictest0-21swapper/022:10:380
961999201,11cyclictest0-21swapper/022:00:230
961999201,11cyclictest0-21swapper/021:43:300
961999201,11cyclictest0-21swapper/000:33:370
961999201,11cyclictest0-21swapper/000:05:590
961999201,10cyclictest32697-21diskstats23:40:170
961999201,10cyclictest0-21swapper/022:48:580
963699198,9cyclictest0-21swapper/320:35:173
963699198,3cyclictest13664-21sshd21:41:523
963699197,3cyclictest23556-21/usr/sbin/munin23:20:173
963699197,3cyclictest23556-21/usr/sbin/munin23:20:173
963699197,2cyclictest87350irq/126-enp1s0-22:50:263
963699196,3cyclictest87650irq/129-enp1s0-21:15:183
963699191,10cyclictest1146-21sshd22:27:553
963199196,3cyclictest30-21rcuc/222:25:432
963199195,5cyclictest0-21swapper/222:52:162
963199195,3cyclictest87550irq/128-enp1s0-20:00:122
963199194,3cyclictest87650irq/129-enp1s0-21:42:232
963199193,6cyclictest0-21swapper/222:45:582
963199193,3cyclictest32-21ksoftirqd/222:30:132
963199192,9cyclictest32-21ksoftirqd/222:37:042
963199192,7cyclictest0-21swapper/223:25:422
963199192,6cyclictest30-21rcuc/219:20:012
963199192,6cyclictest0-21swapper/221:29:432
963199191,6cyclictest12565-21/usr/sbin/munin22:55:222
961999196,5cyclictest0-21swapper/023:10:040
961999192,6cyclictest14688-21munin-run19:20:020
961999191,4cyclictest0-21swapper/021:16:080
961999191,10cyclictest30885-21diskmemload23:20:100
961999191,10cyclictest30885-21diskmemload23:20:100
961999191,10cyclictest2536-21/usr/sbin/munin20:05:140
961999191,10cyclictest19307-21/usr/sbin/munin19:30:130
961999191,10cyclictest0-21swapper/023:35:210
961999191,10cyclictest0-21swapper/021:54:110
961999191,10cyclictest0-21swapper/021:30:300
961999191,10cyclictest0-21swapper/000:25:500
961999191,10cyclictest0-21swapper/000:01:270
963699187,5cyclictest0-21swapper/320:15:163
963699181,6cyclictest0-21swapper/320:50:133
963699181,5cyclictest0-21swapper/321:45:363
963699181,5cyclictest0-21swapper/300:08:573
963699181,3cyclictest89150irq/134-enp2s0-19:55:013
963699181,3cyclictest89150irq/134-enp2s0-19:55:013
9636991813,3cyclictest4217-21ntp_states21:20:223
963699181,15cyclictest0-21swapper/322:56:093
963699181,15cyclictest0-21swapper/300:15:123
9636991811,4cyclictest31028-21cat21:10:013
963699181,10cyclictest0-21swapper/323:35:153
9636991810,2cyclictest15106-21taskset00:14:113
963699180,6cyclictest41-21ksoftirqd/300:33:363
963199184,7cyclictest87650irq/129-enp1s0-22:15:162
963199182,3cyclictest30-21rcuc/223:05:142
963199181,4cyclictest0-21swapper/200:19:142
9631991812,3cyclictest4959-21cat20:10:132
961999182,10cyclictest7-21ksoftirqd/020:02:350
961999181,9cyclictest0-21swapper/023:25:140
961999181,7cyclictest22158-21grep19:35:170
961999181,7cyclictest0-21swapper/020:40:370
961999181,10cyclictest12950-21fschecks_count22:55:140
961999181,10cyclictest0-21swapper/022:25:260
168092181,2sleep00-21swapper/019:24:120
16152181,2sleep30-21swapper/323:40:513
963699172,12cyclictest9129-21/usr/sbin/munin20:20:163
963699172,12cyclictest11942-21grep20:25:173
963699171,7cyclictest0-21swapper/322:41:103
963699171,6cyclictest0-21swapper/319:20:573
963699171,5cyclictest21806-21ntp_states22:00:213
963699171,5cyclictest19802-21/usr/sbin/munin00:25:173
9636991713,3cyclictest0-21swapper/319:32:563
963699171,13cyclictest24422-21taskset19:40:173
9636991711,3cyclictest19144-21cut23:10:013
963699171,10cyclictest0-21swapper/321:26:563
9636991710,4cyclictest13305-21grep19:15:153
9636991710,4cyclictest10899-21ntpq19:10:233
963699170,7cyclictest87350irq/126-enp1s0-23:25:203
963699170,4cyclictest0-21swapper/320:55:133
963199173,2cyclictest89050irq/133-enp2s0-19:20:172
963199171,5cyclictest0-21swapper/220:30:262
963199171,3cyclictest0-21swapper/219:47:152
9631991711,3cyclictest17390-21expr19:25:152
961999178,2cyclictest11119-21awk20:25:010
961999171,6cyclictest0-21swapper/019:45:150
961999171,15cyclictest0-21swapper/020:58:070
961999171,15cyclictest0-21swapper/020:16:060
961999171,11cyclictest0-21swapper/019:55:230
963699165,3cyclictest87350irq/126-enp1s0-00:05:013
963699161,5cyclictest0-21swapper/320:05:123
963699161,4cyclictest13528-21/usr/sbin/munin20:30:163
963699161,4cyclictest0-21swapper/320:45:183
9636991612,3cyclictest0-21swapper/322:36:293
963699161,13cyclictest0-21swapper/323:49:413
963699161,12cyclictest839-21lldpd19:27:073
963199169,4cyclictest22907-21gltestperf20:50:152
963199169,4cyclictest0-21swapper/221:00:122
963199169,3cyclictest25668-21dump-pmu-power19:45:012
963199163,10cyclictest19324-21cpu19:30:122
9631991612,3cyclictest0-21swapper/220:08:042
9631991612,3cyclictest0-21swapper/219:35:492
9631991610,3cyclictest10862-21ntpq19:10:212
961999169,4cyclictest909-21sssd_nss20:30:140
961999161,5cyclictest0-21swapper/020:35:270
961999161,2cyclictest0-21swapper/019:25:210
9619991611,3cyclictest4733-21/usr/sbin/munin20:10:260
284952161,3sleep30-21swapper/321:02:393
961999159,4cyclictest1-21systemd21:00:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional