You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 02:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Jul 23, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7052235162,60sleep00-21swapper/019:06:230
6972235167,22sleep20-21swapper/219:06:172
8302204168,24sleep30-21swapper/319:07:273
7152202165,25sleep10-21swapper/119:06:281
2924921690,5sleep2115099cyclictest20:10:182
3069621620,4sleep3115699cyclictest21:26:003
1150997470,3cyclictest0-21swapper/222:56:322
1150996360,2cyclictest0-21swapper/219:14:452
92872550,3sleep29292-21gltestperf19:25:152
79522530,2sleep081ktimersoftd/021:50:130
44142510,1sleep30-21swapper/319:15:133
241752470,1sleep30-21swapper/323:45:263
193522470,3sleep20-21swapper/222:15:152
12052460,3sleep230-21rcuc/221:35:132
114499451,35cyclictest27522-21hddtemp_smartct21:20:171
288872440,3sleep3401ktimersoftd/320:10:143
121222390,1sleep10-21swapper/119:30:241
114499391,3cyclictest20318-21munin-run23:40:001
114499376,3cyclictest14070-21awk22:05:011
114499375,29cyclictest22462-21gltestperf19:55:171
1144993710,20cyclictest4920-21cat23:04:561
114499361,7cyclictest0-21swapper/121:40:131
114499361,3cyclictest17208-21latency_hist19:45:021
114499361,3cyclictest10638-21cat00:35:021
114499361,31cyclictest13415-21cat20:49:561
114499361,26cyclictest12103-21diskstats23:20:151
114499361,22cyclictest12876-21cat00:40:021
1144993611,17cyclictest4636-21latency_hist20:30:021
114499357,2cyclictest6529-21taskset20:31:101
114499353,3cyclictest4198-21/usr/sbin/munin00:20:171
113899359,13cyclictest817-21munin-run21:35:010
113899359,13cyclictest817-21munin-run21:35:010
114499334,3cyclictest29148-21cat00:00:011
114499334,3cyclictest29148-21cat00:00:011
1144993315,15cyclictest8639-21grep20:35:281
114499331,3cyclictest9264-21uptime21:50:251
114499331,3cyclictest8380-21netstat23:10:201
1144993311,19cyclictest9559-21/usr/sbin/munin23:15:271
1144993311,19cyclictest6390-21/usr/sbin/munin00:25:171
1144993310,20cyclictest15855-21/usr/sbin/munin20:55:241
114499330,6cyclictest23-21ksoftirqd/122:15:331
114499328,17cyclictest826-21awk21:35:011
114499328,17cyclictest826-21awk21:35:011
114499325,2cyclictest30062-21kworker/u8:120:51:331
114499321,5cyclictest0-21swapper/123:27:181
114499321,4cyclictest808-21snmpd00:05:261
114499321,4cyclictest23418-21diskstats22:25:131
114499321,29cyclictest24722-21awk23:50:011
114499321,28cyclictest7410-21cat19:20:191
114499321,28cyclictest3899-21if_enp2s019:15:021
114499321,28cyclictest30695-21sleep22:50:011
114499321,28cyclictest30695-21sleep22:50:011
114499321,23cyclictest419-21latency_hist22:55:021
1144993211,18cyclictest14296-21/usr/sbin/munin19:35:241
1144993210,11cyclictest10296-21fschecks_count21:55:161
114499316,3cyclictest0-21swapper/123:54:291
114499315,2cyclictest30499-21cat21:25:251
114499315,2cyclictest23-21ksoftirqd/122:35:421
114499311,6cyclictest21841-21cat22:20:191
114499311,6cyclictest19405-21latency_hist19:50:021
114499311,3cyclictest18336-21expr21:00:111
1144993111,17cyclictest19634-21/usr/sbin/munin19:50:151
1144993110,18cyclictest28614-21cat20:10:131
114499310,24cyclictest0-21swapper/121:05:191
114499310,22cyclictest0-21swapper/120:40:131
115099309,3cyclictest0-21swapper/219:17:292
114499308,19cyclictest14051-21latency_hist22:05:021
114499302,4cyclictest5513-21diskstats23:05:141
114499302,24cyclictest6722-21wc21:45:231
114499301,4cyclictest28099-21/usr/sbin/munin22:40:241
114499301,3cyclictest26237-21/usr/sbin/munin20:05:291
114499301,2cyclictest0-21swapper/122:55:161
114499301,26cyclictest30758-21expr20:15:121
1144993012,15cyclictest16478-21/usr/sbin/munin22:10:271
1144993010,17cyclictest17093-21missed_timers23:30:191
115699292,17cyclictest22105-21kworker/u8:021:30:233
115699292,17cyclictest22105-21kworker/u8:021:30:233
1156992913,5cyclictest0-21swapper/322:19:283
115699291,19cyclictest0-21swapper/321:46:133
115099291,11cyclictest22226-21diskmemload00:01:562
1150992910,11cyclictest0-21swapper/223:33:372
114499297,2cyclictest2146-21/usr/sbin/munin21:35:221
114499294,4cyclictest763-21dbus-daemon21:10:201
114499292,4cyclictest1749-21swap20:20:221
114499292,4cyclictest1749-21swap20:20:221
114499291,6cyclictest0-21swapper/119:26:331
326182280,1sleep20-21swapper/220:20:012
115099288,3cyclictest7059-21grep19:20:142
1150992811,9cyclictest0-21swapper/222:43:212
115099281,14cyclictest0-21swapper/221:19:092
115099281,10cyclictest0-21swapper/223:52:112
114499284,2cyclictest25011-21/usr/sbin/munin21:15:161
114499283,3cyclictest1448-21grep19:10:121
114499281,5cyclictest30432-40updatedb00:00:211
114499281,4cyclictest2164-21/usr/sbin/munin00:15:111
1144992811,3cyclictest25375-21taskset22:31:371
113899282,14cyclictest25041-21munin-run22:30:010
115699273,8cyclictest0-21swapper/300:25:213
115699271,20cyclictest0-21swapper/321:41:103
115699271,19cyclictest0-21swapper/323:44:563
115699271,19cyclictest0-21swapper/322:30:223
115699271,19cyclictest0-21swapper/321:58:343
115699271,18cyclictest0-21swapper/323:17:573
115699271,18cyclictest0-21swapper/323:00:543
115699271,18cyclictest0-21swapper/322:44:583
115699271,18cyclictest0-21swapper/322:38:133
115699271,18cyclictest0-21swapper/321:15:123
115699271,18cyclictest0-21swapper/300:34:283
115699271,18cyclictest0-21swapper/300:17:313
115099278,3cyclictest0-21swapper/220:39:492
115099272,13cyclictest11519-21kworker/u8:222:38:132
1150992711,8cyclictest0-21swapper/219:56:572
115099271,13cyclictest0-21swapper/221:20:572
115099271,12cyclictest3637-21hddtemp_smartct23:00:172
115099271,11cyclictest0-21swapper/219:43:052
115099271,10cyclictest0-21swapper/200:35:532
114499273,4cyclictest24946-21/usr/sbin/munin20:00:191
114499271,4cyclictest20626-21cpuspeed_turbos23:40:141
114499270,9cyclictest0-21swapper/100:12:451
113899270,12cyclictest30405-21munin-run20:15:010
1156992614,3cyclictest0-21swapper/322:54:083
115699261,19cyclictest0-21swapper/322:58:513
115699261,18cyclictest0-21swapper/321:36:433
115699261,18cyclictest0-21swapper/321:20:213
115699261,18cyclictest0-21swapper/300:09:503
115699261,17cyclictest0-21swapper/323:55:053
115699261,17cyclictest0-21swapper/323:55:053
115699261,17cyclictest0-21swapper/300:22:143
115699261,17cyclictest0-21swapper/300:01:243
115099269,9cyclictest0-21swapper/219:38:012
115099269,3cyclictest0-21swapper/223:06:372
115099269,3cyclictest0-21swapper/222:24:292
115099269,3cyclictest0-21swapper/219:48:052
115099268,3cyclictest0-21swapper/220:22:172
115099268,3cyclictest0-21swapper/220:22:172
115099268,3cyclictest0-21swapper/220:01:292
115099261,17cyclictest32064-21irqstats21:30:182
115099261,17cyclictest32064-21irqstats21:30:182
1150992611,3cyclictest60922chrt00:24:132
115099261,13cyclictest0-21swapper/222:30:452
115099261,12cyclictest0-21swapper/223:59:042
115099261,12cyclictest0-21swapper/223:59:042
115099261,11cyclictest0-21swapper/222:52:292
115099261,11cyclictest0-21swapper/222:11:052
115099261,11cyclictest0-21swapper/220:26:012
115099261,10cyclictest30113-21ntp_states21:25:202
1150992610,9cyclictest0-21swapper/219:31:212
1150992610,8cyclictest0-21swapper/223:19:492
1150992610,8cyclictest0-21swapper/220:40:332
1150992610,3cyclictest22226-21diskmemload00:34:092
1150992610,3cyclictest0-21swapper/221:09:052
113899261,13cyclictest12823-21munin-run19:35:010
115699255,13cyclictest7018-21kworker/u8:023:20:473
115699251,18cyclictest0-21swapper/322:08:553
115699251,17cyclictest0-21swapper/323:53:153
115699251,17cyclictest0-21swapper/300:11:013
115099259,3cyclictest0-21swapper/223:26:452
115099259,3cyclictest0-21swapper/220:07:252
115099259,3cyclictest0-21swapper/219:52:332
115099259,3cyclictest0-21swapper/200:11:052
115099258,3cyclictest0-21swapper/223:47:292
115099258,3cyclictest0-21swapper/222:06:372
115099258,3cyclictest0-21swapper/220:50:332
115099251,11cyclictest0-21swapper/221:45:212
1150992510,3cyclictest0-21swapper/223:10:452
1150992510,3cyclictest0-21swapper/220:31:452
1150992510,3cyclictest0-21swapper/200:29:492
115099250,11cyclictest0-21swapper/222:28:492
115099250,11cyclictest0-21swapper/220:46:372
115699241,17cyclictest0-21swapper/323:30:153
115699241,17cyclictest0-21swapper/323:12:403
115699241,17cyclictest0-21swapper/300:35:493
115699241,16cyclictest0-21swapper/323:39:303
115699241,16cyclictest0-21swapper/323:29:503
115699241,16cyclictest0-21swapper/321:50:253
115099249,3cyclictest0-21swapper/220:56:332
115099248,3cyclictest0-21swapper/222:46:052
115099248,3cyclictest0-21swapper/222:46:052
115099248,3cyclictest0-21swapper/221:50:572
115099248,3cyclictest0-21swapper/200:07:202
115099248,2cyclictest0-21swapper/221:13:492
115099247,3cyclictest0-21swapper/221:01:532
115099241,8cyclictest13143-21ntp_states22:00:212
115099241,10cyclictest0-21swapper/221:42:222
1150992410,3cyclictest0-21swapper/223:37:372
1150992410,3cyclictest0-21swapper/223:20:132
115099240,12cyclictest0-21swapper/221:56:592
113899242,4cyclictest22233-21munin-run21:10:010
113899241,11cyclictest24029-21/usr/sbin/munin20:00:150
115699233,4cyclictest21942-21ntp_states22:20:213
1156992313,2cyclictest87350irq/126-enp1s0-19:20:253
115699231,16cyclictest0-21swapper/322:26:513
115699231,15cyclictest0-21swapper/321:11:003
115099238,3cyclictest0-21swapper/223:42:052
113899239,3cyclictest29326-21/usr/sbin/munin00:00:140
113899231,10cyclictest31547-21/usr/sbin/munin00:05:270
115699224,9cyclictest8759-21chrt20:36:363
115699222,8cyclictest87350irq/126-enp1s0-21:10:013
115699221,16cyclictest0-21swapper/323:05:083
115699221,16cyclictest0-21swapper/322:12:273
115699221,15cyclictest0-21swapper/322:47:013
115699221,15cyclictest0-21swapper/322:47:013
115699221,15cyclictest0-21swapper/322:02:493
115099221,11cyclictest0-21swapper/200:19:132
113899228,3cyclictest11458-21/usr/sbin/munin20:45:150
113899221,10cyclictest18673-21/usr/sbin/munin22:15:230
113899220,20cyclictest0-21swapper/023:00:230
113899220,20cyclictest0-21swapper/022:10:220
113899220,1cyclictest0-21swapper/022:55:220
115699213,9cyclictest2249-21cat19:10:183
115699212,10cyclictest20027-21latency_hist21:05:023
1156992118,2cyclictest0-21swapper/320:15:243
1156992115,3cyclictest20242-21grep19:50:143
115699211,11cyclictest4872-21/usr/sbin/munin20:30:133
115699211,11cyclictest11760-21memory19:30:183
113899219,2cyclictest87550irq/128-enp1s0-22:30:360
113899214,15cyclictest87350irq/126-enp1s0-23:45:170
113899211,9cyclictest808-21snmpd19:59:160
1138992116,3cyclictest4773-21grep21:40:250
113899210,19cyclictest0-21swapper/021:10:200
113899210,19cyclictest0-21swapper/020:50:220
115699209,3cyclictest21832-21/usr/sbin/munin19:55:203
115699201,9cyclictest0-21swapper/320:28:273
115699201,7cyclictest17441-21/usr/sbin/munin19:45:123
1156992017,2cyclictest0-21swapper/320:24:403
1156992017,2cyclictest0-21swapper/320:24:403
113899206,3cyclictest773-21NetworkManager21:03:080
113899205,3cyclictest27735-21gltestperf23:55:170
113899205,3cyclictest27735-21gltestperf23:55:170
113899201,8cyclictest0-21swapper/022:51:550
1138992013,4cyclictest6163-21dump-pmu-power00:25:010
113899201,16cyclictest6853-21basename20:35:010
113899201,16cyclictest28717-21/usr/sbin/munin22:45:160
113899201,16cyclictest28717-21/usr/sbin/munin22:45:160
113899200,19cyclictest0-21swapper/019:40:230
115699195,3cyclictest87350irq/126-enp1s0-19:40:123
1156991917,1cyclictest0-21swapper/319:35:233
1156991916,2cyclictest0-21swapper/320:47:253
115699191,10cyclictest8641-21sshd19:25:053
115699190,9cyclictest0-21swapper/320:56:263
115699190,10cyclictest0-21swapper/320:05:203
113899198,3cyclictest3983-21awk00:20:010
113899195,3cyclictest87550irq/128-enp1s0-20:24:240
113899195,3cyclictest87550irq/128-enp1s0-20:24:240
113899195,3cyclictest808-21snmpd19:24:100
113899192,13cyclictest8978-21fschecks_count00:30:140
113899191,6cyclictest10234-21hddtemp_smartct23:15:170
113899191,5cyclictest1-21systemd22:20:210
113899191,15cyclictest9880-21/usr/sbin/munin21:55:170
113899191,15cyclictest9261-21/usr/sbin/munin20:40:110
113899191,15cyclictest29050-21/usr/sbin/munin21:25:240
113899191,15cyclictest12078-21/usr/sbin/munin22:00:200
140762180,2sleep00-21swapper/022:05:030
115699184,4cyclictest26015-21perf20:05:003
115699184,3cyclictest87650irq/129-enp1s0-20:50:183
115699181,5cyclictest9261-21/usr/sbin/munin20:40:153
113899188,7cyclictest7320-21expr20:35:130
113899187,2cyclictest0-21swapper/000:35:220
113899185,2cyclictest21765-21gzip23:40:250
113899184,3cyclictest22226-21diskmemload23:05:040
113899183,2cyclictest0-21swapper/021:35:200
113899181,7cyclictest15855-21/usr/sbin/munin20:55:180
113899181,5cyclictest4254-21/usr/sbin/munin19:15:150
113899181,5cyclictest26850-21/usr/sbin/munin21:20:260
113899181,4cyclictest808-21snmpd20:19:230
113899181,3cyclictest87350irq/126-enp1s0-20:25:160
1138991812,3cyclictest28209-21perf20:10:010
113899181,14cyclictest808-21snmpd19:29:110
113899180,9cyclictest0-21swapper/021:45:160
137142171,2sleep00-21swapper/023:25:010
113899175,3cyclictest87350irq/126-enp1s0-23:10:240
113899173,3cyclictest22226-21diskmemload23:30:370
113899171,8cyclictest0-21swapper/022:44:510
113899171,8cyclictest0-21swapper/000:14:490
113899171,6cyclictest0-21swapper/023:27:090
113899171,5cyclictest18345-21/usr/sbin/munin23:35:170
113899171,4cyclictest18470-21ntp_kernel_err19:45:220
113899171,4cyclictest13046-21/usr/sbin/munin19:35:210
1138991711,4cyclictest25988-21sadc23:50:210
113899164,2cyclictest87350irq/126-enp1s0-00:25:100
1138991612,3cyclictest0-21swapper/022:36:260
1138991611,3cyclictest1932-21cut19:10:140
1138991610,3cyclictest21063-21timerwakeupswit19:50:240
113899160,15cyclictest0-21swapper/021:17:540
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional