You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 10:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Jun 18, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
106752239163,23sleep20-21swapper/219:05:532
109582233159,60sleep00-21swapper/019:09:340
109742223194,19sleep10-21swapper/119:09:471
108522204165,26sleep30-21swapper/319:08:123
2891821410,4sleep11108299cyclictest21:00:181
176392560,4sleep230-21rcuc/200:15:202
11086995046,3cyclictest0-21swapper/220:52:512
234562470,2sleep10-21swapper/100:30:121
11082994114,12cyclictest15185-21awk21:45:011
11082993915,12cyclictest3577-21awk20:05:011
11082993915,12cyclictest3577-21awk20:05:011
11082993912,23cyclictest30703-21uniq21:05:141
11082993718,11cyclictest20600-21/usr/sbin/munin21:55:171
11082993715,13cyclictest32369-21/usr/sbin/munin21:10:151
11082993715,13cyclictest32369-21/usr/sbin/munin21:10:151
11082993715,12cyclictest790-21in:imjournal21:40:011
11082993711,13cyclictest2265-21cpu21:15:111
11082993710,15cyclictest5842-21timerwakeupswit21:20:231
1107499379,13cyclictest5775-21awk20:10:010
11082993613,20cyclictest25544-21awk20:55:011
11082993610,12cyclictest8566-21latency_hist21:30:011
11082993610,11cyclictest9407-21/usr/sbin/munin22:45:251
1108299358,19cyclictest16562-21perl23:00:151
11082993517,15cyclictest8491-21hddtemp_smartct23:55:151
11082993511,11cyclictest20180-21latency_hist23:10:011
1108299348,18cyclictest0-21swapper/123:51:531
11082993417,14cyclictest6073-21/usr/sbin/munin22:35:211
11082993415,16cyclictest12025-21awk00:05:011
11082993415,11cyclictest9350-21proc_pri20:15:211
11082993414,17cyclictest7558-21diskstats22:40:151
11082993411,11cyclictest0-21swapper/121:50:191
1107499346,13cyclictest26178-21idleruntime-cro22:10:010
1108699331,16cyclictest0-21swapper/223:53:032
1108699331,16cyclictest0-21swapper/223:30:072
1108699331,16cyclictest0-21swapper/223:30:072
11082993315,5cyclictest21672-21smart_sda23:10:241
11082993313,6cyclictest26345-21/usr/sbin/munin00:35:161
11082993311,6cyclictest32441-21ntpq23:35:201
1108699321,24cyclictest0-21swapper/221:39:172
1108699321,15cyclictest6392-21latency_hist21:25:002
1108699321,13cyclictest0-21swapper/222:00:142
11082993215,14cyclictest26747-21users19:40:271
11082993214,14cyclictest15790-21munin-run23:00:021
11082993213,15cyclictest28933-21diskstats22:15:131
11082993212,17cyclictest18955-21perf20:40:021
11082993211,9cyclictest10381-21/usr/sbin/munin20:20:231
11082993210,4cyclictest29871-21hddtemp_smartct23:30:151
11082993210,4cyclictest29871-21hddtemp_smartct23:30:151
11082993210,19cyclictest14554-21turbostat.cron20:30:021
11082993210,11cyclictest19912-21ntp_states00:20:211
1108299317,22cyclictest0-21swapper/100:05:141
11082993115,10cyclictest31854-21ntp_states22:20:211
11082993115,10cyclictest31854-21ntp_states22:20:211
11082993114,14cyclictest9869-21ntp_states21:30:211
11082993112,7cyclictest22960-21/usr/sbin/munin19:35:131
11082993112,6cyclictest808-21snmpd20:33:181
11082993112,16cyclictest13535-21unixbench_singl19:15:021
11082993111,17cyclictest28354-21crond22:15:011
11082993110,8cyclictest30312-21if_enp2s019:50:181
1108699301,13cyclictest0-21swapper/220:47:192
1108299307,7cyclictest21585-21fschecks_time00:25:141
1108299307,4cyclictest26173-21timerandwakeup23:20:251
1108299306,12cyclictest0-21swapper/123:25:201
11082993013,9cyclictest16312-21taskset00:13:471
11082993012,8cyclictest11612-21/usr/sbin/munin22:50:201
11082993011,16cyclictest21784-21fschecks_count20:45:141
11082993011,16cyclictest16800-21cut00:15:121
11082993010,17cyclictest524-21/usr/sbin/munin22:25:111
11082993010,17cyclictest14174-21/usr/sbin/munin19:15:111
1107499305,12cyclictest4195-21idleruntime-cro21:20:010
1108699291,15cyclictest0-21swapper/223:46:232
1108699291,13cyclictest0-21swapper/221:27:192
1108699291,13cyclictest0-21swapper/221:16:142
1108699291,13cyclictest0-21swapper/220:00:502
1108699291,13cyclictest0-21swapper/220:00:502
1108699291,13cyclictest0-21swapper/219:41:262
1108699291,11cyclictest0-21swapper/222:05:122
1108699291,11cyclictest0-21swapper/219:53:582
1108299299,12cyclictest87550irq/128-enp1s0-20:10:171
11082992912,10cyclictest766-21ntpd23:46:151
11082992910,11cyclictest5758-21latency_hist20:10:021
1107499293,13cyclictest24579-21munin-run23:20:010
272062280,4sleep2311ktimersoftd/222:10:182
1108699281,13cyclictest0-21swapper/221:44:542
1108699281,13cyclictest0-21swapper/200:01:512
1108299286,12cyclictest0-21swapper/119:25:181
11082992820,5cyclictest20766-21users20:40:271
11082992819,6cyclictest32083-21/usr/sbin/munin19:55:141
11082992813,4cyclictest23677-21ntp_states23:15:211
11082992811,14cyclictest2694-21unixbench-2d23:40:241
11082992810,7cyclictest21986-21/usr/sbin/munin22:00:161
11082992810,15cyclictest26157-21sh22:10:011
1107499282,13cyclictest6986-21munin-run22:40:010
1108699271,13cyclictest0-21swapper/223:36:072
1108699271,12cyclictest0-21swapper/222:26:112
1108699271,10cyclictest0-21swapper/222:35:162
1108699270,13cyclictest0-21swapper/221:14:072
1108699270,13cyclictest0-21swapper/221:14:072
11082992719,5cyclictest3168-21diskstats22:30:131
11082992718,6cyclictest16958-21/usr/sbin/munin19:20:171
11082992717,7cyclictest26682-21memory20:55:181
11082992714,10cyclictest29330-21awk19:50:011
11082992710,14cyclictest15553-21cpuspeed_turbos21:45:111
1107499274,11cyclictest4804-21perf22:35:010
1107499272,9cyclictest24469-21kworker/0:223:53:020
1107499272,13cyclictest20539-21munin-run19:30:020
1107499271,13cyclictest12023-21munin-run00:05:020
1108699261,13cyclictest0-21swapper/223:05:112
1108699261,12cyclictest827-21runrttasks21:54:382
1108699261,11cyclictest0-21swapper/200:34:152
11082992619,4cyclictest21575-21/usr/sbin/munin19:30:151
1107499268,15cyclictest14226-21munin-run00:10:020
1107499262,12cyclictest15169-21munin-run21:45:020
1107499260,12cyclictest27143-21perf19:45:010
196692250,1sleep3401ktimersoftd/323:05:253
1109299256,10cyclictest0-21swapper/322:47:383
1108699251,16cyclictest16683-21hddtemp_smartct23:00:152
1108699251,15cyclictest0-21swapper/219:55:342
1107499252,10cyclictest24469-21kworker/0:223:24:160
1107499251,9cyclictest24469-21kworker/0:223:43:170
11074992511,11cyclictest30569-21basename22:20:010
1107499250,12cyclictest22745-21awk19:35:010
1109299240,9cyclictest0-21swapper/321:56:093
1108699241,9cyclictest0-21swapper/222:23:162
1108699241,9cyclictest0-21swapper/222:23:162
1108699241,9cyclictest0-21swapper/220:35:032
1108699241,15cyclictest0-21swapper/220:59:072
1108699241,11cyclictest0-21swapper/223:19:052
1107499248,3cyclictest2786-21kworker/0:020:10:130
1107499247,15cyclictest19567-21munin-run21:55:020
1107499241,12cyclictest13267-21basename19:15:010
11074992410,13cyclictest0-21swapper/020:25:010
1109299231,9cyclictest0-21swapper/321:12:093
1109299231,9cyclictest0-21swapper/321:12:093
1109299231,10cyclictest0-21swapper/322:36:403
1109299230,10cyclictest0-21swapper/322:19:293
11086992313,3cyclictest0-21swapper/222:51:252
1108699231,15cyclictest0-21swapper/219:18:472
1108699231,14cyclictest0-21swapper/219:34:192
1108699231,12cyclictest0-21swapper/222:17:022
1108699231,12cyclictest0-21swapper/219:48:362
1108699231,10cyclictest0-21swapper/221:08:372
11086992310,3cyclictest0-21swapper/220:31:332
1107499235,10cyclictest0-21swapper/023:30:240
1107499235,10cyclictest0-21swapper/023:30:240
1109299220,9cyclictest0-21swapper/321:30:113
1108699221,12cyclictest0-21swapper/220:40:272
1108699220,13cyclictest10868-21fschecks_time20:20:152
1107499226,14cyclictest20827-21perf00:25:010
1107499225,5cyclictest827-21runrttasks22:50:140
1107499221,11cyclictest808-21snmpd21:25:140
1107499221,11cyclictest18357-21perf19:25:010
11074992210,3cyclictest1687-21ntp_states22:25:200
79322210,2sleep30-21swapper/322:40:163
1109299213,14cyclictest11762-21grep22:50:113
1109299211,6cyclictest0-21swapper/321:42:443
1109299211,10cyclictest0-21swapper/300:34:143
1109299210,9cyclictest0-21swapper/322:10:483
1108699218,3cyclictest0-21swapper/221:58:002
1108699212,14cyclictest763-21dbus-daemon00:14:232
1108699211,5cyclictest0-21swapper/222:46:022
1108699211,5cyclictest0-21swapper/200:25:342
1108699211,12cyclictest0-21swapper/220:07:262
1108699211,12cyclictest0-21swapper/219:37:032
1108699210,15cyclictest0-21swapper/200:06:362
1107499215,9cyclictest0-21swapper/021:21:400
1107499215,8cyclictest25933-21kworker/0:022:10:190
1107499215,3cyclictest17787-21kworker/0:123:10:170
1107499214,4cyclictest18641-21kworker/0:020:48:560
1107499213,8cyclictest827-21runrttasks00:33:460
1107499212,7cyclictest22955-21kworker/0:219:35:210
1107499212,11cyclictest10551-21kworker/0:222:00:210
1109299209,2cyclictest0-21swapper/320:05:183
1109299201,7cyclictest0-21swapper/323:57:413
1109299201,7cyclictest0-21swapper/323:42:033
1109299201,11cyclictest31388-21/usr/sbin/munin23:35:163
1109299201,10cyclictest352-21ntp_states19:55:223
1109299201,10cyclictest21400-21needreboot23:10:193
1108699205,3cyclictest0-21swapper/200:35:092
1108699203,3cyclictest0-21swapper/223:26:382
1108699201,5cyclictest0-21swapper/222:56:212
1108699201,15cyclictest0-21swapper/220:11:032
1108699201,13cyclictest0-21swapper/200:20:142
1108699201,12cyclictest0-21swapper/223:14:202
1108699201,12cyclictest0-21swapper/219:12:212
1107499207,2cyclictest10551-21kworker/0:221:35:410
1107499206,6cyclictest0-21swapper/021:46:110
1107499204,3cyclictest20449-21kworker/0:200:28:330
1107499203,8cyclictest0-21swapper/023:35:020
1107499203,4cyclictest31458-21kworker/0:219:56:130
1107499203,15cyclictest0-21swapper/021:10:010
1107499202,3cyclictest18641-21kworker/0:021:10:130
1107499202,3cyclictest18641-21kworker/0:021:10:130
1107499202,10cyclictest9895-21kworker/0:220:18:230
1107499201,9cyclictest25338-21kworker/0:220:56:440
1107499201,10cyclictest8961-21kworker/0:022:56:290
1109299196,11cyclictest87350irq/126-enp1s0-19:35:163
1109299194,8cyclictest0-21swapper/323:54:513
1109299192,7cyclictest18245-1kworker/u9:223:49:023
1109299191,9cyclictest12399-21chrt21:35:243
1109299191,6cyclictest2070-21sed22:25:243
11092991914,3cyclictest30590-21grep21:05:153
11092991912,4cyclictest3239-21fschecks_count22:30:133
1109299191,10cyclictest5987-21/usr/sbin/munin20:10:243
1108699197,2cyclictest0-21swapper/221:01:092
1108699191,5cyclictest0-21swapper/222:41:252
1108699191,15cyclictest0-21swapper/223:24:472
1108699191,14cyclictest0-21swapper/221:32:152
1107499198,3cyclictest27965-21/usr/sbin/munin21:00:210
1107499198,3cyclictest24469-21kworker/0:223:25:100
1107499198,3cyclictest0-21swapper/000:37:070
1107499194,3cyclictest11926-21kworker/0:200:11:290
1107499192,5cyclictest8506-21kworker/u8:122:20:210
1107499192,5cyclictest8506-21kworker/u8:122:20:210
1107499191,4cyclictest0-21swapper/023:03:170
11074991913,3cyclictest25538-21ls20:55:000
1107499191,12cyclictest0-21swapper/020:00:220
1107499191,12cyclictest0-21swapper/020:00:220
1107499191,10cyclictest8760-21sed23:55:190
1107499191,10cyclictest3427-21/usr/sbin/munin23:45:140
215612181,3sleep00-21swapper/021:57:370
172192181,2sleep30-21swapper/321:48:173
166392180,2sleep00-21swapper/000:15:070
1109299188,3cyclictest25249-21ntp_offset22:05:203
1109299188,2cyclictest0-21swapper/319:48:173
1109299183,9cyclictest0-21swapper/320:02:413
1109299183,9cyclictest0-21swapper/320:02:403
1109299181,9cyclictest808-21snmpd20:50:073
1109299181,6cyclictest23418-21irqstats23:15:173
1109299181,6cyclictest14649-21kernelversion22:55:173
1109299181,3cyclictest25872-21fschecks_count00:35:133
11092991811,4cyclictest8122-21tail21:25:263
1109299181,10cyclictest0-21swapper/320:45:233
1109299180,12cyclictest0-21swapper/323:25:213
1108699187,3cyclictest0-21swapper/220:15:172
1108699184,11cyclictest311ktimersoftd/223:56:232
1108699181,2cyclictest0-21swapper/219:21:312
1108699181,14cyclictest0-21swapper/222:31:552
1108699181,14cyclictest0-21swapper/221:49:302
1108699181,13cyclictest0-21swapper/220:28:392
1107499184,5cyclictest0-21swapper/021:30:040
1107499183,2cyclictest18641-21kworker/0:020:44:440
1107499181,15cyclictest19874-21taskset23:06:110
1107499181,10cyclictest9679-21cstates22:45:120
1107499181,10cyclictest9113-21sshd22:44:180
1107499180,7cyclictest14203-21kworker/0:320:29:420
1109299178,3cyclictest0-21swapper/320:25:173
1109299177,7cyclictest26151-21needreboot19:40:213
1109299177,3cyclictest87350irq/126-enp1s0-23:23:323
1109299174,6cyclictest19177-21/usr/sbin/munin20:40:173
1109299172,12cyclictest839-21lldpd19:50:183
1109299171,6cyclictest15026-21perl00:10:153
1109299171,6cyclictest0-21swapper/322:00:093
1109299171,6cyclictest0-21swapper/320:35:183
1109299171,6cyclictest0-21swapper/300:16:593
1109299171,4cyclictest808-21snmpd20:30:023
1109299171,4cyclictest20763-21/usr/sbin/munin19:30:223
11092991712,3cyclictest8184-21acpi20:15:093
11092991712,3cyclictest13422-21processes00:05:263
1109299171,14cyclictest0-21swapper/321:50:103
11092991711,3cyclictest23003-21latency_hist00:30:013
11092991711,3cyclictest13392-21munin-node19:15:043
11092991711,3cyclictest10381-21/usr/sbin/munin20:20:173
1108699171,6cyclictest0-21swapper/223:44:472
1108699171,13cyclictest0-21swapper/219:26:192
1107499174,3cyclictest16436-21kworker/0:120:31:100
1107499172,2cyclictest87650irq/129-enp1s0-19:15:030
1107499172,2cyclictest87350irq/126-enp1s0-20:37:560
1107499171,15cyclictest0-21swapper/019:49:380
116582160,2sleep30-21swapper/300:00:453
1109299161,9cyclictest0-21swapper/319:26:223
1109299161,7cyclictest0-21swapper/323:02:143
1109299161,5cyclictest3420-21processes21:15:233
1109299161,5cyclictest0-21swapper/323:30:093
1109299161,5cyclictest0-21swapper/323:30:093
1109299161,2cyclictest32505-21kworker/3:021:20:163
11092991611,3cyclictest27965-21/usr/sbin/munin21:00:223
11092991611,3cyclictest11278-21/usr/sbin/munin19:10:173
11092991610,3cyclictest25765-21/usr/sbin/munin20:55:133
1109299160,14cyclictest0-21swapper/322:20:293
1109299160,14cyclictest0-21swapper/322:20:283
11074991610,3cyclictest30421-21cat19:50:190
1109299151,5cyclictest19888-21ntp_kernel_pll_00:20:213
1109299149,3cyclictest16674-21cat19:20:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional