You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 08:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri Jan 02, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
92092235164,58sleep20-21swapper/219:09:022
90882202166,24sleep10-21swapper/119:07:281
91122201164,24sleep30-21swapper/319:07:463
90702200163,25sleep00-21swapper/019:07:150
9379997269,2cyclictest0-21swapper/221:29:002
9379997269,2cyclictest0-21swapper/221:28:592
9379995753,3cyclictest0-21swapper/200:36:362
256142570,2sleep1937499cyclictest20:55:271
268202560,2sleep30-21swapper/321:00:183
213122550,6sleep121-21rcuc/119:35:131
92612520,4sleep0936999cyclictest00:00:230
195402510,1sleep00-21swapper/019:30:140
937499468,31cyclictest2240-21pmu-power22:30:251
937499452,31cyclictest775-21dbus-daemon00:23:391
937499411,31cyclictest23345-21latency22:05:221
937499411,25cyclictest2471-21turbostat.cron21:20:011
937499406,15cyclictest17869-21munin-run21:55:011
937499401,29cyclictest20051-21latency_hist22:00:021
937499401,27cyclictest23084-21sa100:30:281
938499391,27cyclictest23983-21unixbench_multi22:05:303
937499395,5cyclictest751-21rngd21:05:451
937499392,27cyclictest589-21awk20:00:171
937499391,26cyclictest0-21swapper/122:50:151
937499391,25cyclictest18484-21latency_hist23:10:011
938499383,6cyclictest0-21swapper/322:40:413
937499387,21cyclictest577-21systemd-journal21:40:011
937499381,23cyclictest27264-21latency_hist23:30:011
938499373,6cyclictest23178-40dnf19:39:443
9384993712,3cyclictest5870-21users20:10:273
937499376,23cyclictest0-21swapper/121:30:011
937499376,23cyclictest0-21swapper/121:30:011
937499376,22cyclictest15662-21latency_hist21:50:011
937499376,19cyclictest16625-21latency_hist19:25:011
937499372,19cyclictest0-21swapper/123:15:391
937499371,32cyclictest28239-21latency_hist21:05:011
937499371,27cyclictest11865-21needreboot20:25:231
937499371,24cyclictest87150irq/128-enp1s0-20:05:181
937499361,22cyclictest31663-21latency_hist23:40:011
9379993514,3cyclictest28676-21kworker/2:122:58:262
937499351,31cyclictest22313-21cat22:05:011
937499351,27cyclictest801-21snmpd23:44:391
937499351,25cyclictest0-21swapper/120:50:251
937499351,21cyclictest25876-21expr19:45:141
9369993511,11cyclictest32653-21awk21:15:000
938499345,4cyclictest12298-21cstates22:55:133
938499341,4cyclictest1548-21ntp_states21:15:213
937499343,22cyclictest87150irq/128-enp1s0-23:45:261
937499343,22cyclictest87150irq/128-enp1s0-23:45:261
937499343,21cyclictest87150irq/128-enp1s0-23:55:011
937499342,21cyclictest9599-21smart_sda00:00:271
937499341,5cyclictest0-21swapper/122:42:171
937499341,22cyclictest0-21swapper/100:05:151
937499341,18cyclictest13395-21cstates20:30:171
937499341,18cyclictest0-21swapper/122:45:161
937499340,24cyclictest0-21swapper/120:15:171
937499340,24cyclictest0-21swapper/120:15:161
937499340,1cyclictest0-21swapper/121:40:281
938499337,4cyclictest0-21swapper/323:10:263
938499332,5cyclictest30614-21memory23:35:203
9384993310,3cyclictest1934-21irqstats22:30:193
937999331,13cyclictest28676-21kworker/2:120:00:432
9379993310,4cyclictest28676-21kworker/2:120:48:202
937499332,14cyclictest0-21swapper/122:59:111
937499331,29cyclictest6204-21tr23:55:151
937499331,22cyclictest0-21swapper/100:15:261
937499330,19cyclictest23743-21/usr/sbin/munin00:35:141
936999337,12cyclictest0-21swapper/000:00:020
936999335,13cyclictest6882-21awk21:30:010
936999335,13cyclictest6882-21awk21:30:010
938499329,3cyclictest31296-21sensors_temp19:55:243
938499329,2cyclictest835-21runrttasks23:19:493
938499324,7cyclictest32259-21users21:10:283
938499324,7cyclictest12568-21tr19:15:113
938499322,9cyclictest10116-21irqstats21:35:213
938499321,9cyclictest9918-21/usr/sbin/munin22:50:263
9384993212,3cyclictest28913-21date22:20:013
9384993211,3cyclictest24061-21/usr/sbin/munin20:55:193
9384993210,3cyclictest15569-21sensors_temp23:00:243
937999329,3cyclictest28676-21kworker/2:120:39:152
9379993222,3cyclictest28676-21kworker/2:119:55:192
937499322,26cyclictest16348-21ntp_states20:35:271
937499322,25cyclictest16855-21/usr/sbin/munin19:25:281
937499321,28cyclictest28377-21cat23:30:191
937499321,28cyclictest19499-21if_err_enp2s023:10:181
937499321,27cyclictest31475-21expr22:25:151
9374993212,18cyclictest13431-21hddtemp_smartct00:10:171
938499319,4cyclictest16554-21/usr/sbin/munin20:35:293
938499314,8cyclictest29130-21gltestperf21:05:163
938499311,9cyclictest904-21sssd_nss21:20:133
938499311,3cyclictest18216-21ntp_states00:20:233
9384993112,3cyclictest8326-21/usr/sbin/munin00:00:143
9384993110,3cyclictest19668-21/usr/sbin/munin20:45:243
9384993110,3cyclictest18500-21ntp_kernel_err20:40:213
938499310,23cyclictest0-21swapper/322:00:413
9379993120,3cyclictest28676-21kworker/2:123:00:102
937499311,4cyclictest27986-21grep19:50:131
937499311,4cyclictest0-21swapper/121:30:071
937499311,2cyclictest30670-21/usr/sbin/munin21:10:161
937499311,16cyclictest9254-21perl20:20:161
937499310,3cyclictest5356-21ntpq20:10:221
936999316,12cyclictest23521-21munin-run00:35:000
936999315,12cyclictest0-21swapper/020:40:010
938499309,4cyclictest27608-21latency_hist19:50:013
938499309,3cyclictest21862-21/usr/sbin/munin20:50:143
938499309,2cyclictest18088-21/usr/sbin/munin21:55:173
938499308,4cyclictest0-21swapper/321:30:173
938499308,3cyclictest15207-21tail21:45:283
938499308,3cyclictest12837-21/usr/sbin/munin21:40:273
938499304,6cyclictest751-21rngd21:50:143
938499304,3cyclictest7461-21kworker/u8:023:05:143
938499301,8cyclictest86950irq/126-enp1s0-23:24:193
9384993013,14cyclictest3715-21sleep23:50:013
9384993013,14cyclictest3715-21sleep23:50:013
9384993010,3cyclictest5267-21cat23:50:253
9384993010,3cyclictest16031-21open_files00:15:213
9384993010,3cyclictest15916-21sensors_temp19:20:243
938499300,3cyclictest0-21swapper/322:28:023
938499300,2cyclictest0-21swapper/323:30:183
938499300,2cyclictest0-21swapper/321:28:023
938499300,2cyclictest0-21swapper/321:28:023
9379993020,3cyclictest28676-21kworker/2:120:16:152
9379993020,3cyclictest28676-21kworker/2:120:16:152
9379993017,2cyclictest28676-21kworker/2:122:10:212
937499301,4cyclictest14432-21grep23:00:121
937499301,3cyclictest25161-21cat23:25:011
937499301,26cyclictest19530-21cstates00:25:131
937499301,25cyclictest0-21swapper/120:40:121
937499301,15cyclictest751-21rngd19:58:251
937499301,13cyclictest0-21swapper/119:15:101
9369993011,16cyclictest9697-21munin-run22:50:010
938499299,4cyclictest24686-21sendmail_mailst19:40:243
938499298,9cyclictest0-21swapper/319:31:443
938499298,2cyclictest86950irq/126-enp1s0-23:40:123
938499294,3cyclictest20161-21irqstats00:25:183
938499291,8cyclictest11575-21netstat00:05:203
938499291,5cyclictest8049-21df22:45:143
938499291,4cyclictest6132-21/usr/sbin/munin23:55:173
9384992910,3cyclictest26552-21mailstats23:25:273
938499290,3cyclictest0-21swapper/322:39:463
938499290,3cyclictest0-21swapper/322:39:463
9379992919,3cyclictest28676-21kworker/2:123:12:092
9379992919,2cyclictest28676-21kworker/2:100:27:582
9379992911,3cyclictest28676-21kworker/2:122:35:312
9379992911,3cyclictest28676-21kworker/2:122:35:312
937999290,16cyclictest28676-21kworker/2:100:04:412
937499291,26cyclictest11629-21sort19:15:011
937499291,1cyclictest0-21swapper/122:35:281
937499291,1cyclictest0-21swapper/122:35:281
936999292,10cyclictest28831-21kworker/0:022:32:510
936999291,13cyclictest26664-21munin-run22:15:010
938499289,4cyclictest2441-21entropy20:05:143
938499281,7cyclictest13125-21awk00:10:143
938499281,5cyclictest29995-21/usr/sbin/munin22:20:193
938499281,3cyclictest25002-21ls22:10:123
9384992811,15cyclictest25716-21awk00:40:003
9384992810,3cyclictest10811-21expr19:10:243
937999289,3cyclictest28676-21kworker/2:119:15:312
937999288,4cyclictest28676-21kworker/2:123:46:112
937999288,4cyclictest28676-21kworker/2:123:46:112
9379992823,2cyclictest28676-21kworker/2:121:06:102
9379992810,3cyclictest28676-21kworker/2:121:30:432
937499282,2cyclictest0-21swapper/120:45:121
937499281,3cyclictest27980-21/usr/sbin/munin22:15:231
937499281,3cyclictest0-21swapper/121:20:031
937499281,19cyclictest0-21swapper/122:20:251
937499281,15cyclictest0-21swapper/119:30:051
936999289,16cyclictest16645-21awk19:25:010
936999281,11cyclictest19165-21kworker/0:220:46:030
938499279,3cyclictest11102-21expr20:25:153
938499278,3cyclictest22088-21cut00:30:163
938499271,5cyclictest0-21swapper/319:50:283
937999279,3cyclictest28676-21kworker/2:120:30:032
9379992714,6cyclictest28676-21kworker/2:123:20:132
9379992714,3cyclictest28676-21kworker/2:123:55:172
937999271,20cyclictest28676-21kworker/2:100:24:092
9379992711,4cyclictest28676-21kworker/2:119:39:312
937999271,14cyclictest28676-21kworker/2:100:10:232
9379992710,3cyclictest28676-21kworker/2:120:56:422
9379992710,3cyclictest28676-21kworker/2:120:28:032
937499271,3cyclictest25391-21/usr/sbin/munin22:10:161
936999272,12cyclictest19135-21awk00:25:010
936999271,10cyclictest15433-21kworker/0:321:48:110
936999270,12cyclictest12866-21awk20:30:010
938499262,4cyclictest14145-21ntp_states20:30:253
938499260,3cyclictest0-21swapper/320:15:403
938499260,3cyclictest0-21swapper/320:15:403
937999269,4cyclictest28676-21kworker/2:120:23:512
937999263,15cyclictest28676-21kworker/2:123:40:252
9379992615,3cyclictest28676-21kworker/2:123:27:492
937999261,13cyclictest28676-21kworker/2:121:40:132
937999261,10cyclictest28676-21kworker/2:120:52:282
9379992610,3cyclictest28676-21kworker/2:119:42:112
937499261,4cyclictest23743-21cstates19:40:121
936999262,9cyclictest3420-21kworker/0:023:50:280
936999262,11cyclictest11217-21kworker/0:119:11:320
936999261,14cyclictest12050irq/123-ahci[0019:20:010
936999261,13cyclictest20514-21kworker/0:219:40:510
936999261,11cyclictest1176-21kworker/0:123:44:200
938499251,6cyclictest0-21swapper/320:00:153
937999259,3cyclictest28676-21kworker/2:119:23:402
9379992511,2cyclictest28676-21kworker/2:119:45:552
937999251,12cyclictest28676-21kworker/2:119:10:132
9379992510,3cyclictest28676-21kworker/2:123:05:392
9379992510,3cyclictest28676-21kworker/2:121:38:592
936999252,9cyclictest31910-21kworker/0:220:00:200
936999252,10cyclictest28831-21kworker/0:022:29:390
936999252,10cyclictest20566-21kworker/0:223:32:440
936999251,9cyclictest29661-21kworker/0:019:57:560
936999251,9cyclictest1176-21kworker/0:123:45:080
936999251,9cyclictest1176-21kworker/0:123:45:080
936999251,12cyclictest20063-21munin-run22:00:020
9384992410,3cyclictest9491-21irqstats20:20:173
938499240,3cyclictest0-21swapper/319:29:283
937999249,3cyclictest28676-21kworker/2:122:28:592
937999249,3cyclictest28676-21kworker/2:120:06:592
937999246,3cyclictest28676-21kworker/2:123:36:312
9379992413,3cyclictest28676-21kworker/2:122:07:532
9379992413,3cyclictest28676-21kworker/2:120:11:532
937999241,15cyclictest28676-21kworker/2:120:40:172
9379992410,3cyclictest28676-21kworker/2:119:51:232
936999242,11cyclictest11859-21kworker/0:122:56:110
936999242,10cyclictest9051-21kworker/0:121:44:030
936999241,9cyclictest25033-21kworker/0:023:27:310
936999241,12cyclictest6478-21kworker/0:120:15:160
936999241,12cyclictest6478-21kworker/0:120:15:150
936999241,11cyclictest20665-21kworker/0:019:35:150
936999241,10cyclictest1802-21kworker/0:120:08:440
118612240,9sleep2311ktimersoftd/200:05:262
937999239,4cyclictest28676-21kworker/2:121:01:242
937999239,3cyclictest28676-21kworker/2:121:15:392
937999239,3cyclictest28676-21kworker/2:119:34:352
937999238,3cyclictest28676-21kworker/2:122:19:232
937999238,3cyclictest28676-21kworker/2:119:29:552
937999238,3cyclictest28676-21kworker/2:100:34:522
937999231,10cyclictest28676-21kworker/2:121:52:512
9379992310,3cyclictest28676-21kworker/2:122:40:512
936999232,9cyclictest2155-21kworker/0:121:23:550
936999232,7cyclictest20514-21kworker/0:219:25:390
936999230,11cyclictest5305-21perf22:40:010
936999230,11cyclictest5305-21perf22:40:010
937999229,3cyclictest28676-21kworker/2:100:18:592
937999227,4cyclictest28676-21kworker/2:122:01:392
937999227,3cyclictest28676-21kworker/2:121:56:272
9379992210,3cyclictest28676-21kworker/2:122:52:362
9379992210,2cyclictest28676-21kworker/2:122:30:592
936999228,2cyclictest14417-21kworker/0:100:13:270
936999222,8cyclictest28831-21kworker/0:022:25:000
936999222,12cyclictest6777-21kworker/0:221:37:470
936999222,10cyclictest28831-21kworker/0:022:40:030
936999221,11cyclictest2155-21kworker/0:121:17:230
937999218,3cyclictest28676-21kworker/2:123:51:032
937999218,3cyclictest28676-21kworker/2:121:20:012
937999216,3cyclictest28676-21kworker/2:123:30:532
937999216,3cyclictest28676-21kworker/2:121:13:312
9379992114,2cyclictest28676-21kworker/2:122:22:452
9379992110,3cyclictest28676-21kworker/2:123:16:592
936999219,11cyclictest0-21swapper/021:05:000
9369992115,3cyclictest19995-21gltestperf00:25:160
936999211,4cyclictest0-21swapper/022:19:550
936999211,12cyclictest25892-21kworker/0:121:08:110
936999210,5cyclictest30421-1kworker/u9:200:38:430
937999209,3cyclictest28676-21kworker/2:122:46:352
9379992015,3cyclictest14299-21grep21:45:172
936999201,9cyclictest4274-21kworker/0:220:14:280
936999201,12cyclictest0-21swapper/023:15:180
936999201,10cyclictest9572-21kworker/0:022:53:390
936999191,8cyclictest15329-21kworker/0:222:00:040
9369991913,4cyclictest30348-21sed23:35:180
936999191,11cyclictest16915-21kworker/0:020:40:270
936999191,10cyclictest20410-21sshd23:11:400
147092190,1sleep00-21swapper/023:00:140
936999189,2cyclictest25647-21/usr/sbin/munin19:45:240
936999189,2cyclictest18136-21sshd23:05:400
936999182,6cyclictest17641-21kworker/0:022:08:200
936999181,9cyclictest0-21swapper/019:53:410
936999181,14cyclictest24715-21cut20:55:160
177682180,3sleep00-21swapper/021:53:580
936999171,9cyclictest801-21snmpd20:50:270
936999171,8cyclictest0-21swapper/021:33:190
936999171,7cyclictest30421-1kworker/u9:200:15:220
936999171,7cyclictest24802-21sshd23:21:400
936999171,6cyclictest801-21snmpd20:20:210
150292171,3sleep00-21swapper/020:34:560
936999161,14cyclictest0-21swapper/000:07:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional