You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-18 - 22:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Aug 18, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
173832236161,26sleep00-21swapper/007:08:380
172772222193,19sleep20-21swapper/207:07:162
173282204168,24sleep10-21swapper/107:07:541
173942203166,24sleep30-21swapper/307:08:463
1758799106103,2cyclictest0-21swapper/210:58:222
306002570,3sleep330603-21unixbench-2d11:15:263
140832550,1sleep00-21swapper/008:10:260
58032540,2sleep00-21swapper/011:35:130
95352530,2sleep10-21swapper/109:15:201
132192520,5sleep313222-21cpuspeed_turbos09:25:143
254972380,8sleep081ktimersoftd/012:20:120
17587993611,12cyclictest17855-21df_inode09:35:152
17587993610,8cyclictest11214-21smart_sda11:45:242
17576993611,12cyclictest9730-21munin-run11:45:010
17587993510,16cyclictest26078-21munin-run09:55:012
1757699357,13cyclictest15713-21awk10:45:020
17587993410,9cyclictest87550irq/128-enp1s0-11:50:122
1757699346,13cyclictest4724-21awk10:20:010
17587993319,5cyclictest17598-21/usr/sbin/munin12:00:202
17587993311,10cyclictest6298-21perf09:10:012
17587993310,9cyclictest11256-21df_inode09:20:142
1757699336,12cyclictest20722-21idleruntime-cro12:10:010
1757699331,24cyclictest22571-21kworker/0:312:15:480
1758799329,14cyclictest23155-21/usr/sbin/munin12:15:212
1758799328,14cyclictest30071-21missed_timers11:15:202
1758799327,16cyclictest425-21sed10:10:012
1758799324,18cyclictest579-21systemd-journal08:15:012
1758799319,14cyclictest26661-21wc12:20:242
1758799317,16cyclictest1150-21users12:35:282
1758799317,10cyclictest2111-21/usr/sbin/munin09:00:202
17587993111,11cyclictest19105-21sed07:10:232
17587993110,7cyclictest9932-21grep10:30:172
1758799310,14cyclictest0-21swapper/210:40:422
1757699311,18cyclictest13763-21kworker/0:011:51:080
1758799309,5cyclictest808-21snmpd08:00:412
1758799309,12cyclictest20383-21hddtemp_smartct09:40:162
1758799307,6cyclictest2513-21latency_hist10:15:012
1758799307,6cyclictest2513-21latency_hist10:15:012
1758799306,16cyclictest7629-21latency_hist11:40:022
17587993011,7cyclictest18874-21awk08:25:022
17587993011,10cyclictest766-21ntpd07:46:142
1758199303,15cyclictest31982-40dnf08:54:251
1757699302,13cyclictest6300-21munin-run09:10:000
1758799299,6cyclictest25170-21cpuspeed_turbos07:25:132
1758799298,9cyclictest0-21swapper/211:25:572
1758799295,13cyclictest0-21swapper/209:33:442
1758799294,14cyclictest12438-21cat10:35:192
17587992919,6cyclictest5547-40dnf07:53:412
17587992919,6cyclictest5547-40dnf07:53:412
17587992912,5cyclictest16309-21sshd08:15:412
17587992911,10cyclictest8865-21grep11:40:222
17587992910,6cyclictest0-21swapper/210:20:022
1757699294,12cyclictest5667-21munin-run07:55:010
1757699294,12cyclictest5667-21munin-run07:55:010
17576992910,16cyclictest22317-21idleruntime-cro11:00:000
1758799288,12cyclictest15069-21taskset09:30:012
17587992820,5cyclictest6200-21smart_sda10:20:252
17587992819,5cyclictest21633-21diskstats08:30:142
17587992812,7cyclictest26644-21df_inode09:55:142
17587992811,14cyclictest1015-21grep11:25:012
17587992811,14cyclictest1015-21grep11:25:012
17581992813,13cyclictest550-21kworker/u8:208:00:241
1757699281,22cyclictest22173-21kworker/0:111:08:510
1757699281,19cyclictest15675-21kworker/0:010:48:000
1758799279,7cyclictest22299-1kworker/2:0H12:34:562
1758799279,7cyclictest22299-1kworker/2:0H12:34:562
1758799277,8cyclictest19185-21ntpq10:50:202
1758799275,10cyclictest0-21swapper/212:10:002
1758799273,14cyclictest0-21swapper/210:00:242
17587992719,5cyclictest21741-21ntp_states07:15:222
17587992718,6cyclictest4620-21smart_sda11:30:252
17587992716,7cyclictest159722sleep211:55:422
1758199271,17cyclictest3810-21ntp_states10:15:201
1757699272,13cyclictest27038-21munin-run07:30:010
1757699272,12cyclictest6927-21idleruntime-cro10:25:010
1757699272,12cyclictest31666-21kworker/0:212:35:590
1757699272,10cyclictest15675-21kworker/0:010:50:110
1757699271,11cyclictest12831-21kworker/0:209:33:240
1757699271,10cyclictest10052-21kworker/0:208:05:200
1758799269,6cyclictest8721-21/usr/sbin/munin09:15:142
1758799264,11cyclictest0-21swapper/209:10:132
17587992616,7cyclictest23948-21/usr/sbin/munin08:35:132
17587992614,8cyclictest808-21snmpd12:25:262
1757699262,12cyclictest17243-21kworker/0:009:40:000
1757699262,11cyclictest18634-21kworker/0:008:23:420
1757699262,10cyclictest28911-21kworker/0:207:35:200
1757699261,18cyclictest25784-21kworker/0:309:52:480
17576992611,2cyclictest31200-21kworker/0:107:40:200
1757699261,12cyclictest10696-21munin-run09:20:000
1757699261,10cyclictest658-21kworker/0:111:25:160
1759399251,4cyclictest21809-21sed10:55:273
1758799256,16cyclictest24244-21ls07:20:242
1758799255,8cyclictest23637-21taskset09:47:082
17587992518,4cyclictest634-21irqstats08:55:192
17587992518,4cyclictest28176-21/usr/sbin/munin11:10:242
17587992518,4cyclictest16275-21diskstats10:45:152
17587992517,5cyclictest30072-21/usr/sbin/munin08:50:222
17587992516,6cyclictest25534-21/usr/sbin/munin11:05:172
17587992516,6cyclictest21305-21diskstats12:10:152
17587992516,6cyclictest20682-21/usr/sbin/munin08:25:292
17587992516,6cyclictest10713-21/usr/sbin/munin08:05:142
17587992514,7cyclictest7595-21taskset07:56:452
17587992514,6cyclictest909-21sssd_nss08:45:152
1758199254,12cyclictest0-21swapper/110:20:241
1758199252,19cyclictest6230-21df07:55:151
1757699258,14cyclictest31727-21awk12:35:010
1757699258,14cyclictest31727-21awk12:35:010
1757699251,11cyclictest28115-21kworker/0:110:00:120
17587992416,5cyclictest25680-21/usr/sbin/munin08:40:222
17587992415,6cyclictest28044-21/usr/sbin/munin07:30:182
1758199242,11cyclictest4098-21munin-run09:05:011
1758199241,6cyclictest0-21swapper/112:35:271
1758199241,6cyclictest0-21swapper/110:09:141
1758199241,3cyclictest28357-21irqstats12:25:181
1757699242,12cyclictest799-21kworker/0:311:46:360
1757699242,10cyclictest13763-21kworker/0:012:03:400
1757699241,13cyclictest22571-21kworker/0:312:13:120
1757699241,10cyclictest26923-21kworker/0:211:11:110
1757699241,10cyclictest12831-21kworker/0:209:29:030
1759399235,6cyclictest0-21swapper/311:10:313
17593992313,7cyclictest28187-21systemd-journal09:59:053
17587992316,4cyclictest29793-21df07:35:142
17587992316,4cyclictest22872-21df_inode11:00:142
17587992315,6cyclictest7139-21/usr/sbin/munin10:25:142
17587992315,5cyclictest32719-21ntp_states07:40:202
1758199239,8cyclictest0-21swapper/111:18:421
1758199238,3cyclictest20483-21if_err_enp1s009:40:171
1758199232,12cyclictest22869-21df_inode11:00:141
1758199231,16cyclictest0-21swapper/112:13:371
1758199231,16cyclictest0-21swapper/111:52:241
1758199231,16cyclictest0-21swapper/110:28:061
1758199231,11cyclictest0-21swapper/111:37:341
1758199231,11cyclictest0-21swapper/111:11:431
1758199231,11cyclictest0-21swapper/109:32:101
1757699232,12cyclictest28115-21kworker/0:110:07:520
17576992311,3cyclictest799-21kworker/0:311:30:520
1757699231,10cyclictest26923-21kworker/0:211:24:040
1757699231,10cyclictest26923-21kworker/0:211:24:040
23482220,8sleep123-21ksoftirqd/107:45:191
1759399229,5cyclictest0-21swapper/310:20:173
1759399221,12cyclictest12149-21/usr/sbin/munin11:50:153
1759399221,11cyclictest6300-21munin-run09:10:013
1759399221,10cyclictest0-21swapper/308:40:523
1758199228,4cyclictest0-21swapper/109:58:211
1758199227,7cyclictest0-21swapper/109:45:111
1758199223,10cyclictest6289-21diskmemload12:15:171
1758199222,11cyclictest763-21dbus-daemon09:50:201
1758199221,5cyclictest0-21swapper/112:07:181
17581992214,2cyclictest0-21swapper/111:55:261
1758199221,3cyclictest1607-21ntp_states10:10:221
1758199221,3cyclictest1607-21ntp_states10:10:221
1758199221,2cyclictest23267-21perf08:35:021
17581992211,3cyclictest11360-21ntp_states08:05:211
1758199221,12cyclictest0-21swapper/112:33:061
1758199221,12cyclictest0-21swapper/112:33:061
1758199221,11cyclictest0-21swapper/110:53:011
1757699227,8cyclictest17767-21kworker/0:207:11:560
1757699227,3cyclictest26774-21kworker/0:007:34:280
1757699222,9cyclictest8402-21kworker/0:009:23:190
1757699222,9cyclictest28911-21kworker/0:207:46:160
1757699222,11cyclictest15987-21kworker/0:111:56:480
1757699222,10cyclictest8894-21kworker/0:110:38:440
1757699221,9cyclictest19197-21kworker/0:211:00:270
17576992211,3cyclictest8506-21awk09:15:000
17576992211,2cyclictest25784-21kworker/0:309:59:080
1757699221,12cyclictest23521-21kworker/0:209:49:240
1757699221,10cyclictest26923-21kworker/0:211:15:550
84462211,16sleep18560-21interrupts11:40:161
23142210,4sleep121-21rcuc/111:25:221
1759399216,2cyclictest87350irq/126-enp1s0-10:37:433
17593992111,7cyclictest4151-21if_err_enp2s011:30:183
1759399211,12cyclictest0-21swapper/310:53:273
1759399211,11cyclictest6368-21irqstats11:35:183
1759399211,11cyclictest16548-21/usr/sbin/munin12:00:143
1758199212,11cyclictest17815-21smart_sda12:00:241
17581992113,3cyclictest0-21swapper/110:34:381
1758199211,17cyclictest4577-21cstates09:05:141
1758199211,15cyclictest0-21swapper/110:01:491
1758199211,14cyclictest0-21swapper/111:05:121
1758199211,12cyclictest0-21swapper/111:30:161
1758199211,11cyclictest17228-21df_inode08:20:151
1758199211,11cyclictest0-21swapper/109:25:241
1758199210,6cyclictest0-21swapper/109:35:201
1758199210,16cyclictest0-21swapper/112:20:251
1757699217,3cyclictest14239-21kworker/0:308:15:240
1757699217,2cyclictest24573-21kworker/0:307:24:070
1757699216,3cyclictest1565-21kworker/0:209:01:190
1757699212,8cyclictest20970-21kworker/0:208:37:160
1757699211,8cyclictest1112-21kworker/0:307:56:200
1757699211,11cyclictest8894-21kworker/0:110:31:520
1759399209,3cyclictest6289-21diskmemload10:47:333
1759399203,2cyclictest0-21swapper/311:07:253
1759399201,11cyclictest17760-21cstates09:35:143
1759399201,10cyclictest30734-21head12:30:203
1759399201,10cyclictest30734-21head12:30:193
1759399201,10cyclictest0-21swapper/308:45:133
1759399200,8cyclictest0-21swapper/309:10:033
1759399200,11cyclictest0-21swapper/307:55:243
1759399200,10cyclictest15213-21unixbench-2d10:40:283
1758199209,3cyclictest14770-21ntp_kernel_pll_10:40:221
1758199202,14cyclictest87550irq/128-enp1s0-11:23:361
1758199202,14cyclictest87550irq/128-enp1s0-11:23:351
1758199201,8cyclictest0-21swapper/111:45:031
1758199201,18cyclictest0-21swapper/110:45:221
1758199201,18cyclictest0-21swapper/108:59:181
1758199201,17cyclictest0-21swapper/107:28:591
1758199201,11cyclictest19829-21if_enp1s008:25:191
1758199201,11cyclictest0-21swapper/110:55:141
1758199201,11cyclictest0-21swapper/108:40:161
1758199200,18cyclictest0-21swapper/107:36:181
1757699206,8cyclictest25419-21kworker/0:008:52:150
1757699202,10cyclictest2329-21kworker/0:310:25:080
1757699201,7cyclictest1-21systemd09:40:210
1757699201,11cyclictest19091-21/usr/sbin/munin08:25:290
1757699201,10cyclictest30393-21kworker/0:010:12:000
1757699201,10cyclictest30393-21kworker/0:010:12:000
36782190,2sleep30-21swapper/307:50:033
36782190,2sleep30-21swapper/307:50:033
1759399198,3cyclictest21903-21/usr/sbin/munin09:45:143
1759399198,2cyclictest24145-21grep12:15:193
1759399194,3cyclictest0-21swapper/309:24:093
1759399193,4cyclictest18988-21cstates12:05:133
1759399193,3cyclictest0-21swapper/309:31:283
1759399191,8cyclictest0-21swapper/309:15:263
1759399191,7cyclictest0-21swapper/310:12:153
1759399191,7cyclictest0-21swapper/310:12:153
1759399191,6cyclictest22695-21/usr/sbin/munin11:00:133
17593991913,3cyclictest6289-21diskmemload10:25:133
1759399191,11cyclictest87350irq/126-enp1s0-11:25:183
1759399191,11cyclictest0-21swapper/312:20:083
1759399191,10cyclictest6289-21diskmemload11:42:313
1759399191,10cyclictest28500-21/usr/sbin/munin10:00:193
1759399191,10cyclictest2741-21/usr/sbin/munin10:15:193
1759399191,10cyclictest24755-21gltestperf09:50:143
1759399191,10cyclictest0-21swapper/308:22:133
17593991910,2cyclictest0-21swapper/307:40:263
1759399190,5cyclictest0-21swapper/311:23:253
1759399190,5cyclictest0-21swapper/311:23:253
1759399190,5cyclictest0-21swapper/310:07:573
1759399190,11cyclictest0-21swapper/312:10:223
1758199198,3cyclictest23307-21tune2fs07:20:141
1758199191,9cyclictest20813-21tr07:15:121
17581991915,3cyclictest0-21swapper/108:48:591
1758199191,3cyclictest0-21swapper/108:19:211
17581991912,4cyclictest6518-21/usr/sbin/munin09:10:261
1758199191,13cyclictest0-21swapper/107:30:231
1758199191,10cyclictest12300-21if_enp2s010:35:181
1757699191,15cyclictest26282-21grep08:40:160
1757699191,10cyclictest32273-21/usr/sbin/munin08:55:260
1757699190,9cyclictest18634-21kworker/0:008:32:000
1759399189,3cyclictest0-21swapper/308:15:133
1759399186,3cyclictest3103-21head09:00:193
1759399182,7cyclictest10076-21if_enp1s010:30:173
1759399182,5cyclictest0-21swapper/312:28:233
1759399181,9cyclictest9953-21/usr/sbin/munin11:45:153
17593991811,4cyclictest30692-21fgrep07:35:233
1759399181,10cyclictest0-21swapper/311:56:013
1759399180,10cyclictest1474-21/usr/sbin/munin07:45:263
1758199181,16cyclictest0-21swapper/109:21:581
1758199181,16cyclictest0-21swapper/108:35:321
1758199181,16cyclictest0-21swapper/108:11:341
1758199181,16cyclictest0-21swapper/107:12:111
1758199180,16cyclictest0-21swapper/107:50:031
1758199180,16cyclictest0-21swapper/107:50:031
1758199180,16cyclictest0-21swapper/107:40:131
1757699186,2cyclictest17767-21kworker/0:207:16:080
1757699185,2cyclictest27249-21kworker/0:112:25:480
1757699181,7cyclictest7826-21kworker/0:108:03:440
12692180,2sleep30-21swapper/312:36:503
1759399176,3cyclictest41-21ksoftirqd/308:05:163
1759399174,4cyclictest30320-21cat08:50:133
1759399171,6cyclictest0-21swapper/308:30:113
1759399171,2cyclictest19294-21sed07:10:263
17593991712,3cyclictest23633-21if_err_enp1s007:20:163
17593991712,3cyclictest19891-21munin-node08:25:133
1759399171,13cyclictest8087-21/usr/sbin/munin08:00:173
1759399171,13cyclictest28820-21unixbench_singl07:30:273
1759399171,13cyclictest26539-21unixbench-2d07:25:273
1759399171,10cyclictest0-21swapper/309:41:223
1759399161,14cyclictest0-21swapper/308:57:143
17593991611,3cyclictest24423-21missed_timers08:35:163
1757699161,2cyclictest0-21swapper/008:47:310
1759399159,3cyclictest22211-21unixbench_singl07:15:263
1759399154,3cyclictest12050irq/123-ahci[0008:10:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional