You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 17:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Wed Jan 07, 2026 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
225757421240,7sleep2331rcuc/207:05:122
23539272770,3sleep00-21swapper/011:35:130
197399750,14rtkit-daemon4497-21PK-Backend10:13:361
197399710,4rtkit-daemon4497-21PK-Backend09:22:050
197399710,4rtkit-daemon0-21swapper/011:20:200
225829227045,16sleep10-21swapper/107:05:241
22611372650,5sleep10-21swapper/107:15:031
225781026343,12sleep30-21swapper/307:05:133
23711072620,13chrt2371109-21cpuspeed_turbos12:25:143
225865326141,14sleep00-21swapper/007:06:020
197399600,8rtkit-daemon1972-21rtkit-daemon09:35:290
197399590,9rtkit-daemon1972-21rtkit-daemon10:23:190
197399580,9rtkit-daemon1972-21rtkit-daemon09:10:231
197399580,8rtkit-daemon1972-21rtkit-daemon07:19:150
197399570,8rtkit-daemon1972-21rtkit-daemon08:40:190
197399560,9rtkit-daemon1972-21rtkit-daemon09:06:240
225912299530,48cyclictest1972-21rtkit-daemon09:55:220
197399530,8rtkit-daemon1972-21rtkit-daemon09:56:521
197399530,8rtkit-daemon1972-21rtkit-daemon09:17:191
2259131995220,10cyclictest2321294-21systemd-udevd10:03:292
197399520,8rtkit-daemon1972-21rtkit-daemon11:31:051
2259127995010,19cyclictest26-21ksoftirqd/111:55:131
225912299500,46cyclictest1972-21rtkit-daemon10:49:080
197399500,6rtkit-daemon1972-21rtkit-daemon07:26:071
2259127994919,26cyclictest2286095-21/usr/sbin/munin08:25:271
23455742480,9sleep126-21ksoftirqd/111:10:301
23543442470,6sleep30-21swapper/311:35:243
225912299470,43cyclictest1972-21rtkit-daemon11:32:190
225912299470,43cyclictest1972-21rtkit-daemon10:06:130
2259131994522,10cyclictest34-21ksoftirqd/210:15:162
2259122994526,15cyclictest1-21systemd10:20:010
2259133994425,15cyclictest1054-21snmpd07:48:153
23399462430,5sleep2331rcuc/210:55:302
2259122994327,12cyclictest2321450-21cron10:05:000
2259133994229,9cyclictest42-21ksoftirqd/311:28:363
2259133994218,20cyclictest2306884-21sshd09:20:453
2259131994222,17cyclictest2347379-21sshd11:15:462
22903402410,17sleep234-21ksoftirqd/208:35:252
2259133994122,15cyclictest42-21ksoftirqd/311:55:133
2259133994122,13cyclictest42-21ksoftirqd/309:25:463
2259133994116,21cyclictest2294512-21latency_hist08:50:033
2259131994122,6cyclictest34-21ksoftirqd/207:45:012
2259127994124,12cyclictest438-20systemd-journal07:44:591
2259127994120,12cyclictest26-21ksoftirqd/107:55:161
225912299412,20cyclictest438-20systemd-journal10:55:000
225912299411,11cyclictest2351596-21turbostat11:29:590
2259133994023,13cyclictest42-21ksoftirqd/310:20:273
2259133994023,13cyclictest42-21ksoftirqd/307:37:593
2259133994023,13cyclictest42-21ksoftirqd/307:37:583
2259133994021,15cyclictest1054-21snmpd09:13:313
2259133994019,18cyclictest1054-21snmpd08:38:243
2259131994024,12cyclictest923-21dbus-daemon08:45:002
2259131994022,12cyclictest2308620-21systemd-udevd09:25:452
2259131994012,8cyclictest34-21ksoftirqd/211:23:072
2259127994021,12cyclictest26-21ksoftirqd/111:24:271
225912299402,11cyclictest2366118-21df_inode12:10:140
225912299402,11cyclictest2366118-21df_inode12:10:140
2259133993927,7cyclictest2276107-21grep07:55:223
2259133993922,5cyclictest42-21ksoftirqd/309:45:033
2259133993922,5cyclictest42-21ksoftirqd/309:45:023
2259133993920,14cyclictest42-21ksoftirqd/311:20:003
2259131993926,9cyclictest2372392-21cron12:30:002
2259131993923,12cyclictest1-21systemd07:50:012
2259131993913,9cyclictest34-21ksoftirqd/207:33:372
225912799395,5cyclictest141rcu_preempt08:00:221
225912799395,5cyclictest141rcu_preempt08:00:221
2259127993922,13cyclictest1-21systemd10:40:001
225912299395,7cyclictest2333588-21turbostat10:40:000
225912299392,32cyclictest2297144-21df_inode08:55:160
2259122993913,7cyclictest23437962cat11:05:300
2259133993826,8cyclictest2280346-21cron08:09:593
2259133993824,6cyclictest42-21ksoftirqd/309:48:353
2259133993820,14cyclictest42-21ksoftirqd/307:55:003
2259133993817,17cyclictest1054-21snmpd08:23:223
2259133993816,12cyclictest42-21ksoftirqd/312:05:133
2259131993821,12cyclictest2282080-21apt-get08:15:002
2259131993820,13cyclictest438-20systemd-journal07:39:592
2259131993820,13cyclictest438-20systemd-journal07:39:592
225913199381,33cyclictest2293418-21df_inode08:45:142
2259127993827,7cyclictest2323194-21cron10:10:011
225912799381,6cyclictest141rcu_preempt10:03:301
2259127993814,5cyclictest141rcu_preempt10:15:471
2259127993811,6cyclictest141rcu_preempt09:25:461
225912299383,12cyclictest13-21ksoftirqd/009:45:030
225912299383,12cyclictest13-21ksoftirqd/009:45:020
2259122993826,7cyclictest2359529-21ntp_states11:50:250
225912299382,15cyclictest2340494-21apt-get11:00:070
2259122993821,13cyclictest2347816-21munin-run11:20:010
2259122993821,12cyclictest2329380-21ntpq10:25:240
197399380,31rtkit-daemon1920-21kerneloops12:01:511
2259133993719,12cyclictest2291794-21awk08:40:203
2259133993718,15cyclictest1054-21snmpd07:13:083
2259133993718,15cyclictest1054-21snmpd07:13:083
2259133993716,13cyclictest42-21ksoftirqd/310:05:003
225913399371,21cyclictest141rcu_preempt11:40:463
225913399371,13cyclictest23455732cat11:10:313
2259133993710,23cyclictest2266442-21cron07:30:003
2259131993725,9cyclictest34-21ksoftirqd/211:28:362
2259131993724,9cyclictest2313407-21needreboot09:40:232
2259131993724,9cyclictest2313407-21needreboot09:40:232
2259131993722,11cyclictest2317767-21sshd09:52:452
2259131993722,10cyclictest2369741-21sort12:20:202
2259131993722,10cyclictest2361585-21sort11:55:302
225913199372,20cyclictest2300856-21memory09:05:222
2259131993721,13cyclictest2328923-21cstates10:25:162
2259131993721,12cyclictest2310233-21expr09:30:292
2259131993720,13cyclictest2338039-21ntpq10:50:242
225913199371,8cyclictest4497-21PK-Backend08:55:272
2259131993713,5cyclictest141rcu_preempt10:12:412
2259127993720,13cyclictest26-21ksoftirqd/110:45:001
2259127993720,13cyclictest2360136-21sshd11:53:451
2259127993719,13cyclictest26-21ksoftirqd/107:15:011
2259127993719,13cyclictest26-21ksoftirqd/107:15:011
2259127993719,13cyclictest2267471-21ntp_states07:30:271
225912299372,13cyclictest141rcu_preempt12:35:140
225912299372,13cyclictest141rcu_preempt12:35:140
2259122993721,12cyclictest2302974-21ntp_kernel_err09:10:240
2259122993720,13cyclictest2345-21gdbus07:39:570
2259122993720,13cyclictest2345-21gdbus07:39:560
225912299371,14cyclictest2358510-21dump-pmu-power11:49:590
23353062360,5sleep2331rcuc/210:45:002
23148192360,5sleep0161rcuc/009:45:180
2259133993626,6cyclictest42-21ksoftirqd/312:10:133
2259133993626,6cyclictest42-21ksoftirqd/312:10:123
2259133993622,11cyclictest42-21ksoftirqd/310:55:213
2259133993621,11cyclictest2324910-21cron10:15:003
2259133993620,5cyclictest42-21ksoftirqd/312:15:133
2259133993620,12cyclictest2289322-21/usr/sbin/munin08:35:003
2259133993619,5cyclictest42-21ksoftirqd/312:35:223
2259133993619,5cyclictest42-21ksoftirqd/312:35:223
2259133993619,12cyclictest1074-21lldpd09:50:153
2259133993610,22cyclictest0-21swapper/310:30:013
225913199362,6cyclictest2330786-21hddtemp_smartct10:30:192
2259131993621,11cyclictest976-21rs:main2
2259131993621,11cyclictest2285858-21ls08:25:002
225913199361,9cyclictest0-21swapper/208:50:052
2259131993619,12cyclictest2282783-21expr08:15:152
2259131993617,14cyclictest2326890-21seq10:20:032
225912799362,30cyclictest2301145-21python309:05:291
2259127993621,11cyclictest2374129-21cron12:35:001
225912799361,9cyclictest0-21swapper/110:49:321
225912799361,9cyclictest0-21swapper/107:23:271
2259127993619,13cyclictest2317367-21python309:50:261
225912799361,31cyclictest2315872-21taskset09:46:071
225912299367,9cyclictest141rcu_preempt08:25:140
225912299365,9cyclictest13-21ksoftirqd/011:10:260
225912299361,7cyclictest438-20systemd-journal09:33:450
225912299361,31cyclictest2221-21packagekitd10:33:290
197399360,29rtkit-daemon0-21swapper/112:38:591
197399360,29rtkit-daemon0-21swapper/112:38:591
2259133993524,7cyclictest2300019-21sleep09:05:003
2259133993523,8cyclictest2094-21JS10:49:093
2259133993523,7cyclictest42-21ksoftirqd/310:35:203
2259133993521,10cyclictest2305491-21cron09:20:003
2259133993520,11cyclictest42-21ksoftirqd/309:05:123
2259133993518,13cyclictest2357798-21ntp_states11:45:263
2259133993518,13cyclictest2270980-21ntp_states07:40:263
2259133993517,13cyclictest2373418-21ntp_states12:30:243
2259133993514,17cyclictest1054-21snmpd10:18:443
225913399351,29cyclictest2295189-21ls08:50:193
225913199356,10cyclictest0-21swapper/209:11:582
225913199354,5cyclictest141rcu_preempt10:05:232
225913199352,6cyclictest2337072-21munin-run10:50:012
2259131993519,12cyclictest2375520-21sshd12:35:462
2259131993519,12cyclictest2375520-21sshd12:35:462
2259131993519,12cyclictest2366804-21sed12:10:292
2259131993519,12cyclictest2366804-21sed12:10:282
2259131993519,12cyclictest2306764-21sed09:20:282
2259131993518,13cyclictest2298920-21diskstats09:00:172
2259131993512,6cyclictest141rcu_preempt07:25:022
2259127993524,7cyclictest2273865-21cpuspeed_turbos07:50:151
225912799351,9cyclictest4497-21PK-Backend09:00:341
2259127993519,12cyclictest26-21ksoftirqd/108:30:171
2259127993517,14cyclictest26-21ksoftirqd/109:45:031
2259127993517,14cyclictest26-21ksoftirqd/109:45:021
225912799351,13cyclictest4497-21PK-Backend12:25:061
225912299356,5cyclictest141rcu_preempt09:25:450
225912299352,11cyclictest2259888-21memory07:10:230
225912299352,11cyclictest2259888-21memory07:10:220
225912299352,10cyclictest2278635-21munin-run08:05:010
225912299352,10cyclictest2278635-21munin-run08:05:000
2259122993519,11cyclictest438-20systemd-journal07:45:000
225912299351,30cyclictest2294518-21turbostat08:50:010
225912299351,11cyclictest141rcu_preempt10:10:210
2259133993425,6cyclictest42-21ksoftirqd/310:40:563
2259133993423,6cyclictest42-21ksoftirqd/312:20:283
2259133993423,5cyclictest42-21ksoftirqd/311:55:003
2259133993422,7cyclictest2266883-21tr07:30:143
2259133993418,12cyclictest2282577-21mktemp08:15:033
2259133993417,13cyclictest2312482-21basename09:40:003
2259133993417,12cyclictest438-20systemd-journal10:49:563
2259133993417,12cyclictest2309847-21cat09:30:203
2259133993417,12cyclictest1074-21lldpd11:25:013
225913399341,15cyclictest0-21swapper/309:59:553
225913199349,8cyclictest141rcu_preempt11:30:222
2259131993419,11cyclictest2274294-21wc07:50:222
2259131993419,10cyclictest2287776-21date08:30:032
225913199341,8cyclictest0-21swapper/211:12:002
2259131993418,12cyclictest3692986-21snapd12:16:092
2259131993418,12cyclictest2279522-21cat08:05:212
2259131993417,13cyclictest2355900-21memory11:40:182
225913199341,29cyclictest2288442-21wc08:30:212
225912799346,11cyclictest26-21ksoftirqd/108:06:511
225912799344,5cyclictest141rcu_preempt08:25:001
2259127993423,7cyclictest2307219-21cron09:24:591
225912799341,9cyclictest4497-21PK-Backend10:30:381
225912799341,9cyclictest4497-21PK-Backend09:32:051
2259127993419,11cyclictest2269202-21ntp_states07:35:271
2259127993419,11cyclictest2269202-21ntp_states07:35:261
2259127993418,12cyclictest2327660-21awk10:20:241
2259127993417,12cyclictest2297525-21ntpq08:55:261
2259127993413,6cyclictest141rcu_preempt11:30:001
225912799341,30cyclictest4497-21PK-Backend08:45:461
225912799341,30cyclictest0-21swapper/108:44:131
225912299342,6cyclictest0-21swapper/007:30:140
225912299342,10cyclictest2364419-21hddtemp_smartct12:05:190
2259122993419,11cyclictest2339665-21cat10:55:230
2259122993419,11cyclictest2290652-21cat08:35:290
2259122993417,13cyclictest2317206-21awk09:50:220
2259122993415,14cyclictest2334212-21df10:40:160
2259133993323,6cyclictest42-21ksoftirqd/311:05:143
2259133993323,6cyclictest42-21ksoftirqd/307:20:233
2259133993322,6cyclictest42-21ksoftirqd/310:50:143
225913399332,10cyclictest2342281-40lsb_release11:02:063
2259133993317,12cyclictest2277279-21cpuspeed_turbos08:00:153
2259133993317,12cyclictest2277279-21cpuspeed_turbos08:00:153
225913399331,28cyclictest4497-21PK-Backend11:33:173
225913399331,28cyclictest2330805-21chrt10:30:203
225913399331,10cyclictest1913-21cron08:15:003
225913199331,9cyclictest4497-21PK-Backend12:05:182
2259131993319,10cyclictest34-21ksoftirqd/212:00:212
2259131993318,11cyclictest2341924-21grep11:00:282
2259131993318,11cyclictest2275865-21seq07:55:192
2259131993317,12cyclictest2311307-21expr09:35:142
2259131993317,12cyclictest2278049-21awk08:00:302
2259131993317,12cyclictest2278049-21awk08:00:292
2259131993315,13cyclictest1074-21lldpd12:32:332
2259127993323,6cyclictest26-21ksoftirqd/111:05:271
2259127993323,6cyclictest26-21ksoftirqd/110:25:161
2259127993323,5cyclictest26-21ksoftirqd/108:11:551
2259127993322,7cyclictest2365910-21tr12:10:121
2259127993322,7cyclictest2365910-21tr12:10:121
2259127993321,7cyclictest26-21ksoftirqd/111:05:001
2259127993318,10cyclictest26-21ksoftirqd/108:55:021
2259127993317,12cyclictest26-21ksoftirqd/107:45:191
2259127993315,13cyclictest26-21ksoftirqd/108:40:011
225912799331,29cyclictest4497-21PK-Backend09:36:071
225912799331,29cyclictest0-21swapper/111:44:191
225912799331,12cyclictest0-21swapper/112:05:141
225912299339,8cyclictest141rcu_preempt07:55:140
225912299333,10cyclictest2296371-21apt-config08:55:030
225912299332,9cyclictest2356059-21ntp_states11:40:220
225912299332,9cyclictest2304403-21df09:15:170
225912299332,9cyclictest141rcu_preempt12:00:140
225912299332,8cyclictest2368063-21memory12:15:210
2259122993326,4cyclictest0-21swapper/007:21:100
2259122993321,8cyclictest2069-21gnome-shell08:21:100
225912299331,9cyclictest2373038-21df12:30:140
2259122993318,11cyclictest2272449-21awk07:45:170
2259122993317,12cyclictest923-21dbus-daemon07:55:010
2259122993317,12cyclictest2283057-21if_enp2s008:15:220
225912299331,4cyclictest141rcu_preempt11:55:100
225912299331,16cyclictest2279939-21grep08:05:280
2259122993310,6cyclictest141rcu_preempt08:15:010
197399330,26rtkit-daemon1051-21thermald11:18:581
2259133993225,4cyclictest0-21swapper/308:25:213
2259133993222,6cyclictest42-21ksoftirqd/310:06:443
2259133993218,10cyclictest2261028-21seq07:15:033
225913399321,27cyclictest2296774-21dpkg08:55:133
225913199328,9cyclictest141rcu_preempt09:15:152
2259131993221,8cyclictest2359977-21pool-evolution-11:51:492
2259131993220,8cyclictest2069-21gnome-shell10:36:092
2259131993220,7cyclictest2343687-21swap11:05:282
2259131993216,11cyclictest438-20systemd-journal11:36:452
2259131993216,11cyclictest2357258-21cstates11:45:152
2259131993215,13cyclictest1054-21snmpd07:15:122
225913199321,28cyclictest4497-21PK-Backend07:25:492
2259127993221,7cyclictest2340544-21unin-run11:00:021
2259127993221,6cyclictest26-21ksoftirqd/111:45:251
2259127993221,5cyclictest26-21ksoftirqd/108:15:041
2259127993219,6cyclictest26-21ksoftirqd/112:20:501
2259122993217,11cyclictest2265790-21ps07:25:260
2259122993216,12cyclictest923-21dbus-daemon09:05:000
225912299321,28cyclictest4497-21PK-Backend12:26:230
225912299321,11cyclictest2288880-21timerwakeupswit08:30:280
2259131993124,4cyclictest4497-21PK-Backend09:56:142
2259131993115,12cyclictest1054-21snmpd07:12:112
2259131993115,12cyclictest1054-21snmpd07:12:112
225912799319,6cyclictest141rcu_preempt10:50:291
2259127993125,4cyclictest0-21swapper/112:15:551
2259127993121,5cyclictest26-21ksoftirqd/111:35:171
225912299312,9cyclictest2370296-21users12:20:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional