You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 12:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Jan 07, 2026 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
196819621600,26sleep013-21ksoftirqd/019:05:160
202813421070,7sleep14497-21PK-Backend21:55:001
202813421070,7sleep14497-21PK-Backend21:55:001
1973991000,21rtkit-daemon0-21swapper/223:53:082
20636382970,5sleep30-21swapper/323:35:153
20761952960,15sleep126-21ksoftirqd/100:10:021
20564992920,8sleep20-21swapper/223:15:132
196906227940,14sleep30-21swapper/319:07:413
196910727051,12sleep20-21swapper/219:08:172
20128592670,5chrt4497-21PK-Backend21:10:142
196923826341,14sleep10-21swapper/119:09:571
20014842540,6sleep342-21ksoftirqd/320:35:303
20229712520,6sleep30-21swapper/321:40:013
20044162520,7chrt4497-21PK-Backend20:45:210
19942742520,7sleep00-21swapper/020:15:260
20838982500,5sleep20-21swapper/200:30:152
19888802500,7sleep14497-21PK-Backend20:00:221
1969370995025,20cyclictest2073434-21irqstats00:00:202
1969370995021,25cyclictest2005366-21unin-run20:50:012
19991072480,17sleep013-21ksoftirqd/020:30:190
1969370994628,14cyclictest34-21ksoftirqd/220:50:132
1969370994628,14cyclictest34-21ksoftirqd/220:40:022
1969370994627,14cyclictest34-21ksoftirqd/220:30:002
1969370994619,23cyclictest34-21ksoftirqd/222:00:002
1969363994624,13cyclictest13-21ksoftirqd/021:40:020
20682882450,8chrt4497-21PK-Backend23:45:293
1969370994528,13cyclictest34-21ksoftirqd/221:35:222
1969370994527,14cyclictest34-21ksoftirqd/223:10:202
1969370994433,6cyclictest34-21ksoftirqd/221:30:212
1969370994426,6cyclictest34-21ksoftirqd/219:55:222
1969370994426,14cyclictest34-21ksoftirqd/221:46:152
1969370994424,14cyclictest34-21ksoftirqd/222:10:272
1969370994422,11cyclictest34-21ksoftirqd/221:40:162
1969363994429,11cyclictest1996664-21cron20:25:000
20864032430,7sleep32212-21gvfs-afc-volume00:35:313
20864032430,7sleep32212-21gvfs-afc-volume00:35:303
20820932430,7sleep20-21swapper/200:25:202
20819292430,8sleep01969358-21cyclictest00:25:130
1969370994325,14cyclictest34-21ksoftirqd/223:30:152
1969370994325,14cyclictest34-21ksoftirqd/220:56:032
1969370994325,14cyclictest34-21ksoftirqd/220:20:142
1969370994324,6cyclictest34-21ksoftirqd/220:40:132
1969370994324,6cyclictest34-21ksoftirqd/200:15:002
1969370994324,14cyclictest34-21ksoftirqd/223:10:022
1969370994323,7cyclictest34-21ksoftirqd/222:15:032
1969370994323,7cyclictest34-21ksoftirqd/222:15:022
1969370994224,6cyclictest34-21ksoftirqd/222:00:162
1969370994224,6cyclictest34-21ksoftirqd/219:15:292
1969370994224,14cyclictest34-21ksoftirqd/223:25:192
1969370994224,14cyclictest34-21ksoftirqd/223:25:192
1969370994224,14cyclictest34-21ksoftirqd/222:20:152
1969370994224,14cyclictest34-21ksoftirqd/221:55:012
1969370994224,14cyclictest34-21ksoftirqd/221:55:012
1969370994224,14cyclictest34-21ksoftirqd/221:20:142
1969370994224,14cyclictest34-21ksoftirqd/220:05:282
1969370994224,14cyclictest34-21ksoftirqd/219:20:212
1969370994224,13cyclictest34-21ksoftirqd/222:40:202
1969370994224,13cyclictest34-21ksoftirqd/221:25:122
1969370994224,13cyclictest34-21ksoftirqd/221:25:122
1969370994224,13cyclictest34-21ksoftirqd/219:50:302
1969370994223,6cyclictest34-21ksoftirqd/223:00:282
1969370994223,6cyclictest34-21ksoftirqd/220:00:292
1969370994223,14cyclictest34-21ksoftirqd/222:55:252
1969370994223,14cyclictest34-21ksoftirqd/222:25:152
1969370994223,14cyclictest34-21ksoftirqd/221:15:212
1969370994223,14cyclictest34-21ksoftirqd/219:15:022
1969363994231,7cyclictest1987717-21turbostat.cron20:00:000
19974972410,8sleep342-21ksoftirqd/320:25:233
1969373994117,19cyclictest42-21ksoftirqd/321:05:023
1969370994129,9cyclictest34-21ksoftirqd/223:38:052
1969370994124,6cyclictest34-21ksoftirqd/221:05:272
1969370994124,6cyclictest34-21ksoftirqd/220:10:372
1969370994123,14cyclictest34-21ksoftirqd/223:45:032
1969370994123,14cyclictest34-21ksoftirqd/222:55:022
1969370994123,14cyclictest34-21ksoftirqd/219:50:012
1969370994123,14cyclictest34-21ksoftirqd/200:10:012
1969370994122,14cyclictest34-21ksoftirqd/219:30:292
1969370994121,5cyclictest34-21ksoftirqd/222:10:022
1969370994121,5cyclictest34-21ksoftirqd/220:30:212
1969370994120,13cyclictest34-21ksoftirqd/200:25:002
196936899417,15cyclictest141rcu_preempt22:40:001
196936899417,15cyclictest141rcu_preempt22:40:001
1969368994122,14cyclictest26-21ksoftirqd/121:30:331
1969368994111,11cyclictest141rcu_preempt23:45:131
1969363994126,11cyclictest2061324-21latency_hist23:30:000
1969363994126,11cyclictest2061324-21latency_hist23:29:590
196936399411,37cyclictest0-21swapper/000:37:190
196936399411,37cyclictest0-21swapper/000:37:190
196936399411,11cyclictest2072709-21/usr/sbin/munin00:00:010
19800382400,7sleep3421ksoftirqd/319:35:293
1969373994025,11cyclictest2009087-21cron21:00:013
1969373994018,7cyclictest1137-21runrttasks19:47:403
1969373994018,18cyclictest42-21ksoftirqd/320:17:093
1969373994012,6cyclictest141rcu_preempt19:15:003
1969370994023,5cyclictest34-21ksoftirqd/219:45:022
1969370994023,13cyclictest34-21ksoftirqd/222:45:122
1969370994022,6cyclictest34-21ksoftirqd/223:55:222
1969370994022,6cyclictest34-21ksoftirqd/219:29:412
1969370994022,6cyclictest34-21ksoftirqd/219:29:412
1969370994022,6cyclictest34-21ksoftirqd/200:37:472
1969370994022,6cyclictest34-21ksoftirqd/200:37:472
1969370994022,5cyclictest34-21ksoftirqd/223:40:302
1969370994022,13cyclictest34-21ksoftirqd/220:15:572
196936899401,17cyclictest20838992expr00:30:151
196936399409,5cyclictest141rcu_preempt23:10:170
1969363994024,12cyclictest2066476-40apt-check23:40:530
1969363994023,12cyclictest1036-21gdbus23:49:570
1969363994021,15cyclictest13-21ksoftirqd/022:10:000
1969363994019,17cyclictest1991661-21latency_hist20:10:010
196936399401,11cyclictest2050922-21cron23:00:000
20603632390,6sleep3411rcuc/323:25:213
20603632390,6sleep3411rcuc/323:25:213
20098002390,5chrt0-21swapper/021:00:220
20098002390,5chrt0-21swapper/021:00:210
1969373993913,22cyclictest2032923-21timerwakeupswit22:05:293
196937399391,10cyclictest1986253-21apt19:55:113
1969370993922,5cyclictest34-21ksoftirqd/221:01:362
1969370993922,5cyclictest34-21ksoftirqd/221:01:362
1969370993922,5cyclictest34-21ksoftirqd/200:16:512
1969370993921,6cyclictest34-21ksoftirqd/223:23:392
1969370993921,6cyclictest34-21ksoftirqd/222:30:472
196936899399,8cyclictest2036404-21processes22:15:261
196936899399,8cyclictest2036404-21processes22:15:261
1969368993925,10cyclictest2067083-21date23:45:021
1969368993924,11cyclictest2026392-21cron21:50:011
196936899392,17cyclictest1979781-21ntp_states19:35:251
196936899391,10cyclictest2059575-21cron23:25:001
1969368993910,14cyclictest26-21ksoftirqd/123:52:591
196936399398,15cyclictest141rcu_preempt19:50:170
1969363993930,6cyclictest0-21swapper/019:10:100
1969363993923,11cyclictest2021202-21cron21:35:000
1969363993922,12cyclictest923-21dbus-daemon22:15:000
1969363993922,12cyclictest2064808-21cron23:40:000
1969363993920,14cyclictest438-20systemd-journal19:45:000
1969363993920,14cyclictest1975330-21pluginstate19:25:020
1969363993920,14cyclictest13-21ksoftirqd/020:35:230
196936399391,34cyclictest0-21swapper/019:35:030
196936399391,15cyclictest2074776-21seq00:05:140
1969363993910,13cyclictest141rcu_preempt21:10:270
20075762380,13chrt26-21ksoftirqd/120:55:101
19906622380,9sleep342-21ksoftirqd/320:05:233
196937399382,32cyclictest2073537-21memory00:00:203
1969373993822,12cyclictest2038698-21sshd22:23:373
1969373993822,12cyclictest1984521-21/usr/sbin/munin19:50:113
1969373993820,13cyclictest438-20systemd-journal22:15:373
1969373993820,13cyclictest438-20systemd-journal22:15:363
1969373993810,6cyclictest141rcu_preempt23:45:023
196936899386,15cyclictest141rcu_preempt21:10:001
1969368993821,5cyclictest26-21ksoftirqd/100:17:331
1969368993820,13cyclictest1043-21gdbus23:38:041
1969368993814,6cyclictest141rcu_preempt22:55:281
196936399384,11cyclictest13-21ksoftirqd/022:26:500
196936399384,10cyclictest13-21ksoftirqd/019:28:140
196936399384,10cyclictest13-21ksoftirqd/019:28:140
196936399383,11cyclictest13-21ksoftirqd/023:07:270
1969363993827,6cyclictest2078899-21ntpq00:15:270
1969363993820,14cyclictest13-21ksoftirqd/020:55:000
1969363993820,14cyclictest1-21systemd21:50:000
1969363993820,13cyclictest13-21ksoftirqd/022:55:000
1969363993820,13cyclictest13-21ksoftirqd/022:15:030
1969363993820,13cyclictest13-21ksoftirqd/022:15:030
1969363993819,15cyclictest13-21ksoftirqd/021:00:020
196936399381,33cyclictest2062072-21chrt23:30:210
196937399374,20cyclictest141rcu_preempt22:10:123
1969373993719,13cyclictest438-20systemd-journal21:25:013
1969370993721,5cyclictest34-21ksoftirqd/219:35:152
196936899379,12cyclictest141rcu_preempt20:25:191
1969368993722,11cyclictest26-21ksoftirqd/123:12:551
1969368993722,10cyclictest1975417-21cat19:25:021
1969368993721,11cyclictest26-21ksoftirqd/122:43:171
1969368993720,13cyclictest26-21ksoftirqd/121:38:221
1969368993710,11cyclictest141rcu_preempt22:25:221
196936399379,5cyclictest141rcu_preempt21:40:290
1969363993719,14cyclictest2073313-21df00:00:140
1969363993719,13cyclictest2011831-21ntpq21:05:240
1969363993719,13cyclictest13-21ksoftirqd/021:20:000
1969363993715,18cyclictest1993247-21unin-run20:15:010
196937399369,5cyclictest141rcu_preempt21:50:013
196937399366,9cyclictest1973710-21apt-key19:20:033
1969373993626,6cyclictest2045754-21basename22:45:003
1969373993620,12cyclictest976-21rs:main3
1969373993620,12cyclictest2047236-21sshd22:47:103
1969373993612,12cyclictest141rcu_preempt00:09:553
1969370993619,5cyclictest34-21ksoftirqd/222:37:392
1969370993619,5cyclictest34-21ksoftirqd/222:37:382
196936899368,12cyclictest141rcu_preempt22:05:171
196936899363,16cyclictest2013413-21memory21:10:221
1969368993626,7cyclictest2077884-21basename00:15:011
1969368993622,10cyclictest1990240-21cstates20:05:151
196936899362,18cyclictest2028503-21tr21:55:141
1969368993621,11cyclictest2085752-21diskstats00:35:191
1969368993621,11cyclictest2085752-21diskstats00:35:191
1969368993621,11cyclictest2031625-21systemd-run22:05:001
1969368993620,11cyclictest1977749-21cat19:30:201
196936899361,14cyclictest2003598-21cron20:45:001
196936899361,12cyclictest1985230-21awk19:50:231
196936399366,12cyclictest141rcu_preempt00:23:410
1969363993624,6cyclictest13-21ksoftirqd/021:25:150
1969363993624,6cyclictest13-21ksoftirqd/021:25:140
1969363993620,12cyclictest923-21dbus-daemon00:35:000
1969363993620,12cyclictest2016869-21load21:20:220
1969363993619,13cyclictest13-21ksoftirqd/019:50:010
196936399361,30cyclictest1978250-21find19:30:290
196937399353,19cyclictest141rcu_preempt23:21:343
196937399352,14cyclictest2029274-21python321:55:293
196937399352,13cyclictest2020149-21hddtemp_smartct21:30:183
1969373993521,10cyclictest42-21ksoftirqd/320:53:203
1969373993520,11cyclictest2078536-21awk00:15:193
1969373993519,11cyclictest438-20systemd-journal23:16:383
1969373993512,19cyclictest4497-21PK-Backend00:25:093
196937399351,14cyclictest2069530-21/usr/sbin/munin23:50:133
196937399351,10cyclictest0-21swapper/323:01:563
196936899355,9cyclictest26-21ksoftirqd/123:00:241
196936899353,16cyclictest3692989-21snapd00:21:101
196936899353,16cyclictest0-21swapper/123:15:221
196936899352,8cyclictest2001153-21ntp_states20:35:241
1969368993525,6cyclictest2019487-21latency_hist21:30:001
1969368993525,6cyclictest2019487-21latency_hist21:30:001
1969368993519,12cyclictest1975778-21cat19:25:161
1969368993519,12cyclictest1975778-21cat19:25:151
196936899351,30cyclictest2082281-21strings00:25:221
196936899351,13cyclictest1986735-21irqstats19:55:191
196936399357,11cyclictest141rcu_preempt20:04:530
1969363993524,6cyclictest13-21ksoftirqd/019:20:010
1969363993521,10cyclictest1997841-21cat20:25:310
1969363993517,14cyclictest13-21ksoftirqd/020:41:090
196936399351,12cyclictest2030865-21grep22:00:250
20534082340,5sleep1251rcuc/123:05:191
196937399349,5cyclictest141rcu_preempt21:10:303
196937399346,12cyclictest4497-21PK-Backend00:20:313
196937399343,11cyclictest1999259-21load20:30:213
1969373993423,8cyclictest1991926-21tr20:10:133
1969373993421,8cyclictest1989000-21ntp_states20:00:253
1969373993418,12cyclictest2055227-21cat23:10:213
1969373993418,12cyclictest2049034-21seq22:53:023
1969373993418,12cyclictest1972517-21ntpq19:15:233
1969373993417,12cyclictest438-20systemd-journal20:46:383
1969373993416,6cyclictest42-21ksoftirqd/323:56:023
1969373993416,14cyclictest42-21ksoftirqd/322:28:583
196937399341,29cyclictest2062296-21ntp_kernel_err23:30:223
196937399341,28cyclictest438-20systemd-journal23:09:593
196936899349,6cyclictest141rcu_preempt00:00:021
196936899343,14cyclictest2048104-21df22:50:141
1969368993422,8cyclictest2069-21gnome-shell21:44:081
196936899342,10cyclictest1971994-21tr19:15:141
1969368993420,10cyclictest26-21ksoftirqd/123:25:161
1969368993420,10cyclictest26-21ksoftirqd/123:25:161
1969368993419,11cyclictest2041143-21expr22:30:151
1969368993418,12cyclictest976-21rs:main1
1969368993418,11cyclictest1992937-21sshd20:11:371
196936899341,7cyclictest4497-21PK-Backend22:45:101
196936899341,30cyclictest0-21swapper/119:14:501
196936899341,29cyclictest1993485-21apt20:15:121
1969368993412,7cyclictest141rcu_preempt19:40:291
196936899341,14cyclictest2069-21gnome-shell19:46:591
196936399342,11cyclictest2047474-21turbostat22:50:000
1969363993419,11cyclictest2042524-21/usr/sbin/munin22:35:200
1969363993419,11cyclictest2042524-21/usr/sbin/munin22:35:190
1969363993416,14cyclictest13-21ksoftirqd/021:50:120
1969363993416,14cyclictest13-21ksoftirqd/021:50:110
196936399341,4cyclictest0-21swapper/023:54:550
196936399341,4cyclictest0-21swapper/023:23:060
196936399341,29cyclictest4497-21PK-Backend21:55:170
196937399338,5cyclictest141rcu_preempt22:30:273
196937399337,13cyclictest141rcu_preempt00:30:263
196937399335,24cyclictest2002224-21cpuspeed20:40:123
1969373993321,8cyclictest2069-21gnome-shell22:36:593
1969373993321,8cyclictest2069-21gnome-shell22:36:583
1969373993319,10cyclictest42-21ksoftirqd/300:10:193
1969373993318,11cyclictest2031319-21seq22:01:393
1969373993317,12cyclictest2023216-21apt21:40:123
1969373993314,6cyclictest141rcu_preempt19:30:153
196937399331,29cyclictest4497-21PK-Backend21:03:403
196937399331,29cyclictest4497-21PK-Backend21:03:393
196937399331,29cyclictest0-21swapper/322:56:343
1969373993310,4cyclictest141rcu_preempt19:30:023
1969373993310,4cyclictest141rcu_preempt19:30:023
196936899339,6cyclictest141rcu_preempt20:55:011
196936899334,10cyclictest26-21ksoftirqd/120:34:221
196936899333,9cyclictest141rcu_preempt21:03:191
196936899333,9cyclictest141rcu_preempt21:03:191
1969368993318,11cyclictest2014558-21sh21:15:101
196936899331,29cyclictest4497-21PK-Backend21:22:451
196936899331,29cyclictest4497-21PK-Backend20:46:281
196936899331,13cyclictest4497-21PK-Backend20:20:201
1969363993322,6cyclictest13-21ksoftirqd/022:40:390
1969363993318,11cyclictest923-21dbus-daemon23:20:010
1969363993316,12cyclictest438-20systemd-journal23:05:000
196936399331,29cyclictest0-21swapper/022:32:100
196936399331,12cyclictest13-21ksoftirqd/022:23:560
196937399322,8cyclictest1995966-21ntp_states20:20:253
196937399322,25cyclictest2018768-21ntp_states21:25:253
196937399322,25cyclictest2018768-21ntp_states21:25:243
196937399321,4cyclictest0-21swapper/321:52:273
196937399321,4cyclictest0-21swapper/321:52:263
196936899321,4cyclictest4497-21PK-Backend00:02:511
196936899321,12cyclictest2037856-21kernelversion22:20:211
1969368993210,5cyclictest141rcu_preempt22:10:291
196937399311,4cyclictest4497-21PK-Backend19:44:523
1969363993119,8cyclictest2076573-21tr00:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional