You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-11 - 22:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Jan 11, 2026 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4137652960,6sleep013-21ksoftirqd/009:25:280
4057022940,5sleep20-21swapper/209:05:132
36398228140,13sleep20-21swapper/207:08:252
197399770,9rtkit-daemon3037-21gmain10:18:121
4554892760,5sleep2331rcuc/211:25:142
197399750,8rtkit-daemon1051-21thermald12:37:061
197399750,60rtkit-daemon0-21swapper/107:06:351
197399740,7rtkit-daemon4497-21PK-Backend11:17:083
197399740,7rtkit-daemon4497-21PK-Backend11:17:083
197399740,10rtkit-daemon2746-21gmain10:20:061
3742662720,3sleep20-21swapper/207:35:232
197399710,5rtkit-daemon0-21swapper/311:37:013
36400226243,13sleep00-21swapper/007:08:380
36400026241,14sleep30-21swapper/307:08:373
197399590,9rtkit-daemon1972-21rtkit-daemon07:42:273
197399560,8rtkit-daemon1972-21rtkit-daemon12:28:380
197399560,8rtkit-daemon1972-21rtkit-daemon12:28:370
197399550,9rtkit-daemon1972-21rtkit-daemon12:38:060
4267062540,7sleep24497-21PK-Backend10:05:022
4243452540,3sleep20-21swapper/209:55:322
3873262540,4sleep20-21swapper/208:10:282
197399540,8rtkit-daemon1972-21rtkit-daemon12:23:510
197399540,8rtkit-daemon1972-21rtkit-daemon10:44:040
4575662530,11sleep1457572-21needreboot11:30:201
364283995320,29cyclictest382255-21seq07:59:001
364283995319,29cyclictest461482-21sed11:40:301
364283995318,30cyclictest375255-21date07:40:011
197399530,8rtkit-daemon1972-21rtkit-daemon08:19:513
36429199518,9cyclictest42-21ksoftirqd/312:05:223
36428399515,40cyclictest437459-21cat10:35:131
364283995114,32cyclictest471457-21cat12:10:201
364283995020,26cyclictest448547-21expr11:05:141
364283995015,30cyclictest447119-21wc11:00:221
364283995013,32cyclictest459167-21wc11:35:201
364283995013,32cyclictest439329-21expr10:40:141
364283995012,33cyclictest367931-21seq07:17:271
36428399501,11cyclictest438-20systemd-journal08:46:261
197399500,7rtkit-daemon1972-21rtkit-daemon10:50:540
364291994926,7cyclictest141rcu_preempt11:50:303
364283994914,30cyclictest421667-21expr09:50:141
364283994914,30cyclictest417837-21cut09:40:011
364283994914,30cyclictest417837-21cut09:40:011
364283994913,32cyclictest443189-21basename10:50:201
364283994913,31cyclictest470705-21cat12:10:011
36428399482,40cyclictest393539-21tr08:30:141
364283994819,26cyclictest396483-21sshd08:39:281
364283994814,29cyclictest402444-21if_enp2s008:55:201
364283994814,29cyclictest370723-21load07:25:211
364283994813,31cyclictest403984-21expr09:00:151
364283994813,30cyclictest444729-21find10:55:111
364283994812,32cyclictest434857-21cat10:25:321
364283994812,32cyclictest434857-21cat10:25:321
364283994812,32cyclictest410128-21seq09:16:541
36428399477,34cyclictest382687-21cat08:00:141
36428399474,38cyclictest384513-21cut08:05:141
364283994711,31cyclictest418901-21timerwakeupswit09:40:291
364287994622,19cyclictest381758-21python307:55:272
36428399468,33cyclictest413102-21basename09:25:181
36428399468,32cyclictest436068-21cat10:30:191
364283994614,27cyclictest407562-21df_inode09:10:141
364283994614,22cyclictest456034-21processes11:25:261
364283994613,29cyclictest1-21systemd12:05:011
364283994613,28cyclictest450879-21switchtime11:10:281
364283994612,29cyclictest441442-21if_enp2s010:45:201
364283994611,31cyclictest420725-21seq09:45:411
364283994610,30cyclictest473763-21missed_timers12:20:181
364291994516,12cyclictest141rcu_preempt10:11:133
364291994512,17cyclictest42-21ksoftirqd/308:45:013
364283994519,22cyclictest4497-21PK-Backend08:13:181
364283994517,24cyclictest1137-21runrttasks10:03:111
364283994513,27cyclictest438-20systemd-journal08:30:001
364283994513,27cyclictest423482-21tr09:55:161
36429199449,19cyclictest42-21ksoftirqd/309:55:003
364291994417,23cyclictest401072-21ps08:50:273
36428399449,30cyclictest414478-21cpuspeed09:30:131
36428399448,31cyclictest463986-21cpuspeed_turbos11:50:131
364283994417,23cyclictest405674-21cstates09:05:141
364283994411,28cyclictest1043-21gdbus10:05:241
36429199435,5cyclictest141rcu_preempt07:50:003
364291994315,15cyclictest42-21ksoftirqd/308:13:593
364291994314,15cyclictest42-21ksoftirqd/310:38:293
36428399439,29cyclictest378976-21cat07:50:011
36428399437,11cyclictest369384-21sed07:20:271
364283994318,22cyclictest1137-21runrttasks11:18:091
364283994318,22cyclictest1137-21runrttasks11:18:091
364283994316,23cyclictest401309-21sed08:50:291
364283994313,26cyclictest477058-21idleruntime12:30:191
364283994311,29cyclictest379356-21cstates07:50:131
364283994310,28cyclictest375701-21cpu07:40:131
364291994228,9cyclictest42-21ksoftirqd/310:05:253
364291994216,8cyclictest42-21ksoftirqd/312:13:163
364287994221,15cyclictest410923-21seq09:20:132
36428399425,11cyclictest463359-21seq11:46:581
364283994216,21cyclictest372833-21grep07:30:301
364283994213,25cyclictest429700-21vmstat10:10:301
364283994210,28cyclictest390941-21ls08:20:281
36429199415,19cyclictest42-21ksoftirqd/310:15:293
364291994117,10cyclictest42-21ksoftirqd/309:05:133
36428399412,14cyclictest454323-21python311:20:271
364283994111,26cyclictest475343-21if_enp2s012:25:191
364283994111,26cyclictest475343-21if_enp2s012:25:191
36427999415,31cyclictest401724-21cron08:54:590
364279994122,6cyclictest13-21ksoftirqd/009:20:080
4373182400,6sleep24497-21PK-Backend10:35:132
36429199409,7cyclictest42-21ksoftirqd/312:15:163
364291994013,10cyclictest42-21ksoftirqd/308:25:383
36429199401,22cyclictest379556-21taskset07:50:183
36429199401,17cyclictest4497-21PK-Backend09:15:013
364287994010,14cyclictest141rcu_preempt07:49:592
364283994010,26cyclictest397301-21grep08:40:201
36427999402,6cyclictest0-21swapper/011:20:130
364279994023,13cyclictest378293-21sensors_temp07:45:270
197399400,28rtkit-daemon4497-21PK-Backend11:09:202
36429199392,14cyclictest373394-21kworker/3:210:47:453
364291993912,10cyclictest42-21ksoftirqd/307:56:023
364291993912,10cyclictest42-21ksoftirqd/307:15:533
364287993927,7cyclictest460357-21systemd-udevd11:40:072
36428399391,7cyclictest438-20systemd-journal08:20:001
364283993912,23cyclictest1137-21runrttasks12:18:151
36428399391,15cyclictest466108-21grep11:55:201
36428399391,15cyclictest365618-21taskset07:11:171
36428399391,15cyclictest365618-21taskset07:11:171
36427999392,33cyclictest434673-21python310:25:300
36427999392,33cyclictest434673-21python310:25:300
364279993922,5cyclictest13-21ksoftirqd/008:59:520
364279993920,14cyclictest923-21dbus-daemon11:40:070
197399390,31rtkit-daemon4497-21PK-Backend09:26:353
197399390,27rtkit-daemon4497-21PK-Backend11:20:022
197399390,27rtkit-daemon4497-21PK-Backend11:20:012
36429199388,11cyclictest42-21ksoftirqd/307:34:033
36429199383,12cyclictest42-21ksoftirqd/308:00:303
36429199381,14cyclictest454859-21taskset11:24:303
36428799382,31cyclictest397025-21cstates08:40:132
36428799381,15cyclictest390045-21cut08:20:132
36428399389,24cyclictest412472-21cut09:25:011
364279993822,12cyclictest427640-21open_inodes10:05:270
364279993822,11cyclictest438-20systemd-journal09:36:280
364279993822,11cyclictest438-20systemd-journal09:36:280
364279993818,16cyclictest2069-21gnome-shell08:11:090
197399380,27rtkit-daemon4497-21PK-Backend12:01:412
36429199374,18cyclictest42-21ksoftirqd/309:45:133
36429199373,11cyclictest42-21ksoftirqd/309:20:153
364291993729,5cyclictest460360-21colord-sane11:40:093
36429199372,31cyclictest924-21NetworkManager09:34:593
364291993712,8cyclictest42-21ksoftirqd/311:30:283
36429199371,13cyclictest416119-21taskset09:35:043
36429199371,13cyclictest416119-21taskset09:35:043
364291993710,10cyclictest42-21ksoftirqd/307:12:263
364291993710,10cyclictest42-21ksoftirqd/307:12:263
364279993726,7cyclictest476702-21cpuspeed12:30:120
364279993720,14cyclictest449991-21apt11:10:110
36427999371,31cyclictest406207-21ntp_states09:05:250
364279993712,7cyclictest4287412cpuspeed_turbos10:10:140
197399370,29rtkit-daemon374753-21users07:35:313
197399370,27rtkit-daemon4497-21PK-Backend11:47:102
197399370,25rtkit-daemon0-21swapper/310:03:123
197399370,25rtkit-daemon0-21swapper/210:11:432
4343312360,6sleep20-21swapper/210:25:222
4343312360,6sleep20-21swapper/210:25:222
36429199363,13cyclictest42-21ksoftirqd/307:23:533
36429199363,12cyclictest42-21ksoftirqd/310:45:013
36429199362,10cyclictest446261-21munin-run10:59:593
364291993618,14cyclictest477666-21grep12:30:293
36429199361,31cyclictest42-21ksoftirqd/311:05:233
36429199361,11cyclictest0-21swapper/311:58:543
364287993618,14cyclictest2345-21gdbus08:19:572
36427999362,9cyclictest424497-40lsb_release09:56:040
364279993625,7cyclictest470552-21turbostat12:09:590
364279993620,12cyclictest425960-21awk10:00:260
364279993620,11cyclictest438-20systemd-journal11:45:210
364279993619,13cyclictest3693053-21snapd12:11:090
364279993613,18cyclictest466576-21timerandwakeup11:55:300
197399360,29rtkit-daemon0-21swapper/010:30:450
36429199355,9cyclictest42-21ksoftirqd/311:10:133
36429199355,14cyclictest0-21swapper/310:35:013
36429199355,13cyclictest0-21swapper/309:55:113
36429199354,6cyclictest141rcu_preempt12:40:013
364291993519,12cyclictest475646-21grep12:25:253
364291993519,12cyclictest475646-21grep12:25:253
364291993512,6cyclictest408899-21/usr/sbin/munin09:15:223
36429199351,14cyclictest395667-21memory08:35:223
36429199351,10cyclictest4497-21PK-Backend12:20:153
36429199351,10cyclictest4497-21PK-Backend10:25:333
36429199351,10cyclictest4497-21PK-Backend10:25:323
36428799359,9cyclictest974-21in:imuxsock10:09:592
364287993521,10cyclictest478752-21diskstats12:35:172
364287993518,12cyclictest454803-21sshd11:23:452
364287993518,12cyclictest401236-21grep08:50:302
364279993519,12cyclictest455244-21cat11:25:140
364279993519,11cyclictest438-20systemd-journal10:47:060
36427999351,31cyclictest0-21swapper/009:43:140
36427999351,29cyclictest438-20systemd-journal11:40:010
364279993510,22cyclictest368145-21perf07:20:000
197399350,25rtkit-daemon0-21swapper/212:19:212
197399350,24rtkit-daemon0-21swapper/210:59:032
364291993421,10cyclictest394159-21sshd08:30:263
364291993419,11cyclictest447339-21cut11:00:283
364291993419,11cyclictest399472-21cat08:45:303
36429199341,7cyclictest42-21ksoftirqd/310:22:203
364291993417,13cyclictest463598-21cron11:49:593
364291993417,12cyclictest438-20systemd-journal11:29:553
36429199341,16cyclictest4497-21PK-Backend08:20:153
36428799349,10cyclictest141rcu_preempt07:25:212
36428799346,9cyclictest34-21ksoftirqd/210:15:132
364287993423,7cyclictest394819-21turbostat.cron08:35:002
364287993419,11cyclictest465673-21awk11:55:132
364287993419,11cyclictest413674-21wc09:25:282
364287993419,11cyclictest1-21systemd09:35:002
364287993417,13cyclictest367553-21python307:15:272
36428799341,30cyclictest4497-21PK-Backend07:20:112
364279993426,5cyclictest0-21swapper/010:35:090
364279993422,8cyclictest2093-21JS10:49:090
364279993421,8cyclictest389938-21cpu08:20:110
364279993419,11cyclictest445694-21cat10:55:290
364279993419,11cyclictest415299-21python309:30:280
364279993419,11cyclictest398316-21grep08:45:020
364279993419,11cyclictest374334-21awk07:35:250
364279993418,12cyclictest431440-21seq10:15:340
364279993418,11cyclictest448770-21cut11:05:210
36427999341,30cyclictest4497-21PK-Backend09:19:480
197399340,27rtkit-daemon0-21swapper/208:08:092
197399340,23rtkit-daemon0-21swapper/211:12:482
36429199334,10cyclictest370120-21/usr/sbin/munin07:25:213
36429199332,9cyclictest42-21ksoftirqd/310:54:593
36429199331,9cyclictest4497-21PK-Backend12:02:183
36429199331,9cyclictest4497-21PK-Backend09:42:223
364291993318,11cyclictest402880-21smartctl08:55:283
364291993311,8cyclictest42-21ksoftirqd/309:03:483
36429199331,13cyclictest4497-21PK-Backend08:05:263
36428799337,9cyclictest141rcu_preempt10:50:262
364287993319,9cyclictest410686-21date09:20:022
364287993319,10cyclictest459714-21sed11:35:302
364287993318,11cyclictest976-21rs:main2
364287993317,12cyclictest474045-21python312:20:232
364287993315,13cyclictest371870-21awk07:30:112
36428799331,30cyclictest0-21swapper/211:53:102
36428799331,29cyclictest0-21swapper/207:10:292
36428799331,29cyclictest0-21swapper/207:10:282
364279993325,5cyclictest0-21swapper/009:10:440
364279993321,8cyclictest447258-21tr11:00:250
364279993321,8cyclictest389620-21sh08:20:010
364279993319,10cyclictest923-21dbus-daemon08:40:010
364279993318,11cyclictest369421-21sed07:20:280
364279993318,11cyclictest13-21ksoftirqd/008:00:170
364279993316,13cyclictest403913-21grep09:00:140
36427999331,29cyclictest0-21swapper/007:30:570
36427999331,14cyclictest13-21ksoftirqd/008:26:400
197399330,6rtkit-daemon4497-21PK-Backend12:11:052
364287993222,7cyclictest28750irq/122-ahci[0009:14:502
364287993221,6cyclictest34-21ksoftirqd/209:35:102
364287993221,6cyclictest34-21ksoftirqd/209:35:102
364287993220,8cyclictest28750irq/122-ahci[0010:47:072
364287993218,10cyclictest475404-21cut12:25:222
364287993218,10cyclictest475404-21cut12:25:222
364287993216,11cyclictest438-20systemd-journal08:49:562
36428799321,4cyclictest4497-21PK-Backend12:05:232
36428799321,4cyclictest4497-21PK-Backend11:30:402
364279993225,5cyclictest0-21swapper/008:06:000
364279993218,9cyclictest376835-21package-system-07:41:070
364279993218,10cyclictest381751-21gunzip07:55:260
364279993218,10cyclictest13-21ksoftirqd/011:50:140
364279993217,11cyclictest421640-21expr09:50:130
364279993216,12cyclictest468229-21smartctl12:00:280
364279993216,11cyclictest438-20systemd-journal08:30:010
364279993215,12cyclictest432222-21tr10:20:120
36427999321,28cyclictest0-21swapper/012:16:470
197399320,22rtkit-daemon0-21swapper/211:01:002
36428799319,5cyclictest141rcu_preempt10:30:212
36428799317,5cyclictest141rcu_preempt10:45:032
364287993116,11cyclictest383352-21awk08:00:252
364287993116,11cyclictest1074-21lldpd08:58:232
36428799311,5cyclictest141rcu_preempt09:55:012
364287993113,14cyclictest395725-21cat08:35:222
364287993113,14cyclictest376062-21which07:40:182
36428799311,27cyclictest4497-21PK-Backend10:20:182
36428799311,13cyclictest4497-21PK-Backend07:50:102
364279993116,11cyclictest420518-21cat09:45:290
364287993020,6cyclictest419350-21sh09:45:002
364287993019,7cyclictest477391-21sh12:30:252
36427999309,5cyclictest141rcu_preempt07:13:170
36427999309,5cyclictest141rcu_preempt07:13:160
364279993015,10cyclictest370325-21tr07:25:140
364279993014,12cyclictest452393-21uname11:15:230
364279993014,12cyclictest452393-21uname11:15:220
364279993014,12cyclictest380556-21seq07:54:260
36427999301,25cyclictest4497-21PK-Backend08:47:470
36428799293,12cyclictest141rcu_preempt09:05:012
36428799281,12cyclictest141rcu_preempt09:45:552
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional