You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 22:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 10, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991170,8rtkit-daemon13-21ksoftirqd/009:40:471
6004821010,23sleep12197014-21PK-Backend08:55:031
1374192980,7sleep1251rcuc/112:35:141
2247729246,38sleep30-21swapper/307:09:563
2263199780,62cyclictest1622-21rtkit-daemon11:21:463
2263199771,52cyclictest2181-21gdbus09:25:013
2242127540,10sleep20-21swapper/207:09:152
22631997415,29cyclictest60552-21if_enp2s008:55:223
2225427447,17sleep10-21swapper/107:07:081
2220627240,25sleep00-21swapper/007:06:310
162399680,9rtkit-daemon1622-21rtkit-daemon07:27:302
162399630,9rtkit-daemon1622-21rtkit-daemon09:07:293
162399630,9rtkit-daemon1622-21rtkit-daemon09:07:293
22631996210,35cyclictest72869-21find09:30:113
162399620,10rtkit-daemon1622-21rtkit-daemon07:16:023
162399620,10rtkit-daemon1622-21rtkit-daemon07:16:023
162399610,9rtkit-daemon1622-21rtkit-daemon10:08:253
162399610,9rtkit-daemon1622-21rtkit-daemon08:12:013
254702600,9sleep01753-21gnome-shell07:15:160
254702600,9sleep01753-21gnome-shell07:15:150
162399600,9rtkit-daemon1622-21rtkit-daemon09:04:433
162399600,8rtkit-daemon1622-21rtkit-daemon07:25:243
162399590,9rtkit-daemon1622-21rtkit-daemon12:39:313
162399590,9rtkit-daemon1622-21rtkit-daemon09:49:033
162399590,9rtkit-daemon1622-21rtkit-daemon08:00:113
2263199582,8cyclictest1753-21gnome-shell10:40:153
162399580,9rtkit-daemon1622-21rtkit-daemon09:54:533
2262799570,54cyclictest1622-21rtkit-daemon08:30:112
162399560,9rtkit-daemon1622-21rtkit-daemon08:17:273
1056732560,20sleep013-21ksoftirqd/011:05:010
913252550,4sleep10-21swapper/110:20:311
162399550,8rtkit-daemon1622-21rtkit-daemon10:44:342
1336742550,14sleep234-21ksoftirqd/212:25:012
22622995415,35cyclictest1753-21gnome-shell12:10:230
162399540,44rtkit-daemon1622-21rtkit-daemon08:10:382
22631995316,31cyclictest50381-21cut08:25:233
22631995314,34cyclictest37905-21expr07:50:163
22631995312,36cyclictest132810-21cat12:20:223
162399530,9rtkit-daemon1622-21rtkit-daemon09:18:433
162399530,8rtkit-daemon1622-21rtkit-daemon09:59:053
2263199522,44cyclictest1753-21gnome-shell07:21:163
22631995220,27cyclictest55628-21ntpq08:40:283
22631995220,27cyclictest55628-21ntpq08:40:273
22631995217,30cyclictest94812-21cat10:30:323
22631995216,31cyclictest116984-21hddtemp_smartct11:35:193
22631995216,31cyclictest116984-21hddtemp_smartct11:35:193
22631995213,35cyclictest0-21swapper/311:40:183
162399520,8rtkit-daemon1622-21rtkit-daemon08:21:163
162399520,8rtkit-daemon1622-21rtkit-daemon08:21:153
2263199513,42cyclictest134091-21cut12:25:133
22631995117,29cyclictest2962346-21in:imuxsock11:17:003
22631995117,29cyclictest131660-21cron12:17:003
2263199511,45cyclictest855-21dbus-daemon09:27:533
2263199511,45cyclictest855-21dbus-daemon09:27:523
2263199511,10cyclictest1-21systemd07:15:013
2263199502,44cyclictest1753-21gnome-shell12:33:183
2263199501,45cyclictest99815-21python310:45:313
22631995012,33cyclictest92587-21missed_timers10:25:223
2263199501,16cyclictest1753-21gnome-shell09:14:093
344382490,13sleep11753-21gnome-shell07:40:161
2263199492,43cyclictest1753-21gnome-shell12:12:403
2263199492,42cyclictest1753-21gnome-shell10:17:123
2263199492,41cyclictest53538-21df_inode08:35:163
2263199491,10cyclictest34202-21taskset07:40:143
2263199489,36cyclictest0-21swapper/310:20:293
2263199482,8cyclictest1753-21gnome-shell09:38:483
2263199482,42cyclictest1753-21gnome-shell10:01:313
2263199482,41cyclictest1753-21gnome-shell10:53:023
2263199482,40cyclictest108084-21irqstats11:10:213
22631994817,26cyclictest1753-21gnome-shell10:55:313
22631994817,26cyclictest1753-21gnome-shell07:45:333
22631994816,28cyclictest1069-21runrttasks11:06:463
2263199481,42cyclictest2197014-21PK-Backend09:41:043
22631994813,30cyclictest116519-21sed11:35:023
22631994812,30cyclictest43097-21cat08:05:143
22631994811,33cyclictest51806-21df_inode08:30:173
1097342480,7sleep02197014-21PK-Backend11:15:200
2263199472,7cyclictest1753-21gnome-shell08:45:143
22622994723,20cyclictest73488-40lsb_release09:30:230
2263199469,19cyclictest1753-21gnome-shell07:32:143
2263199462,39cyclictest1753-21gnome-shell11:55:023
22631994619,23cyclictest32268-21/usr/sbin/munin07:35:163
2263199461,10cyclictest113506-21awk11:25:193
22625994618,23cyclictest3242254-21gmain10:15:021
396842450,6chrt2197014-21PK-Backend07:55:203
2263199452,7cyclictest59452-21users08:50:323
2263199452,7cyclictest127568-21memory12:05:233
22631994517,23cyclictest2197014-21PK-Backend11:04:133
22631994516,25cyclictest96060-21missed_timers10:35:233
22631994515,25cyclictest120639-21missed_timers11:45:243
22631994411,28cyclictest123586-21cpuspeed11:55:143
22631994411,28cyclictest123586-21cpuspeed11:55:143
22627994425,14cyclictest438-20systemd-journal08:28:022
2262599448,16cyclictest141rcu_preempt12:31:361
2262299444,16cyclictest17531gnome-shell10:25:000
162399440,33rtkit-daemon1622-21rtkit-daemon09:53:272
2263199432,17cyclictest1753-21gnome-shell10:11:243
22625994321,18cyclictest70535-21cron09:29:551
22625994321,18cyclictest70535-21cron09:29:551
162399430,32rtkit-daemon1622-21rtkit-daemon09:19:542
2263199421,36cyclictest126399-21sshd12:01:133
2262799426,8cyclictest60074-21sshd08:55:122
22627994227,11cyclictest73261-21df_inode09:30:172
2262599426,12cyclictest141rcu_preempt09:05:021
22625994212,8cyclictest26-21ksoftirqd/109:55:261
22622994227,10cyclictest77544-21sshd09:41:120
22622994221,16cyclictest81357-21pool-org.gnome.09:55:010
22622994218,19cyclictest123278-21unin-run11:55:010
2262299421,12cyclictest141rcu_preempt11:20:030
22627994118,18cyclictest34440-21cstates07:40:152
22625994128,9cyclictest1617-10pipewire-media-12:25:141
2262599412,23cyclictest1753-21gnome-shell09:05:111
2262599412,23cyclictest1753-21gnome-shell09:05:111
22625994113,12cyclictest26-21ksoftirqd/108:45:181
22625994111,12cyclictest26-21ksoftirqd/107:30:011
22625994111,11cyclictest26-21ksoftirqd/110:45:001
2262299414,5cyclictest141rcu_preempt10:40:330
2262299412,8cyclictest1753-21gnome-shell12:04:290
507552400,7sleep02197014-21PK-Backend08:25:300
22625994013,14cyclictest26-21ksoftirqd/111:25:161
22625994013,11cyclictest26-21ksoftirqd/110:05:181
22622994029,7cyclictest1069-21runrttasks08:36:380
2262299402,8cyclictest1753-21gnome-shell09:19:140
22622994027,9cyclictest1753-21gnome-shell08:42:220
22622994027,9cyclictest1753-21gnome-shell08:42:210
2262299402,33cyclictest856-21gmain09:08:180
2262299402,33cyclictest856-21gmain09:08:180
2262299402,33cyclictest1753-21gnome-shell09:57:050
22622994023,13cyclictest66692-21sshd09:14:120
2262299402,10cyclictest1753-21gnome-shell10:38:220
2262299401,13cyclictest102213-21grep10:55:020
22627993924,11cyclictest86514-21cron10:10:002
22627993922,12cyclictest86852-21cpuspeed10:10:142
22627993922,12cyclictest438-20systemd-journal11:44:562
2262599396,15cyclictest141rcu_preempt10:25:101
2262599395,12cyclictest26-21ksoftirqd/111:55:091
2262599395,12cyclictest26-21ksoftirqd/111:55:091
22625993923,12cyclictest855-21dbus-daemon12:20:021
22625993912,11cyclictest125917-21cut12:00:251
22625993910,13cyclictest141rcu_preempt07:46:141
2262299392,32cyclictest1753-21gnome-shell08:51:250
22622993921,13cyclictest47723-21pool-org.gnome.08:20:020
22622993920,14cyclictest43131-21df_inode08:05:150
22622993920,14cyclictest123888-21df_inode11:55:180
22622993920,14cyclictest123888-21df_inode11:55:170
2262299391,34cyclictest13-21ksoftirqd/010:00:280
162399390,27rtkit-daemon2197014-21PK-Backend12:31:250
926722380,11chrt92677-21needreboot10:25:242
22627993827,7cyclictest331rcuc/209:26:072
22627993827,7cyclictest331rcuc/209:26:062
22627993826,7cyclictest2197014-21PK-Backend11:26:122
22627993822,11cyclictest131914-21sh12:20:002
22627993821,13cyclictest1-21systemd10:00:012
2262599387,11cyclictest26-21ksoftirqd/111:40:111
2262599387,11cyclictest26-21ksoftirqd/108:40:311
2262599387,11cyclictest26-21ksoftirqd/108:40:311
22625993825,9cyclictest1617-10pipewire-media-07:30:221
22625993822,11cyclictest25680-21sh07:15:181
22625993822,11cyclictest25680-21sh07:15:171
22625993821,12cyclictest438-20systemd-journal09:50:001
22625993820,14cyclictest81357-21pool-org.gnome.09:55:011
22625993820,14cyclictest1-21systemd07:25:001
22625993814,5cyclictest141rcu_preempt08:10:021
22625993814,20cyclictest26-21ksoftirqd/107:55:171
2262599381,23cyclictest52979-21latency_hist08:35:021
22625993811,11cyclictest26-21ksoftirqd/112:24:141
22625993810,15cyclictest1753-21gnome-shell11:20:231
2262299382,6cyclictest1753-21gnome-shell07:13:260
22622993824,10cyclictest76123-21sh09:40:000
22622993821,12cyclictest100384-21cron10:50:000
2262299381,28cyclictest0-21swapper/007:58:150
162399380,31rtkit-daemon0-21swapper/209:35:202
162399380,30rtkit-daemon34-21ksoftirqd/209:05:182
162399380,30rtkit-daemon34-21ksoftirqd/209:05:172
1210362380,5chrt2197014-21PK-Backend11:45:322
2262799376,5cyclictest141rcu_preempt12:02:522
2262799372,31cyclictest1753-21gnome-shell07:36:032
22627993723,10cyclictest34-21ksoftirqd/212:35:202
2262799372,30cyclictest1753-21gnome-shell07:30:072
22627993720,13cyclictest63316-21awk09:05:002
22627993720,12cyclictest95985-21cut10:35:222
22627993719,13cyclictest68974-21gdbus09:20:032
22627993718,14cyclictest438-20systemd-journal10:19:592
2262599378,11cyclictest26-21ksoftirqd/108:00:511
2262599377,17cyclictest141rcu_preempt09:30:271
2262599377,11cyclictest26-21ksoftirqd/111:30:121
2262599377,11cyclictest26-21ksoftirqd/108:10:151
22625993723,10cyclictest123197-21sshd11:54:391
22625993721,12cyclictest24394-21rm07:15:011
22625993721,12cyclictest103852-21cron11:00:011
22625993720,12cyclictest95892-21sort10:35:201
22625993719,14cyclictest118046-21grep11:40:021
22625993719,14cyclictest118046-21grep11:40:021
22625993719,13cyclictest468-21systemd-udevd08:29:501
22625993712,5cyclictest141rcu_preempt12:11:021
22625993710,11cyclictest141rcu_preempt08:59:381
22622993726,7cyclictest0-21swapper/010:30:140
22622993720,13cyclictest49278-21seq08:23:210
22622993720,13cyclictest49278-21seq08:23:210
2262299371,6cyclictest141rcu_preempt08:10:370
22622993715,18cyclictest131321-21python312:15:300
2262299371,17cyclictest355392cat07:44:550
952262361,31sleep295257-21latency_hist10:35:022
2262799366,11cyclictest141rcu_preempt08:40:342
2262799366,11cyclictest141rcu_preempt08:40:332
2262799362,8cyclictest1753-21gnome-shell08:20:002
22627993626,6cyclictest1369772sleep212:33:262
22627993625,7cyclictest3242253-21nautilus11:15:012
22627993623,9cyclictest1753-21gnome-shell07:19:582
22627993623,9cyclictest1753-21gnome-shell07:19:582
2262799362,20cyclictest1-21systemd08:05:012
22627993620,12cyclictest855-21dbus-daemon08:55:002
2262799361,9cyclictest2197014-21PK-Backend07:10:272
22627993619,12cyclictest104853-21ntp_kernel_pll_11:00:232
22627993618,13cyclictest1-21systemd10:55:022
22627993615,5cyclictest28726-21latency_hist07:25:012
2262799361,30cyclictest109587-21cstates11:15:162
22627993610,5cyclictest141rcu_preempt10:00:192
2262599368,11cyclictest107680-21cat11:10:121
2262599367,11cyclictest26-21ksoftirqd/110:49:411
2262599362,31cyclictest1753-21gnome-shell11:09:041
22625993623,10cyclictest26-21ksoftirqd/107:54:161
22625993620,12cyclictest1-21systemd07:40:011
22625993619,12cyclictest127341-21cat12:05:151
22625993613,5cyclictest26-21ksoftirqd/109:35:201
22625993610,6cyclictest141rcu_preempt09:20:061
22625993610,5cyclictest141rcu_preempt10:32:061
22625993610,13cyclictest141rcu_preempt11:17:061
2262299362,7cyclictest1753-21gnome-shell07:28:110
22622993625,7cyclictest116989-21smartctl11:35:200
22622993625,7cyclictest116989-21smartctl11:35:190
22622993624,8cyclictest1753-21gnome-shell12:28:280
22622993622,9cyclictest1753-21gnome-shell11:31:080
22622993622,10cyclictest60248-21tr08:55:160
22622993621,10cyclictest1753-21gnome-shell07:31:380
22622993620,12cyclictest88102-21seq10:13:140
2262299361,9cyclictest2197014-21PK-Backend12:36:510
2262299361,11cyclictest0-21swapper/011:45:110
22627993524,7cyclictest117235-21grep11:35:232
22627993524,7cyclictest117235-21grep11:35:232
2262799352,11cyclictest34-21ksoftirqd/211:58:042
2262799352,11cyclictest34-21ksoftirqd/211:58:042
2262799351,9cyclictest2197014-21PK-Backend11:07:582
2262799351,9cyclictest2197014-21PK-Backend07:59:012
22627993515,6cyclictest53170-21apt08:35:132
22627993514,7cyclictest112001-21needreboot11:20:232
22627993513,6cyclictest141rcu_preempt07:49:052
2262799351,17cyclictest2197014-21PK-Backend09:50:022
2262799351,14cyclictest2197014-21PK-Backend09:42:572
2262599359,5cyclictest141rcu_preempt10:00:411
2262599358,5cyclictest141rcu_preempt08:20:231
2262599358,5cyclictest141rcu_preempt08:20:231
2262599356,11cyclictest26-21ksoftirqd/110:50:041
2262599355,3cyclictest0-21swapper/111:49:441
2262599354,11cyclictest26-21ksoftirqd/109:10:131
22625993510,12cyclictest141rcu_preempt08:35:251
22622993523,8cyclictest92652-21grep10:25:230
22622993523,8cyclictest70191-40apt-check09:21:210
22622993522,9cyclictest13-21ksoftirqd/011:10:160
22622993520,10cyclictest114559-21cron11:30:000
22622993519,12cyclictest36051-21expr07:45:170
22622993519,11cyclictest127959-21cat12:05:310
22622993518,12cyclictest438-20systemd-journal10:08:120
22622993517,13cyclictest107362-21pool-org.gnome.11:10:020
162399350,24rtkit-daemon0-21swapper/010:18:430
162399350,23rtkit-daemon1622-21rtkit-daemon09:13:232
2262799341,8cyclictest58099-21munin-run08:50:002
2262799341,7cyclictest2197014-21PK-Backend12:07:312
22627993417,7cyclictest1753-21gnome-shell11:50:242
2262599349,5cyclictest141rcu_preempt11:00:071
2262599347,5cyclictest141rcu_preempt08:15:171
2262299345,14cyclictest51433-21apt08:30:120
2262299342,5cyclictest0-21swapper/007:20:450
22622993420,9cyclictest1753-21gnome-shell07:37:280
2262299341,5cyclictest141rcu_preempt08:00:180
22622993411,5cyclictest141rcu_preempt09:45:250
162399340,5rtkit-daemon0-21swapper/008:49:450
2262799332,9cyclictest43127-21df08:05:152
22627993322,7cyclictest102379-21/usr/sbin/munin10:55:162
22627993320,10cyclictest49416-21cron08:25:012
22627993320,10cyclictest49416-21cron08:25:002
2262799331,8cyclictest2197014-21PK-Backend10:47:172
2262799331,5cyclictest2197014-21PK-Backend07:50:442
2262799331,20cyclictest134031-21ls12:25:132
2262599334,11cyclictest141rcu_preempt09:15:151
22625993322,7cyclictest89390-21gunzip10:15:301
22622993319,10cyclictest62220-21/usr/sbin/munin09:00:190
22622993318,11cyclictest38796-21seq07:52:190
22622993318,10cyclictest133881-21date12:25:030
2262299331,28cyclictest2197014-21PK-Backend11:44:020
2262299331,28cyclictest2197014-21PK-Backend10:56:300
2262799328,9cyclictest141rcu_preempt10:20:112
2262799327,9cyclictest141rcu_preempt11:32:552
22627993221,7cyclictest1753-21gnome-shell12:10:292
22622993221,7cyclictest71734-21switchtime09:25:300
22622993221,7cyclictest71734-21switchtime09:25:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional