You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 21:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Thu Nov 27, 2025 12:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10208662950,8sleep30-21swapper/307:05:163
11358472920,6sleep30-21swapper/312:27:283
10670762880,5sleep00-21swapper/009:15:130
162399800,8rtkit-daemon1622-21rtkit-daemon09:48:522
162399670,8rtkit-daemon1622-21rtkit-daemon12:03:042
162399660,53rtkit-daemon0-21swapper/207:07:242
102197126644,14sleep10-21swapper/107:07:111
162399640,9rtkit-daemon1622-21rtkit-daemon09:57:583
102203726449,9sleep00-21swapper/007:08:040
162399630,10rtkit-daemon1622-21rtkit-daemon10:02:462
162399630,10rtkit-daemon1622-21rtkit-daemon10:02:462
162399600,9rtkit-daemon1622-21rtkit-daemon11:08:273
10579152600,8sleep32197014-21PK-Backend08:50:003
162399580,9rtkit-daemon1622-21rtkit-daemon08:42:432
162399580,8rtkit-daemon1622-21rtkit-daemon08:13:362
162399580,8rtkit-daemon1622-21rtkit-daemon08:13:352
162399570,9rtkit-daemon1622-21rtkit-daemon07:52:242
162399570,8rtkit-daemon1622-21rtkit-daemon11:53:002
162399570,8rtkit-daemon1622-21rtkit-daemon11:53:002
162399570,8rtkit-daemon1622-21rtkit-daemon10:50:042
162399560,8rtkit-daemon1622-21rtkit-daemon10:58:292
1022349995632,20cyclictest1753-21gnome-shell09:35:253
162399550,9rtkit-daemon1622-21rtkit-daemon09:39:112
102234499554,28cyclictest34-21ksoftirqd/210:30:262
1022344995517,18cyclictest34-21ksoftirqd/211:40:002
1022344995517,18cyclictest34-21ksoftirqd/211:40:002
102234499550,51cyclictest1622-21rtkit-daemon09:06:332
102234499550,51cyclictest1622-21rtkit-daemon07:19:252
162399540,9rtkit-daemon1622-21rtkit-daemon11:19:232
162399540,9rtkit-daemon1622-21rtkit-daemon08:16:252
162399540,8rtkit-daemon1622-21rtkit-daemon10:41:122
162399540,8rtkit-daemon1622-21rtkit-daemon10:41:112
1022349995429,20cyclictest1753-21gnome-shell08:00:123
162399530,9rtkit-daemon1622-21rtkit-daemon11:47:442
162399530,9rtkit-daemon1622-21rtkit-daemon07:13:022
162399530,9rtkit-daemon1622-21rtkit-daemon07:13:022
162399530,8rtkit-daemon1622-21rtkit-daemon12:31:282
162399530,8rtkit-daemon1622-21rtkit-daemon12:17:162
162399530,8rtkit-daemon1622-21rtkit-daemon10:35:172
162399530,8rtkit-daemon1622-21rtkit-daemon09:47:273
162399530,7rtkit-daemon1622-21rtkit-daemon08:31:443
162399530,7rtkit-daemon1622-21rtkit-daemon08:31:433
10894242530,6chrt1753-21gnome-shell10:20:133
10894242530,6chrt1753-21gnome-shell10:20:123
10315982530,7chrt2197014-21PK-Backend07:35:023
10283772530,14sleep31028381-21cpuspeed_turbos07:25:123
102234499530,49cyclictest1622-21rtkit-daemon08:57:042
102234499530,49cyclictest1622-21rtkit-daemon08:57:032
102234499530,49cyclictest1622-21rtkit-daemon08:45:482
11033052520,9sleep126-21ksoftirqd/110:59:551
1022349995229,19cyclictest1753-21gnome-shell08:10:243
1022349995229,19cyclictest1753-21gnome-shell08:10:243
1022349995127,20cyclictest1753-21gnome-shell12:20:263
1022344995113,21cyclictest34-21ksoftirqd/207:37:152
1022344995014,20cyclictest34-21ksoftirqd/212:25:002
102234499501,11cyclictest1099535-21chrt10:46:042
1022349994922,22cyclictest42-21ksoftirqd/307:39:563
102234499491,17cyclictest34-21ksoftirqd/212:00:002
162399480,8rtkit-daemon141rcu_preempt09:53:103
162399480,8rtkit-daemon1029455-21sshd07:27:122
11305292480,5sleep126-21ksoftirqd/112:15:141
102234499483,12cyclictest34-21ksoftirqd/208:30:132
102234499483,12cyclictest34-21ksoftirqd/208:30:122
1022344994816,17cyclictest34-21ksoftirqd/208:10:002
162399470,6rtkit-daemon1622-21rtkit-daemon12:00:363
1022344994718,13cyclictest34-21ksoftirqd/207:46:502
102234499471,19cyclictest860997-21snmpd10:29:302
1022340994720,16cyclictest26-21ksoftirqd/109:24:591
1022344994612,17cyclictest34-21ksoftirqd/207:45:012
1022340994619,23cyclictest1100287-21cpuspeed_turbos10:50:121
1022335994634,9cyclictest13-21ksoftirqd/011:55:020
1022335994634,9cyclictest13-21ksoftirqd/011:55:020
1022335994630,12cyclictest1056145-21sh08:45:000
162399450,38rtkit-daemon856-21gmain08:37:072
11157962450,5chrt0-21swapper/011:35:000
1022344994513,8cyclictest34-21ksoftirqd/209:28:582
1022344994511,17cyclictest34-21ksoftirqd/209:20:012
162399440,37rtkit-daemon2197014-21PK-Backend09:40:262
102234499447,5cyclictest141rcu_preempt12:10:022
102234499447,5cyclictest141rcu_preempt12:10:022
102234499447,20cyclictest34-21ksoftirqd/211:42:182
102234499444,11cyclictest34-21ksoftirqd/210:10:552
1022344994423,8cyclictest34-21ksoftirqd/212:30:022
1022340994424,15cyclictest3242253-21nautilus12:10:021
1022340994424,15cyclictest3242253-21nautilus12:10:011
1022335994423,17cyclictest1052348-21apt-get08:35:110
1022349994320,11cyclictest42-21ksoftirqd/312:10:283
102234499439,19cyclictest34-21ksoftirqd/209:58:042
102234499438,18cyclictest34-21ksoftirqd/208:05:022
1022344994311,18cyclictest34-21ksoftirqd/211:30:212
1022344994311,11cyclictest34-21ksoftirqd/207:24:592
102234099433,16cyclictest1753-21gnome-shell10:01:461
102234099433,16cyclictest1753-21gnome-shell10:01:461
1022340994325,6cyclictest26-21ksoftirqd/107:45:011
1022340994324,14cyclictest26-21ksoftirqd/107:45:131
1022340994321,17cyclictest2962348-21rs:main1
1022335994326,13cyclictest1-21systemd12:15:010
10309802420,7sleep20-21swapper/207:30:312
1022349994227,11cyclictest1088900-21cron10:16:593
1022349994226,11cyclictest438-20systemd-journal08:50:363
102234499428,19cyclictest34-21ksoftirqd/211:07:552
102234499425,18cyclictest34-21ksoftirqd/209:20:152
1022344994221,7cyclictest34-21ksoftirqd/208:53:342
1022340994225,13cyclictest1079553-21latency09:50:211
1022335994226,12cyclictest1128533-21/usr/sbin/munin12:10:010
1022335994226,12cyclictest1128533-21/usr/sbin/munin12:10:000
1022335994225,13cyclictest1099845-21cron10:50:000
162399410,34rtkit-daemon2197014-21PK-Backend11:20:312
102234499417,4cyclictest141rcu_preempt11:00:292
102234499417,14cyclictest34-21ksoftirqd/210:20:192
102234499417,14cyclictest34-21ksoftirqd/210:20:182
102234499415,19cyclictest34-21ksoftirqd/211:26:152
102234499415,19cyclictest34-21ksoftirqd/211:11:532
102234499414,7cyclictest34-21ksoftirqd/208:25:002
1022344994111,12cyclictest2197014-21PK-Backend09:31:132
1022344994110,17cyclictest34-21ksoftirqd/207:55:042
1022340994124,6cyclictest26-21ksoftirqd/107:25:001
1022340994123,6cyclictest26-21ksoftirqd/109:15:201
102234099412,35cyclictest1753-21gnome-shell08:21:141
102234099412,18cyclictest1753-21gnome-shell08:59:091
102234099412,18cyclictest1753-21gnome-shell08:59:081
102234099411,4cyclictest0-21swapper/111:42:181
102234099411,36cyclictest1130175-21sshd12:14:131
1022335994131,7cyclictest1069-21runrttasks07:44:090
1022335994121,15cyclictest1119574-21pool-org.gnome.11:45:020
1022335994111,6cyclictest141rcu_preempt12:20:230
162399400,33rtkit-daemon42-21ksoftirqd/309:18:393
162399400,32rtkit-daemon1022329-21cyclictest08:27:252
102234999404,5cyclictest10384652munin-plugin-st07:55:023
1022349994029,7cyclictest10962112sleep310:37:523
102234999402,34cyclictest1753-21gnome-shell11:12:533
1022349994022,13cyclictest1905-21gdbus08:35:043
1022349994022,13cyclictest1093244-40apt-check10:30:183
1022349994011,25cyclictest42-21ksoftirqd/310:27:093
102234099406,7cyclictest10548642expr08:40:141
1022340994025,11cyclictest1078739-21sh09:49:591
1022340994024,12cyclictest1113804-21sshd11:27:121
1022340994023,6cyclictest26-21ksoftirqd/111:03:171
1022340994023,12cyclictest438-20systemd-journal09:05:121
1022340994023,12cyclictest1073969-21grep09:35:111
1022340994022,6cyclictest26-21ksoftirqd/108:09:261
1022340994022,14cyclictest1134928-21df_inode12:25:141
102234099401,15cyclictest11239512sh11:55:201
1022340994011,18cyclictest26-21ksoftirqd/108:15:081
1022335994027,9cyclictest1617-10pipewire-media-10:30:130
1022335994025,11cyclictest54-21kcompactd008:59:470
1022335994025,11cyclictest54-21kcompactd008:59:470
1022335994022,13cyclictest438-20systemd-journal07:35:000
1022335994021,15cyclictest1058991-21processes08:50:290
1022335994017,19cyclictest1-21systemd10:40:000
10293562390,14chrt26-21ksoftirqd/107:25:551
102234999395,12cyclictest2197014-21PK-Backend10:06:173
1022349993924,11cyclictest42-21ksoftirqd/310:00:153
1022349993924,11cyclictest42-21ksoftirqd/310:00:153
1022349993922,13cyclictest1046809-21pool-org.gnome.08:16:063
1022349993921,13cyclictest438-20systemd-journal07:25:003
102234499398,15cyclictest34-21ksoftirqd/209:10:322
102234499396,17cyclictest34-21ksoftirqd/209:54:062
102234499394,18cyclictest34-21ksoftirqd/212:15:012
102234499393,8cyclictest34-21ksoftirqd/212:35:132
1022344993919,8cyclictest34-21ksoftirqd/210:16:192
102234099396,11cyclictest26-21ksoftirqd/108:50:231
1022340993922,6cyclictest26-21ksoftirqd/109:55:311
1022340993922,13cyclictest1-21systemd10:15:011
1022340993922,12cyclictest1052350-21munin-run08:35:011
1022340993922,12cyclictest1052350-21munin-run08:35:011
1022340993916,7cyclictest1054032-40apt-check08:36:191
1022335993927,9cyclictest13-21ksoftirqd/010:30:010
1022335993923,11cyclictest1085348-21sshd10:05:560
1022335993923,11cyclictest1040599-21grep08:00:100
1022335993922,12cyclictest438-20systemd-journal09:42:120
1022335993922,12cyclictest1136737-21fw_packets12:30:180
1022335993922,12cyclictest1073030-21threads09:30:320
1022335993921,13cyclictest1088400-21ntpq10:15:250
102233599391,7cyclictest1121290-21turbostat11:50:000
1022335993912,11cyclictest141rcu_preempt10:10:220
102234999382,8cyclictest1753-21gnome-shell10:45:593
1022349993815,6cyclictest141rcu_preempt12:15:213
1022349993813,21cyclictest1753-21gnome-shell09:30:583
1022349993812,6cyclictest141rcu_preempt11:55:043
102234999381,17cyclictest1115033-21awk11:30:233
1022344993830,4cyclictest331rcuc/210:08:282
1022344993822,12cyclictest1062485-21sensors_temp09:00:272
1022340993821,5cyclictest26-21ksoftirqd/111:20:011
1022340993820,6cyclictest26-21ksoftirqd/108:25:271
1022340993818,15cyclictest1800-21upowerd07:37:141
1022340993815,6cyclictest1044306-21/usr/sbin/munin08:10:171
1022340993815,6cyclictest1044306-21/usr/sbin/munin08:10:171
1022340993814,20cyclictest1097627-21expr10:40:301
1022340993814,20cyclictest1097627-21expr10:40:301
102233599386,9cyclictest13-21ksoftirqd/010:40:340
102233599386,9cyclictest13-21ksoftirqd/010:40:340
102233599384,11cyclictest2197014-21PK-Backend10:50:200
102233599382,7cyclictest1753-21gnome-shell09:06:090
1022335993821,12cyclictest1113650-21uptime11:25:330
1022335993821,12cyclictest1108138-21cat11:10:310
1022335993821,12cyclictest1079766-21ntp_states09:50:260
1022335993820,14cyclictest1632-21dbus-daemon07:37:130
1022335993820,13cyclictest1130832-21cat12:15:160
1022335993820,13cyclictest1126099-21pluginstate12:00:260
1022335993820,13cyclictest1069196-21memory09:20:230
1022335993819,14cyclictest1103014-21sshd10:56:110
1022335993810,5cyclictest141rcu_preempt07:55:400
162399370,30rtkit-daemon42-21ksoftirqd/310:44:273
162399370,30rtkit-daemon42-21ksoftirqd/310:44:263
11229972371,31sleep33242253-21nautilus11:55:003
11229972371,31sleep33242253-21nautilus11:55:003
102234999375,15cyclictest141rcu_preempt08:20:253
1022349993723,11cyclictest42-21ksoftirqd/312:05:243
1022349993723,11cyclictest42-21ksoftirqd/312:05:243
1022349993722,12cyclictest42-21ksoftirqd/307:40:323
102234999372,11cyclictest141rcu_preempt08:40:263
1022349993721,12cyclictest1039600-21python307:55:283
1022349993719,14cyclictest1119122-21users11:40:313
1022349993712,6cyclictest141rcu_preempt11:00:283
102234099372,8cyclictest1753-21gnome-shell10:08:481
1022340993726,7cyclictest1069-21runrttasks11:47:111
1022340993726,7cyclictest1030462-21kernelversion07:30:211
1022340993725,8cyclictest1136518-21tr12:30:131
1022340993722,11cyclictest1134258-21apt-config12:25:021
1022340993720,6cyclictest26-21ksoftirqd/111:35:151
1022340993720,6cyclictest26-21ksoftirqd/111:35:151
1022340993719,14cyclictest3242254-21gmain08:50:011
1022340993719,14cyclictest1095292-21df10:35:151
102234099371,28cyclictest0-21swapper/107:59:201
1022340993710,23cyclictest1115751-21anacron11:34:561
102233599372,32cyclictest0-21swapper/012:35:590
102233599372,31cyclictest1753-21gnome-shell11:38:090
102233599372,31cyclictest1753-21gnome-shell11:38:090
1022335993721,12cyclictest855-21dbus-daemon08:30:000
1022335993721,12cyclictest1025785-21sort07:15:310
1022335993721,12cyclictest1024024-21systemd-run07:15:010
1022335993721,12cyclictest1024024-21systemd-run07:15:010
102233599371,9cyclictest2197014-21PK-Backend09:45:270
102233599371,9cyclictest2197014-21PK-Backend07:50:200
1022335993718,15cyclictest1083232-21ntp_states10:00:260
1022335993718,15cyclictest1083232-21ntp_states10:00:260
102234999362,30cyclictest1753-21gnome-shell09:29:153
1022349993622,10cyclictest1025970-21cron07:17:003
1022349993619,12cyclictest1117509-21apt-get11:40:013
1022349993619,12cyclictest1117509-21apt-get11:40:013
1022349993614,10cyclictest141rcu_preempt12:30:273
102234999361,31cyclictest1753-21gnome-shell09:05:173
102234999361,14cyclictest1086822-21python310:10:303
102234099362,29cyclictest1111176-21diskstats11:20:181
1022340993622,10cyclictest1075549-21sshd09:40:121
1022340993621,11cyclictest1738-21gdbus09:11:561
1022340993620,12cyclictest1105318-21basename11:05:111
1022340993619,13cyclictest1098114-21cron10:49:551
1022340993619,13cyclictest1037718-21ntpq07:50:271
1022340993619,12cyclictest438-20systemd-journal09:02:121
102234099361,13cyclictest104110750journal-offline08:00:201
102233599367,11cyclictest141rcu_preempt08:24:530
102233599362,7cyclictest1081125-21diskstats09:55:160
1022335993624,8cyclictest1062966-21taskset09:03:400
102233599362,31cyclictest0-21swapper/008:15:490
1022335993620,12cyclictest1111949-21nice11:21:130
1022335993620,12cyclictest1065485-21fw_packets09:10:180
1022335993620,11cyclictest1109508-21ntp_kernel_err11:15:250
102233599361,9cyclictest0-21swapper/012:27:390
1022335993619,12cyclictest1124316-21open_inodes11:55:270
1022335993618,13cyclictest1090142-21ntpq10:20:240
1022335993618,13cyclictest1090142-21ntpq10:20:240
1022335993617,14cyclictest1051606-21ntpq08:30:240
1022335993617,14cyclictest1051606-21ntpq08:30:240
102234999358,6cyclictest141rcu_preempt07:10:183
102234999358,6cyclictest141rcu_preempt07:10:173
102234999358,10cyclictest1617-10pipewire-media-08:25:133
102234999357,5cyclictest141rcu_preempt11:20:143
102234999352,16cyclictest141rcu_preempt08:05:213
1022349993521,10cyclictest42-21ksoftirqd/311:15:133
102234099352,16cyclictest1091151-21apt10:25:101
1022340993520,11cyclictest1025042-21cat07:15:151
102234099351,9cyclictest1087836-21cpuspeed_turbos10:15:151
1022340993519,6cyclictest26-21ksoftirqd/110:23:551
1022340993519,6cyclictest26-21ksoftirqd/110:23:551
102233599353,5cyclictest141rcu_preempt08:05:210
102233599352,30cyclictest0-21swapper/009:26:480
1022335993522,9cyclictest1753-21gnome-shell11:10:020
1022335993519,12cyclictest855-21dbus-daemon07:21:560
1022335993519,11cyclictest1044637-21ntp_kernel_err08:10:230
1022335993519,11cyclictest1044637-21ntp_kernel_err08:10:230
102234999349,9cyclictest141rcu_preempt12:36:163
102234999348,6cyclictest141rcu_preempt11:00:023
102234999348,22cyclictest1632-21dbus-daemon10:55:013
1022349993422,8cyclictest1753-21gnome-shell09:11:353
1022349993421,9cyclictest1753-21gnome-shell11:25:373
1022349993418,12cyclictest1069694-21users09:20:323
102234999341,30cyclictest2197014-21PK-Backend09:44:043
102234999341,16cyclictest2197014-21PK-Backend08:55:153
102234999341,16cyclictest2197014-21PK-Backend08:55:153
1022340993423,7cyclictest1107807-21ntp_states11:10:251
1022340993423,7cyclictest1069-21runrttasks12:35:111
1022340993420,10cyclictest1023444-21sed07:10:301
1022340993420,10cyclictest1023444-21sed07:10:301
102234099341,9cyclictest1070347-21apt09:25:121
1022340993418,12cyclictest1121548-21apt11:50:121
1022340993418,12cyclictest1121548-21apt11:50:121
1022335993422,7cyclictest1753-21gnome-shell08:45:070
1022335993419,11cyclictest1074457-21cat09:35:200
102233599341,5cyclictest141rcu_preempt07:45:240
1022349993318,11cyclictest1036756-21pluginstate07:50:013
1022349993316,13cyclictest1063115-21pool-org.gnome.09:05:003
102234999331,29cyclictest2197014-21PK-Backend11:49:103
102234099332,15cyclictest1093244-31apt-check10:30:121
102234099331,29cyclictest0-21swapper/112:04:301
102233599334,9cyclictest2197014-21PK-Backend07:25:120
102233599332,5cyclictest141rcu_preempt11:00:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional