You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 12:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Nov 20, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
95507521040,5sleep0943366-21kworker/0:120:20:330
162399880,8rtkit-daemon1622-21rtkit-daemon19:26:330
939157998324,42cyclictest855-21dbus-daemon21:21:562
939161998115,40cyclictest1031191-21/usr/sbin/munin00:00:313
162399760,45rtkit-daemon1753-21gnome-shell00:04:091
939157997422,37cyclictest1753-21gnome-shell20:34:442
93893927454,13sleep00-21swapper/019:09:130
939161997315,38cyclictest1069-21runrttasks20:08:113
93915799731,46cyclictest980255-21taskset21:35:092
93916199721,46cyclictest941212-21taskset19:22:103
93916199721,46cyclictest941212-21taskset19:22:103
93916199711,37cyclictest1005662-21/usr/sbin/munin22:45:213
93916199701,45cyclictest1045012-21taskset00:38:413
93916199701,44cyclictest1022065-21taskset23:32:483
93915799702,49cyclictest1753-21gnome-shell23:51:352
93890127038,26sleep20-21swapper/219:08:432
93915799691,36cyclictest1004694-21taskset22:42:252
93899026947,15sleep10-21swapper/119:09:541
93916199672,46cyclictest1753-21gnome-shell22:15:583
93916199671,46cyclictest966377-21taskset20:55:033
93916199671,38cyclictest1008056-21seq22:51:093
93916199671,38cyclictest1008056-21seq22:51:093
93915799672,45cyclictest1753-21gnome-shell00:30:372
93915799671,49cyclictest994420-40apt-check22:15:192
93893726739,22sleep30-21swapper/319:09:123
93916199661,37cyclictest951665-21seq20:12:593
93916199652,42cyclictest1753-21gnome-shell23:08:553
93916199652,40cyclictest1753-21gnome-shell00:19:573
93916199651,42cyclictest1753-21gnome-shell23:39:373
93916199651,39cyclictest941698-21taskset19:25:243
93916199642,44cyclictest1753-21gnome-shell19:17:553
93916199642,43cyclictest1753-21gnome-shell21:17:433
93916199642,42cyclictest1753-21gnome-shell20:28:433
93916199642,42cyclictest1753-21gnome-shell20:28:433
93916199641,44cyclictest997417-21taskset22:21:563
93915799644,38cyclictest1044159-21kernelversion00:35:222
939157996426,33cyclictest997666-21apt-get22:25:012
93916199632,36cyclictest965817-21colord-sane20:51:203
93915799631,35cyclictest1021664-21/usr/sbin/munin23:30:302
93915799631,34cyclictest1031795-21load00:00:222
93916199621,38cyclictest1011491-21seq23:00:433
93916199621,36cyclictest1004816-21seq22:43:593
93915799621,50cyclictest175350gnome-shell23:15:162
93915799621,41cyclictest2197014-21PK-Backend20:25:222
93915799621,41cyclictest2197014-21PK-Backend20:25:222
939151996221,36cyclictest986804-21sshd21:53:210
93916199612,36cyclictest982378-21/usr/sbin/munin21:40:133
93916199612,32cyclictest1753-21gnome-shell00:23:063
93916199611,36cyclictest1041008-21seq00:25:433
93916199611,35cyclictest1034191-21seq00:07:243
93915799612,41cyclictest971980-21/usr/sbin/munin21:10:172
939157996122,34cyclictest1007716-21wc22:50:272
939157996122,34cyclictest1007716-21wc22:50:272
93915799611,41cyclictest2197014-21PK-Backend00:15:032
93915799611,36cyclictest978520-21taskset21:30:092
93915799611,33cyclictest983862-21/usr/sbin/munin21:45:112
93916199601,35cyclictest945423-21taskset19:52:493
93916199601,31cyclictest1024217-21taskset23:40:053
93916199601,30cyclictest1003070-21taskset22:38:503
93916199601,30cyclictest1003070-21taskset22:38:503
93915799602,41cyclictest1753-21gnome-shell20:44:302
93915799602,31cyclictest1753-21gnome-shell19:19:112
162399600,9rtkit-daemon1622-21rtkit-daemon20:00:011
939161995910,32cyclictest0-21swapper/321:10:283
93915799592,33cyclictest1753-21gnome-shell23:35:262
93915799591,40cyclictest860997-21snmpd00:14:322
93915799591,35cyclictest990085-21taskset22:01:002
93915599592,37cyclictest3242253-21nautilus19:45:011
93915199592,7cyclictest1753-21gnome-shell23:32:060
93916199582,31cyclictest1753-21gnome-shell22:10:173
93916199581,34cyclictest1030838-21seq23:59:113
93916199581,34cyclictest1028861-21taskset23:51:053
93916199581,32cyclictest984884-21taskset21:45:583
93916199581,31cyclictest1-21systemd20:05:013
93916199581,29cyclictest1026943-21/usr/sbin/munin23:45:353
93915799581,40cyclictest860997-21snmpd23:59:162
93915199580,53cyclictest1622-21rtkit-daemon19:20:180
93915199580,53cyclictest1622-21rtkit-daemon19:20:180
93916199571,33cyclictest960422-21timerwakeupswit20:35:313
93916199571,32cyclictest979661-21taskset21:30:413
939161995710,25cyclictest943243-21taskset19:39:083
939157995720,32cyclictest977692-21python321:25:312
93915799571,35cyclictest967620-21taskset20:56:522
93915799571,33cyclictest998749-21latency22:25:202
93915799571,29cyclictest2197014-21PK-Backend21:59:582
93915799571,28cyclictest1753-21gnome-shell23:10:012
93915799571,27cyclictest2197014-21PK-Backend20:01:172
93916199566,26cyclictest0-21swapper/319:45:513
93916199562,30cyclictest860997-21snmpd20:30:143
93915799563,48cyclictest1026132-21cpuspeed_turbos23:45:192
939157995618,33cyclictest973645-21expr21:15:142
93915199562,48cyclictest1753-21gnome-shell20:55:240
9981752550,9chrt1753-21gnome-shell22:25:101
939161995522,28cyclictest1043001-21cat00:30:363
93916199551,31cyclictest953607-21taskset20:16:433
93916199551,31cyclictest860997-21snmpd21:53:223
93916199551,29cyclictest2197014-21PK-Backend00:14:583
93916199551,28cyclictest1069-21runrttasks19:31:323
939157995518,32cyclictest965015-21df_inode20:50:182
93915799551,33cyclictest1069-21runrttasks20:13:012
93915799551,30cyclictest2197014-21PK-Backend22:10:142
93915799551,27cyclictest2197014-21PK-Backend19:44:392
93915799551,27cyclictest1753-21gnome-shell19:38:332
93916199541,27cyclictest2197014-21PK-Backend19:10:053
939157995416,33cyclictest970838-21wc21:05:302
939157995414,35cyclictest982673-21tr21:40:212
93915799541,28cyclictest2197014-21PK-Backend19:11:232
939151995420,29cyclictest995070-21cat22:15:240
939161995315,34cyclictest855-21dbus-daemon20:45:013
93916199531,29cyclictest969450-21taskset21:03:083
93916199531,29cyclictest1001078-21taskset22:30:343
93915799532,46cyclictest1753-21gnome-shell22:40:022
93915799532,46cyclictest1753-21gnome-shell22:40:022
93915799531,49cyclictest0-21swapper/223:45:012
93915799531,30cyclictest2197014-21PK-Backend20:24:552
93915799531,28cyclictest2197014-21PK-Backend23:13:532
939157995310,38cyclictest1753-21gnome-shell22:56:562
939157995310,38cyclictest1753-21gnome-shell22:56:562
93916199521,29cyclictest981575-21taskset21:37:573
93915799522,45cyclictest1753-21gnome-shell19:49:512
939157995217,30cyclictest991441-21ntpq22:05:252
939157995216,31cyclictest963011-21tr20:45:132
939157995216,31cyclictest963011-21tr20:45:132
93915799521,27cyclictest2197014-21PK-Backend00:20:592
93915799521,27cyclictest0-21swapper/200:09:552
162399520,8rtkit-daemon1622-21rtkit-daemon20:00:441
93915799511,27cyclictest0-21swapper/222:45:592
93915799511,26cyclictest0-21swapper/219:23:192
93915799511,26cyclictest0-21swapper/219:23:192
93916199501,45cyclictest1753-21gnome-shell22:04:303
93915799501,29cyclictest0-21swapper/200:26:112
93915799501,28cyclictest0-21swapper/219:25:122
93915799501,27cyclictest0-21swapper/222:33:312
93915199502,7cyclictest1753-21gnome-shell23:45:010
93916199493,41cyclictest1069-21runrttasks20:20:033
93916199492,42cyclictest1753-21gnome-shell22:05:563
93916199492,42cyclictest1753-21gnome-shell21:57:513
93916199492,42cyclictest1753-21gnome-shell21:27:453
93915799492,42cyclictest1019024-21/usr/sbin/munin23:25:322
93915799491,44cyclictest1010770-21taskset23:00:182
93915799491,26cyclictest0-21swapper/219:55:132
939157994911,33cyclictest968391-21expr21:00:152
93915199499,36cyclictest1753-21gnome-shell22:43:050
93915199498,36cyclictest1753-21gnome-shell21:45:390
93915199495,18cyclictest964107-21systemd-udevd20:47:120
93915199495,18cyclictest964107-21systemd-udevd20:47:120
939151994920,24cyclictest1017495-21grep23:20:120
939151994910,35cyclictest1753-21gnome-shell21:33:540
9530802480,9sleep11753-21gnome-shell20:15:261
93916199489,34cyclictest1753-21gnome-shell21:05:293
93916199482,41cyclictest1753-21gnome-shell20:49:103
93916199482,41cyclictest1753-21gnome-shell20:49:103
939161994813,31cyclictest999488-21package-system-22:27:273
93915799481,29cyclictest1069-21runrttasks20:05:382
939157994810,33cyclictest959582-21expr20:35:152
93915199489,35cyclictest1753-21gnome-shell22:50:570
93915199489,35cyclictest1753-21gnome-shell22:50:570
93915199488,35cyclictest1753-21gnome-shell20:26:380
93915199488,35cyclictest1753-21gnome-shell20:26:380
93915199481,43cyclictest1753-21gnome-shell22:13:180
93916199472,40cyclictest1014325-21irqstats23:10:203
93916199472,29cyclictest1753-21gnome-shell22:56:123
93916199472,29cyclictest1753-21gnome-shell22:56:113
939161994712,30cyclictest1020060-21cat23:25:333
939157994716,26cyclictest34-21ksoftirqd/220:15:172
939157994714,29cyclictest1069-21runrttasks21:50:592
939157994710,33cyclictest1753-21gnome-shell19:33:192
939155994723,20cyclictest1027482-21unin-run23:50:011
93915199479,34cyclictest1753-21gnome-shell00:27:410
93915199478,34cyclictest1753-21gnome-shell19:51:120
93915199475,38cyclictest1033683-21tr00:05:250
939151994716,27cyclictest1020543-21unin-run23:30:010
939151994716,27cyclictest1017095-21cut23:20:020
93915199471,42cyclictest1753-21gnome-shell00:12:300
93915199471,41cyclictest1753-21gnome-shell21:37:450
93915199471,41cyclictest1753-21gnome-shell20:30:290
939151994713,30cyclictest943022-21seq19:36:120
93916199462,39cyclictest1753-21gnome-shell19:42:503
939157994615,27cyclictest945614-21latency_hist19:55:012
939157994613,28cyclictest1018780-21latency_hist23:25:012
93915599461,8cyclictest1753-21gnome-shell23:21:431
93915199468,34cyclictest1753-21gnome-shell20:09:300
93915199468,34cyclictest1753-21gnome-shell00:35:410
93915199466,16cyclictest1753-21gnome-shell22:45:070
93915199466,16cyclictest1753-21gnome-shell22:38:580
93915199466,16cyclictest1753-21gnome-shell22:38:580
939151994625,17cyclictest0-21swapper/020:03:120
939151994619,23cyclictest1069-21runrttasks21:14:460
939151994619,22cyclictest1069-21runrttasks23:57:010
939151994615,27cyclictest998268-21cpuspeed22:25:120
939151994615,27cyclictest992140-21cron22:10:000
93915199461,40cyclictest1753-21gnome-shell23:50:200
939151994613,28cyclictest939349-21sshd19:10:370
939151994610,31cyclictest974659-21seq21:18:150
10419792460,7sleep02197014-21PK-Backend00:30:180
93915199459,32cyclictest1753-21gnome-shell20:38:220
93915199458,32cyclictest989223-21expr22:00:160
93915199458,32cyclictest1753-21gnome-shell22:31:090
93915199457,34cyclictest1753-21gnome-shell21:20:280
93915199452,38cyclictest1031741-21wc00:00:210
93915199451,40cyclictest1753-21gnome-shell21:41:400
939151994512,29cyclictest1037466-21timerwakeupswit00:15:310
939151994512,28cyclictest953158-21ntp_states20:15:290
93916199442,38cyclictest976164-21users21:20:303
939161994419,20cyclictest1753-21gnome-shell19:55:113
939161994414,26cyclictest1018780-21latency_hist23:25:003
93915199448,31cyclictest1753-21gnome-shell23:10:420
93915199444,16cyclictest1753-21gnome-shell21:04:020
939151994415,25cyclictest997705-21sh22:25:010
939151994414,26cyclictest1010563-21tr23:00:140
93915199441,39cyclictest1753-21gnome-shell20:12:430
93915199441,38cyclictest1753-21gnome-shell20:50:300
939151994413,26cyclictest1069-21runrttasks21:30:030
93915199438,29cyclictest1753-21gnome-shell19:45:230
93915199431,4cyclictest2197014-21PK-Backend23:45:130
939151994314,24cyclictest940289-21sort19:15:060
162399430,7rtkit-daemon1753-21gnome-shell21:37:041
939151994216,22cyclictest1069-21runrttasks19:41:330
93915199421,36cyclictest1753-21gnome-shell19:57:030
939151994213,25cyclictest987995-21open_inodes21:55:270
939161994118,19cyclictest2197014-21PK-Backend23:15:273
939155994123,13cyclictest961518-21/usr/sbin/munin20:40:191
93915199415,16cyclictest1753-21gnome-shell19:31:300
93915199412,14cyclictest1022893-21diskstats23:35:160
93915199411,15cyclictest1038878-21awk00:20:240
162399410,5rtkit-daemon0-21swapper/120:36:411
93915199403,8cyclictest1753-21gnome-shell23:00:020
93915199403,8cyclictest1753-21gnome-shell23:00:020
162399400,6rtkit-daemon2197014-21PK-Backend23:58:491
939155993923,11cyclictest1041392-21systemctl00:29:591
939155993922,12cyclictest438-20systemd-journal22:10:211
939155993918,16cyclictest1003423-21/usr/sbin/munin22:40:281
93915599385,29cyclictest1069-21runrttasks00:13:441
939155993826,8cyclictest1029178-21sh23:55:001
939155993821,12cyclictest438-20systemd-journal21:23:201
93915199382,13cyclictest961111-21chrt20:40:120
93915199381,8cyclictest0-21swapper/023:06:510
9514902378,25sleep11753-21gnome-shell20:10:481
93915599372,31cyclictest1753-21gnome-shell00:36:281
939155993720,13cyclictest1012985-21python323:05:291
93915199371,10cyclictest970983-21seq21:05:350
162399370,7rtkit-daemon2197014-21PK-Backend20:27:091
162399370,7rtkit-daemon2197014-21PK-Backend20:27:091
162399370,26rtkit-daemon0-21swapper/123:01:121
93915599365,27cyclictest1042052-21tr00:30:181
939155993622,10cyclictest1025708-21sh23:45:001
939155993621,11cyclictest1022832-21expr23:35:131
93915599362,10cyclictest987619-21hddtemp_smartct21:55:191
939155993617,14cyclictest3242253-21nautilus22:25:011
93915599359,21cyclictest1-21systemd19:15:021
939155993524,7cyclictest1753-21gnome-shell23:10:081
93915599352,28cyclictest1753-21gnome-shell19:30:091
939155993520,11cyclictest992154-21awk22:10:001
939155993520,11cyclictest977413-21missed_timers21:25:251
93915599351,17cyclictest1753-21gnome-shell21:00:151
93915599351,15cyclictest2197014-21PK-Backend22:02:511
162399350,22rtkit-daemon9405692sleep119:16:591
93915599348,22cyclictest1016296-21ntp_kernel_pll_23:15:251
939155993427,4cyclictest2197014-21PK-Backend19:39:231
939155993424,6cyclictest1008376-21munin-run22:55:001
939155993424,6cyclictest1008376-21munin-run22:54:591
939155993420,10cyclictest2631797-21snapd00:05:181
939155993419,11cyclictest974676-21seq21:18:271
939155993419,10cyclictest26-21ksoftirqd/119:46:191
939155993418,12cyclictest1036560-21cut00:15:141
939155993417,12cyclictest964107-21systemd-udevd20:47:111
939155993417,12cyclictest964107-21systemd-udevd20:47:111
93915599341,30cyclictest2197014-21PK-Backend20:08:281
10212842340,6sleep1231irq_work/123:30:221
93915599331,29cyclictest0-21swapper/123:29:321
93915599331,29cyclictest0-21swapper/120:58:481
939155993225,4cyclictest2197014-21PK-Backend19:24:031
939155993225,4cyclictest2197014-21PK-Backend19:24:031
939155993219,9cyclictest26-21ksoftirqd/120:53:371
939155993219,9cyclictest1753-21gnome-shell21:50:291
939155993217,11cyclictest958856-21sshd20:32:191
939155993217,11cyclictest1009815-21sshd22:56:231
939155993217,11cyclictest1009815-21sshd22:56:231
939155993217,11cyclictest1000585-21cat22:30:231
93915599321,28cyclictest0-21swapper/121:11:571
939155993212,16cyclictest3242253-21nautilus22:20:001
939155993124,5cyclictest0-21swapper/122:36:041
939155993124,5cyclictest0-21swapper/122:36:031
939155993124,5cyclictest0-21swapper/121:48:321
93915599312,25cyclictest1039635-21munin-run00:25:001
939155993116,11cyclictest2962348-21rs:main1
93915599311,27cyclictest0-21swapper/121:43:131
939155993019,7cyclictest1069-21runrttasks21:05:311
939155993016,10cyclictest945718-21cut19:55:011
939155992919,7cyclictest2197014-21PK-Backend21:31:531
939155992912,12cyclictest438-20systemd-journal20:25:001
939155992813,12cyclictest707415-21gmain22:47:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional