You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-12 - 05:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Nov 12, 2025 00:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
55381722330,10sleep00-21swapper/019:05:140
1623991120,8rtkit-daemon1622-21rtkit-daemon23:23:023
1623991000,9rtkit-daemon1622-21rtkit-daemon21:51:393
5622562920,5sleep02197014-21PK-Backend19:25:300
55348627953,16sleep10-21swapper/119:05:121
55468227762,9sleep20-21swapper/219:06:292
162399770,6rtkit-daemon1622-21rtkit-daemon19:21:493
55508799762,52cyclictest1753-21gnome-shell21:15:171
162399750,6rtkit-daemon2197014-21PK-Backend20:24:500
162399740,5rtkit-daemon0-21swapper/000:04:130
6656972730,7sleep02197014-21PK-Backend00:20:160
5779012730,5sleep20-21swapper/220:10:312
5779012730,5sleep20-21swapper/220:10:312
162399730,6rtkit-daemon2197014-21PK-Backend22:45:250
162399730,6rtkit-daemon2197014-21PK-Backend22:45:250
162399730,6rtkit-daemon2197014-21PK-Backend22:26:140
55483827248,16sleep30-21swapper/319:08:273
162399720,6rtkit-daemon2197014-21PK-Backend22:20:210
162399720,5rtkit-daemon0-21swapper/022:12:190
162399720,5rtkit-daemon0-21swapper/021:32:500
162399720,5rtkit-daemon0-21swapper/021:26:250
162399710,6rtkit-daemon2197014-21PK-Backend21:08:530
162399710,6rtkit-daemon2197014-21PK-Backend20:18:030
555087996817,27cyclictest641336-21if_enp2s023:10:201
555087996817,27cyclictest641336-21if_enp2s023:10:191
555089996715,26cyclictest1069-21runrttasks22:16:362
55508999652,36cyclictest1753-21gnome-shell22:39:462
555087996520,23cyclictest26-21ksoftirqd/121:56:401
55508799642,31cyclictest1753-21gnome-shell22:35:451
55508799632,33cyclictest1753-21gnome-shell19:17:261
55508799632,33cyclictest1753-21gnome-shell19:17:261
555087996314,25cyclictest0-21swapper/122:55:271
55508799631,39cyclictest1753-21gnome-shell00:37:501
55508999622,35cyclictest1753-21gnome-shell20:04:232
55508999622,35cyclictest1753-21gnome-shell20:04:232
55508999621,39cyclictest665141-21taskset00:18:252
555087996216,41cyclictest611569-21sh21:45:271
5866802610,5chrt0-21swapper/020:35:290
55508999612,31cyclictest1753-21gnome-shell22:54:562
55508999611,35cyclictest615758-21taskset21:57:022
55508799612,35cyclictest1753-21gnome-shell23:59:531
55508799612,33cyclictest1753-21gnome-shell19:20:051
55508799612,32cyclictest1753-21gnome-shell21:04:531
55508799612,32cyclictest1753-21gnome-shell21:04:531
162399610,8rtkit-daemon1622-21rtkit-daemon21:52:422
55508999602,35cyclictest573163-21taskset19:59:382
55508999602,33cyclictest1753-21gnome-shell22:45:002
55508999601,32cyclictest649213-21sshd23:33:302
55508799602,32cyclictest1753-21gnome-shell23:38:591
55508799602,31cyclictest1753-21gnome-shell19:13:051
55508799602,31cyclictest1753-21gnome-shell19:13:051
55508999592,33cyclictest607070-21cron21:35:002
55508799592,34cyclictest1753-21gnome-shell23:00:401
55508799592,31cyclictest1753-21gnome-shell23:51:371
55508799592,31cyclictest1753-21gnome-shell23:15:431
55508799592,31cyclictest1753-21gnome-shell22:22:221
55508799592,30cyclictest1753-21gnome-shell19:27:071
55508799591,33cyclictest563266-21cpuspeed_turbos19:30:141
162399590,8rtkit-daemon1622-21rtkit-daemon00:37:202
55508799582,31cyclictest1753-21gnome-shell23:47:341
162399580,9rtkit-daemon1622-21rtkit-daemon23:29:542
162399580,8rtkit-daemon1622-21rtkit-daemon19:37:252
162399580,8rtkit-daemon1622-21rtkit-daemon19:37:242
55508999571,34cyclictest860997-21snmpd20:42:412
55508999571,34cyclictest860997-21snmpd20:42:402
55508799572,32cyclictest1753-21gnome-shell21:32:391
55508799572,30cyclictest1753-21gnome-shell20:09:451
55508799572,30cyclictest1753-21gnome-shell00:13:181
55508799572,30cyclictest1753-21gnome-shell00:13:181
55508799572,29cyclictest1753-21gnome-shell00:32:561
55508999562,30cyclictest1753-21gnome-shell21:28:132
555089995620,31cyclictest576853-21date20:10:022
55508999561,30cyclictest587401-21latency_hist20:40:022
555087995625,26cyclictest603296-21apt-get21:25:011
55508799562,33cyclictest634447-21ntp_states22:50:261
5700972550,7chrt2197014-21PK-Backend19:50:130
55508999552,28cyclictest662837-21ntp_states00:10:272
55508999552,28cyclictest662837-21ntp_states00:10:262
55508999551,30cyclictest609074-21awk21:40:132
55508799552,29cyclictest1753-21gnome-shell22:16:291
55509399540,51cyclictest1622-21rtkit-daemon23:11:143
55509399540,51cyclictest1622-21rtkit-daemon23:11:143
55508799542,32cyclictest1753-21gnome-shell20:42:541
55508799542,32cyclictest1753-21gnome-shell20:42:541
555087995419,30cyclictest646728-21load23:25:251
55508999532,29cyclictest1753-21gnome-shell23:23:002
55509399525,9cyclictest42-21ksoftirqd/322:15:013
55509399520,48cyclictest1622-21rtkit-daemon19:43:123
55508999524,42cyclictest585185-21wc20:30:312
55508999523,45cyclictest619522-21latency_hist22:10:012
555089995219,28cyclictest557942-21basename19:15:222
555089995219,28cyclictest557942-21basename19:15:212
555089995218,29cyclictest642440-21pool-org.gnome.23:15:022
555089995218,29cyclictest642440-21pool-org.gnome.23:15:022
555087995212,23cyclictest565120-21expr19:35:191
555087995212,23cyclictest565120-21expr19:35:191
162399520,35rtkit-daemon2197014-21PK-Backend20:05:230
55509399510,28cyclictest1753-21gnome-shell22:33:533
555089995122,24cyclictest580154-21/usr/sbin/munin20:20:012
555089995117,29cyclictest855-21dbus-daemon23:00:012
555089995117,29cyclictest855-21dbus-daemon22:45:042
555089995117,29cyclictest855-21dbus-daemon22:45:042
555089995110,37cyclictest1753-21gnome-shell19:45:192
55508799512,30cyclictest1753-21gnome-shell19:55:211
555087995119,27cyclictest627148-21basename22:30:211
55509399507,17cyclictest42-21ksoftirqd/321:19:083
555093995041,6cyclictest411rcuc/319:17:123
555093995041,6cyclictest411rcuc/319:17:123
55508999502,43cyclictest1753-21gnome-shell22:25:292
555089995018,27cyclictest637930-21awk23:00:272
55508999501,25cyclictest658025-21sh00:00:012
162399500,6rtkit-daemon1622-21rtkit-daemon00:26:022
5951892490,7sleep02197014-21PK-Backend21:00:160
5951892490,7sleep02197014-21PK-Backend21:00:160
555093994940,6cyclictest411rcuc/322:21:593
55508999497,38cyclictest1753-21gnome-shell20:25:262
555089994918,26cyclictest438-20systemd-journal22:33:392
55508999491,42cyclictest570066-21cpuspeed19:50:132
55508799492,42cyclictest660897-21memory00:05:241
55508799492,41cyclictest1753-21gnome-shell23:05:181
162399490,34rtkit-daemon651353-21apt23:40:131
55509399489,19cyclictest42-21ksoftirqd/321:50:013
55509399487,13cyclictest42-21ksoftirqd/322:40:583
55508999489,34cyclictest1753-21gnome-shell22:10:322
55508999485,39cyclictest1069-21runrttasks22:00:452
55508999485,38cyclictest593337-21cstates20:55:152
55508999484,39cyclictest1069-21runrttasks23:36:212
55508999482,42cyclictest1753-21gnome-shell21:00:152
55508999482,42cyclictest1753-21gnome-shell21:00:152
555089994816,27cyclictest610958-21grep21:45:152
55508999481,43cyclictest1753-21gnome-shell21:10:542
555089994810,33cyclictest855-21dbus-daemon19:22:402
55508799481,42cyclictest591787-21grep20:50:281
55508799481,41cyclictest570939-21sort19:50:311
55508799481,31cyclictest604051-21cstates21:25:171
6030992470,11sleep0555077-21cyclictest21:22:380
555093994739,5cyclictest411rcuc/300:02:393
555093994737,7cyclictest411rcuc/323:34:563
555093994723,9cyclictest42-21ksoftirqd/320:45:313
555093994717,17cyclictest42-21ksoftirqd/300:17:003
555093994712,19cyclictest42-21ksoftirqd/322:59:193
55508999478,35cyclictest1753-21gnome-shell21:18:442
55508999478,34cyclictest1753-21gnome-shell20:47:382
555089994720,24cyclictest1753-21gnome-shell21:05:292
555089994716,27cyclictest581583-21seq20:21:282
555089994714,28cyclictest438-20systemd-journal23:20:002
555089994713,28cyclictest623266-21cpu22:20:142
555089994713,28cyclictest561467-21tr19:25:142
555089994711,31cyclictest651868-21latency23:40:222
55508799472,41cyclictest1753-21gnome-shell23:34:181
555087994723,20cyclictest644604-21tr23:20:171
555087994722,20cyclictest1753-21gnome-shell22:00:261
555087994719,25cyclictest1069-21runrttasks20:57:511
555087994716,27cyclictest1-21systemd00:30:011
55508799471,42cyclictest665001-21sshd00:16:391
55508799471,42cyclictest620310-21latency22:10:221
55508799471,40cyclictest581528-21seq20:20:431
162399470,4rtkit-daemon0-21swapper/019:10:050
162399470,4rtkit-daemon0-21swapper/019:10:040
55509399466,9cyclictest42-21ksoftirqd/300:35:293
55509399465,17cyclictest644269-21ls23:20:013
55509399463,21cyclictest584944-21ntp_states20:30:253
555093994627,14cyclictest594353-40apt-check20:57:163
55508999468,33cyclictest1753-21gnome-shell21:35:332
55508999462,39cyclictest1069-21runrttasks23:05:532
555089994615,27cyclictest1753-21gnome-shell00:01:522
55508799467,34cyclictest582787-21/usr/sbin/munin20:25:161
55508799462,39cyclictest610578-21munin-run21:45:011
55508799462,39cyclictest1753-21gnome-shell19:40:221
55508799461,40cyclictest1753-21gnome-shell20:30:301
55508799461,39cyclictest577995-21grep20:10:321
55508799461,39cyclictest577995-21grep20:10:321
55509399457,11cyclictest42-21ksoftirqd/320:15:133
55509399456,19cyclictest42-21ksoftirqd/321:06:003
55509399456,17cyclictest42-21ksoftirqd/320:05:163
55509399451,41cyclictest1622-21rtkit-daemon23:51:373
55508999452,37cyclictest592723-21dpkg20:55:022
555089994520,21cyclictest1069-21runrttasks00:20:262
555089994517,23cyclictest653136-21cpu23:45:152
555089994515,25cyclictest555667-21sh19:10:172
555089994515,25cyclictest555667-21sh19:10:172
55508799452,38cyclictest1753-21gnome-shell20:16:381
55508799452,29cyclictest1753-21gnome-shell20:01:201
55508799452,29cyclictest1753-21gnome-shell20:01:191
55508799452,28cyclictest1753-21gnome-shell22:25:111
55508799451,30cyclictest1753-21gnome-shell22:05:361
55508799451,29cyclictest667053-21munin-run00:25:001
6317122440,6sleep10-21swapper/122:45:021
55509399449,18cyclictest42-21ksoftirqd/322:45:043
55509399449,18cyclictest42-21ksoftirqd/322:45:043
555093994423,7cyclictest42-21ksoftirqd/300:31:203
555093994420,20cyclictest1069-21runrttasks21:13:233
555093994414,16cyclictest42-21ksoftirqd/319:29:143
555093994412,19cyclictest571719-21taskset19:55:113
55509399441,12cyclictest588257-21memory20:40:293
55509399441,12cyclictest588257-21memory20:40:293
55509399441,12cyclictest42-21ksoftirqd/320:15:023
55509399441,12cyclictest42-21ksoftirqd/320:15:023
55509399441,10cyclictest42-21ksoftirqd/320:40:023
555089994420,20cyclictest566520-21/usr/sbin/munin19:40:292
555089994418,22cyclictest661899-21latency_hist00:10:022
55508999441,39cyclictest1753-21gnome-shell19:32:212
555087994415,24cyclictest855-21dbus-daemon00:00:051
55508799441,38cyclictest587244-21taskset20:38:161
55508799441,38cyclictest1753-21gnome-shell21:05:111
55508799441,28cyclictest608673-21taskset21:38:251
162399440,30rtkit-daemon1753-21gnome-shell20:14:160
162399440,30rtkit-daemon1753-21gnome-shell20:14:160
55509399437,13cyclictest42-21ksoftirqd/300:20:213
55509399436,9cyclictest42-21ksoftirqd/319:31:563
55509399436,9cyclictest42-21ksoftirqd/300:10:243
55509399436,9cyclictest42-21ksoftirqd/300:10:233
55509399433,22cyclictest614469-21gpgconf21:55:033
555093994326,12cyclictest438-20systemd-journal23:27:223
55509399431,18cyclictest566154-21taskset19:38:393
55509399431,18cyclictest566154-21taskset19:38:393
55509399431,15cyclictest622904-21taskset22:19:063
55509399431,14cyclictest629204-21/usr/sbin/munin22:35:283
55509399431,12cyclictest1753-21gnome-shell21:38:573
55509399431,11cyclictest557061-21munin-node19:15:023
55509399431,11cyclictest557061-21munin-node19:15:023
55508999436,32cyclictest602903-21cat21:20:312
55508999432,36cyclictest656196-21taskset23:54:052
55508799438,31cyclictest1753-21gnome-shell22:45:161
55508799438,31cyclictest1753-21gnome-shell22:45:161
555082994330,9cyclictest13-21ksoftirqd/000:00:030
55509399427,19cyclictest42-21ksoftirqd/322:00:143
55509399426,9cyclictest42-21ksoftirqd/321:20:173
55509399426,13cyclictest42-21ksoftirqd/319:47:403
55509399426,10cyclictest42-21ksoftirqd/320:00:203
55509399426,10cyclictest42-21ksoftirqd/320:00:203
55509399425,33cyclictest605096-40lsb_release21:27:233
55509399425,10cyclictest42-21ksoftirqd/321:00:323
55509399425,10cyclictest42-21ksoftirqd/321:00:323
55509399425,10cyclictest42-21ksoftirqd/300:25:273
55509399423,13cyclictest42-21ksoftirqd/323:44:593
555093994211,10cyclictest42-21ksoftirqd/321:40:123
55508799428,31cyclictest1753-21gnome-shell20:47:591
55508799422,25cyclictest1753-21gnome-shell21:50:051
555087994220,18cyclictest568647-21sh19:45:191
555082994226,11cyclictest438-20systemd-journal20:50:000
55508299421,9cyclictest141rcu_preempt20:50:210
55509399415,9cyclictest42-21ksoftirqd/322:25:233
55509399415,10cyclictest42-21ksoftirqd/322:05:183
55509399414,18cyclictest650425-21proc_pri23:35:283
55509399414,12cyclictest42-21ksoftirqd/319:50:343
55509399411,13cyclictest606597-21/usr/sbin/munin21:30:313
55508799411,26cyclictest1753-21gnome-shell21:14:461
555082994133,4cyclictest2197014-21PK-Backend21:39:280
555082994130,8cyclictest13-21ksoftirqd/019:22:390
162399410,29rtkit-daemon2197014-21PK-Backend21:46:540
55509399407,13cyclictest42-21ksoftirqd/300:06:003
55509399405,17cyclictest42-21ksoftirqd/320:28:093
555093994018,6cyclictest141rcu_preempt23:59:173
555089994016,20cyclictest670499-21cut00:35:022
555082994028,8cyclictest572227-21irqstats19:55:220
555082994023,6cyclictest13-21ksoftirqd/021:15:070
555082994022,6cyclictest13-21ksoftirqd/023:15:180
162399400,27rtkit-daemon2197014-21PK-Backend21:40:130
55509399397,10cyclictest42-21ksoftirqd/322:50:283
55509399396,9cyclictest2197014-21PK-Backend20:23:233
55509399393,20cyclictest42-21ksoftirqd/323:05:273
55509399391,22cyclictest653975-21/usr/sbin/munin23:45:303
55508299392,8cyclictest1753-21gnome-shell00:28:020
555082993925,11cyclictest635174-21awk22:54:590
555082993925,10cyclictest631375-21sshd22:41:380
555082993924,11cyclictest558786-21sshd19:17:360
555082993924,11cyclictest558786-21sshd19:17:360
55508299392,33cyclictest1753-21gnome-shell00:30:320
555082993922,13cyclictest13-21ksoftirqd/000:35:140
555082993921,14cyclictest615627-21users21:55:320
55509399381,12cyclictest1753-21gnome-shell20:52:403
555082993827,7cyclictest1753-21gnome-shell00:05:490
555082993821,13cyclictest13-21ksoftirqd/023:01:560
555082993821,12cyclictest438-20systemd-journal23:52:380
555082993820,6cyclictest13-21ksoftirqd/000:15:010
555082993820,6cyclictest13-21ksoftirqd/000:15:010
162399380,27rtkit-daemon2197014-21PK-Backend21:14:590
5750122370,5sleep0161rcuc/020:05:020
5750122370,5sleep0161rcuc/020:05:020
55509399375,9cyclictest2197014-21PK-Backend23:01:013
55508299376,5cyclictest141rcu_preempt19:43:250
555082993727,6cyclictest2197014-21PK-Backend20:25:120
162399370,24rtkit-daemon13-21ksoftirqd/000:16:330
55508299362,18cyclictest1753-21gnome-shell23:40:070
55508299362,11cyclictest614176-21apt-config21:55:020
555082993621,11cyclictest13-21ksoftirqd/022:37:440
555082993618,13cyclictest13-21ksoftirqd/019:30:140
55508299361,4cyclictest2197014-21PK-Backend23:30:320
162399360,24rtkit-daemon2197014-21PK-Backend22:57:210
555082993524,7cyclictest1753-21gnome-shell20:43:060
555082993524,7cyclictest1753-21gnome-shell20:43:060
555082993524,6cyclictest13-21ksoftirqd/023:45:180
555082993520,11cyclictest594398-40apt-check20:57:260
55508299351,17cyclictest1069-21runrttasks23:05:120
555082993423,6cyclictest13-21ksoftirqd/023:14:130
555082993423,6cyclictest13-21ksoftirqd/023:14:120
555082993423,6cyclictest13-21ksoftirqd/022:16:200
555082993422,9cyclictest13-21ksoftirqd/023:40:010
55508299342,18cyclictest1753-21gnome-shell19:38:520
55508299342,18cyclictest1753-21gnome-shell19:38:520
55508299341,9cyclictest2197014-21PK-Backend23:26:520
555082993419,11cyclictest584444-21cat20:30:150
555082993313,6cyclictest141rcu_preempt19:48:240
55508299331,13cyclictest628029-21taskset22:33:090
6170662320,8sleep02197014-21PK-Backend22:00:250
55508299321,9cyclictest2197014-21PK-Backend23:20:250
55508299321,28cyclictest2197014-21PK-Backend22:06:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional