You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 12:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Thu Nov 27, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
73771322640,8sleep00-21swapper/019:05:120
77550321200,8sleep21753-21gnome-shell20:55:302
1623991050,24rtkit-daemon881143-21gmain23:02:043
7547382990,16sleep342-21ksoftirqd/319:50:223
7695962960,6sleep10-21swapper/120:40:111
7983942950,5sleep10-21swapper/122:00:321
7527962950,6sleep31753-21gnome-shell19:45:203
162399840,9rtkit-daemon1622-21rtkit-daemon19:48:430
162399830,8rtkit-daemon1622-21rtkit-daemon23:30:200
73878528162,13sleep30-21swapper/319:06:123
162399790,5rtkit-daemon0-21swapper/222:40:222
8154462770,4chrt814933-21/usr/sbin/munin22:50:220
73875227246,17sleep20-21swapper/219:05:482
8212432710,14sleep2341ksoftirqd/223:05:292
8212432710,14sleep2341ksoftirqd/223:05:292
73879526443,13sleep10-21swapper/119:06:191
162399630,17rtkit-daemon1622-21rtkit-daemon23:56:570
162399620,17rtkit-daemon1622-21rtkit-daemon00:11:450
162399600,9rtkit-daemon1622-21rtkit-daemon21:51:450
162399600,9rtkit-daemon1622-21rtkit-daemon21:15:281
162399600,10rtkit-daemon1622-21rtkit-daemon20:40:500
7837422590,5sleep32197014-21PK-Backend21:20:213
162399590,9rtkit-daemon1622-21rtkit-daemon23:37:190
162399590,9rtkit-daemon1622-21rtkit-daemon23:14:210
162399590,8rtkit-daemon1622-21rtkit-daemon19:20:160
162399580,9rtkit-daemon1622-21rtkit-daemon19:31:000
162399580,9rtkit-daemon1622-21rtkit-daemon00:27:001
162399580,9rtkit-daemon1622-21rtkit-daemon00:24:570
7851692570,11sleep0785053-21cstates21:25:130
73922699573,20cyclictest13-21ksoftirqd/021:40:140
162399570,9rtkit-daemon1622-21rtkit-daemon21:51:381
162399570,9rtkit-daemon1622-21rtkit-daemon20:38:340
162399570,9rtkit-daemon1622-21rtkit-daemon20:38:340
162399570,9rtkit-daemon1622-21rtkit-daemon20:14:210
162399570,8rtkit-daemon1622-21rtkit-daemon23:16:490
162399560,9rtkit-daemon1622-21rtkit-daemon21:43:271
8127312550,7sleep30-21swapper/322:45:013
739228995520,13cyclictest751817-40apt-check19:42:131
739226995518,20cyclictest13-21ksoftirqd/021:05:210
162399550,8rtkit-daemon1622-21rtkit-daemon22:29:450
162399540,8rtkit-daemon1622-21rtkit-daemon23:09:491
162399540,8rtkit-daemon1622-21rtkit-daemon23:09:481
162399540,8rtkit-daemon1622-21rtkit-daemon22:56:211
739234995331,18cyclictest42-21ksoftirqd/319:10:253
162399520,8rtkit-daemon1622-21rtkit-daemon22:16:181
73922699515,42cyclictest1622-21rtkit-daemon20:56:170
162399510,8rtkit-daemon1622-21rtkit-daemon23:33:541
162399510,27rtkit-daemon854-21avahi-daemon20:49:152
7636332500,9sleep31753-21gnome-shell20:15:293
739234995021,25cyclictest42-21ksoftirqd/322:38:463
739232995018,9cyclictest2197014-21PK-Backend21:55:102
73922699501,5cyclictest0-21swapper/022:32:350
162399500,8rtkit-daemon1622-21rtkit-daemon21:58:431
162399500,6rtkit-daemon1622-21rtkit-daemon21:03:151
73922699495,9cyclictest13-21ksoftirqd/022:45:010
73922699495,14cyclictest788203-21wc21:35:010
73922699491,14cyclictest751468-21swap19:40:300
162399490,36rtkit-daemon0-21swapper/122:26:471
73923499486,18cyclictest42-21ksoftirqd/323:10:253
73923499485,20cyclictest42-21ksoftirqd/322:05:283
73923499485,20cyclictest42-21ksoftirqd/322:05:273
739234994830,13cyclictest42-21ksoftirqd/300:25:163
73922699489,16cyclictest1753-21gnome-shell00:37:220
73922699484,9cyclictest13-21ksoftirqd/022:50:020
73922699483,13cyclictest784719-21latency_hist21:25:020
162399480,6rtkit-daemon1622-21rtkit-daemon22:05:521
162399480,6rtkit-daemon1622-21rtkit-daemon22:05:521
8187362470,5sleep2331rcuc/223:00:152
739234994723,13cyclictest42-21ksoftirqd/319:55:533
739234994717,26cyclictest803479-21expr22:15:293
73923499471,19cyclictest860997-21snmpd21:10:223
73922699479,15cyclictest1753-21gnome-shell22:19:540
73922699473,21cyclictest1753-21gnome-shell00:05:270
739226994717,16cyclictest13-21ksoftirqd/021:05:010
162399470,40rtkit-daemon856-21gmain19:29:021
739234994624,14cyclictest42-21ksoftirqd/322:50:063
739234994623,14cyclictest42-21ksoftirqd/320:10:003
739234994621,15cyclictest42-21ksoftirqd/321:34:083
73922699468,11cyclictest13-21ksoftirqd/023:45:230
73922699463,16cyclictest801179-21hddtemp_smartct22:10:190
73922699463,12cyclictest827885-21ntp_states23:25:270
739226994612,14cyclictest1753-21gnome-shell00:29:230
7916022450,4sleep30-21swapper/321:42:283
73923499456,20cyclictest42-21ksoftirqd/323:05:253
73923499456,20cyclictest42-21ksoftirqd/323:05:253
739234994527,7cyclictest42-21ksoftirqd/321:17:373
739234994525,16cyclictest42-21ksoftirqd/321:30:013
739234994524,13cyclictest42-21ksoftirqd/323:16:253
73923499452,39cyclictest757939-21diskstats20:00:183
73923499452,39cyclictest757939-21diskstats20:00:183
739234994519,16cyclictest42-21ksoftirqd/319:25:293
739232994518,11cyclictest1753-21gnome-shell21:29:592
73923299451,33cyclictest0-21swapper/220:19:262
73922699459,19cyclictest13-21ksoftirqd/020:30:190
73922699459,19cyclictest13-21ksoftirqd/020:30:190
73922699457,11cyclictest1753-21gnome-shell19:35:220
73922699456,19cyclictest13-21ksoftirqd/022:59:400
73922699453,9cyclictest792517-21hddtemp_smartct21:45:190
739226994511,13cyclictest13-21ksoftirqd/023:24:140
739226994510,15cyclictest1753-21gnome-shell23:05:410
739226994510,15cyclictest1753-21gnome-shell23:05:410
8157692440,7sleep22197014-21PK-Backend22:50:282
73923499444,5cyclictest141rcu_preempt21:48:303
739234994423,13cyclictest42-21ksoftirqd/300:02:203
739234994422,14cyclictest42-21ksoftirqd/300:18:443
739234994422,13cyclictest42-21ksoftirqd/300:21:333
739234994420,13cyclictest42-21ksoftirqd/323:35:263
739232994431,9cyclictest1753-21gnome-shell00:10:002
739228994422,17cyclictest810582-21users22:35:321
739228994420,20cyclictest823094-21sshd23:11:191
73922699449,19cyclictest13-21ksoftirqd/020:50:150
73922699449,10cyclictest13-21ksoftirqd/022:10:020
73922699449,10cyclictest13-21ksoftirqd/022:10:020
73922699447,10cyclictest13-21ksoftirqd/021:40:030
73922699447,10cyclictest13-21ksoftirqd/021:40:020
73922699444,16cyclictest742022-21df_inode19:15:150
73922699444,10cyclictest757323-21munin-run20:00:020
739226994435,6cyclictest161rcuc/000:01:280
739226994414,14cyclictest1358-1kworker/u9:123:41:290
73922699441,10cyclictest2197014-21PK-Backend22:00:590
7652742430,7sleep32197014-21PK-Backend20:20:243
739234994324,12cyclictest42-21ksoftirqd/323:45:273
739234994322,12cyclictest42-21ksoftirqd/319:41:093
739234994321,15cyclictest42-21ksoftirqd/300:07:203
739234994321,14cyclictest42-21ksoftirqd/320:45:593
739234994321,13cyclictest42-21ksoftirqd/319:31:183
739234994318,13cyclictest42-21ksoftirqd/322:01:563
739234994314,13cyclictest42-21ksoftirqd/321:52:083
73923499431,18cyclictest1-21systemd23:00:013
73923299439,29cyclictest34-21ksoftirqd/222:20:002
739232994330,9cyclictest751545-21grep19:40:302
739232994325,6cyclictest34-21ksoftirqd/200:02:402
739232994313,15cyclictest2631797-21snapd22:35:172
73922699436,19cyclictest13-21ksoftirqd/020:21:560
73922699433,15cyclictest1753-21gnome-shell19:51:350
73922699431,16cyclictest818473-21grep23:00:130
73922699431,15cyclictest1753-21gnome-shell21:10:480
162399430,35rtkit-daemon26-21ksoftirqd/123:36:331
7477702420,6chrt2197014-21PK-Backend19:30:302
73923499424,13cyclictest141rcu_preempt22:14:533
739234994225,7cyclictest42-21ksoftirqd/300:35:143
739234994222,13cyclictest42-21ksoftirqd/323:51:583
73923499422,19cyclictest1753-21gnome-shell20:55:483
739234994221,14cyclictest42-21ksoftirqd/323:55:213
739234994220,11cyclictest42-21ksoftirqd/320:29:473
739234994218,14cyclictest42-21ksoftirqd/319:21:143
73923299429,4cyclictest141rcu_preempt19:59:172
739232994225,6cyclictest34-21ksoftirqd/220:30:212
739232994225,6cyclictest34-21ksoftirqd/220:30:202
739232994224,6cyclictest34-21ksoftirqd/223:59:302
739232994224,6cyclictest34-21ksoftirqd/223:35:102
739232994223,6cyclictest34-21ksoftirqd/219:20:262
739232994220,6cyclictest787195-21/usr/sbin/munin21:30:212
739232994216,10cyclictest1753-21gnome-shell19:11:372
739228994214,9cyclictest808985-21systemd-udevd22:32:361
73922699428,11cyclictest13-21ksoftirqd/021:15:260
73922699427,11cyclictest13-21ksoftirqd/019:26:150
73922699425,12cyclictest0-21swapper/020:49:000
73922699421,9cyclictest0-21swapper/000:33:190
739226994214,11cyclictest13-21ksoftirqd/020:09:560
73922699421,11cyclictest1753-21gnome-shell22:20:140
73922699420,24cyclictest1753-21gnome-shell19:10:540
7501722410,18sleep126-21ksoftirqd/119:40:031
739234994129,8cyclictest1753-21gnome-shell20:42:063
739234994129,8cyclictest1753-21gnome-shell00:10:283
73923499412,34cyclictest3242253-21nautilus22:00:013
739234994121,6cyclictest42-21ksoftirqd/322:24:453
739234994121,13cyclictest42-21ksoftirqd/300:30:483
739234994120,14cyclictest42-21ksoftirqd/320:34:313
739234994120,14cyclictest42-21ksoftirqd/320:34:303
739234994120,13cyclictest42-21ksoftirqd/320:10:183
739234994119,14cyclictest42-21ksoftirqd/322:32:413
739234994119,13cyclictest42-21ksoftirqd/321:03:513
73923299418,5cyclictest141rcu_preempt21:20:242
73923299415,7cyclictest141rcu_preempt20:43:262
73923299412,9cyclictest0-21swapper/219:15:122
739232994124,13cyclictest753853-21systemd-run19:50:002
739232994123,6cyclictest34-21ksoftirqd/223:15:582
739232994114,8cyclictest34-21ksoftirqd/223:30:442
73923299411,37cyclictest1753-21gnome-shell23:46:462
73923299411,15cyclictest761900-21/usr/sbin/munin20:10:312
739232994111,13cyclictest0-21swapper/222:27:012
739228994120,17cyclictest754302-21cstates19:50:131
739228994115,22cyclictest757221-21sshd19:59:201
73922699417,16cyclictest13-21ksoftirqd/022:35:160
73922699414,15cyclictest796449-21python321:55:270
73922699414,10cyclictest1753-21gnome-shell20:26:480
73922699413,13cyclictest763163-21hddtemp_smartct20:15:200
73922699413,10cyclictest844983-21irqstats00:15:220
73922699412,19cyclictest13-21ksoftirqd/020:04:080
73922699412,19cyclictest13-21ksoftirqd/020:04:070
739234994024,6cyclictest42-21ksoftirqd/323:26:133
739234994023,6cyclictest42-21ksoftirqd/322:29:253
739234994021,7cyclictest42-21ksoftirqd/323:20:303
739234994020,13cyclictest42-21ksoftirqd/319:38:073
73923499401,12cyclictest1753-21gnome-shell22:45:153
739232994022,6cyclictest34-21ksoftirqd/200:25:312
739232994018,19cyclictest837281-21awk23:55:002
739232994013,8cyclictest34-21ksoftirqd/222:32:572
739232994011,12cyclictest0-21swapper/200:18:312
739232994011,11cyclictest141rcu_preempt22:01:312
739228994024,12cyclictest855-21dbus-daemon19:25:001
739228994015,21cyclictest780641-21sed21:10:301
162399400,29rtkit-daemon0-21swapper/320:54:183
8452842390,13sleep1251rcuc/100:15:271
739234993922,6cyclictest42-21ksoftirqd/321:35:433
739234993922,6cyclictest42-21ksoftirqd/321:35:423
73923499392,19cyclictest1753-21gnome-shell19:15:293
739234993919,6cyclictest42-21ksoftirqd/320:35:463
739234993919,6cyclictest42-21ksoftirqd/320:35:463
739234993919,12cyclictest42-21ksoftirqd/321:07:303
739234993918,5cyclictest42-21ksoftirqd/323:32:503
739232993927,9cyclictest817622-21awk22:55:292
739232993922,13cyclictest34-21ksoftirqd/220:23:302
73923299391,5cyclictest2197014-21PK-Backend21:50:022
739228993925,10cyclictest772295-21sshd20:48:191
739228993923,12cyclictest802260-21/usr/sbin/munin22:15:001
8363022380,5chrt0-21swapper/123:50:241
73923299389,7cyclictest739221-21cyclictest22:22:282
73923299387,16cyclictest1753-21gnome-shell00:11:132
73923299382,8cyclictest1753-21gnome-shell22:47:212
739232993828,6cyclictest860997-21snmpd23:11:282
739232993827,7cyclictest1753-21gnome-shell23:24:162
739232993827,7cyclictest1753-21gnome-shell00:32:092
739232993826,8cyclictest1753-21gnome-shell22:06:552
739232993826,8cyclictest1753-21gnome-shell22:06:542
739232993822,12cyclictest846614-21entropy00:20:182
739232993820,6cyclictest34-21ksoftirqd/221:10:132
73923299381,8cyclictest0-21swapper/221:40:082
739232993816,18cyclictest778101-21tr21:05:132
739232993813,7cyclictest34-21ksoftirqd/221:51:132
73923299381,21cyclictest1753-21gnome-shell20:01:182
73923299381,21cyclictest1753-21gnome-shell20:01:182
739232993812,13cyclictest34-21ksoftirqd/220:27:582
73923299381,20cyclictest1358-1kworker/u9:122:10:202
739228993827,7cyclictest760742-21cron20:09:591
739228993821,12cyclictest438-20systemd-journal20:54:551
739228993820,13cyclictest843489-21ntpq00:10:231
739228993820,13cyclictest438-20systemd-journal00:10:011
739228993817,17cyclictest783121-21cpu21:20:121
739228993814,20cyclictest758144-21load20:00:241
739228993814,20cyclictest758144-21load20:00:231
73922899381,28cyclictest0-21swapper/119:10:491
73922699388,10cyclictest13-21ksoftirqd/023:52:200
739232993726,7cyclictest1753-21gnome-shell20:35:562
739232993726,7cyclictest1753-21gnome-shell20:35:562
739232993721,13cyclictest34-21ksoftirqd/221:39:192
739232993721,13cyclictest34-21ksoftirqd/221:39:192
739232993721,13cyclictest34-21ksoftirqd/200:38:212
739232993720,13cyclictest34-21ksoftirqd/223:40:132
739232993714,5cyclictest141rcu_preempt20:50:252
739228993721,12cyclictest779152-21sshd21:07:191
73922899372,10cyclictest747218-21df19:30:151
739228993720,12cyclictest438-20systemd-journal00:34:561
739228993719,14cyclictest3242255-21gdbus20:20:011
739228993714,19cyclictest815113-21grep22:50:141
73923499361,11cyclictest2197014-21PK-Backend23:40:373
739232993625,6cyclictest34-21ksoftirqd/219:35:142
73923299362,13cyclictest0-21swapper/219:54:252
739232993620,13cyclictest34-21ksoftirqd/221:16:262
739232993618,6cyclictest34-21ksoftirqd/223:27:202
73922899367,25cyclictest847686-21cron00:25:001
739228993626,7cyclictest64850irq/126-i91522:45:151
739228993626,6cyclictest768322-21cron20:35:001
739228993626,6cyclictest768322-21cron20:35:001
739228993623,10cyclictest793597-21awk21:50:001
739228993621,11cyclictest789391-21uptime21:35:291
739228993621,11cyclictest789391-21uptime21:35:291
739228993620,12cyclictest854-21avahi-daemon00:00:041
73922899361,9cyclictest0-21swapper/120:23:191
73922899361,8cyclictest0-21swapper/121:29:521
73922899361,17cyclictest826858-21cron23:25:001
739228993611,21cyclictest824063-21hddtemp_smartct23:15:181
739232993524,7cyclictest776847-21grep21:00:222
739232993524,6cyclictest34-21ksoftirqd/219:25:242
73923299351,5cyclictest141rcu_preempt20:05:132
73922899359,10cyclictest811213-21sort22:40:101
739228993520,11cyclictest839039-40ogrotate00:00:011
739228993520,11cyclictest753559-21chrt19:46:461
73922899351,9cyclictest0-21swapper/123:49:141
739228993519,12cyclictest769102-21cat20:40:031
739228993519,12cyclictest769102-21cat20:40:031
739228993518,13cyclictest788059-21/usr/sbin/munin21:35:011
739228993423,7cyclictest775901-21dump-pmu-power21:00:001
739228993422,9cyclictest804226-21/usr/sbin/munin22:20:321
739228993422,8cyclictest743118-21perf19:20:011
739228993419,11cyclictest855-21dbus-daemon23:01:561
739228993418,12cyclictest851410-21sshd00:35:121
739228993418,12cyclictest828532-21sshd23:29:181
739228993319,10cyclictest1069-21runrttasks20:10:251
739228993316,12cyclictest438-20systemd-journal23:40:021
739228993315,14cyclictest3242255-21gdbus20:30:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional