You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 08:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Nov 26, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16972723340,8sleep32197014-21PK-Backend19:10:003
2022922990,5sleep10-21swapper/120:40:021
162399990,5rtkit-daemon0-21swapper/121:01:500
162399920,11rtkit-daemon1753-21gnome-shell20:57:050
162399890,10rtkit-daemon1753-21gnome-shell00:12:590
162399880,6rtkit-daemon2197014-21PK-Backend00:18:040
162399860,8rtkit-daemon1622-21rtkit-daemon22:11:192
16945328468,9sleep00-21swapper/019:07:130
2542252810,4chrt253553-21/usr/sbin/munin23:05:262
162399780,10rtkit-daemon1753-21gnome-shell23:24:020
162399770,9rtkit-daemon1622-21rtkit-daemon23:03:452
162399770,8rtkit-daemon218606-21/usr/sbin/munin21:25:200
162399770,6rtkit-daemon2197014-21PK-Backend20:53:440
162399760,6rtkit-daemon2197014-21PK-Backend19:29:220
162399760,6rtkit-daemon2197014-21PK-Backend00:25:510
162399750,6rtkit-daemon2197014-21PK-Backend21:57:400
162399750,6rtkit-daemon2197014-21PK-Backend21:57:400
162399750,6rtkit-daemon2197014-21PK-Backend19:20:230
162399740,6rtkit-daemon2197014-21PK-Backend21:53:330
162399740,5rtkit-daemon0-21swapper/023:36:420
162399740,5rtkit-daemon0-21swapper/021:20:380
1682292730,8sleep2168233-21cpu19:05:132
162399730,6rtkit-daemon2197014-21PK-Backend20:22:360
162399730,6rtkit-daemon0-21swapper/023:41:150
162399730,6rtkit-daemon0-21swapper/023:41:150
16965627240,9sleep10-21swapper/119:09:521
162399720,6rtkit-daemon2197014-21PK-Backend21:40:340
162399720,6rtkit-daemon2197014-21PK-Backend21:36:050
162399710,6rtkit-daemon0-21swapper/023:27:430
162399710,5rtkit-daemon0-21swapper/022:23:440
162399710,5rtkit-daemon0-21swapper/019:53:520
162399710,5rtkit-daemon0-21swapper/019:53:520
162399690,5rtkit-daemon0-21swapper/022:46:460
162399680,5rtkit-daemon0-21swapper/019:12:590
162399680,5rtkit-daemon0-21swapper/019:12:580
169811996717,28cyclictest1069-21runrttasks23:36:192
16981199642,7cyclictest1753-21gnome-shell21:43:312
16981199632,7cyclictest1753-21gnome-shell23:56:462
16981199632,7cyclictest1753-21gnome-shell23:56:452
16981199632,7cyclictest1753-21gnome-shell23:26:202
16981199632,7cyclictest0-21swapper/223:48:162
16981199631,17cyclictest1753-21gnome-shell20:13:462
16981199622,9cyclictest1753-21gnome-shell23:11:292
16981199622,7cyclictest1753-21gnome-shell21:31:332
16981199622,7cyclictest1753-21gnome-shell19:47:352
16981199622,7cyclictest1753-21gnome-shell00:29:002
169815996115,15cyclictest0-21swapper/320:02:043
16981199612,8cyclictest1753-21gnome-shell22:06:302
16981199612,8cyclictest1753-21gnome-shell00:19:372
16981199612,7cyclictest1753-21gnome-shell23:34:212
16981199612,7cyclictest1753-21gnome-shell22:02:142
16981199611,7cyclictest860997-21snmpd20:34:542
16981199611,7cyclictest249546-21runrttasks22:54:522
16981199609,28cyclictest0-21swapper/220:49:262
16981199602,8cyclictest1753-21gnome-shell21:53:592
16981199602,7cyclictest1753-21gnome-shell22:20:082
16981199602,7cyclictest1753-21gnome-shell21:20:282
16981199602,7cyclictest1753-21gnome-shell19:52:432
16981199602,7cyclictest1753-21gnome-shell19:52:422
16981199602,7cyclictest1753-21gnome-shell19:27:362
169815995920,12cyclictest207058-21sshd20:51:313
169815995919,11cyclictest231323-21cat22:00:223
169815995910,16cyclictest1753-21gnome-shell20:30:333
16981199592,8cyclictest1753-21gnome-shell23:50:472
16981199592,7cyclictest1753-21gnome-shell22:29:212
16981199592,7cyclictest1753-21gnome-shell21:25:552
16981199592,7cyclictest1753-21gnome-shell20:37:552
16981199592,7cyclictest1753-21gnome-shell19:21:342
2119902580,6sleep1251rcuc/121:05:141
16981599582,52cyclictest285446-21memory00:35:243
16981599582,50cyclictest1753-21gnome-shell23:20:203
169815995818,7cyclictest178437-21ntp_kernel_err19:30:253
16981199582,7cyclictest1753-21gnome-shell20:28:382
16981199582,10cyclictest1753-21gnome-shell22:32:122
169805995821,20cyclictest13-21ksoftirqd/020:32:160
169815995722,31cyclictest173406-21sshd19:16:313
169815995722,30cyclictest1753-21gnome-shell21:45:233
169815995721,32cyclictest263031-21ntp_states23:30:253
169815995720,13cyclictest173965-21apt19:20:113
169815995719,8cyclictest279755-21cpuspeed_turbos00:20:143
169815995713,12cyclictest282037-21needreboot00:25:243
169815995712,12cyclictest184014-21cat19:45:323
16981199572,8cyclictest1753-21gnome-shell00:00:492
16981199572,8cyclictest1753-21gnome-shell00:00:492
16981199572,7cyclictest1753-21gnome-shell20:05:392
16981199571,7cyclictest214572-21taskset21:11:072
16981599569,19cyclictest225081-21sshd21:42:233
16981599562,50cyclictest182706-21munin-run19:45:023
16981599562,26cyclictest1753-21gnome-shell22:10:153
169815995617,11cyclictest219706-21cat21:25:303
169815995615,13cyclictest438-20systemd-journal22:56:423
16981599561,16cyclictest1753-21gnome-shell23:17:023
169815995516,10cyclictest264555-21cat23:35:213
169815995515,11cyclictest238271-21cat22:20:233
169815995515,10cyclictest276779-21cat00:10:233
169815995514,12cyclictest236903-21cat22:15:293
16981599551,25cyclictest1753-21gnome-shell21:30:153
169811995522,29cyclictest286376-21cat00:40:022
2722122540,20chrt26-21ksoftirqd/123:57:321
2722122540,20chrt26-21ksoftirqd/123:57:321
16981599549,17cyclictest1753-21gnome-shell19:15:013
16981599549,17cyclictest1753-21gnome-shell19:15:013
16981599548,18cyclictest1753-21gnome-shell23:57:093
16981599548,18cyclictest1753-21gnome-shell23:57:093
16981599543,10cyclictest1753-21gnome-shell23:40:193
16981599543,10cyclictest1753-21gnome-shell23:40:193
169815995419,6cyclictest249732-21grep22:55:023
169815995416,7cyclictest273548-21processes00:00:333
169815995416,7cyclictest273548-21processes00:00:333
169815995413,9cyclictest1825-21JS14:31:443
169815995413,9cyclictest1825-21JS14:31:443
16981599541,15cyclictest1753-21gnome-shell19:25:533
169815995411,12cyclictest230282-21sshd21:57:253
169815995411,12cyclictest230282-21sshd21:57:253
169815995410,16cyclictest1753-21gnome-shell20:35:243
169811995419,30cyclictest1632-21dbus-daemon21:10:022
169805995430,7cyclictest141rcu_preempt20:28:400
169805995428,8cyclictest13-21ksoftirqd/020:05:430
162399540,8rtkit-daemon1622-21rtkit-daemon23:02:473
16981599532,20cyclictest212549-21sort21:05:293
169815995317,7cyclictest1069-21runrttasks22:47:403
169815995316,8cyclictest2197014-21PK-Backend22:31:383
169815995314,12cyclictest253541-21dpkg23:05:053
169815995313,12cyclictest267860-21df_inode23:45:153
169811995323,25cyclictest265476-21cron23:40:022
169811995323,25cyclictest265476-21cron23:40:022
16981599527,16cyclictest1753-21gnome-shell23:10:353
16981599522,8cyclictest1753-21gnome-shell00:30:273
169815995221,8cyclictest1753-21gnome-shell23:27:403
169815995219,8cyclictest1753-21gnome-shell00:16:303
169815995218,7cyclictest245318-21grep22:40:253
169815995214,9cyclictest218396-21seq21:25:013
169815995214,10cyclictest208731-21cut20:55:283
16981199522,7cyclictest1753-21gnome-shell19:41:392
16981599518,18cyclictest1753-21gnome-shell21:10:493
16981599512,25cyclictest855-21dbus-daemon19:57:033
169815995122,24cyclictest232881-21diskstats22:05:183
169815995119,8cyclictest1753-21gnome-shell20:20:313
169815995115,7cyclictest180260-21proc_pri19:35:283
16981599511,22cyclictest1753-21gnome-shell23:50:193
169815995110,16cyclictest1753-21gnome-shell21:54:133
16981199512,9cyclictest1069-21runrttasks00:20:592
16981199512,8cyclictest1753-21gnome-shell19:13:522
16981199512,8cyclictest1753-21gnome-shell19:13:522
169811995117,29cyclictest244743-21sort22:40:162
169811995115,31cyclictest206368-21uname20:50:222
16980599519,14cyclictest13-21ksoftirqd/019:32:240
169805995127,8cyclictest13-21ksoftirqd/019:49:360
16980599512,12cyclictest13-21ksoftirqd/000:00:010
16980599512,12cyclictest13-21ksoftirqd/000:00:010
169815995017,6cyclictest391irq_work/321:35:153
16981599501,6cyclictest205650-21latency_hist20:50:013
169815995014,12cyclictest275039-21memory00:05:233
169815995013,13cyclictest2962348-21rs:main3
169815995010,16cyclictest1753-21gnome-shell20:06:003
16981199502,8cyclictest1753-21gnome-shell22:49:522
169811995022,24cyclictest187917-21latency_hist20:00:012
169811995016,29cyclictest209108-40apt-check20:57:132
169815994917,7cyclictest859-21NetworkManager20:43:403
16981199492,8cyclictest1753-21gnome-shell21:00:032
169811994918,26cyclictest189414-21sshd20:02:042
169811994915,30cyclictest257428-21expr23:15:162
169811994914,30cyclictest242988-21cat22:35:132
169811994913,32cyclictest250710-21python322:55:282
16980599496,23cyclictest438-20systemd-journal21:19:590
16981599487,16cyclictest1753-21gnome-shell21:00:583
169815994817,8cyclictest1753-21gnome-shell20:16:463
169815994810,14cyclictest1753-21gnome-shell20:29:223
169811994817,26cyclictest172945-21awk19:15:222
16981199481,43cyclictest1-21systemd19:40:002
169811994813,31cyclictest274716-21expr00:05:142
169811994810,34cyclictest216062-21cat21:15:302
16980599481,10cyclictest173702-21munin-run19:20:010
16981599471,4cyclictest854-21avahi-daemon21:18:053
16981199477,17cyclictest1753-21gnome-shell00:14:542
169809994721,22cyclictest178333-40lsb_release19:30:251
169805994722,8cyclictest13-21ksoftirqd/022:41:260
169805994710,18cyclictest13-21ksoftirqd/021:49:310
16981199468,34cyclictest1753-21gnome-shell20:17:492
16981199463,11cyclictest1753-21gnome-shell21:48:412
169811994617,24cyclictest177535-21dpkg19:30:042
169809994617,12cyclictest187465-21sed19:55:311
16980599467,18cyclictest13-21ksoftirqd/000:00:250
16980599467,18cyclictest13-21ksoftirqd/000:00:250
169805994610,19cyclictest13-21ksoftirqd/022:25:190
2708732450,7sleep00-21swapper/023:55:020
2544522450,7sleep02197014-21PK-Backend23:05:310
169815994517,6cyclictest391irq_work/320:13:593
16981199457,34cyclictest283100-21cat00:30:102
169811994518,23cyclictest1069-21runrttasks23:20:402
169811994514,27cyclictest236883-21switchtime22:15:302
16981199451,40cyclictest230106-21sed21:55:312
16981199451,40cyclictest230106-21sed21:55:302
169809994517,24cyclictest26-21ksoftirqd/121:15:021
169805994521,12cyclictest141rcu_preempt19:38:440
169805994519,13cyclictest141rcu_preempt22:52:370
169805994510,18cyclictest13-21ksoftirqd/021:06:180
16980999441,4cyclictest141rcu_preempt20:32:151
16980599446,10cyclictest13-21ksoftirqd/020:40:000
169805994420,6cyclictest141rcu_preempt20:00:230
169805994419,9cyclictest13-21ksoftirqd/000:34:120
169805994418,8cyclictest13-21ksoftirqd/022:09:260
162399440,31rtkit-daemon2197014-21PK-Backend23:15:300
162399440,30rtkit-daemon1753-21gnome-shell22:51:091
16980599438,19cyclictest13-21ksoftirqd/022:10:150
169805994320,6cyclictest141rcu_preempt23:12:250
169805994318,8cyclictest13-21ksoftirqd/021:34:570
169805994314,14cyclictest13-21ksoftirqd/019:41:320
169809994231,8cyclictest26-21ksoftirqd/121:05:001
16980599425,19cyclictest13-21ksoftirqd/022:55:220
16980599422,7cyclictest1753-21gnome-shell22:19:580
16980599422,7cyclictest1753-21gnome-shell20:42:560
169805994223,5cyclictest141rcu_preempt00:39:580
169805994220,6cyclictest141rcu_preempt20:49:280
169805994214,9cyclictest13-21ksoftirqd/020:15:160
16980599421,13cyclictest13-21ksoftirqd/000:05:310
162399420,30rtkit-daemon1753-21gnome-shell20:43:461
162399420,29rtkit-daemon2197014-21PK-Backend20:53:501
16981199415,32cyclictest222731-21missed_timers21:35:262
16981199414,17cyclictest1753-21gnome-shell20:22:122
169811994112,25cyclictest0-21swapper/220:40:032
16980999411,9cyclictest0-21swapper/123:03:421
169809994118,19cyclictest1753-21gnome-shell00:13:151
169809994116,21cyclictest219632-21sed21:25:291
16980599418,19cyclictest13-21ksoftirqd/022:05:010
16980599418,18cyclictest13-21ksoftirqd/000:20:280
169805994130,7cyclictest1753-21gnome-shell21:13:580
169805994120,7cyclictest13-21ksoftirqd/023:30:360
169805994120,5cyclictest141rcu_preempt22:34:060
169805994115,10cyclictest13-21ksoftirqd/019:55:180
169805994110,16cyclictest13-21ksoftirqd/022:38:500
162399410,24rtkit-daemon0-21swapper/121:51:071
169809994021,15cyclictest1069-21runrttasks23:18:391
16980999401,9cyclictest0-21swapper/100:31:321
16980599408,17cyclictest13-21ksoftirqd/020:10:200
16980599405,15cyclictest252782-21pool-udisksd23:01:560
169805994017,9cyclictest13-21ksoftirqd/023:45:120
169809993921,14cyclictest3242253-21nautilus00:25:021
16980999391,8cyclictest0-21swapper/120:47:151
169809993915,20cyclictest26-21ksoftirqd/100:03:081
169809993915,20cyclictest26-21ksoftirqd/100:03:071
16980999391,16cyclictest1924302open_files20:10:301
16980999391,10cyclictest0-21swapper/122:24:421
162399390,27rtkit-daemon2197014-21PK-Backend23:07:541
16981599382,17cyclictest1753-21gnome-shell22:26:403
169809993824,10cyclictest226811-21sshd21:47:231
16980999382,32cyclictest1753-21gnome-shell22:18:131
169809993822,12cyclictest285738-21sshd00:35:321
169809993822,11cyclictest438-20systemd-journal20:15:281
169809993820,14cyclictest26-21ksoftirqd/121:36:201
169809993812,9cyclictest141rcu_preempt22:55:141
16980999381,13cyclictest208832-21awk20:55:301
162399380,26rtkit-daemon0-21swapper/121:18:101
169809993722,11cyclictest247120-21sh22:45:251
169809993721,12cyclictest261139-21memory23:25:241
169809993720,13cyclictest216615-21cron21:24:551
162399370,25rtkit-daemon2197014-21PK-Backend22:33:341
162399370,25rtkit-daemon0-21swapper/122:04:151
162399370,24rtkit-daemon0-21swapper/123:33:091
169809993626,7cyclictest1069-21runrttasks19:53:331
169809993626,7cyclictest1069-21runrttasks19:53:321
169809993617,15cyclictest26-21ksoftirqd/119:39:361
16980999361,5cyclictest231irq_work/122:11:391
162399360,24rtkit-daemon0-21swapper/121:56:251
162399360,24rtkit-daemon0-21swapper/121:56:251
169809993524,7cyclictest1753-21gnome-shell22:05:331
169809993522,9cyclictest1753-21gnome-shell20:24:141
169809993520,11cyclictest269777-21sort23:50:211
169809993520,11cyclictest243785-21swap22:35:291
169809993519,12cyclictest225021-21seq21:41:401
169809993513,18cyclictest26-21ksoftirqd/120:07:171
16980999351,11cyclictest181183-21/usr/sbin/munin19:40:241
162399350,24rtkit-daemon0-21swapper/123:36:151
16980999349,21cyclictest1-21systemd21:32:321
169809993426,5cyclictest0-21swapper/119:11:321
169809993426,5cyclictest0-21swapper/119:11:311
169809993423,6cyclictest26-21ksoftirqd/119:25:171
169809993422,9cyclictest189451-21sshd20:02:301
162399340,24rtkit-daemon0-21swapper/120:25:501
162399340,23rtkit-daemon2197014-21PK-Backend23:41:291
162399340,23rtkit-daemon2197014-21PK-Backend23:41:281
162399340,22rtkit-daemon0-21swapper/123:20:271
16980999332,8cyclictest0-21swapper/100:25:201
169809993322,7cyclictest183098-21cpuspeed_turbos19:45:141
169809993319,10cyclictest255761-21sort23:10:201
169809993318,11cyclictest173316-21vmstat19:15:291
169809993317,12cyclictest274813-21cstates00:05:151
169809993315,13cyclictest244640-21awk22:40:131
16980999331,12cyclictest141rcu_preempt22:25:141
16980999331,11cyclictest141rcu_preempt00:15:251
169809993220,9cyclictest1753-21gnome-shell19:20:171
16980999321,28cyclictest2197014-21PK-Backend23:45:461
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional