You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 10:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Dec 05, 2025 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109375921960,5sleep00-21swapper/019:05:180
113508121020,7sleep22197014-21PK-Backend21:00:302
162399840,5rtkit-daemon2197014-21PK-Backend23:12:590
162399800,9rtkit-daemon869-21irqbalance23:28:540
162399800,5rtkit-daemon0-21swapper/123:15:051
109448427864,9sleep10-21swapper/119:06:401
162399770,6rtkit-daemon0-21swapper/021:45:570
109453127447,17sleep30-21swapper/319:07:193
162399690,25rtkit-daemon0-21swapper/023:36:110
109489299672,7cyclictest1753-21gnome-shell20:52:252
109489299662,7cyclictest1753-21gnome-shell19:45:392
109464626648,13sleep20-21swapper/219:08:422
109489299652,7cyclictest0-21swapper/200:04:362
109489299642,6cyclictest0-21swapper/221:44:362
109489299632,8cyclictest1753-21gnome-shell22:47:332
109489299632,7cyclictest1753-21gnome-shell20:11:222
109489299631,18cyclictest1753-21gnome-shell21:45:522
162399620,9rtkit-daemon1622-21rtkit-daemon20:02:371
162399620,9rtkit-daemon1622-21rtkit-daemon19:42:220
109489299622,7cyclictest1753-21gnome-shell22:27:012
109489299612,8cyclictest1753-21gnome-shell20:25:032
109489299612,7cyclictest1753-21gnome-shell21:50:172
109489299612,7cyclictest1753-21gnome-shell20:19:102
109489299612,7cyclictest1753-21gnome-shell00:33:232
162399600,9rtkit-daemon1622-21rtkit-daemon20:29:561
109489299602,8cyclictest1753-21gnome-shell23:40:482
109489299602,8cyclictest1753-21gnome-shell23:21:082
1094888996017,13cyclictest438-20systemd-journal19:18:261
162399590,9rtkit-daemon1622-21rtkit-daemon19:56:150
109489299592,8cyclictest1753-21gnome-shell23:56:352
109489299592,16cyclictest1753-21gnome-shell22:09:242
109489299591,15cyclictest1753-21gnome-shell20:06:122
1094888995920,11cyclictest1171531-21cat22:45:211
11012332580,8chrt2197014-21PK-Backend19:25:202
109489299582,8cyclictest1753-21gnome-shell23:15:072
109489299582,8cyclictest1753-21gnome-shell22:21:432
109489299582,8cyclictest1753-21gnome-shell19:12:272
109489299582,7cyclictest0-21swapper/219:31:592
1094892995816,36cyclictest1200971-21cat00:05:332
1094888995820,13cyclictest1124349-21which20:30:261
1094888995813,21cyclictest1135952-21taskset21:05:021
162399570,8rtkit-daemon1622-21rtkit-daemon00:37:062
109489299572,8cyclictest1753-21gnome-shell20:43:072
109489299572,7cyclictest1753-21gnome-shell19:43:252
1094888995714,21cyclictest1177106-40lsb_release23:00:241
109488399571,14cyclictest2197014-21PK-Backend23:25:010
162399560,42rtkit-daemon1753-21gnome-shell20:41:291
109489299562,8cyclictest1753-21gnome-shell19:53:232
109488899562,16cyclictest1121601-40lsb_release20:21:211
109488899562,16cyclictest1121601-40lsb_release20:21:211
109489299554,46cyclictest1140358-21grep21:15:312
109489299553,47cyclictest1186549-21/usr/sbin/munin23:25:242
109489299551,14cyclictest860997-21snmpd00:28:212
1094888995519,12cyclictest1192906-21cat23:45:161
162399540,9rtkit-daemon1622-21rtkit-daemon19:53:140
109489299542,7cyclictest1753-21gnome-shell20:55:032
1094892995412,37cyclictest3242255-21gdbus23:15:022
1094892995411,38cyclictest438-20systemd-journal19:35:252
109489299540,49cyclictest1622-21rtkit-daemon00:14:302
1094888995414,18cyclictest1178754-21if_enp2s023:05:231
1094892995320,28cyclictest1147444-21timerandwakeup21:35:292
109489299530,48cyclictest1622-21rtkit-daemon21:23:382
162399520,9rtkit-daemon1622-21rtkit-daemon19:33:580
162399520,8rtkit-daemon1622-21rtkit-daemon19:39:191
109489299522,8cyclictest1753-21gnome-shell23:00:302
1094892995218,29cyclictest1205010-21latency_hist00:20:012
1094892995211,35cyclictest1111490-21expr19:55:172
1094892995210,37cyclictest884-21gdbus23:55:012
162399510,9rtkit-daemon1622-21rtkit-daemon19:46:101
11965562510,5chrt0-21swapper/123:55:171
109489299519,17cyclictest1753-21gnome-shell23:34:142
109489299518,17cyclictest1753-21gnome-shell22:14:242
1094892995117,29cyclictest1127147-21latency_hist20:40:022
1094892995116,31cyclictest1130188-21timerwakeupswit20:45:322
1094883995130,6cyclictest141rcu_preempt20:55:130
109489299502,43cyclictest1753-21gnome-shell22:15:232
1094892995017,29cyclictest2197014-21PK-Backend20:23:162
1094892995017,29cyclictest2197014-21PK-Backend20:23:162
1094892995017,28cyclictest34-21ksoftirqd/221:30:202
1094892995015,29cyclictest1097428-21cpuspeed_turbos19:15:132
109489299499,16cyclictest1753-21gnome-shell21:08:242
109489299499,16cyclictest1753-21gnome-shell21:08:232
109489299498,17cyclictest1753-21gnome-shell23:09:202
109489299498,16cyclictest1753-21gnome-shell00:20:122
109489299494,16cyclictest1753-21gnome-shell20:31:382
1094892994920,24cyclictest1173023-21cstates22:50:152
1094892994917,28cyclictest34-21ksoftirqd/220:01:512
109488399491,21cyclictest13-21ksoftirqd/022:15:230
1094892994814,29cyclictest1143637-21missed_timers21:25:252
109488399487,21cyclictest13-21ksoftirqd/000:06:260
1094893994735,8cyclictest1114422-21unin-run20:05:013
1094893994725,17cyclictest3242253-21nautilus23:00:003
109489399471,18cyclictest42-21ksoftirqd/300:10:183
109489299470,42cyclictest1622-21rtkit-daemon23:46:192
1094883994722,8cyclictest13-21ksoftirqd/022:49:360
1094893994632,10cyclictest1110443-21grep19:50:313
109489299462,17cyclictest1753-21gnome-shell21:59:092
1094892994613,29cyclictest2197014-21PK-Backend22:42:412
109488399465,19cyclictest13-21ksoftirqd/000:00:030
1094883994624,9cyclictest13-21ksoftirqd/022:30:250
162399450,5rtkit-daemon0-21swapper/123:42:321
1094893994533,8cyclictest1170079-21grep22:40:273
109489299452,17cyclictest1753-21gnome-shell22:01:082
1094888994527,13cyclictest1170094-21ntp_states22:40:271
1094888994522,20cyclictest1162173-21unin-run22:20:001
1094883994522,6cyclictest141rcu_preempt19:39:440
11900872440,19chrt42-21ksoftirqd/323:35:283
11080982440,5sleep00-21swapper/019:45:180
1094893994421,18cyclictest1117073-21/usr/sbin/munin20:10:193
1094893994415,24cyclictest438-20systemd-journal21:15:293
109489299449,31cyclictest2197014-21PK-Backend19:23:392
109489299449,31cyclictest2197014-21PK-Backend19:23:382
109489299441,13cyclictest1166212-21expr22:30:192
109489299441,12cyclictest860997-21snmpd21:14:202
1094883994432,8cyclictest1753-21gnome-shell22:41:570
1094883994428,12cyclictest1114378-21apt-get20:05:000
1094883994418,8cyclictest13-21ksoftirqd/000:25:270
162399430,36rtkit-daemon906-21gmain21:09:193
162399430,36rtkit-daemon906-21gmain21:09:193
162399430,31rtkit-daemon2197014-21PK-Backend23:02:450
11735862430,8sleep02197014-21PK-Backend22:50:280
11203922430,17chrt42-21ksoftirqd/320:20:073
11203922430,17chrt42-21ksoftirqd/320:20:073
11045582430,6chrt0-21swapper/319:35:143
109489399437,32cyclictest1753-21gnome-shell21:30:183
1094893994316,8cyclictest1753-21gnome-shell21:47:123
109489299432,9cyclictest1753-21gnome-shell23:40:012
109488399438,18cyclictest13-21ksoftirqd/019:30:010
1094883994325,8cyclictest13-21ksoftirqd/021:50:190
1094883994314,16cyclictest13-21ksoftirqd/000:20:160
1094893994227,11cyclictest1116363-21sshd20:09:253
1094893994215,23cyclictest1753-21gnome-shell20:44:123
109488399424,18cyclictest13-21ksoftirqd/021:40:130
1094883994222,15cyclictest3242253-21nautilus22:10:010
1094883994221,6cyclictest141rcu_preempt22:28:200
1094883994219,20cyclictest13-21ksoftirqd/019:10:140
1094883994216,13cyclictest141rcu_preempt23:08:380
162399410,28rtkit-daemon1151690-21apt21:50:111
1094893994122,6cyclictest42-21ksoftirqd/323:15:173
1094893994121,16cyclictest1176712-21cpuspeed_turbos23:00:143
1094893994117,20cyclictest1191536-21interrupts23:40:203
109489299412,9cyclictest1753-21gnome-shell22:55:282
1094888994117,19cyclictest1200767-21grep00:05:251
1094888994115,21cyclictest1099041-21cat19:20:111
1094888994115,21cyclictest1099041-21cat19:20:111
1094883994124,12cyclictest1121920-21cron20:25:010
1094883994124,12cyclictest1121920-21cron20:25:000
1094883994121,6cyclictest141rcu_preempt21:08:460
1094883994121,6cyclictest141rcu_preempt21:08:460
109488399411,20cyclictest13-21ksoftirqd/021:20:010
1094893994028,8cyclictest42-21ksoftirqd/300:05:023
1094893994023,6cyclictest42-21ksoftirqd/300:25:003
1094893994019,12cyclictest42-21ksoftirqd/321:55:023
1094893994017,20cyclictest1753-21gnome-shell22:02:493
1094888994026,9cyclictest1129797-21tail20:45:241
1094888994021,14cyclictest1204402-21sensors_temp00:15:281
1094888994018,7cyclictest1159119-21kernelversion22:10:241
1094888994016,20cyclictest1212350-21cat00:40:031
109488899401,13cyclictest438-20systemd-journal21:40:001
1094883994026,9cyclictest13-21ksoftirqd/020:45:010
1094883994011,16cyclictest13-21ksoftirqd/023:30:440
162399390,26rtkit-daemon2197014-21PK-Backend21:58:331
12105922390,4sleep10-21swapper/100:35:011
109489399398,13cyclictest1753-21gnome-shell20:59:003
1094893993922,13cyclictest1-21systemd20:50:013
1094893993920,7cyclictest42-21ksoftirqd/300:05:313
1094893993916,19cyclictest1123928-21acpi20:30:173
1094893993915,20cyclictest391irq_work/321:36:273
1094888993922,12cyclictest438-20systemd-journal00:21:271
109488899391,9cyclictest1074983-21kworker/1:219:44:011
109488399398,12cyclictest13-21ksoftirqd/021:35:120
109488399397,18cyclictest13-21ksoftirqd/021:28:320
1094883993922,6cyclictest13-21ksoftirqd/020:31:260
1094883993922,12cyclictest438-20systemd-journal23:50:000
1094883993920,16cyclictest13-21ksoftirqd/000:15:470
109488399391,8cyclictest1753-21gnome-shell22:05:000
1094883993918,6cyclictest141rcu_preempt20:18:100
1094883993910,12cyclictest13-21ksoftirqd/020:51:190
109488399390,36cyclictest1622-21rtkit-daemon19:23:290
109488399390,36cyclictest1622-21rtkit-daemon19:23:280
162399380,26rtkit-daemon2197014-21PK-Backend23:16:540
109489399382,32cyclictest1162891-21irqstats22:20:223
109489399382,30cyclictest1753-21gnome-shell22:45:093
1094893993820,13cyclictest438-20systemd-journal21:25:003
1094893993820,13cyclictest42-21ksoftirqd/322:10:003
1094893993819,15cyclictest1753-21gnome-shell19:32:263
109489299381,6cyclictest11688982sleep222:37:542
1094888993819,16cyclictest1753-21gnome-shell21:05:251
1094888993819,16cyclictest1753-21gnome-shell21:05:251
1094888993817,7cyclictest1069-21runrttasks22:28:541
1094888993811,23cyclictest1131275-21hddtemp_smartct20:50:221
109488399385,6cyclictest141rcu_preempt22:14:250
109488399385,16cyclictest13-21ksoftirqd/020:05:200
109488399383,17cyclictest13-21ksoftirqd/020:10:170
1094883993828,6cyclictest13-21ksoftirqd/023:40:480
109488399382,7cyclictest1753-21gnome-shell19:19:010
1094883993824,10cyclictest13-21ksoftirqd/000:35:080
109488399382,32cyclictest1753-21gnome-shell22:35:070
1094883993822,12cyclictest1129392-21cat20:45:150
109488399382,16cyclictest1358-1kworker/u9:121:01:570
1094883993821,13cyclictest13-21ksoftirqd/023:51:190
1094883993821,13cyclictest13-21ksoftirqd/000:31:430
1094883993821,12cyclictest884-21gdbus00:00:030
1094883993811,9cyclictest13-21ksoftirqd/022:23:010
1094893993726,8cyclictest1753-21gnome-shell23:53:323
1094893993726,7cyclictest1753-21gnome-shell22:54:583
1094893993724,9cyclictest1753-21gnome-shell23:20:033
109489399372,31cyclictest1753-21gnome-shell23:11:583
109489399372,13cyclictest3242253-21nautilus00:35:013
1094893993717,10cyclictest42-21ksoftirqd/321:10:173
1094893993713,7cyclictest141rcu_preempt20:37:133
1094893993713,6cyclictest391irq_work/320:25:343
1094888993727,6cyclictest1102702-21tr19:30:131
1094888993726,7cyclictest1133011-21hddtemp_smartct20:55:211
1094888993722,11cyclictest1203251-21cron00:15:011
1094888993721,12cyclictest1119892-21apt-get20:20:001
1094888993720,12cyclictest1169083-21cut22:40:001
1094888993715,18cyclictest1117181-21kernelversion20:10:241
109488899371,15cyclictest11017662/usr/sbin/munin19:25:291
1094883993725,8cyclictest1753-21gnome-shell22:55:440
109488399372,31cyclictest1753-21gnome-shell21:32:530
1094883993721,6cyclictest13-21ksoftirqd/021:59:290
1094883993720,6cyclictest13-21ksoftirqd/021:22:370
109489399364,28cyclictest1134327-21sort21:00:173
109489399363,13cyclictest438-20systemd-journal23:09:283
1094893993621,11cyclictest1196394-21grep23:55:173
1094893993612,20cyclictest0-21swapper/323:32:353
109488899362,8cyclictest1-21systemd22:35:021
1094888993622,10cyclictest26-21ksoftirqd/122:00:281
1094888993622,10cyclictest1753-21gnome-shell23:36:581
1094888993619,13cyclictest1163154-21ntp_states22:20:261
1094888993618,13cyclictest1116454-21systemd-run20:10:011
1094883993613,6cyclictest141rcu_preempt21:13:350
109488399361,13cyclictest13-21ksoftirqd/000:12:190
162399350,24rtkit-daemon0-21swapper/123:21:471
1094893993524,8cyclictest1069-21runrttasks19:25:263
1094893993523,8cyclictest1753-21gnome-shell22:26:573
1094893993522,9cyclictest1753-21gnome-shell21:43:063
1094893993522,9cyclictest1753-21gnome-shell20:53:343
1094893993520,11cyclictest1095981-21wc19:10:303
1094893993520,10cyclictest1162337-21wc22:20:023
1094893993519,12cyclictest1111896-21awk19:55:263
1094893993518,13cyclictest3242254-21gmain19:25:023
1094893993518,13cyclictest3242254-21gmain19:25:013
1094893993518,12cyclictest438-20systemd-journal00:30:013
1094893993510,21cyclictest3242253-21nautilus23:30:003
109488899358,11cyclictest1069-21runrttasks23:13:051
109488899355,8cyclictest26-21ksoftirqd/119:52:051
1094888993526,6cyclictest26-21ksoftirqd/122:59:591
1094888993523,7cyclictest1-21systemd22:55:001
1094888993520,10cyclictest1198197-21taskset00:00:161
1094888993519,12cyclictest1144428-21seq21:29:301
1094888993519,12cyclictest1126917-21sshd20:37:261
1094888993518,13cyclictest1069-21runrttasks21:16:221
109488399352,17cyclictest13-21ksoftirqd/020:25:260
109488399351,19cyclictest13-21ksoftirqd/020:36:170
1094893993424,6cyclictest42-21ksoftirqd/300:20:013
109489399342,27cyclictest1753-21gnome-shell22:10:193
1094893993420,11cyclictest42-21ksoftirqd/319:45:313
1094893993419,11cyclictest42-21ksoftirqd/300:35:263
1094893993418,12cyclictest1168048-21if_enp2s022:35:213
1094893993417,13cyclictest1097508-21cstates19:15:133
109489399341,30cyclictest0-21swapper/319:44:183
1094893993412,19cyclictest42-21ksoftirqd/323:45:253
1094888993420,10cyclictest1146257-21wc21:35:011
1094888993419,11cyclictest1095198-21cat19:10:131
1094888993419,10cyclictest1151607-21cat21:50:031
1094888993418,11cyclictest1156980-21ls22:05:161
1094888993417,12cyclictest1195188-21tr23:50:231
1094893993323,5cyclictest42-21ksoftirqd/322:00:013
1094893993322,7cyclictest1096597-21munin-node22:30:143
1094893993320,9cyclictest1143828-21ps21:25:303
109488899338,6cyclictest141rcu_preempt00:25:281
1094888993319,11cyclictest1142157-21python321:20:301
1094888993317,12cyclictest1138786-21sed21:10:291
1094888993316,12cyclictest1187675-21cat23:30:141
109489399321,27cyclictest2197014-21PK-Backend20:16:563
109488899321,28cyclictest0-21swapper/119:58:231
1094888993119,8cyclictest1187341-21perf23:30:011
109488899291,24cyclictest1753-21gnome-shell21:40:281
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional