You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Wed Aug 27, 2025 00:44:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
162399980,17rtkit-daemon0-21swapper/121:27:520
29078482950,8sleep02197014-21PK-Backend23:25:190
281646999932,7cyclictest0-21swapper/222:58:192
281646999932,6cyclictest0-21swapper/223:07:232
281646999922,87cyclictest2197014-21PK-Backend23:11:172
281646999922,87cyclictest0-21swapper/223:16:542
28681782910,17sleep013-21ksoftirqd/021:35:030
28681782910,17sleep013-21ksoftirqd/021:35:020
2816469999183,5cyclictest0-21swapper/223:21:142
2816469999182,5cyclictest2909970-21strings23:30:232
281646999912,7cyclictest0-21swapper/200:33:452
2816469999083,5cyclictest0-21swapper/223:42:252
2816469999083,5cyclictest0-21swapper/223:26:352
2816469999082,5cyclictest0-21swapper/223:51:152
2816469999082,5cyclictest0-21swapper/223:39:492
2816469999082,5cyclictest0-21swapper/223:04:052
2816469999082,5cyclictest0-21swapper/222:49:482
2816469999081,5cyclictest2197014-21PK-Backend23:46:582
2816469998981,5cyclictest0-21swapper/200:37:542
2816469998981,5cyclictest0-21swapper/200:05:162
2816469998980,5cyclictest0-21swapper/200:25:082
2816469998779,5cyclictest0-21swapper/200:11:142
281646999862,19cyclictest1753-21gnome-shell00:23:192
2816463998512,53cyclictest1753-21gnome-shell00:25:370
2816472998436,44cyclictest1753-21gnome-shell00:26:493
281646999842,16cyclictest1753-21gnome-shell00:02:182
2816472998372,7cyclictest1753-21gnome-shell23:18:203
28994882820,4chrt2898815-21/usr/sbin/munin23:00:241
2816466998210,19cyclictest26-21ksoftirqd/100:10:121
2816472998170,7cyclictest1753-21gnome-shell00:23:053
2816472998133,44cyclictest1753-21gnome-shell00:38:203
2816472998131,46cyclictest1753-21gnome-shell23:39:223
281646699814,12cyclictest26-21ksoftirqd/100:38:231
281646399802,55cyclictest1753-21gnome-shell23:45:160
281646399798,8cyclictest141rcu_preempt23:53:210
281646399791,14cyclictest0-21swapper/000:33:200
281614827964,9sleep20-21swapper/219:07:342
162399790,7rtkit-daemon1753-21gnome-shell22:51:020
281646999782,56cyclictest1753-21gnome-shell22:50:242
281646399782,55cyclictest1753-21gnome-shell23:03:210
281646399772,15cyclictest1753-21gnome-shell00:15:480
281646399771,55cyclictest1753-21gnome-shell00:11:530
162399770,5rtkit-daemon0-21swapper/000:08:440
2816472997626,9cyclictest1753-21gnome-shell23:53:003
281646999762,12cyclictest1753-21gnome-shell00:15:362
2816463997640,8cyclictest1753-21gnome-shell23:11:000
281646399762,47cyclictest1753-21gnome-shell00:21:370
281646399762,45cyclictest1753-21gnome-shell23:30:150
281646399762,12cyclictest1753-21gnome-shell23:09:110
281646399762,10cyclictest1753-21gnome-shell23:41:260
2816463997610,13cyclictest141rcu_preempt22:49:460
162399760,31rtkit-daemon1622-21rtkit-daemon00:16:523
28701802750,5sleep12870184-21latency_hist21:40:021
2816472997564,7cyclictest1753-21gnome-shell00:02:043
2816463997539,7cyclictest2918043-21cpu23:55:140
281646399753,10cyclictest1753-21gnome-shell23:15:360
281646399752,42cyclictest0-21swapper/000:35:030
281646399752,10cyclictest1753-21gnome-shell22:55:530
162399750,6rtkit-daemon2197014-21PK-Backend20:49:230
162399750,5rtkit-daemon0-21swapper/020:43:500
162399750,5rtkit-daemon0-21swapper/019:56:290
281646999742,10cyclictest1753-21gnome-shell23:55:162
2816466997423,47cyclictest2914523-21/usr/sbin/munin23:45:191
162399740,6rtkit-daemon2197014-21PK-Backend20:20:030
162399740,5rtkit-daemon0-21swapper/019:10:590
162399740,5rtkit-daemon0-21swapper/019:10:580
281646399733,47cyclictest1753-21gnome-shell23:22:390
162399730,6rtkit-daemon2197014-21PK-Backend21:16:550
162399730,6rtkit-daemon2197014-21PK-Backend20:05:520
162399730,4rtkit-daemon0-21swapper/019:17:130
2816466997220,48cyclictest26-21ksoftirqd/123:56:571
281646399722,12cyclictest2921110-21taskset00:02:580
162399720,58rtkit-daemon0-21swapper/019:05:520
281646399718,7cyclictest141rcu_preempt23:37:410
2816472997055,5cyclictest42-21ksoftirqd/323:48:553
281646699703,7cyclictest141rcu_preempt23:40:021
281646699702,15cyclictest0-21swapper/123:18:241
2816466997016,44cyclictest26-21ksoftirqd/100:06:011
281615427054,10sleep30-21swapper/319:07:403
162399700,46rtkit-daemon2197014-21PK-Backend21:14:110
2816466996937,10cyclictest26-21ksoftirqd/100:30:281
2816472996858,6cyclictest42-21ksoftirqd/323:04:363
2816472996857,7cyclictest1753-21gnome-shell00:06:113
2816472996855,9cyclictest1753-21gnome-shell23:25:213
2816472996816,47cyclictest1753-21gnome-shell22:55:013
2816466996811,42cyclictest141rcu_preempt23:54:511
2816472996720,38cyclictest42-21ksoftirqd/323:08:583
281647299671,15cyclictest2197014-21PK-Backend00:10:173
281646699671,24cyclictest1753-21gnome-shell23:30:341
281610426739,22sleep10-21swapper/119:07:141
162399670,34rtkit-daemon1753-21gnome-shell22:51:291
281647299662,22cyclictest1753-21gnome-shell23:14:213
2816466996658,5cyclictest26-21ksoftirqd/122:57:291
2816466996658,5cyclictest26-21ksoftirqd/100:23:191
281647299658,6cyclictest141rcu_preempt00:32:433
2816472996535,8cyclictest141rcu_preempt23:30:273
281647299651,22cyclictest1753-21gnome-shell23:22:333
281646699652,59cyclictest1753-21gnome-shell23:10:341
2816472996433,6cyclictest141rcu_preempt23:57:223
281647299642,44cyclictest0-21swapper/322:57:463
281647299642,20cyclictest1753-21gnome-shell23:40:373
281646699643,47cyclictest1753-21gnome-shell00:25:511
2816472996329,18cyclictest141rcu_preempt22:49:273
281646699632,56cyclictest1753-21gnome-shell23:28:251
162399630,10rtkit-daemon1622-21rtkit-daemon22:31:043
2816466996240,17cyclictest1753-21gnome-shell23:43:011
281646699622,56cyclictest1753-21gnome-shell22:49:401
281646699612,54cyclictest2900777-21cpuspeed_turbos23:05:161
2816466996051,5cyclictest26-21ksoftirqd/100:15:431
281646699602,15cyclictest869-21irqbalance23:20:551
281646699588,8cyclictest2920321-21cut00:00:231
162399580,9rtkit-daemon1622-21rtkit-daemon22:38:563
162399580,9rtkit-daemon1622-21rtkit-daemon22:38:553
28889272570,7sleep126-21ksoftirqd/122:30:341
162399560,44rtkit-daemon881143-21gmain21:40:201
162399560,44rtkit-daemon881143-21gmain21:40:201
2816469995522,28cyclictest2827268-21cat19:35:332
281646999532,4cyclictest854-21avahi-daemon20:33:372
2816469995120,26cyclictest2844540-21sshd20:25:312
2816469995117,28cyclictest438-20systemd-journal21:31:542
2816469995117,28cyclictest438-20systemd-journal21:31:532
2816469995112,33cyclictest2879318-21grep22:05:152
2816469995019,26cyclictest2840313-21cpuspeed_turbos20:15:172
2816469995019,26cyclictest2840313-21cpuspeed_turbos20:15:162
2816469995015,30cyclictest2822931-21grep19:25:222
2816469995015,30cyclictest2822931-21grep19:25:222
2816469994921,24cyclictest2962348-21rs:main2
2816469994920,25cyclictest2876598-21awk21:55:292
2816469994917,28cyclictest2878511-21cat22:00:342
2816469994917,28cyclictest2878511-21cat22:00:342
2816469994917,28cyclictest1753-21gnome-shell21:00:222
2816469994913,32cyclictest2849455-21cat20:40:222
2816469994912,33cyclictest2874815-21cat21:50:292
28193782480,6sleep2331rcuc/219:15:222
2816469994814,30cyclictest2823870-21cron19:30:112
162399480,33rtkit-daemon2197014-21PK-Backend20:10:100
28221572470,8chrt1753-21gnome-shell19:25:002
281646999472,41cyclictest2829913-21ls19:45:162
281646999471,42cyclictest1753-21gnome-shell22:30:152
281646999471,42cyclictest1753-21gnome-shell21:23:062
281646999471,42cyclictest1753-21gnome-shell21:23:062
2816469994713,29cyclictest1753-21gnome-shell20:55:022
2816469994711,32cyclictest2885541-21cat22:20:332
2816469994618,24cyclictest855-21dbus-daemon20:10:022
2816469994614,28cyclictest2839199-21open_inodes20:10:302
2816469994613,29cyclictest2871085-21latency21:40:272
2816469994613,29cyclictest2871085-21latency21:40:262
2816466994620,22cyclictest2847152-21awk20:35:171
2816466994620,22cyclictest2847152-21awk20:35:171
2816463994624,17cyclictest3242253-21nautilus21:35:030
281646999459,31cyclictest2816894-21seq19:10:142
281646999459,31cyclictest2816894-21seq19:10:132
2816469994515,26cyclictest2873574-21sshd21:47:282
2816469994513,28cyclictest2884380-21latency_hist22:20:012
2816469994512,27cyclictest994-21lldpd20:55:192
2816466994528,13cyclictest1-21systemd20:10:011
162399450,7rtkit-daemon1358-1kworker/u9:121:45:291
162399450,31rtkit-daemon1753-21gnome-shell19:52:360
281646999449,31cyclictest2833493-21expr19:55:172
281646999449,30cyclictest1753-21gnome-shell20:20:172
281646999447,18cyclictest1753-21gnome-shell20:35:232
281646999447,18cyclictest1753-21gnome-shell20:35:222
2816469994417,23cyclictest2851097-21if_err_enp2s020:45:262
2816469994414,26cyclictest2861350-21cut21:15:162
2816469994413,27cyclictest2829000-21timerwakeupswit19:40:312
2816469994412,28cyclictest2889760-21/usr/sbin/munin22:35:212
2816469994412,28cyclictest2889760-21/usr/sbin/munin22:35:212
2816469994411,29cyclictest2869999-21grep21:40:002
2816469994410,30cyclictest1753-21gnome-shell21:05:362
2816466994427,14cyclictest1753-21gnome-shell22:06:511
281646399442,7cyclictest1753-21gnome-shell19:23:440
2816463994426,13cyclictest438-20systemd-journal22:38:540
2816463994426,13cyclictest438-20systemd-journal22:38:530
2816472994324,6cyclictest42-21ksoftirqd/321:20:023
281646999432,16cyclictest1753-21gnome-shell21:14:212
2816469994319,20cyclictest2882527-21latency_hist22:15:012
2816469994317,22cyclictest1069-21runrttasks19:52:022
2816466994320,19cyclictest1069-21runrttasks19:50:131
162399430,6rtkit-daemon0-21swapper/119:13:561
162399430,6rtkit-daemon0-21swapper/119:13:561
162399430,30rtkit-daemon1753-21gnome-shell22:13:161
162399430,29rtkit-daemon0-21swapper/020:30:240
2816469994216,21cyclictest34-21ksoftirqd/222:25:172
2816469994211,26cyclictest1069-21runrttasks21:25:182
281646699421,17cyclictest28511862wc20:45:271
2816472994126,11cyclictest855-21dbus-daemon21:21:023
2816472994126,11cyclictest855-21dbus-daemon21:21:023
2816472994114,23cyclictest2872232-21latency_hist21:45:033
2816472994114,23cyclictest2872232-21latency_hist21:45:023
2816469994115,23cyclictest2892906-21sshd22:44:052
2816466994128,9cyclictest1617-10pipewire-media-21:32:301
2816466994128,9cyclictest1617-10pipewire-media-21:32:301
2816472994026,10cyclictest2851751-21sshd20:45:433
2816463994028,8cyclictest2823276-21sensors19:25:290
2816463994028,8cyclictest2823276-21sensors19:25:290
281646399402,6cyclictest2823895-21apt-get19:30:040
28593132390,5sleep30-21swapper/321:10:013
28470592390,5sleep10-21swapper/120:35:031
28220852392,32sleep11753-21gnome-shell19:24:321
2816472993924,11cyclictest42-21ksoftirqd/322:18:423
281647299392,33cyclictest1753-21gnome-shell19:36:243
2816472993920,14cyclictest42-21ksoftirqd/319:45:013
2816472993915,19cyclictest2875331-21sshd21:52:323
2816466993922,13cyclictest2887771-21/usr/sbin/munin22:30:001
2816466993921,13cyclictest2891899-21df22:40:211
2816466993920,14cyclictest26-21ksoftirqd/122:35:171
2816466993920,14cyclictest26-21ksoftirqd/122:35:161
2816466993917,18cyclictest2864046-21systemd-udevd21:21:021
2816466993917,18cyclictest2864046-21systemd-udevd21:21:011
281646399392,33cyclictest1753-21gnome-shell22:28:400
2816463993922,12cyclictest438-20systemd-journal22:23:260
281646399391,14cyclictest2857397-21turbostat21:05:000
2816463993910,7cyclictest141rcu_preempt20:50:290
162399390,27rtkit-daemon2197014-21PK-Backend22:03:130
162399390,27rtkit-daemon2197014-21PK-Backend22:03:130
162399390,27rtkit-daemon2197014-21PK-Backend21:44:560
162399390,27rtkit-daemon2197014-21PK-Backend21:44:550
162399390,27rtkit-daemon0-21swapper/121:27:171
162399390,26rtkit-daemon0-21swapper/119:47:141
162399390,25rtkit-daemon0-21swapper/121:53:401
2816472993821,6cyclictest42-21ksoftirqd/321:36:503
2816472993821,13cyclictest2867188-21ntpq21:30:263
2816472993821,13cyclictest2867188-21ntpq21:30:263
2816472993820,14cyclictest2839171-21ntp_states20:10:293
2816472993815,5cyclictest141rcu_preempt20:30:153
2816472993814,19cyclictest2877087-21sshd21:57:433
281647299381,15cyclictest2830051-21taskset19:45:183
2816466993827,7cyclictest1753-21gnome-shell21:16:301
2816466993827,7cyclictest1753-21gnome-shell20:59:401
2816466993827,7cyclictest1753-21gnome-shell19:15:271
281646699382,32cyclictest1753-21gnome-shell21:08:331
2816466993820,12cyclictest2824686-21expr19:30:161
2816463993820,13cyclictest438-20systemd-journal22:33:290
2816463993816,17cyclictest2962348-21rs:main0
281646399381,33cyclictest1753-21gnome-shell19:42:370
162399380,26rtkit-daemon2197014-21PK-Backend20:03:530
281647299372,31cyclictest855-21dbus-daemon19:35:023
2816472993722,11cyclictest42-21ksoftirqd/320:53:123
281647299372,16cyclictest1753-21gnome-shell20:57:473
2816472993719,14cyclictest42-21ksoftirqd/320:03:353
2816472993715,19cyclictest391irq_work/322:43:163
2816466993726,7cyclictest1753-21gnome-shell19:25:401
2816466993726,7cyclictest1753-21gnome-shell19:25:391
2816466993725,8cyclictest1753-21gnome-shell20:21:011
2816466993720,12cyclictest438-20systemd-journal22:18:251
281646399379,6cyclictest141rcu_preempt20:55:260
281646399372,7cyclictest1753-21gnome-shell21:08:550
2816463993719,13cyclictest2892613-21cat22:40:380
162399370,25rtkit-daemon2197014-21PK-Backend20:16:250
162399370,25rtkit-daemon2197014-21PK-Backend20:16:250
28799872361,9sleep02197014-21PK-Backend22:05:260
281647299365,14cyclictest1753-21gnome-shell19:51:303
2816472993622,10cyclictest1753-21gnome-shell21:14:173
2816472993621,11cyclictest2881756-21awk22:10:233
2816472993621,11cyclictest2856989-21cat21:00:333
2816472993613,6cyclictest391irq_work/320:37:013
2816472993613,6cyclictest391irq_work/320:37:013
281647299361,31cyclictest2823213-21ps19:25:283
281647299361,31cyclictest2823213-21ps19:25:283
281647299361,30cyclictest1753-21gnome-shell22:05:083
281647299361,21cyclictest175350gnome-shell22:20:263
281647299361,20cyclictest175350gnome-shell19:10:233
281647299361,20cyclictest175350gnome-shell19:10:233
2816472993610,22cyclictest0-21swapper/320:15:173
2816472993610,22cyclictest0-21swapper/320:15:173
2816466993626,7cyclictest1753-21gnome-shell19:56:351
2816466993625,7cyclictest1753-21gnome-shell22:00:311
2816466993625,7cyclictest1753-21gnome-shell22:00:311
2816466993625,7cyclictest1753-21gnome-shell20:25:191
2816466993625,7cyclictest1753-21gnome-shell20:11:471
2816466993617,16cyclictest1753-21gnome-shell22:20:331
2816466993610,8cyclictest26-21ksoftirqd/120:50:301
2816463993622,10cyclictest13-21ksoftirqd/022:15:190
2816463993620,11cyclictest2863150-21expr21:20:160
2816463993620,11cyclictest2863150-21expr21:20:160
2816463993613,12cyclictest141rcu_preempt20:35:210
2816463993613,12cyclictest141rcu_preempt20:35:210
162399360,25rtkit-daemon0-21swapper/119:37:131
162399360,24rtkit-daemon0-21swapper/120:16:181
162399360,24rtkit-daemon0-21swapper/120:16:181
2816472993525,6cyclictest42-21ksoftirqd/319:19:323
2816472993523,8cyclictest1069-21runrttasks20:25:433
2816472993520,8cyclictest1753-21gnome-shell21:45:513
2816472993519,12cyclictest42-21ksoftirqd/321:25:173
2816472993518,6cyclictest42-21ksoftirqd/320:05:153
281647299351,15cyclictest1753-21gnome-shell19:55:223
2816466993524,7cyclictest2829441-21cut19:45:001
2816466993519,12cyclictest2848984-21sort20:40:131
2816466993519,12cyclictest2835700-21open_inodes20:00:281
2816466993519,11cyclictest438-20systemd-journal21:04:591
2816463993523,7cyclictest1753-21gnome-shell19:48:390
2816463993518,12cyclictest2845165-21pluginstate20:30:010
281646399351,31cyclictest0-21swapper/021:59:030
2816472993416,13cyclictest438-20systemd-journal22:05:013
2816472993416,13cyclictest438-20systemd-journal22:05:003
281647299341,13cyclictest2842792-21python320:20:313
281646699348,5cyclictest141rcu_preempt21:55:191
2816466993419,10cyclictest2860282-21sshd21:10:301
281646399342,28cyclictest0-21swapper/019:39:590
2816472993318,12cyclictest2887776-21awk22:29:593
2816472993316,12cyclictest42-21ksoftirqd/320:40:163
2816463993319,10cyclictest2874332-21expr21:50:190
2816472993214,13cyclictest42-21ksoftirqd/319:20:213
2816463993219,9cyclictest2873914-21wc21:50:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional