You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 11:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Nov 16, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
284835322630,8sleep32848318-21munin-run19:10:003
289288421100,17sleep126-21ksoftirqd/121:14:551
284844999932,87cyclictest0-21swapper/319:42:203
284844999921,19cyclictest1753-21gnome-shell19:58:503
284844999912,85cyclictest0-21swapper/320:01:123
284844999912,85cyclictest0-21swapper/320:01:123
28869192890,5sleep32197014-21PK-Backend20:55:293
2848446998880,5cyclictest0-21swapper/219:42:202
2848446998779,5cyclictest0-21swapper/220:01:122
2848446998779,5cyclictest0-21swapper/220:01:122
284844199872,17cyclictest1753-21gnome-shell19:15:211
2848441998576,5cyclictest2197014-21PK-Backend19:31:021
284844999822,22cyclictest1753-21gnome-shell19:47:223
284844999822,21cyclictest1753-21gnome-shell19:24:213
284844999822,10cyclictest1753-21gnome-shell19:50:213
284844999821,17cyclictest1753-21gnome-shell19:36:103
284844999812,18cyclictest1753-21gnome-shell19:18:343
284844999802,18cyclictest1753-21gnome-shell19:14:423
284801728048,22sleep10-21swapper/119:06:211
284844999792,10cyclictest1753-21gnome-shell19:28:383
284844999792,10cyclictest1753-21gnome-shell19:28:383
2848437997955,8cyclictest13-21ksoftirqd/019:15:000
284844999772,13cyclictest1753-21gnome-shell19:30:073
284844199773,47cyclictest0-21swapper/119:14:121
2848437997721,52cyclictest13-21ksoftirqd/019:43:410
2848446997669,4cyclictest0-21swapper/219:11:422
284844199762,9cyclictest1753-21gnome-shell19:40:271
2848437997626,7cyclictest13-21ksoftirqd/020:00:410
2848437997626,7cyclictest13-21ksoftirqd/020:00:410
2848437997521,50cyclictest13-21ksoftirqd/019:18:290
2848437997519,14cyclictest13-21ksoftirqd/019:35:390
284800827547,18sleep20-21swapper/219:06:142
284844699742,67cyclictest2863147-21sed19:50:022
284844199742,7cyclictest0-21swapper/120:07:021
284843799747,7cyclictest141rcu_preempt19:25:180
284843799747,7cyclictest141rcu_preempt19:25:180
2848437997419,13cyclictest13-21ksoftirqd/019:54:530
284810727442,22sleep00-21swapper/019:07:290
284844199733,10cyclictest1753-21gnome-shell20:03:021
284844199733,10cyclictest1753-21gnome-shell20:03:021
2848441997325,42cyclictest2858440-21expr19:35:171
2848437997318,50cyclictest13-21ksoftirqd/019:20:440
2848437997317,14cyclictest13-21ksoftirqd/019:58:480
2848441997264,5cyclictest0-21swapper/119:51:341
2848437997162,5cyclictest13-21ksoftirqd/019:45:360
2848437997020,8cyclictest1753-21gnome-shell20:05:180
2848441996940,25cyclictest1753-21gnome-shell19:20:531
284844199692,46cyclictest1753-21gnome-shell19:55:331
284844199692,33cyclictest2862373-21ntp_states19:45:281
2848441996914,51cyclictest855-21dbus-daemon19:25:281
2848441996914,51cyclictest855-21dbus-daemon19:25:281
284844699673,9cyclictest2855847-21munin-run19:30:012
284844699673,9cyclictest2855847-21munin-run19:30:012
284844699672,22cyclictest1753-21gnome-shell19:55:162
2848437996758,5cyclictest13-21ksoftirqd/019:31:410
2848446996655,7cyclictest2857115-21timerandwakeup19:30:312
29333712650,16sleep234-21ksoftirqd/223:05:202
284844699642,57cyclictest1753-21gnome-shell19:52:092
284844999632,58cyclictest0-21swapper/320:05:233
284844699632,49cyclictest2868719-21apt-config20:05:022
284844699632,20cyclictest1753-21gnome-shell19:36:162
284844699632,19cyclictest1753-21gnome-shell19:23:032
284844699611,20cyclictest1753-21gnome-shell19:16:502
2848449996010,32cyclictest1753-21gnome-shell00:09:213
284844199602,54cyclictest1753-21gnome-shell21:44:551
28901632570,16chrt26-21ksoftirqd/121:05:211
2848441995718,10cyclictest2907987-21basename21:55:131
2848441995716,11cyclictest2904485-21expr21:45:161
2848441995713,11cyclictest2889310-21systemd-udevd21:03:561
162399570,8rtkit-daemon1622-21rtkit-daemon22:44:381
29419532560,7sleep12197014-21PK-Backend23:30:191
28987322560,8sleep31753-21gnome-shell21:30:143
284844699562,15cyclictest1753-21gnome-shell20:35:022
2848441995615,12cyclictest884-21gdbus21:17:011
2848441995615,12cyclictest884-21gdbus21:17:011
284844199561,4cyclictest2197014-21PK-Backend21:38:591
162399560,8rtkit-daemon1622-21rtkit-daemon22:16:351
2848449995516,34cyclictest2875575-21grep20:25:013
284844699554,27cyclictest1753-21gnome-shell00:33:412
2848441995519,12cyclictest2951891-21ls00:00:031
2848441995516,11cyclictest2960576-21gpgv00:25:031
2848449995421,27cyclictest438-20systemd-journal23:43:363
2848449995421,27cyclictest438-20systemd-journal23:43:363
2848441995415,9cyclictest2943495-21tr23:35:161
2848441995415,11cyclictest438-20systemd-journal21:24:591
2848441995415,10cyclictest2949327-21which23:50:281
2848441995414,10cyclictest2964709-21cat00:35:161
162399540,8rtkit-daemon1622-21rtkit-daemon23:40:150
162399540,8rtkit-daemon1622-21rtkit-daemon23:40:150
2848441995319,9cyclictest2935166-21latency23:10:231
2848441995315,8cyclictest1753-21gnome-shell20:15:331
2848441995312,13cyclictest2884609-21cpuspeed_turbos20:50:141
2848441995312,13cyclictest2884609-21cpuspeed_turbos20:50:141
2848441995312,11cyclictest2947730-21wc23:45:311
162399530,9rtkit-daemon1622-21rtkit-daemon00:17:250
2848446995215,16cyclictest2912551-21uptime22:05:312
284844699521,28cyclictest2964507-21cpu00:35:152
284844699521,26cyclictest0-21swapper/223:30:322
162399520,9rtkit-daemon1622-21rtkit-daemon21:41:150
162399520,8rtkit-daemon1622-21rtkit-daemon22:28:411
2848449995114,32cyclictest2931141-21sort23:00:123
2848446995120,26cyclictest2915835-40lsb_release22:15:242
284844199512,20cyclictest2883325-21memory20:45:251
2848441995116,8cyclictest1753-21gnome-shell22:48:571
2848441995114,11cyclictest2875933-21cut20:25:161
284844199511,27cyclictest0-21swapper/123:30:001
162399510,8rtkit-daemon1622-21rtkit-daemon22:35:221
162399510,6rtkit-daemon1622-21rtkit-daemon20:31:571
2848449995025,21cyclictest2900637-21tr21:35:133
284844999502,43cyclictest1753-21gnome-shell23:22:063
284844999502,43cyclictest1753-21gnome-shell23:22:063
284844999502,42cyclictest1753-21gnome-shell20:15:193
2848449995018,28cyclictest2929581-21tr22:55:183
2848449995018,28cyclictest2929581-21tr22:55:183
284844999501,43cyclictest2913611-21which22:10:203
2848449995014,31cyclictest2928140-21cat22:50:233
2848449995013,32cyclictest2947765-21timerandwakeup23:45:313
284844699501,16cyclictest2955897-21df_inode00:10:212
284844199508,17cyclictest1753-21gnome-shell21:50:201
2848441995017,8cyclictest1753-21gnome-shell22:10:391
2848441995017,7cyclictest2197014-21PK-Backend23:06:121
2848441995013,13cyclictest1753-21gnome-shell22:05:191
284844199501,19cyclictest2896944-21/usr/sbin/munin21:25:111
162399500,25rtkit-daemon2197014-21PK-Backend22:50:260
284844999499,16cyclictest1753-21gnome-shell21:08:413
284844999492,8cyclictest1753-21gnome-shell20:35:163
284844999492,43cyclictest1753-21gnome-shell22:46:463
2848449994922,22cyclictest1753-21gnome-shell21:57:053
284844999492,14cyclictest2902946-21ntp_states21:40:303
2848449994915,30cyclictest2961579-21wc00:25:253
284844999491,43cyclictest1753-21gnome-shell23:15:093
2848446994938,7cyclictest2868386-21kworker/2:121:10:012
284844699491,31cyclictest2899583-21/usr/sbin/munin21:30:302
284844199498,16cyclictest1753-21gnome-shell21:30:331
284844199491,23cyclictest2939554-21perf23:25:011
284844199491,23cyclictest2939554-21perf23:25:011
284844999489,35cyclictest1753-21gnome-shell23:09:043
284844999489,34cyclictest1753-21gnome-shell23:51:553
284844999482,42cyclictest1753-21gnome-shell23:58:423
284844999482,42cyclictest1753-21gnome-shell23:26:553
284844999482,42cyclictest1753-21gnome-shell23:12:563
284844999482,42cyclictest1753-21gnome-shell00:32:013
284844999482,41cyclictest1753-21gnome-shell00:37:383
2848449994821,23cyclictest42-21ksoftirqd/300:22:373
2848449994821,22cyclictest2881159-21cpuspeed_turbos20:40:173
2848449994811,32cyclictest1753-21gnome-shell21:50:033
2848449994811,31cyclictest2916185-21cat22:15:303
284844699482,27cyclictest1753-21gnome-shell23:12:012
284844699482,26cyclictest1753-21gnome-shell23:52:462
284844699482,26cyclictest1753-21gnome-shell20:21:312
284844699482,22cyclictest1753-21gnome-shell00:16:022
284844699481,20cyclictest860997-21snmpd23:25:242
284844199481,21cyclictest1753-21gnome-shell20:40:231
2848441994811,8cyclictest0-21swapper/100:00:211
29555312470,8sleep31753-21gnome-shell00:10:173
284844999472,41cyclictest1753-21gnome-shell22:44:073
284844999472,41cyclictest1753-21gnome-shell20:34:113
2848449994720,23cyclictest2871496-21wc20:10:323
2848449994716,27cyclictest2919431-21idleruntime22:25:233
2848449994715,26cyclictest438-20systemd-journal22:24:553
284844999471,41cyclictest1753-21gnome-shell22:35:213
284844699472,27cyclictest1753-21gnome-shell23:41:052
284844699472,27cyclictest1753-21gnome-shell23:41:052
284844699472,27cyclictest1753-21gnome-shell21:11:312
284844699472,18cyclictest1753-21gnome-shell22:42:572
2848446994719,10cyclictest1753-21gnome-shell21:45:072
284844199471,27cyclictest0-21swapper/122:33:421
2848441994711,7cyclictest0-21swapper/100:25:151
284844999469,32cyclictest2882834-21ls20:45:173
284844999467,16cyclictest1753-21gnome-shell20:25:163
2848449994621,21cyclictest1753-21gnome-shell21:13:533
2848449994615,26cyclictest42-21ksoftirqd/321:27:273
284844999461,40cyclictest2920730-21/usr/sbin/munin22:30:253
2848449994613,28cyclictest2893850-21needreboot21:15:253
2848449994613,28cyclictest2893850-21needreboot21:15:253
2848449994610,31cyclictest2944662-21seq23:38:233
284844699462,23cyclictest1753-21gnome-shell20:10:242
284844699462,17cyclictest1753-21gnome-shell22:53:192
284844699462,17cyclictest1753-21gnome-shell20:59:582
284844699462,16cyclictest2930177-21ntp_states22:55:302
284844699462,16cyclictest2930177-21ntp_states22:55:292
284844699462,16cyclictest2879757-21irqstats20:35:242
284844699461,16cyclictest2907705-21taskset21:54:492
2848449994514,27cyclictest2895161-21cstates21:20:143
2848449994514,27cyclictest1069-21runrttasks21:04:323
2848449994513,28cyclictest1795-21gmain21:50:013
2848449994512,29cyclictest2911098-21seq22:03:523
2848449994510,31cyclictest1753-21gnome-shell00:17:173
284844699455,16cyclictest2908651-21sed21:55:252
284844699452,17cyclictest1753-21gnome-shell00:04:282
284844699452,16cyclictest2950643-21df23:55:162
284844699451,22cyclictest860997-21snmpd22:10:272
284844699451,18cyclictest2903292-21vmstat21:40:352
284844199451,11cyclictest1753-21gnome-shell23:40:241
284844199451,11cyclictest1753-21gnome-shell23:40:231
2848449994412,28cyclictest2952577-21grep00:00:223
284844699443,17cyclictest1753-21gnome-shell23:19:452
284844699442,18cyclictest1753-21gnome-shell22:33:022
284844699441,19cyclictest2877194-21taskset20:29:172
284844699441,17cyclictest2918392-40apt-check22:21:232
284844699441,14cyclictest2872562-21cstates20:15:172
284844199442,7cyclictest2956005-21irqstats00:10:251
284844199441,8cyclictest28718112sleep120:12:001
2848437994433,8cyclictest13-21ksoftirqd/021:17:010
2848437994433,8cyclictest13-21ksoftirqd/021:17:010
2848437994426,7cyclictest13-21ksoftirqd/021:54:460
162399440,37rtkit-daemon2197014-21PK-Backend23:37:580
162399440,35rtkit-daemon881143-21gmain00:35:040
162399440,21rtkit-daemon0-21swapper/021:38:570
284844999432,15cyclictest2942425-21python323:30:283
2848449994312,26cyclictest2911681-21cstates22:05:153
284844699432,25cyclictest1753-21gnome-shell22:27:572
284844699432,24cyclictest1753-21gnome-shell21:23:062
284844699432,17cyclictest1753-21gnome-shell22:39:422
284844199431,21cyclictest1753-21gnome-shell23:15:201
284844199431,21cyclictest1753-21gnome-shell22:03:141
2848437994331,7cyclictest1753-21gnome-shell20:26:120
2848437994324,15cyclictest1753-21gnome-shell21:00:550
284844699422,24cyclictest1753-21gnome-shell23:02:212
284844699422,24cyclictest1753-21gnome-shell00:21:592
284844699422,15cyclictest2926827-21python322:45:292
2848446994221,16cyclictest438-20systemd-journal20:45:472
284844699421,22cyclictest2888338-21smartctl21:00:202
284844699421,15cyclictest2954539-21ntp_states00:05:282
2848441994219,19cyclictest2958710-21cron00:20:001
2848437994225,7cyclictest1753-21gnome-shell22:02:280
2848437994223,6cyclictest13-21ksoftirqd/000:00:040
2848437994223,14cyclictest13-21ksoftirqd/022:45:320
284844699412,17cyclictest2947396-21memory23:45:242
284844699412,16cyclictest2893920-21ntp_states21:15:272
284844699412,16cyclictest2893920-21ntp_states21:15:272
284844699412,16cyclictest1753-21gnome-shell21:27:052
284844699411,17cyclictest0-21swapper/223:20:142
284844699411,17cyclictest0-21swapper/223:20:142
284844199411,9cyclictest0-21swapper/122:50:171
2848437994124,6cyclictest13-21ksoftirqd/022:15:040
2848437994121,11cyclictest13-21ksoftirqd/020:11:050
2848437994120,8cyclictest13-21ksoftirqd/020:32:270
284844999401,34cyclictest2197014-21PK-Backend20:50:123
284844999401,34cyclictest2197014-21PK-Backend20:50:123
284844699402,17cyclictest1753-21gnome-shell23:39:132
284844699401,16cyclictest2882481-21munin-run20:45:002
2848441994013,21cyclictest2931255-21sort23:00:141
2848437994023,13cyclictest13-21ksoftirqd/021:35:010
2848437994022,7cyclictest13-21ksoftirqd/023:30:210
162399400,31rtkit-daemon13-21ksoftirqd/022:58:150
162399400,31rtkit-daemon13-21ksoftirqd/022:58:150
2848441993927,8cyclictest2955280-21pool-org.gnome.00:10:011
2848437993926,9cyclictest1753-21gnome-shell00:25:030
2848437993922,6cyclictest13-21ksoftirqd/022:35:280
2848437993922,5cyclictest13-21ksoftirqd/023:50:200
2848437993920,7cyclictest13-21ksoftirqd/020:38:490
2848437993920,7cyclictest13-21ksoftirqd/000:11:520
29328842380,7sleep02197014-21PK-Backend23:05:090
2848446993823,11cyclictest2901812-21sshd21:38:462
284844699381,16cyclictest2909716-21/usr/sbin/munin22:00:192
2848441993814,19cyclictest2880328-21sort20:35:341
284843799388,6cyclictest141rcu_preempt23:55:020
284843799384,9cyclictest13-21ksoftirqd/020:24:530
2848437993828,6cyclictest1753-21gnome-shell23:20:170
2848437993828,6cyclictest1753-21gnome-shell23:20:170
2848437993827,7cyclictest1753-21gnome-shell21:22:140
2848437993827,7cyclictest1753-21gnome-shell00:30:540
2848437993824,10cyclictest13-21ksoftirqd/021:06:000
2848437993820,6cyclictest13-21ksoftirqd/022:35:010
284843799381,33cyclictest2891165-21dump-pmu-power21:14:550
162399380,30rtkit-daemon13-21ksoftirqd/021:59:140
162399380,30rtkit-daemon13-21ksoftirqd/000:05:260
284844699372,30cyclictest0-21swapper/220:50:462
284844699372,30cyclictest0-21swapper/220:50:462
284843799372,7cyclictest1753-21gnome-shell22:06:350
2848437993726,7cyclictest1753-21gnome-shell22:10:080
2848437993726,7cyclictest1753-21gnome-shell20:54:450
2848437993726,7cyclictest1753-21gnome-shell20:54:450
2848437993722,11cyclictest13-21ksoftirqd/023:00:280
2848437993720,6cyclictest13-21ksoftirqd/022:25:010
2848446993624,8cyclictest2961114-21tr00:25:162
2848441993618,13cyclictest2886964-21ntpq20:55:311
2848437993625,7cyclictest1753-21gnome-shell23:16:160
2848437993625,7cyclictest1753-21gnome-shell23:14:180
2848437993625,7cyclictest1753-21gnome-shell20:48:570
284843799362,30cyclictest0-21swapper/022:25:020
2848437993621,11cyclictest2886546-21expr20:55:190
2848437993620,6cyclictest13-21ksoftirqd/023:46:450
2848441993523,7cyclictest2929592-21cpuspeed_turbos22:55:181
2848441993523,7cyclictest2929592-21cpuspeed_turbos22:55:171
284843799352,28cyclictest1753-21gnome-shell21:25:040
2848437993519,5cyclictest13-21ksoftirqd/000:25:010
2848437993516,5cyclictest141rcu_preempt22:41:550
2848437993423,6cyclictest13-21ksoftirqd/020:17:590
2848437993418,11cyclictest2903664-21apt-get21:45:020
2848441993322,7cyclictest2962917-21tr00:30:161
2848437993325,4cyclictest2197014-21PK-Backend23:28:060
2848437993322,7cyclictest2882471-21sleep20:45:000
284844199321,28cyclictest2197014-21PK-Backend20:22:401
284844199311,27cyclictest0-21swapper/122:22:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional