You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 22:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Nov 22, 2025 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
235917822540,6sleep00-21swapper/007:05:220
236520221160,5sleep00-21swapper/007:20:300
241297221120,6sleep30-21swapper/309:35:173
24413942930,7sleep30-21swapper/310:55:153
24639102900,6sleep0856-21gmain11:56:410
24587542900,6sleep00-21swapper/011:45:030
162399850,10rtkit-daemon1753-21gnome-shell08:36:492
162399840,11rtkit-daemon1753-21gnome-shell09:49:352
162399810,7rtkit-daemon2197014-21PK-Backend12:12:372
162399810,6rtkit-daemon2197014-21PK-Backend12:19:142
162399800,5rtkit-daemon0-21swapper/208:59:302
162399790,6rtkit-daemon2197014-21PK-Backend09:06:052
162399790,5rtkit-daemon0-21swapper/209:02:532
162399780,6rtkit-daemon2197014-21PK-Backend12:26:432
24162632770,6sleep22416266-21latency_hist09:45:022
235865927650,17sleep20-21swapper/207:05:082
162399760,7rtkit-daemon2197014-21PK-Backend10:17:042
235984327458,9sleep10-21swapper/107:06:151
236008327043,20sleep30-21swapper/307:09:243
24507102590,8sleep22197014-21PK-Backend11:20:212
236028299596,19cyclictest855-21dbus-daemon11:05:011
162399590,9rtkit-daemon1753-21gnome-shell12:22:102
162399590,9rtkit-daemon1753-21gnome-shell12:22:102
162399590,9rtkit-daemon1622-21rtkit-daemon08:18:551
24667212560,5sleep0161rcuc/012:05:230
2360286995329,20cyclictest1753-21gnome-shell11:46:162
162399530,7rtkit-daemon1622-21rtkit-daemon08:01:381
162399530,7rtkit-daemon1622-21rtkit-daemon08:01:371
236028299526,25cyclictest2395298-21idleruntime08:45:191
24198892510,6sleep1231irq_work/109:55:151
162399510,8rtkit-daemon0-21swapper/211:34:022
162399510,43rtkit-daemon1506-21systemd08:12:130
2360291995017,9cyclictest2432080-21awk10:30:013
236028299504,20cyclictest2466395-21cstates12:05:151
162399500,6rtkit-daemon2197014-21PK-Backend11:18:023
162399500,42rtkit-daemon1506-21systemd08:16:540
162399500,42rtkit-daemon1506-21systemd07:27:250
2360291994820,8cyclictest1753-21gnome-shell11:25:243
2360286994824,20cyclictest1753-21gnome-shell12:00:312
2360286994824,20cyclictest1753-21gnome-shell12:00:312
236028299488,16cyclictest2363053-21expr07:15:161
236028299488,13cyclictest2390773-21seq08:34:161
236028299487,26cyclictest2417290-21python309:45:351
2360282994810,19cyclictest1753-21gnome-shell12:30:241
2360280994829,14cyclictest1-21systemd09:55:020
236028299473,11cyclictest2386293-21diskstats08:20:181
2360280994723,15cyclictest13-21ksoftirqd/007:50:250
236029199461,19cyclictest2437349-21taskset10:42:263
2360286994628,14cyclictest34-21ksoftirqd/207:15:082
2360286994627,7cyclictest1753-21gnome-shell07:20:272
236028299468,14cyclictest2469102-21taskset12:11:251
236028299468,10cyclictest1753-21gnome-shell10:13:351
236028299467,14cyclictest2361835-21taskset07:13:491
236028299465,25cyclictest2423616-21taskset10:05:131
2360282994634,8cyclictest1069-21runrttasks07:30:491
236028299461,17cyclictest1753-21gnome-shell11:18:401
162399460,38rtkit-daemon2360275-21cyclictest07:58:011
24530932450,6sleep234-21ksoftirqd/211:25:322
2360291994523,8cyclictest42-21ksoftirqd/310:19:263
236028699459,32cyclictest1753-21gnome-shell09:10:292
2360286994526,6cyclictest34-21ksoftirqd/207:35:262
2360286994526,14cyclictest34-21ksoftirqd/211:00:232
2360286994525,15cyclictest34-21ksoftirqd/209:58:252
2360286994523,13cyclictest34-21ksoftirqd/211:40:022
236028299456,14cyclictest2437433-21runrttasks10:43:321
236028299455,17cyclictest2440052-21kernelversion10:50:231
236028299455,17cyclictest2440052-21kernelversion10:50:221
236028299452,16cyclictest2418531-21entropy09:50:211
236028299451,16cyclictest2392580-40lsb_release08:36:261
236028299450,25cyclictest1753-21gnome-shell08:10:011
236028299450,16cyclictest2434142-21wc10:35:011
236028099452,20cyclictest13-21ksoftirqd/012:05:010
236028099452,20cyclictest13-21ksoftirqd/012:05:010
162399450,31rtkit-daemon0-21swapper/208:51:152
236029199441,15cyclictest2443828-40lsb_release11:00:223
2360286994424,14cyclictest34-21ksoftirqd/208:40:272
2360286994424,14cyclictest34-21ksoftirqd/208:34:012
2360286994421,6cyclictest34-21ksoftirqd/210:45:022
2360286994419,17cyclictest34-21ksoftirqd/210:10:462
236028299449,9cyclictest2372190-21ntp_states07:40:281
236028299449,9cyclictest2372190-21ntp_states07:40:271
236028299449,10cyclictest1753-21gnome-shell09:33:501
236028299449,10cyclictest1753-21gnome-shell09:33:491
236028299448,9cyclictest2477005-21hddtemp_smartct12:35:211
236028299448,11cyclictest2405612-21taskset09:15:121
236028299448,10cyclictest2403259-40lsb_release09:06:211
236028299447,9cyclictest2452383-21df_inode11:25:181
236028299445,14cyclictest2465134-21/usr/sbin/munin12:00:241
236028299445,14cyclictest2465134-21/usr/sbin/munin12:00:231
2360282994420,21cyclictest2389144-21unin-run08:30:011
2360280994413,12cyclictest1753-21gnome-shell12:15:190
236028099441,12cyclictest141rcu_preempt08:31:330
162399440,30rtkit-daemon2197014-21PK-Backend09:25:252
23632912430,6sleep00-21swapper/007:15:220
2360291994322,16cyclictest42-21ksoftirqd/308:15:233
2360291994310,29cyclictest1753-21gnome-shell09:11:013
2360291994310,20cyclictest1753-21gnome-shell08:51:403
2360286994325,6cyclictest34-21ksoftirqd/210:05:082
2360286994325,6cyclictest34-21ksoftirqd/208:05:162
2360286994324,7cyclictest34-21ksoftirqd/212:39:122
2360286994324,7cyclictest34-21ksoftirqd/210:00:132
2360286994324,7cyclictest34-21ksoftirqd/209:30:042
2360286994324,7cyclictest34-21ksoftirqd/209:30:042
2360286994324,7cyclictest34-21ksoftirqd/208:24:432
2360286994324,6cyclictest34-21ksoftirqd/211:50:302
2360286994324,6cyclictest34-21ksoftirqd/211:13:572
2360286994324,6cyclictest34-21ksoftirqd/210:30:232
2360286994324,6cyclictest34-21ksoftirqd/208:25:162
2360286994324,6cyclictest34-21ksoftirqd/207:55:312
2360286994324,6cyclictest34-21ksoftirqd/207:48:222
2360286994324,6cyclictest34-21ksoftirqd/207:48:212
2360286994324,15cyclictest34-21ksoftirqd/209:39:242
2360286994324,14cyclictest34-21ksoftirqd/212:33:022
2360286994324,14cyclictest34-21ksoftirqd/211:08:222
2360286994324,14cyclictest34-21ksoftirqd/210:56:412
2360286994324,14cyclictest34-21ksoftirqd/210:50:222
2360286994324,14cyclictest34-21ksoftirqd/210:50:222
2360286994324,14cyclictest34-21ksoftirqd/210:35:262
2360286994324,14cyclictest34-21ksoftirqd/208:45:322
2360286994324,14cyclictest34-21ksoftirqd/207:31:572
2360286994324,14cyclictest34-21ksoftirqd/207:14:072
2360286994323,6cyclictest34-21ksoftirqd/208:14:592
2360286994323,15cyclictest34-21ksoftirqd/211:55:192
2360286994323,15cyclictest2431342-21ntp_states10:25:252
236028299438,13cyclictest2464099-21taskset11:59:101
236028299438,12cyclictest2374477-21taskset07:48:151
236028299438,12cyclictest2374477-21taskset07:48:151
236028299437,12cyclictest2431531-21sed10:25:291
236028299437,11cyclictest2438239-21hddtemp_smartct10:45:221
236028299437,11cyclictest2401212-21runrttasks09:00:491
2360282994325,13cyclictest2457353-21grep11:40:141
236028299430,27cyclictest1753-21gnome-shell09:20:401
236028299430,16cyclictest1753-21gnome-shell08:58:501
236028099439,11cyclictest141rcu_preempt10:24:450
2360280994324,14cyclictest13-21ksoftirqd/008:25:010
2360280994311,9cyclictest2360275-21cyclictest08:00:060
2360280994311,9cyclictest2360275-21cyclictest08:00:060
24223592420,7chrt0-21swapper/110:00:261
236029199428,30cyclictest1753-21gnome-shell09:47:193
236029199427,31cyclictest1753-21gnome-shell12:01:543
236029199427,31cyclictest1753-21gnome-shell12:01:533
236029199424,25cyclictest141rcu_preempt07:25:123
2360291994227,10cyclictest42-21ksoftirqd/308:00:143
2360291994227,10cyclictest42-21ksoftirqd/308:00:133
2360286994224,6cyclictest34-21ksoftirqd/207:50:252
2360286994224,14cyclictest34-21ksoftirqd/209:50:322
2360286994224,14cyclictest34-21ksoftirqd/208:15:122
2360286994224,14cyclictest34-21ksoftirqd/207:40:582
2360286994224,14cyclictest34-21ksoftirqd/207:40:582
2360286994224,13cyclictest34-21ksoftirqd/210:45:022
2360286994223,6cyclictest34-21ksoftirqd/211:15:232
2360286994223,6cyclictest34-21ksoftirqd/208:00:232
2360286994223,6cyclictest34-21ksoftirqd/208:00:232
2360286994223,14cyclictest34-21ksoftirqd/212:05:282
2360286994222,7cyclictest1753-21gnome-shell11:40:282
236028299428,18cyclictest1753-21gnome-shell11:24:361
236028299428,13cyclictest2369415-21kworker/1:008:10:081
236028299428,11cyclictest2456991-21sed11:40:021
236028299427,10cyclictest1753-21gnome-shell10:20:291
236028299426,15cyclictest1753-21gnome-shell09:42:291
236028299426,10cyclictest2459191-21/usr/sbin/munin11:45:241
236028299425,11cyclictest2446245-21seq11:06:271
2360282994227,11cyclictest2455013-21sshd11:32:421
2360282994210,10cyclictest2375393-21irqstats07:50:261
236028299420,26cyclictest1753-21gnome-shell12:16:531
236028299420,16cyclictest1753-21gnome-shell09:25:321
2360280994231,8cyclictest13-21ksoftirqd/009:10:200
2360280994220,18cyclictest2432052-21cron10:30:010
2360280994211,9cyclictest141rcu_preempt10:05:370
236029199419,19cyclictest141rcu_preempt12:19:003
236029199412,5cyclictest141rcu_preempt12:06:423
2360286994124,6cyclictest34-21ksoftirqd/209:20:012
2360286994123,6cyclictest34-21ksoftirqd/207:29:332
2360286994123,14cyclictest34-21ksoftirqd/210:23:422
236028299417,10cyclictest1753-21gnome-shell10:38:281
236028299416,10cyclictest2370929-21runrttasks07:37:561
236028299416,10cyclictest2364927-21irqstats07:20:251
236028299412,10cyclictest0-21swapper/109:10:501
236028299411,11cyclictest0-21swapper/112:24:071
236028299411,11cyclictest0-21swapper/112:24:071
236028099419,9cyclictest141rcu_preempt10:48:590
2360280994128,9cyclictest13-21ksoftirqd/009:30:010
2360280994127,10cyclictest3242253-21nautilus09:40:010
2360280994125,12cyclictest2394611-21sh08:45:010
2360280994123,7cyclictest13-21ksoftirqd/008:27:240
2360280994110,9cyclictest2360275-21cyclictest11:31:450
2360280994110,10cyclictest141rcu_preempt08:51:420
162399410,9rtkit-daemon1753-21gnome-shell10:47:463
24080992400,5chrt0-21swapper/009:20:240
236029199407,15cyclictest1753-21gnome-shell11:09:433
236029199405,17cyclictest1753-21gnome-shell11:43:213
236029199402,34cyclictest1753-21gnome-shell08:10:233
236029199402,11cyclictest141rcu_preempt08:41:373
2360291994019,9cyclictest42-21ksoftirqd/310:55:023
2360291994019,9cyclictest42-21ksoftirqd/310:55:023
2360291994017,10cyclictest42-21ksoftirqd/309:53:253
2360291994014,8cyclictest1753-21gnome-shell11:10:213
2360286994022,6cyclictest34-21ksoftirqd/209:20:312
236028299407,12cyclictest2394325-21seq08:41:251
236028299407,11cyclictest2462066-21users11:50:321
236028299407,11cyclictest2442401-21taskset10:55:341
236028299402,16cyclictest2447203-21hddtemp_smartct11:10:191
236028299401,16cyclictest1753-21gnome-shell07:26:411
236028299400,16cyclictest1753-21gnome-shell10:15:261
2360280994025,11cyclictest2360633-21cut07:10:150
2360280994024,12cyclictest1753-21gnome-shell12:26:530
2360280994022,13cyclictest13-21ksoftirqd/008:35:020
2360280994021,14cyclictest13-21ksoftirqd/012:14:590
236028099401,8cyclictest2459184-40apt-check11:45:170
2360280994012,11cyclictest141rcu_preempt11:15:510
2360280994011,12cyclictest141rcu_preempt10:58:030
236029199397,19cyclictest1753-21gnome-shell10:36:263
2360291993925,11cyclictest42-21ksoftirqd/310:25:003
2360291993922,6cyclictest42-21ksoftirqd/311:55:303
2360291993921,13cyclictest2424765-40apt-check10:06:193
2360291993920,14cyclictest2408095-21ntpq09:20:243
236029199391,5cyclictest141rcu_preempt07:47:283
236029199391,5cyclictest141rcu_preempt07:47:273
2360282993923,11cyclictest438-20systemd-journal08:53:011
236028099398,5cyclictest141rcu_preempt11:00:060
236028099392,9cyclictest0-21swapper/010:42:140
2360280993924,11cyclictest13-21ksoftirqd/010:33:150
2360280993921,13cyclictest13-21ksoftirqd/009:00:560
2360280993917,18cyclictest2425783-21hddtemp_smartct10:10:200
2360280993916,14cyclictest13-21ksoftirqd/008:06:420
236029199388,17cyclictest1753-21gnome-shell12:38:533
236029199387,15cyclictest42-21ksoftirqd/309:55:333
236029199384,21cyclictest141rcu_preempt12:22:443
236029199384,21cyclictest141rcu_preempt12:22:443
2360291993830,5cyclictest2197014-21PK-Backend09:30:453
2360291993830,5cyclictest2197014-21PK-Backend09:30:453
2360291993827,8cyclictest1753-21gnome-shell09:00:153
236029199382,32cyclictest1753-21gnome-shell10:12:013
2360291993811,6cyclictest141rcu_preempt09:05:123
2360291993811,23cyclictest0-21swapper/312:10:323
236028099389,5cyclictest141rcu_preempt10:50:280
236028099389,5cyclictest141rcu_preempt10:50:280
2360280993827,7cyclictest1753-21gnome-shell07:33:300
2360280993826,8cyclictest13-21ksoftirqd/007:40:010
2360280993819,15cyclictest13-21ksoftirqd/007:42:270
2360280993819,15cyclictest13-21ksoftirqd/007:42:260
2360280993815,6cyclictest24278052/usr/sbin/munin10:15:250
2360280993812,21cyclictest2411510-40lsb_release09:30:200
2360280993812,21cyclictest2411510-40lsb_release09:30:200
2360280993810,12cyclictest141rcu_preempt12:38:210
24517722370,6chrt2197014-21PK-Backend11:25:013
236029199376,16cyclictest141rcu_preempt08:32:373
236029199376,10cyclictest42-21ksoftirqd/312:30:283
236029199375,5cyclictest141rcu_preempt08:00:013
236029199373,15cyclictest1753-21gnome-shell08:37:283
2360291993726,7cyclictest2455200-21cron11:35:003
2360291993726,7cyclictest2370635-21grep07:35:313
2360291993721,11cyclictest2459646-21idleruntime11:45:263
2360291993715,11cyclictest141rcu_preempt07:53:013
236029199371,24cyclictest141rcu_preempt08:24:353
2360291993711,6cyclictest141rcu_preempt11:55:003
2360282993719,13cyclictest2413289-21systemd-udevd09:35:201
2360280993726,7cyclictest161rcuc/011:36:290
236028099372,5cyclictest141rcu_preempt11:50:540
2360280993721,12cyclictest2378104-21/usr/sbin/munin08:00:010
2360280993721,11cyclictest2450435-21grep11:20:140
2360280993720,13cyclictest13-21ksoftirqd/011:13:470
2360280993720,12cyclictest438-20systemd-journal12:33:050
2360280993720,12cyclictest2399739-21seq08:59:110
2360280993718,14cyclictest13-21ksoftirqd/009:06:220
2360280993715,5cyclictest2472200-21/usr/sbin/munin12:20:280
2360280993715,5cyclictest2472200-21/usr/sbin/munin12:20:270
236029199369,23cyclictest0-21swapper/310:30:303
236029199365,14cyclictest42-21ksoftirqd/307:24:293
236029199365,12cyclictest0-21swapper/308:58:503
2360291993621,11cyclictest42-21ksoftirqd/308:46:563
2360291993621,11cyclictest2368390-21awk07:30:213
2360291993621,10cyclictest2457117-21cut11:40:033
2360291993620,12cyclictest2474640-21wc12:30:023
2360291993619,12cyclictest2381902-21pool-org.gnome.08:10:003
2360291993618,5cyclictest141rcu_preempt07:11:113
236028299365,10cyclictest2474633-21systemd-run12:30:001
2360280993626,6cyclictest1753-21gnome-shell11:26:570
236028099362,5cyclictest141rcu_preempt09:57:150
2360280993625,7cyclictest161rcuc/009:49:180
2360280993624,8cyclictest1753-21gnome-shell11:05:530
2360280993619,12cyclictest2405839-21cstates09:15:170
2360280993614,5cyclictest2395160-21expr08:45:150
2360280993612,12cyclictest141rcu_preempt09:42:530
162399360,30rtkit-daemon0-21swapper/007:45:250
162399360,30rtkit-daemon0-21swapper/007:45:240
236029199355,5cyclictest141rcu_preempt09:15:273
236029199354,5cyclictest141rcu_preempt10:05:013
2360291993523,8cyclictest1753-21gnome-shell07:44:583
2360291993523,8cyclictest1753-21gnome-shell07:44:573
2360291993522,8cyclictest1753-21gnome-shell07:17:323
2360291993520,11cyclictest2409756-21needreboot09:25:233
2360291993519,12cyclictest2415534-21cut09:40:293
2360280993511,12cyclictest141rcu_preempt10:03:170
2360280993510,10cyclictest141rcu_preempt10:35:570
2360291993218,10cyclictest42-21ksoftirqd/308:27:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional