You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackfslot7s.osadl.org (updated Wed Aug 27, 2025 00:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
292495499419519,4163cyclictest2999019-21kworker/1:1+events23:15:011
292496099313114,3113cyclictest2931550-21kworker/3:1+events19:45:213
29249469930419,3028cyclictest3008717-21kworker/0:0+events23:31:250
29249469929229,2909cyclictest3027678-21kworker/0:2+events00:30:220
292494699292014,2902cyclictest2962163-21kworker/0:2+events21:05:200
292496099291611,2900cyclictest2951061-21kworker/3:1+events20:35:183
292494699289014,2871cyclictest3008717-21kworker/0:0+events23:55:140
292496099283412,2815cyclictest3018711-21kworker/3:1+events00:10:143
292495699282514,2807cyclictest3000734-21kworker/2:0+events23:05:202
292495699280513,2787cyclictest2963436-21kworker/2:2+events21:20:182
292495499280413,2789cyclictest2989244-21kworker/1:2+events22:30:251
292494699278712,2771cyclictest3008717-21kworker/0:0+events23:50:000
292496099273210,2718cyclictest3030311-21kworker/3:2+events00:35:193
292494699271913,2703cyclictest3002210-21kworker/0:1+events23:10:330
292494699271313,2696cyclictest2947532-21kworker/0:2+events20:24:390
292496099269811,2683cyclictest3003810-21kworker/3:0+events23:30:183
292495699262612,2609cyclictest3013592-21kworker/2:0+events23:45:232
292494699261611,2601cyclictest2952561-21kworker/0:2+events20:40:130
292495499260910,2595cyclictest3008509-21kworker/1:2+events23:38:411
292494699259713,2580cyclictest2954464-21kworker/0:0+events20:47:280
292495699257911,2564cyclictest2934897-21kworker/2:2+events19:50:142
292495699257012,2554cyclictest3011965-21kworker/2:2+events23:57:542
292495699255510,2542cyclictest2963436-21kworker/2:2+events21:15:372
292495699255212,2538cyclictest2995851-21kworker/2:1+events23:11:382
292495699254711,2534cyclictest2941084-21kworker/2:1+events20:05:302
292494699252711,2512cyclictest2957060-21kworker/0:1+events21:01:100
29249569925228,2510cyclictest2931581-21kworker/2:1+events19:40:472
292495699251010,2497cyclictest2954129-21kworker/2:2+events20:40:362
292494699249813,2481cyclictest2978032-21kworker/0:2+events21:54:430
292495699249712,2481cyclictest2995851-21kworker/2:1+events23:18:292
292495699249712,2481cyclictest2994225-21kworker/2:0+events22:43:532
292495699249612,2480cyclictest2966676-21kworker/2:1+events21:37:282
292495699249512,2479cyclictest2949296-21kworker/2:0+events20:32:152
292494699249512,2479cyclictest3007071-21kworker/0:2+events23:24:570
292495499249412,2478cyclictest2963627-21kworker/1:1+events21:10:151
292495699249113,2474cyclictest2976544-21kworker/2:0+events22:19:182
292495699249112,2476cyclictest2949296-21kworker/2:0+events20:47:512
292495699249112,2475cyclictest2974924-21kworker/2:1+events21:50:522
292495699249111,2476cyclictest2995851-21kworker/2:1+events22:52:212
292495499249012,2475cyclictest2954176-21kworker/1:0+events20:42:071
292495699248911,2474cyclictest2976544-21kworker/2:0+events22:36:132
292495499248812,2472cyclictest2926059-21kworker/1:0+events19:25:141
29249569924858,2474cyclictest2947650-21kworker/2:1+events20:23:292
292495699248512,2470cyclictest2933181-21kworker/2:0+events19:36:352
292495699248412,2469cyclictest3028438-21kworker/2:1+events00:28:472
292495499248412,2469cyclictest2968287-21kworker/1:0+events21:27:341
292495699247911,2464cyclictest2961840-21kworker/2:0+events21:03:502
292495699247812,2463cyclictest3031680-21kworker/2:2+events00:37:592
292495699247710,2463cyclictest3018466-21kworker/2:1+events00:18:142
292496099247511,2460cyclictest2951061-21kworker/3:1+events20:43:173
292495699247511,2460cyclictest2976544-21kworker/2:0+events22:02:022
292494699247412,2458cyclictest2950793-21kworker/0:1+events20:28:500
292494699247312,2458cyclictest2934593-21kworker/0:0+events19:52:570
292495499247212,2456cyclictest2929661-21kworker/1:2+events20:03:571
292494699247111,2456cyclictest2957060-21kworker/0:1+events21:11:240
29249469924706,2460cyclictest2976337-21kworker/0:1+events21:55:300
292494699247012,2454cyclictest2976337-21kworker/0:1+events21:48:120
292495699246711,2453cyclictest2934897-21kworker/2:2+events20:00:342
292496099246612,2450cyclictest2986284-21kworker/3:2+events22:44:163
29249469924589,2446cyclictest2991079-21kworker/0:1+events22:37:230
292496099245311,2438cyclictest2976441-21kworker/3:1+events21:51:143
292495699245312,2438cyclictest2976544-21kworker/2:0+events22:26:112
292496099244912,2433cyclictest2958713-21kworker/3:0+events20:58:533
292495699244810,2434cyclictest2937668-21kworker/2:0+events19:49:052
292494699244712,2431cyclictest3002210-21kworker/0:1+events23:26:060
292496099244412,2428cyclictest2979825-21kworker/3:1+events22:22:223
292494699244012,2425cyclictest2934593-21kworker/0:0+events19:48:420
292496099243512,2419cyclictest2949355-21kworker/3:0+events20:25:243
29249609924349,2422cyclictest2998836-21kworker/3:1+events22:59:373
292495499242810,2414cyclictest2938044-21kworker/1:1+events19:54:051
292495499242712,2411cyclictest2999019-21kworker/1:1+events23:28:461
292496099242612,2410cyclictest3009793-21kworker/3:2+events23:44:403
292495499242212,2406cyclictest2963627-21kworker/1:1+events21:23:451
292495499242012,2406cyclictest2929661-21kworker/1:2+events20:15:051
292495499242012,2404cyclictest2979514-21kworker/1:0+events22:56:351
292494699241712,2401cyclictest2918105-21kworker/0:2+events19:20:190
292496099241512,2399cyclictest2986284-21kworker/3:2+events22:48:083
292494699241512,2400cyclictest2991079-21kworker/0:1+events23:01:070
292495499241212,2396cyclictest3028558-21kworker/1:0+events00:31:311
292495499241111,2397cyclictest2952497-21kworker/1:1+events20:49:221
292495499241012,2396cyclictest2979514-21kworker/1:0+events22:51:581
292495499240912,2394cyclictest2936368-21kworker/1:0+events19:49:291
292496099240812,2392cyclictest3015249-21kworker/3:1+events23:53:223
292495499240712,2391cyclictest2979514-21kworker/1:0+events22:29:141
292494699240712,2392cyclictest2979650-21kworker/0:0+events22:00:530
292495499240612,2390cyclictest2999019-21kworker/1:1+events23:09:511
292495499240612,2390cyclictest2973247-21kworker/1:0+events21:39:251
292495499240611,2391cyclictest2979514-21kworker/1:0+events22:02:251
292494699240612,2391cyclictest3023464-21kworker/0:2+events00:17:040
292495499240412,2388cyclictest2929661-21kworker/1:2+events20:06:411
292495499240411,2389cyclictest3022068-21kworker/1:0+events00:11:231
292494699240412,2388cyclictest2986099-21kworker/0:1+events22:23:300
292495499240311,2388cyclictest2969966-21kworker/1:2+events21:31:251
29249549924029,2390cyclictest2999019-21kworker/1:1+events23:16:551
292495499240210,2388cyclictest2992519-21kworker/1:1+events22:44:401
292496099240112,2386cyclictest3016924-21kworker/3:0+events00:23:303
292495499240112,2385cyclictest2995776-21kworker/1:2+events22:48:541
292495499240010,2387cyclictest3030161-21kworker/1:2+events00:36:291
292495499239911,2385cyclictest3005816-21kworker/1:0+events23:22:181
292494699239911,2384cyclictest2970265-21kworker/0:1+events21:39:490
292495499239811,2384cyclictest2950881-21kworker/1:2+events20:30:001
292494699239811,2383cyclictest2966928-21kworker/0:2+events21:19:550
292494699239612,2381cyclictest3011949-21kworker/0:1+events23:50:440
292494699239612,2381cyclictest2939524-21kworker/0:2+events20:10:290
292495499239511,2381cyclictest3015466-21kworker/1:1+events00:19:221
292494699239512,2379cyclictest2945915-21kworker/0:1+events20:19:380
292495499239412,2378cyclictest2992519-21kworker/1:1+events22:37:461
292494699239412,2378cyclictest2978032-21kworker/0:2+events22:05:510
292494699239410,2382cyclictest2978032-21kworker/0:2+events22:10:020
292494699239312,2379cyclictest2991079-21kworker/0:1+events22:55:020
292494699239312,2378cyclictest3000537-21kworker/0:2+events23:08:220
292494699239311,2378cyclictest2932976-21kworker/0:1+events19:42:410
292496099239212,2376cyclictest3009793-21kworker/3:2+events23:35:403
292494699239110,2377cyclictest2966928-21kworker/0:2+events21:41:430
29249469923902,2385cyclictest3011949-21kworker/0:1+events23:37:530
292494699239011,2377cyclictest3008717-21kworker/0:0+events00:20:070
292494699239011,2375cyclictest3027678-21kworker/0:2+events00:26:320
292495499238912,2373cyclictest2933066-21kworker/1:0+events19:37:221
292494699238912,2373cyclictest2932976-21kworker/0:1+events19:37:440
292494699238912,2373cyclictest2918105-21kworker/0:2+events19:28:120
292494699238911,2374cyclictest2939524-21kworker/0:2+events20:05:530
292494699238711,2372cyclictest2926375-21kworker/0:0+events19:19:340
292494699238710,2373cyclictest3019919-21kworker/0:2+events00:03:250
292496099238612,2370cyclictest2942830-21kworker/3:1+events20:15:283
292494699238611,2371cyclictest3008717-21kworker/0:0+events00:06:270
292494699238611,2371cyclictest3008717-21kworker/0:0+events00:06:270
292494699238512,2370cyclictest2989481-21kworker/0:2+events22:33:550
292495699238313,2367cyclictest3023700-21kworker/2:0+events00:24:162
292494699238212,2367cyclictest2954464-21kworker/0:0+events20:50:510
292496099238012,2364cyclictest2965270-21kworker/3:0+events21:25:173
292494699237912,2364cyclictest2939524-21kworker/0:2+events20:02:510
292494699237812,2362cyclictest2997217-21kworker/0:2+events22:53:540
292495699237612,2361cyclictest2957483-21kworker/2:1+events20:57:432
292495499237412,2358cyclictest3015466-21kworker/1:1+events00:04:121
292495699237212,2356cyclictest3018466-21kworker/2:1+events00:09:062
292495699237212,2356cyclictest3018466-21kworker/2:1+events00:09:062
292496099237110,2357cyclictest2960495-21kworker/3:1+events21:16:473
292496099236411,2349cyclictest2990893-21kworker/3:1+events22:35:513
292495499236210,2349cyclictest3015466-21kworker/1:1+events00:09:301
292495499236210,2349cyclictest3015466-21kworker/1:1+events00:09:291
29249469923629,2350cyclictest2950793-21kworker/0:1+events20:30:440
292496099236112,2345cyclictest2924732-21kworker/3:0+events19:26:443
292495699235910,2347cyclictest2999115-21kworker/2:2+events23:00:022
292496099235410,2341cyclictest2929841-21kworker/3:2+events19:23:453
292496099235312,2338cyclictest3007190-21kworker/3:1+events23:29:083
292496099235312,2337cyclictest3018711-21kworker/3:1+events00:03:043
292496099235311,2338cyclictest3000474-21kworker/3:0+events23:02:393
292495499235312,2338cyclictest2941219-21kworker/1:0+events19:58:161
292496099235113,2334cyclictest2973081-21kworker/3:0+events22:14:373
292496099235112,2336cyclictest2996019-21kworker/3:0+events22:51:123
292496099235110,2338cyclictest2986284-21kworker/3:2+events22:28:303
292496099235012,2334cyclictest3005494-21kworker/3:2+events23:23:043
292496099234912,2335cyclictest3016924-21kworker/3:0+events00:26:083
292496099234912,2334cyclictest3016924-21kworker/3:0+events00:30:433
292496099234911,2335cyclictest2962288-21kworker/3:2+events21:07:123
292496099234911,2334cyclictest3003810-21kworker/3:0+events23:16:333
292496099234812,2333cyclictest2966898-21kworker/3:2+events21:21:053
292496099234810,2334cyclictest2958713-21kworker/3:0+events21:03:283
292496099234612,2332cyclictest2941190-21kworker/3:2+events20:11:383
292494699234412,2328cyclictest2958809-21kworker/0:2+events20:55:490
292496099234212,2326cyclictest3009793-21kworker/3:2+events23:57:303
292494699234010,2327cyclictest2970265-21kworker/0:1+events21:31:010
292496099233812,2322cyclictest2942830-21kworker/3:1+events20:22:443
292495699233510,2322cyclictest3008460-21kworker/2:1+events23:41:362
29249609923349,2321cyclictest2965270-21kworker/3:0+events21:14:273
29249609923319,2319cyclictest3002122-21kworker/3:2+events23:10:123
292494699233112,2316cyclictest2963773-21kworker/0:0+events21:24:310
292494699232811,2314cyclictest2932976-21kworker/0:1+events19:33:550
292495699232610,2312cyclictest2973279-21kworker/2:2+events21:49:202
292494699232610,2312cyclictest2939524-21kworker/0:2+events19:55:130
292496099232412,2309cyclictest2951061-21kworker/3:1+events20:46:193
292495699232212,2307cyclictest2987740-21kworker/2:1+events22:24:172
292496099231410,2300cyclictest3009793-21kworker/3:2+events23:48:263
292495499231211,2297cyclictest3015466-21kworker/1:1+events23:51:531
292495499229711,2282cyclictest2931260-21kworker/1:1+events19:30:301
292495699228812,2272cyclictest3030037-21kworker/2:0+events00:33:462
292495699228711,2272cyclictest2929945-21kworker/2:2+events19:29:442
292496099228411,2269cyclictest2979825-21kworker/3:1+events22:08:563
292495499228310,2270cyclictest2950881-21kworker/1:2+events20:31:521
292495499228211,2269cyclictest2926059-21kworker/1:0+events19:15:021
292495699227310,2259cyclictest2957483-21kworker/2:1+events21:05:432
292495699226711,2252cyclictest2926301-21kworker/2:2+events19:15:242
292495699226610,2253cyclictest2978157-21kworker/2:2+events21:55:542
292495699226310,2250cyclictest2949296-21kworker/2:0+events20:54:412
292495699226310,2249cyclictest2976544-21kworker/2:0+events22:14:142
292495699226211,2247cyclictest2929945-21kworker/2:2+events19:32:462
292495699226211,2246cyclictest3015219-21kworker/2:1+events23:52:382
29249569922606,2250cyclictest2941084-21kworker/2:1+events20:10:522
292495699226011,2245cyclictest2934897-21kworker/2:2+events19:57:522
292495499226011,2246cyclictest2979514-21kworker/1:0+events22:13:531
292495699225911,2245cyclictest2976544-21kworker/2:0+events22:09:392
29249469922598,2249cyclictest2991079-21kworker/0:1+events22:40:280
292495699225711,2243cyclictest2968320-21kworker/2:0+events21:25:392
292495699225710,2243cyclictest3008460-21kworker/2:1+events23:39:242
292496099225411,2239cyclictest2931550-21kworker/3:1+events19:41:113
292495699225411,2239cyclictest2976544-21kworker/2:0+events22:34:172
29249609922506,2241cyclictest2973081-21kworker/3:0+events21:48:333
29249569922509,2237cyclictest2973279-21kworker/2:2+events21:43:352
29249569922449,2232cyclictest3008460-21kworker/2:1+events23:29:332
292496099224310,2230cyclictest2941190-21kworker/3:2+events19:59:473
292495499223611,2222cyclictest2979514-21kworker/1:0+events22:07:461
29249569922356,2226cyclictest2952533-21kworker/2:1+events20:36:282
29249569922349,2221cyclictest2990998-21kworker/2:2+events22:45:492
29249549922348,2222cyclictest3008509-21kworker/1:2+events23:32:351
292495499223410,2220cyclictest2957444-21kworker/1:2+events21:16:241
29249549922269,2214cyclictest3015466-21kworker/1:1+events23:55:381
29249549922269,2214cyclictest2945976-21kworker/1:0+events20:22:211
292495699222511,2210cyclictest2995851-21kworker/2:1+events22:58:512
292495499222211,2207cyclictest3028558-21kworker/1:0+events00:29:111
29249469922229,2210cyclictest3008717-21kworker/0:0+events00:12:100
292494699222211,2208cyclictest2978032-21kworker/0:2+events22:19:400
292495499221811,2204cyclictest2929661-21kworker/1:2+events19:42:181
292495499221710,2204cyclictest2986253-21kworker/1:1+events22:22:451
292496099221611,2202cyclictest3016924-21kworker/3:0+events00:08:423
292496099221611,2202cyclictest3016924-21kworker/3:0+events00:08:413
292495499221210,2199cyclictest3008509-21kworker/1:2+events23:44:171
292495499220810,2195cyclictest2973247-21kworker/1:0+events21:44:441
29249549922077,2197cyclictest2926059-21kworker/1:0+events19:20:411
292495499220710,2193cyclictest2962015-21kworker/1:0+events21:04:121
292495499220510,2192cyclictest2952497-21kworker/1:1+events20:35:411
29249549922048,2192cyclictest2957444-21kworker/1:2+events21:07:581
292495499220411,2190cyclictest3013727-21kworker/1:0+events23:47:401
292495699220110,2188cyclictest3006837-21kworker/2:2+events23:31:042
292495499219910,2186cyclictest3015466-21kworker/1:1+events00:23:521
29249469921969,2184cyclictest3027678-21kworker/0:2+events00:35:420
29249549921959,2183cyclictest2999019-21kworker/1:1+events23:04:121
29249549921897,2179cyclictest2974944-21kworker/1:1+events21:52:241
292496099218810,2175cyclictest2924732-21kworker/3:0+events19:17:403
292496099218710,2174cyclictest2929841-21kworker/3:2+events19:34:173
292494699218710,2174cyclictest3008717-21kworker/0:0+events23:40:530
292494699218611,2172cyclictest2994435-21kworker/0:0+events22:47:210
292494699218510,2171cyclictest2986099-21kworker/0:1+events22:26:360
29249609921847,2173cyclictest2974724-21kworker/3:2+events21:42:493
292495499218111,2167cyclictest2974944-21kworker/1:1+events21:45:301
292496099218012,2164cyclictest2941190-21kworker/3:2+events20:04:193
292495499218011,2167cyclictest2983011-21kworker/1:2+events22:15:021
292495499218011,2167cyclictest2954176-21kworker/1:0+events20:50:081
292495499218010,2166cyclictest2957444-21kworker/1:2+events20:59:391
29249549921799,2166cyclictest2974944-21kworker/1:1+events21:59:421
292495499217811,2164cyclictest2929661-21kworker/1:2+events20:12:001
292494699216711,2152cyclictest2950793-21kworker/0:1+events20:37:370
292494699216610,2153cyclictest3003842-21kworker/0:0+events23:17:200
292496099216211,2148cyclictest2921462-21kworker/3:2+events19:14:173
292496099216210,2149cyclictest2990893-21kworker/3:1+events22:33:083
29249609921619,2150cyclictest2966898-21kworker/3:2+events21:30:383
29249609921608,2149cyclictest2942830-21kworker/3:1+events20:07:263
29249609921588,2147cyclictest2951061-21kworker/3:1+events20:32:373
292496099215711,2143cyclictest3002122-21kworker/3:2+events23:07:583
292496099215710,2143cyclictest2978094-21kworker/3:2+events21:56:183
29249609921529,2139cyclictest2933202-21kworker/3:0+events19:38:083
292496099215211,2138cyclictest2951061-21kworker/3:1+events20:50:293
292496099214310,2130cyclictest3023371-21kworker/3:2+events00:15:573
292495699213211,2118cyclictest3006837-21kworker/2:2+events23:23:472
292496099212611,2112cyclictest2979825-21kworker/3:1+events22:03:113
29249569921087,2098cyclictest2918046-21kworker/2:1+events19:11:582
292495699210811,2094cyclictest3016846-21kworker/2:0+events00:01:142
292495699210310,2090cyclictest2947650-21kworker/2:1+events20:27:412
29249569920699,2057cyclictest3018466-21kworker/2:1+events00:13:172
292495499206510,2052cyclictest2923153-21kworker/1:2+events19:13:311
29249569920413,2035cyclictest2928321-21kworker/2:0+events19:24:082
292494699203910,2026cyclictest2924642-21kworker/0:1+events19:13:070
29249469920306,2021cyclictest2963773-21kworker/0:0+events21:29:060
292496099200210,1989cyclictest2931550-21kworker/3:1+events19:51:453
29249609920007,1989cyclictest2971403-21kworker/3:1+events21:36:443
29249609919749,1962cyclictest2979825-21kworker/3:1+events22:18:553
29244342164134,25sleep00-21swapper/019:06:170
292495699162158,2cyclictest2944390-21kworker/2:2+pm20:19:512
29244902149108,36sleep30-21swapper/319:06:593
29246072144109,31sleep10-21swapper/119:08:171
29246132141106,29sleep20-21swapper/219:08:222
292495699170,15cyclictest2971549-21runrttasks21:32:172
2924956991513,1cyclictest2964130-21/usr/sbin/munin21:10:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional