You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 14:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Mar 10, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27261275306,7512sleep319412-21kworker/3:100:11:193
27261275306,7512sleep319412-21kworker/3:100:11:193
4329219348,1900sleep023547-21kworker/u8:222:58:420
1575927047,684sleep315807-21unin-run21:53:263
3124265313,622sleep23191-21df19:28:412
3124265313,622sleep23191-21df19:28:412
12661263879,299sleep312682-21awk21:43:253
12661263879,299sleep312682-21awk21:43:253
11371235910,331sleep011433-21/usr/sbin/munin19:53:540
13514227119,238sleep213516-21sshd20:00:382
18134223018,195sleep318149-21latency_hist23:43:263
18134223018,195sleep318149-21latency_hist23:43:263
10598222913,130sleep310615-21ls19:53:253
3163122128,161sleep331608-21latency_hist22:43:273
23583220413,95sleep30-21swapper/320:33:413
23583220413,95sleep30-21swapper/320:33:413
1116722020,27sleep328747-21cyclictest21:38:273
22739219528,149sleep022766-21munin-plugin-st23:58:240
29245219328,81sleep30-21swapper/322:33:553
1632218977,97sleep31755-21timerwakeupswit22:48:563
1978721889,159sleep319895-21timerandwakeup22:03:553
1893821648,142sleep319042-21latency_hist22:03:273
11621215812,129sleep311774-21grep19:53:573
29571215715,57sleep329710-21latency_hist20:53:273
19387215450,86sleep30-21swapper/320:18:523
2034721517,133sleep00-21swapper/022:07:150
8062214451,78sleep30-21swapper/319:43:483
2872321430,8sleep30-21swapper/319:08:273
22498214311,55sleep322537-21proc_pri20:28:533
16323214219,43sleep30-21swapper/320:09:073
16323214219,43sleep30-21swapper/320:09:073
28496213548,21sleep30-21swapper/322:33:263
28496213548,21sleep30-21swapper/322:33:263
12256213516,24sleep30-21swapper/323:23:443
12256213516,24sleep30-21swapper/323:23:443
7308213311,28sleep30-21swapper/321:23:553
109521339,102sleep31182-21sort21:03:573
29558213125,59sleep20-21swapper/200:18:452
29558213125,59sleep20-21swapper/200:18:452
2380221300,42sleep223804-21latency_hist22:18:272
6816212817,20sleep00-21swapper/021:23:440
5913212752,17sleep30-21swapper/323:03:423
9082212616,91sleep09180-21/usr/sbin/munin21:28:570
5997212614,94sleep26015-21latency_hist19:38:262
5997212614,94sleep26015-21latency_hist19:38:262
29046212525,81sleep30-21swapper/319:08:483
429621247,50sleep32117-1kworker/3:1H22:58:373
10326212313,46sleep30-21swapper/323:18:273
3863212225,76sleep14004-21unixbench_singl19:28:571
3863212225,76sleep14004-21unixbench_singl19:28:571
2344212112,72sleep32423-21timerandwakeup00:33:563
23845212012,93sleep223857-21kernelversion20:33:492
23845212012,93sleep223857-21kernelversion20:33:492
22360211811,92sleep322407-21fschecks_count22:13:443
19694211617,82sleep30-21swapper/323:48:263
19694211617,82sleep30-21swapper/323:48:263
20576211522,67sleep020614-21grep20:23:460
2027211518,37sleep00-21swapper/021:08:380
2027211518,37sleep00-21swapper/021:08:380
3687211411,84sleep30-21swapper/321:13:463
26902211431,67sleep30-21swapper/322:28:263
26902211431,67sleep30-21swapper/322:28:263
22751211317,76sleep122809-21/usr/sbin/munin22:13:541
20417211313,81sleep220468-21/usr/sbin/munin23:48:542
766211129,28sleep3914-21cat22:48:283
458211122,62sleep0477-21apt-get21:03:380
1733121118,84sleep30-21swapper/320:13:393
6153210917,74sleep16245-21/usr/sbin/munin23:03:481
1104721099,85sleep211109-21/usr/sbin/munin23:18:542
1104721099,85sleep211109-21/usr/sbin/munin23:18:542
5954210815,76sleep05965-21/usr/sbin/munin23:03:430
29712210812,74sleep029861-21/usr/sbin/munin19:09:060
23745210841,14sleep30-21swapper/323:58:563
4946210719,72sleep04903-21latency_hist21:18:270
24329210728,59sleep00-21swapper/000:03:250
2147821077,84sleep30-21swapper/323:53:393
5786210610,78sleep30-21swapper/321:18:553
27372210521,67sleep027442-21df19:03:410
25123210520,71sleep30-21swapper/322:20:263
15999210524,60sleep116149-21grep23:33:551
15999210524,60sleep116149-21grep23:33:551
18469210317,70sleep118524-21fschecks_count23:43:441
18469210317,70sleep118524-21fschecks_count23:43:441
16728210319,62sleep116876-21unixbench_multi21:53:561
12018210316,71sleep212148-21wc21:38:562
12018210316,71sleep212148-21wc21:38:562
30868210221,54sleep30-21swapper/319:13:373
9617210115,57sleep30-21swapper/321:33:283
2490229910,10sleep30-21swapper/300:03:483
780229729,48sleep30-21swapper/323:08:503
1738229728,57sleep317383-21munin-run21:58:263
2911429414,67sleep229138-21netstat22:33:492
276402945,78sleep30-21swapper/320:44:153
2180429318,65sleep30-21swapper/322:11:063
214072936,16sleep30-21swapper/320:24:053
183529327,34sleep20-21swapper/221:08:282
1253529321,15sleep30-21swapper/319:58:423
3108229222,54sleep131149-21munin-run20:58:261
242929212,63sleep22476-21users19:23:572
242929212,63sleep22476-21users19:23:572
327182919,41sleep30-21swapper/300:28:433
2806929130,39sleep228227-21sed19:03:552
1410829114,60sleep114181-21fschecks_time20:03:441
132872919,70sleep00-21swapper/023:26:430
132872919,70sleep00-21swapper/023:26:430
3130929022,49sleep30-21swapper/300:23:463
3130929022,49sleep30-21swapper/300:23:463
3043229013,63sleep330551-21timerwakeupswit20:53:563
251029010,68sleep32551-21grep22:53:403
105632908,18sleep30-21swapper/323:18:393
105632908,18sleep30-21swapper/323:18:393
287328921,12sleep30-21swapper/319:28:263
287328921,12sleep30-21swapper/319:28:263
1998928914,56sleep00-21swapper/020:20:430
219472889,65sleep122011-21ps23:53:531
3170228713,18sleep00-21swapper/000:23:550
3170228713,18sleep00-21swapper/000:23:550
148762879,14sleep30-21swapper/320:04:033
12872878,67sleep31-21systemd19:23:263
12872878,67sleep31-21systemd19:23:263
3150728615,58sleep231629-21cat19:13:562
2467628621,53sleep124753-21switchtime22:18:591
1194928520,47sleep112017-21/usr/sbin/munin21:38:551
1194928520,47sleep112017-21/usr/sbin/munin21:38:551
45628413,57sleep0495-21expr00:28:480
66892837,35sleep30-21swapper/319:38:523
66892837,35sleep30-21swapper/319:38:523
2871428314,54sleep128733-21apt-get22:33:361
1697328323,41sleep017186-21latency_hist20:13:280
1697328323,41sleep017186-21latency_hist20:13:280
804128219,28sleep20-21swapper/223:08:572
204822820,7sleep20-21swapper/222:08:252
149682827,41sleep314998-21munin-run23:33:243
1420828214,46sleep214211-21cron21:48:242
75342810,9sleep10-21swapper/119:43:251
75342810,9sleep10-21swapper/119:43:251
295812807,36sleep30-21swapper/300:19:023
295812807,36sleep30-21swapper/300:19:023
2899028023,35sleep20-21swapper/200:18:242
1083628021,44sleep010896-21latency23:18:480
1083628021,44sleep010896-21latency23:18:480
46682796,16sleep30-21swapper/319:33:393
46682796,16sleep30-21swapper/319:33:393
23252797,36sleep30-21swapper/321:08:493
23252797,36sleep30-21swapper/321:08:493
156742787,13sleep30-21swapper/323:33:493
156742787,13sleep30-21swapper/323:33:493
3178027713,41sleep131930-21unixbench_multi00:23:561
3178027713,41sleep131930-21unixbench_multi00:23:561
2919627715,26sleep10-21swapper/119:08:501
284372776,18sleep328578-21awk00:13:573
1031227713,46sleep110374-21/usr/sbin/munin21:33:541
58822769,12sleep00-21swapper/019:37:480
58822769,12sleep00-21swapper/019:37:480
2923027613,47sleep129010-21latency_hist00:18:271
2561927613,40sleep10-21swapper/120:38:521
2332627616,41sleep223340-21grep23:58:472
154427618,32sleep10-21swapper/119:23:361
154427618,32sleep10-21swapper/119:23:361
1426227626,27sleep10-21swapper/121:48:271
182627524,28sleep00-21swapper/000:33:440
3080727420,34sleep130872-21awk22:38:531
3036127420,32sleep20-21swapper/220:53:552
2695527423,32sleep127083-21latency_hist22:28:281
2695527423,32sleep127083-21latency_hist22:28:281
255622748,53sleep325604-21ntpq20:38:503
2542727414,33sleep00-21swapper/020:38:470
1439227418,16sleep20-21swapper/223:28:562
1378027419,21sleep00-21swapper/023:28:410
1869727313,16sleep20-21swapper/220:18:272
1347027319,33sleep113520-21wc23:28:261
1347027319,33sleep113520-21wc23:28:261
788627217,25sleep10-21swapper/123:08:521
644227220,36sleep16448-21modprobe21:23:251
393027218,24sleep20-21swapper/222:58:282
3179027226,33sleep031802-21vmstat19:13:580
3107027223,31sleep031080-21modprobe20:58:250
1865127218,36sleep018666-21kernelversion23:43:470
1865127218,36sleep018666-21kernelversion23:43:470
68542718,11sleep20-21swapper/223:04:462
317862716,30sleep30-21swapper/320:58:483
3086127114,15sleep00-21swapper/000:23:270
3086127114,15sleep00-21swapper/000:23:270
307852718,17sleep00-21swapper/022:38:520
200062717,47sleep30-21swapper/323:48:423
135582719,41sleep30-21swapper/321:43:573
942427017,33sleep09431-21/usr/sbin/munin19:48:460
2562727023,25sleep20-21swapper/220:38:532
1738527017,26sleep00-21swapper/021:58:270
564226910,46sleep05649-21open_files21:18:510
40162698,14sleep10-21swapper/121:13:521
3026426913,36sleep230352-21munin-node19:13:272
2914226919,31sleep029191-21munin-run00:18:260
1319526925,21sleep10-21swapper/119:58:561
1292626916,33sleep012945-21apt-get21:43:380
943226819,32sleep10-21swapper/119:48:471
910426821,24sleep20-21swapper/219:48:272
907326812,40sleep09179-21tune2fs23:13:440
3074026813,40sleep230776-21ntp_states22:38:502
1604726814,29sleep20-21swapper/221:53:412
1397226820,25sleep10-21swapper/123:28:461
1037526814,38sleep210453-21switchtime21:33:552
846926713,16sleep00-21swapper/019:43:570
835226718,27sleep10-21swapper/121:28:411
675226723,21sleep20-21swapper/219:38:552
675226723,21sleep20-21swapper/219:38:552
421326720,19sleep20-21swapper/221:13:572
3214326721,26sleep10-21swapper/122:43:501
3245126624,24sleep10-21swapper/119:18:351
3245126624,24sleep10-21swapper/119:18:351
1909726612,31sleep10-21swapper/120:18:461
3196026513,34sleep232035-21fschecks_time22:43:472
30922655,28sleep30-21swapper/319:28:343
30922655,28sleep30-21swapper/319:28:343
2930826520,20sleep00-21swapper/020:50:420
2761726513,36sleep127640-21irqstats19:03:461
2637326527,17sleep20-21swapper/222:23:572
2637326527,17sleep20-21swapper/222:23:572
2791726416,15sleep10-21swapper/122:28:561
2791726416,15sleep10-21swapper/122:28:561
257162646,11sleep00-21swapper/000:06:190
2299226419,24sleep00-21swapper/022:13:580
174872645,12sleep10-21swapper/121:58:281
93826319,28sleep0957-21apt-get22:48:370
937226316,27sleep10-21swapper/123:13:491
2203526317,21sleep20-21swapper/222:13:272
1718826315,27sleep10-21swapper/120:13:291
1097226316,25sleep20-21swapper/219:53:442
2469226218,25sleep124701-21sshd00:03:421
1581026218,21sleep10-21swapper/121:53:281
118462628,12sleep10-21swapper/123:23:051
118462628,12sleep10-21swapper/123:23:051
3185226115,19sleep20-21swapper/220:58:512
3147026117,20sleep10-21swapper/120:58:421
2855126118,27sleep028701-21modprobe22:33:270
2282526117,28sleep222894-21sed23:58:262
173426118,26sleep11731-21apt-get00:33:371
1511326118,19sleep20-21swapper/221:48:572
325822608,38sleep00-21swapper/021:02:290
2633626020,23sleep00-21swapper/000:08:460
2633626020,23sleep00-21swapper/000:08:460
2398326012,18sleep10-21swapper/120:33:521
2398326012,18sleep10-21swapper/120:33:521
1954526013,30sleep119591-21/usr/sbin/munin22:03:491
1693526014,23sleep20-21swapper/223:38:442
1693526014,23sleep20-21swapper/223:38:442
165172606,42sleep20-21swapper/223:38:242
165172606,42sleep20-21swapper/223:38:242
1418226018,26sleep214240-21/usr/sbin/munin20:03:452
210422595,18sleep00-21swapper/023:51:300
1732425913,26sleep00-21swapper/020:13:360
82525817,23sleep2923-21users19:18:572
82525817,23sleep2923-21users19:18:572
671325815,27sleep16724-21apt-get21:23:371
26582587,38sleep011-21ksoftirqd/019:26:090
26582587,38sleep011-21ksoftirqd/019:26:090
2618825812,26sleep026256-21/usr/sbin/munin22:23:550
2618825812,26sleep026256-21/usr/sbin/munin22:23:550
1915225817,23sleep219209-21irqstats20:18:472
1584025820,18sleep10-21swapper/120:08:491
1584025820,18sleep10-21swapper/120:08:491
1273425819,19sleep00-21swapper/019:58:470
1108525818,23sleep00-21swapper/021:38:250
954825710,21sleep20-21swapper/223:13:542
32625714,26sleep2396-21/usr/sbin/munin00:28:462
1993425716,26sleep119926-21apt-get23:48:401
1533125715,19sleep00-21swapper/020:08:280
692825620,20sleep00-21swapper/019:38:570
692825620,20sleep00-21swapper/019:38:570
1815425612,18sleep00-21swapper/023:43:270
1815425612,18sleep00-21swapper/023:43:270
1557225613,27sleep215635-21/usr/sbin/munin20:08:422
1557225613,27sleep215635-21/usr/sbin/munin20:08:422
1002725618,20sleep20-21swapper/219:49:002
509025520,16sleep10-21swapper/122:59:201
3006625518,22sleep130103-21ps00:19:321
3006625518,22sleep130103-21ps00:19:321
1967525516,24sleep219693-21munin-run23:48:252
1967525516,24sleep219693-21munin-run23:48:252
180625518,19sleep10-21swapper/121:08:271
2719025414,24sleep027184-21apt-get22:28:390
2719025414,24sleep027184-21apt-get22:28:390
2670725411,26sleep126701-21apt-get20:43:411
2203625411,23sleep20-21swapper/220:28:402
712725316,21sleep20-21swapper/221:23:502
3131125314,22sleep131348-21ntp_states19:13:511
282572539,18sleep20-21swapper/220:48:422
277172537,37sleep20-21swapper/220:45:152
2598825313,23sleep226128-21modprobe00:08:272
2598825313,23sleep226128-21modprobe00:08:272
230932535,36sleep00-21swapper/020:30:480
2247125314,25sleep122490-21ntp_states20:28:511
2103025313,23sleep021095-21memory22:08:490
2049225314,15sleep10-21swapper/120:23:421
827825215,21sleep28272-21apt-get21:28:382
45425219,17sleep00-21swapper/022:45:420
2458925211,18sleep00-21swapper/022:18:580
2380225215,20sleep223989-21apt-config22:18:282
2119425212,28sleep120702-21/usr/sbin/munin22:08:541
209925215,21sleep12127-21df21:08:431
209925215,21sleep12127-21df21:08:431
1811525216,18sleep20-21swapper/221:58:542
1762125211,24sleep017615-21apt-get21:58:400
1471525215,20sleep014768-21irqstats21:48:470
1215925214,22sleep212153-21apt-get23:23:392
1215925214,22sleep212153-21apt-get23:23:392
247032518,17sleep20-21swapper/200:03:432
2346425111,23sleep123511-21cat23:58:501
2179825113,16sleep20-21swapper/220:28:282
1571225112,22sleep015718-21ntp_states23:33:500
1571225112,22sleep015718-21ntp_states23:33:500
10392516,36sleep00-21swapper/019:20:200
10392516,36sleep00-21swapper/019:20:200
37702509,24sleep03846-21apt-config22:58:260
3156625014,16sleep20-21swapper/200:23:532
3156625014,16sleep20-21swapper/200:23:532
205025011,23sleep22046-21apt-get21:08:402
205025011,23sleep22046-21apt-get21:08:402
1224125014,16sleep10-21swapper/119:58:271
645024913,21sleep21-21systemd21:23:262
373724910,23sleep00-21swapper/019:28:560
373724910,23sleep00-21swapper/019:28:560
3264624912,24sleep132647-21cpu00:28:411
2735724914,15sleep20-21swapper/222:28:452
2735724914,15sleep20-21swapper/222:28:452
121124910,22sleep21234-21irqstats22:48:472
909424811,23sleep19103-21pluginstate19:48:261
2405124815,15sleep00-21swapper/020:33:550
2405124815,15sleep00-21swapper/020:33:550
196224812,19sleep22007-21irqstats00:33:462
506024714,16sleep10-21swapper/119:33:491
506024714,16sleep10-21swapper/119:33:491
863424616,18sleep00-21swapper/023:12:430
384924615,16sleep10-21swapper/122:58:271
2834224612,19sleep10-21swapper/120:48:451
2712524611,17sleep00-21swapper/020:43:520
523524411,21sleep25258-21users22:59:212
171112439,15sleep10-21swapper/123:38:471
171112439,15sleep10-21swapper/123:38:471
10142435,15sleep10-21swapper/122:48:421
262412429,16sleep10-21swapper/100:08:421
262412429,16sleep10-21swapper/100:08:421
113992429,15sleep00-21swapper/021:38:370
113992429,15sleep00-21swapper/021:38:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional