You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-12 - 05:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Thu Nov 12, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
247692871110,746sleep024778-21/usr/sbin/munin22:34:310
2345827509,730sleep14535-1kworker/1:0H22:30:001
27015245185,185sleep20-21swapper/219:14:522
12138244815,130sleep212211-21switchtime20:10:002
12138244815,130sleep212211-21switchtime20:10:002
18328242310,395sleep218451-21timerwakeupswit23:55:012
1240924239,324sleep212471-21grep21:54:552
31331235048,286sleep231306-21latency_hist19:29:342
21716231812,291sleep121723-21df_inode22:24:471
29335228379,186sleep229467-21unixbench_singl00:30:022
28303226996,119sleep228332-21awk00:29:302
370225012,221sleep31-21systemd19:34:313
5801224913,216sleep25836-21proc_pri19:49:572
32547223834,182sleep20-21swapper/221:14:572
8120222879,73sleep28186-21sendmail_mailqu21:39:572
253022227135,17sleep20-21swapper/200:19:322
2245222215,168sleep20-21swapper/219:39:472
3765221242,80sleep20-21swapper/219:44:402
21415221114,182sleep221519-21timerandwakeup00:05:032
14841220980,22sleep20-21swapper/223:44:512
13492219911,175sleep213003-21/usr/sbin/munin23:39:592
13492219911,175sleep213003-21/usr/sbin/munin23:39:592
2402121989,77sleep20-21swapper/220:49:332
29983219322,142sleep330047-21fschecks_time22:49:483
25720218524,125sleep225761-21ntpq19:09:552
11237218313,154sleep011267-21latency_hist23:34:320
18231217512,142sleep218247-21modprobe22:14:302
31239217015,72sleep231329-21sort22:54:322
29651216932,70sleep20-21swapper/222:49:312
27894216347,100sleep227966-21mailstats20:59:592
22413216041,23sleep00-21swapper/020:44:300
3184321509,101sleep231890-21ntpq22:54:522
32684214611,120sleep032759-21find21:14:590
7472214543,37sleep20-21swapper/219:55:022
8123213613,105sleep14535-1kworker/1:0H23:24:311
69321348,43sleep00-21swapper/019:34:450
16205213245,66sleep20-21swapper/223:49:482
16205213245,66sleep20-21swapper/223:49:482
5091213114,92sleep20-21swapper/221:30:482
740921298,68sleep27440-21latency_hist21:39:332
23881212952,64sleep223943-21find19:04:592
28649212820,35sleep128682-21basename21:04:301
6573212610,54sleep26688-21latency_hist23:19:322
6573212610,54sleep26688-21latency_hist23:19:322
23426212634,32sleep223424-21apt-get22:29:402
17616212617,90sleep317733-21cat22:09:593
23036212540,68sleep223180-21gpgv22:29:322
30261212348,63sleep230240-21latency_hist21:09:322
23201212320,71sleep00-21swapper/000:10:050
17989212140,69sleep217516-21apt-get20:29:342
31398212012,67sleep231411-21awk21:10:052
29144211946,59sleep229266-21cat19:20:012
14964211842,64sleep214991-21cut20:19:532
14964211842,64sleep214991-21cut20:19:532
1113921157,71sleep211183-21ntpq21:49:552
24731211421,68sleep124927-21latency_hist19:09:341
26275211313,47sleep226306-21awk22:39:302
19231211235,37sleep20-21swapper/223:59:412
9279211010,64sleep29374-21perl21:44:502
1133211030,12sleep20-21swapper/219:34:552
3675210935,35sleep20-21swapper/223:09:422
3675210935,35sleep20-21swapper/223:09:422
2752210948,45sleep22875-21grep23:04:582
16729210910,60sleep216872-21latency_hist22:09:332
32542210712,79sleep132546-21ntpq21:14:561
580210551,40sleep223169-21cyclictest22:59:452
25343210515,74sleep125352-21cpu19:09:461
21406210518,71sleep321491-21latency_hist22:24:323
21406210518,71sleep321491-21latency_hist22:24:323
19857210515,76sleep319933-21sed22:19:323
10457210521,48sleep20-21swapper/220:04:552
2989210411,80sleep23003-21kernelversion21:24:512
8727210311,65sleep00-21swapper/023:24:550
32483210212,75sleep332528-21cat21:14:543
1663421026,11sleep20-21swapper/220:24:542
24222210111,74sleep024315-21cut00:14:540
19291210119,37sleep20-21swapper/222:15:022
24572210039,44sleep20-21swapper/220:49:542
12054210010,75sleep212142-21cat23:34:582
128629842,41sleep20-21swapper/221:19:502
128629842,41sleep20-21swapper/221:19:502
678829710,67sleep00-21swapper/023:19:420
2651829617,66sleep226665-21grep20:55:022
1104029614,63sleep011107-21/usr/sbin/munin21:49:530
91462959,32sleep29256-21timerwakeupswit20:00:022
283222949,69sleep00-21swapper/022:44:410
32229315,63sleep1358-21latency_hist22:59:321
432829210,67sleep04433-21sort21:29:330
2182229213,66sleep021877-21cat22:24:500
3214529010,42sleep232238-21cat19:30:022
84382897,13sleep20-21swapper/223:24:492
449128819,40sleep34011-21apt-get21:29:403
232628713,58sleep02370-21if_err_eth023:04:480
2313528740,26sleep00-21swapper/019:04:320
1518328738,33sleep20-21swapper/222:04:322
2680728612,57sleep126775-21latency_hist00:24:321
893328419,30sleep30-21swapper/319:59:593
2246528412,36sleep20-21swapper/220:44:322
1341128416,53sleep113427-21kernelversion20:14:511
1341128416,53sleep113427-21kernelversion20:14:511
2415828216,50sleep324176-21/usr/sbin/munin22:30:463
200872827,14sleep20-21swapper/222:19:432
200872827,14sleep20-21swapper/222:19:432
196092829,12sleep20-21swapper/220:34:472
2971728116,20sleep00-21swapper/021:05:020
2704328122,22sleep00-21swapper/000:24:430
2172828129,32sleep321817-21grep22:24:493
2710028028,30sleep10-21swapper/120:59:301
269772807,15sleep20-21swapper/222:39:542
1672128020,38sleep116092-21/usr/sbin/munin23:49:591
1672128020,38sleep116092-21/usr/sbin/munin23:49:591
1503928014,47sleep015078-21processes23:44:580
1420728011,14sleep20-21swapper/220:16:542
1420728011,14sleep20-21swapper/220:16:542
1440827913,52sleep314435-21sh20:19:303
1440827913,52sleep314435-21sh20:19:303
1091027935,10sleep20-21swapper/223:31:152
2575227813,13sleep20-21swapper/200:19:532
21252786,23sleep10-21swapper/123:04:411
2007127816,32sleep30-21swapper/320:35:003
53392778,51sleep10-21swapper/119:49:441
2303927620,34sleep323194-21grep00:10:043
1195627617,35sleep111994-21/usr/sbin/munin20:09:551
1195627617,35sleep111994-21/usr/sbin/munin20:09:551
2871527513,15sleep00-21swapper/021:04:330
2008927516,42sleep020152-21sendmail_mailqu20:35:010
806527432,10sleep20-21swapper/223:24:242
1778327410,49sleep117790-21diskstats23:54:481
154212746,50sleep00-21swapper/022:04:470
148922740,7sleep30-21swapper/323:44:513
1020527410,46sleep110272-21memory23:29:521
967027318,26sleep10-21swapper/123:29:321
2249527323,32sleep122629-21grep19:00:011
1490627313,41sleep114963-21/usr/sbin/munin20:19:521
1490627313,41sleep114963-21/usr/sbin/munin20:19:521
1386927319,33sleep113866-21apt-get21:59:431
825027216,22sleep10-21swapper/119:59:331
807127220,30sleep38080-21modprobe23:24:293
264032725,58sleep00-21swapper/019:11:510
1537827215,16sleep30-21swapper/320:20:023
1537827215,16sleep30-21swapper/320:20:023
394827113,41sleep14014-21wc23:09:531
394827113,41sleep14014-21wc23:09:531
2895027122,28sleep10-21swapper/121:04:431
2720027116,39sleep127278-21sed19:14:551
2632627114,39sleep026413-21grep22:39:320
235942716,30sleep20-21swapper/200:14:032
222227122,29sleep30-21swapper/323:04:473
2798127010,26sleep30-21swapper/321:00:003
2498927014,24sleep10-21swapper/122:34:401
3116026918,25sleep00-21swapper/022:54:300
2242926919,28sleep10-21swapper/100:09:501
1730126910,47sleep00-21swapper/023:53:300
1730126910,47sleep00-21swapper/023:53:300
1278926914,26sleep30-21swapper/321:55:043
2859026812,30sleep30-21swapper/319:19:503
2387226812,14sleep10-21swapper/120:48:151
2350026813,39sleep323610-21/usr/sbin/munin20:45:023
2263326822,25sleep20-21swapper/219:00:022
1737726818,31sleep317434-21modprobe23:54:293
1737726818,31sleep317434-21modprobe23:54:293
571226712,42sleep05721-21uname23:14:570
571226712,42sleep05721-21uname23:14:570
2805926714,28sleep10-21swapper/122:44:331
251026717,26sleep10-21swapper/119:39:541
2440226716,22sleep30-21swapper/320:49:503
2104526740,14sleep10-21swapper/100:04:531
1808326723,22sleep00-21swapper/023:54:550
1221526714,28sleep30-21swapper/320:10:013
1221526714,28sleep30-21swapper/320:10:013
1133626718,31sleep311415-21find21:50:003
702226618,27sleep00-21swapper/019:54:510
2844326613,23sleep10-21swapper/122:44:541
1428226616,33sleep014296-21modprobe23:44:290
1428226616,33sleep014296-21modprobe23:44:290
294502657,12sleep10-21swapper/119:20:531
2787326514,27sleep30-21swapper/322:44:313
2361526517,25sleep30-21swapper/319:04:513
412826417,31sleep34323-21latency_hist21:29:323
2998026417,27sleep00-21swapper/000:34:320
2796326419,18sleep00-21swapper/022:44:320
2741326413,16sleep30-21swapper/319:15:013
2471526419,31sleep024727-21munin-node19:09:330
2117926413,33sleep021224-21/usr/sbin/munin00:04:560
1292126412,37sleep113003-21awk21:55:071
919326314,29sleep10-21swapper/121:44:451
275126317,28sleep02845-21grep19:40:020
2078226317,25sleep320801-21apt-get00:04:433
132426315,25sleep30-21swapper/323:00:023
835326213,22sleep10-21swapper/121:40:001
2966126213,22sleep30-21swapper/322:49:323
2307826223,21sleep30-21swapper/319:04:313
1587226214,24sleep10-21swapper/123:49:311
77526121,18sleep30-21swapper/319:34:483
470626118,23sleep10-21swapper/119:45:031
315132617,42sleep00-21swapper/021:11:200
2452526115,18sleep10-21swapper/120:49:531
2067926118,25sleep10-21swapper/122:20:021
2067926118,25sleep10-21swapper/122:20:021
2026026114,32sleep320261-21sshd00:01:163
1904526110,38sleep0367-21snmpd20:31:220
151662619,27sleep00-21swapper/020:19:590
151662619,27sleep00-21swapper/020:19:590
116226115,30sleep1666-21/usr/sbin/munin19:34:581
1050126110,34sleep00-21swapper/021:49:320
45226014,25sleep00-21swapper/019:34:320
3051126017,25sleep330593-21mailstats19:25:023
222142604,13sleep10-21swapper/120:41:571
633525915,23sleep10-21swapper/121:34:511
505425915,21sleep00-21swapper/023:14:330
505425915,21sleep00-21swapper/023:14:330
3096225916,29sleep130985-21/usr/sbin/munin21:10:001
286782597,14sleep00-21swapper/019:19:510
2597825914,29sleep326044-21fw_packets20:54:513
2431625920,19sleep10-21swapper/100:14:551
181662597,12sleep30-21swapper/323:54:583
1802425916,27sleep30-21swapper/320:29:433
15312596,11sleep00-21swapper/023:00:320
1423925914,18sleep00-21swapper/021:59:540
755725818,19sleep30-21swapper/319:55:033
2903625816,28sleep329058-21fschecks_count21:04:493
22872585,11sleep00-21swapper/021:22:310
22872585,11sleep00-21swapper/021:22:310
1837325815,26sleep118441-21ls20:30:501
1715725814,17sleep10-21swapper/122:09:481
137925810,26sleep30-21swapper/321:19:513
137925810,26sleep30-21swapper/321:19:513
1127525817,26sleep111282-21modprobe20:09:301
904725720,16sleep10-21swapper/120:00:011
552725723,18sleep30-21swapper/319:49:503
404725720,21sleep30-21swapper/323:09:553
404725720,21sleep30-21swapper/323:09:553
399225717,26sleep14019-21munin-run21:29:301
27102575,11sleep30-21swapper/319:40:013
2559825715,17sleep00-21swapper/000:19:510
1342825711,20sleep30-21swapper/323:39:563
1342825711,20sleep30-21swapper/323:39:563
1279125712,25sleep00-21swapper/023:39:320
1199925716,25sleep112003-21cron21:54:301
533825614,25sleep05333-21apt-get19:49:430
2877625613,26sleep328829-21if_err_eth000:29:513
2589525617,16sleep30-21swapper/319:10:003
203612566,12sleep10-21swapper/100:02:311
1900425615,25sleep119010-21ntp_states22:14:571
1414425615,23sleep30-21swapper/321:59:533
1041025617,20sleep30-21swapper/323:29:583
956125520,20sleep30-21swapper/321:44:553
881425515,24sleep38820-21ntp_states23:24:583
846925514,26sleep08465-21apt-get19:59:440
618525516,24sleep36280-21perl21:34:493
577025517,18sleep30-21swapper/323:15:003
577025517,18sleep30-21swapper/323:15:003
53522558,14sleep10-21swapper/123:14:501
53522558,14sleep10-21swapper/123:14:501
3147325513,24sleep031467-21apt-get22:54:420
2999925515,20sleep10-21swapper/119:24:441
2715625515,15sleep00-21swapper/020:59:330
174212558,34sleep017455-21ps22:09:560
171562555,11sleep30-21swapper/320:25:033
1153925514,24sleep011558-21apt-get20:09:410
1153925514,24sleep011558-21apt-get20:09:410
1069225513,25sleep39990-21/usr/sbin/munin20:05:003
70682545,37sleep00-21swapper/021:35:470
3050025412,26sleep330496-21apt-get21:09:453
273125414,24sleep02798-21df21:24:450
2479025413,24sleep024861-21sensors_temp20:50:000
2290925412,27sleep022961-21/usr/sbin/munin20:44:510
1763325416,20sleep00-21swapper/020:29:320
2859425312,25sleep128588-21apt-get00:29:451
2557925315,23sleep325591-21fschecks_count00:19:493
1854925311,26sleep018543-21apt-get22:14:440
1667525316,22sleep316709-21modprobe22:09:303
809825213,23sleep08104-21ntp_states21:39:550
456325214,18sleep30-21swapper/319:45:013
3154625212,25sleep331555-21/usr/sbin/munin22:54:453
1661025215,20sleep10-21swapper/120:24:531
139482526,16sleep10-21swapper/123:40:101
139482526,16sleep10-21swapper/123:40:101
57925111,24sleep1574-21apt-get22:59:441
378925112,22sleep03785-21apt-get19:44:420
3029625112,20sleep130337-21ntpq22:49:551
1187825118,16sleep10-21swapper/123:34:531
1011325111,23sleep010222-21perl20:04:490
578225013,23sleep10-21swapper/121:34:311
2749525011,16sleep30-21swapper/300:24:543
2703925012,22sleep127036-21apt-get00:24:401
2587025011,15sleep30-21swapper/322:35:033
2419725012,22sleep324220-21irqstats00:14:533
1672625014,21sleep116694-21/usr/sbin/munin22:09:321
163625013,21sleep11642-21ntp_states21:19:571
163625013,21sleep11642-21ntp_states21:19:571
2716524911,20sleep027199-21irqstats19:14:540
2017324910,16sleep00-21swapper/022:19:490
2017324910,16sleep00-21swapper/022:19:490
1193324912,20sleep311939-21ntp_states23:34:553
3046824812,22sleep030479-21ntp_states19:24:590
2121424813,20sleep321225-21df_inode20:39:453
127252489,16sleep00-21swapper/021:55:010
89042478,14sleep30-21swapper/321:44:303
1314924712,18sleep00-21swapper/020:14:420
1314924712,18sleep00-21swapper/020:14:420
208652468,19sleep120893-21munin-run20:39:311
193272469,15sleep00-21swapper/023:59:480
315672449,19sleep131562-21apt-get19:29:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional