You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-12 - 14:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Mon Jan 12, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28952178118,1747sleep028508-1kworker/0:0H23:20:430
7765210678,1049sleep221850-1kworker/2:2H23:32:132
9734210197,1001sleep39767-21ntpq20:15:043
9734210197,1001sleep39767-21ntpq20:15:043
389129228,904sleep028508-1kworker/0:0H23:21:080
2222526485,633sleep028508-1kworker/0:0H22:35:320
444625116,493sleep328623-21kworker/3:223:22:243
506824617,444sleep330095-21snmpd21:43:073
2967324315,419sleep028508-1kworker/0:0H21:18:130
2967324315,419sleep028508-1kworker/0:0H21:18:130
12839240290,217sleep313029-21fschecks_time20:26:273
26129239610,308sleep326261-21unixbench_singl00:30:123
12411236510,347sleep221850-1kworker/2:2H20:24:322
1487232686,186sleep31643-21unixbench_multi21:30:113
21236225570,92sleep30-21swapper/319:09:583
19192225278,157sleep30-21swapper/320:45:013
6857224451,107sleep36873-21fschecks_count23:30:563
21798223040,16sleep30-21swapper/320:54:423
31654222355,106sleep331705-21sort21:24:543
15514220935,156sleep315525-21modprobe20:34:393
23396220180,46sleep30-21swapper/319:15:153
11878219621,145sleep111984-21wc20:21:081
2190321887,168sleep321975-21find22:35:193
2466421795,139sleep36192-1kworker/3:2H19:19:473
13320217757,101sleep313231-21/usr/sbin/munin22:09:563
24352216776,24sleep30-21swapper/321:00:113
23214216613,140sleep023295-21grep19:15:120
1905521640,7sleep10-21swapper/120:44:581
18996215816,121sleep218902-21/usr/sbin/munin20:44:552
19123215343,25sleep30-21swapper/300:09:493
31391215130,106sleep331450-21find19:40:073
2827421509,87sleep30-21swapper/319:30:093
17703214724,109sleep317822-21latency_hist22:24:413
15269214516,65sleep315309-21proc_pri22:15:093
20809214110,114sleep120914-21apt-config22:34:411
25222213849,49sleep30-21swapper/321:04:553
20898213816,108sleep321066-21latency_hist19:09:423
26801213314,103sleep226808-21sh21:09:552
25145213315,104sleep06209-1kworker/0:2H00:29:400
2889621319,63sleep329046-21latency_hist22:59:423
2651821286,109sleep326600-21switchtime22:50:093
13008212816,93sleep013023-21latency_hist22:09:410
30918212712,98sleep230955-21grep23:05:022
10606212718,89sleep210626-21latency_hist20:19:392
10606212718,89sleep210626-21latency_hist20:19:392
24207212626,24sleep324246-21ntpq00:25:033
6241212321,16sleep30-21swapper/320:04:543
4685212319,86sleep14748-21/usr/sbin/munin19:59:561
10087212012,87sleep310108-21kernelversion23:40:013
30595211927,72sleep030647-21tr19:39:390
4390211715,16sleep20-21swapper/221:40:072
30735211615,82sleep330828-21fschecks_time23:04:593
14724211418,78sleep114740-21kernelversion23:55:011
4791211332,18sleep34660-21apt-get23:25:193
20055211322,72sleep120136-21/usr/sbin/munin19:05:081
11366211310,43sleep30-21swapper/323:44:543
17890211111,76sleep30-21swapper/320:40:123
9942211017,77sleep310068-21latency_hist21:59:423
16042211036,17sleep316041-21apt-get23:59:523
26427210913,64sleep326461-21basename19:25:043
11125210813,12sleep30-21swapper/320:19:443
5174210727,16sleep30-21swapper/320:00:103
29829210725,66sleep329970-21grep23:00:103
21496210714,77sleep121545-21if_eth022:35:091
6243210612,20sleep00-21swapper/020:04:550
14172210611,21sleep30-21swapper/323:54:393
32656210526,16sleep30-21swapper/319:44:593
15220210523,59sleep30-21swapper/323:55:113
12187210512,77sleep3286-21dbus-daemon22:05:073
25656210417,72sleep125670-21uname00:30:021
23549210414,75sleep323581-21grep00:24:393
26933210328,19sleep326929-21apt-get00:34:523
7478210220,63sleep07617-21unixbench_multi23:31:100
20687210223,65sleep320689-21apt-get00:14:503
26002210026,46sleep126122-21latency_hist19:24:421
10819210017,17sleep10-21swapper/120:19:421
10819210017,17sleep10-21swapper/120:19:421
905629913,72sleep09117-21/usr/sbin/munin21:55:070
3046629918,46sleep30-21swapper/319:38:013
74992988,76sleep37557-21find21:50:053
472129813,67sleep131588-1kworker/1:2H23:25:171
36629813,14sleep20-21swapper/219:45:022
3010429810,46sleep30-21swapper/321:19:543
2078429623,53sleep020821-21fschecks_count00:14:590
1901229625,48sleep019051-21grep20:44:580
101882969,10sleep10-21swapper/123:40:031
2211229523,16sleep30-21swapper/320:54:583
241229420,13sleep30-21swapper/319:50:123
93922935,11sleep30-21swapper/323:38:123
282502937,11sleep20-21swapper/219:30:082
1971329319,18sleep20-21swapper/200:10:082
547529210,41sleep30-21swapper/321:44:533
1767829218,13sleep30-21swapper/300:04:573
168329217,14sleep30-21swapper/323:15:533
2839029113,32sleep228004-21apt-get21:14:432
2839029113,32sleep228004-21apt-get21:14:432
301529043,29sleep22977-21apt-get23:20:452
2910629015,60sleep129167-21/usr/sbin/munin21:15:351
2910629015,60sleep129167-21/usr/sbin/munin21:15:351
282029021,41sleep22843-21modprobe19:54:402
2751229012,61sleep127627-21/usr/sbin/munin21:10:101
2286929012,15sleep30-21swapper/322:39:563
2990628911,14sleep30-21swapper/321:19:413
244192896,70sleep324412-21apt-get22:44:563
51228834,43sleep028508-1kworker/0:0H21:29:400
224028837,26sleep10-21swapper/123:16:081
950528721,41sleep19523-21sendmail23:39:391
544728718,32sleep20-21swapper/220:00:132
283732860,7sleep10-21swapper/100:39:421
2516828621,51sleep225219-21sort21:04:542
2204828617,50sleep121125-21/usr/sbin/munin19:10:121
118328613,56sleep01222-21/usr/sbin/munin21:30:040
7772856,15sleep20-21swapper/221:29:502
673528520,50sleep16768-21awk21:49:391
3265828415,50sleep232693-21proc_pri23:10:072
297052846,13sleep30-21swapper/321:18:363
2493028423,31sleep20-21swapper/221:04:422
1429828412,55sleep014329-21df20:29:550
905028310,35sleep30-21swapper/320:14:393
56328314,57sleep0637-21switchtime19:45:090
444028325,22sleep34471-21latency_hist19:59:413
444028325,22sleep34471-21latency_hist19:59:413
322672827,34sleep30-21swapper/323:09:563
2669328212,57sleep126704-21sh19:25:111
3247928114,17sleep10-21swapper/123:10:021
2518228118,47sleep125328-21latency_hist00:29:411
112432816,65sleep30-21swapper/322:02:253
990928017,48sleep09985-21switchtime20:15:090
990928017,48sleep09985-21switchtime20:15:090
37132788,12sleep00-21swapper/019:55:100
37132788,12sleep00-21swapper/019:55:100
2636127813,51sleep030095-21snmpd00:31:020
2256027815,31sleep20-21swapper/200:20:022
1769827820,33sleep10-21swapper/100:04:591
2384027722,32sleep20-21swapper/200:24:542
132427631,27sleep10-21swapper/119:49:421
262227516,30sleep10-21swapper/121:35:021
2249827514,42sleep122013-21/usr/sbin/munin20:55:081
1803327515,35sleep118121-21grep22:25:001
2791227417,42sleep127953-21ntp_offset22:55:041
2038927416,28sleep20-21swapper/222:30:142
378927318,36sleep13929-21unixbench_multi19:55:111
378927318,36sleep13929-21unixbench_multi19:55:111
24772737,30sleep30-21swapper/321:35:003
211727313,45sleep12250-21sed21:34:411
1957127311,46sleep30-21swapper/322:29:583
1396927319,36sleep013984-21apt20:29:390
2036527212,41sleep020463-21users19:05:110
2013727218,41sleep220234-21sed19:05:092
181772726,31sleep30-21swapper/322:25:023
168522726,10sleep30-21swapper/320:36:573
3214027114,15sleep10-21swapper/121:25:081
3173727119,26sleep20-21swapper/221:24:572
2355127112,39sleep123689-21grep22:40:141
1325527118,31sleep013250-21apt-get22:09:530
1244427122,22sleep20-21swapper/222:05:102
1167827110,13sleep10-21swapper/122:04:491
310192705,13sleep00-21swapper/021:20:150
2964527025,21sleep10-21swapper/123:00:081
2525127017,27sleep10-21swapper/121:04:571
1651527018,36sleep216552-21ps00:00:112
1446127015,38sleep214455-21apt-get23:54:542
1081327012,24sleep10-21swapper/122:00:131
1022227018,31sleep20-21swapper/221:59:582
762826926,21sleep10-21swapper/121:50:081
758826919,24sleep20-21swapper/220:09:422
453226916,23sleep10-21swapper/121:40:101
278372696,10sleep30-21swapper/321:12:423
2095726918,22sleep20-21swapper/220:50:082
1821126912,42sleep018289-21sed22:25:030
670426823,28sleep10-21swapper/120:05:091
526026815,36sleep05213-21latency_hist21:44:420
1799226816,38sleep218015-21df22:24:572
1507426821,26sleep10-21swapper/122:15:021
910126715,39sleep19238-21latency_hist20:14:421
472826714,37sleep24789-21ls23:25:182
3195826716,26sleep00-21swapper/023:09:400
2788026717,33sleep027911-21/usr/sbin/munin22:55:030
2303526711,40sleep023182-21awk00:20:120
2538826618,27sleep20-21swapper/200:29:552
2074826622,23sleep00-21swapper/022:34:390
1021926617,28sleep210261-21awk23:40:042
864826525,20sleep10-21swapper/123:36:161
361226510,34sleep23616-21cron21:39:392
2517926524,23sleep20-21swapper/219:20:112
2136526513,37sleep221374-21diskstats22:35:022
1334026512,27sleep10-21swapper/122:09:581
1255526516,23sleep00-21swapper/020:26:080
239842645,11sleep20-21swapper/222:42:492
1643926418,23sleep00-21swapper/020:35:120
1451626417,21sleep20-21swapper/222:14:392
1042326412,16sleep00-21swapper/022:00:040
3048226313,20sleep20-21swapper/221:20:052
2842326312,14sleep10-21swapper/119:30:111
1638626312,14sleep10-21swapper/100:00:061
699726215,32sleep27016-21apt-get21:49:482
2633326214,27sleep20-21swapper/219:25:032
225142628,12sleep20-21swapper/220:55:092
1632326213,26sleep00-21swapper/000:00:050
1579626217,29sleep015807-21munin-run23:59:410
1318226226,16sleep10-21swapper/123:50:011
614726115,21sleep00-21swapper/023:29:240
1765626116,21sleep00-21swapper/020:40:050
1765226119,18sleep10-21swapper/122:24:391
1556626113,21sleep10-21swapper/120:34:411
991126021,23sleep20-21swapper/221:59:412
2917426017,19sleep20-21swapper/222:59:572
2320426012,32sleep223210-21diskstats19:15:102
1601626017,20sleep20-21swapper/220:35:022
954425920,25sleep29582-21munin-run23:39:402
3245825914,30sleep032478-21kernelversion23:10:010
2675125913,30sleep026839-21grep19:25:140
2212125914,23sleep00-21swapper/020:54:590
1766725916,26sleep217677-21df_inode00:04:562
1222225917,24sleep112269-21users23:45:121
1144525914,30sleep211341-21/usr/sbin/munin23:44:572
828625811,21sleep20-21swapper/220:10:102
315925820,20sleep20-21swapper/219:54:562
315925820,20sleep20-21swapper/219:54:562
3127525811,33sleep231286-21netstat19:40:022
2806725817,22sleep00-21swapper/021:14:410
2751025815,25sleep227503-21apt-get22:54:532
27402586,10sleep00-21swapper/019:53:370
2360225818,20sleep10-21swapper/100:24:411
2004525812,27sleep10-21swapper/100:10:121
176862588,13sleep00-21swapper/000:04:580
990725717,20sleep00-21swapper/023:39:580
595225715,28sleep05978-21munin-run20:04:390
547425714,26sleep25471-21apt-get21:44:522
2721225711,25sleep00-21swapper/000:35:050
2499525715,18sleep10-21swapper/122:45:151
2082525716,21sleep10-21swapper/100:15:001
1615525713,20sleep00-21swapper/022:19:420
3056625618,20sleep10-21swapper/121:20:081
2778825614,26sleep027782-21apt-get19:29:540
2360125617,21sleep10-21swapper/119:15:211
1346825617,21sleep20-21swapper/220:26:382
1136925614,18sleep00-21swapper/023:44:550
305122559,32sleep130095-21snmpd19:38:351
2463225511,27sleep024688-21/usr/sbin/munin22:45:060
2417825519,19sleep00-21swapper/022:44:410
2940525415,24sleep229411-21df_inode19:34:572
2468725414,24sleep024690-21apt-get19:19:520
2066925413,18sleep20-21swapper/200:14:492
1715225415,23sleep117296-21latency_hist20:39:411
1548025416,17sleep20-21swapper/222:15:122
1425625413,24sleep214250-21apt-get20:29:532
2957425315,21sleep00-21swapper/023:00:070
281612536,38sleep20-21swapper/200:39:112
2541125310,18sleep00-21swapper/021:05:000
2361025311,25sleep123604-21apt-get20:59:521
138652535,36sleep20-21swapper/223:50:422
1291225314,24sleep012907-21apt-get23:49:510
864925213,25sleep18655-21diskstats21:54:561
828425213,24sleep08298-21apt21:54:390
1818125212,21sleep218285-21users20:40:152
1639225212,25sleep016385-21apt-get22:19:540
325325115,21sleep03265-21vmstat21:35:130
3116725113,23sleep031180-21irqstats19:40:000
2671425118,17sleep10-21swapper/122:50:111
2596425112,23sleep025957-21apt-get22:49:540
2487625111,24sleep224882-21ntp_states22:45:122
2158625114,15sleep20-21swapper/219:10:052
2114725112,24sleep021144-21apt-get19:09:510
1747525111,16sleep10-21swapper/120:40:021
623425015,20sleep16371-21cat23:29:411
201725010,23sleep12023-21ntp_states19:50:061
1914325010,16sleep00-21swapper/000:09:500
2726424910,22sleep227273-21munin-run22:54:402
2045924912,21sleep120471-21apt-get20:49:511
1646424914,20sleep216471-21df_inode22:19:562
1339524910,21sleep10-21swapper/120:26:371
80752489,15sleep00-21swapper/020:10:040
3116924810,15sleep10-21swapper/123:05:091
593024711,21sleep05936-21ntp_states21:45:060
238582478,15sleep00-21swapper/021:00:000
2056124611,16sleep00-21swapper/020:49:580
541624512,16sleep20-21swapper/223:25:382
3221924514,15sleep10-21swapper/119:44:411
933023612,12sleep00-21swapper/023:37:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional