You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 03:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Jul 03, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
142362182217,1787sleep114237-21ntp_states23:24:541
20965293211,909sleep120127-21kworker/1:220:20:061
5650269720,623sleep35780-21grep19:30:183
534026177,594sleep35149-21latency_hist00:39:293
24930253414,508sleep26028-21kworker/2:220:32:072
13786252011,494sleep013787-21sshd19:57:040
12555251437,441sleep312582-21ntpq23:19:523
494525047,486sleep27750-1kworker/2:0H19:29:442
30540231240,211sleep330690-21unixbench_singl00:14:583
9407230616,273sleep19433-21netstat23:09:491
9407230616,273sleep19433-21netstat23:09:491
8319225963,33sleep30-21swapper/321:24:283
291182245111,62sleep30-21swapper/322:29:453
16581221054,140sleep316609-21idleruntime-cro23:34:253
16010221019,164sleep016169-21unixbench_multi23:29:580
19026220912,178sleep319040-21fschecks_count20:14:433
1069722078,187sleep1364-21snmpd19:47:071
25899220011,175sleep025978-21sendmail_mailtr20:34:550
25757219731,151sleep325836-21xargs23:59:573
25839217420,131sleep025970-21grep23:59:580
11074217116,138sleep311149-21mailstats23:14:553
3073921679,147sleep330808-21idleruntime22:34:463
3233216588,19sleep30-21swapper/321:05:063
27973216285,64sleep3278-21dbus-daemon00:09:253
29531216120,121sleep129641-21wc19:04:561
32315215984,57sleep332458-21grep20:54:573
755215611,46sleep30-21swapper/320:59:413
1015621568,10sleep30-21swapper/323:12:373
1015621568,10sleep30-21swapper/323:12:373
9022215516,118sleep30-21swapper/321:24:573
31136215325,25sleep30-21swapper/319:09:593
21119215315,27sleep30-21swapper/320:21:013
16954215015,118sleep216967-21fschecks_count23:34:442
6493214679,20sleep30-21swapper/323:00:023
6493214679,20sleep30-21swapper/323:00:023
2918214629,16sleep30-21swapper/319:24:283
2918214629,16sleep30-21swapper/319:24:283
17108214331,99sleep317134-21munin-run20:09:263
24584213816,100sleep024730-21sed20:29:590
3271821359,108sleep30-21swapper/300:24:273
2469213510,105sleep30-21swapper/300:29:523
1934321338,113sleep319356-21awk23:39:583
13700213343,24sleep30-21swapper/321:39:553
2281213215,28sleep10-21swapper/119:19:581
2281213215,28sleep10-21swapper/119:19:581
22308213220,94sleep30-21swapper/320:24:413
5617213067,48sleep35665-21cat21:14:473
21621213014,102sleep221664-21ntp_kernel_err22:04:492
17602212610,59sleep30-21swapper/320:09:473
9039212413,98sleep09058-21cpu23:09:390
9039212413,98sleep09058-21cpu23:09:390
32324212211,50sleep30-21swapper/319:14:353
32324212211,50sleep30-21swapper/319:14:353
20963212012,84sleep30-21swapper/322:04:263
1687021208,14sleep30-21swapper/323:34:413
1632212031,17sleep31668-21ntpq22:44:513
827211910,94sleep1834-21df_inode20:59:431
17272211921,75sleep117432-21unixbench_multi21:50:101
12688211985,18sleep30-21swapper/319:54:363
500211821,16sleep30-21swapper/300:24:433
18240211815,86sleep018328-21/usr/sbin/munin21:54:460
2683121178,21sleep30-21swapper/322:20:003
2683121178,21sleep30-21swapper/322:20:003
20212211712,82sleep2225-21systemd-journal20:19:272
3803211612,89sleep23336-21apt-get00:34:312
22047211518,13sleep30-21swapper/323:49:593
24833211420,75sleep124910-21systemd-run23:59:261
17120211431,14sleep30-21swapper/321:50:083
7861211321,13sleep30-21swapper/323:04:503
7861211321,13sleep30-21swapper/323:04:503
12335211310,82sleep00-21swapper/021:34:590
30354211212,88sleep330418-21cat20:49:483
4325211124,13sleep30-21swapper/321:09:513
2371121117,38sleep30-21swapper/323:55:003
3257211022,13sleep30-21swapper/322:49:523
28794211018,67sleep228920-21wc22:29:282
26551211017,68sleep125910-21/usr/sbin/munin22:19:581
26551211017,68sleep125910-21/usr/sbin/munin22:19:581
18401211020,14sleep30-21swapper/321:54:493
15085211013,81sleep215227-21latency_hist23:29:272
1107221107,35sleep30-21swapper/321:30:423
29073210810,54sleep30-21swapper/320:44:563
2545021086,33sleep30-21swapper/322:17:183
2385621089,16sleep30-21swapper/320:29:413
1609821088,28sleep316200-21cut21:49:283
25387210720,12sleep30-21swapper/320:34:383
14364210721,12sleep30-21swapper/319:59:453
4820210615,16sleep30-21swapper/322:54:533
16523210614,75sleep216648-21cat20:04:572
8749210511,82sleep00-21swapper/023:09:230
8749210511,82sleep00-21swapper/023:09:230
3241210415,73sleep13327-21cut19:24:461
1474021036,29sleep3360-21ntpd23:25:413
30885210215,71sleep10-21swapper/119:09:521
22923210135,41sleep022990-21fschecks_time22:09:480
2280121008,22sleep322791-21apt-get22:09:433
806429967,16sleep30-21swapper/319:39:383
550729825,50sleep15520-21fschecks_count21:14:451
269182979,75sleep0364-21snmpd22:20:540
269182979,75sleep0364-21snmpd22:20:540
3109029621,58sleep231136-21readlink00:19:252
129792957,18sleep312984-21systemd-run21:39:263
278529421,48sleep00-21swapper/022:49:370
864629316,63sleep28669-21cut21:24:482
2740629010,62sleep30-21swapper/320:39:513
2175529016,55sleep121270-21apt-get23:49:371
3019228911,64sleep030237-21ntp_kernel_pll_00:14:510
292672898,11sleep30-21swapper/300:11:113
2829028912,60sleep028439-21cut20:44:280
64822888,63sleep30-21swapper/319:34:383
64822888,63sleep30-21swapper/319:34:383
2354428810,58sleep00-21swapper/023:54:370
2101528717,55sleep121105-21sort22:04:281
1242528714,33sleep10-21swapper/119:54:261
41722867,67sleep30-21swapper/319:26:223
3106728614,16sleep20-21swapper/219:09:582
2793828620,14sleep30-21swapper/322:24:533
2301428622,46sleep223159-21/usr/sbin/munin20:24:562
1177728611,57sleep011889-21cat19:49:570
2897628528,31sleep00-21swapper/019:04:430
2485128516,48sleep024324-21/usr/sbin/munin22:14:570
210802856,11sleep30-21swapper/323:47:133
1572628516,54sleep215771-21ntp_offset23:29:522
138382856,12sleep10-21swapper/121:39:571
9012835,11sleep30-21swapper/322:43:303
224332836,10sleep30-21swapper/322:08:253
122122835,10sleep30-21swapper/319:51:463
3105628213,54sleep031093-21ps22:34:550
2926728215,47sleep129411-21grep20:44:581
1323328210,42sleep20-21swapper/221:39:422
105322815,43sleep30-21swapper/319:45:033
1164128024,28sleep20-21swapper/219:49:552
1144428021,36sleep211592-21latency_hist21:34:292
76152798,11sleep00-21swapper/019:37:420
76152798,11sleep00-21swapper/019:37:420
657727924,23sleep20-21swapper/219:34:452
657727924,23sleep20-21swapper/219:34:452
569727920,17sleep20-21swapper/222:59:282
3099227913,48sleep131034-21ntpq22:34:521
2299227714,50sleep1364-21snmpd22:09:491
331127627,27sleep00-21swapper/000:34:250
1705427624,35sleep217118-21ps21:50:072
776227518,41sleep17800-21wc23:04:481
776227518,41sleep17800-21wc23:04:481
304332757,12sleep20-21swapper/222:34:292
2853227520,29sleep10-21swapper/100:09:481
2152227514,30sleep00-21swapper/022:04:470
1653527514,44sleep016066-21apt-get21:49:370
1377527510,17sleep00-21swapper/023:24:410
102527515,45sleep01160-21latency_hist22:44:280
57872740,8sleep20-21swapper/221:14:492
518327417,24sleep00-21swapper/021:14:280
2894327419,32sleep229093-21unixbench_multi00:09:572
2726927417,34sleep227337-21wc00:04:562
943527319,29sleep20-21swapper/223:09:502
943527319,29sleep20-21swapper/223:09:502
293327318,31sleep10-21swapper/122:49:461
2629827315,45sleep226307-21/usr/sbin/munin22:19:512
2629827315,45sleep226307-21/usr/sbin/munin22:19:512
252127316,40sleep12547-21munin-run22:49:261
231727210,17sleep00-21swapper/021:04:430
2030027216,34sleep220389-21timerandwakeup21:59:572
1538527227,25sleep00-21swapper/021:44:560
516827117,36sleep15256-21systemd-run00:39:271
449427117,29sleep24638-21latency_hist19:29:282
411927126,27sleep20-21swapper/222:54:272
338827125,27sleep20-21swapper/200:34:272
2356727117,36sleep023576-21modprobe20:29:260
1880627119,30sleep20-21swapper/223:39:492
71527021,29sleep2477-21/usr/sbin/munin00:24:502
327542706,10sleep10-21swapper/120:57:371
3035527026,22sleep10-21swapper/122:34:281
2057026920,24sleep20-21swapper/223:44:582
186426915,29sleep01877-21irqstats19:19:490
186426915,29sleep01877-21irqstats19:19:490
170126916,38sleep21797-21grep19:19:472
170126916,38sleep21797-21grep19:19:472
440626810,14sleep20-21swapper/221:09:542
3266826828,22sleep10-21swapper/100:24:251
2432326810,12sleep10-21swapper/122:14:371
3045426713,27sleep10-21swapper/120:49:501
2406526712,27sleep20-21swapper/223:55:152
1348626713,34sleep013508-21grep23:24:250
434626618,29sleep10-21swapper/121:09:531
3235126617,36sleep232401-21basename22:39:462
2914126616,21sleep00-21swapper/020:44:570
2707526623,25sleep10-21swapper/120:39:451
1476026611,41sleep214825-21awk19:59:562
136026616,20sleep20-21swapper/220:59:572
96926521,28sleep10-21swapper/100:24:581
612926513,23sleep20-21swapper/222:59:522
612926513,23sleep20-21swapper/222:59:522
2548326515,34sleep125490-21diskstats20:34:441
2136226522,25sleep00-21swapper/023:49:280
816426420,31sleep28268-21fschecks_time19:39:452
692026412,33sleep16944-21ntp_states19:34:521
692026412,33sleep16944-21ntp_states19:34:521
2796826417,28sleep10-21swapper/122:24:551
2794626413,26sleep00-21swapper/022:24:540
2502426411,26sleep20-21swapper/222:14:592
1978326417,26sleep019937-21latency_hist23:44:270
1819726418,20sleep20-21swapper/221:54:442
1707926417,21sleep00-21swapper/020:09:070
493526317,26sleep10-21swapper/119:29:391
3140926314,25sleep20-21swapper/200:19:422
3016326322,21sleep00-21swapper/020:49:450
1865426317,30sleep218679-21munin-run20:14:262
3274526215,25sleep10-21swapper/119:14:491
3274526215,25sleep10-21swapper/119:14:491
2919626213,34sleep029252-21idleruntime22:29:460
1686726218,27sleep016863-21apt-get23:34:380
1019326222,23sleep00-21swapper/019:44:560
680726114,25sleep00-21swapper/021:19:290
656226116,23sleep00-21swapper/023:00:030
656226116,23sleep00-21swapper/023:00:030
2534826119,24sleep10-21swapper/123:59:461
2038326119,20sleep00-21swapper/020:19:290
152062615,38sleep00-21swapper/020:00:020
1058926116,27sleep010584-21apt-get23:14:400
728726015,30sleep17437-21cat23:04:281
454326017,22sleep00-21swapper/022:54:460
2929126023,23sleep329321-21awk19:04:503
1008026014,20sleep00-21swapper/021:29:380
937825912,24sleep00-21swapper/019:44:280
620425913,32sleep06232-21munin-run19:34:260
1329025918,21sleep20-21swapper/219:54:562
1254325915,26sleep212554-21/usr/sbin/munin23:19:512
1175225917,23sleep10-21swapper/121:34:481
603925820,19sleep10-21swapper/122:59:511
383125814,27sleep03838-21dpkg00:34:370
3216925813,29sleep10-21swapper/122:39:411
3211725815,19sleep00-21swapper/020:54:540
2805525818,17sleep10-21swapper/100:09:271
273632587,12sleep20-21swapper/220:39:502
229672586,17sleep20-21swapper/223:53:252
1547425818,18sleep10-21swapper/121:44:581
1205225812,31sleep212090-21grep21:34:532
998525716,21sleep20-21swapper/219:44:502
981425715,26sleep09831-21apt21:29:260
334425715,21sleep10-21swapper/100:34:261
315425711,25sleep00-21swapper/019:24:410
3117725720,19sleep10-21swapper/100:19:291
278762575,15sleep00-21swapper/000:08:090
1870625716,19sleep00-21swapper/020:14:280
182742575,12sleep10-21swapper/123:39:281
75725614,25sleep0826-21df20:59:420
367625616,18sleep10-21swapper/121:09:281
315902566,41sleep00-21swapper/019:12:050
273225614,24sleep22768-21ntp_states21:04:532
2540425614,25sleep225398-21apt-get20:34:412
2260125616,15sleep10-21swapper/122:09:281
199982565,41sleep10-21swapper/120:16:431
1944825614,19sleep10-21swapper/121:59:271
1322325616,25sleep113289-21/usr/sbin/munin19:54:551
830925514,25sleep28318-21munin-run21:24:272
79025515,24sleep0874-21memory00:24:520
749225512,20sleep20-21swapper/223:04:412
749225512,20sleep20-21swapper/223:04:412
726325518,20sleep10-21swapper/121:19:501
2964225519,16sleep20-21swapper/219:04:582
2425225514,25sleep124286-21/usr/sbin/munin20:29:511
172325517,16sleep20-21swapper/222:44:542
1278525512,20sleep10-21swapper/123:19:581
913625417,17sleep10-21swapper/121:24:591
3196425416,17sleep20-21swapper/219:14:302
3196425416,17sleep20-21swapper/219:14:302
3029825412,24sleep230353-21irqstats20:49:472
262882546,10sleep20-21swapper/200:02:422
234682545,13sleep10-21swapper/120:28:101
2006425414,18sleep00-21swapper/021:59:500
1739425413,24sleep117388-21apt-get20:09:401
1579125417,17sleep10-21swapper/123:29:551
116982546,36sleep10-21swapper/123:17:231
1028925412,25sleep110326-21idleruntime21:29:471
806725311,25sleep18061-21apt-get19:39:401
2848425310,15sleep00-21swapper/000:09:470
1454225317,15sleep00-21swapper/021:44:280
3206925212,22sleep00-21swapper/019:14:320
3206925212,22sleep00-21swapper/019:14:320
2288225213,24sleep222886-21diskstats22:09:462
2139725222,17sleep20-21swapper/220:21:082
1730325214,22sleep117322-21ntp_states23:34:521
3171625113,22sleep231720-21diskstats20:54:442
3140725118,18sleep00-21swapper/000:19:400
167602514,11sleep10-21swapper/120:05:001
126125117,18sleep00-21swapper/022:44:400
1095425113,25sleep210960-21needreboot23:14:502
80142506,35sleep00-21swapper/021:21:100
443425012,18sleep10-21swapper/122:54:431
2948725013,23sleep229488-21ntp_states22:29:522
2852525013,23sleep228521-21apt-get20:44:382
1459425012,23sleep114661-21memory19:59:511
2207224810,22sleep022166-21memory23:50:000
159972479,26sleep2364-21snmpd21:48:542
2004424510,19sleep120071-21df23:44:451
213112398,14sleep00-21swapper/020:21:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional