You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-16 - 11:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Tue Dec 16, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
383772212361,58sleep30-21swapper/319:07:463
54043121180,2sleep30-21swapper/321:35:153
38378728559,20sleep00-21swapper/019:07:560
38385328353,9sleep10-21swapper/119:08:491
38383928252,9sleep20-21swapper/219:08:382
384067992724,1cyclictest0-21swapper/300:05:133
38406799252,19cyclictest638501-21taskset23:08:293
384067992118,1cyclictest0-21swapper/320:10:003
38406799198,8cyclictest486684-21chrt20:45:103
38406799192,11cyclictest454682-21taskset20:15:113
384067991916,1cyclictest0-21swapper/322:40:013
384067991916,1cyclictest0-21swapper/321:10:203
384067991916,1cyclictest0-21swapper/320:40:123
384067991916,1cyclictest0-21swapper/320:40:023
384067991915,2cyclictest0-21swapper/323:30:143
384067991915,2cyclictest0-21swapper/319:55:133
38406799191,15cyclictest0-21swapper/322:20:113
38406799191,14cyclictest5288742chrt21:25:133
38406799191,13cyclictest581047-21turbostat22:15:003
38406799191,11cyclictest720516-21taskset00:25:173
38406799191,11cyclictest405000-21taskset19:25:173
384067991815,1cyclictest0-21swapper/322:40:163
384067991814,2cyclictest0-21swapper/320:25:003
38406799181,14cyclictest0-21swapper/322:45:123
38406799181,13cyclictest565617-21chrt22:00:123
38406799181,13cyclictest549362-21turbostat21:45:003
38406799181,13cyclictest467353-21taskset20:25:183
38406799181,13cyclictest0-21swapper/323:20:123
38406799181,13cyclictest0-21swapper/322:25:123
38406799181,13cyclictest0-21swapper/319:45:103
38406799181,11cyclictest553198-21taskset21:45:173
38406799180,14cyclictest0-21swapper/323:35:173
5975242171,5sleep1598913-21cut22:30:171
4124262171,4sleep2412445-21apt-get19:35:112
384067991714,1cyclictest0-21swapper/322:50:143
384067991714,1cyclictest0-21swapper/321:15:163
38406799171,13cyclictest0-21swapper/323:15:143
38406799171,13cyclictest0-21swapper/319:35:153
38406799171,12cyclictest0-21swapper/320:55:113
38406799171,12cyclictest0-21swapper/319:30:113
38406799171,10cyclictest422489-21taskset19:43:013
38406799170,10cyclictest639152-21taskset23:10:063
4549392161,10sleep0454989-21fschecks_count20:15:130
38406799168,4cyclictest6865792sleep323:55:113
38406799166,6cyclictest443623-21taskset20:04:453
38406799161,9cyclictest679272-21taskset23:45:163
38406799161,9cyclictest581286-21taskset22:15:083
38406799161,9cyclictest555117-21taskset21:50:023
384067991612,2cyclictest0-21swapper/322:55:163
38406799161,13cyclictest0-21swapper/322:30:153
38406799161,12cyclictest0-21swapper/321:20:553
38406799161,11cyclictest560839-21chrt21:55:153
38406799161,11cyclictest502448-21chrt21:00:123
38406799161,11cyclictest0-21swapper/320:50:183
38406799161,10cyclictest706965-21taskset00:10:293
38406799161,10cyclictest389163-21chrt19:14:353
38406799160,13cyclictest0-21swapper/319:50:153
38406799160,13cyclictest0-21swapper/319:15:163
38406799160,13cyclictest0-21swapper/300:30:163
7271632151,5sleep1727251-21ntpq00:30:171
5135082151,10sleep1514281-21cut21:10:181
4753372151,9sleep0475836-21gpg-connect-age20:35:010
38406799151,9cyclictest733199-21taskset00:36:353
38406799151,9cyclictest685976-21taskset23:50:423
38406799151,9cyclictest511514-21taskset21:05:173
384067991512,1cyclictest0-21swapper/319:20:153
384067991512,1cyclictest0-21swapper/300:15:153
384067991512,1cyclictest0-21swapper/300:00:283
384067991511,2cyclictest0-21swapper/323:25:133
38406799151,11cyclictest717823-21turbostat00:25:003
38406799150,13cyclictest0-21swapper/320:10:013
38406799150,12cyclictest0-21swapper/323:40:213
38406799150,12cyclictest0-21swapper/323:00:103
38406799150,12cyclictest0-21swapper/320:30:113
6132352141,8sleep1613533-21grep22:45:161
5603612141,9sleep0560473-21cut21:55:140
38406799141,10cyclictest575833-21turbostat22:10:003
38406799140,12cyclictest0-21swapper/321:30:143
6767192131,8sleep10-21swapper/123:45:161
5760842131,8sleep2576289-21expr22:10:142
5077022131,8sleep1507912-21expr21:05:121
3917272130,9sleep2391958-21cat19:15:152
6497112122,5sleep1649730-21apt-get23:20:101
6323902121,7sleep2632445-21modprobe23:00:212
6237402121,7sleep0624447-21kthreadcore22:55:170
4330992121,3sleep2433102-21sshd19:54:552
4072202121,7sleep0407407-21cstates19:30:120
4023492121,7sleep0402518-21if_err_enp1s019:25:150
3939792121,5sleep1395263-21memory19:15:181
3847912121,7sleep0384848-21kthreadcore19:10:170
7238752111,6sleep2724795-21kthreadcore00:30:132
7180742111,4sleep1718093-21apt-get00:25:111
7070942110,1sleep10-21swapper/100:12:091
6391272111,6sleep2638631-21apt-get23:10:022
6244482111,6sleep1626175-21kthreadcore22:55:181
5855582110,7sleep1581292-21/usr/sbin/munin22:15:271
5760622112,4sleep1576081-21apt-get22:10:131
4716582112,5sleep1473402-21cut20:30:171
4656382110,7sleep10-21swapper/120:25:161
4546602111,5sleep20-21swapper/220:15:122
4479522111,5sleep2448009-21ntpq20:05:212
4182392111,6sleep1419181-21kthreadcore19:40:151
4016132111,7sleep0401614-21sshd19:23:550
7323122100,1sleep20-21swapper/200:35:182
7131412101,4sleep2713351-21/usr/sbin/munin00:20:152
7115852101,4sleep0711634-21ntpq00:15:200
7026282101,5sleep2702673-21fschecks_count00:10:132
6958722101,4sleep0695921-21ntpq00:00:210
6816502101,5sleep2681753-21perl23:50:132
6696672101,4sleep0669714-21ntpq23:35:210
6552542101,5sleep0655395-21expr23:25:150
6219522101,4sleep0622001-21ntpq22:50:210
6019252102,4sleep0602138-21latency_hist22:35:020
5865272101,4sleep1586546-21apt-get22:20:111
5813142101,4sleep0581570-21df22:15:130
5705552100,2sleep0570558-21sshd22:04:550
5126782101,4sleep0512699-21munin-run21:10:000
4959372101,4sleep0495987-21ntpq20:50:210
4866862101,3sleep00-21swapper/020:45:110
4864282101,4sleep2486158-21apt-get20:45:012
4761622101,4sleep2476375-21hddtemp_smartct20:35:162
4230692101,4sleep1423192-21diskstats19:45:121
3969862101,4sleep1397089-21perl19:20:121
3845222101,5sleep2384589-21tune2fs19:10:142
691907291,3sleep1692163-21df00:00:141
691576291,4sleep2691604-21munin-run00:00:002
689143291,4sleep1690416-21memory23:55:181
681369291,3sleep00-21swapper/023:50:110
680415291,4sleep2680447-21sed23:45:252
639180291,4sleep0639361-21ls23:10:110
592467291,4sleep1594134-21cut22:25:161
586823290,1sleep00-21swapper/022:20:140
585112291,4sleep2585231-21modprobe22:15:202
554821290,1sleep20-21swapper/221:49:262
548690291,4sleep1548966-21sed21:40:271
544493291,3sleep0544616-21diskstats21:40:140
543033291,4sleep2543080-21ntpq21:35:192
507420291,4sleep1507462-21true21:05:011
501095291,4sleep2501143-21ntpq20:55:192
486151291,4sleep1486427-21latency_hist20:45:001
459920291,3sleep2459939-21apt-get20:20:122
449463291,3sleep1449669-21cstates20:10:131
437369291,4sleep2437414-21collect219:55:192
428118291,3sleep1428137-21apt-get19:50:131
423193291,3sleep2423402-21hddtemp_smartct19:45:132
729360281,3sleep00-21swapper/000:35:160
713571281,3sleep0715246-21tr00:20:180
702350281,3sleep0702369-21apt-get00:10:110
682621281,3sleep1684361-21cut23:50:161
672521281,3sleep0674279-21kthreadcore23:40:170
660528280,1sleep10-21swapper/123:30:121
655536281,3sleep2656579-21kthreadcore23:25:162
648421281,3sleep1648468-21ntpq23:15:221
634218281,3sleep2635335-21kthreadcore23:05:172
633388281,3sleep1633411-21sh23:05:001
623466281,3sleep2623558-21fschecks_time22:55:152
611425281,3sleep1611471-21ntpq22:40:201
592253281,3sleep0592466-21kthreadcore22:25:150
570319280,1sleep10-21swapper/122:01:551
537913281,3sleep21-21systemd21:30:232
534423281,3sleep0536051-21kthreadcore21:30:160
528599281,2sleep10-21swapper/121:25:121
522011280,4sleep0522099-21ntpq21:15:210
514282281,3sleep2516018-21kthreadcore21:10:192
486708281,3sleep1486931-21cstates20:45:121
485496281,3sleep0485736-21grep20:40:260
470068280,1sleep00-21swapper/020:29:560
469715281,3sleep2800-21snmpd20:25:302
459626280,1sleep10-21swapper/120:19:561
449441281,2sleep00-21swapper/020:10:120
449420280,2sleep20-21swapper/220:10:022
443907280,1sleep00-21swapper/020:05:120
428414280,1sleep00-21swapper/019:50:160
419184281,3sleep2420939-21kthreadcore19:40:162
385296281,3sleep1386995-21kthreadcore19:10:181
728095270,1sleep00-21swapper/000:32:570
717467270,1sleep10-21swapper/100:20:381
712281270,1sleep10-21swapper/100:16:341
711686271,3sleep2711825-21basename00:15:232
701937270,1sleep00-21swapper/000:08:030
697139270,2sleep10-21swapper/100:05:121
687448271,2sleep00-21swapper/023:55:160
665208270,1sleep20-21swapper/223:32:022
660447270,1sleep00-21swapper/023:30:030
650383271,2sleep20-21swapper/223:20:152
648299270,3sleep0648420-21ntpq23:15:210
645729271,2sleep20-21swapper/223:15:192
606488270,4sleep10-21swapper/122:35:261
591190270,1sleep20-21swapper/222:21:242
571518271,3sleep2573185-21cut22:05:182
570827271,2sleep10-21swapper/122:05:131
570227270,1sleep20-21swapper/222:00:442
564428271,2sleep2564547-21gzip21:55:232
560076270,1sleep20-21swapper/221:54:592
555968271,2sleep10-21swapper/121:50:171
555130271,2sleep0555149-21apt-get21:50:120
544326270,2sleep2544345-21apt-get21:40:122
543622270,4sleep10-21swapper/121:35:261
538550270,1sleep10-21swapper/121:32:011
512547270,1sleep20-21swapper/221:08:212
502426270,2sleep20-21swapper/221:00:122
481740271,2sleep2482585-21tr20:40:162
470336270,1sleep20-21swapper/220:30:102
464702270,1sleep10-21swapper/120:22:551
444469271,3sleep1445270-21cut20:05:171
442923270,1sleep20-21swapper/220:00:242
438086270,1sleep00-21swapper/019:56:100
423476271,2sleep0424899-21cat19:45:150
422531270,1sleep00-21swapper/019:43:350
417363270,1sleep10-21swapper/119:39:431
403255271,2sleep20-21swapper/219:25:172
401964270,2sleep10-21swapper/119:25:121
389238271,3sleep00-21swapper/019:15:010
718835261,2sleep20-21swapper/200:25:162
697712260,2sleep2421rcuc/200:05:172
696617260,1sleep20-21swapper/200:01:442
675728260,1sleep20-21swapper/223:42:522
674280261,2sleep10-21swapper/123:40:181
667746261,2sleep10-21swapper/123:35:181
665995261,2sleep20-21swapper/223:35:142
659918260,2sleep1660315-21latency_hist23:30:001
643476261,2sleep10-21swapper/123:10:251
638551260,4sleep00-21swapper/023:09:070
638344260,1sleep10-21swapper/123:06:321
633259260,1sleep00-21swapper/023:03:220
622596260,4sleep20-21swapper/222:50:552
617626260,1sleep00-21swapper/022:49:110
612402260,4sleep00-21swapper/022:44:190
607485261,2sleep20-21swapper/222:40:122
603074261,2sleep00-21swapper/022:35:170
596609260,1sleep20-21swapper/222:29:022
553773261,1sleep00-21swapper/021:45:200
550449261,2sleep10-21swapper/121:45:171
543901260,1sleep00-21swapper/021:38:130
529498260,2sleep20-21swapper/221:25:172
528098260,1sleep20-21swapper/221:21:572
519591261,2sleep20-21swapper/221:15:192
518174260,1sleep10-21swapper/121:15:101
500845260,2sleep1501089-21strings20:55:181
499029261,2sleep00-21swapper/020:55:170
496217261,2sleep10-21swapper/120:50:251
496039261,2sleep20-21swapper/220:50:222
479965260,2sleep10-21swapper/120:35:221
470339260,2sleep00-21swapper/020:30:120
464610260,1sleep00-21swapper/020:21:460
438674260,2sleep10-21swapper/120:00:111
438034260,1sleep10-21swapper/119:55:301
412154260,4sleep10-21swapper/119:34:541
401595260,1sleep20-21swapper/219:23:422
722725250,4sleep00-21swapper/000:25:540
650007250,1sleep00-21swapper/023:20:130
622686251,1sleep10-21swapper/122:52:021
617370251,2sleep20-21swapper/222:45:572
602182250,3sleep20-21swapper/222:35:112
597110250,1sleep20-21swapper/222:30:142
576358250,1sleep00-21swapper/022:10:160
574641251,1sleep00-21swapper/022:05:190
560474250,1sleep10-21swapper/121:55:151
524381250,2sleep00-21swapper/021:20:180
523668250,1sleep10-21swapper/121:20:161
505120250,1sleep00-21swapper/021:00:190
442423250,1sleep00-21swapper/020:00:180
417332250,3sleep00-21swapper/019:39:180
407493250,1sleep20-21swapper/219:30:152
728380240,1sleep10-21swapper/100:35:011
676157240,1sleep00-21swapper/023:45:110
532391240,1sleep00-21swapper/021:25:190
513333240,1sleep00-21swapper/021:10:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional