You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 10:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Fri Jan 02, 2026 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
410778211751,61sleep30-21swapper/319:06:053
41102628454,9sleep20-21swapper/219:09:132
40631828263,9sleep10-21swapper/119:05:151
41080827454,14sleep00-21swapper/019:06:260
411196992925,2cyclictest0-21swapper/319:20:003
411196992320,1cyclictest0-21swapper/322:05:003
411196992320,1cyclictest0-21swapper/321:55:133
411196992320,1cyclictest0-21swapper/319:15:003
411196992320,1cyclictest0-21swapper/300:25:233
411196992118,1cyclictest0-21swapper/323:57:243
411196992118,1cyclictest0-21swapper/320:35:163
411196992118,1cyclictest0-21swapper/320:25:003
411196992115,3cyclictest486813-21sleep20:20:003
411196992114,4cyclictest728374-21sleep00:10:003
41119699209,6cyclictest528747-21turbostat20:59:593
41119699208,7cyclictest702082-21turbostat23:45:003
41119699205,13cyclictest0-21swapper/321:10:283
411196992017,1cyclictest0-21swapper/322:29:593
411196992016,2cyclictest0-21swapper/300:00:303
41119699201,11cyclictest759103-21taskset00:35:223
411196991916,1cyclictest0-21swapper/323:30:143
41119699191,14cyclictest0-21swapper/321:45:133
6172222181,8sleep1617280-21sed22:20:231
41119699186,6cyclictest455730-21taskset19:50:123
411196991815,1cyclictest0-21swapper/323:35:143
411196991815,1cyclictest0-21swapper/322:50:183
411196991815,1cyclictest0-21swapper/322:30:163
411196991815,1cyclictest0-21swapper/321:35:293
411196991815,1cyclictest0-21swapper/321:25:183
411196991815,1cyclictest0-21swapper/319:30:013
411196991814,2cyclictest0-21swapper/323:50:233
41119699181,13cyclictest0-21swapper/319:40:103
41119699181,11cyclictest554980-21taskset21:25:003
41119699180,16cyclictest0-21swapper/322:59:153
7182782171,11sleep0718347-21grep00:00:130
5699502171,5sleep2569996-21ntpq21:35:222
411196991713,2cyclictest0-21swapper/321:05:173
411196991713,2cyclictest0-21swapper/300:20:003
41119699171,12cyclictest634104-21chrt22:40:123
41119699171,12cyclictest633821-21turbostat22:39:593
41119699171,11cyclictest471399-21taskset20:05:123
41119699170,14cyclictest440291-21chrt19:35:143
41119699170,11cyclictest675321-21chrt23:16:553
5138292161,10sleep2514593-21tr20:45:152
41119699167,5cyclictest497610-21taskset20:30:093
411196991614,1cyclictest0-21swapper/300:30:183
411196991613,1cyclictest0-21swapper/323:45:233
411196991613,1cyclictest0-21swapper/320:50:113
411196991612,2cyclictest0-21swapper/323:25:153
41119699161,13cyclictest0-21swapper/319:45:193
41119699161,11cyclictest639316-21chrt22:45:133
41119699160,13cyclictest0-21swapper/323:20:213
41119699160,13cyclictest0-21swapper/323:20:213
41119699160,13cyclictest0-21swapper/322:20:263
41119699160,13cyclictest0-21swapper/321:30:143
41119699160,13cyclictest0-21swapper/321:15:133
41119699160,13cyclictest0-21swapper/321:15:123
41119699160,13cyclictest0-21swapper/319:55:223
41119699160,13cyclictest0-21swapper/319:30:193
41119699160,12cyclictest423985-21chrt19:20:113
41119699160,12cyclictest0-21swapper/320:45:123
41119699160,10cyclictest659753-21taskset23:03:003
4559872151,9sleep0456009-21/usr/sbin/munin19:50:150
41119699159,3cyclictest576132-21chrt21:44:123
41119699151,9cyclictest744222-21taskset00:22:563
41119699151,9cyclictest744222-21taskset00:22:563
41119699151,9cyclictest733353-21taskset00:11:543
41119699151,9cyclictest530286-21taskset21:00:193
41119699151,9cyclictest476877-21taskset20:10:133
411196991512,2cyclictest0-21swapper/321:50:113
411196991512,1cyclictest0-21swapper/320:25:163
411196991511,2cyclictest0-21swapper/320:00:163
41119699151,11cyclictest612893-21sleep22:19:593
41119699151,10cyclictest668084-21chrt23:10:183
41119699151,10cyclictest508078-21chrt20:40:113
41119699150,9cyclictest597449-21taskset22:04:593
41119699150,12cyclictest0-21swapper/322:10:203
5814002147,5sleep1411188-21cyclictest21:49:351
5758412141,5sleep0571248-21/usr/sbin/munin21:40:290
41119699140,11cyclictest0-21swapper/323:05:243
7399692131,7sleep0741069-21tr00:20:180
7399692131,7sleep0741069-21tr00:20:180
7230382131,6sleep1723056-21munin-run00:05:001
7023362131,8sleep2534031-21kworker/2:0+events23:45:132
6381452131,7sleep1638227-21wc22:40:231
4911722131,8sleep1491242-21grep20:20:221
4342182131,5sleep2434633-21readlink19:30:012
4242422131,8sleep0424270-21fschecks_count19:20:120
7537852120,2sleep10-21swapper/100:30:211
6976642121,3sleep2698578-21tr23:40:162
6915562121,7sleep0691680-21latency_hist23:35:010
6027192121,6sleep1602875-21cstates22:10:121
5874742121,7sleep0587742-21cat21:55:160
5829212121,6sleep2583739-21kthreadcore21:50:172
5505392121,7sleep2551087-21grep21:20:152
5129562120,1sleep20-21swapper/220:44:242
4923322121,6sleep0492351-21apt-get20:25:130
4815382121,8sleep0481541-21cron20:15:000
7485492111,5sleep1748597-21ntpq00:25:191
7288892111,5sleep2729009-21tune2fs00:10:142
6863432111,5sleep2686037-21apt-get23:30:022
6796982111,4sleep2679811-21ntpq23:20:202
6796982111,4sleep2679811-21ntpq23:20:192
6706392111,5sleep1670897-21df_inode23:15:141
5962262111,5sleep0596273-21ntpq22:00:210
5254602111,6sleep2527180-21kthreadcore20:55:172
4780152111,6sleep2479711-21kthreadcore20:10:172
4596132111,5sleep2459651-21ntpq19:50:202
4471762111,6sleep0448949-21kthreadcore19:40:170
4119582111,6sleep2413246-21kthreadcore19:10:172
7591512101,4sleep0759207-21ntp_states00:35:250
7452482101,5sleep2746778-21cut00:25:162
7395762101,4sleep1739687-21diskstats00:20:151
7395762101,4sleep1739687-21diskstats00:20:141
7325822101,4sleep0732620-21ntpq00:10:190
7233202101,4sleep2723576-21df00:05:132
7220692101,4sleep2722121-21ntpq00:00:242
6811612101,4sleep1681374-21df_inode23:25:161
6759392101,4sleep0676142-21diskstats23:20:140
6759392101,4sleep0676142-21diskstats23:20:140
6640852101,4sleep0664109-21ntpq23:05:230
6398032102,4sleep2639877-21uname22:45:172
6276292101,4sleep0627685-21ntp_states22:30:230
6233522101,4sleep1623403-21munin-run22:30:011
6118442100,2sleep10-21swapper/122:15:221
5976262101,4sleep2597754-21diskstats22:05:152
5766372101,4sleep0576748-21diskstats21:45:140
5557682101,4sleep1555976-21hddtemp_smartct21:25:151
5543812101,4sleep1554566-21users21:20:261
5452022101,4sleep2545271-21irqstats21:15:142
5452022101,4sleep2545271-21irqstats21:15:142
5295802102,4sleep2530285-21tr21:00:192
5028722101,4sleep0503125-21df20:35:160
4763282101,4sleep1476349-21munin-run20:10:001
4616972101,5sleep1462963-21kthreadcore19:55:171
4347772101,4sleep1435030-21df19:30:111
4295582101,4sleep0429768-21hddtemp_smartct19:25:140
4292642101,4sleep1429283-21apt-get19:25:111
759063291,3sleep2759101-21ntpq00:35:222
753946291,4sleep0754061-21ps00:30:240
738483290,1sleep20-21swapper/200:15:352
717103291,3sleep1717320-21users23:55:301
712782291,4sleep2712903-21grep23:55:152
696716290,1sleep20-21swapper/223:38:462
686588290,5sleep10-21swapper/123:30:131
665769290,6sleep0665935-21if_err_enp1s023:10:160
655507291,4sleep2655825-21cut23:00:162
650301291,4sleep1651704-21cat22:55:181
644540290,1sleep20-21swapper/222:50:142
638089291,4sleep0638138-21ntpq22:40:210
633113291,4sleep0628828-21/usr/sbin/munin22:35:290
632644291,3sleep1632785-21ntpq22:35:221
611763291,3sleep0611843-21ntpq22:15:210
561030291,4sleep2561105-21grep21:30:132
560771291,4sleep1561029-21df_inode21:30:121
508630291,4sleep0509846-21kthreadcore20:40:170
487060291,4sleep2487079-21apt-get20:20:112
482335291,4sleep1482575-21tr20:15:161
475385291,4sleep0475440-21modprobe20:05:210
467467291,4sleep0469130-21cut20:00:170
455448291,3sleep2455496-21/usr/sbin/munin19:50:012
454471291,4sleep1454521-21ntpq19:45:231
445557291,4sleep1445766-21hddtemp_smartct19:40:141
434755291,3sleep00-21swapper/019:30:110
418656291,4sleep0418901-21expr19:15:120
411385291,2sleep10-21swapper/119:10:111
738768280,1sleep00-21swapper/000:19:100
737945281,3sleep1738029-21ps00:15:221
728342281,3sleep0728343-21cron00:10:000
712275280,1sleep10-21swapper/123:52:131
708094281,3sleep0709054-21kthreadcore23:50:150
704978280,1sleep00-21swapper/023:45:190
700304281,3sleep0700940-21missed_timers23:40:180
697095281,2sleep10-21swapper/123:40:111
696704280,1sleep00-21swapper/023:38:370
664110280,3sleep1664159-21ntpq23:05:241
644257281,3sleep1644279-21munin-run22:50:011
628849281,3sleep2629026-21cstates22:35:152
622381281,3sleep0622430-21ntpq22:25:230
608190281,3sleep20-21swapper/222:15:142
607359280,1sleep00-21swapper/022:11:240
606958280,1sleep20-21swapper/222:10:262
598033281,3sleep1598789-21cut22:05:201
570769280,1sleep00-21swapper/021:37:320
564722280,5sleep00-21swapper/021:30:190
555472280,3sleep2555491-21apt-get21:25:122
538184281,3sleep2538233-21modprobe21:05:202
504622281,3sleep1504872-21kthreadcore20:35:201
502292280,1sleep00-21swapper/020:31:190
496631280,1sleep10-21swapper/120:25:261
488905281,3sleep0490637-21cut20:20:170
440030280,1sleep20-21swapper/219:35:142
440006281,3sleep0440014-21/usr/sbin/munin19:35:120
751018271,2sleep20-21swapper/200:30:182
723299270,2sleep10-21swapper/100:05:121
717540270,1sleep00-21swapper/023:58:090
711556271,3sleep20-21swapper/223:50:222
702713270,1sleep10-21swapper/123:45:161
680644270,4sleep10-21swapper/123:22:431
680644270,4sleep10-21swapper/123:22:421
665936271,3sleep10-21swapper/123:10:161
654368270,1sleep20-21swapper/222:56:072
649748271,3sleep00-21swapper/022:55:130
649373270,1sleep10-21swapper/122:53:481
624173271,2sleep20-21swapper/222:30:182
597461271,3sleep0800-21snmpd22:05:100
575986270,4sleep20-21swapper/221:42:242
566029270,2sleep10-21swapper/121:35:121
559518270,2sleep00-21swapper/021:25:210
554732270,1sleep00-21swapper/021:22:260
545528271,2sleep00-21swapper/021:15:160
545528271,2sleep00-21swapper/021:15:150
539492271,2sleep1539511-21apt-get21:10:121
539224271,2sleep1539279-21munin-run21:10:001
528396270,1sleep10-21swapper/120:55:451
523736270,2sleep00-21swapper/020:55:120
518500271,2sleep0518629-21tr20:50:110
508050270,1sleep10-21swapper/120:40:061
502351270,4sleep10-21swapper/120:32:051
486532270,1sleep20-21swapper/220:16:582
471378271,2sleep20-21swapper/220:05:122
470928270,1sleep20-21swapper/220:02:362
470812270,1sleep10-21swapper/120:01:101
459757271,3sleep10-21swapper/119:50:241
423963271,2sleep20-21swapper/219:20:102
420674271,3sleep10-21swapper/119:15:171
416719271,3sleep20-21swapper/219:15:022
755087260,1sleep10-21swapper/100:35:101
749375260,4sleep00-21swapper/000:27:150
729768261,2sleep10-21swapper/100:10:161
696549260,1sleep10-21swapper/123:36:421
675236260,1sleep20-21swapper/223:15:512
670623261,2sleep00-21swapper/023:15:130
666436261,2sleep2668082-21cut23:10:182
664161260,2sleep20-21swapper/223:05:252
658941261,1sleep00-21swapper/023:00:210
643309261,2sleep00-21swapper/022:45:230
634082260,2sleep20-21swapper/222:40:122
623044260,1sleep20-21swapper/222:26:092
617953260,1sleep00-21swapper/022:22:540
613692260,2sleep20-21swapper/222:20:162
596898260,1sleep10-21swapper/122:01:111
592571261,1sleep20-21swapper/222:00:152
587743261,2sleep10-21swapper/121:55:161
586374260,1sleep10-21swapper/121:51:211
572936261,2sleep10-21swapper/121:40:181
540059260,2sleep20-21swapper/221:10:172
539008260,4sleep00-21swapper/021:07:190
533868260,1sleep10-21swapper/121:03:471
533747260,1sleep00-21swapper/021:02:180
513266260,3sleep10-21swapper/120:45:101
507772260,4sleep20-21swapper/220:39:582
496408261,2sleep2496466-21modprobe20:25:242
480522260,2sleep10-21swapper/120:10:201
465601260,1sleep20-21swapper/219:56:222
460918260,2sleep00-21swapper/019:55:120
451044261,2sleep00-21swapper/019:45:180
445262260,1sleep20-21swapper/219:40:102
440557261,2sleep10-21swapper/119:35:171
438783260,2sleep20-21swapper/219:30:202
416282260,4sleep00-21swapper/019:13:390
681695250,1sleep20-21swapper/223:25:202
648541251,1sleep00-21swapper/022:50:230
590946250,1sleep20-21swapper/221:55:212
585800250,2sleep00-21swapper/021:50:230
576458250,1sleep20-21swapper/221:45:112
544991250,1sleep10-21swapper/121:15:131
544991250,1sleep10-21swapper/121:15:131
522786250,1sleep20-21swapper/220:50:252
519970250,2sleep10-21swapper/120:50:161
501871250,1sleep20-21swapper/220:30:292
486688250,3sleep00-21swapper/020:18:520
428754250,3sleep10-21swapper/119:21:491
739386240,1sleep20-21swapper/200:20:042
739386240,1sleep20-21swapper/200:20:042
681107240,1sleep00-21swapper/023:25:150
654959240,1sleep10-21swapper/123:00:101
623615240,1sleep10-21swapper/122:30:131
539782240,1sleep00-21swapper/021:10:140
513255240,1sleep00-21swapper/020:45:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional