You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 21:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Aug 27, 2025 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
112433522430,4sleep0221rcuc/012:17:400
100560121710,1sleep31005602-21grep11:05:153
7141622155109,15sleep20-21swapper/207:06:392
714351214497,16sleep00-21swapper/007:09:000
7140952143101,34sleep30-21swapper/307:05:483
7141482131109,15sleep10-21swapper/107:06:301
714554994035,3cyclictest1118102-21cpuspeed_turbos12:15:133
10751972390,3sleep0171ktimers/011:50:020
714554993027,2cyclictest1062498-21grep11:40:193
714554992920,6cyclictest877525-21cut09:35:183
714554992824,2cyclictest1074681-21idleruntime-cro11:50:003
714554992824,2cyclictest0-21swapper/310:50:163
714554992819,6cyclictest840868-21cut09:10:183
714554992819,6cyclictest757549-21cut07:50:143
714554992723,3cyclictest1091873-21sendmail12:00:003
714554992620,1cyclictest191rcu_preempt12:20:483
714554992522,2cyclictest719496-21idleruntime-cro07:15:003
714554992521,3cyclictest746681-40tracker-extract07:39:023
714554992521,2cyclictest1160350-21sh12:40:013
714554992519,1cyclictest191rcu_preempt09:15:013
714554992518,2cyclictest191rcu_preempt12:25:153
714554992518,1cyclictest191rcu_preempt12:10:013
714554992516,6cyclictest982754-21sh10:49:193
71455499251,20cyclictest898539-21cut09:50:173
714554992419,1cyclictest191rcu_preempt10:25:213
714554992418,4cyclictest870909-21grep09:30:233
714554992418,1cyclictest191rcu_preempt10:06:533
714554992415,6cyclictest994563-21uname10:55:203
714554992412,9cyclictest925892-21tr10:10:163
714554992319,2cyclictest736-21gdbus10:25:003
714554992318,2cyclictest191rcu_preempt11:53:443
714554992311,8cyclictest884482-21tr09:40:173
10486552230,2sleep10-21swapper/111:32:151
8355342220,2sleep10-21swapper/109:05:201
71455499229,9cyclictest1028372-21tr11:20:143
714554992218,2cyclictest0-21swapper/312:00:243
714554992214,6cyclictest1016593-21ntpq11:10:213
714554992212,7cyclictest822967-21hddtemp_smartct08:55:143
71455499221,18cyclictest969448-21diskstats10:40:143
71455499221,17cyclictest831466-21/usr/sbin/munin09:00:213
714554992210,5cyclictest974917-21kworker/u16:3+nfsiod11:26:523
9475892210,8sleep1947788-21gltestperf10:25:131
71455499218,7cyclictest52-21ksoftirqd/310:00:023
71455499214,11cyclictest1001375-21kthreadcore11:00:173
71455499214,11cyclictest1001375-21kthreadcore11:00:163
714554992118,1cyclictest0-21swapper/308:00:273
71455499211,6cyclictest908280-21ssh09:56:123
714554992116,3cyclictest852895-21runrttasks09:20:143
714554992115,5cyclictest0-21swapper/311:38:143
714554992113,2cyclictest191rcu_preempt10:15:023
714554992110,8cyclictest889039-21ls09:45:143
714554992110,7cyclictest956482-21tr10:30:173
71455499210,7cyclictest1144112-21cut12:30:143
71455499205,4cyclictest997889-21kworker/u16:1+xprtiod11:18:343
714554992016,2cyclictest0-21swapper/310:35:243
714554992016,2cyclictest0-21swapper/308:30:233
714554992012,2cyclictest191rcu_preempt12:05:173
714554992012,2cyclictest191rcu_preempt11:30:193
714554992011,8cyclictest737185-21fschecks_count07:30:153
71455499201,17cyclictest782204-21apt-get08:15:113
71455499200,18cyclictest766152-21grep07:55:243
71455499196,2cyclictest191rcu_preempt08:09:563
714554991910,7cyclictest792367-21sh08:25:123
11490552191,2sleep20-21swapper/212:31:222
11411322191,3sleep20-21swapper/212:27:052
9179302183,8sleep2917955-21apt-get10:05:152
7671602186,8sleep2767178-21apt-get08:00:132
71455499188,7cyclictest787328-21grep08:20:143
71455499187,7cyclictest721748-21ntpq07:15:033
714554991813,1cyclictest191rcu_preempt09:05:143
714554991813,1cyclictest191rcu_preempt08:35:133
714554991811,5cyclictest866548-21sh09:29:193
9684202172,5sleep0837378-21diskmemload10:39:030
7308072170,2sleep00-21swapper/007:20:230
71455499179,1cyclictest191rcu_preempt07:20:153
71455499178,6cyclictest802445-21packagekitd08:40:063
71455499178,6cyclictest262-21systemd-journal07:40:373
714554991713,2cyclictest0-21swapper/307:45:163
714554991713,2cyclictest0-21swapper/307:45:153
714554991712,3cyclictest817704-21cpuspeed_turbos08:50:143
714554991710,1cyclictest191rcu_preempt08:45:163
10485222171,4sleep0837378-21diskmemload11:32:030
10469052171,10sleep21046970-21/usr/sbin/munin11:30:252
9159222161,2sleep10-21swapper/110:01:531
8719832161,4sleep0837378-21diskmemload09:31:240
8458102161,10sleep1847130-21cut09:15:191
71455499169,2cyclictest191rcu_preempt08:10:143
10794512161,3sleep10-21swapper/111:50:241
9839202156,2sleep0191rcu_preempt10:50:180
8426862150,1sleep10-21swapper/109:10:451
8035682150,2sleep00-21swapper/008:35:170
714554991510,3cyclictest735716-21ntpq07:25:203
11006672151,3sleep20-21swapper/212:05:032
9965722141,3sleep00-21swapper/010:58:160
9387712141,8sleep2837378-21diskmemload10:18:012
7467552141,8sleep2746699-31find07:39:122
7200282141,9sleep0721504-21sh07:15:020
11078832140,1sleep00-21swapper/012:08:470
10032662141,7sleep0837378-21diskmemload11:01:570
10032662141,7sleep0837378-21diskmemload11:01:570
9242422131,8sleep1916953-21kworker/u16:3+rpciod10:08:501
8516062131,7sleep2837378-21diskmemload09:18:412
11218872130,3sleep10-21swapper/112:15:221
11195432131,8sleep21121317-21cut12:15:182
10731422131,3sleep20-21swapper/211:48:012
10582502131,6sleep01058287-21munin-run11:40:000
10267382131,7sleep2837378-21diskmemload11:19:592
9806672121,7sleep2980732-21/usr/sbin/munin10:45:222
8790872121,7sleep2837378-21diskmemload09:36:042
8657902121,7sleep2865795-21ssh09:27:472
8535292121,2sleep20-21swapper/209:20:162
7194282122,7sleep2945-21snmpd07:14:352
11318692121,6sleep0837378-21diskmemload12:21:010
11095162122,5sleep21109532-21apt-get12:10:132
10898302121,8sleep21089844-21ssh11:57:182
10644252121,2sleep00-21swapper/011:42:210
10515792121,7sleep11053236-21kthreadcore11:35:171
9882252111,3sleep10-21swapper/110:50:551
9817182111,3sleep10-21swapper/110:47:011
9761182110,7sleep0976124-21rm10:44:370
9617852110,4sleep2961818-21munin-run10:35:012
9545932111,4sleep1954262-21apt-get10:30:031
9438522111,5sleep2943930-21ntpq10:20:222
9312422111,7sleep1931251-21ssh10:13:151
9167122111,5sleep0837378-21diskmemload10:03:320
9105782111,6sleep20-21swapper/210:00:122
9081552110,7sleep1908165-21ssh09:56:001
9034712111,6sleep0903504-21ssh09:55:140
8651272111,3sleep10-21swapper/109:26:251
8584932110,1sleep10-21swapper/109:23:001
8163122111,5sleep1816358-21ntpq08:45:211
8162722111,5sleep0816310-21ntpq08:45:200
7366772111,3sleep00-21swapper/007:29:200
11332482111,3sleep20-21swapper/212:22:522
11077342110,1sleep10-21swapper/112:08:351
10885022111,3sleep10-21swapper/111:55:321
10591522111,6sleep11060003-21cut11:40:181
10421302110,7sleep21042140-21ssh11:29:492
10317122111,3sleep11031722-21ssh11:20:301
10057702114,3sleep00-21swapper/011:05:170
9956432100,3sleep10-21swapper/110:56:311
9907322101,4sleep2990756-21apt-get10:55:122
9836272101,4sleep2983756-21perl10:50:162
9621692101,3sleep10-21swapper/110:35:151
9528432101,6sleep0837378-21diskmemload10:27:040
9449342101,3sleep10-21swapper/110:21:051
9393462101,3sleep10-21swapper/110:19:211
9302452101,4sleep2837378-21diskmemload10:11:232
9006932101,3sleep10-21swapper/109:50:381
8965112100,6sleep0897119-21kthreadcore09:50:170
8956362101,3sleep00-21swapper/009:49:570
8939752101,3sleep10-21swapper/109:46:301
8736642101,4sleep1837378-21diskmemload09:34:421
8419082101,4sleep2841967-21ntp_states09:10:232
8418112101,4sleep0841851-21ntpq09:10:200
8273112100,6sleep2827486-21latency_hist09:00:012
8267222101,4sleep1826939-21users08:55:271
7662512100,6sleep0766500-21awk07:55:250
7467482100,4sleep0746699-31find07:39:060
7443322101,4sleep1745569-21memory07:35:181
7357172101,5sleep1735763-21ntpq07:25:211
11592672101,4sleep0837378-21diskmemload12:38:380
11589312101,4sleep2837378-21diskmemload12:38:122
11522422100,2sleep10-21swapper/112:35:141
11516782101,3sleep10-21swapper/112:34:491
11405972101,3sleep00-21swapper/012:26:230
11391512101,5sleep11139190-21ntpq12:25:201
11312812101,6sleep10-21swapper/112:20:271
11173562101,4sleep0837378-21diskmemload12:14:440
10996202101,3sleep00-21swapper/012:04:020
10969892101,2sleep20-21swapper/212:00:352
10881822101,4sleep01088438-21users11:55:270
10721672101,2sleep10-21swapper/111:46:421
10345652101,4sleep0837378-21diskmemload11:25:060
10341642101,3sleep20-21swapper/211:24:512
10270572101,2sleep00-21swapper/011:20:050
10188952101,2sleep10-21swapper/111:13:431
10184742101,4sleep0837378-21diskmemload11:12:590
10101962101,4sleep2837378-21diskmemload11:06:082
10030372101,4sleep1837378-21diskmemload11:01:341
10030372101,4sleep1837378-21diskmemload11:01:331
975115291,3sleep2837378-21diskmemload10:42:552
974378290,1sleep10-21swapper/110:41:331
962192291,4sleep2962356-21diskstats10:35:162
952981291,3sleep2837378-21diskmemload10:27:192
945010291,2sleep00-21swapper/010:21:120
938281291,4sleep0837378-21diskmemload10:17:060
921912291,4sleep0837378-21diskmemload10:05:240
908588291,2sleep20-21swapper/209:56:532
901416291,3sleep2837378-21diskmemload09:51:552
894722291,2sleep20-21swapper/209:48:112
887756291,2sleep20-21swapper/209:43:292
886771291,3sleep0837378-21diskmemload09:41:350
881711291,2sleep10-21swapper/109:40:111
879722291,2sleep10-21swapper/109:37:121
878910290,2sleep00-21swapper/009:35:460
873592291,3sleep20-21swapper/209:34:352
856594291,4sleep0856719-21grep09:20:220
821300291,4sleep0821330-21ntpq08:50:200
777194291,2sleep00-21swapper/008:10:130
757350291,3sleep2757560-21hddtemp_smartct07:50:152
727419291,4sleep2727889-21cut07:20:182
1150311291,2sleep00-21swapper/012:32:590
1115209291,2sleep10-21swapper/112:11:531
1098249291,2sleep10-21swapper/112:02:151
1081839290,2sleep20-21swapper/211:52:582
1063853290,2sleep20-21swapper/211:41:272
1055002291,2sleep20-21swapper/211:35:262
1042209291,4sleep11042215-21ssh11:29:561
1025971290,3sleep00-21swapper/011:18:400
1018000291,2sleep20-21swapper/211:12:042
1004524291,3sleep2837378-21diskmemload11:04:192
1004524291,3sleep2837378-21diskmemload11:04:182
977157280,5sleep0977963-21grep10:45:170
957076281,3sleep0958085-21ssh10:30:180
932012281,3sleep00-21swapper/010:14:390
864731281,1sleep00-21swapper/009:25:400
851386280,2sleep00-21swapper/009:18:190
827594281,2sleep1827613-21apt-get09:00:111
821338281,3sleep1821387-21ntpq08:50:221
807119280,1sleep20-21swapper/208:38:122
801048281,4sleep2801175-21sendmail_mailqu08:30:232
796551280,1sleep10-21swapper/108:25:281
762140280,3sleep2762159-21apt-get07:55:112
757179281,3sleep00-21swapper/007:50:130
747585281,3sleep2749061-21kthreadcore07:40:172
737446281,4sleep0738251-21tr07:30:170
1071597280,3sleep0837378-21diskmemload11:45:570
1019866280,2sleep10-21swapper/111:15:041
828615271,3sleep2830333-21kthreadcore09:00:172
791940271,3sleep1791989-21unin-run08:25:011
791250271,3sleep0787209-21/usr/sbin/munin08:20:270
786765270,1sleep00-21swapper/008:18:030
776239270,2sleep20-21swapper/208:05:252
772669271,3sleep1774210-21cut08:05:161
752154271,2sleep1752173-21apt-get07:45:131
752154271,2sleep1752173-21apt-get07:45:121
751687270,1sleep00-21swapper/007:42:280
732814271,3sleep2734516-21kthreadcore07:25:192
730573270,2sleep10-21swapper/107:20:211
716007271,3sleep1717755-21kthreadcore07:10:191
714726271,2sleep00-21swapper/007:10:140
1011425270,2sleep10-21swapper/111:08:231
833809261,2sleep00-21swapper/009:05:200
812633260,2sleep20-21swapper/208:45:132
812038260,1sleep10-21swapper/108:41:041
807550260,2sleep20-21swapper/208:40:132
802641261,2sleep10-21swapper/108:35:151
782585261,2sleep20-21swapper/208:15:152
780983260,1sleep10-21swapper/108:10:221
776676260,1sleep00-21swapper/008:06:480
771563260,2sleep10-21swapper/108:00:361
771030261,2sleep00-21swapper/008:00:230
756863260,1sleep20-21swapper/207:49:272
756863260,1sleep20-21swapper/207:49:262
756042261,2sleep00-21swapper/007:45:240
756042261,2sleep00-21swapper/007:45:230
738252260,3sleep1321rcuc/107:30:181
726497260,1sleep10-21swapper/107:17:321
726004261,2sleep2726130-21timerandwakeup07:15:292
836289250,1sleep20-21swapper/209:05:222
822604251,1sleep00-21swapper/008:55:140
810009250,1sleep00-21swapper/008:40:180
800999250,1sleep10-21swapper/108:30:221
795958250,2sleep20-21swapper/208:25:212
791929250,3sleep20-21swapper/208:24:542
766600250,3sleep10-21swapper/107:56:351
750866250,1sleep10-21swapper/107:40:201
741661250,1sleep20-21swapper/207:34:122
827834240,1sleep00-21swapper/009:00:140
819801240,1sleep20-21swapper/208:50:192
796953240,1sleep00-21swapper/008:30:000
792200240,1sleep00-21swapper/008:25:120
785775240,1sleep10-21swapper/108:15:181
777371240,1sleep20-21swapper/208:10:152
760943240,1sleep10-21swapper/107:50:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional