You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 06:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Mon Nov 17, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
817882212161,56sleep30-21swapper/319:07:523
81730328657,6sleep00-21swapper/019:05:240
81795028164,10sleep10-21swapper/119:08:461
81778728156,9sleep20-21swapper/219:06:422
818179992723,2cyclictest0-21swapper/322:05:163
818179992722,2cyclictest0-21swapper/323:31:173
818179992721,2cyclictest0-21swapper/321:45:283
818179992622,2cyclictest0-21swapper/321:30:133
818179992621,3cyclictest0-21swapper/323:45:213
818179992621,2cyclictest0-21swapper/322:11:133
818179992621,2cyclictest0-21swapper/322:11:123
818179992521,2cyclictest0-21swapper/322:20:173
818179992520,3cyclictest0-21swapper/300:10:173
818179992520,2cyclictest0-21swapper/300:00:213
11126952251,3sleep00-21swapper/022:51:550
818179992419,3cyclictest0-21swapper/323:40:133
818179992318,2cyclictest0-21swapper/322:26:023
81817999231,17cyclictest1216908-21taskset23:57:533
818179992219,1cyclictest0-21swapper/322:03:583
818179992219,1cyclictest0-21swapper/321:10:203
818179992219,1cyclictest0-21swapper/300:25:153
818179992218,2cyclictest0-21swapper/322:38:523
818179992218,2cyclictest0-21swapper/322:30:223
818179992218,2cyclictest0-21swapper/321:50:413
818179992218,2cyclictest0-21swapper/321:25:243
818179992218,1cyclictest0-21swapper/323:11:383
818179992218,1cyclictest0-21swapper/323:11:373
818179992217,2cyclictest0-21swapper/300:20:303
818179992217,2cyclictest0-21swapper/300:15:203
818179992214,5cyclictest66550irq/131-enp1s0-TxRx-322:40:243
81817999221,17cyclictest1024373-21taskset21:55:423
12325642220,9sleep244-21ksoftirqd/200:06:542
81817999216,2cyclictest0-21swapper/321:19:503
81817999216,2cyclictest0-21swapper/321:19:503
81817999212,14cyclictest0-21swapper/323:35:133
81817999212,13cyclictest1103818-21taskset22:45:323
818179992119,1cyclictest0-21swapper/321:40:013
818179992118,1cyclictest0-21swapper/320:55:143
818179992115,3cyclictest942109-21turbostat21:04:593
818179992111,6cyclictest66550irq/131-enp1s0-TxRx-300:05:143
10768252211,14sleep21078524-21grep22:30:222
81817999202,13cyclictest1129873-21taskset23:03:533
81817999202,12cyclictest1162554-21taskset23:25:143
81817999201,14cyclictest1154569-21chrt23:20:073
12313022204,5sleep01231442-21cat00:05:270
11207872200,4sleep20-21swapper/222:57:022
81817999193,12cyclictest9464052chrt21:05:223
81817999192,12cyclictest1207820-21taskset23:51:143
818179991916,1cyclictest0-21swapper/320:15:273
818179991916,1cyclictest0-21swapper/320:15:273
818179991916,1cyclictest0-21swapper/319:50:183
818179991915,2cyclictest0-21swapper/322:55:003
818179991915,2cyclictest0-21swapper/322:50:223
818179991915,2cyclictest0-21swapper/319:40:193
818179991915,2cyclictest0-21swapper/319:25:003
818179991915,2cyclictest0-21swapper/300:35:203
818179991914,3cyclictest0-21swapper/323:15:203
81817999191,14cyclictest0-21swapper/322:17:323
81817999191,13cyclictest0-21swapper/320:40:133
9614382181,10sleep0947354-21diskmemload21:16:550
9614382181,10sleep0947354-21diskmemload21:16:540
9104052181,5sleep2910433-21munin-run20:35:012
81817999183,11cyclictest1272774-21taskset00:32:433
81817999182,11cyclictest968021-21taskset21:20:253
818179991815,1cyclictest0-21swapper/319:55:123
818179991814,2cyclictest0-21swapper/320:25:143
81817999181,13cyclictest0-21swapper/323:05:013
12549492181,8sleep0947354-21diskmemload00:20:240
11532062181,3sleep10-21swapper/123:18:151
10551572181,5sleep01055218-21ntpq22:15:190
10481842181,11sleep1947354-21diskmemload22:10:481
10481842181,11sleep1947354-21diskmemload22:10:471
10032172181,5sleep2947354-21diskmemload21:44:592
9856542171,3sleep2947354-21diskmemload21:32:242
9773112171,3sleep1947354-21diskmemload21:26:471
9692672171,3sleep10-21swapper/121:21:361
8988212171,10sleep1898863-21ntpq20:20:211
818179991714,1cyclictest0-21swapper/320:50:153
818179991714,1cyclictest0-21swapper/320:10:183
818179991714,1cyclictest0-21swapper/320:00:193
81817999171,10cyclictest896957-21taskset20:20:173
81817999171,10cyclictest1001186-21taskset21:41:223
12485322171,3sleep20-21swapper/200:16:572
12227562171,11sleep21222824-21ntpq00:00:202
11941142171,3sleep00-21swapper/023:44:590
11923772171,3sleep10-21swapper/123:42:111
11620862171,2sleep00-21swapper/023:24:540
11288452171,4sleep1947354-21diskmemload23:02:061
10887912171,2sleep00-21swapper/022:37:280
10196342171,10sleep21019769-21diskstats21:55:182
10017082171,3sleep00-21swapper/021:42:240
9933782161,2sleep20-21swapper/221:37:102
9706872161,7sleep20-21swapper/221:23:582
818179991614,1cyclictest0-21swapper/320:30:173
818179991614,1cyclictest0-21swapper/319:25:173
818179991613,1cyclictest0-21swapper/320:45:163
818179991610,2cyclictest0-21swapper/319:30:133
81817999160,13cyclictest0-21swapper/320:35:243
81817999160,13cyclictest0-21swapper/320:05:303
81817999160,13cyclictest0-21swapper/319:35:273
81817999160,12cyclictest818373-21chrt19:10:153
81817999160,10cyclictest862687-21chrt19:48:193
12073662161,10sleep01207377-21ssh23:50:320
11999792161,10sleep1947354-21diskmemload23:46:331
11436712161,2sleep20-21swapper/223:10:382
11436712161,2sleep20-21swapper/223:10:372
11358612161,10sleep1947354-21diskmemload23:05:431
10826952160,3sleep10-21swapper/122:35:051
10668612161,2sleep10-21swapper/122:25:021
10590022161,4sleep21059016-21ssh22:20:072
10356212161,9sleep01035780-21grep22:05:170
9539812153,2sleep10-21swapper/121:12:431
81817999150,12cyclictest0-21swapper/319:15:193
12744902151,8sleep01273986-21apt-get00:35:020
12660722150,2sleep00-21swapper/000:29:050
12317902150,3sleep1947354-21diskmemload00:05:451
11309382151,9sleep21131073-21grep23:05:162
10731812151,3sleep2947354-21diskmemload22:27:482
10643352152,8sleep0947354-21diskmemload22:21:080
10486352151,8sleep0947354-21diskmemload22:11:300
10486352151,8sleep0947354-21diskmemload22:11:300
10273802151,7sleep21027406-21apt-get22:00:142
10053742151,9sleep21007150-21tr21:45:202
9712182141,8sleep0947354-21diskmemload21:24:520
9633552141,8sleep1963497-21cat21:20:011
9633552141,8sleep1963497-21cat21:20:001
11857822141,9sleep01185790-21ssh23:39:340
11767402140,3sleep2421rcuc/223:32:542
11758952141,8sleep1947354-21diskmemload23:31:251
11695672148,3sleep01169578-21ssh23:29:110
11538322141,9sleep2947354-21diskmemload23:19:202
11521292141,5sleep00-21swapper/023:16:250
11150172140,3sleep00-21swapper/022:55:120
10969132141,9sleep01096920-21sh22:42:450
9784192131,5sleep2947354-21diskmemload21:28:362
9541702131,5sleep00-21swapper/021:13:020
9414152131,8sleep1941639-21grep21:00:291
12250972131,8sleep1947354-21diskmemload00:02:561
12090282131,3sleep20-21swapper/223:53:112
11864282131,3sleep21186448-21apt-get23:40:122
11063262131,8sleep0802-21snmpd22:49:350
10981822131,3sleep21098193-21ssh22:44:492
10747792131,6sleep10-21swapper/122:30:081
10401902131,8sleep1947354-21diskmemload22:05:391
10186962130,1sleep00-21swapper/021:54:230
9950892121,7sleep1995426-21wc21:40:011
9930522121,7sleep0993061-21diskmemload21:36:360
9867712121,4sleep1947354-21diskmemload21:34:151
9854882120,5sleep00-21swapper/021:32:100
8882762121,3sleep1888326-21ldconfig20:10:221
8802842121,7sleep0882000-21kthreadcore20:05:210
8223712121,7sleep2822415-21collect219:10:232
12486072121,4sleep1947354-21diskmemload00:17:031
12115682121,3sleep11213189-21kthreadcore23:55:191
12035342121,7sleep11205148-21kthreadcore23:50:171
12000812121,4sleep2947354-21diskmemload23:46:422
11838692121,4sleep2947354-21diskmemload23:36:282
11836232121,4sleep1947354-21diskmemload23:36:031
11674982126,3sleep10-21swapper/123:25:451
11458922121,7sleep11145902-21ssh23:14:181
11458922121,7sleep11145902-21ssh23:14:171
11429412121,2sleep01142998-21ntpq23:10:230
11429412121,2sleep01142998-21ntpq23:10:230
11280362121,4sleep0947354-21diskmemload23:00:490
11119232121,4sleep2947354-21diskmemload22:50:422
10975192121,4sleep1947354-21diskmemload22:43:421
10179432121,4sleep2947354-21diskmemload21:53:042
10086182121,4sleep1947354-21diskmemload21:45:471
10027512121,6sleep11002757-21ssh21:44:131
9777192111,3sleep0947354-21diskmemload21:27:260
9527942111,3sleep20-21swapper/221:10:482
8945202111,4sleep1894591-21munin-run20:20:011
8945202111,4sleep1894591-21munin-run20:20:001
8900232111,6sleep2890323-21kthreadcore20:15:172
8900232111,6sleep2890323-21kthreadcore20:15:162
8618502111,6sleep2861893-21ntpq19:45:222
8532502111,6sleep1854905-21kthreadcore19:40:211
12799332111,4sleep2947354-21diskmemload00:36:172
12798822111,3sleep1947354-21diskmemload00:36:111
12723272111,4sleep1947354-21diskmemload00:31:511
12667842111,3sleep00-21swapper/000:30:060
12661692111,4sleep2947354-21diskmemload00:29:152
12562732111,2sleep10-21swapper/100:21:121
12485082111,3sleep0947354-21diskmemload00:16:540
12257622111,3sleep0947354-21diskmemload00:03:590
12111592110,7sleep01211566-21cut23:55:170
11764732111,4sleep0947354-21diskmemload23:32:240
11693512111,4sleep2947354-21diskmemload23:28:502
11619022111,4sleep1947354-21diskmemload23:24:351
11545562110,3sleep20-21swapper/223:20:072
11383502111,3sleep00-21swapper/023:09:550
11191052111,4sleep11119140-21ntpq22:55:211
11072892110,1sleep10-21swapper/122:50:161
10815002111,3sleep00-21swapper/022:33:270
10672472111,4sleep0947354-21diskmemload22:25:130
10574612111,3sleep2947354-21diskmemload22:18:022
10497522110,1sleep20-21swapper/222:13:222
10497522110,1sleep20-21swapper/222:13:212
10336792111,3sleep10-21swapper/122:02:571
10331712110,5sleep0947354-21diskmemload22:02:080
10194862111,3sleep1947354-21diskmemload21:55:171
10086092111,3sleep00-21swapper/021:45:450
9558052101,3sleep2947354-21diskmemload21:15:172
9558052101,3sleep2947354-21diskmemload21:15:162
9423832101,4sleep2942551-21expr21:05:142
9225802101,4sleep1924307-21cut20:45:171
9162732101,4sleep1916486-21hddtemp_smartct20:40:171
9109532101,4sleep2911077-21perl20:35:162
8478672101,4sleep0849501-21grep19:35:180
8416612101,3sleep0841680-21apt-get19:30:130
12591512100,1sleep10-21swapper/100:25:131
12559272100,2sleep20-21swapper/200:20:412
12400592101,3sleep0947354-21diskmemload00:11:100
12391552101,5sleep21239259-21sh00:10:272
12175842101,2sleep20-21swapper/223:58:572
11984872101,4sleep01198616-21ntpq23:45:230
11282902101,3sleep2947354-21diskmemload23:01:122
11033382100,3sleep2947354-21diskmemload22:45:242
10886292101,3sleep2947354-21diskmemload22:37:102
10622182101,2sleep1947354-21diskmemload22:20:181
10572972101,2sleep10-21swapper/122:17:451
10405412101,2sleep20-21swapper/222:06:122
10243832101,2sleep00-21swapper/021:55:430
10181422101,3sleep1947354-21diskmemload21:53:271
947170290,1sleep00-21swapper/021:07:480
935805291,3sleep2935853-21ntpq20:55:202
900122291,3sleep1900141-21apt-get20:25:131
882002291,4sleep1882872-21sed20:05:221
877707291,3sleep0877747-21ntpq20:00:230
875437291,4sleep1877199-21kthreadcore20:00:201
872454291,2sleep00-21swapper/019:55:230
867855290,1sleep10-21swapper/119:51:461
861803291,4sleep1861844-21ntpq19:45:211
852210291,4sleep1852224-21dump-pmu-power19:40:001
1271885291,2sleep20-21swapper/200:30:552
1240633291,4sleep11240643-21ssh00:12:061
1099351291,4sleep11099526-21cat22:45:151
935458280,4sleep0935753-21strings20:55:180
914725280,3sleep1914772-21ntpq20:35:231
914685281,3sleep0914723-21ntpq20:35:220
906635281,3sleep0908318-21grep20:30:180
900703281,3sleep2902153-21kthreadcore20:25:182
884261281,2sleep00-21swapper/020:10:130
868665281,3sleep1868818-21grep19:55:151
868116280,4sleep244-21ksoftirqd/219:55:002
865761281,3sleep2866957-21memory19:50:202
863084280,3sleep0863103-21apt-get19:50:130
853016281,3sleep0853249-21cut19:40:190
842233281,3sleep1843611-21kthreadcore19:30:191
841826281,3sleep2841949-21diskstats19:30:162
831219280,1sleep10-21swapper/119:20:131
829713281,3sleep2829759-21ntpq19:15:202
942361270,2sleep10-21swapper/121:05:131
931330270,1sleep00-21swapper/020:52:260
930587271,3sleep2930647-21ld20:50:242
926178270,1sleep20-21swapper/220:49:152
925923270,1sleep00-21swapper/020:46:000
899767270,1sleep20-21swapper/220:23:512
883857270,4sleep20-21swapper/220:08:152
873984271,2sleep2841064-21kworker/u16:0+flush-8:020:00:172
851051270,2sleep2851161-21strings19:35:202
840493271,3sleep20-21swapper/219:25:222
836935271,2sleep00-21swapper/019:25:160
836070270,1sleep20-21swapper/219:24:392
830731271,3sleep11-21systemd19:20:001
823994271,2sleep00-21swapper/019:15:020
823178270,1sleep00-21swapper/019:12:220
938673260,2sleep00-21swapper/021:00:190
931794261,1sleep1931813-21apt-get20:55:111
931234260,1sleep10-21swapper/120:51:151
920891260,1sleep20-21swapper/220:43:472
899673260,4sleep00-21swapper/020:22:400
893571261,2sleep00-21swapper/020:15:230
893571261,2sleep00-21swapper/020:15:220
889063260,1sleep20-21swapper/220:12:432
861948261,2sleep00-21swapper/019:45:240
840440261,2sleep10-21swapper/119:25:211
835101260,2sleep00-21swapper/019:20:200
823317260,1sleep10-21swapper/119:14:081
910165250,1sleep10-21swapper/120:32:031
903908250,1sleep00-21swapper/020:25:200
856576250,1sleep20-21swapper/219:40:262
937089240,1sleep20-21swapper/221:00:142
916161240,1sleep00-21swapper/020:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional