You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 15:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Sep 16, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32071210778,9sleep10-21swapper/107:08:351
3194829683,9sleep30-21swapper/307:07:003
3189328068,8sleep00-21swapper/007:06:170
3203527767,6sleep20-21swapper/207:08:082
191472540,1sleep20-21swapper/210:02:382
3227899270,1cyclictest0-21swapper/212:07:012
3227899260,24cyclictest0-21swapper/210:21:582
3227599261,2cyclictest224-21systemd-journal07:55:001
3227599241,2cyclictest0-21swapper/107:15:171
3227599240,2cyclictest0-21swapper/110:40:151
3227599233,3cyclictest25117-21diskmemload10:49:031
3227599231,3cyclictest224-21systemd-journal08:40:011
3227899220,22cyclictest0-21swapper/210:51:512
32270992219,2cyclictest21031-21smartctl11:25:180
3227599218,12cyclictest32365-21ssh09:45:241
3227599213,3cyclictest13185-21ps11:45:221
3227599211,18cyclictest0-21swapper/110:16:191
3227599210,20cyclictest0-21swapper/109:51:361
32281992018,1cyclictest8455-21grep10:20:283
3227899200,19cyclictest0-21swapper/211:23:492
3227599203,4cyclictest11434-21sh10:23:531
3227599201,3cyclictest14380-21grep11:20:171
32275992010,9cyclictest25117-21diskmemload12:27:361
3227599200,17cyclictest0-21swapper/111:50:391
3227899190,19cyclictest0-21swapper/211:18:332
3227599191,4cyclictest21242-21ssh12:19:261
3228199185,10cyclictest0-21swapper/311:05:143
3228199183,9cyclictest0-21swapper/309:05:143
32281991817,1cyclictest0-21swapper/312:33:043
32281991816,1cyclictest15030-21apt-config12:15:003
32281991816,1cyclictest10836-21apt-config10:50:003
32278991817,1cyclictest0-21swapper/210:16:072
3227899181,3cyclictest16205-21/usr/sbin/munin09:30:202
3227599189,8cyclictest30690-21cron09:45:001
32275991810,7cyclictest19351-21sh10:30:131
3227099187,6cyclictest4861-21apt-get07:20:000
32281991712,4cyclictest31289-21sshd11:34:363
3227599179,7cyclictest25150-21tail08:00:171
3227599178,2cyclictest28500-21fgrep10:10:231
3227599178,2cyclictest28500-21fgrep10:10:231
3227599174,4cyclictest24166-21cut11:00:231
3227599172,5cyclictest8048-21/usr/sbin/munin11:15:131
3227599171,6cyclictest951-21runrttasks12:02:331
32275991710,6cyclictest4834-21sed09:20:161
3227099174,3cyclictest224-21systemd-journal08:15:010
3228199160,15cyclictest0-21swapper/310:26:093
3227899160,16cyclictest0-21swapper/212:10:482
3227899160,16cyclictest0-21swapper/211:56:062
3227599168,7cyclictest5568-21tr07:20:161
3227599168,2cyclictest7238-21ssh12:07:531
3227599168,2cyclictest29380-21date12:00:011
3227599168,2cyclictest24040-21taskset11:27:471
3227599168,2cyclictest19644-21ssh10:56:191
3227599168,2cyclictest13026-21ssh09:57:131
3227599168,2cyclictest11304-21rm12:10:381
3227599168,1cyclictest26869-21date10:10:001
3227599161,4cyclictest224-21systemd-journal11:40:001
3227599161,2cyclictest0-21swapper/107:35:001
3227599161,1cyclictest0-21swapper/111:10:141
3227099164,6cyclictest9652-21idleruntime-cro09:55:010
32278991514,1cyclictest37850irq/131-snd_hda10:45:202
3227899150,14cyclictest0-21swapper/210:14:162
3227899150,14cyclictest0-21swapper/210:14:162
3227599159,5cyclictest25-21ksoftirqd/109:10:241
3227599158,2cyclictest1576-21grep07:10:231
3227599158,2cyclictest14962-21rm12:39:411
3227599158,2cyclictest14073-21ssh10:51:401
3227599158,2cyclictest10160-21ssh11:43:281
3227599158,1cyclictest27130-21cat11:30:251
3227599157,2cyclictest4987-21ssh12:31:231
3227599157,2cyclictest20731-21ssh09:35:041
3227599157,2cyclictest15590-21head07:40:201
3227599157,1cyclictest29858-21tr08:10:211
3227599151,4cyclictest224-21systemd-journal08:55:001
32275991513,1cyclictest12951-21apt-get10:25:151
3227599150,9cyclictest224-21systemd-journal08:35:001
3227599150,4cyclictest224-21systemd-journal09:20:001
3227599150,14cyclictest28946-21ssh10:37:101
3227099157,7cyclictest3685-21munin-run08:25:010
3227099153,6cyclictest6073-21awk08:30:000
3228199149,3cyclictest27002-21wc09:10:263
32281991411,2cyclictest15958-21grep11:20:283
32281991411,2cyclictest0-21swapper/310:13:213
32281991411,2cyclictest0-21swapper/310:13:203
32281991410,3cyclictest8390-21tr11:15:153
32281991410,3cyclictest778-21grep08:15:273
3228199140,14cyclictest0-21swapper/311:42:443
3228199140,13cyclictest0-21swapper/309:33:003
32278991410,3cyclictest20331-21sh08:59:592
3227899140,14cyclictest0-21swapper/212:34:372
3227899140,14cyclictest0-21swapper/211:40:122
3227599148,1cyclictest247912sleep109:06:071
3227599147,2cyclictest31775-21ssh11:07:181
3227599147,2cyclictest27733-21wc08:05:191
3227599147,2cyclictest26088-21ssh12:23:221
3227599147,2cyclictest17238-21ssh09:30:341
3227599147,1cyclictest14326-21cat08:45:191
3227599143,6cyclictest0-21swapper/109:25:021
32275991410,3cyclictest20323-21cron08:59:591
3227599140,9cyclictest22509-21taskset09:02:211
3227599140,4cyclictest224-21systemd-journal08:40:001
3227099145,1cyclictest30923-21apt-get09:15:140
32270991414,0cyclictest0-21swapper/007:30:150
32270991411,2cyclictest32689-21apache210:08:390
3227099140,12cyclictest2261-21sh11:10:150
3228199139,3cyclictest9287-21cron09:25:013
3228199139,3cyclictest2969-21grep11:10:183
32281991312,1cyclictest0-21swapper/310:19:303
32281991312,1cyclictest0-21swapper/308:53:563
3228199131,11cyclictest0-21swapper/309:59:133
32281991310,2cyclictest24117-21basename08:00:003
3228199130,2cyclictest0-21swapper/310:02:463
3228199130,13cyclictest0-21swapper/311:49:263
3228199130,12cyclictest25935-21cat08:00:253
3227899138,3cyclictest23601-21wc11:55:002
32278991310,3cyclictest32-21ksoftirqd/211:25:362
32278991310,2cyclictest30721-21taskset11:33:562
32278991310,2cyclictest30104-21sed11:05:272
32278991310,2cyclictest27408-21grep10:35:282
32278991310,2cyclictest17358-21grep07:45:172
32278991310,2cyclictest17079-21head08:50:222
32278991310,2cyclictest15495-21wc12:40:002
32278991310,2cyclictest11321-21head07:30:242
3227899130,11cyclictest26821-21expr09:40:202
3227599138,1cyclictest9816-21taskset07:29:561
3227599137,1cyclictest17011-21ssh10:00:271
3227599131,5cyclictest14584-21cron07:40:001
3227099139,3cyclictest11475-21cat09:25:260
32270991311,1cyclictest1243-21smartctl11:35:280
32270991310,2cyclictest26831-21head11:30:220
32270991310,2cyclictest26310-21grep10:35:190
32270991310,2cyclictest23617-21sed11:00:180
32270991310,2cyclictest2120-21ls10:15:230
3227099130,8cyclictest29326-21perf12:00:000
3227099130,6cyclictest224-21systemd-journal07:55:000
3227099130,1cyclictest0-21swapper/011:22:040
3228199129,2cyclictest9439-21sed07:25:253
3228199129,2cyclictest7411-21grep08:30:223
3228199129,2cyclictest6996-21grep07:20:273
3228199129,2cyclictest5502-21cat10:45:213
3228199129,2cyclictest4591-21cron10:44:593
3228199129,2cyclictest352-21sed09:15:283
3228199129,2cyclictest29254-21sh12:00:003
3228199129,2cyclictest27424-21grep08:05:173
3228199129,2cyclictest23197-21sed10:05:323
3228199129,2cyclictest22051-21cat09:35:223
3228199129,2cyclictest20327-21cron08:59:593
3228199129,2cyclictest17513-21tr07:45:173
32281991210,1cyclictest31282sleep312:05:003
32281991210,1cyclictest0-21swapper/312:12:243
3227899129,2cyclictest9799-21grep08:35:202
3227899129,2cyclictest8020-21grep08:30:282
3227899129,2cyclictest5167-21rm09:50:172
3227899129,2cyclictest5007-21ls08:25:212
3227899129,2cyclictest4849-21dump-pmu-power07:20:002
3227899129,2cyclictest3727-21cat09:20:002
3227899129,2cyclictest3183-21grep08:20:282
3227899129,2cyclictest29884-21grep12:25:252
3227899129,2cyclictest25134-21awk09:10:002
3227899129,2cyclictest22398-21tr12:20:172
3227899129,2cyclictest14208-21head08:45:182
3227899128,3cyclictest26146-21taskset08:00:312
3227899123,7cyclictest15252-21sh11:47:192
3227899121,1cyclictest32274-21df_inode10:40:172
3227899120,10cyclictest0-21swapper/209:35:042
3227599127,1cyclictest3591-21taskset08:24:031
3227599120,5cyclictest19069-21sshd07:46:261
3227099129,2cyclictest30098-21grep11:05:260
3227099129,2cyclictest21141-21head10:30:270
3227099124,4cyclictest224-21systemd-journal12:39:430
3227099123,6cyclictest0-21swapper/007:35:110
3227099120,9cyclictest0-21swapper/009:44:310
3227099120,12cyclictest0-21swapper/009:30:330
3228199119,1cyclictest10976-21cat08:40:003
3228199118,2cyclictest8967-21ssh12:35:063
3228199118,2cyclictest4884-21basename07:20:003
3228199118,2cyclictest347-21cut11:35:203
3228199118,2cyclictest30495-21tail12:00:183
3228199118,2cyclictest29870-21head08:10:203
3228199118,2cyclictest2840-21ssh09:48:453
3228199118,2cyclictest28310-21tr12:25:133
3228199118,2cyclictest22731-21ssh11:26:113
3228199118,2cyclictest22289-21grep09:00:303
3228199118,2cyclictest18829-21ntpq11:50:203
3228199118,2cyclictest1759-21taskset07:10:263
3228199118,2cyclictest16895-21ssh10:55:033
3228199118,2cyclictest11682-21grep09:25:273
3228199117,3cyclictest5318-21wc09:50:183
3228199114,3cyclictest25117-21diskmemload11:00:273
3228199113,3cyclictest224-21systemd-journal08:50:003
3228199110,4cyclictest0-21swapper/310:30:133
3228199110,10cyclictest29339-21sh10:37:393
3228199110,10cyclictest14276-21chrt07:36:093
3227899118,2cyclictest9392-21tail07:25:252
3227899118,2cyclictest707-21ssh12:02:182
3227899118,2cyclictest32191-21basename11:35:182
3227899118,2cyclictest29432-21tr08:10:172
3227899118,2cyclictest23783-21chrt07:56:002
3227899118,2cyclictest22706-21sshd09:04:512
3227899118,2cyclictest2048-21grep07:10:282
3227899118,2cyclictest18825-21ssh12:16:582
3227899118,2cyclictest17979-21if_enp0s31f610:55:202
3227899118,2cyclictest12787-21sed08:40:242
3227899118,2cyclictest12771-21ssh10:25:052
3227899118,2cyclictest12528-21rm09:56:112
3227899118,2cyclictest1146-21chrt08:18:312
3227899117,3cyclictest30243-21ssh09:14:432
3227899116,4cyclictest9885-21taskset09:25:122
3227899113,4cyclictest224-21systemd-journal07:38:392
3227899110,9cyclictest0-21swapper/209:45:152
3227899110,4cyclictest224-21systemd-journal10:35:002
3227899110,10cyclictest5763-21ssh11:12:432
3227899110,0cyclictest0-21swapper/208:05:162
3227599118,2cyclictest786-21grep08:15:271
3227599115,2cyclictest5204-21wc08:25:221
3227099119,1cyclictest27555-21tail08:05:170
3227099118,2cyclictest6073-21expr11:40:150
3227099118,2cyclictest4885-21ssh12:31:150
3227099118,2cyclictest3692-21ssh10:43:330
3227099118,2cyclictest31401-21grep09:45:180
3227099118,2cyclictest29437-21ssh10:11:030
3227099118,2cyclictest29437-21ssh10:11:030
3227099118,2cyclictest27339-21ssh12:24:420
3227099118,2cyclictest26882-21ssh09:10:240
3227099118,2cyclictest21929-21cut09:35:210
3227099118,2cyclictest21773-21ntpq09:00:240
3227099118,2cyclictest20360-21ssh11:51:270
3227099118,2cyclictest18875-21sort08:55:170
3227099118,2cyclictest17725-21grep10:55:170
3227099118,2cyclictest17005-21sh10:27:540
3227099118,2cyclictest15659-21grep12:15:160
3227099118,2cyclictest15360-21ssh10:00:060
3227099118,2cyclictest14199-21grep08:45:190
3227099118,2cyclictest11938-21grep09:55:280
3227099118,2cyclictest11083-21grep12:10:280
3227099118,2cyclictest10964-21ssh11:16:390
3227099117,3cyclictest7451-21grep10:20:180
3227099113,7cyclictest23003-21apt-get09:05:150
3227099113,4cyclictest224-21systemd-journal12:04:590
3227099110,10cyclictest79452sleep012:09:030
3227099110,10cyclictest12739-21cat10:50:260
157862110,1sleep00-21swapper/011:47:590
3228199108,1cyclictest4890-21cut08:25:193
3228199108,1cyclictest3565-21taskset08:23:423
3228199108,1cyclictest261822sleep309:40:033
3228199108,1cyclictest19586-21taskset07:50:073
3228199108,1cyclictest16788-21chrt07:42:383
3228199108,1cyclictest10246-21egrep07:30:163
3228199107,2cyclictest22158-21sh12:20:153
3228199107,2cyclictest11852-21cut08:40:153
3227899108,1cyclictest7352-21taskset07:23:542
3227899107,2cyclictest3914-21sh09:20:072
3227899107,2cyclictest24714-21ssh11:00:302
3227899107,2cyclictest21772-21expr10:05:202
3227899107,2cyclictest15587-21grep07:40:212
3227099108,1cyclictest9033-21gzip07:25:210
3227099108,1cyclictest8773-21awk08:35:140
3227099108,1cyclictest5674-21ssh10:45:210
3227099108,1cyclictest5229-21dpkg07:20:000
3227099108,1cyclictest4006-21seq09:20:120
3227099108,1cyclictest32317-21expr08:15:180
3227099108,1cyclictest31462-21ssh12:26:570
3227099108,1cyclictest262332sleep008:01:390
3227099108,1cyclictest15883-21wget08:50:140
3227099107,2cyclictest696-21expr07:10:150
3227099107,2cyclictest6641-21cat08:30:170
3227099107,2cyclictest23399-21sshd07:55:260
3227099107,2cyclictest18632-21tr07:45:280
3227099107,2cyclictest15270-21expr07:40:180
3227099107,2cyclictest11196-21cut08:40:120
322789996,2cyclictest21256-21grep07:50:322
322759997,1cyclictest22432-21expr07:55:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional