You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-21 - 22:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jun 21, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27026210376,9sleep10-21swapper/107:08:311
26839210375,9sleep30-21swapper/307:06:053
2703029063,9sleep00-21swapper/007:08:340
2687328267,11sleep20-21swapper/207:06:312
37182620,0sleep10-21swapper/111:50:371
84712560,0sleep20-21swapper/211:54:292
2722299280,28cyclictest0-21swapper/212:03:322
27222992423,1cyclictest0-21swapper/207:28:172
2721999240,24cyclictest0-21swapper/010:28:060
2722299230,23cyclictest0-21swapper/211:41:452
2722299230,23cyclictest0-21swapper/211:17:062
2722299220,22cyclictest0-21swapper/211:23:482
2721999221,2cyclictest0-21swapper/010:55:160
2721999220,17cyclictest0-21swapper/011:15:130
27222992119,1cyclictest19735-21kworker/2:012:22:382
2722299210,21cyclictest0-21swapper/211:59:422
2722299210,20cyclictest0-21swapper/210:12:022
2722299200,20cyclictest0-21swapper/210:33:592
27220992019,1cyclictest0-21swapper/109:54:131
2721999203,12cyclictest7028-21apache_processe09:25:150
2721999200,19cyclictest0-21swapper/010:31:240
2723099191,17cyclictest25447-21cron11:45:003
27222991913,5cyclictest37850irq/131-snd_hda08:40:152
2722299191,1cyclictest141rcu_preempt12:13:542
2722299190,19cyclictest0-21swapper/210:58:132
2722299190,18cyclictest0-21swapper/212:31:472
2721999192,2cyclictest0-21swapper/010:05:180
27219991918,1cyclictest0-21swapper/010:15:420
27219991917,2cyclictest13-21ksoftirqd/010:24:160
2722299180,18cyclictest0-21swapper/210:27:072
2722299180,18cyclictest0-21swapper/209:34:082
2722299180,15cyclictest0-21swapper/210:41:042
2722099185,3cyclictest224-21systemd-journal11:43:571
2723099170,0cyclictest0-21swapper/311:38:403
2722299172,2cyclictest0-21swapper/210:37:482
27222991716,1cyclictest61750irq/125-enp2s0-10:03:132
27222991714,1cyclictest1446-21switchtime07:20:272
2722299170,14cyclictest0-21swapper/211:13:352
2723099161,1cyclictest0-21swapper/312:22:493
27222991615,1cyclictest26277-21kworker/2:109:25:182
27222991615,1cyclictest19079-21kworker/2:211:00:382
2722299160,15cyclictest0-21swapper/210:08:102
2721999162,7cyclictest17179-21awk11:59:590
27219991615,1cyclictest0-21swapper/012:10:150
27219991614,2cyclictest0-21swapper/010:37:240
27230991512,2cyclictest18311-21anacron12:00:213
27230991511,4cyclictest22667-21apache207:30:573
2723099150,2cyclictest0-21swapper/309:15:443
2722299153,11cyclictest16019-21sed09:00:192
27222991514,1cyclictest4949-21kworker/2:011:39:582
27222991514,1cyclictest4596-21kworker/2:012:35:382
27222991514,1cyclictest15944-21kworker/2:209:37:582
27222991514,1cyclictest0-21swapper/208:45:292
2722299150,13cyclictest0-21swapper/212:18:052
2722099154,9cyclictest22654-21apache210:15:081
2722099150,13cyclictest9384-21sh10:45:191
27219991512,2cyclictest12392-21ssh11:13:500
2721999150,1cyclictest0-21swapper/011:34:120
2723099145,6cyclictest7894-21expr11:10:183
27230991413,1cyclictest0-21swapper/311:52:303
27230991412,1cyclictest19751-21diskmemload11:22:393
27230991412,1cyclictest0-21swapper/309:59:053
27230991410,3cyclictest23900-21sh10:05:003
2723099140,14cyclictest0-21swapper/310:26:103
27222991413,1cyclictest24311-21kworker/2:212:25:382
27222991413,1cyclictest0-21swapper/209:47:382
2722299141,13cyclictest12441-21kworker/2:209:10:482
2722299141,12cyclictest579-21ssh11:29:302
2722299140,1cyclictest0-21swapper/210:48:042
2722299140,14cyclictest0-21swapper/210:51:502
2722299140,14cyclictest0-21swapper/210:15:412
2722299140,13cyclictest28471-21taskset11:45:522
2722099148,4cyclictest25774-21head10:05:261
2721999140,11cyclictest0-21swapper/011:02:440
2723099134,7cyclictest21730-21python09:10:273
27230991310,2cyclictest4135-21grep08:35:193
27230991310,2cyclictest3726-21tr10:40:203
27230991310,2cyclictest29443-21head08:20:203
27230991310,2cyclictest2070-21cat09:20:213
27230991310,2cyclictest12830-21grep10:20:273
2723099130,12cyclictest30329-21grep07:15:153
2722299139,3cyclictest6651-21wc09:25:012
27222991312,1cyclictest0-21swapper/209:56:182
27222991311,2cyclictest0-21swapper/211:08:232
27222991310,2cyclictest18969-21mailstats09:05:242
27222991310,2cyclictest0-21swapper/209:50:172
2722299130,1cyclictest19751-21diskmemload12:06:162
2722099139,3cyclictest12305-21tr09:55:201
2722099139,3cyclictest10217-21cron08:49:591
27220991311,2cyclictest25-21ksoftirqd/110:14:241
27220991310,2cyclictest7089-21ssh09:25:161
27220991310,2cyclictest14552-21grep11:15:211
27220991310,2cyclictest1449-21wc07:20:271
27220991310,2cyclictest14012-21tr08:55:241
27220991310,2cyclictest13149-21grep07:45:271
27220991310,2cyclictest10510-21tune2fs11:35:171
2722099130,1cyclictest0-21swapper/109:49:571
2722099130,12cyclictest2983-21cat10:40:011
27219991312,1cyclictest0-21swapper/009:22:080
27219991310,2cyclictest26848-21grep11:45:190
27219991310,2cyclictest18373-21sed07:55:290
2723099129,2cyclictest330-21cut09:45:193
2723099129,2cyclictest30567-21ssh10:35:223
2723099129,2cyclictest28301-21grep07:10:183
2723099129,2cyclictest23751-21wc08:10:003
2723099129,2cyclictest14904-21perf10:49:593
2723099129,2cyclictest14633-21grep11:15:213
2723099129,2cyclictest10879-21mailstats07:40:243
2723099129,1cyclictest0-21swapper/311:45:573
2723099128,3cyclictest447-21munin-run11:05:003
2723099128,2cyclictest700-21latency_hist08:30:003
2723099128,2cyclictest14400-21grep07:50:163
2723099126,2cyclictest278-21systemd-udevd10:32:213
2723099126,2cyclictest224-21systemd-journal12:20:003
2723099125,3cyclictest8124-21apt-get08:45:143
2723099120,12cyclictest0-21swapper/312:34:463
2722299129,2cyclictest4111-21head08:35:202
2722299129,2cyclictest31383-21wc07:15:242
2722299129,2cyclictest28539-21sed08:20:002
2722299129,2cyclictest2021-21tr11:30:162
2722299120,1cyclictest0-21swapper/209:15:192
2722299120,11cyclictest14498-21sed08:55:292
2722099129,2cyclictest6586-21apt09:25:001
2722099129,2cyclictest5063-21head08:35:281
2722099129,2cyclictest31633-21basename09:45:001
2722099129,2cyclictest27926-21grep11:45:281
2722099129,2cyclictest21485-21sed09:35:291
2722099129,2cyclictest20917-21tr11:20:251
2722099129,2cyclictest20210-21grep09:10:141
2722099129,2cyclictest18293-21expr12:00:211
2722099129,2cyclictest15685-21ssh10:50:181
2722099129,2cyclictest13509-21wc11:15:001
2722099129,2cyclictest11688-21sort10:20:181
2722099128,3cyclictest28913-21grep07:10:231
2722099120,12cyclictest0-21swapper/111:33:161
2722099120,11cyclictest31732-21sed07:15:281
2721999129,2cyclictest5494-21grep07:30:200
2721999129,2cyclictest3180-21head07:25:190
2721999129,2cyclictest27820-21grep09:15:220
2721999129,2cyclictest26950-21grep08:15:180
2721999129,2cyclictest22239-21ssh11:21:550
2721999129,2cyclictest17597-21rm11:40:010
2721999129,2cyclictest15895-21ssh12:39:120
2721999129,2cyclictest12952-21ssh10:47:290
2721999129,2cyclictest12422-21ssh09:55:200
2721999129,2cyclictest10682-21strings12:15:230
2721999128,3cyclictest9024-21wc12:35:000
2721999128,3cyclictest29985-21ssh09:43:050
2721999127,2cyclictest15987-21ssh11:38:550
2721999121,8cyclictest27682-21apt-get07:10:150
27219991210,1cyclictest17174-21grep10:50:290
2723099119,1cyclictest4547-21ssh10:15:143
2723099118,2cyclictest9740-21ssh11:55:173
2723099118,2cyclictest8845-21grep07:35:283
2723099118,2cyclictest6999-21ps08:40:243
2723099118,2cyclictest6462-21grep09:50:203
2723099118,2cyclictest30958-21sort10:10:173
2723099118,2cyclictest29015-21ssh12:26:333
2723099118,2cyclictest28632-21ssh11:25:473
2723099118,2cyclictest27983-21sh12:05:383
2723099118,2cyclictest27957-21grep08:15:293
2723099118,2cyclictest26773-21sed09:40:203
2723099118,2cyclictest26216-21sed10:05:323
2723099118,2cyclictest2204-21taskset11:05:233
2723099118,2cyclictest20868-21sh09:35:223
2723099118,2cyclictest16767-21cat09:00:413
2723099118,2cyclictest13647-21tr09:30:203
2723099118,2cyclictest13095-21grep08:55:183
2723099118,2cyclictest12713-21wc08:55:003
2723099118,2cyclictest12029-21ssh12:36:013
2723099118,2cyclictest1188-21ssh12:10:053
2723099118,2cyclictest10223-21ssh09:27:133
2723099117,3cyclictest2131-21expr11:30:173
2723099116,2cyclictest278-21systemd-udevd09:06:213
2723099113,4cyclictest224-21systemd-journal10:55:003
2722299119,1cyclictest27661-21basename07:10:132
2722299119,1cyclictest11454-21wc08:50:212
2722299118,2cyclictest19292-21grep08:00:172
2722299118,2cyclictest11654-21ssh10:20:172
2722299112,2cyclictest0-21swapper/209:40:152
2722099118,2cyclictest9382-21ssh12:15:141
2722099118,2cyclictest7135-21bc07:35:131
2722099118,2cyclictest4113-21ssh12:31:081
2722099118,2cyclictest3651-21ssh12:10:411
2722099118,2cyclictest3459-21ssh10:40:171
2722099118,2cyclictest3026-21ssh11:05:401
2722099118,2cyclictest28692-21tr08:20:171
2722099118,2cyclictest28332-21ssh11:25:321
2722099118,2cyclictest27381-21sh11:00:151
2722099118,2cyclictest25943-21ssh12:25:211
2722099118,2cyclictest25781-21ssh12:05:171
2722099118,2cyclictest23430-21sshd08:07:571
2722099118,2cyclictest20946-21chrt12:21:381
2722099118,2cyclictest20381-21ssh10:26:581
2722099118,2cyclictest19741-21sort08:00:191
2722099118,2cyclictest19434-21ssh10:00:271
2722099118,2cyclictest17100-21ssh09:32:371
2722099118,2cyclictest15003-21grep07:50:191
2722099118,2cyclictest14270-21ssh11:57:391
2722099118,2cyclictest11787-21ssh12:35:491
2722099111,6cyclictest10499-21apt-get08:50:151
2722099110,10cyclictest28782-21sort09:15:311
2721999119,2cyclictest2739-21ssh09:46:520
2721999119,2cyclictest24448-21ssh09:38:500
2721999119,1cyclictest9528-21cat07:40:130
2721999119,1cyclictest77752sleep008:44:100
2721999119,1cyclictest734-21ssh11:05:040
2721999119,1cyclictest5683-21date08:40:010
2721999119,1cyclictest26434-21expr12:05:230
2721999119,1cyclictest2575-21date08:30:270
2721999119,1cyclictest19534-21taskset09:07:280
2721999119,1cyclictest18260-21expr10:00:180
2721999119,1cyclictest17517-21rm12:00:060
2721999119,1cyclictest17364-21ssh12:20:070
2721999119,1cyclictest1516-21ssh11:50:120
2721999119,1cyclictest14670-21expr07:50:170
2721999119,1cyclictest13928-21cat08:55:240
2721999119,1cyclictest13132-21ssh09:30:030
2721999119,1cyclictest10949-21cat08:50:170
2721999118,2cyclictest5495-21taskset09:50:040
2721999118,2cyclictest3730-21ssh10:14:390
2721999118,2cyclictest26387-21basename11:25:170
2721999118,2cyclictest25673-21taskset08:11:140
2721999118,2cyclictest25673-21taskset08:11:130
2721999118,2cyclictest23603-21sh08:10:010
2721999118,2cyclictest22260-21ssh09:10:390
2721999118,2cyclictest1698-21sed07:20:290
2721999118,2cyclictest15841-21df09:00:180
2721999118,2cyclictest13640-21sed07:45:310
2721999110,10cyclictest31154-21basename08:25:160
2723099108,1cyclictest2786-21expr07:25:173
2723099108,1cyclictest24750-21cat08:10:243
2723099108,1cyclictest24750-21cat08:10:243
2723099108,1cyclictest19272sleep307:22:273
2723099108,1cyclictest12177-21tr07:45:193
2723099107,2cyclictest20005-21cut08:00:223
2723099106,3cyclictest16538-21uname10:50:233
2723099102,4cyclictest224-21systemd-journal08:35:003
2722299109,1cyclictest89692sleep207:36:252
2722299109,1cyclictest54792sleep207:30:192
2722299109,1cyclictest32542-21sort08:25:282
2722299109,1cyclictest21567-21cut08:05:172
2722299109,1cyclictest167592sleep207:55:152
2722299108,2cyclictest29027-21ls08:20:182
2722299108,1cyclictest16250-21chrt07:53:112
2722299108,1cyclictest10321-21cut07:40:192
2722299107,2cyclictest24525-21grep08:10:222
2722299107,2cyclictest24525-21grep08:10:222
2722299107,2cyclictest1748-21sed08:30:192
2722299107,2cyclictest12678-21sed07:45:212
2722099108,1cyclictest5919-21grep07:30:231
2722099108,1cyclictest32535-21wc08:25:271
2722099108,1cyclictest2784-21expr07:25:171
2722099108,1cyclictest26920-21cut08:15:181
2722099108,1cyclictest24217-21expr08:10:211
2722099108,1cyclictest24217-21expr08:10:201
2722099108,1cyclictest1979-21sed08:30:211
2722099107,2cyclictest24545-21cut10:30:221
2722099107,2cyclictest21146-21expr10:55:191
2722099107,2cyclictest18107-21expr09:05:171
2722099107,2cyclictest15401-21grep09:00:161
2722099107,2cyclictest10334-21sed07:40:191
2722099100,9cyclictest7683-21chrt08:42:581
2721999109,1cyclictest25872-21basename12:25:210
2721999108,1cyclictest8941-21idleruntime08:45:190
2721999108,1cyclictest31151-21tr07:15:220
2721999108,1cyclictest30907-21grep08:25:010
2721999107,2cyclictest5157-21grep10:40:350
2721999107,2cyclictest19027-21seq08:00:100
2721999101,3cyclictest0-21swapper/007:35:140
272309997,1cyclictest16979-21cat07:55:163
272209997,1cyclictest16864-21wc07:55:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional